Convert a Hex dump to Binary data with xxd

In my last blog post, I demonstarted how to dump a binary file, using either 'hexdump' or 'xxd'.
In summary, we can do this:

# ls -l hash-file.bin

-rw-r--r-- 1 root root 20 Jul  4 10:05 hash-file.bin

# xxd -p hash-file.bin

57443a4c052350a44638835d64fd66822f813319

Ok, what if we want to do the reverse of this.
That is to take a string of hex characters and create a binary file!
The utility 'xxd' can also do this.

To demonstrate, lets first save the hex representation into a separate file:

# xxd -p hash-file.bin > hash-file.hex

# ls -l hash-file.hex

-rw-r--r-- 1 root root 41 Jul  4 12:04 hash-file.hex

# cat hash-file.hex

57443a4c052350a44638835d64fd66822f813319

Ok, here's the clever part. Note we now pass the '-r' option to xdd to 'reverse' it's operation.

# xxd -p -r hash-file.hex > new-file.bin

# ls -l new-file.bin

-rw-r--r-- 1 root root 20 Jul  4 12:19 new-file.bin

# xxd -p new-file.bin

57443a4c052350a44638835d64fd66822f813319

# cmp hash-file.bin new-file.bin

# 

So we recreated the binary file, as 'new-file.bin' from the hex dump and we proved it is identical to the file we started with.

You can also feed into xdd from the pipeline, like this:

# cat hash-file.hex | xxd -p -r > new-file.bin

# xxd -p new-file.bin

57443a4c052350a44638835d64fd66822f813319

..and here are some more examples:

# echo -n 'password' | xxd -p | xxd -p -r | hexdump -C
00000000  70 61 73 73 77 6f 72 64                           |password|
00000008

# echo -n 'c0a06003' | xxd -p -r | hexdump -C
00000000  c0 a0 60 03                                       |..`.|
00000004

hexdump and xxd output compared

Here I'm working with a Centos Linux system, and I've found two seperate utility programs that can produced a hex dump. One is called hexdump and the other is xxd, and they are provided as part of two seperate packages.

# which hexdump

/usr/bin/hexdump

# rpm -qf /usr/bin/hexdump

util-linux-2.13-0.59.el5



# which xxd

/usr/bin/xxd

# rpm -qf /usr/bin/xxd

vim-common-7.0.109-7.el5

Ok, so lets compare the output format of the hex dump with these utilities. But first we need some test data, which I'm going to generate with 'openssl'. Conveniently, openssl can output the data in either hex (the default) or binary.

# echo -n 'value' | openssl dgst -sha1 -hmac "key"

57443a4c052350a44638835d64fd66822f813319

# echo -n 'value' | openssl dgst -sha1 -hmac "key" -binary > hash-file.bin

# ls -l hash-file.bin

-rw-r--r-- 1 root root 20 Jul  4 10:05 hash-file.bin

So we have our test data in a file called 'hash-file.bin'. It's 20 bytes long as it contains a 160 bit sha1 hash. Ok, let's try dumping that file and see what we get:

# hexdump hash-file.bin
0000000 4457 4c3a 2305 a450 3846 5d83 fd64 8266
0000010 812f 1933
0000014

# xxd hash-file.bin
0000000: 5744 3a4c 0523 50a4 4638 835d 64fd 6682  WD:L.#P.F8.]d.f.
0000010: 2f81 3319                                /.3.

Um, interesting!
Note the default output is grouping the bytes into 16-bit values and there is a difference in how to interpret the endianness.

Let's try customizing the output with some options:

# hexdump -C hash-file.bin
00000000  57 44 3a 4c 05 23 50 a4  46 38 83 5d 64 fd 66 82  |WD:L.#P.F8.]d.f.|
00000010  2f 81 33 19                                       |/.3.|
00000014

# xxd -g 1 hash-file.bin
0000000: 57 44 3a 4c 05 23 50 a4 46 38 83 5d 64 fd 66 82  WD:L.#P.F8.]d.f.
0000010: 2f 81 33 19                                      /.3.

# xxd -p hash-file.bin
57443a4c052350a44638835d64fd66822f813319

So this is 'better' for seeing the byte sequence.

Openssl and the "(stndin)= " prefix

Recently, I've been playing around with the openssl utility. Doing things like this:

# echo -n 'value' | openssl dgst -sha1 -hmac 'key'

57443a4c052350a44638835d64fd66822f813319



# rpm -qa | grep openssl

openssl-0.9.8e-22.el5_8.3



# cat /etc/redhat-release

CentOS release 5.8 (Final)

Note that I show the version and operating system I'm using. Here's why. Look at the following:

# echo -n 'value' | openssl dgst -sha1 -hmac 'key'

(stdin)= 57443a4c052350a44638835d64fd66822f813319



# rpm -qa | grep openssl

openssl-1.0.0-20.el6_2.5.x86_64



# cat /etc/redhat-release

CentOS release 6.2 (Final)

So on a more recent system, openssl has prefixed it's output with "(stndin)= ".
I'm not finding this particularly helpful for my application.
One workaround, suggested here, is to use sed to remove it, like this:

# echo -n 'value' | openssl dgst -sha1 -hmac 'key' | sed 's/^.*= //'

57443a4c052350a44638835d64fd66822f813319

Comparing RPM packages installed on two hosts, without using temporary files

When setting up a new Linux server, it's often interesting to compare the list of packages that are installed on the new server, with the list of packages installed on an existing server. You can use the following command line, which makes use of Bash supports for process substitution, to show the difference between packages installed on the local host and on the remote host.

 # ssh root@remotehost 'rpm -qa | sort' | diff -u <(rpm -qa | sort) -

If the above command gives no output, it means that the two hosts have identical packages installed.

If you want to disregard the package version differences in the comparison, then you will need to use something like this:

 # ssh root@remotehost 'rpm -qa --queryformat "%{NAME}\n" | sort' | diff -u <(rpm -qa --queryformat "%{NAME}\n" | sort) -

Update(2013-07-02):
Following Paul Waterman's comment, I did try out his rpmscomp Perl script, and I did find it useful. So I would recommend you also give it a try:

https://github.com/pdwaterman/rpmscomp

Also, to help in removing packages, I have found the rpmreader package useful:

https://fedorahosted.org/rpmreaper/

Moving old Nokia ringtones to the Apple iPhone

For the last seven years, I have used the 'bach' ringtone on my old Nokia 6230i. Now I have an iPhone 4s, I wanted the option of using the same ring tone. What follows is the procedure I used. I did this all on a Windows PC, but I believe you can also use 'faac' on Linux. Note that this procedure is 'free'.

I download the Nokia 6230i ringtone "Bach.aac" from here:

http://meramob.com/filedownload/2011/41824/Bach.html

I used this online service...

http://media.io/

... to convert the "Bach.aac" to "Bach.wav".

I then downloaded "FAACv1.28 Binary for Win32" as file "faac-1.28-mod.zip" from this site:

http://www.rarewares.org/aac-encoders.php

..and unziped the archive to extract binary executable "faac.exe".

Here is the 'help' for faac:

C:\Temp>faac.exe --help

Freeware Advanced Audio Coder

FAAC 1.28



Usage: faac.exe [options] infiles ...

Options:

  -q   Set quantizer quality.

  -b   Set average bitrate to x kbps. (ABR, lower quality mode)

  -c      Set the bandwidth in Hz. (default=automatic)

  -o X          Set output file to X (only for one input file)

  -r            Use RAW AAC output file.

  -P            Raw PCM input mode (default 44100Hz 16bit stereo).

  -R            Raw PCM input rate.

  -B            Raw PCM input sample size (8, 16 (default), 24 or 32bits).

  -C            Raw PCM input channels.

  -X            Raw PCM swap input bytes

  -I      Input channel config, default is 3,4 (Center third, LF fourth)



MP4 specific options:

  -w            Wrap AAC data in MP4 container. (default for *.mp4 and *.m4a)

  -s            Optimize MP4 container layout after encoding

  --artist X    Set artist to X

  --writer X    Set writer to X

  --title X     Set title to X

  --genre X     Set genre to X

  --album X     Set album to X

  --compilation Set compilation

  --track X     Set track to X (number/total)

  --disc X      Set disc to X (number/total)

  --year X      Set year to X

  --cover-art X Read cover art from file X

  --comment X   Set comment to X



Documentation:

  --license     Show the FAAC license.

  --help        Show this abbreviated help.

  --long-help   Show complete help.



  More tips can be found in the audiocoding.com Knowledge Base at

  http://www.audiocoding.com/wiki/

I then used this command line to convert the WAV file to a M4A file:

C:\Temp>faac.exe -b 128 -c 44100 -w Bach.wav

Freeware Advanced Audio Coder

FAAC 1.28



Average bitrate: 128 kbps

Quantization quality: 100

Bandwidth: 16000 Hz

Object type: Low Complexity(MPEG-4) + M/S

Container format: MPEG-4 File Format (MP4)

Encoding Bach.wav to Bach.m4a

   frame          | bitrate | elapsed/estim | play/CPU | ETA

  403/403   (100%)|   74.8  |    0.4/0.4    |   29.51x | 0.0

Note that I used the "-w" option so that the AAC data stream was 'wrapped' in a MP4 container.

I then renamed "Bach.m4a" to "Bach.m4r".
(Note the M4R extension is used to identify the file as a ringtone.)

And I then dragged and dropped the "Bach.m4r" file into the iTunes library.
(If iTunes fails to accept the dropped file, it means that file is not in the correct format.)

Then I synced the ringtone to the iPhone.

The following pages provided inspiration for the above:

• Create Ringtones For Your iPhone Using Linux
• (YouTube) How to Put Free Ringtones On Your Iphone 4S - EASY MODE

Query your Switch using SNMP and Linux

So you want to query a switch using snmp to see which MACs are being seen on which ports. In that case you will need the BRIDGE-MIB. On Centos-5 it is supplied, but as part of the 'libsmi' package.

# rpm -qa | egrep 'libsmi|net-snmp' | sort
libsmi-0.4.5-2.el5
net-snmp-5.3.2.2-14.el5_7.1
net-snmp-libs-5.3.2.2-14.el5_7.1
net-snmp-utils-5.3.2.2-14.el5_7.1

So you need to create file '/etc/snmp/snmp.conf' and configure as follows:

# cat /etc/snmp/snmp.conf
mibdirs +/usr/share/mibs/ietf

Then you should have success with the following command.
Remember to use the correct community string - by default its usually 'public'.
And the mib object name IS case sensitive.

# snmptable -v1 -c public -Cbw 80 192.168.24.1 -IR BRIDGE-MIB::dot1dTpFdbTable
SNMP table: BRIDGE-MIB::dot1dTpFdbTable

           Address Port  Status
   0:c:29:45:21:9e   24 learned
   0:c:29:79:6c:73   24 learned
  0:12:3f:c3:9d:7b   19 learned
  0:14:38:a2:70:7e   15 learned
  0:14:5e:68:74:ed   17 learned
  0:1b:78:22:bc:30   21 learned
  0:26:55:e7:5b:f4   27 learned
  0:50:56:a7:61:32   22 learned
  64:31:50:cd:c7:0    0    self
 9c:8e:99:19:99:a6   23 learned
 9c:8e:99:c4:46:56   13 learned
  c8:4c:75:90:f:e0   28 learned

Status 'self' or 'mgmt' is the MAC of the switch itself - used for the management connection.

If you have a modular switch, you may want to check the logical to physical port name mapping, as show next. If you just have a simple switch, the port names are usually boring!

# snmpwalk -v1 -c public 192.168.24.1 -IR IF-MIB::ifName | head -n 5
IF-MIB::ifName.1 = STRING: 1
IF-MIB::ifName.2 = STRING: 2
IF-MIB::ifName.3 = STRING: 3
IF-MIB::ifName.4 = STRING: 4
IF-MIB::ifName.5 = STRING: 5

Before you run the above commands, it may be useful to ping every host on your subnet, just so that the switch has seen everything. You can use 'nmap' for that:

# nmap -n -sP 192.168.24.0/24

Good luck with the above. I've often found these command very useful!

bash initialisation files

I've been installing Centos 5 into VirtualBox this evening. Due to some mistyping, I ended up in the situation where I needed to manually create the 'home' directory for the user 'nwsmith'. Ok, thats simple, just remember to set the correct user and group, and the correct permissions:

# mkdir /home/nwsmith
# chown nwsmith.nwsmith /home/nwsmith
# chmod 700 /home/nwsmith

But when I logged in as user 'nwsmith' I was not getting the correct bash prompt. It looked like PS1 was not being set correctly. I quickly realised that I was missing the '.bashrc' file from my home directory, so I copied that across from '/etc/skel/.bashrc'.

But still I was not getting the correct prompt. But I found that if I ran the command 'source ./.bashrc' then I did get the correct prompt. Finally the penny dropped, when I googled up this link Initialisation files and configuration from the University of Cambridge Computer Laboratory. I'm just going to quote the paragraph I found really useful:

When bash is invoked as a login shell it first reads and executes commands from the file /etc/profile, if that file exists. This initialises a few environment variables, and calls a set of small initialisation scripts from the directory /etc/profile.d, which will vary depending on the software loaded on the machine (for example, if KDE is loaded there will be one called kde.sh).

After reading that file, it looks for ~/.bash_profile or ~/.bash_login or ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. (The --noprofile option could be used when the shell is started to inhibit this behaviour.)

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. (This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc).

Thus, by default, the file ~/.bashrc is not read automatically when a login shell is started. For this reason it is usual behaviour to add a section to ~/.profile (or ~/.bash_profile or ~/.bash_login) to read and execute it, so that you get uniform behaviour on both login shells and normal interactive shells.

After reading that I realised that I also need a '.bash_profile' file in my home directory, so I copied one across from '/etc/skel/.bash_profile'. And then when I logged in as user 'nwsmith' I got the prompt I was expecting.

By the way, '.bash_profile' and '.bashrc' do NOT need to have 'x' executable permission.