Nigel Smith's Blog

Query your Switch using SNMP and Linux

2011-11-30T17:10:00.003Z

So you want to query a switch using snmp to see which MACs are being seen on which ports. In that case you will need the BRIDGE-MIB. On Centos-5 it is supplied, but as part of the 'libsmi' package.


# rpm -qa | egrep 'libsmi|net-snmp' | sort
libsmi-0.4.5-2.el5
net-snmp-5.3.2.2-14.el5_7.1
net-snmp-libs-5.3.2.2-14.el5_7.1
net-snmp-utils-5.3.2.2-14.el5_7.1

So you need to create file '/etc/snmp/snmp.conf' and configure as follows:


# cat /etc/snmp/snmp.conf
mibdirs +/usr/share/mibs/ietf

Then you should have success with the following command.
Remember to use the correct community string - by default its usually 'public'.
And the mib object name IS case sensitive.


# snmptable -v1 -c public -Cbw 80 192.168.24.1 -IR BRIDGE-MIB::dot1dTpFdbTable
SNMP table: BRIDGE-MIB::dot1dTpFdbTable

           Address Port  Status
   0:c:29:45:21:9e   24 learned
   0:c:29:79:6c:73   24 learned
  0:12:3f:c3:9d:7b   19 learned
  0:14:38:a2:70:7e   15 learned
  0:14:5e:68:74:ed   17 learned
  0:1b:78:22:bc:30   21 learned
  0:26:55:e7:5b:f4   27 learned
  0:50:56:a7:61:32   22 learned
  64:31:50:cd:c7:0    0    self
 9c:8e:99:19:99:a6   23 learned
 9c:8e:99:c4:46:56   13 learned
  c8:4c:75:90:f:e0   28 learned

Status 'self' or 'mgmt' is the MAC of the switch itself - used for the management connection.

If you have a modular switch, you may want to check the logical to physical port name mapping, as show next. If you just have a simple switch, the port names are usually boring!


# snmpwalk -v1 -c public 192.168.24.1 -IR IF-MIB::ifName | head -n 5
IF-MIB::ifName.1 = STRING: 1
IF-MIB::ifName.2 = STRING: 2
IF-MIB::ifName.3 = STRING: 3
IF-MIB::ifName.4 = STRING: 4
IF-MIB::ifName.5 = STRING: 5

Before you run the above commands, it may be useful to ping every host on your subnet, just so that the switch has seen everything. You can use 'nmap' for that:


# nmap -n -sP 192.168.24.0/24

Good luck with the above. I've often found these command very useful!

bash initialisation files

2011-08-05T00:35:00.003+01:00

I've been installing Centos 5 into VirtualBox this evening. Due to some mistyping, I ended up in the situation where I needed to manually create the 'home' directory for the user 'nwsmith'. Ok, thats simple, just remember to set the correct user and group, and the correct permissions:


# mkdir /home/nwsmith
# chown nwsmith.nwsmith /home/nwsmith
# chmod 700 /home/nwsmith

But when I logged in as user 'nwsmith' I was not getting the correct bash prompt. It looked like PS1 was not being set correctly. I quickly realised that I was missing the '.bashrc' file from my home directory, so I copied that across from '/etc/skel/.bashrc'.

But still I was not getting the correct prompt. But I found that if I ran the command 'source ./.bashrc' then I did get the correct prompt. Finally the penny dropped, when I googled up this link Initialisation files and configuration from the University of Cambridge Computer Laboratory. I'm just going to quote the paragraph I found really useful:

When bash is invoked as a login shell it first reads and executes commands from the file /etc/profile, if that file exists. This initialises a few environment variables, and calls a set of small initialisation scripts from the directory /etc/profile.d, which will vary depending on the software loaded on the machine (for example, if KDE is loaded there will be one called kde.sh).

After reading that file, it looks for ~/.bash_profile or ~/.bash_login or ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. (The --noprofile option could be used when the shell is started to inhibit this behaviour.)

When an interactive shell that is not a login shell is started, bash reads and executes commands from ~/.bashrc, if that file exists. (This may be inhibited by using the --norc option. The --rcfile file option will force bash to read and execute commands from file instead of ~/.bashrc).

Thus, by default, the file ~/.bashrc is not read automatically when a login shell is started. For this reason it is usual behaviour to add a section to ~/.profile (or ~/.bash_profile or ~/.bash_login) to read and execute it, so that you get uniform behaviour on both login shells and normal interactive shells.

After reading that I realised that I also need a '.bash_profile' file in my home directory, so I copied one across from '/etc/skel/.bash_profile'. And then when I logged in as user 'nwsmith' I got the prompt I was expecting.

By the way, '.bash_profile' and '.bashrc' do NOT need to have 'x' executable permission.

selinux, ipsec-tools and '/etc/racoon/racoon.conf'

2011-07-22T09:49:00.003+01:00

The racoon daemon failed to start, and this message was logged:


  # grep denied /var/log/messages
Jul 21 17:57:56 hexgate kernel: type=1400 audit(1311267452.973:4): avc:  denied  { getattr } for  pid=2412 comm="racoon" path="/etc/racoon/racoon.conf" dev=md0 ino=1401757 scontext=system_u:system_r:racoon_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

So I guess I had got the wrong security context on the file. To fix it, I did this:


  # ls -lZ /etc/racoon/racoon.conf
  -rw-------  root root root:object_r:user_home_t        /etc/racoon/racoon.conf
  #
  # restorecon -v /etc/racoon/racoon.conf
  restorecon reset /etc/racoon/racoon.conf context root:object_r:user_home_t:s0->system_u:object_r:ipsec_conf_file_t:s0
  #
  # ls -lZ /etc/racoon/racoon.conf
  -rw-------  root root system_u:object_r:ipsec_conf_file_t /etc/racoon/racoon.conf
  #

..and then racoon started Ok.
So what other file security context could be relevant to racoon:


  # grep racoon /etc/selinux/targeted/contexts/files//file_contexts
  /etc/racoon(/.*)?       system_u:object_r:ipsec_conf_file_t:s0
  /var/racoon(/.*)?       system_u:object_r:ipsec_var_run_t:s0
  /var/run/racoon.pid     --      system_u:object_r:ipsec_var_run_t:s0
  /etc/racoon/certs(/.*)? system_u:object_r:ipsec_key_file_t:s0
  /usr/sbin/racoon        --      system_u:object_r:racoon_exec_t:s0
  /etc/racoon/psk\.txt    --      system_u:object_r:ipsec_key_file_t:s0
  #

By the way, this was on the following system:


  # cat /etc/redhat-release
  CentOS release 5.5 (Final)
  #
  # rpm -qa | grep selinux-policy
  selinux-policy-2.4.6-279.el5_5.2
  selinux-policy-targeted-2.4.6-279.el5_5.2

Installing TweetDeck with Adobe AIR on Fedora 15

2011-06-30T23:38:00.008+01:00

Unfortunately, there is a problem installing TweakDeck on Fedora 15, and even more unfortunately, it does not look like it will be fixed.

Using the 'TechnoZeal' instructions, I had no difficulty installing Adobe AIR on Fedora 15 (64-bit), although AIR seems to be a 32-bit application and so you will need to install many 32-bit library files.


[nwsmith@fed15-x64 ~]$ rpm -qa | grep -i adobe
adobe-release-i386-1.0-1.noarch
adobeair-2.6.0-19170.i386

I downloaded the latest TweakDeck '.air' file:


[nwsmith@fed15-x64 Downloads]$ ll
-rw-rw-r--. 1 nwsmith nwsmith 2373967 Jun 30 20:46 TweetDeck_0_37.6.air
[nwsmith@fed15-x64 Downloads]$ md5sum -b TweetDeck_0_37.6.air 
d8dc9c0fbff37c31c8af6a788854cb25 *TweetDeck_0_37.6.air

..but when you try to install it, it aborts with error message:


Sorry, an error has occured.
The application could not be installed because the installer file is
damaged. Try obtaining a new installer file from the application author.

Eventually I discovered that the 'Adobe AIR Application Installer' was recording a log file:


[nwsmith@fed15-x64 ~]$ cd /home/nwsmith/.appdata/Adobe/AIR/Logs/
[nwsmith@fed15-x64 Logs]$ ll
-rw-rw-r--. 1 nwsmith nwsmith 83863 Jun 30 21:41 Install.log
[nwsmith@fed15-x64 Logs]$ cat Install.log 
[Adobe AIR Application Installer:4156][INFO] Application Installer begin with version 2.6.0.19170 on Linux x86
[Adobe AIR Application Installer:4156][INFO] Commandline is: 
[Adobe AIR Application Installer:4156][INFO] Installed runtime (2.6.0.19170) located at /opt/Adobe AIR
[Adobe AIR Application Installer:4156][INFO] Cancel request received
[Adobe AIR Application Installer:4156][INFO] Application Installer end with exit code 6
[Adobe AIR Application Installer:4192][INFO] Application Installer begin with version 2.6.0.19170 on Linux x86
[Adobe AIR Application Installer:4192][INFO] Commandline is: 
[Adobe AIR Application Installer:4192][INFO] Installed runtime (2.6.0.19170) located at /opt/Adobe AIR
[Adobe AIR Application Installer:4192][INFO] Unpackaging file:///home/nwsmith/Downloads/TweetDeck_0_37.6.air to /tmp/FlashTmp.NhdYfW
[Adobe AIR Application Installer:4192][INFO] Application signature verified
[Adobe AIR Application Installer:4192][INFO] Unpackaging/validation complete
[Adobe AIR Application Installer:4192][INFO] No app located for appID 'TweetDeckFast' and pubID 'FFF259DC0CE2657847BBB4AFF0E62062EFC56543.1'
[Adobe AIR Application Installer:4192][INFO] Converting unpackaged application to a native installation package in /tmp/FlashTmp.wkEIg0
[Adobe AIR Application Installer:4192][ERR] Native installation package creation failed: [ErrorEvent type="error" bubbles=false cancelable=false eventPhase=2 text="Unhandled exception Error: Error creating the package /opt/Adobe AIR/Versions/1.0/Resources/rpmbuilder error : symbol not found: parseSpec, /usr/lib/librpmbuild.so: undefined symbol: parseSpec error :
<-- SNIP-->
File not found: /tmp/FlashTmp.wkEIg0/build/usr/share/app-install/desktop/tweetdeckfast.fff259dc0ce2657847bbb4aff0e62062efc56543.1.desktop" errorID=1]
[Adobe AIR Application Installer:4192][ERR] Application Installer end with exit code 7

So Googling on "symbol not found: parseSpec" we find these links:

Problem with air application installer on Fedora 15 linux

Bug 692381 - missing parseSpec in librpmbuild.so

...where Panu Matilainen summaries:


"Yes, the entire librpmbuild API has changed in rpm 4.9.x and parseSpec() no
longer exists, it's been replaced by a new saner interface known as
rpmSpecParse(). Air (whatever it is, apparently some Adobe proprietary app)
appears to be cheating against soname changes by using the non-versioned -devel
.so symlink and now gets burned because of it. It'd be much better off
launching rpmbuild than using the librpmbuild API.

And yes, Fedora 15 is using rpm 4.9.x:


[nwsmith@fed15-x64 ~]$ rpm -qa | grep ^rpm | sort
rpm-4.9.0-9.fc15.x86_64
rpm-build-4.9.0-9.fc15.x86_64
rpm-build-libs-4.9.0-9.fc15.i686
rpm-build-libs-4.9.0-9.fc15.x86_64
rpm-devel-4.9.0-9.fc15.i686
rpm-devel-4.9.0-9.fc15.x86_64
rpm-libs-4.9.0-9.fc15.i686
rpm-libs-4.9.0-9.fc15.x86_64
rpm-python-4.9.0-9.fc15.x86_64

So it looks like Adobe need to fix it. Unfortunately, we read here that:


"Note: Beginning June 14 2011, Adobe AIR is no longer supported for desktop Linux distributions."

So I've given up on this. A pity as it would have been really nice to run TweetDeck on Fedora Linux. Anyway now I guess I can remove all those 32-bit libraries I just install, and revert to a 'clean' 64-bit environment.

Status Update: 2011-07-15
Please read Sérgio Basto's comments to this post. Sérgio seems to have found a way around the problem, using Mock. I've not yet tried this for myself, but I will give it a try when I have a few minutes to spare. If anyone else tried this method, please post a comment with your result.

Status Update: 2011-07-22
Chris Beckenbach is reporting success with Sergio's method.

DTrace for Linux: Progress, but still a work in progress

2011-06-14T23:21:00.006+01:00

A couple of months back, I blog posted on Paul Fox's port of DTrace to Linux. I've continued to play around with it since then, and so I think its about time for another blog post about it.

Certainly now there is no problem compiling DTrace for Linux on Fedora 14 with the 2.6.35 kernel, provided you install the necessary dependencies that I listed in the previous post. Since my initial report, I've exchanged many emails with Paul, and he had been able to incorporate in the code the necessary fixes to support compiling on Fedora.

Paul has spent a lot of time recently on the cross-call (aka xcall) problem, and you can read the details on Paul's blog. Certainly this is a challenging area, and although there are still some issues, it looks to me like Paul is determined to fully resolve the issues.

Ok, so let's give DTrace for Linux a bit of a work out and discover a little bit more about it.

After compiling, with 'make all' and installing with 'make install', you load
the DTrace module, with 'make load'.


# uname -a
Linux fed14-64 2.6.35.13-91.fc14.x86_64 #1 SMP Tue May 3 13:23:06 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
# make load
tools/load.pl
21:41:23 Syncing...
21:41:24 Loading: build/driver/dtracedrv.ko
21:41:25 Preparing symbols...
21:41:25 Probes available: 319433
21:41:30 Time: 7s

So you can see there are a lot of probes available. Let's look at a break down:


# dtrace -l | awk '{ print $2 }' | sort | uniq -c | sort -n
      1 PROVIDER
      3 dtrace
      4 io
     17 profile
   1270 syscall
  63808 fbt
 254330 instr

You can see that most of the probes come from the 'instr' provider, which I believe is unique to the Linux port of DTrace. And note that the number of providers available is less than what is available on Solaris and FreeBSD.

Let's list the probes, where there are only a small number of probes for the providers:


[root@fed14-64 ~]# dtrace -l -n dtrace:::
   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
[root@fed14-64 ~]# dtrace -l -n io:::
   ID   PROVIDER            MODULE                          FUNCTION NAME
    4         io      do_sync_read                                   start
    5         io      do_sync_read                                   done
    6         io     do_sync_write                                   start
    7         io     do_sync_write                                   done
[root@fed14-64 ~]# dtrace -l -n profile:::
   ID   PROVIDER            MODULE                          FUNCTION NAME
254912    profile                                                     profile-97
254913    profile                                                     profile-199
254914    profile                                                     profile-499
254915    profile                                                     profile-997
254916    profile                                                     profile-1999
254917    profile                                                     profile-4001
254918    profile                                                     profile-4999
254919    profile                                                     tick-1
254920    profile                                                     tick-10
254921    profile                                                     tick-100
254922    profile                                                     tick-500
254923    profile                                                     tick-1000
254924    profile                                                     tick-5000
319429    profile                                                     tick-1ms
319430    profile                                                     tick-1s
319431    profile                                                     tick-5s
319432    profile                                                     tick-10s

Now let's give a breakdown of the number of probes available for each modules for the FBT (Function Boundary Tracing) provider. This will give you an idea of what kernel modules I have loaded on my test system.


[root@fed14-64 dtrace-20110404]# dtrace -l -P fbt | awk '{print $3}' | sort | uniq -c | sort -n
      1 MODULE
     12 freq_table
     13 output
     14 speedstep_lib
     15 iTCO_vendor_support
     15 p4_clockmod
     23 i2c_i801
     29 ppdev
     30 i2c_algo_bit
     31 iTCO_wdt
     33 snd_page_alloc
     37 snd_seq_device
     37 soundcore
     41 dcdbas
     41 snd_hwdep
     61 sata_sil24
     63 microcode
     79 snd_hda_intel
     83 parport_pc
     99 snd_timer
    100 drm_kms_helper
    109 video
    138 scsi_transport_spi
    141 i2c_core
    155 parport
    199 snd_hda_codec_idt
    337 ttm
    357 snd
    357 tg3
    359 snd_seq
    420 fuse
    424 aic7xxx
    490 snd_hda_codec
    495 snd_pcm
    744 e1000e
    867 drm
   1671 ipv6
   1875 nouveau
  52615 kernel

As you can see from the above, I have two different network interface cards in my system. One is a Broadcom, supported by the 'tg3' module, and the other is an Intel card, supported by the 'e1000e' module.

Lets list the probes available for one of the FBT modules:


# dtrace -l -n fbt:p4_clockmod::entry
   ID   PROVIDER            MODULE                          FUNCTION NAME
310254        fbt       p4_clockmod                    cpufreq_p4_get entry
310256        fbt       p4_clockmod                  cpufreq_p4_setdc entry
310258        fbt       p4_clockmod                 cpufreq_p4_target entry
310260        fbt       p4_clockmod                 cpufreq_p4_verify entry
310262        fbt       p4_clockmod               cpufreq_p4_cpu_init entry
310264        fbt       p4_clockmod                   cpufreq_p4_exit entry
310266        fbt       p4_clockmod               cpufreq_p4_cpu_exit entry
310268        fbt       p4_clockmod                    cleanup_module entry
# dtrace -l -n fbt:p4_clockmod::return
   ID   PROVIDER            MODULE                          FUNCTION NAME
310255        fbt       p4_clockmod                    cpufreq_p4_get return
310257        fbt       p4_clockmod                  cpufreq_p4_setdc return
310259        fbt       p4_clockmod                 cpufreq_p4_target return
310261        fbt       p4_clockmod                 cpufreq_p4_verify return
310263        fbt       p4_clockmod               cpufreq_p4_cpu_init return
310265        fbt       p4_clockmod                   cpufreq_p4_exit return
310267        fbt       p4_clockmod               cpufreq_p4_cpu_exit return

Ok, enough of listing probes! Let show some simple DTrace one-liners in action on Linux:


# dtrace -n 'syscall:::entry { @num[execname] = count(); }'
dtrace: description 'syscall:::entry ' matched 635 probes
^C
  audispd                                                           1
  udisks-daemon                                                     6
  rtkit-daemon                                                      8
  gnome-settings-                                                   9
  abrtd                                                            10
  console-kit-dae                                                  10
  hald-addon-acpi                                                  12
  sendmail                                                         14
  gnome-screensav                                                  15
  sshd                                                             17
  crond                                                            18
  dbus-daemon                                                      18
  gnome-panel                                                      33
  gnome-power-man                                                  38
  clock-applet                                                     49
  gvfs-afc-volume                                                  60
  metacity                                                         68
  irqbalance                                                      120
  nautilus                                                        157
  gedit                                                           362
  dtrace                                                         1553
  Xorg                                                          10039
  firefox                                                       12083
#
# dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'
dtrace: description 'syscall:::entry ' matched 635 probes
^C
  rt_sigreturn                                                      1
  brk                                                               2
  getdents                                                          2
  select                                                            6
  stat                                                              8
  nanosleep                                                        11
  rt_sigaction                                                     16
  munmap                                                           18
  close                                                            21
  fstat                                                            21
  mmap                                                             21
  inotify_add_watch                                                26
  open                                                             32
  rt_sigprocmask                                                   32
  ioctl                                                            49
  write                                                           195
  poll                                                            525
  read                                                            736
  futex                                                          1283
#
# dtrace -n 'fbt:e1000e::entry { @num[probefunc] = count(); }'
dtrace: description 'fbt:e1000e::entry ' matched 373 probes
^C
  e1000_clean                                                      17
  e1000_clean_rx_irq                                               17
  e1000_clean_tx_irq                                               17
  e1000_desc_unused.clone.26                                       17
  e1000_intr_msi                                                   17
  e1000_irq_enable                                                 17
  e1000_watchdog                                                   17
  e1000_watchdog_task                                              17
  e1000e_get_laa_state_82571                                       17
  e1000e_has_link                                                  17
  e1000e_update_adaptive                                           17
  e1000e_update_stats                                              17
  napi_schedule_prep                                               17
  test_and_set_bit                                                 17
  e1000_ioctl                                                      34
  e1000_get_stats                                                 411

Here is the 'instr' provider in action:


# dtrace -n 'instr:e1000e:: { @num[probefunc] = count(); }'
dtrace: description 'instr:e1000e:: ' matched 3472 probes
^C

  e1000_get_drvinfo-je                                              1
  e1000_get_drvinfo-jne                                             1
  e1000e_get_sset_count-ja                                          3
  e1000_get_drvinfo-callr                                           5
  e1000_clean_rx_irq-callr                                         27
  e1000_clean_rx_irq-je                                            27
  e1000_desc_unused.clone.26-jbe                                   27
  e1000_irq_enable-je                                              27
  e1000_watchdog-callr                                             27
  e1000e_get_laa_state_82571-jne                                   27
  e1000e_update_adaptive-jbe                                       27
  e1000e_update_adaptive-je                                        27
  e1000e_update_adaptive-jne                                       27
  e1000e_update_stats-jbe                                          27
  napi_schedule_prep-callr                                         27
  napi_schedule_prep-jne                                           27
  test_and_set_bit-lock                                            27
  e1000_clean-jne                                                  54
  e1000_intr_msi-callr                                             54
  e1000_intr_msi-je                                                54
  e1000e_has_link-je                                               54
  e1000e_update_stats-ja                                           54
  e1000_clean-callr                                                81
  e1000_clean-je                                                   81
  e1000_watchdog_task-jne                                          81
  e1000_clean_tx_irq-je                                           108
  e1000_watchdog_task-je                                          108
  e1000_watchdog_task-callr                                       189

One of the aspects that Paul has been developing recently is a suite of test programs that can be used to stress test the system, and expose any problems with the DTrace for Linux port. To try this out, run 'make test'.

Ok, I'm out of time for tonight. Hopefully I can find time for some more post to this blog soon.

Trying DTrace for Linux on Fedora 14

2011-03-31T22:26:00.028+01:00

Some months, or even years ago, I noticed that Paul Fox, the author of the CRiSP editor, was working on a port of DTrace to Linux. Paul seems to have started work on this port back at the start of 2008. He has made regular progress reports on his website blog, and from June 2010, he also started using Google's blogger, which did seem to have taken over as Paul's main blog, but now I notice he has put all the blogger posts back on his website blog!

In February 2011, I finally go around to trying out the code.

Paul seems to be developing on/for Ubuntu, but I wanted to try it on Fedora 14. I had a number of problems, first indentifying which RPM packages I needed to install for the build. And then there were various compile problems, and it took me a while to figure out how to resolve these problems, as I would not claim to be any sort of expert on this.

Here is a list of the RPM packages I needed to install to get the code to compile:


binutils-devel
elfutils-libelf-devel
kernel-devel
libdwarf
libdwarf-devel
libdwarf-static
zlib-devel

I was eventually able to work out the necessary patches to get it to compile on Fedora 14. I did try emailing Paul to give him some feedback, and I think he may have read my emails, but so far I have never received a reply back from him!

Here for posterity, I publish my patch, just to prove I did some work! But note that this is already out of date, as Paul's DTrace for Linux port has continued to moved on.

$ diff -uNr dtrace-20110215 dtrace-20110215.fed14/
diff -uNr dtrace-20110215/cmd/ctfconvert/dwarf.c dtrace-20110215.fed14/cmd/ctfconvert/dwarf.c
--- dtrace-20110215/cmd/ctfconvert/dwarf.c      2010-08-24 23:34:26.000000000 +0100
+++ dtrace-20110215.fed14/cmd/ctfconvert/dwarf.c        2011-03-18 01:19:57.786335906 +0000
@@ -88,9 +88,9 @@
 #include <errno.h>
 #include "linux.h"
 #include <libelf.h>
-#include <libdwarf.h>
+#include <libdwarf/libdwarf.h>
 #include <libgen.h>
-#include <dwarf.h>
+#include <libdwarf/dwarf.h>
 
 #include "ctf_headers.h"
 #include "ctftools.h"
diff -uNr dtrace-20110215/cmd/ctfconvert/makefile dtrace-20110215.fed14/cmd/ctfconvert/makefile
--- dtrace-20110215/cmd/ctfconvert/makefile     2010-10-17 17:55:26.000000000 +0100
+++ dtrace-20110215.fed14/cmd/ctfconvert/makefile       2011-03-18 01:19:23.255448742 +0000
@@ -44,7 +44,7 @@
 LIBS = ../../build/libctf.a -ldwarf -lbfd -lelf -lz

 all:
-       @if [ ! -f /usr/lib/libdwarf.a ]; then \
+       @if [ ! -f /usr/lib64/libdwarf.a ]; then \
                echo "=================================================================" ; \
                echo "=== You need /usr/lib/libdwarf.a and /usr/lib/libbfd.a installed to build." ; \
                echo "=== " ; \
diff -uNr dtrace-20110215/tools/build.pl dtrace-20110215.fed14/tools/build.pl
--- dtrace-20110215/tools/build.pl      2011-01-26 23:36:06.000000000 +0000
+++ dtrace-20110215.fed14/tools/build.pl        2011-03-19 14:16:48.725192127 +0000
@@ -59,7 +59,7 @@
        }

        if (! -f "/usr/include/gelf.h") {
-               print "Error: you dont appear to have /usr/include/elf.h, which means\n";
+               print "Error: you dont appear to have /usr/include/gelf.h, which means\n";
                print "compilation will fail. You should add the libelf-dev package to\n";
                print "your system and retry the 'make all'.\n";
                exit(1);
diff -uNr dtrace-20110215/tools/mkctf.sh dtrace-20110215.fed14/tools/mkctf.sh
--- dtrace-20110215/tools/mkctf.sh      2010-08-26 10:03:21.000000000 +0100
+++ dtrace-20110215.fed14/tools/mkctf.sh        2011-03-18 01:21:59.600884493 +0000
@@ -19,11 +19,11 @@
 cd $build_dir
 cmd=`grep '^cmd_' $pwd/build/driver/.cpu_x86.o.cmd |
 sed -e 's/^.* := //' |
-sed -e 's/-Wp,-MD[^ ]* /-g /' |
+sed -e 's/-Wp,-MD[^ ]* /-gdwarf-2 /' |
 sed -e 's/\\\\//g' |
 sed -e 's/cpu_x86/ctf_struct/g' `
 #echo $cmd
 eval $cmd
 cd $pwd
-build/ctfconvert -L label -o build/linux-$BUILD_KERNEL.ctf build/driver/.tmp_ctf_struct.o
+build/ctfconvert -L label -o build/linux-$BUILD_KERNEL.ctf build/driver/ctf_struct.o
 ls -l build/linux-$BUILD_KERNEL.ctf

And here is a very quick and basic test, to show DTrace running:


# uname -a:
Linux fed14-64 2.6.35.11-83.fc14.x86_64 #1 SMP Mon Feb 7 07:06:44 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
# make load
tools/load.pl
01:26:50 Syncing...
01:26:50 Loading: build/driver/dtracedrv.ko
01:26:52 Preparing symbols...
01:26:52 Probes available: 314700
01:26:56 Time: 6s

# dtrace -n BEGIN
dtrace: description 'BEGIN' matched 1 probe
CPU     ID                    FUNCTION:NAME
  1      1                           :BEGIN
^C

Admittedly, that is the most basic DTrace one-liner you can possibly try. But I did also try a few 'syscall' one-liners, mostly with success.

Anyway, once I got this going, I wanted to try with Paul's latest release of the code, to get his latest bug fixes, but I was stopped in my tracks, as his FTP site was down. And it continued to be down, for over a week, which was quite frustrating. I tried emailing him, and leaving a blog comment, but again, I got no response. And there was no mention of this problem on his web sites, or blog.

Eventualy, while Googling, I discovered that Paul has a Twitter account, and that he had announced there, that he has a new host name for his FTP site. There is no mention of Paul's Twitter account on his web sites or blogs, at this moment. So I will publicize it here, as he does seem to make good use of it to report progress with DTrace for Linux:

http://twitter.com/#!/crispeditor

And for anyone wanting to try compiling the DTrace for Linux code, please note the new download site is:

ftp://crisp.dyndns-server.com/pub/release/website/dtrace/

Ok, so now I guess I had better download the latest release of the code, and try it out.

selinux, hwclock and /etc/adjtime

2011-01-27T17:00:00.002Z

If you are having problems with hwclock, you may want to remove the file '/etc/adjtime' and let hwclock recreate the file, for reasons discussed here:

"hwclock never updates drift factor in /etc/adjtime if time is always set more than once a day"

However, if you have selinux enabled, with an enforcing targeted policy, this will result in the wrong file security context. Here is how to fix it:


  # ls -Z /etc/adjtime
  -rw-r--r--  root root root:object_r:etc_t:s0           /etc/adjtime

  # restorecon -v /etc/adjtime
  restorecon reset /etc/adjtime context root:object_r:etc_t:s0->system_u:object_r:adjtime_t:s0

  # ls -Z /etc/adjtime
  -rw-r--r--  root root system_u:object_r:adjtime_t:s0   /etc/adjtime

  # grep adjtime /etc/selinux/targeted/contexts/files/file_contexts
  /etc/adjtime    --      system_u:object_r:adjtime_t:s0

Running Access 97 on Windows 7

2010-07-29T17:10:00.003+01:00

Today I had to get Access 97 running on a Windows 7 64bit PC. When I arrived on site, my colleague had already successfully installed Office 97, and the SR-1 patch (sr1off97.exe). So I guess that the well known "Hatten" file issue had already been resolved.


 "There is no license" error message starting Microsoft Access
 http://support.microsoft.com/kb/141373

On further investigation, I found that an attempt to install the SR-2 patch had failed, and also that the on-line help for Access 97 was not working at all. And I could see that the "Programming help" file had not been installed.

To install "Programming help", I ran the maintenance setup, but it soon failed, reporting message "Setup Error 825 - Object 9807". This issue is covered by the Microsoft support Article ID: 156638


 OFF97: International English SR-1 Patch Offpro97.inf
 http://support.microsoft.com/kb/156638

Basically, you need to download file "offpro97.exe" and extract from that the file "off97pro.inf" and copy it across to the 'setup' directory, replacing the existing file.

To get the help working, you need to install support for "winhelp32.exe".
This issue is covered by the Microsoft support Article ID: 917607


 I cannot open Help files that require the Windows Help (WinHlp32.exe) program
 http://support.microsoft.com/kb/917607

But then when you try to use the Access 97 help files, you will get error "There was a problem running the macro. (1037)". The fix for this is covered in the same Microsoft support article.

Basically, you need to run "C:\WINDOWS\syswow64\regedit32.exe" and create key "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WinHelp" and then create DWORD "AllowProgrammaticMacros" and set it to value "1".

In order to install the SR-2 patch (sr2bof97.exe), you need to replace some of the Windows 7 font files, with older versions copied from Windows XP. These are the font files, from Windows XP, that I used:


17/07/2004  11:39   127,596 comic.ttf
29/08/2002  13:00   111,476 comicbd.ttf
21/11/1996  00:00    41,408 hatten.ttf
17/07/2004  22:55   383,140 tahoma.ttf
17/07/2004  22:55   355,436 tahomabd.ttf

Finally, I installed file "Jet35sp3.exe" to make sure the Jet Engine was at the final version.

Ok, so that's it. I tried testing a few things with Access 97, and it seems to run fine on Windows 7.

Patching vmxnet to disable LRO

2010-07-19T15:18:00.005+01:00

We've been playing with two Centos 5.5 virtual machines connected over a virtual switch on VMware ESX 4.0 update 2. Unfortunately we experienced very poor TCP performance. I think the same issue affects ESX 4.1 as we found this reference in the release notes:

Poor TCP performance can occur in traffic-forwarding virtual machines with LRO enabled
Some Linux modules cannot handle LRO-generated packets.
As a result, having LRO enabled on a VMXNET 2 or VMXNET 3 device in a traffic
forwarding virtual machine running a Linux guest operating system can cause
poor TCP performance. LRO is enabled by default on these devices.

Workaround: In traffic-forwarding virtual machines running Linux guests,
set the module load time parameter for the VMXNET 2 or VMXNET 3 Linux driver
to include disable_lro=1.

We found that this works:

# rmmod vmxnet
# modprobe vmxnet disable_lro=1

BUT the problem is how to make vmxnet default to disabling LRO when the VM is first booted.

We tried editing '/etc/modprobe.conf' and adding 'options vmxnet disable_lro=1' but this was not sufficent. It seems that the vmxnet module is first loaded during bootup from the initrd.

Our conclusion was that patching the source code of vmxnet was the best way. So here is what we did:

# cd /usr/lib/vmware-tools/modules/source/
# tar xvf vmxnet.tar
# cd vmxnet-only
# grep -n disable_lro *
vmxnet.c:155:static int disable_lro = 0;
vmxnet.c:157:   module_param(disable_lro, int, 0);
vmxnet.c:159:   MODULE_PARM(disable_lro, "i");
vmxnet.c:931:       !disable_lro) {
# cp vmxnet.c vmxnet.c.orig
# chmod +w vmxnet.c
# vi vmxnet.c
# diff -u vmxnet.c.orig vmxnet.c
--- vmxnet.c.orig       2010-07-19 13:20:44.000000000 +0100
+++ vmxnet.c    2010-07-19 13:57:43.000000000 +0100
@@ -152,7 +152,7 @@
#endif // VMXNET_DO_ZERO_COPY

#ifdef VMXNET_DO_TSO
-static int disable_lro = 0;
+static int disable_lro = 1;
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2, 6, 9)
module_param(disable_lro, int, 0);
#else
@@ -932,6 +932,14 @@
  lp->lpd = TRUE;
  printk(" lpd");
}
+
+   if (disable_lro) {
+     printk(" disable_lro:1");
+   }
+   else {
+     printk(" disable_lro:0");
+   }
+
#endif
#endif

# cd ..
# mv vmxnet.tar vmxnet.tar.orig
# tar cvf vmxnet.tar vmxnet-only/
# vmware-config-tools.pl -c

The change to the source code was just to change the value of the disable_lro variable from zero to one. And we also added some code to report the value of the variable when the module loads.

To recompile the modules, you will need these rpm packages installed:
gcc, binutils, kernel-devel, kernel-headers

For reference, the version of vmware tools we were using was:
VMwareTools-8195-261974

Of course, if you update vmware-tools, you may need to review this fix, and re-patch the file.

If you are using the vmxnet3 driver, the fix should be similar to the above.

I'd like to thank Michael Melling for helping test the above patch.
Regards
Nigel Smith

DSCP QoS & Microsoft Office Live Meeting

2009-08-21T16:06:00.005+01:00

Today I've double checked the QoS situation with Live Meeting, with the help of my colleague Michael Melling, and here are our conclusions:

DSCP QosEnabled works fine with Office Communication 2007 R2 on Windows XP, Vista and Windows 7, provided you use the latest updated 3.5.6907.37 version. You can use WireShark to confirm the correct DSCP values are being used, as per my earlier blog post. The audio packets and the video packets will have a different DSCP value, so you can give them a appropriate priority over your network.

Now what about Office Live Meeting? We are using the 8.0.6362.128 version, which AFAIK is the latest. If you are running Window XP, then no problem. If you are trying to use Vista or Window 7, then there is a problem, as with those operating systems, Live Meeting fails to set a DCSP code on the packets.

In our tests today, we started a video call in Office Communicator, then switched into 'Share Information Using Live Meeting' mode. This opens the Live Meeting window, but leaves audio & video going through Office Communicator. But as we are in conference mode, all A/V packets are
now going via the OCS server. At this stage DSCP is still working fine. (We also have DCSP enabled on our OCS 2007 R2 server).

Now by clicking the WebCam icon in Live Meeting, you can flip the Video into a separate windows called 'Voice & Video'. (Don't forget you also need to click the 'Join Audio' button). Now Live Meeting is handling the Video, and that's fine if you are using Windows XP. But if you use Vista or Windows 7, a check with WireShark will show none of the UDP packets being sent by Live Meeting have a DSCP value - they are all marked 0x00.

To me that looks like a bug in Live Meeting.

There is a partial work around. Using Group Policy on Vista and Window 7, you can force all packets generated by a particular application to use a DSCP value that you specify. Unfortunately this does not allow you to differentiate between Audio and Video.

Here are a couple of screen-shots were I setup a local group policy on a Vista PC, so that any UDP packets sent by Live Meeting are given a DSCP value of 56'd. I used that value just while testing, to differentiate then from the A/V packets from Office Communicator.

Microsoft Office Communicator 2007 R2 and DSCP QoS

2009-07-29T19:14:00.012+01:00

Anyone trying to get DSCP working with Office Communicator 2007 R2 on Windows XP needs to grab the 'July' update (v3.5.6907.37) otherwise you will be having problems getting video to work. Here are the links:


"Description of the update for Communicator 2007 R2: July 2009"
http://support.microsoft.com/kb/969695/

"Video frames are not displayed in Office Communicator 2007 R2 on a Windows XP-based computer"
http://support.microsoft.com/kb/971846/

(BTW, I'd like to thank Michael Melling for spotting those links, and for helping with the testing that I describe below.)

Before the update, with Office Communicator versions v3.5.6907.0 or v3.5.6907.34, on Windows XP, after setting QoSEnabled to 1, you find audio continues to work fine, but video fails. Most of the time we just got a black video windows, sometimes it was a grey window, and sometimes a frozen image.

Using OCS's "Monitoring Server Reports" you can see whats going wrong. Click the "User Call List" link, and then filter by one of the users in the call, and this will give you a detailed statistical report of all finished calls. For the Video Stream you will see a "Packet loss Rate" of about 60% !

Office Communicator, with QoSEnabled, works fine on Vista and Windows 7, without this 'July' update. I think that Microsoft must have done the majority of their testing with their latest operating systems, and have only recently got around to testing with Windows XP.

We have been battling this problem for a couple of weeks, so this fix comes just in time! In trying to investigate what was goingwrong, we have learned a great deal, so not all the time was wasted, just some!

DSCP QoS is enabled in the registry, like this:


HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\RTC\Transport
"QoSEnabled"=dword:00000001

.. as per these instructions:


"Enabling DSCP Marking"
http://technet.microsoft.com/en-us/library/dd441192(office.13).aspx

BTW, you need to exit (not just close) Office Communicator and restart it, for this registry setting to take effect.

Once Office Communicator has setup the call using SIP (over TCP), using Office Communications Server 2007 R2, all audio & video network traffic is sent directly between the two clients using RTP (Real Time Protocol) packets in UDP packets.

Using WireShark, we could see the packets being tagged correctly:


Audio - DSCP 0x28: Class Selector CS5
Video - DSCP 0x18: Class Selector CS3

BTW, these are the default values. You can change them using a group policy.

In WireShark, to just grab the audio & video traffic, use a capture filter like this:


  udp and ip[1]!=0

This will match any UDP packets that have a non-zero DSCP field in the IP header.You can use the following display filters to select either Audio or Video packets:


For Audio, use:
    ip.dsfield.dscp == 40

For Video, use:
    ip.dsfield.dscp == 24

BTW, if you use tcpdump to capture the packets, remember that the DSCP values are multiplied by four. So use these capture filters with tcpdump:


For Audio:
    ip[1]=0xA0
    ip[1]=160

For Video:
    ip[1]=0x60
    ip[1]=96

It is very useful to be able to see the RTP headers in the packets. To do this in WireShark, right-click on any of the packets of a UDP stream, and select 'Decode As..', the select 'Transport' as RTP.

Once you have WireShark decoding the RTP header, you can see a couple of interesting fields in the header. The first is 'Payload type',and this is set as follows:


  Type  97 - RT-Audio encoded redundant data
  Type 111 - 'Siren' audio - used by 'LiveMeeting'
  Type 114 - RTaudio (x-msrta)
  Type 118 - 'Comfort Noise'
  Type 121 - RTvideo (x-rtvc1)

The second interesting RTP header field is the 'Sequence number'. This is a 16-bit counter field, that increments by 1 for each packet sent in a particular stream, and is useful for detecting dropped or duplicated packets. (The sequence number seems to start at a random value.)

There is a useful feature in WireShark which will do 'Stream Anaysis..'. On WireShark v1.2.0 you will find this under the 'Telephony > RTP' menu. This will highlight where the sequence number goes wrong.

Ok, so why was the video failing, (before the 'July' update for Office Communication). We had two laptops, running Office Communicator, connected together via a HP Procurve switch,and enabled port mirroring on the switch, to a third laptop to see all network traffic. Running WireShark on the client sending video, we saw a lot more video packet, then were being received at the other end, on the receiving client. The third monitoring laptop confirmed that packets were being dropped before they emerged 'onto the wire' from the sending laptop. Looking closely at the WireShark capture from the laptop sending video, and using the 'Stream Analysis..' we saw many video packets with duplicate RTP sequence numbers! Correlating with the capture on the receiving client, we saw that any packet that had been duplicated was missing. It was not a case of just the duplicated packet being missing, but also the original packet. Thus from the point of view of the receiving packet, there was a massive level of packet loss, hence it could not display the video stream.

BTW, on Windows XP, you can use 'tcmon.exe' from the resource kit to see what QoS policies are in effect, and this confirms packet loss onthe video stream. Here are some screen shots:

Unfortunately tcmon does not run on Vista or Windows 7, which is very disapointing. Let's hope that Microsoft get around to fixing this!

BTW, if you are using Linux to priorities the audio & video traffic as it is sent into the WAN, like we are, you will find the following iptables & 'tc filters' useful to match the packets:


# Audio:
iptables -t mangle -A POSTROUTING -o eth0 -p udp -m dscp --dscp 40 -j mark1
iptables -t mangle -A POSTROUTING -o eth0 -p udp -m dscp --dscp-class CS5 -j mark1
tc filter add dev eth0 parent 10:0 prio 1 protocol ip u32 match ip tos 0xA0 0xff flowid 10:200

# Video:
iptables -t mangle -A POSTROUTING -o eth0 -p udp -m dscp --dscp 24 -j mark2
iptables -t mangle -A POSTROUTING -o eth0 -p udp -m dscp --dscp-class CS3 -j mark2
tc filter add dev eth0 parent 10:0 prio 1 protocol ip u32 match ip tos 0x60 0xff flowid 10:210

Best Regards
Nigel Smith

OpenSolaris 2009-06 and the Comstar iScsi Target

2009-07-29T00:46:00.002+01:00

Having upgraded from OpenSolaris 2008-11 to 2009-06, I wanted to try the new Comstar iscsi target, to see how well it worked, and to see if it was ready to replace the 'old' iscsi target. And I wanted to see how far I could get with the package versions in the 'production-quality' release repository, based on snv_111. To start with, I just wanted to get something basic going as quickly as possible.

The first tasks was to disable the old iscsi target (to avoid a conflict with the new target which would also want to listen on TCP port 3260), and to enable the 'SCSI Target Mode Framework' - stmf.


# svcs iscsitgt
online         Jul_09   svc:/system/iscsitgt:default
# svcs stmf
disabled       Jul_09   svc:/system/stmf:default
# svcadm disable iscsitgt
# svcs iscsitgt
STATE          STIME    FMRI
disabled       21:29:57 svc:/system/iscsitgt:default
# svcadm enable stmf
# svcs stmf
STATE          STIME    FMRI
online         21:30:55 svc:/system/stmf:default
# stmfadm list-state
Operational Status: online
Config Status     : initialized

Next I created a new small zfs volume.
(My zfs pool is named 'rz2pool' - it's Raidz2 - dual parity.)


# zfs create -V 4g rz2pool/iscsi_lun1

And then created a 'logical unit', using the zvol as it's backing store.


# sbdadm create-lu /dev/zvol/rdsk/rz2pool/iscsi_lun1

Created the following LU:

             GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f00008278f04694a5cf0980001      4294901760       /dev/zvol/rdsk/rz2pool/iscsi_lun1

# stmfadm list-lu -v
LU Name: 600144F00008278F04694A5CF0980001
   Operational Status: Online
   Provider Name     : sbd
   Alias             : /dev/zvol/rdsk/rz2pool/iscsi_lun1
   View Entry Count  : 0

Now using the GUID of the logical unit, we make it visible to any initiators, by adding a 'view'


# stmfadm add-view 600144f00008278f04694a5cf0980001
# stmfadm list-view -l 600144f00008278f04694a5cf0980001
View Entry: 0
   Host group   : All
   Target group : All
   LUN          : 0

Next we need to do the iscsi part, for which we need to install the iscsi target and the 'itadm' command. These were not already installed so I had to add the 'SUNWiscsit' package.


# pkg install -v SUNWiscsit
Creating Plan | Before evaluation:
UNEVALUATED:
+pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z

After evaluation:
None -> pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z
None -> pkg:/SUNWiscsidm@0.5.11,5.11-0.111:20090508T161041Z
Actuators:
     restart_fmri: svc:/system/manifest-import:default
None
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    2/2       23/23     0.66/0.66

PHASE                                        ACTIONS
Install Phase                                  76/76
PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                                2/2

# pkg list -v SUNWiscsit
FMRI                                                             STATE      UFIX
pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z               installed  ----
# pkg list -v SUNWiscsidm
FMRI                                                             STATE      UFIX
pkg:/SUNWiscsidm@0.5.11,5.11-0.111:20090508T161041Z              installed  ----

On the next step I had a small problem.
(BTW, I am not using NWAM to configure my network card.)


# svcadm enable -r iscsi/target:default
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

# svcs -a | grep network/physical
disabled       Jul_09   svc:/network/physical:nwam
online         Jul_09   svc:/network/physical:default

# svcadm enable  iscsi/target
# svcs -a | grep iscsi
disabled       Jul_09   svc:/network/iscsi_initiator:default
disabled       21:29:57 svc:/system/iscsitgt:default
maintenance    22:29:13 svc:/network/iscsi/target:default

# svcs -xv
svc:/network/iscsi/target:default (iscsi target)
State: maintenance since Tue Jul 14 22:29:13 2009
Reason: Start method failed repeatedly, last exited with status 4.
  See: http://sun.com/msg/SMF-8000-KS
  See: man -M /usr/share/man -s 1M itadm
  See: /var/svc/log/network-iscsi-target:default.log
Impact: This service is not running.

# tail /var/svc/log/network-iscsi-target\:default.log
[ Jul 14 22:29:13 Enabled. ]
[ Jul 14 22:29:13 Executing start method ("/lib/svc/method/iscsi-target start"). ]
iscsi-target: Requesting to enable iscsi target
open failed: INVALIDUnable to open device /devices/pseudo/iscsit@0:iscsit[ Jul 14 22:29:13 Method "start" exited with status 4. ]

Rebooting solved this problem.


# svcs -a | grep iscsi
disabled       19:25:15 svc:/network/iscsi_initiator:default
disabled       19:25:18 svc:/system/iscsitgt:default
online         19:25:37 svc:/network/iscsi/target:default

So now we create the iscsi target and check it looks ok:


# itadm create-target
Target iqn.1986-03.com.sun:02:7e969860-cb8d-475a-ddbb-97bf2f22ce7b successfully created

# itadm list-target -v
TARGET NAME                                                  STATE    SESSIONS
iqn.1986-03.com.sun:02:7e969860-cb8d-475a-ddbb-97bf2f22ce7b  online   0
       alias:                  -
       auth:                   none (defaults)
       targetchapuser:         -
       targetchapsecret:       unset
       tpg-tags:               default

I was then able to discover and logon to this iscsi target from a Windows 2003 server, using the Microsoft iscsi initiator.

The above is just the simplest possible configuration, just to allow some simple tests. For a more practical setup, you may well want multiple logical units, and want to put some restrictions on the views. But details on how to do that will have to wait for a follow-up post to this blog!

I did notice a couple of things fairly quickly. I saw the connection between the initiator and target was dropping every 50 seconds, followed immediately by a re-logon. Checking with WireShark showed that it was the target starting this with a [FIN, ACK]. With the old iscsi target I would have used DTrace to confirm this, but the iscsi target provider for the new iscsi target was not available.

I looked at the source code for the comstar iscsi target:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/comstar/port/iscsit/iscsit.c

.. and from the history link, I could see that a couple of likely fixes had already been committed:


26-May-2009 Priya Krishnan - 6809997 COMSTAR iscsi target DTrace Provider needed
08-May-2009 Peter Dunlap - 6755803 win2003 initiator numerous iscsi connection lost and connection retries mesgs to iscsi target

Now normally, I would hyper-link those bug number for you. But in this case there is no point, because Sun has chosen not to allows details of those bugs to be publicly available. (Or any of the other comstar iscsi target bugs AFAIK.)

Having those fixes is important for me, so it looks like I will have to update my OpenSolaris 2009-06 using the development repository to snv_118.

Using SNMP with a Cisco router to troubleshoot & diagnose ADSL problems

2009-06-25T13:14:00.004+01:00

We've been having numerous errors and problem with an ADSL line at the company where I work. We're trying to run an Ipsec tunnel across the ADSL line to connect to one of our small branch offices. Every few hours we would have problems, with the tunnel dropping out. Restarting the router would often fix the problem, but the problems never went away for long.

We were using good quality Cisco 1801 routers. The most useful command for debugging is 'show dsl interface atm0' and 'show int atm0'. Below is some typical output, while the ADSL line was faulty:


#show int atm 0
ATM0 is up, line protocol is up
  Hardware is MPC ATMSAR (with Alcatel ADSL Module)
  MTU 4470 bytes, sub MTU 4470, BW 832 Kbit/sec, DLY 440 usec,
     reliability 255/255, txload 41/255, rxload 6/255
  Encapsulation ATM, loopback not set
  Encapsulation(s): AAL5  AAL2, PVC mode
  23 maximum active VCs, 1024 VCs per VP, 1 current VCCs
  VC Auto Creation Disabled.
  VC idle disconnect time: 300 seconds
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 48
  Queueing strategy: Per VC Queueing
  5 minute input rate 22000 bits/sec, 20 packets/sec
  5 minute output rate 137000 bits/sec, 24 packets/sec
     322340 packets input, 50047091 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 182 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     345336 packets output, 145468959 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 output buffer failures, 0 output buffers swapped out

#show dsl interface atm0
ATM0
Alcatel 20190 chipset information
                ATU-R (DS)                      ATU-C (US)
Modem Status:    Showtime (DMTDSL_SHOWTIME)
DSL Mode:        ITU G.992.1 (G.DMT) Annex A
ITU STD NUM:     0x03                            0x2
Vendor ID:       'STMI'                          'P   '
Vendor Specific: 0x0000                          0x0000
Vendor Country:  0x0F                            0xB5
Chip ID:         C196 (0)
DFE BOM:         DFE3.0 Annex A (1)
Capacity Used:   100%                            85%
Noise Margin:     1.0 dB                         11.0 dB
Output Power:    20.0 dBm                        12.5 dBm
Attenuation:     34.0 dB                         17.0 dB
Defect Status:   None                            None
Last Fail Code:  None
Watchdog Counter: 0xA7
Watchdog Resets: 0
Selftest Result: 0x00
Subfunction:     0x00
Interrupts:      4155 (0 spurious)
PHY Access Err:  0
Activations:     3
LED Status:      ON
LED On Time:     100
LED Off Time:    100
Init FW:         init_AMR-3.0.014_no_bist.bin
Operation FW:    AMR-3.0.014.bin
FW Source:       embedded
FW Version:      3.0.14

                 Interleave             Fast    Interleave              Fast
Speed (kbps):          4640                0           832                 0
Cells:               716617                0     108647003                 0
Reed-Solomon EC:      43244                0             9                 9
CRC Errors:           25362                0            22                14
Header Errors:        15617                0             9                 7
Total BER:                1610E-8                0E-0
Leakage Average BER:      2734E-8                0E-0
                        ATU-R (DS)      ATU-C (US)
Bitswap:               enabled            enabled
Bitswap success:          0                   0
Bitswap failure:          0                   0

You can see from the above that the 'Down Stream' (DS) noise margin is very low at 1dB. We have also seen 0.5dB reported, or even 0db! You can also see the error counts are high. In the output of 'show int atm 0', the figure to monitor is the number of CRC errors - in the output above it's 182. It is also useful to set 'debug atm errors' on the cisco. The in the syslog's you will see messages like:


*Jun 19 10:22:01.894: ATM0: AAL5 rx errors (status = 0C030000)
*Jun 19 10:22:01.974: ATM0: AAL5 rx errors (status = 2C030000)
*Jun 19 10:22:02.758: ATM0: AAL5 rx errors (status = 0C030000)
*Jun 19 10:22:04.058: ATM0: AAL5 rx errors (status = 2C070000)
*Jun 19 10:22:04.922: ATM0: AAL5 rx errors (status = 0C030000)
*Jun 19 10:22:06.174: ATM0: AAL5 rx errors (status = 0C030000)
*Jun 19 10:22:07.822: ATM0: AAL5 rx errors (status = 0C030000)

To try and dig deeper, I used SNMP (from a Linux box on the same LAN), to query the router:


[root@hexdns1 snmp_queries]# snmpwalk -v1 -c public 192.168.6.1  1.3.6.1.2.1.10.94
SNMPv2-SMI::transmission.94.1.1.1.1.1.19 = INTEGER: 2
SNMPv2-SMI::transmission.94.1.1.1.1.2.19 = INTEGER: 3
SNMPv2-SMI::transmission.94.1.1.2.1.2.19 = STRING: "P"
SNMPv2-SMI::transmission.94.1.1.2.1.3.19 = STRING: "0"
SNMPv2-SMI::transmission.94.1.1.2.1.4.19 = Gauge32: 110
SNMPv2-SMI::transmission.94.1.1.2.1.5.19 = Gauge32: 175
SNMPv2-SMI::transmission.94.1.1.2.1.7.19 = Gauge32: 200
SNMPv2-SMI::transmission.94.1.1.2.1.8.19 = Gauge32: 4492000
SNMPv2-SMI::transmission.94.1.1.3.1.1.19 = ""
SNMPv2-SMI::transmission.94.1.1.3.1.2.19 = STRING: "STMI"
SNMPv2-SMI::transmission.94.1.1.3.1.3.19 = STRING: "0"
SNMPv2-SMI::transmission.94.1.1.3.1.4.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.3.1.5.19 = Gauge32: 340
SNMPv2-SMI::transmission.94.1.1.3.1.7.19 = Gauge32: 130
SNMPv2-SMI::transmission.94.1.1.3.1.8.19 = Gauge32: 967000
SNMPv2-SMI::transmission.94.1.1.4.1.2.19 = Gauge32: 4448000
SNMPv2-SMI::transmission.94.1.1.5.1.2.19 = Gauge32: 832000
SNMPv2-SMI::transmission.94.1.1.6.1.1.19 = Counter32: 697
SNMPv2-SMI::transmission.94.1.1.6.1.2.19 = Counter32: 449
SNMPv2-SMI::transmission.94.1.1.6.1.4.19 = Counter32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.5.19 = Counter32: 1820
SNMPv2-SMI::transmission.94.1.1.6.1.6.19 = Counter32: 31
SNMPv2-SMI::transmission.94.1.1.6.1.10.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.11.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.13.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.14.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.15.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.17.19 = Gauge32: 565
SNMPv2-SMI::transmission.94.1.1.6.1.18.19 = Gauge32: 381
SNMPv2-SMI::transmission.94.1.1.6.1.20.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.6.1.21.19 = Gauge32: 1034
SNMPv2-SMI::transmission.94.1.1.6.1.22.19 = Gauge32: 10
SNMPv2-SMI::transmission.94.1.1.7.1.1.19 = Counter32: 45
SNMPv2-SMI::transmission.94.1.1.7.1.2.19 = Counter32: 28
SNMPv2-SMI::transmission.94.1.1.7.1.3.19 = Counter32: 0
SNMPv2-SMI::transmission.94.1.1.7.1.4.19 = Counter32: 15043
SNMPv2-SMI::transmission.94.1.1.7.1.8.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.7.1.9.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.7.1.10.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.7.1.11.19 = Gauge32: 328
SNMPv2-SMI::transmission.94.1.1.7.1.13.19 = Gauge32: 44
SNMPv2-SMI::transmission.94.1.1.7.1.14.19 = Gauge32: 27
SNMPv2-SMI::transmission.94.1.1.7.1.15.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.7.1.16.19 = Gauge32: 3717
SNMPv2-SMI::transmission.94.1.1.10.1.3.19 = Counter32: 1864592813
SNMPv2-SMI::transmission.94.1.1.10.1.4.19 = Counter32: 804238
SNMPv2-SMI::transmission.94.1.1.10.1.10.19 = Gauge32: 2538597
SNMPv2-SMI::transmission.94.1.1.10.1.11.19 = Gauge32: 11493
SNMPv2-SMI::transmission.94.1.1.10.1.15.19 = Gauge32: 804265846
SNMPv2-SMI::transmission.94.1.1.10.1.16.19 = Gauge32: 400641
SNMPv2-SMI::transmission.94.1.1.11.1.3.19 = Counter32: 31565
SNMPv2-SMI::transmission.94.1.1.11.1.4.19 = Counter32: 56466
SNMPv2-SMI::transmission.94.1.1.11.1.10.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.11.1.11.19 = Gauge32: 0
SNMPv2-SMI::transmission.94.1.1.11.1.15.19 = Gauge32: 18799
SNMPv2-SMI::transmission.94.1.1.11.1.16.19 = Gauge32: 44213

If you have the ADSL-LINE-MIB in place, these OID's can be translated:


# snmptranslate 1.3.6.1.2.1.10.94
SNMPv2-SMI::transmission.94
# export MIBS=ALL
# snmptranslate 1.3.6.1.2.1.10.94
ADSL-LINE-MIB::adslMIB

For a guide on how to use Linux SNMP, see my snmp tutorial.


# snmpwalk -v1 -c public 192.168.6.1 1.3.6.1.2.1.10.94
ADSL-LINE-MIB::adslLineCoding.19 = INTEGER: 2
ADSL-LINE-MIB::adslLineType.19 = INTEGER: interleavedOnly(3)
ADSL-LINE-MIB::adslAtucInvVendorID.19 = STRING: P
ADSL-LINE-MIB::adslAtucInvVersionNumber.19 = STRING: 0
ADSL-LINE-MIB::adslAtucCurrSnrMgn.19 = Wrong Type (should be INTEGER): Gauge32: 110
ADSL-LINE-MIB::adslAtucCurrAtn.19 = Gauge32: 170 tenth dB
ADSL-LINE-MIB::adslAtucCurrOutputPwr.19 = Wrong Type (should be INTEGER): Gauge32: 200
ADSL-LINE-MIB::adslAtucCurrAttainableRate.19 = Gauge32: 4686000 bps
ADSL-LINE-MIB::adslAturInvSerialNumber.19 = STRING:
ADSL-LINE-MIB::adslAturInvVendorID.19 = STRING: STMI
ADSL-LINE-MIB::adslAturInvVersionNumber.19 = STRING: 0
ADSL-LINE-MIB::adslAturCurrSnrMgn.19 = Wrong Type (should be INTEGER): Gauge32: 15
ADSL-LINE-MIB::adslAturCurrAtn.19 = Gauge32: 340 tenth dB
ADSL-LINE-MIB::adslAturCurrOutputPwr.19 = Wrong Type (should be INTEGER): Gauge32: 130
ADSL-LINE-MIB::adslAturCurrAttainableRate.19 = Gauge32: 978000 bps
ADSL-LINE-MIB::adslAtucChanCurrTxRate.19 = Gauge32: 4640000 bps
ADSL-LINE-MIB::adslAturChanCurrTxRate.19 = Gauge32: 832000 bps
ADSL-LINE-MIB::adslAtucPerfLofs.19 = Counter32: 453
ADSL-LINE-MIB::adslAtucPerfLoss.19 = Counter32: 13
ADSL-LINE-MIB::adslAtucPerfLprs.19 = Counter32: 0
ADSL-LINE-MIB::adslAtucPerfESs.19 = Counter32: 1997
ADSL-LINE-MIB::adslAtucPerfInits.19 = Counter32: 18
ADSL-LINE-MIB::adslAtucPerfCurr15MinLofs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAtucPerfCurr15MinLoss.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAtucPerfCurr15MinLprs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAtucPerfCurr15MinESs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAtucPerfCurr15MinInits.19 = Gauge32: 0
ADSL-LINE-MIB::adslAtucPerfCurr1DayLofs.19 = Gauge32: 445 seconds
ADSL-LINE-MIB::adslAtucPerfCurr1DayLoss.19 = Gauge32: 14 seconds
ADSL-LINE-MIB::adslAtucPerfCurr1DayLprs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAtucPerfCurr1DayESs.19 = Gauge32: 1425 seconds
ADSL-LINE-MIB::adslAtucPerfCurr1DayInits.19 = Gauge32: 9
ADSL-LINE-MIB::adslAturPerfLofs.19 = Counter32: 22 seconds
ADSL-LINE-MIB::adslAturPerfLoss.19 = Counter32: 15 seconds
ADSL-LINE-MIB::adslAturPerfLprs.19 = Counter32: 0 seconds
ADSL-LINE-MIB::adslAturPerfESs.19 = Counter32: 95908 seconds
ADSL-LINE-MIB::adslAturPerfCurr15MinLofs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAturPerfCurr15MinLoss.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAturPerfCurr15MinLprs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAturPerfCurr15MinESs.19 = Gauge32: 345 seconds
ADSL-LINE-MIB::adslAturPerfCurr1DayLofs.19 = Gauge32: 22 seconds
ADSL-LINE-MIB::adslAturPerfCurr1DayLoss.19 = Gauge32: 15 seconds
ADSL-LINE-MIB::adslAturPerfCurr1DayLprs.19 = Gauge32: 0 seconds
ADSL-LINE-MIB::adslAturPerfCurr1DayESs.19 = Gauge32: 21932 seconds
ADSL-LINE-MIB::adslAtucChanCorrectedBlks.19 = Counter32: 4045523380
ADSL-LINE-MIB::adslAtucChanUncorrectBlks.19 = Counter32: 23129426
ADSL-LINE-MIB::adslAtucChanPerfCurr15MinCorrectedBlks.19 = Gauge32: 6278668
ADSL-LINE-MIB::adslAtucChanPerfCurr15MinUncorrectBlks.19 = Gauge32: 17489
ADSL-LINE-MIB::adslAtucChanPerfCurr1DayCorrectedBlks.19 = Gauge32: 1783723602
ADSL-LINE-MIB::adslAtucChanPerfCurr1DayUncorrectBlks.19 = Gauge32: 10061120
ADSL-LINE-MIB::adslAturChanCorrectedBlks.19 = Counter32: 4235
ADSL-LINE-MIB::adslAturChanUncorrectBlks.19 = Counter32: 9371
ADSL-LINE-MIB::adslAturChanPerfCurr15MinCorrectedBlks.19 = Gauge32: 0
ADSL-LINE-MIB::adslAturChanPerfCurr15MinUncorrectBlks.19 = Gauge32: 0
ADSL-LINE-MIB::adslAturChanPerfCurr1DayCorrectedBlks.19 = Gauge32: 1538
ADSL-LINE-MIB::adslAturChanPerfCurr1DayUncorrectBlks.19 = Gauge32: 5374

Many of the figure in the output of 'show dsl interface atm0' are available via SNMP. For example 'Down Stream Noise Margin' is 'adslAturCurrSnrMgn' quoted in tenths of a decibel. So a reading of 15 means 1.5dB. To get an idea of how these values were changing with time, I used Cacti and RRD to poll the SNMP and produce some nice graphs. I Googles and found a Cacti 'ADSL Line Data' templates that allows you to graph Attenuation, Noise margin, and Output Power. Of those, only Noise Margin is interesting, because I found the Attenuation and Output Power to remain constant. But the thing the template did not graph was the number of uncorrectable errors, labelled 'adslAtucChanUncorrectBlks' in the MIB, and this is what I was really interested in, as when we were having problems with the ADSL line, the count of uncorrectable errors was high.

So I modified '/var/www/cacti/resource/snmp_queries/adslLine.xml' to include another OID. I chose 'ADSL-LINE-MIB::adslAtucChanPerfCurr15MinUncorrectBlks' which the MIB decribes as follows:


# snmptranslate -Td ADSL-LINE-MIB::adslAtucChanPerfCurr15MinUncorrectBlks
ADSL-LINE-MIB::adslAtucChanPerfCurr15MinUncorrectBlks
adslAtucChanPerfCurr15MinUncorrectBlks OBJECT-TYPE
  -- FROM       ADSL-LINE-MIB
  MAX-ACCESS    read-only
  STATUS        current
  DESCRIPTION   "Count of all blocks received with uncorrectable
              errors on this channel within the current 15 minute
              interval."

I modified the XML for the data query as follows:


# snmptranslate -On ADSL-LINE-MIB::adslAtucChanPerfCurr15MinUncorrectBlks
.1.3.6.1.2.1.10.94.1.1.10.1.11

# diff -u adslLine.orig.xml adslLine.xml
--- adslLine.orig.xml   2009-05-01 16:57:18.000000000 +0100
+++ adslLine.xml    2009-06-15 18:53:58.000000000 +0100
@@ -27,6 +27,13 @@
          <direction>input</direction>
          <oid>.1.3.6.1.2.1.10.94.1.1.3.1.2</oid>
        </adslAtucInvVendorID>
+      <adslAtuc15mUncrBlks>
+         <name>Local Curr 15min Uncorrectable Blocks</name>
+         <method>walk</method>
+         <source>value</source>
+         <direction>output</direction>
+         <oid>.1.3.6.1.2.1.10.94.1.1.10.1.11</oid>
+       </adslAtuc15mUncrBlks>
       <adslAtucCurrSnrMgn>
          <name>Local Noise Margin (tenth dB)</name>
          <method>walk</method>

When I started to graph this value, I was surprised by the range of values I saw. Often it was low, a few hundred, but some time it would shoot up to 60,000! I realised that to graph this range of values it would be better to use a logarithmic scale.

After graphing uncorrectable errors for a while I began to question what does 'within the current 15 minute interval.' actually mean. So let's look in detail at the OID's related to uncorrectable errors. Here are some (high) starting figures:


# snmpwalk -v1 -c public 192.168.6.1  1.3.6.1.2.1.10.94.1.1.10.1
SNMPv2-SMI::transmission.94.1.1.10.1.3.19 = Counter32: 293561906
SNMPv2-SMI::transmission.94.1.1.10.1.4.19 = Counter32: 1493720
SNMPv2-SMI::transmission.94.1.1.10.1.10.19 = Gauge32: 6671894
SNMPv2-SMI::transmission.94.1.1.10.1.11.19 = Gauge32: 80028
SNMPv2-SMI::transmission.94.1.1.10.1.15.19 = Gauge32: 293561906
SNMPv2-SMI::transmission.94.1.1.10.1.16.19 = Gauge32: 1493720

You can see that "adslAtucChanCorrectedBlks" and "adslAtucChanPerfCurr1DayUncorrectBlks" are identical values, and they were increasing exactly in step, so they are both SNMP 'Counters'. A few minutes latter, we get these reading:


# snmpwalk -v1 -c public 192.168.6.1  1.3.6.1.2.1.10.94.1.1.10.1
SNMPv2-SMI::transmission.94.1.1.10.1.3.19 = Counter32: 293628501
SNMPv2-SMI::transmission.94.1.1.10.1.4.19 = Counter32: 1495372
SNMPv2-SMI::transmission.94.1.1.10.1.10.19 = Gauge32: 27251
SNMPv2-SMI::transmission.94.1.1.10.1.11.19 = Gauge32: 1383
SNMPv2-SMI::transmission.94.1.1.10.1.15.19 = Gauge32: 293628501
SNMPv2-SMI::transmission.94.1.1.10.1.16.19 = Gauge32: 1495372

In the period between taking these readings, the '15Min' figures, have reset to zero, and are counting up again. If you keep a constant watch on these figures, you can see that every 15 minutes they reset back to zero. I found the following command to be useful to monitor errors occuring in real time.


# watch -n 5 'snmpwalk -v1 -c public 192.168.6.1  1.3.6.1.2.1.10.94.1.1.10.1'

So with the benefit of hindsight, I may have been better using Cacti to graph 'adslAtucChanUncorrectBlks' as a 'COUNTER', rather than 'adslAtucChanPerfCurr15MinUncorrectBlks' as a 'GAUGE'.

Eventually we got BT to come on site to check the ADSL line, and they located
a bad connection. Since they fixed that, the noise margin has remained consistently above 5dB and the uncorrected blocks has been very low.

Here are some graph, where you can that the ADSL line has been fixed:

A new way to search & browse the OpenSolaris Mail Lists

2008-10-07T00:29:00.005+01:00

You can now search and browse through over a quarter million messages posted to the OpenSolaris Mail-lists using a free service called 'MarkMail'.

Head over to http://opensolaris.markmail.org/ and take a look.

I stumbled across the MarkMail web site last week,via Google. It seems to have been running for about 12 months, and in that time they have loaded the mail archives of thousands of open-source projects into their system.

But until last week, they did not include anything on OpenSolaris. A shocking omission! So I left feedback, and I was quickly contacted by MarkMail's Jason Hunter. Jason wasted no time in subscribing their system to most of the OpenSolaris lists, and then within a few days they had loaded the 'MailMan' archive files. (If you would like to see further lists added then leave feedback on the MarkMail site via their form.)

By going to this link http://opensolaris.markmail.org/search/?q= you can see the latest emails that have been added into the system. I find this a useful way of just getting some idea of the vast amount of activity being generated around the OpenSolaris communities and projects. If your browser has 'flash' enabled, you will also see a graph of email activity per month, which shows the story of tremendous growth in interest around OpenSolaris. You will also see some fascinating statistics which tell you which are the most popular mail lists and which individuals have made the most posts.

Progress on FISHworks?

2008-09-16T22:04:00.005+01:00

Very little has been publicly announced by Sun about the FISHworks project.
After some initial news in early 2007, it seemed to go quiet.

Based on those reports, I would speculate that FISHworks appears to be a project to create a NAS/SAN appliance, based on existing OpenSolaris source code for ZFS, CIFS server, NFS server and the iScsi target, using DTrace observability to instrument under the hood.

At the start of September 2008, some more details about FISHworks emerged
when one of Sun's top engineers Mike Shapiro gave a video interview, bizarrely, on the noisy streets of San Francisco! It's a pity that this interview was conducted outside of a bar, as the background noise is somewhat distracting!

So when can we expect a proper announcement from Sun on FISHworks. Well according to the New York Time, that will be in November.

So do you want to know what progress is being made?

Then I suggest you check out the OpenSolaris bugs database.
This link shows bug activity in the iScsi target code for the last 30 days.

In the 'Reported Against' field, you will often see things like fw_36. This I believe is referring to FISHworks build 36.

Looking at the bugs for the last 30 days, there seems to be extensive & intensive testing happening of the OpenSolaris iScsi target, using various iScsi initiators and various operating systems, like Vista & Linux.

Some of the bugs looks quite obscure, occurring in relatively complex scenarios. Many of the bug comments refer to 'GRITS' and 'Diskomizer'. These seem to be Sun's internals tools for stress testing storage systems. Again, there does not seem to be much public information on these tools, but I googled and found these snippets on GRITS and Diskomizer.

It's good to see that in GRITS and Diskomizer, Sun seems to have the tools that
can expose these bugs. It's also good that these bugs are being fixed quickly.
Sun's Tim Szeto looks to have been particularly busy, and making good progress.

Some of the bug descriptions mention the 'Iwashi NAS appliance'
-So what it that? I would speculate maybe the code name for a Sun appliance?

'Java.exe' and Oracle SQL Developer

2008-08-20T12:14:00.001+01:00

I've just upgraded to v1.5.1 of 'Oracle SQL Developer'. When you first start the program, it will ask for the path to 'java.exe'. If you need to change or reset that path to a different value, you need to locate text file 'sqldeveloper.conf' and edit the value of setting 'SetJavaHome'. You may want to change this setting if you try to use an 'Unsupported Java Version'.

OpenSolaris and the HP ML-115

2008-08-15T00:27:00.007+01:00

The HP ML-115 is a very low cost server, that is ideal for trying out VMware ESX:
"Using the HP ML115 as a Test Server"
"Building a Low Cost (Cheap) VMware ESX Test Server"

So maybe the ML-115 would also be suitable for OpenSolaris.
So I tried it out, but there are some driver issues.
The good news is that these should be fixed very soon.

Problem #1 -If you try to boot the Open Solaris 2008.05 live CD, it fails, with error "Console login service(s) cannot run". See this thread.

The problem is that the ML-115's DVD drive is connected to a SATA port on the nVidia MCP55 chipset, and the nv_sata driver has only support for disk drives and not CD/DVD drives. But now Bug 6595488 is fixed, if you use build 95.

Ok, so with build 95, you should be able to boot from the DVD drive and install OpenSolaris. I tried this with "Solaris Express Community Edition" and was successful. You should then see something like this:


# cfgadm -a | grep sata | grep configured
sata0/0::dsk/c0t0d0            disk         connected    configured   ok
sata1/0::dsk/c4t0d0            disk         connected    configured   ok
sata2/0::dsk/c1t0d0            cd/dvd       connected    configured   ok
# rmformat -l
Looking for devices...
     1. Logical Node: /dev/rdsk/c1t0d0p0
        Physical Node: /pci@0,0/pci103c,1714@5,2/cdrom@0,0
        Connected Device: HL-DT-ST DVD-RAM GH15L    FA01
        Device Type: DVD Reader/Writer
 Size: 2.9 GB
# cdrw -l
Looking for CD devices...
    Node            Connected Device            Device type
----------------------+--------------------------------+-----------------
 cdrom0               | HL-DT-ST DVD-RAM GH15L    FA01 | CD Reader/Writer
# modinfo | grep nv_sata
 45 fffffffff7883000   6458 207   1  nv_sata (Nvidia ck804/mcp55 HBA v1.7)

Problem #2 - X will not run to give a graphical interface. If you check the Xerrors file, you will see:


(EE) MGA(0): Given color and fb depth combination not supported by this driver
(EE) Screen(s) found, but none have a usable configuration.

The graphics chipset on ML-115 is identified as "MGA G200e [Pilot] ServerEngines"
aka "mgag200 SE A PCI" aka "vendor 0x102b device 0x0522" and uses the MGA driver, in file "mga_drv.so". The version of the MGA driver supplier with OpenSolaris is version 1.4.6

I tried booting a Linux LiveCD, GRML v1.1, which uses
version 1.4.7 of the MGA driver and this worked without problem on the ML-115.

The OpenSolaris MGA driver has recently been updated. See Bug 6735095 and that fix should be available in build 97. Let's hope that fixes the X problem.

Problem #3 - No network card detected. The ML-115 network card is identified as
"Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express" aka "vendor 0x14e4 device 0x165a". The fix for this should be available in build 96 - see Bug 6726056.

Updated ISO for OpenSolaris to build 95 are available now. For an update to build 97, I guess that means a wait of a further 4 weeks.

ISDN Caller ID and Cisco ISDN routers

2008-06-20T15:16:00.004+01:00

We recently had a ISDN-2e line installed at work. But I forgot to ask BT (British Telecom) for 'caller ID' aka CLID to be enabled. BT terminology for this is the 'Calling Line Identity Presentation' or CLIP service, and BT will only enable it if you specifically ask for it, and there is an additional charge for the service.

Without the number of the calling party, the Cisco router does not know which 'Dialer profile' to associate with the incoming call, and rejects the call.

I kept a record of the log from the Cisco router, before and after enabling CLID, so here we can see the differences.

While doing the testing & troubleshooting, I had debugging enable on the cisco router as follows:


#show debug
Dial on demand:
  Dial on demand events debugging is on

The following ISDN debugs are enabled on all DSLs:

debug isdn error is             ON.
debug isdn event is             ON.
debug isdn q931 is              ON.

First an incoming data call, before BT enabled CLID on the ISDN line:


ISDN BR0 Q931: RX <- SETUP pd = 8  callref = 0x01
        Sending Complete
        Bearer Capability i = 0x8890
                Standard = CCITT
                Transfer Capability = Unrestricted Digital
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0x89
        Called Party Number i = 0x81, '67xxxx'
                Plan:ISDN, Type:Unknown
ISDN BR0 EVENT: process_rxstate: ces/callid 1/0x14 calltype 1 HOST_INCOMING_CALL
ISDN BR0:1: Incoming call rejected, unbindable
ISDN BR0 **ERROR**: host_incoming_call: DIALER ERROR 0x1: b channel 0, call id 0x14
ISDN BR0 EVENT: process_rxstate: ces/callid 1/0x14 calltype 1 HOST_DISCONNECT_ACK
ISDN BR0 Q931: TX -> RELEASE_COMP pd = 8  callref = 0x81
        Cause i = 0x8095 - Call rejected

After BT enabled CLID, which was done within a couple of hours, the log shows this:


ISDN BR0 Q931: RX <- SETUP pd = 8  callref = 0x01
        Sending Complete
        Bearer Capability i = 0x8890
                Standard = CCITT
                Transfer Capability = Unrestricted Digital
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0x89
        Calling Party Number i = 0x2183, '142761xxxx'
                Plan:ISDN, Type:National
        Called Party Number i = 0x81, '67xxxx'
                Plan:ISDN, Type:Unknown
ISDN BR0 EVENT: process_rxstate: ces/callid 1/0x17 calltype 1 HOST_INCOMING_CALL
BR0:1 DDR: Caller id 142761xxxx matched to profile
%DIALER-6-BIND: Interface BR0:1 bound to profile Di1

I also tried making a voice call to the ISDN number, which I knew would fail, just to see what it looked like. Here is before CLID was enabled:


ISDN BR0 Q931: RX <- SETUP pd = 8  callref = 0x01
        Sending Complete
        Bearer Capability i = 0x8090A3
                Standard = CCITT
                Transfer Capability = Speech
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0x89
        Called Party Number i = 0x81, '67xxxx'
                Plan:ISDN, Type:Unknown
ISDN BR0 EVENT: process_rxstate: ces/callid 1/0x11 calltype 2 HOST_INCOMING_CALL
ISDN BR0 **ERROR**: host_incoming_call: Received a call with a bad bearer cap from <unknown> on B1

And here is a voice call, after CLID enabled:


ISDN BR0 Q931: RX <- SETUP pd = 8  callref = 0x01
        Sending Complete
        Bearer Capability i = 0x8090A3
                Standard = CCITT
                Transfer Capability = Speech
                Transfer Mode = Circuit
                Transfer Rate = 64 kbit/s
        Channel ID i = 0x89
        Calling Party Number i = 0x2183, '142768xxxx'
                Plan:ISDN, Type:National
        Called Party Number i = 0x81, '67xxxx'
                Plan:ISDN, Type:Unknown
ISDN BR0 EVENT: process_rxstate: ces/callid 1/0x19 calltype 2 HOST_INCOMING_CALL
ISDN BR0 **ERROR**: host_incoming_call: Received a call with a bad bearer cap from 142768xxxx on B1

By the way, the Cisco router was a 1721 using IOS v12.3


#show version
Cisco Internetwork Operating System Software
IOS (tm) C1700 Software (C1700-Y-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by cisco Systems, Inc.
Compiled Mon 17-Mar-08 14:24 by dchih

ROM: System Bootstrap, Version 12.2(7r)XM1, RELEASE SOFTWARE (fc1)
System returned to ROM by power-on
System restarted at 12:53:16 GMT Wed May 21 2008
System image file is "flash:c1700-y-mz.123-26.bin"

cisco 1721 (MPC860P) processor (revision 0x100) with 28231K/4537K bytes of memory.
MPC860P processor: part number 5, mask 2
Bridging software.
X.25 software, Version 3.0.0.
Basic Rate ISDN software, Version 1.1.
1 FastEthernet/IEEE 802.3 interface(s)
1 Serial(sync/async) network interface(s)
1 ISDN Basic Rate interface(s)
32K bytes of non-volatile configuration memory.
16384K bytes of processor board System flash (Read/Write)

Oracle, Access & ODBC: Problems & Solution

2008-06-12T11:00:00.002+01:00

I was using Microsoft Access, in this case Access 97, to connect to an Oracle 10g database, using the ability of Access to set linked tables using an ODBC data source.

For the client, I was using the Oracle Instant Client v10.1.0.4 and setting an environmental variable NLS_LANG.

I was seeing two problems. First, when trying to link to the tables/views on the Oracle server, after selecting the ODBC data source DSN, Access would list all the tables/views on the Oracle server, but It would only display the first character of the name of the tables/views, making it impossible to find the correct table/view.

Second, when I tried setting up the Oracle linked tables in the Access MDB file on another PC, where everything was working ok, and then going back to the original problem PC and opening same MDB file, when opening the Oracle linked tables, all rows and all fields contained #Deleted.

I did a Google and found this Microsoft KnowledgeBase article - Q913070:
#Deleted is displayed in the records when you open a linked ODBC table from an Oracle 10g database in Access 2003, in Access 2002, or in Access 2000

To cut a long story short, I had set the client side environmental variable

NLS_LANG = ENGLISH_UNITED KINGDOM.WE8ISO8859P1

but this did not match the setting on the Oracle database, and I was advised
by our Oracle DBA that I should have been using:

NLS_LANG = ENGLISH_UNITED KINGDOM.WE8MSWIN1252

Apparently, our Oracle DBA had changed the NLS_CHARACTERSET of the database at some stage in the past.

After I corrected the value of NLS_LANG, everything starting working ok. It seems WE8MSWIN1252 is a binary super set of WE8ISO8859P1.

See these links for further background:

Oracle NLS_LANG FAQ

Quotation marks changing in web forms, from being pasted from MS Word

Changing Database Character Set

My new motherboard: Discovering hardware with 'grml'

2008-05-14T23:59:00.003+01:00

I've bought a new motherboard for a new PC I'm building. It's intended to run OpenSolaris with the ZFS file system and the Solaris iScsi target. But to kick-off, I'm going to start by taking a look at the system, running a Linux Live-CD distribution called 'grml'.

Gigabyte seem to have generated some confusion over the numbering of this motherboard. My board it labelled 'GA-P35C-DS3R v2.1'. A very similar motherboard, the 'GA-EP35C-DS3R' has been reviewed this month in 'PCpro' magazine here in the UK, and was selected as 'Labs Winner'.

So what is the difference between these two similarly named motherboards. Other are asking the same question, and the answer seems to be 'there is no difference'. They are both 'dynamic energy saver' motherboards.

Grml originates from Germany, and is based on a Debian. It boot's directly to a command line prompt, which I love, as this is where the real work is done.

On any new hardware I get my hands on, I like to boot up grml, and use it to poke around & identify what sort of hardware we have for hard drive interfaces and networking.

One of the reasons I picked this Gigabyte motherboard was that it used the Intel ICH9R chipset to provide six AHCI compatible SATA interfaces. In addition it has a JMicron interface chip, which can either provide two more AHCI sata ports, or alternatively run in legacy mode with old style IDE drives.

Again to kick off, I am running the JMicron in legacy mode, with a DVD drive
and a IDE drive hanging off of the ribbon cable.

Ok, lets take a look:


root@grml ~ # uname -a
Linux grml 2.6.23-grml64 #1 SMP PREEMPT Mon Feb 11 16:49:32 CET 2008 x86_64 GNU/Linux
root@grml ~ # lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller [8086:29c0] (rev 02)
00:01.0 PCI bridge [0604]: Intel Corporation 82G33/G31/P35/P31 Express PCI Express Root Port [8086:29c1] (rev 02)
00:1a.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 [8086:2937] (rev 02)
00:1a.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 [8086:2938] (rev 02)
00:1a.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 [8086:2939] (rev 02)
00:1a.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 [8086:293c] (rev 02)
00:1b.0 Audio device [0403]: Intel Corporation 82801I (ICH9 Family) HD Audio Controller [8086:293e] (rev 02)
00:1c.0 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1 [8086:2940] (rev 02)
00:1c.3 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 4 [8086:2946] (rev 02)
00:1c.4 PCI bridge [0604]: Intel Corporation 82801I (ICH9 Family) PCI Express Port 5 [8086:2948] (rev 02)
00:1d.0 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 [8086:2934] (rev 02)
00:1d.1 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 [8086:2935] (rev 02)
00:1d.2 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 [8086:2936] (rev 02)
00:1d.7 USB Controller [0c03]: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 [8086:293a] (rev 02)
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 92)
00:1f.0 ISA bridge [0601]: Intel Corporation 82801IR (ICH9R) LPC Interface Controller [8086:2916] (rev 02)
00:1f.2 SATA controller [0106]: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA AHCI Controller [8086:2922] (rev 02)
00:1f.3 SMBus [0c05]: Intel Corporation 82801I (ICH9 Family) SMBus Controller [8086:2930] (rev 02)
01:00.0 VGA compatible controller [0300]: nVidia Corporation NV44 [Quadro NVS 285] [10de:0165] (rev a1)
03:00.0 SATA controller [0106]: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller [197b:2363] (rev 02)
03:00.1 IDE interface [0101]: JMicron Technologies, Inc. JMicron 20360/20363 AHCI Controller [197b:2363] (rev 02)
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 01)

Here are some 'highlights' from the 'dmesg' command, showing I have a
Samsung SATA drive on one of the ICH9 ports, and a Western Digital drive
on the JMicron IDE port.


root@grml ~ # dmesg
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
JMB363: IDE controller at PCI slot 0000:03:00.1
PCI: Enabling device 0000:03:00.1 (0000 -> 0001)
ACPI: PCI Interrupt 0000:03:00.1[B] -> GSI 16 (level, low) -> IRQ 16
JMB363: chipset revision 2
JMB363: 100% native mode on irq 16
Probing IDE interface ide0...
hda: SAMSUNG CD-ROM SC-148C, ATAPI CD/DVD-ROM drive
hdb: WDC WD800JB-00CRA1, ATA DISK drive
hda: Disabling (U)DMA for SAMSUNG CD-ROM SC-148C (blacklisted)
hdb: selected mode 0x45
ahci 0000:00:1f.2: version 2.3
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 19
ahci 0000:00:1f.2: AHCI 0001.0200 32 slots 6 ports 3 Gbps 0x3f impl SATA mode
ahci 0000:00:1f.2: flags: 64bit ncq sntf stag pm led clo pmp pio slum part
PCI: Setting latency timer of device 0000:00:1f.2 to 64
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
scsi4 : ahci
scsi5 : ahci
ata1: SATA max UDMA/133 cmd 0xffffc20000026100 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 507
ata2: SATA max UDMA/133 cmd 0xffffc20000026180 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 507
ata3: SATA max UDMA/133 cmd 0xffffc20000026200 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 507
ata4: SATA max UDMA/133 cmd 0xffffc20000026280 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 507
ata5: SATA max UDMA/133 cmd 0xffffc20000026300 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 507
ata6: SATA max UDMA/133 cmd 0xffffc20000026380 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 507
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: ATA-7: SAMSUNG HD400LJ, ZZ100-15, max UDMA7
ata1.00: 781422768 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata1.00: configured for UDMA/133
ata2: SATA link down (SStatus 0 SControl 300)
ata3: SATA link down (SStatus 0 SControl 300)
ata4: SATA link down (SStatus 0 SControl 300)
ata5: SATA link down (SStatus 0 SControl 300)
ata6: SATA link down (SStatus 0 SControl 300)
scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HD400LJ  ZZ10 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 781422768 512-byte hardware sectors (400088 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 19 (level, low) -> IRQ 19
ahci 0000:03:00.0: AHCI 0001.0000 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
ahci 0000:03:00.0: flags: 64bit ncq pm led clo pmp pio slum part
PCI: Setting latency timer of device 0000:03:00.0 to 64
scsi6 : ahci
scsi7 : ahci
ata7: SATA max UDMA/133 cmd 0xffffc20000028100 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 19
ata8: SATA max UDMA/133 cmd 0xffffc20000028180 ctl 0x0000000000000000 bmdma 0x0000000000000000 irq 19
ata7: SATA link down (SStatus 0 SControl 300)
ata8: SATA link down (SStatus 0 SControl 300)
r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded
eth0: RTL8168b/8111b at 0xffffc20000030000, 00:1a:4d:5e:b9:ba, XID 38500000 IRQ 16
r8169: eth0: link up

Ok, that's enough excitement for this time, as I'm out of time.
See you next time...

Vista SP1 crashes Linux L2TP daemon

2008-03-25T15:56:00.006Z

In the last week, one of our users has had problems connecting to our VPN, which uses Smoothwall and the Linux L2TP daemon. Until recently this user had not experienced a problem.

I checked our Smoothwall box, and found that the L2TP daemon (l2tpd) was not running. But after restarting it, it crashed again as soon as the user tried to connect. Other users could connect without problem.

Here is the tail end of the log from /var/log/messages:


l2tpd version 0.70-smoothwall started on smoothwallAF1 PID:28094
Linux version 2.4.34.5-up on a i686, listening on IP address 0.0.0.0, port 1701
<--snip-->
handle_avps: handling avp's for tunnel 61598, call 0
message_type_avp: message type 10 (Incoming-Call-Request)
message_type_avp: new incoming call
ourcid = 13433, entropy_buf = 3479
assigned_session_avp: assigned session id: 1
call_serno_avp: serial number is 0
bearer_type_avp: peer bears: analog
handle_avps: Bad exit status handling attribute 1 (Result Code).
Segmentation fault      (core dumped) /modules/tunnel/usr/sbin/l2tpd

So, why suddenly a problem?
The Windows XP users could connect ok. The Windows Vista users could connect ok.

Ahh! The user who was having problems had recently applied Vista SP1.

Using tcpdump, I captured the packets arriving at Smoothwall on UDP port 1701, and then examined the capture file using WireShark. The problem seems to occur on the ICRQ packet, which is the 'Incomming Call Request', transmitted by the Windows client to the Smoothwall server.

For Windows XP clients, we see the following AVP's (Attribute Value pair):
'Control Message', 'Assigned Session', 'Call Serial Number' and 'Bearer Type'.

For the Windows Vista SP1 client, there was an extra AVP tagged onto the end. This is a 'Vendor-Specific' AVP of 'Type 1', specifying a 'Vendor ID' of 311 (0x0137), meaning 'Microsoft'.

So what is this extra Microsoft AVP? A quick google finds a Cisco document, implying it may be a related to RADIUS. It talks about Vendor-Specific Attributes (VSAs). Table 36 lists Vendor-Specific RADIUS IETF Attributes with Vendor company code 311 and Sub-type 1 which is a "MSCHAP-Response" attribute.

Ok, so it seems the version of l2tpd we are using does not like the presence of this extra AVP, and mistakes it for a 'Result Code' AVP, which should only be present in CDN and StopCCN messages.

The relavent RFC is: RFC2261 - "Layer Two Tunneling Protocol".

Looking at the top of page 50 of the RFC, for an ICRQ, it lists the AVPs that MUST be present, and the AVPs that MAY be present. This would seem to indicate that a vendor specific AVP is NOT valid at this stage.

Looking at page 12 of the RFC, section 4.1, which details the format of the AVP, implies that the problem could be fixed if l2tpd were to look at the 'Vendor ID' field, and ignore the AVP in the ICRQ where the Vendor ID was non zero.

Looking at the code for l2tpd, in the file "avp.c" and in function "handle_avps()" there is no code that checks to confirm that (avp->vendorid) is zero.

At the moment we do not have a fix, other than to un-install Vista SP1.

Windbg Symbols for ntoskrnl.exe

2008-02-28T15:57:00.003Z

In this case the OS was Windows Vista 64-bit with SP1 RTM applied,
so the kernel was "6001.18000.amd64fre.longhorn_rtm.080118-1840".

I tried setting WinDbg to load sysmbols across the Internet
from Microsoft's symbol server. But I got errors like the following:


 ERROR: Module load completed but symbols could not be loaded for ntoskrnl.exe
 Kernel symbols are WRONG. Please fix symbols to do analysis.
 Your debugger is not using the correct symbols
 Type referenced: nt!_KPRCB

So I decided to download the symbols, from here:
http://www.microsoft.com/whdc/devtools/debugging/symbolpkg.mspx

I extracted them onto my hard drive, updated my symbols path,
but I still got the same errors.

I discovered that to fix this, you need to go into
the SYMBOLS\EXE folder and copy file 'ntkrnlmp.pdb'
to 'ntoskrnl.pdb'.

Calling Oracle Stored Procedure with PHP

2008-01-18T11:39:00.000Z

I've just been following this excellent tutorial article
on the Oracle WebSite:
Calling Oracle Stored Procedure with PHP

I'm only just starting to use Oracle PL/SQL and PHP,
and it's still early days. So I'm making really stupid
mistakes and getting errors like this:

Warning: oci_execute() [function.oci-execute]: ORA-06550: line 1, column 7:
PLS-00201: identifier 'ROLLROUTE' must be declared ORA-06550: line 1, column 7:

Here was my code:

 $sql = "BEGIN RollRoute(:PartNo); END;";

 $s = oci_parse($c, $sql );
 oci_bind_by_name($s,':PartNo',$PartNo,32);
 oci_execute($s);

The fix was simple:

 $sql = "BEGIN RROUTING_API.RollRoute(:PartNo); END;";

Yes, the Procedure is in a Package.
So you need to prefix the procedure name, with the package name.
Being a newbie, it look me a while to spot that!

WMI and "Permission denied"

2007-08-30T12:30:00.000+01:00

Today we were trying to use vbscript and WMI to audit the software installed on PCs connected to our network. The PC's were running Windows XP Pro, and were members of an Active Directory Domain.

The script worked fine in most cases, but we occasionally got errors like these:

Error 462 on GetObject()
"The remote server machine does not exist or is unavailable"

Error 70 on GetObject()
"Permission denied"

The line of code in question was:

Set objReg = GetObject("winmgmts:{impersonationLevel=impersonate,authenticationLevel=Pkt}!//" & strComputer & sDNS & "/root/default:StdRegProv")

To cut a long story short, we used WireShark to capture the conversation between the source and target PC's.

Here is a RPC packet sent from the source to the target PC.
The target PC is "PC-ONE-LAP".

DCE RPC Bind, Fragment: Single, FragLen: 1440, Call: 2
    Version: 5
    Version (minor): 0
    Packet type: Bind (11)
    Packet Flags: 0x03
    Data Representation: 10000000
    Frag Length: 1440
    Auth Length: 1360
    Call ID: 2
    Max Xmit Frag: 5840
    Max Recv Frag: 5840
    Assoc Group: 0x0000b254
    Num Ctx Items: 1
    Ctx Item[1]: ID:1
        Context ID: 1
        Num Trans Items: 1
        Abstract Syntax: ISystemActivator V0.0
            Interface: ISystemActivator UUID: 000001a0-0000-0000-c000-000000000046
            Interface Ver: 0
            Interface Ver Minor: 0
        Transfer Syntax[1]: 8a885d04-1ceb-11c9-9fe8-08002b104860 V2
    Auth type: SPNEGO (9)
    Auth level: Connect (2)
    Auth pad len: 0
    Auth Rsrvd: 0
    Auth Context ID: 1277840
    GSS-API Generic Security Service Application Program Interface
        OID: 1.3.6.1.5.5.2 (SPNEGO - Simple Protected Negotiation)
        SPNEGO
            negTokenInit
                mechTypes: 3 items
                    Item: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5)
                    Item: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
                    Item: 1.3.6.1.4.1.311.2.2.10 (NTLMSSP - Microsoft NTLM Security Support Provider)
                mechToken: 6E82050A30820506A003020105A10302010EA20703050020...
                krb5_blob: 6E82050A30820506A003020105A10302010EA20703050020...
                    Kerberos AP-REQ
                        Pvno: 5
                        MSG Type: AP-REQ (14)
                        Padding: 0
                        APOptions: 20000000 (Mutual required)
                        Ticket
                            Tkt-vno: 5
                            Realm: COMPANY.NET
                            Server Name (Service and Instance): RPCSS/PC-ONE-LAP.company.net
                            enc-part rc4-hmac
                        Authenticator rc4-hmac

You can see it is using kerberos to authenticate.
Here is the response from the target PC:

DCE RPC Bind_ack, Fragment: Single, FragLen: 199, Call: 2
    Version: 5
    Version (minor): 0
    Packet type: Bind_ack (12)
    Packet Flags: 0x03
    Data Representation: 10000000
    Frag Length: 199
    Auth Length: 131
    Call ID: 2
    Max Xmit Frag: 5840
    Max Recv Frag: 5840
    Assoc Group: 0x0000b254
    Scndry Addr len: 4
    Scndry Addr: 135
    Num results: 1
    Context ID[1]
    Auth type: SPNEGO (9)
    Auth level: Connect (2)
    Auth pad len: 0
    Auth Rsrvd: 0
    Auth Context ID: 1277840
    GSS-API Generic Security Service Application Program Interface
        SPNEGO
            negTokenTarg
                negResult: accept-incomplete (1)
                supportedMech: 1.2.840.48018.1.2.2 (MS KRB5 - Microsoft Kerberos 5)
                responseToken: 606606092A864886F71201020203007E573055A003020105...
                krb5_blob: 606606092A864886F71201020203007E573055A003020105...
                    KRB5 OID: 1.2.840.113554.1.2.2 (KRB5 - Kerberos 5)
                    krb5_tok_id: KRB5_ERROR (0x0003)
                    Kerberos KRB-ERROR
                        Pvno: 5
                        MSG Type: KRB-ERROR (30)
                        stime: 2007-08-30 10:19:06 (Z)
                        susec: 851504
                        error_code: KRB5KRB_AP_ERR_MODIFIED (41)
                        Realm: COMPANY.NET
                        Server Name (Principal): PC-TWO-LAP$

The response show an Kerberos error "KRB5KRB_AP_ERR_MODIFIED", as it is basically saying - hold on, my name is "PC-TWO-LAP" and not "PC-ONE-LAP".

In the System Event Log for the source PC, you will see the following:

Event Type: Error
Event Source: DCOM
Event Category: None
Event ID: 10009
Date:  30/08/2007
Time:  10:22:52
User:  COMPANY\USER
Computer: SOURCEPC
Description:
DCOM was unable to communicate with the computer PC-ONE-LAP.company.net
 using any of the configured protocols.

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 4
Date:  30/08/2007
Time:  10:22:52
User:  N/A
Computer: SOURCEPC
Description:
The kerberos client received a KRB_AP_ERR_MODIFIED error from the
 server PC-TWO-LAP$.  This indicates that the password used to encrypt
 the kerberos service ticket is different than that on the target server.
 Commonly, this is due to identically named  machine accounts in the
  target realm (COMPANY.NET), and the client realm.

The explanation is that the DNS server was returning the same IP address for both "PC-ONE-LAP" and "PC-TWO-LAP". Only "PC-TWO-LAP" was actually connected to the network, but when we tried "PC-ONE-LAP", it was actually "PC-TWO-LAP" that responded.

Kerberos authenticates both the User and the Computer, so that is why we got the error. If you get this error, then check your DNS records.

For other similar problems you may find this link useful:
Failed to get remote resources: Remote server is unavailable. The RPC server is unavailable.

A problem with wget on OpenSolaris

2007-08-27T16:26:00.000+01:00

Ok, here is a stupid little problem.I waisted an hour trying to figure this one out. Maybe this will help someone to avoid the same mistake!

I was using 'wget' to try to download a file from a http server onto my OpenSolaris PC. Here is what I did:

# uname -a
SunOS solaris 5.11 snv_60 i86pc i386 i86pc
# wget -V
GNU Wget 1.10.2
# cd /home
# /usr/sfw/bin/wget http://www.nwsmith.net/index.htm
--18:31:13--  http://www.nwsmith.net/index.htm
           => `index.htm'
Resolving www.nwsmith.net... NNN.NNN.NNN.NNN
Connecting to www.nwsmith.net|NNN.NNN.NNN.NNN|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11,774 (11K) [text/html]
index.htm: Operation not applicable
Cannot write to `index.htm' (Operation not applicable).

Did you spot my mistake?
I must have thought I was using a Linux PC, because then the '/home' directory would have been fine. But on Solaris...

# ls -ld /home
dr-xr-xr-x   1 root     root           1 Apr 18 00:03 /home

...you cannot write to that directory.
Choose a directory that is writeable, and then wget will work without error.

fdisk, sfdisk, mdadm and SCSI hard drive geometry

2007-08-16T15:27:00.000+01:00

At work, one of our servers, uses Linux software RAIDand we have two mirrored hard drives setup as RAID1.
Smartmontools reported that one of the hard drives was starting to fail:

# smartctl -a /dev/sda
SMART Health Status: LOGICAL UNIT FAILURE PREDICTION THRESHOLD EXCEEDED [asc=5d,ascq=2]
# smartctl -l selftest /dev/sda
SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Failed in segment -->       2 10572  0x         22f0e6c [0x4 0x40 0x85]
# 2  Background long   Failed in segment -->       2 10404  0x         22f0e6c [0x3 0x11 0x0]
# 3  Background long   Failed in segment -->       2 10236  0x         22f0e63 [0x4 0x40 0x85]

So we executed the following commands:

mdadm --manage /dev/md0 --set-faulty  /dev/sda1
mdadm --manage /dev/md0 --remove /dev/sda1
mdadm --manage /dev/md1 --set-faulty /dev/sda2
mdadm --manage /dev/md1 --remove /dev/sda2
mdadm --manage /dev/md2 --set-faulty /dev/sda3
mdadm --manage /dev/md2 --remove /dev/sda3
mdadm --manage /dev/md3 --set-faulty /dev/sda5
mdadm --manage /dev/md3 --remove /dev/sda5

And then hot un-plugged the drive.
When the replacement drive arrived, although it was an identical model
to the original, and had an identical total size, the new drive had a different geometry.

[root@ifsclstr02 ~]# smartctl -i /dev/sda
Device: IBM-ESXS BBD036C3ESTT0ZFN Version: JP86
Device type: disk
Transport protocol: Parallel SCSI (SPI-4)

[root@ifsclstr02 ~]# smartctl -i /dev/sdb
Device: IBM-ESXS BBD036C3ESTT0ZFN Version: JP85
Device type: disk
Transport protocol: Parallel SCSI (SPI-4)

# fdisk -l /dev/sda
Disk /dev/sda: 36.4 GB, 36401479680 bytes
64 heads, 32 sectors/track, 34715 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

# fdisk -l /dev/sdb
Disk /dev/sdb: 36.4 GB, 36401479680 bytes
255 heads, 63 sectors/track, 4425 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

The potential problem was that if we had to specify the start and end of the partition in terms of cylinders, then we would not be able to get an exact match in the size of the partitions between the new disk and the existing working half of the mirror.
After some googling, we concluded that having to align the partition boundaries with the cylinders was a DOS legacy issue, and was not something that would cause a problem for Linux.
So to copy the partitions from the working disk to the new disk we used the following:

# sfdisk -d /dev/sdb | sfdisk --Linux /dev/sda
Checking that no-one is using this disk right now ...
OK
Disk /dev/sda: 34715 cylinders, 64 heads, 32 sectors/track
Old situation:
Units = cylinders of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1          0       -       0          0    0  Empty
/dev/sda2          0       -       0          0    0  Empty
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sda1   *        63    208844     208782  fd  Linux raid autodetect
/dev/sda2        208845  16980704   16771860  fd  Linux raid autodetect
/dev/sda3      16980705  25366634    8385930  fd  Linux raid autodetect
/dev/sda4      25366635  71087624   45720990   5  Extended
/dev/sda5      25366698  71087624   45720927  fd  Linux raid autodetect
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Warning: partition 3 does not start at a cylinder boundary
Warning: partition 3 does not end at a cylinder boundary
Warning: partition 4 does not start at a cylinder boundary
Warning: partition 4 does not end at a cylinder boundary
Warning: partition 5 does not end at a cylinder boundary
Successfully wrote the new partition table
Re-reading the partition table ...
# mdadm --manage /dev/md0 --add /dev/sda1
# mdadm --manage /dev/md1 --add /dev/sda2
# mdadm --manage /dev/md2 --add /dev/sda3
# mdadm --manage /dev/md3 --add /dev/sda5
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda2[0] sdb2[1]
      8385856 blocks [2/2] [UU]

md2 : active raid1 sda3[2] sdb3[1]
      4192896 blocks [2/1] [_U]
        resync=DELAYED
md3 : active raid1 sda5[2] sdb5[1]
      22860352 blocks [2/1] [_U]
      [============>........]  recovery = 61.8% (14148928/22860352) finish=2.1min speed=67707K/sec
md0 : active raid1 sda1[0] sdb1[1]
      104320 blocks [2/2] [UU]

unused devices:

Using the sfdisk command, you can specify the unit of measure when listing the partition table. Use '-uS' for Sectors and '-uC' for Cylinders:

# sfdisk -l -uS /dev/sda
Disk /dev/sda: 34715 cylinders, 64 heads, 32 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sda1   *        63    208844     208782  fd  Linux raid autodetect
/dev/sda2        208845  16980704   16771860  fd  Linux raid autodetect
/dev/sda3      16980705  25366634    8385930  fd  Linux raid autodetect
/dev/sda4      25366635  71087624   45720990   5  Extended
/dev/sda5      25366698  71087624   45720927  fd  Linux raid autodetect

# sfdisk -l -uS /dev/sdb
Disk /dev/sdb: 4425 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *        63    208844     208782  fd  Linux raid autodetect
/dev/sdb2        208845  16980704   16771860  fd  Linux raid autodetect
/dev/sdb3      16980705  25366634    8385930  fd  Linux raid autodetect
/dev/sdb4      25366635  71087624   45720990   5  Extended
/dev/sdb5      25366698  71087624   45720927  fd  Linux raid autodetect

It's only when you think in terms of cylinders, that there appears to be a problem:

# sfdisk -l -uC /dev/sda
Disk /dev/sda: 34715 cylinders, 64 heads, 32 sectors/track
Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.
Units = cylinders of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1   *      0+    101-    102-    104391   fd  Linux raid autodetect
/dev/sda2        101+   8291-   8190-   8385930   fd  Linux raid autodetect
/dev/sda3       8291+  12386-   4095-   4192965   fd  Linux raid autodetect
/dev/sda4      12386+  34710-  22325-  22860495    5  Extended
/dev/sda5      12386+  34710-  22325-  22860463+  fd  Linux raid autodetect

# sfdisk -l -uC /dev/sdb
Disk /dev/sdb: 4425 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdb1   *      0+     12      13-    104391   fd  Linux raid autodetect
/dev/sdb2         13    1056    1044    8385930   fd  Linux raid autodetect
/dev/sdb3       1057    1578     522    4192965   fd  Linux raid autodetect
/dev/sdb4       1579    4424    2846   22860495    5  Extended
/dev/sdb5       1579+   4424    2846-  22860463+  fd  Linux raid autodetect

The pluses and minuses, just mean that the numbers are not exact and are rounded up or down.
For comparison, here is what fdisk reports:

# fdisk -l /dev/sda
Disk /dev/sda: 36.4 GB, 36401479680 bytes
64 heads, 32 sectors/track, 34715 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1         102      104391   fd  Linux raid autodetect
Partition 1 does not end on cylinder boundary.
/dev/sda2             102        8292     8385930   fd  Linux raid autodetect
/dev/sda3            8292       12387     4192965   fd  Linux raid autodetect
/dev/sda4           12387       34711    22860495    5  Extended
/dev/sda5           12387       34711    22860463+  fd  Linux raid autodetect

# fdisk -l /dev/sdb
Disk /dev/sdb: 36.4 GB, 36401479680 bytes
255 heads, 63 sectors/track, 4425 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   fd  Linux raid autodetect
/dev/sdb2              14        1057     8385930   fd  Linux raid autodetect
/dev/sdb3            1058        1579     4192965   fd  Linux raid autodetect
/dev/sdb4            1580        4425    22860495    5  Extended
/dev/sdb5            1580        4425    22860463+  fd  Linux raid autodetect

Smartmontools and fixing Unreadable Disk Sectors

2007-08-08T17:26:00.001+01:00

Smartmontools was showing some problems on the disk.
At least two bad LBAs:

# smartctl -l selftest /dev/hda
smartctl version 5.36 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       20%      1596         44724966
# 2  Extended offline    Completed: read failure       40%      1519         12622427

# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       2
196 Reallocated_Event_Count 0x0008   252   252   000    Old_age   Offline      -       1
197 Current_Pending_Sector  0x0008   253   253   000    Old_age   Offline      -       2
198 Offline_Uncorrectable   0x0008   252   252   000    Old_age   Offline      -       1

I found a document:
"SHOWS HOW TO IDENTIFY THE FILE ASSOCIATED
WITH AN UNREADABLE DISK SECTOR, AND HOW TO
FORCE THAT SECTOR TO REALLOCATE."
and followed the procedure.

Note the LBA values are given as decimal values. The document seems to refer
to an older version of smarctl that gives the LBA as a hexadecimal number.

Lets look at the partition sizes to see where this LBA drops in.

# fdisk -lu /dev/hda

Disk /dev/hda: 255 heads, 63 sectors, 3738 cylinders
Units = sectors of 1 * 512 bytes

Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *        63    160649     80293+  83  Linux
/dev/hda2        160650   1204874    522112+  82  Linux swap
/dev/hda3       1204875  53737424  26266275   83  Linux
/dev/hda4      53737425  60050969   3156772+   f  Win95 Ext'd (LBA)
/dev/hda5      53737488  55841939   1052226   83  Linux
/dev/hda6      55842003  57946454   1052226   83  Linux
/dev/hda7      57946518  60050969   1052226   83  Linux

Ok, so the problem is in '/dev/hda3 '.
What's mounted there? As the partitions are labeled, we need to use:

# grep `e2label /dev/hda3` /etc/fstab
LABEL=/var              /var                    ext3    defaults        1 2

Ok, so the problem is in '/var'.

# tune2fs -l /dev/hda3 | grep Block
Block count:              6566568
Block size:               4096

Let's do the maths:
LBA 12622427 - 1204875 and multiply by (512/4096) equals 1427194.
LBA 44724966 - 1204875 and multiply by (512/4096) equals 5440011.375

Ok, now let's use 'debugfs':

# debugfs
debugfs 1.27 (8-Mar-2002)
debugfs:  open /dev/hda3
debugfs:  icheck 1427194
Block   Inode number
1427194 526482
debugfs:  ncheck 526482
Inode   Pathname
526482  /log/ntp/peers.20070717
debugfs:  icheck 5440011
icheck: Can't read next inode while doing inode scan
debugfs:  quit

So that means LBA 12622427 is in file "/var/log/ntp/peers.20070717".
And it looks like LBA 44724966 is in currently unused space on the disk.

As this file is not critical, I will just overwrite part of it
to force it to be reallocated:

# dd if=/dev/zero of=/dev/hda3 bs=4096 count=1 seek=1427194
1+0 records in
1+0 records out
# sync
# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       1
196 Reallocated_Event_Count 0x0008   252   252   000    Old_age   Offline      -       1
197 Current_Pending_Sector  0x0008   253   253   000    Old_age   Offline      -       1
198 Offline_Uncorrectable   0x0008   252   252   000    Old_age   Offline      -       1

Ok, that seems to have made that error go away for the time being.

Then while googling I found this perl script to help with
automation of badblocks on Linux:

"smartfixdisk - assistant that helps to repair bad LBAs detected by Smartmontools"
developed by the "IT-Support-Group" (ISG.EE),
which is a service organisation of the
"Department of Information Technology and Electrical Engineering" (D-ITET)
of the "Swiss Federal Institute of Technology", Zurich.

I wanted to use it on a old RedHat 9 server.
The script immediately fell over on this line:

open(DISKEND,"</sys/block/$diskname/size") or die "$!";

Not too surprising, as the '/sys' does not exist on my old server!
It seems to be a feature of newer kernels.
On a Centos-5 box I tried this:

# cat /proc/ide/hda/capacity
78165360
# cat /sys/block/hda/size
78165360
# cat /proc/ide/hda/geometry
physical     16383/16/63
logical      65535/16/63

So DISKEND seems to be related to the number of sectors on the hard drive.
The '/proc' version was available on the old Redhat 9 server,
so I change the perl code line like this:

open(DISKEND,"</proc/ide/$diskname/capacity") or die "$!";

..and it was happy.
To figure out what the script is doing, it's useful to add in a few
'print' commands into the script, or just uncomment the ones that
already in place. Here's what the script told me on this server:

# ./smartfixdisk.pl --noaction /dev/hda
Block size = 4096, factor = 0.125
Searching for inode... this may take a while...

LBA 12622427
    Partition and partition type: /dev/hda3 Linux_Ext2
    Status: used
    Comment: EXT2/3: File found at inode 526482: /log/ntp/peers.20070717

LBA 44724966
    Partition and partition type: /dev/hda3 Linux_Ext2
    Status: free
    Comment: block not used in filesystem
    dd if=/dev/zero of=/dev/hda seek=5590620 bs=4096 count=1 conv=sync

Looks good.
Ok, so let's finish off:

# dd if=/dev/zero of=/dev/hda seek=5590620 bs=4096 count=1 conv=sync
1+0 records in
1+0 records out
# sync
# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       1
196 Reallocated_Event_Count 0x0008   252   252   000    Old_age   Offline      -       1
197 Current_Pending_Sector  0x0008   253   253   000    Old_age   Offline      -       0
198 Offline_Uncorrectable   0x0008   252   252   000    Old_age   Offline      -       1

# smartctl -t long /dev/hda

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".
Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 17 minutes for test to complete.
Test will complete after Wed Aug  8 16:38:15 2007

Use smartctl -X to abort test.
# smartctl -l selftest /dev/hda
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      1599         -
# 2  Extended offline    Completed: read failure       20%      1596         44724966
# 3  Extended offline    Completed: read failure       40%      1519         12622427

# smartctl -A /dev/hda | egrep 'Reallocated|Pending|Uncorrectable'
5 Reallocated_Sector_Ct   0x0033   253   253   063    Pre-fail  Always       -       1
196 Reallocated_Event_Count 0x0008   252   252   000    Old_age   Offline      -       1
197 Current_Pending_Sector  0x0008   253   253   000    Old_age   Offline      -       0
198 Offline_Uncorrectable   0x0008   253   252   000    Old_age   Offline      -       0

Ok, that looks to have cleared the errors for the time being.
But I'm going to keep a careful eye on that disk, using smartd and logwatch.

Using 'prtpci' to list PCI devices

2007-03-25T00:30:00.000Z

On Solaris, if you want to check the details of the hardware installed
in your PC, you can use the command 'prtconf -pv'. However, the output is
very long, and gives too great a level of detail. Sun's Dan Mick has written
a perl program, called 'prtpci' which gives a more useful summary.
Here is how I installed the program:


bash-3.00# wget ftp://playground.sun.com/pub/dmick/prtpci.tar.Z
--00:08:09--  ftp://playground.sun.com/pub/dmick/prtpci.tar.Z
           => `prtpci.tar.Z'
Resolving playground.sun.com... 192.9.5.5
Connecting to playground.sun.com|192.9.5.5|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD /pub/dmick ... done.
==> PASV ... done.    ==> RETR prtpci.tar.Z ... done.
Length: 163,050 (159K) (unauthoritative)

100%[====================================>] 163,050       57.69K/s

00:08:16 (57.54 KB/s) - `prtpci.tar.Z' saved [163050]

bash-3.00# gunzip < prtpci.tar.Z | tar xvf -
x prtpci, 5954 bytes, 12 tape blocks
x pciids/pci.ids, 349313 bytes, 683 tape blocks
x pciids/class.ids, 3445 bytes, 7 tape blocks
x pciids/getnew.pci.ids, 81 bytes, 1 tape blocks

bash-3.00# chown -R root:root prtpci pciids/

bash-3.00# ls -l
total 350
drwxr-xr-x   2 root     root         512 Mar 25 00:10 pciids
-r-xr-xr-x   1 root     root        5954 Jul 12  2005 prtpci
-rw-r--r--   1 root     root      163050 Mar 25 00:08 prtpci.tar.Z

bash-3.00# ls -l pciids/
total 714
-rw-rw-r--   1 root     root        3445 Mar 24  2002 class.ids
-rwxrwxr-x   1 root     root          81 Mar 18  2005 getnew.pci.ids
-rw-rw-r--   1 root     root      349313 May 27  2005 pci.ids

And here is the (edited) output of what it shows
on my Dell Precision 380:


bash-3.00# ./prtpci
0/0x0/0 8086,2774 (1028,1a8) rev 0x0
Intel Corporation 955X Memory Controller Hub
class 6/0/0: Bridge/Host bridge

0/0x1/0 8086,2775 0  rev 0x0
Intel Corporation 955X PCI Express Graphics Port
class 6/4/0: Bridge/PCI bridge

1/0x0/0 1002,71d2 (1002,3b02) rev 0x0
ATI Technologies Inc
class 3/0/0: Display controller/VGA compatible controller
BAR[0]: prefetchable 64-bit memory 0xe0000000 0x10000000
BAR[2]: 64-bit memory 0xfe9e0000 0x10000
BAR[4]: I/O 0xdc00 0x100
legacy reg #3: aliased I/O 0x3b0 0xc
legacy reg #4: aliased I/O 0x3c0 0x20
legacy reg #5: 32-bit memory 0xa0000 0x20000

1/0x0/1 1002,71f2 (1002,3b03) rev 0x0
ATI Technologies Inc
class 3/80/0: Display controller/Display controller
BAR[0]: 64-bit memory 0xfe9f0000 0x10000

0/0x1c/0 8086,27d0 1  rev 0x0
Intel Corporation 82801G (ICH7 Family) PCI Express Port 1
class 6/4/0: Bridge/PCI bridge

4/0x0/0 14e4,1677 (1028,1a8) rev 0x1
Broadcom Corporation NetXtreme BCM5751 Gigabit Ethernet PCI Express
class 2/0/0: Network controller/Ethernet controller
BAR[0]: 64-bit memory 0xfe7f0000 0x10000

0/0x1f/0 8086,27b8 1  rev 0x0
Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge
class 6/1/0: Bridge/ISA bridge

0/0x1f/1 8086,27df (1028,1a8) rev 0x1
Intel Corporation 82801G (ICH7 Family) IDE Controller
class 1/1/8a: Mass storage controller/IDE interface
BAR[0]: I/O 0x1f0 0x8
BAR[1]: I/O 0x3f6 0x1
BAR[2]: I/O 0x170 0x8
BAR[3]: I/O 0x376 0x1
BAR[4]: I/O 0xffa0 0x10

0/0x1f/2 8086,27c1 (1028,1a8) rev 0x1
Intel Corporation 82801GR/GH (ICH7 Family) Serial ATA Storage Controllers cc=AHCI
class 1/6/1: Mass storage controller/
BAR[0]: I/O 0xfe00 0x8
BAR[1]: I/O 0xfe10 0x4
BAR[2]: I/O 0xfe20 0x8
BAR[3]: I/O 0xfe30 0x4
BAR[4]: I/O 0xfea0 0x10
BAR[5]: 32-bit memory 0xfebfbc00 0x400

0/0x1f/3 8086,27da (1028,1a8) rev 0x1
Intel Corporation 82801G (ICH7 Family) SMBus Controller
class c/5/0: Serial bus controller/SMBus
BAR[4]: I/O 0xece0 0x20

The above clearly shows the ATI graphics card, the Broadcom Gigabit Ethernet card, and the ICH7 SATA card (using AHCI).

Here's the link to the post on Dan Mick's blog.

Using wget, gcc & gmake on Solaris, by fixing the PATH

2007-03-24T23:58:00.000Z

Solaris has a number of useful commands, like gcc and wget, hidden away in
directory '/usr/sfw/bin'. But, by default, for user root, Solaris does not add this
directory to the path. So assuming you are using Bash as your shell,
try the following:


bash-3.00# echo $PATH
/usr/sbin:/usr/bin
bash-3.00# export PATH=$PATH:/usr/sfw/bin
bash-3.00# echo $PATH
/usr/sbin:/usr/bin:/usr/sfw/bin
bash-3.00# wget -V
GNU Wget 1.10.2
bash-3.00# gcc -v
gcc version 3.4.3 (csl-sol210-3_4-20050802)
bash-3.00# gmake -v
GNU Make 3.80
bash-3.00# uname -a
SunOS solaris 5.11 snv_57 i86pc i386 i86pc

How to waste half a day with EDI

2007-02-23T18:05:00.000Z

Yes, thats EDI or Electronic Data Interchange.

The problem for me, is that one of our big customers wants to stop sending us Odette DELINS messages, and instead start using Edifact DELFOR messages. It's a problem for me, because I need to work out how to do it!

So let's start by seeing if we can translate a simple DELFOR file into the in-house format to feed into our ERP system. Let's created the test file, by copying & pasting the sample provided in the pdf of the specification. Simple... No...it does not work. Why? Every thing looks ok!

After hours of double checking & going around in circles. Finally the answer.

In an EDI file, each line is terminated with a ' character - thats ASCII code 0x27. But I finally realised that the lines in my file were terminated with character code 0x92, which looks very similar in the character set I was using.

When you use the correct terminating character, it all works a lot better!
So todays lesson is - be careful when you copy & paste text from a pdf!

Checking Solaris iScsi Performance with Dtrace

2007-02-21T00:24:00.000Z

I've just discovered David Weibel's Blog and he has a couple of interesting posts regarding using dtrace(1M) to check performance issues with iScsi on Solaris.

iSCSI Kernel Visibilty

iSCSI device reporting lots of iSCSI sessions logging in and out

And David show a way to disable MPxIO for the Solaris iSCSI initiator here.

David worked on the Solaris iScsi initiator code when he was at Sun Microsystems, Inc.

Using the NetBSD iScsi Target code on Solaris

2007-01-17T00:43:00.000Z

The NetBSD iScsi target will, quite happily, compile and run on Solaris. The code was developed by Alistair Crooks, and is based on code released by Intel, under the BSD licence.

To try it out, I used the Belenix LiveCD distribution of OpenSolaris.

(root)# uname -a
SunOS belenix 5.11 BeleniX0.4.3 i86pc i386 i86pc
(root)# gcc -v
Reading specs from /usr/foss/lib/gcc/i386-pc-solaris2.10/3.4.3/specs
Thread model: posix
gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)
(root)# cd tmp
(root)# curl -O http://www.alistaircrooks.co.uk/src/netbsd-iscsi-20060526.tar.gz
(root)# ls -l netbsd-iscsi-20060526.tar.gz
-rw-------   1 root     root      239964 May 27 22:21 netbsd-iscsi-20060526.tar.gz
(root)# gunzip < netbsd-iscsi-20060526.tar.gz | tar xvf -
(root)# cd iscsi
(root)# cd src
(root)# ./configure
(root)# make
(root)# cd ../bin
(root)# ls -l
-rwx------   1 root     root      226580 May 27 22:24 iscsi-harness
 -rwx------   1 root     root      219028 May 27 22:24 iscsi-target

I just wanted to use the iScsi target to confirm if the Microsoft iScsi initiator would work ok with it, and take some Ethereal traces of the initial login sequence. So I just used the following simple configuration file, which exports out a 100 mega byte file.

(root)# cat /etc/iscsi/targets
extent0         /tmp/iscsi-target0      0       100MB

target0         rw      extent0         0.0.0.0/0
(root)# ./iscsi-target &
(root)# ls -l /tmp/iscsi-target0
-rw-------   1 root     root     104857601 May 27 23:17 /tmp/iscsi-target0

I tried the target with v2.03 of the Microdoft iScsi initiator, and and I can report that every thing I tried worked ok.

You can browse the source code of the iScsi target at the NetBSD CVS server. To get the latest version of the code, it may be best to get it from the CVS server.

I've just noted that Alistair make some further changes to the code at the start of 2007, which are noted at this digest.

Linux iScsi Target and the Inquiry LUN Response

2007-01-14T20:58:00.000Z

Ok, last time we completed compiling and installing the Linux iScsi target. Now we need to configure it and start it. All I wanted to do, at this stage, was a simple test. So I just did the minimal amount of configuration. I was going to be running the target on Centos-4, running under VmWare, so it was easy to add a second virtual hard drive to the environment, in this case '/dev/sdb', to export out from the target.

# cat /etc/ietd.conf
#IncomingUser nwsmith secretsecret
Target iqn.2007-01.net.nwsmith:test01
Lun 0 /dev/sdb fileio
Alias iscsitest01

# service iscsi-target start
Starting iSCSI target service:    [  OK  ]

# netstat -ntlp | grep 3260
tcp  0  0  0.0.0.0:3260  0.0.0.0:*  LISTEN 3160/ietd

# tail /var/log/messages
Jan 14 15:59:02 localhost kernel: iSCSI Enterprise Target Software - version 0.4.5
Jan 14 15:59:02 localhost kernel: iotype_init(91) register fileio
Jan 14 15:59:03 localhost kernel: target_param(109) d 1 8192 262144 65536 2 20 8 0
Jan 14 15:59:03 localhost iscsi-target: ietd startup succeeded

The Linux iScsi target seems to work fine with the Microsoft iScsi initiator (v2.03).

Ok, now we have the target running, one of the thing I wanted to look at was how the Linux iScsi target handled the initial login and connection from the Microsoft initiator. I was looking for how it handles the "Inquiry LUN" packet. I used Ethereal to capture and analyse the packets - filter on port 3260.

I had a look at the source code, and found what I was looking for in file "target_disk.c" and in function:

static int build_inquiry_response(struct iscsi_cmnd *cmnd)

By editing the values in this function, you can adjust the following fields in the response packet:

Version, Flags, Vendor Id:, Product Id:, Product Revision Level:

I tried changing some of these fields in the code, recompiling, restarting the target, and then used Ethereal to check that the response (from the target) to the (initiators) 'Inquiry' LUN packet changed as expected. And it did what I expected. (To change the Version returned, edit the line "data[2] = 3;").

The Version field indicates which version of SCSI is supported by the target. You can check which values are valid by reading the specification. The specification is called "SCSI Primary Commands - 3" or SPC-3 and you can download it from www.t10.org.

In the case of the version field, the valid values are shown in table 84:

0x00 - The device does not claim conformance to any standard.
0x02 - Obsolete.
0x03 - The device complies to ANSI INCITS 301-1997 (SPC).
0x04 - The device complies to ANSI INCITS 351-2001 (SPC-2).
0x05 - The device complies to T10/1416-D (SPC-3).

Ok, I think that's enough detail for this post.

Centos 4 and the Linux iScsi Target

2007-01-13T17:07:00.000Z

I wanted to do some tests with the Microsoft iScsi initiator and the Linux iScsi target, but first I had to install and configure the target. I had Centos 4.4 (a RHEL4 Linux clone) setup in VmWare, so i thought that using that would be the easy way to do the test.

I had read in PcPro magazine, last year, issue 137 (March 2006) page 197, that the Linux iScsi target would compile & install on Centos-4, but they did not say exactly which version they used.

Ok, so I download the latest version "iscsitarget-0.4.14.tar.gz".

I knew there was a potential problem, as the README file said it needed kernel version of 2.6.14 or newer, and the CONFIG_CRYPTO_CRC32C option had to be enabled. Centos 4.4 uses kernel 2.6.9, but with many additional patches from latter kernels, and the PCpro people said it would be ok.

Ok, so lets try it:


# cat /etc/redhat-release
CentOS release 4.4 (Final)
# uname -a
Linux localhost.localdomain 2.6.9-42.0.3.EL #1 Fri Oct 6 05:59:54 CDT 2006 i686 i686 i386 GNU/Linux
# cat /usr/src/kernels/2.6.9-42.0.3.EL-i686/crypto/Kconfig | grep -A 6 CRC
config CRYPTO_CRC32C
       tristate "CRC32c CRC algorithm"
       depends on CRYPTO
       select LIBCRC32C
       help
         Castagnoli, et al Cyclic Redundancy-Check Algorithm.  Used
         by iSCSI for header and data digests and by others.
         See Castagnoli93.  This implementation uses lib/libcrc32c.
         Module will be crc32c.

# cd /home
# tar xfz iscsitarget-0.4.14.tar.gz
# chown -R root:root iscsitarget-0.4.14
# ll
drwxr-xr-x  7 root root  4096 Oct 19 10:48 iscsitarget-0.4.14
-rw-r--r--  1 root root 92608 Mar 14  2006 iscsitarget-0.4.14.tar.gz
# cd iscsitarget-0.4.14
# export KERNELSRC=/usr/src/kernels/2.6.9-42.0.3.EL-i686/
# make
make -C usr
make[1]: Entering directory `/home/iscsitarget-0.4.14/usr'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o ietd.o ietd.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o iscsid.o iscsid.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o conn.o conn.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o session.o session.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o target.o target.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o message.o message.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o ctldev.o ctldev.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o log.o log.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o chap.o chap.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o event.o event.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o param.o param.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o plain.o plain.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o isns.o isns.c
cc ietd.o iscsid.o conn.o session.o target.o message.o ctldev.o log.o chap.o event.o param.o plain.o isns.o -o ietd -lcrypto
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o ietadm.o ietadm.c
cc ietadm.o param.o -o ietadm
make[1]: Leaving directory `/home/iscsitarget-0.4.14/usr'
make -C /lib/modules/2.6.9-42.0.3.EL/build SUBDIRS=/home/iscsitarget-0.4.14/kernel modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'
 CC [M]  /home/iscsitarget-0.4.14/kernel/tio.o
 CC [M]  /home/iscsitarget-0.4.14/kernel/iscsi.o
 CC [M]  /home/iscsitarget-0.4.14/kernel/nthread.o
 CC [M]  /home/iscsitarget-0.4.14/kernel/wthread.o
 CC [M]  /home/iscsitarget-0.4.14/kernel/config.o
/home/iscsitarget-0.4.14/kernel/config.c:312: error: unknown field `unlocked_ioctl' specified in initializer
/home/iscsitarget-0.4.14/kernel/config.c:312: warning: initialization from incompatible pointer type
/home/iscsitarget-0.4.14/kernel/config.c:313: error: unknown field `compat_ioctl' specified in initializer
/home/iscsitarget-0.4.14/kernel/config.c:313: warning: initialization from incompatible pointer type
make[2]: *** [/home/iscsitarget-0.4.14/kernel/config.o] Error 1
make[1]: *** [_module_/home/iscsitarget-0.4.14/kernel] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'
make: *** [mods] Error 2

# cat /home/iscsitarget-0.4.14/kernel/config.c | grep -n ioctl
219:static long ioctl(struct file *file, unsigned int cmd, unsigned long arg)
312:    .unlocked_ioctl = ioctl,
313:    .compat_ioctl   = ioctl,

Oops! It failed to compile.
Ok, let's google around and see who else has this problem...

It seems other people are getting the same sort of problem with the target versions 4.13 and 4.12 and the older Centos kernel 2.6.9-22.EL

I found this link, where you can download a patch for the target version 0.4.12, which should allow it to compile on kernel 2.6.9, but I did not try it, as it's a "quick hack - Use at your own risk". Maybe someone else can try this patch and give a report.

Ok, so should I dump Centos-4 for now and install Fedora 6, which has an up-to-date kernel. Or should I try and work out which version of the target it was that the Pcpro guy's were using? Ok, I choose the latter.

I went back and downloaded the older versions of the target. And to cut a long story short, I discovered that this version "iscsitarget-0.4.5.tar.gz" will compile on Centos-4. (If you look at the README for that version of the target, it says it is happy with kernel 2.6.10)
Here is the proof that it compiles:


# tar xfz iscsitarget-0.4.5.tar.gz
# chown -R root:root iscsitarget-0.4.5
# cd iscsitarget-0.4.5
# make
make -C usr
make[1]: Entering directory `/home/iscsitarget-0.4.5/usr'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o ietd.o ietd.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o iscsid.o iscsid.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o conn.o conn.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o session.o session.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o target.o target.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o message.o message.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o ctldev.o ctldev.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o log.o log.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o md5.o md5.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o isns.o isns.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o sha1.o sha1.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o chap.o chap.c
cc ietd.o iscsid.o conn.o session.o target.o message.o ctldev.o log.o md5.o isns.o sha1.o chap.o -o ietd
cc -O2 -fno-inline -Wall -Wstrict-prototypes -g -I../include   -c -o ietadm.o ietadm.c
cc ietadm.o ctldev.o -o ietadm
make[1]: Leaving directory `/home/iscsitarget-0.4.5/usr'
make -C /usr/src/kernels/2.6.9-42.0.3.EL-i686/ SUBDIRS=/home/iscsitarget-0.4.5/kernel modules
make[1]: Entering directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'
  CC [M]  /home/iscsitarget-0.4.5/kernel/tio.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/iscsi.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/nthread.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/wthread.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/config.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/digest.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/conn.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/session.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/target.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/volume.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/iotype.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/file-io.o
  CC [M]  /home/iscsitarget-0.4.5/kernel/target_disk.o
  LD [M]  /home/iscsitarget-0.4.5/kernel/iscsi_trgt.o
  Building modules, stage 2.
  MODPOST
  CC      /home/iscsitarget-0.4.5/kernel/iscsi_trgt.mod.o
  LD [M]  /home/iscsitarget-0.4.5/kernel/iscsi_trgt.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.9-42.0.3.EL-i686'

# make install
`usr/ietd' -> `/usr/sbin/ietd'
`usr/ietadm' -> `/usr/sbin/ietadm'
if [ -f /etc/debian_version ]; then \
        install -v -m 755 etc/initd/initd.debian /etc/init.d/iscsi-target; \
elif [ -f /etc/redhat-release ]; then \
        install -v -m 755 etc/initd/initd.redhat /etc/init.d/iscsi-target; \
elif [ -f /etc/slackware-version ]; then \
        install -v -m 755 etc/initd/initd /etc/rc.d/iscsi-target; \
else \
        install -v -m 755 etc/initd/initd /etc/init.d/iscsi-target; \
fi
`etc/initd/initd.redhat' -> `/etc/init.d/iscsi-target'
install: creating directory `/lib/modules/2.6.9-42.0.3.EL/kernel/iscsi'
`kernel/iscsi_trgt.ko' -> `/lib/modules/2.6.9-42.0.3.EL/kernel/iscsi/iscsi_trgt.ko'
depmod -aq
#

Version 0.4.5 was released on 2005-02-21, so that PcPro article must have been on the editors desk for a year!

Ok, I still need to see if it will work, but that something for another post!

Vista 3-D Flip

2006-11-22T16:57:00.000Z

We installed Vista for the first time today. Of course the 'killer' feature we all wanted to see was the 'Windows Flip 3D', where it stacks the windows side-on & allows you to scroll through them. But how do you activate it?

After much searching, our resident Vista evangelist 'Chris (BrightNorth) Martin' came up with the answer. Press and hold the Control Key, the Windows Key and the Tab Key. Or for a slightly different alternative, press and hold the Windows Key and the Tab Key.

Chris provided this link which is the official explanation of how to do it from Microsoft.

Even latter, we found there was a button on the 'quick launch' bar, called "Switch between Windows". Just click that and it flips the windows into the 3D stack. How did we miss that button??

First post

2006-08-23T01:05:00.000+01:00

Ok, this is something I've been going to do for quite some time - well over a year. I was going to use Wordpress, but I just cannot find the free time to finish getting it going. And I have a load of information waiting to be posted to a blog, so I've (finally) decided to get something up and running quickly with "blogger.com". It looks like I've picked a good time to do this (hopefully), as they have a new Beta version that I'm going to try out. Seems ok so far so lets click the "publish" buttons and take a look...