OpenSolaris 2009-06 and the Comstar iScsi Target

Having upgraded from OpenSolaris 2008-11 to 2009-06, I wanted to try the new Comstar iscsi target, to see how well it worked, and to see if it was ready to replace the 'old' iscsi target. And I wanted to see how far I could get with the package versions in the 'production-quality' release repository, based on snv_111. To start with, I just wanted to get something basic going as quickly as possible.

The first tasks was to disable the old iscsi target (to avoid a conflict with the new target which would also want to listen on TCP port 3260), and to enable the 'SCSI Target Mode Framework' - stmf.

# svcs iscsitgt
online         Jul_09   svc:/system/iscsitgt:default
# svcs stmf
disabled       Jul_09   svc:/system/stmf:default
# svcadm disable iscsitgt
# svcs iscsitgt
STATE          STIME    FMRI
disabled       21:29:57 svc:/system/iscsitgt:default
# svcadm enable stmf
# svcs stmf
STATE          STIME    FMRI
online         21:30:55 svc:/system/stmf:default
# stmfadm list-state
Operational Status: online
Config Status     : initialized

Next I created a new small zfs volume.
(My zfs pool is named 'rz2pool' - it's Raidz2 - dual parity.)

# zfs create -V 4g rz2pool/iscsi_lun1

And then created a 'logical unit', using the zvol as it's backing store.

# sbdadm create-lu /dev/zvol/rdsk/rz2pool/iscsi_lun1

Created the following LU:

             GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f00008278f04694a5cf0980001      4294901760       /dev/zvol/rdsk/rz2pool/iscsi_lun1

# stmfadm list-lu -v
LU Name: 600144F00008278F04694A5CF0980001
   Operational Status: Online
   Provider Name     : sbd
   Alias             : /dev/zvol/rdsk/rz2pool/iscsi_lun1
   View Entry Count  : 0

Now using the GUID of the logical unit, we make it visible to any initiators, by adding a 'view'

# stmfadm add-view 600144f00008278f04694a5cf0980001
# stmfadm list-view -l 600144f00008278f04694a5cf0980001
View Entry: 0
   Host group   : All
   Target group : All
   LUN          : 0

Next we need to do the iscsi part, for which we need to install the iscsi target and the 'itadm' command. These were not already installed so I had to add the 'SUNWiscsit' package.

# pkg install -v SUNWiscsit
Creating Plan | Before evaluation:
UNEVALUATED:
+pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z

After evaluation:
None -> pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z
None -> pkg:/SUNWiscsidm@0.5.11,5.11-0.111:20090508T161041Z
Actuators:
     restart_fmri: svc:/system/manifest-import:default
None
DOWNLOAD                                    PKGS       FILES     XFER (MB)
Completed                                    2/2       23/23     0.66/0.66

PHASE                                        ACTIONS
Install Phase                                  76/76
PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                                2/2

# pkg list -v SUNWiscsit
FMRI                                                             STATE      UFIX
pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z               installed  ----
# pkg list -v SUNWiscsidm
FMRI                                                             STATE      UFIX
pkg:/SUNWiscsidm@0.5.11,5.11-0.111:20090508T161041Z              installed  ----

On the next step I had a small problem.
(BTW, I am not using NWAM to configure my network card.)

# svcadm enable -r iscsi/target:default
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

# svcs -a | grep network/physical
disabled       Jul_09   svc:/network/physical:nwam
online         Jul_09   svc:/network/physical:default

# svcadm enable  iscsi/target
# svcs -a | grep iscsi
disabled       Jul_09   svc:/network/iscsi_initiator:default
disabled       21:29:57 svc:/system/iscsitgt:default
maintenance    22:29:13 svc:/network/iscsi/target:default

# svcs -xv
svc:/network/iscsi/target:default (iscsi target)
State: maintenance since Tue Jul 14 22:29:13 2009
Reason: Start method failed repeatedly, last exited with status 4.
  See: http://sun.com/msg/SMF-8000-KS
  See: man -M /usr/share/man -s 1M itadm
  See: /var/svc/log/network-iscsi-target:default.log
Impact: This service is not running.

# tail /var/svc/log/network-iscsi-target\:default.log
[ Jul 14 22:29:13 Enabled. ]
[ Jul 14 22:29:13 Executing start method ("/lib/svc/method/iscsi-target start"). ]
iscsi-target: Requesting to enable iscsi target
open failed: INVALIDUnable to open device /devices/pseudo/iscsit@0:iscsit[ Jul 14 22:29:13 Method "start" exited with status 4. ]

Rebooting solved this problem.

# svcs -a | grep iscsi
disabled       19:25:15 svc:/network/iscsi_initiator:default
disabled       19:25:18 svc:/system/iscsitgt:default
online         19:25:37 svc:/network/iscsi/target:default

So now we create the iscsi target and check it looks ok:

# itadm create-target
Target iqn.1986-03.com.sun:02:7e969860-cb8d-475a-ddbb-97bf2f22ce7b successfully created

# itadm list-target -v
TARGET NAME                                                  STATE    SESSIONS
iqn.1986-03.com.sun:02:7e969860-cb8d-475a-ddbb-97bf2f22ce7b  online   0
       alias:                  -
       auth:                   none (defaults)
       targetchapuser:         -
       targetchapsecret:       unset
       tpg-tags:               default

I was then able to discover and logon to this iscsi target from a Windows 2003 server, using the Microsoft iscsi initiator.

The above is just the simplest possible configuration, just to allow some simple tests. For a more practical setup, you may well want multiple logical units, and want to put some restrictions on the views. But details on how to do that will have to wait for a follow-up post to this blog!

I did notice a couple of things fairly quickly. I saw the connection between the initiator and target was dropping every 50 seconds, followed immediately by a re-logon. Checking with WireShark showed that it was the target starting this with a [FIN, ACK]. With the old iscsi target I would have used DTrace to confirm this, but the iscsi target provider for the new iscsi target was not available.

I looked at the source code for the comstar iscsi target:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/comstar/port/iscsit/iscsit.c

.. and from the history link, I could see that a couple of likely fixes had already been committed:

26-May-2009 Priya Krishnan - 6809997 COMSTAR iscsi target DTrace Provider needed
08-May-2009 Peter Dunlap - 6755803 win2003 initiator numerous iscsi connection lost and connection retries mesgs to iscsi target

Now normally, I would hyper-link those bug number for you. But in this case there is no point, because Sun has chosen not to allows details of those bugs to be publicly available. (Or any of the other comstar iscsi target bugs AFAIK.)

Having those fixes is important for me, so it looks like I will have to update my OpenSolaris 2009-06 using the development repository to snv_118.

A new way to search & browse the OpenSolaris Mail Lists

You can now search and browse through over a quarter million messages posted to the OpenSolaris Mail-lists using a free service called 'MarkMail'.

Head over to http://opensolaris.markmail.org/ and take a look.

I stumbled across the MarkMail web site last week,via Google. It seems to have been running for about 12 months, and in that time they have loaded the mail archives of thousands of open-source projects into their system.

But until last week, they did not include anything on OpenSolaris. A shocking omission! So I left feedback, and I was quickly contacted by MarkMail's Jason Hunter. Jason wasted no time in subscribing their system to most of the OpenSolaris lists, and then within a few days they had loaded the 'MailMan' archive files. (If you would like to see further lists added then leave feedback on the MarkMail site via their form.)

By going to this link http://opensolaris.markmail.org/search/?q= you can see the latest emails that have been added into the system. I find this a useful way of just getting some idea of the vast amount of activity being generated around the OpenSolaris communities and projects. If your browser has 'flash' enabled, you will also see a graph of email activity per month, which shows the story of tremendous growth in interest around OpenSolaris. You will also see some fascinating statistics which tell you which are the most popular mail lists and which individuals have made the most posts.

Progress on FISHworks?

Very little has been publicly announced by Sun about the FISHworks project.
After some initial news in early 2007, it seemed to go quiet.

Based on those reports, I would speculate that FISHworks appears to be a project to create a NAS/SAN appliance, based on existing OpenSolaris source code for ZFS, CIFS server, NFS server and the iScsi target, using DTrace observability to instrument under the hood.

At the start of September 2008, some more details about FISHworks emerged
when one of Sun's top engineers Mike Shapiro gave a video interview, bizarrely, on the noisy streets of San Francisco! It's a pity that this interview was conducted outside of a bar, as the background noise is somewhat distracting!

So when can we expect a proper announcement from Sun on FISHworks. Well according to the New York Time, that will be in November.

So do you want to know what progress is being made?

Then I suggest you check out the OpenSolaris bugs database.
This link shows bug activity in the iScsi target code for the last 30 days.

In the 'Reported Against' field, you will often see things like fw_36. This I believe is referring to FISHworks build 36.

Looking at the bugs for the last 30 days, there seems to be extensive & intensive testing happening of the OpenSolaris iScsi target, using various iScsi initiators and various operating systems, like Vista & Linux.

Some of the bugs looks quite obscure, occurring in relatively complex scenarios. Many of the bug comments refer to 'GRITS' and 'Diskomizer'. These seem to be Sun's internals tools for stress testing storage systems. Again, there does not seem to be much public information on these tools, but I googled and found these snippets on GRITS and Diskomizer.

It's good to see that in GRITS and Diskomizer, Sun seems to have the tools that
can expose these bugs. It's also good that these bugs are being fixed quickly.
Sun's Tim Szeto looks to have been particularly busy, and making good progress.

Some of the bug descriptions mention the 'Iwashi NAS appliance'
-So what it that? I would speculate maybe the code name for a Sun appliance?

OpenSolaris and the HP ML-115

The HP ML-115 is a very low cost server, that is ideal for trying out VMware ESX:
"Using the HP ML115 as a Test Server"
"Building a Low Cost (Cheap) VMware ESX Test Server"

So maybe the ML-115 would also be suitable for OpenSolaris.
So I tried it out, but there are some driver issues.
The good news is that these should be fixed very soon.

Problem #1 -If you try to boot the Open Solaris 2008.05 live CD, it fails, with error "Console login service(s) cannot run". See this thread.

The problem is that the ML-115's DVD drive is connected to a SATA port on the nVidia MCP55 chipset, and the nv_sata driver has only support for disk drives and not CD/DVD drives. But now Bug 6595488 is fixed, if you use build 95.

Ok, so with build 95, you should be able to boot from the DVD drive and install OpenSolaris. I tried this with "Solaris Express Community Edition" and was successful. You should then see something like this:

# cfgadm -a | grep sata | grep configured
sata0/0::dsk/c0t0d0            disk         connected    configured   ok
sata1/0::dsk/c4t0d0            disk         connected    configured   ok
sata2/0::dsk/c1t0d0            cd/dvd       connected    configured   ok
# rmformat -l
Looking for devices...
     1. Logical Node: /dev/rdsk/c1t0d0p0
        Physical Node: /pci@0,0/pci103c,1714@5,2/cdrom@0,0
        Connected Device: HL-DT-ST DVD-RAM GH15L    FA01
        Device Type: DVD Reader/Writer
 Size: 2.9 GB
# cdrw -l
Looking for CD devices...
    Node            Connected Device            Device type
----------------------+--------------------------------+-----------------
 cdrom0               | HL-DT-ST DVD-RAM GH15L    FA01 | CD Reader/Writer
# modinfo | grep nv_sata
 45 fffffffff7883000   6458 207   1  nv_sata (Nvidia ck804/mcp55 HBA v1.7)

Problem #2 - X will not run to give a graphical interface. If you check the Xerrors file, you will see:

(EE) MGA(0): Given color and fb depth combination not supported by this driver
(EE) Screen(s) found, but none have a usable configuration.

The graphics chipset on ML-115 is identified as "MGA G200e [Pilot] ServerEngines"
aka "mgag200 SE A PCI" aka "vendor 0x102b device 0x0522" and uses the MGA driver, in file "mga_drv.so". The version of the MGA driver supplier with OpenSolaris is version 1.4.6

I tried booting a Linux LiveCD, GRML v1.1, which uses
version 1.4.7 of the MGA driver and this worked without problem on the ML-115.

The OpenSolaris MGA driver has recently been updated. See Bug 6735095 and that fix should be available in build 97. Let's hope that fixes the X problem.

Problem #3 - No network card detected. The ML-115 network card is identified as
"Broadcom Corporation NetXtreme BCM5722 Gigabit Ethernet PCI Express" aka "vendor 0x14e4 device 0x165a". The fix for this should be available in build 96 - see Bug 6726056.

Updated ISO for OpenSolaris to build 95 are available now. For an update to build 97, I guess that means a wait of a further 4 weeks.