Wednesday, July 29, 2009

OpenSolaris 2009-06 and the Comstar iScsi Target

Having upgraded from OpenSolaris 2008-11 to 2009-06, I wanted to try the new Comstar iscsi target, to see how well it worked, and to see if it was ready to replace the 'old' iscsi target. And I wanted to see how far I could get with the package versions in the 'production-quality' release repository, based on snv_111. To start with, I just wanted to get something basic going as quickly as possible.

The first tasks was to disable the old iscsi target (to avoid a conflict with the new target which would also want to listen on TCP port 3260), and to enable the 'SCSI Target Mode Framework' - stmf.

# svcs iscsitgt
online Jul_09 svc:/system/iscsitgt:default
# svcs stmf
disabled Jul_09 svc:/system/stmf:default
# svcadm disable iscsitgt
# svcs iscsitgt
STATE STIME FMRI
disabled 21:29:57 svc:/system/iscsitgt:default
# svcadm enable stmf
# svcs stmf
STATE STIME FMRI
online 21:30:55 svc:/system/stmf:default
# stmfadm list-state
Operational Status: online
Config Status : initialized

Next I created a new small zfs volume.
(My zfs pool is named 'rz2pool' - it's Raidz2 - dual parity.)

# zfs create -V 4g rz2pool/iscsi_lun1

And then created a 'logical unit', using the zvol as it's backing store.

# sbdadm create-lu /dev/zvol/rdsk/rz2pool/iscsi_lun1

Created the following LU:

GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
600144f00008278f04694a5cf0980001 4294901760 /dev/zvol/rdsk/rz2pool/iscsi_lun1

# stmfadm list-lu -v
LU Name: 600144F00008278F04694A5CF0980001
Operational Status: Online
Provider Name : sbd
Alias : /dev/zvol/rdsk/rz2pool/iscsi_lun1
View Entry Count : 0

Now using the GUID of the logical unit, we make it visible to any initiators, by adding a 'view'

# stmfadm add-view 600144f00008278f04694a5cf0980001
# stmfadm list-view -l 600144f00008278f04694a5cf0980001
View Entry: 0
Host group : All
Target group : All
LUN : 0

Next we need to do the iscsi part, for which we need to install the iscsi target and the 'itadm' command. These were not already installed so I had to add the 'SUNWiscsit' package.

# pkg install -v SUNWiscsit
Creating Plan | Before evaluation:
UNEVALUATED:
+pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z

After evaluation:
None -> pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z
None -> pkg:/SUNWiscsidm@0.5.11,5.11-0.111:20090508T161041Z
Actuators:
restart_fmri: svc:/system/manifest-import:default
None
DOWNLOAD PKGS FILES XFER (MB)
Completed 2/2 23/23 0.66/0.66

PHASE ACTIONS
Install Phase 76/76
PHASE ITEMS
Reading Existing Index 8/8
Indexing Packages 2/2

# pkg list -v SUNWiscsit
FMRI STATE UFIX
pkg:/SUNWiscsit@0.5.11,5.11-0.111:20090508T161047Z installed ----
# pkg list -v SUNWiscsidm
FMRI STATE UFIX
pkg:/SUNWiscsidm@0.5.11,5.11-0.111:20090508T161041Z installed ----

On the next step I had a small problem.
(BTW, I am not using NWAM to configure my network card.)

# svcadm enable -r iscsi/target:default
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

# svcs -a | grep network/physical
disabled Jul_09 svc:/network/physical:nwam
online Jul_09 svc:/network/physical:default

# svcadm enable iscsi/target
# svcs -a | grep iscsi
disabled Jul_09 svc:/network/iscsi_initiator:default
disabled 21:29:57 svc:/system/iscsitgt:default
maintenance 22:29:13 svc:/network/iscsi/target:default

# svcs -xv
svc:/network/iscsi/target:default (iscsi target)
State: maintenance since Tue Jul 14 22:29:13 2009
Reason: Start method failed repeatedly, last exited with status 4.
See: http://sun.com/msg/SMF-8000-KS
See: man -M /usr/share/man -s 1M itadm
See: /var/svc/log/network-iscsi-target:default.log
Impact: This service is not running.

# tail /var/svc/log/network-iscsi-target\:default.log
[ Jul 14 22:29:13 Enabled. ]
[ Jul 14 22:29:13 Executing start method ("/lib/svc/method/iscsi-target start"). ]
iscsi-target: Requesting to enable iscsi target
open failed: INVALIDUnable to open device /devices/pseudo/iscsit@0:iscsit[ Jul 14 22:29:13 Method "start" exited with status 4. ]

Rebooting solved this problem.

# svcs -a | grep iscsi
disabled 19:25:15 svc:/network/iscsi_initiator:default
disabled 19:25:18 svc:/system/iscsitgt:default
online 19:25:37 svc:/network/iscsi/target:default

So now we create the iscsi target and check it looks ok:

# itadm create-target
Target iqn.1986-03.com.sun:02:7e969860-cb8d-475a-ddbb-97bf2f22ce7b successfully created

# itadm list-target -v
TARGET NAME STATE SESSIONS
iqn.1986-03.com.sun:02:7e969860-cb8d-475a-ddbb-97bf2f22ce7b online 0
alias: -
auth: none (defaults)
targetchapuser: -
targetchapsecret: unset
tpg-tags: default

I was then able to discover and logon to this iscsi target from a Windows 2003 server, using the Microsoft iscsi initiator.

The above is just the simplest possible configuration, just to allow some simple tests. For a more practical setup, you may well want multiple logical units, and want to put some restrictions on the views. But details on how to do that will have to wait for a follow-up post to this blog!

I did notice a couple of things fairly quickly. I saw the connection between the initiator and target was dropping every 50 seconds, followed immediately by a re-logon. Checking with WireShark showed that it was the target starting this with a [FIN, ACK]. With the old iscsi target I would have used DTrace to confirm this, but the iscsi target provider for the new iscsi target was not available.

I looked at the source code for the comstar iscsi target:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/io/comstar/port/iscsit/iscsit.c

.. and from the history link, I could see that a couple of likely fixes had already been committed:

26-May-2009 Priya Krishnan - 6809997 COMSTAR iscsi target DTrace Provider needed
08-May-2009 Peter Dunlap - 6755803 win2003 initiator numerous iscsi connection lost and connection retries mesgs to iscsi target

Now normally, I would hyper-link those bug number for you. But in this case there is no point, because Sun has chosen not to allows details of those bugs to be publicly available. (Or any of the other comstar iscsi target bugs AFAIK.)

Having those fixes is important for me, so it looks like I will have to update my OpenSolaris 2009-06 using the development repository to snv_118.

4 comments:

Robo said...

This guide is awesome.
I was stuffing around with the old way of doing it and getting no where, I found your guide and it's just so soo sweet!

Thanks

Ryan said...

Thank you for posting solution.
This guide helped me.

Ryan said...

Thank you for your guide.
I can test iSCSI target

Good job

Vladdrac said...

Very nice, but I have a problem, how can you expand that lun if you define a fs with 4G? do you have to expand the fs as well?
thanks in advance.