Network Working Group                                     M. Chadalapaka
Request for Comments: 3783                                    R. Elliott
Category: Informational                              Hewlett-Packard Co.
                                                                May 2004


                Small Computer Systems Interface (SCSI)
               Command Ordering Considerations with iSCSI

Status of this Memo

   This memo provides information for the Internet community.  It does
   not specify an Internet standard of any kind.  Distribution of this
   memo is unlimited.

Copyright Notice

   Copyright (C) The Internet Society (2004).  All Rights Reserved.

Abstract

   Internet Small Computer Systems Interface (iSCSI) is a Small Computer
   Systems Interface (SCSI) transport protocol designed to run on top of
   TCP.  The iSCSI session abstraction is equivalent to the classic SCSI
   "I_T nexus", which represents the logical relationship between an
   Initiator and a Target (I and T) required in order to communicate via
   the SCSI family of protocols.  The iSCSI session provides an ordered
   command delivery from the SCSI initiator to the SCSI target.  This
   document goes into the design considerations that led to the iSCSI
   session model as it is defined today, relates the SCSI command
   ordering features defined in T10 specifications to the iSCSI
   concepts, and finally provides guidance to system designers on how
   true command ordering solutions can be built based on iSCSI.

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  2
   2.  Definitions and Acronyms . . . . . . . . . . . . . . . . . . .  3
       2.1.  Definitions. . . . . . . . . . . . . . . . . . . . . . .  3
       2.2.  Acronyms . . . . . . . . . . . . . . . . . . . . . . . .  4
   3.  Overview of the iSCSI Protocol . . . . . . . . . . . . . . . .  4
       3.1.  Protocol Mapping Description . . . . . . . . . . . . . .  4
       3.2.  The I_T Nexus Model. . . . . . . . . . . . . . . . . . .  5
       3.3.  Ordered Command Delivery . . . . . . . . . . . . . . . .  6
             3.3.1.  Questions. . . . . . . . . . . . . . . . . . . .  6
             3.3.2.  The Session Guarantee. . . . . . . . . . . . . .  6
             3.3.3.  Ordering Onus. . . . . . . . . . . . . . . . . .  7
             3.3.4.  Design Intent. . . . . . . . . . . . . . . . . .  7



Chadalapaka & Elliott        Informational                      [Page 1]

RFC 3783                    Command Ordering                    May 2004


   4.  The Command Ordering Scenario. . . . . . . . . . . . . . . . .  8
       4.1.  SCSI Layer . . . . . . . . . . . . . . . . . . . . . . .  8
             4.1.1.  Command Reference Number (CRN) . . . . . . . . .  8
             4.1.2.  Task Attributes. . . . . . . . . . . . . . . . .  8
             4.1.3.  Auto Contingent Allegiance (ACA) . . . . . . . .  8
             4.1.4.  UA Interlock . . . . . . . . . . . . . . . . . .  9
       4.2.  iSCSI Layer. . . . . . . . . . . . . . . . . . . . . . .  9
   5.  Connection Failure Considerations. . . . . . . . . . . . . . .  9
   6.  Command Ordering System Considerations . . . . . . . . . . . . 10
   7.  Reservation Considerations . . . . . . . . . . . . . . . . . . 11
   8.  Security Considerations. . . . . . . . . . . . . . . . . . . . 12
   9.  References and Bibliography. . . . . . . . . . . . . . . . . . 12
       9.1.  Normative References.. . . . . . . . . . . . . . . . . . 12
       9.2.  Informative References . . . . . . . . . . . . . . . . . 12
   10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 12
   11. Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . 13
   12. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 14

1.  Introduction

   iSCSI is a SCSI transport protocol ([iSCSI]) designed to enable
   running SCSI application protocols on TCP/IP networks, including
   potentially the Internet.  Given the size and scope of the Internet,
   iSCSI thus enables some exciting new SCSI applications.  Potential
   new application areas for exploiting iSCSI's value include the
   following:

      a) Larger (diameter) Storage Area Networks (SANs) than had been
         possible until now
      b) Asynchronous remote mirroring
      c) Remote tape vaulting

   Each of these applications takes advantage of the practically
   unlimited geographical distance that iSCSI enables between a SCSI
   initiator and a SCSI target.  In each of these cases, because of the
   long delays involved, there is a very high incentive for the
   initiator to stream SCSI commands back-to-back without waiting for
   the SCSI status of previous commands.  Command streaming may be
   employed primarily by two classes of applications - while one class
   may not particularly care about ordered command execution, the other
   class does rely on ordered command execution (i.e. there is an
   application-level dependency on the ordering among SCSI commands).
   As an example, cases b) and c) listed earlier clearly require ordered
   command execution.  A mirroring application does not want the writes
   to be committed out of order on the remote SCSI target, so as to






Chadalapaka & Elliott        Informational                      [Page 2]

RFC 3783                    Command Ordering                    May 2004


   preserve the transactional integrity of the data on that target.  To
   summarize, SCSI command streaming, when coupled with the guarantee of
   ordered command execution on the SCSI target, is extremely valuable
   for a critical class of applications in long-latency networks.

   This document reviews the various protocol considerations in
   designing storage solutions that employ SCSI command ordering.  This
   document also analyzes and explains the design intent of [iSCSI] with
   respect to command ordering.

2.  Definitions and Acronyms

2.1.  Definitions

   -  I_T nexus: [SAM2] defines the I_T nexus as a relationship between
      a SCSI initiator port and a SCSI target port.  [iSCSI] defines an
      iSCSI session as the iSCSI representation of an I_T nexus.  In the
      iSCSI context, the I_T nexus (i.e. the iSCSI session) is a
      relationship between an iSCSI initiator's end of the session (SCSI
      Initiator Port) and the iSCSI target's Portal Group (SCSI Target
      Port).

   -  PDU (Protocol Data Unit): An iSCSI initiator and iSCSI target
      communicate using iSCSI protocol messages.  These messages are
      called "iSCSI protocol data units" (iSCSI PDUs).

   -  SCSI device: A SCSI device is an entity that contains one or more
      SCSI ports that are connected to a service delivery subsystem and
      supports SCSI application protocols.  In the iSCSI context, the
      SCSI Device is the component within an iSCSI Node that provides
      the SCSI functionality.  The SCSI Device Name is defined to be the
      iSCSI Name of the node.

   -  Session: A group of logically related iSCSI connections that link
      an initiator with a target form a session (equivalent to a SCSI
      I-T nexus).  The number of participating iSCSI connections within
      an iSCSI session may vary over time.  The multiplicity of
      connections at the iSCSI level is completely hidden for the SCSI
      layer - each SCSI port in an I_T nexus sees only one peer SCSI
      port across all the connections of a session.











Chadalapaka & Elliott        Informational                      [Page 3]

RFC 3783                    Command Ordering                    May 2004


2.2.  Acronyms

   Acronym                      Definition
   --------------------------------------------------------------
   ACA                          Auto Contingent Allegiance
   ASC                          Additional Sense Code
   ASCQ                         Additional Sense Code Qualifier
   CRN                          Command Reference Number
   IETF                         Internet Engineering Task Force
   ISID                         Initiator Session Identifier
   ITT                          Initiator Task Tag
   LU                           Logical Unit
   LUN                          Logical Unit Number
   NIC                          Network Interface Card
   PDU                          Protocol Data Unit
   TMF                          Task Management Function
   TSIH                         Target Session Identifying Handle
   SAN                          Storage Area Network
   SCSI                         Small Computer Systems Interface
   TCP                          Transmission Control Protocol
   UA                           Unit Attention
   WG                           Working Group

3.  Overview of the iSCSI Protocol

3.1.  Protocol Mapping Description

   The iSCSI protocol is a mapping of the SCSI remote procedure
   invocation model (see [SAM2]) over the TCP protocol.

   SCSI's notion of a task maps to an iSCSI task.  Each iSCSI task is
   uniquely identified within that I_T nexus by a 32-bit unique
   identifier called Initiator Task Tag (ITT).  The ITT is both an iSCSI
   identifier of the task and a classic SCSI task tag.

   SCSI commands from the initiator to the target are carried in iSCSI
   requests called SCSI Command PDUs.  SCSI status back to the initiator
   is carried in iSCSI responses called SCSI Response PDUs.  SCSI Data-
   out from the initiator to the target is carried in SCSI Data-Out
   PDUs, and the SCSI Data-in back to the initiator is carried in SCSI
   Data-in PDUs.










Chadalapaka & Elliott        Informational                      [Page 4]

RFC 3783                    Command Ordering                    May 2004


3.2.  The I_T Nexus Model

   In the iSCSI model, the SCSI I_T nexus maps directly to the iSCSI
   session, which is an iSCSI protocol abstraction spanning one or more
   TCP connections.  The iSCSI protocol defines the semantics in order
   to realize one logical flow of bidirectional communication on the I_T
   nexus, potentially spanning multiple TCP connections (as many as
   2^16).  The multiplicity of iSCSI connections is thus completely
   contained at the iSCSI layer, while the SCSI layer is presented with
   a single I_T nexus, even in a multi-connection session.  A session
   between a pair of given iSCSI nodes is identified by the session
   identifier (SSID) and each connection within a given session is
   uniquely identified by a connection identifier (CID) in iSCSI.  The
   SSID itself has two components - Initiator Session Identifier (ISID)
   and a Target Session Identifying Handler (TSIH) - each identifying
   one end of the same session.

   There are four crucial functional facets of iSCSI that together
   present this single logical flow abstraction to the SCSI layer, even
   with an iSCSI session spanning across multiple iSCSI connections.

      a) Ordered command delivery: A sequence of SCSI commands that is
         striped across all the connections in the session is
         "reordered" by the target iSCSI layer into an identical
         sequence based on a Command Sequence Number (CmdSN) that is
         unique across the session.  The goal is to achieve bandwidth
         aggregation from multiple TCP connections, but to still make it
         appear to the target SCSI layer as if all the commands had
         travelled in one flow.

      b) Connection allegiance: All the PDU exchanges for a SCSI
         Command, up to and including the SCSI Response PDU for the
         Command, are required to flow on the same iSCSI connection at
         any given time.  This again is intended to hide the multi-