connection nature of a session because the SCSI layer on either
         side will never see the PDU contents out of order (e.g., status
         cannot bypass read data for an initiator).

      c) Task set management function handling: [iSCSI] specifies an
         ordered sequence of steps for the iSCSI layer on the SCSI
         target in handling the two SCSI task management functions
         (TMFs) that manage SCSI task sets.  The two TMFs are ABORT TASK
         SET that aborts all active tasks in a session, and CLEAR TASK
         SET that clears the tasks in the task set.  The goal of the
         sequence of steps is to guarantee that the initiator receives
         the SCSI Response PDUs of all unaffected tasks before the TMF
         Response itself arrives, regardless of the number of
         connections in the iSCSI session.  This operational model is



Chadalapaka & Elliott        Informational                      [Page 5]

RFC 3783                    Command Ordering                    May 2004


         again intended to preserve the single flow abstraction to the
         SCSI layer.

      d) Immediate task management function handling: Even when a TMF
         request is marked as "immediate" (i.e. only has a position in
         the command stream, but does not consume a CmdSN), [iSCSI]
         defines semantics that require the target iSCSI layer to ensure
         that the TMF request is executed as if the commands and the TMF
         request were all flowing on a single logical channel.  This
         ensures that the TMF request will act on tasks that it was
         meant to manage.

   The following sections will analyze the "Ordered command delivery"
   aspect in more detail, since command ordering is the focus of this
   document.

3.3.  Ordered Command Delivery

3.3.1.  Questions

   A couple of important questions related to iSCSI command ordering
   were considered early on in the design of the iSCSI protocol.  The
   questions were:

      a) What should be the command ordering behavior required of iSCSI
         implementations in the presence of transport errors, such as
         errors that corrupt the data in a fashion that is not detected
         by the TCP checksum (e.g., two offsetting bit flips in the same
         bit position), but is detected by the iSCSI CRC digest?

      b) Should [iSCSI] require both initiators and targets to use
         ordered command delivery?

   Since the answers to these questions are critical to the
   understanding of the ordering behavior required by the iSCSI
   protocol, the following sub-sections consider them in more detail.

3.3.2.  The Session Guarantee

   The final disposition of question a) in section 3.3.1 was reflected
   in [RFC3347], "iSCSI MUST specify strictly ordered delivery of SCSI
   commands over an iSCSI session between an initiator/target pair, even
   in the presence of transport errors."  Stated differently, an iSCSI
   digest failure, or an iSCSI connection termination, must not cause
   the iSCSI layer on a target to allow executing the commands in an
   order different from that intended (as indicated by the CmdSN order)
   by the initiator.  This design choice is enormously helpful in
   building storage systems and solutions that can now always assume



Chadalapaka & Elliott        Informational                      [Page 6]

RFC 3783                    Command Ordering                    May 2004


   command ordering to be a service characteristic of an iSCSI
   substrate.

   Note that by taking the position that an iSCSI session always
   guarantees command ordering, [iSCSI] was indirectly implying that the
   principal reason for the multi-connection iSCSI session abstraction
   was to allow ordered bandwidth aggregation for an I_T nexus.  In
   deployment models where this cross-connection ordering mandated by
   [iSCSI] is deemed expensive, a serious consideration should be given
   to deploying multiple single-connection sessions instead.

3.3.3.  Ordering Onus

   The final resolution of b) in section 3.3.1 by the iSCSI protocol
   designers was in favor of not always requiring the initiators to use
   command ordering.  This resolution is reflected in dropping the
   mandatory ACA usage requirement on the initiators, and allowing an
   ABORT TASK TMF to plug a command hole etc., since these are conscious
   choices an initiator makes in favor of not using ordered command
   delivery.  The net result can be discerned by a careful reader of
   [iSCSI] - the onus of ensuring ordered command delivery is always on
   the iSCSI targets, while the initiators may or may not utilize
   command ordering.  iSCSI targets, being the servers in the client-
   server model, do not really attempt to establish whether or not a
   client (initiator) intends to take advantage of command ordering
   service, but instead simply always provide the guaranteed delivery
   service.  The rationale here is that there are inherent SCSI and
   application-level dependencies, as we shall see in building a command
   ordered solution, that are beyond the scope of [iSCSI], to mandate or
   even discern the intent with respect to the usage of command
   ordering.

3.3.4.  Design Intent

   To summarize the design intent of [iSCSI]:

   The service delivery subsystem (see [SAM2]) abstraction provided by
   an iSCSI session is guaranteed to have the intrinsic property of
   ordered delivery of commands to the target SCSI layer under all
   conditions.  Consequently, the guarantee of the ordered command
   delivery is across the entire I_T nexus spanning all the LUs that the
   nexus is authorized to access.  It is the initiator's discretion as
   to whether or not this property will be used.








Chadalapaka & Elliott        Informational                      [Page 7]

RFC 3783                    Command Ordering                    May 2004


4.  The Command Ordering Scenario

   A storage systems designer working with SCSI and iSCSI has to
   consider the following protocol features in SCSI and iSCSI layers,
   each of which has a role to play in realizing the command ordering
   goal.

4.1.  SCSI Layer

   The SCSI application layer has several tools to enforce ordering.

4.1.1.  Command Reference Number (CRN)

   CRN is an ordered sequence number which, when enabled for a device
   server, increments by one for each I_T_L nexus (see [SAM2]).  The one
   notable drawback with CRN is that there is no SCSI-generic way (such
   as through mode pages) to enable or disable the CRN feature.  [SAM2]
   also leaves the usage semantics of CRN for the SCSI transport
   protocol, such as iSCSI, to specify.  [iSCSI] chose not to support
   the CRN feature for various reasons.

4.1.2.  Task Attributes

   [SAM2] defines the following four task attributes - SIMPLE, ORDERED,
   HEAD OF QUEUE, and ACA.  Each task to an LU may be assigned an
   attribute.  [SAM2] defines the ordering constraints that each of
   these attributes conveys to the device server that is servicing the
   task.  In particular, judicious use of ORDERED and SIMPLE attributes
   applied to a stream of pipelined commands could convey the precise
   execution schema for the commands that the initiator issues, provided
   the commands are received in the same order on the target.

4.1.3.  Auto Contingent Allegiance (ACA)

   ACA is an LU-level condition that is triggered when a command (with
   the NACA bit set to 1) completes with CHECK CONDITION.  When ACA is
   triggered, it prevents all commands other than those with the ACA
   attribute from executing until the CLEAR ACA task management function
   is executed, while blocking all the other tasks already in the task
   set.  See [SAM2] for the detailed semantics of ACA.  Since ACA is
   closely tied to the notion of a task set, one would ideally have to
   select the scope of the task set (by setting the TST bit to 1 in the
   control mode page of the LU) to be per-initiator in order to prevent
   command failures in one I_T_L nexus from impacting other I_T_L
   nexuses through ACA.






Chadalapaka & Elliott        Informational                      [Page 8]

RFC 3783                    Command Ordering                    May 2004


4.1.4.  UA Interlock

   When UA interlock is enabled, the logical unit does not clear any
   standard Unit Attention condition reported with autosense, and in
   addition, establishes a Unit Attention condition when a task is
   terminated with one of BUSY, TASK SET FULL, or RESERVATION CONFLICT
   statuses.  This so-called "interlocked UA" is cleared only when the
   device server executes an explicit REQUEST SENSE ([SPC3]) command
   from the same initiator.  From a functionality perspective, the scope
   of UA interlock today is slightly different from ACA's because it
   enforces ordering behavior for completion statuses other than CHECK
   CONDITION, but otherwise conceptually has the same design intent as
   ACA.  On the other hand, ACA is somewhat more sophisticated because
   it allows special "cleanup" tasks (ones with ACA attribute) to
   execute when ACA is active.  One of the principal reasons UA
   interlock came into being was that SCSI designers wanted a command
   ordering feature without the side effects of using the aforementioned
   TST bit in the control mode page.

4.2.  iSCSI Layer

   As noted in section 3.2 and section 3.3, the iSCSI protocol enforces
   and guarantees ordered command delivery per iSCSI session using the
   CmdSN, and this is an attribute of the SCSI transport layer.  Note
   further that any command ordering solution that seeks to realize
   ordering from the initiator SCSI layer to the target SCSI layer would
   be of practical value only when the command ordering is guaranteed by
   the SCSI transport layer.  In other words, the related SCSI
   application layer protocol features such as ACA etc. are based on the
   premise of an ordered SCSI transport.  Thus, iSCSI's command ordering
   is the last piece in completing the puzzle of building solutions that
   rely on ordered command execution, by providing the crucial guarantee
   that all the commands handed to the initiator iSCSI layer will be
   transported and handed to the target SCSI layer in the same order.

5.  Connection Failure Considerations

   [iSCSI] mandates that when an iSCSI connection fails, the active
   tasks on that connection must be terminated if not recovered within a
   certain negotiated time limit.  When an iSCSI target does terminate
   some subset of tasks due to iSCSI connection dynamics, there is a
   danger that the SCSI layer would simply move on to the next tasks
   waiting to be processed and execute them out-of-order unbeknownst to
   the initiator SCSI layer.  To preclude this danger, [iSCSI] further
   mandates the following:






Chadalapaka & Elliott        Informational                      [Page 9]

RFC 3783                    Command Ordering                    May 2004


      a) The tasks terminated due to the connection failure must be
         internally terminated by the iSCSI target "as if" due to a
         CHECK CONDITION.  While this particular completion status is
         never communicated back to the initiator, the "as if" is still
         meaningful and required because if the initiator were using ACA
         as the command ordering mechanism of choice, a SCSI-level ACA
         will be triggered due to this mandatory CHECK CONDITION.  This
         addresses the aforementioned danger.

      b) After the tasks are terminated due to the connection failure,
         the iSCSI target must report a Unit Attention condition on the
         next command processed on any connection for each affected
         I_T_L nexus of that session.  This is required because if the
         initiator were using UA interlock as the command ordering
         mechanism of choice, a SCSI-level UA will trigger a UA-
         interlock.  This again addresses the aforementioned danger.
         iSCSI targets must report this UA with the status of CHECK