IPFIX Working Group                                            L. PelusoInternet-Draft                                                  T. ZsebyIntended status: Informational                Fraunhofer Institute FOKUSExpires: December 28, 2007                                  S. D'Antonio                                         CINI Consortium/ITeM Laboratory                                                           June 26, 2007                       Flow selection Techniques                 draft-peluso-flowselection-tech-00.txtStatus of this Memo   By submitting this Internet-Draft, each author represents that any   applicable patent or other IPR claims of which he or she is aware   have been or will be disclosed, and any of which he or she becomes   aware will be disclosed, in accordance with Section 6 of BCP 79.   Internet-Drafts are working documents of the Internet Engineering   Task Force (IETF), its areas, and its working groups.  Note that   other groups may also distribute working documents as Internet-   Drafts.   Internet-Drafts are draft documents valid for a maximum of six months   and may be updated, replaced, or obsoleted by other documents at any   time.  It is inappropriate to use Internet-Drafts as reference   material or to cite them other than as "work in progress."   The list of current Internet-Drafts can be accessed at   http://www.ietf.org/ietf/1id-abstracts.txt.   The list of Internet-Draft Shadow Directories can be accessed at   http://www.ietf.org/shadow.html.   This Internet-Draft will expire on December 28, 2007.Copyright Notice   Copyright (C) The IETF Trust (2007).Abstract   Flow selection is the process in charge of electing a limited number   of flows from all of those accounted at an observation point to be   considered into the measurement process chain.  The flow selection   process can be enabled at different stages of the monitoring   reference model by directly acting on the metering process after that   packet classification is performed, i.e. flow state dependent packetPeluso, et al.          Expires December 28, 2007               [Page 1]Internet-Draft          Flow selection Techniques              June 2007   sampling, or on the exporting process by limiting the number of flows   to be stored and/or exported to the collector applications.  This   document describes the motivations which might lead flow selection to   be performed and a categorization of the related techniques.  The   document furthermore provides the basis for the definition of   information models for configuring flow selection techniques.Requirements Language   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this   document are to be interpreted as described in RFC 2119 [RFC2119].Table of Contents   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3   2.  Scope  . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3   3.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  3     3.1.  General terminology  . . . . . . . . . . . . . . . . . . .  3     3.2.  Selection process related terminology  . . . . . . . . . .  6   4.  Motivation . . . . . . . . . . . . . . . . . . . . . . . . . .  7   5.  Flow selection techniques  . . . . . . . . . . . . . . . . . .  7     5.1.  Flow selection on flow record content  . . . . . . . . . . 10     5.2.  Flow selection on flow record arrival time . . . . . . . . 11     5.3.  Flow selection on external events  . . . . . . . . . . . . 11   6.  Solutions for flow cache data structure  . . . . . . . . . . . 11   7.  Information model for flow selection configuration . . . . . . 11   8.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 12   9.  Security Considerations  . . . . . . . . . . . . . . . . . . . 12   10. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 12   11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 12     11.1. Normative References . . . . . . . . . . . . . . . . . . . 12     11.2. Informative References . . . . . . . . . . . . . . . . . . 12   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 13   Intellectual Property and Copyright Statements . . . . . . . . . . 15Peluso, et al.          Expires December 28, 2007               [Page 2]Internet-Draft          Flow selection Techniques              June 20071.  Introduction   <Text for this section>2.  Scope   The main aim of this document is to describe and analyse the flow   selection that can be performed inside an IPFIX device.  This   document does not intend to deal with the flow selection that might   result from the sampling of packets in the metering process before   that the classification process is performed.  Although that approach   leads to a natural selection of the flows generated after the   classification process, packet sampling techniques are widely   analysed in [PSAMP-TECH] and, therefore, outside the scope of this   document.  Instead, it describes those selection techniques that   might be considered in order to enable flow selection by directly   acting at flow level within the metering process and/or the exporting   process.3.  Terminology   The terminology used here is fully consistent with all terms listed   in [IPFIX-ARCH] and [PSAMP-TECH] and includes additional terms   required for the description of flow selection techniques.  For the   sake of clarity, the definitions of the terms here used are below   reproposed.3.1.  General terminology   * Observation Point      An Observation Point is a location in the network where IP packets      can be observed.  Examples include:        (i)   a line to which a probe is attached;        (ii)  shared medium, such as an Ethernet-based LAN;        (iii) a single port of a router, or a set of interfaces              (physical or logical) of a router;        (iv)  an embedded measurement subsystem within an interface.      Note that every Observation Point is associated with an      Observation Domain, and that one Observation Point may be a      superset of several other Observation Points.  For example onePeluso, et al.          Expires December 28, 2007               [Page 3]Internet-Draft          Flow selection Techniques              June 2007      Observation Point can be an entire line card.  That would be the      superset of the individual Observation Points at the line card's      interfaces.   * Observation Domain      An Observation Domain is the largest set of Observation Points for      which Flow information can be aggregated by a Metering Process.      For example, a router line card may be an observation domain if it      is composed of several interfaces, each of which is an Observation      Point.  Each Observation Domain presents itself to the Collecting      Process using an Observation Domain ID to identify the IPFIX      Messages it generates.  Every Observation Point is associated with      an Observation Domain.  It is recommended that Observation Domain      IDs are also unique per IPFIX Device.   * Observed Packet Stream      The Observed Packet Stream is the set of all packets observed at      the Observation Point.   * IP Traffic Flow or Flow      There are several definitions of the term 'flow' being used by the      Internet community.  Within the context of IPFIX we use the      following definition: A Flow is defined as a set of IP packets      passing an Observation Point in the network during a certain time      interval.  All packets belonging to a particular Flow have a set      of common properties.  Each property is defined as the result of      applying a function to the values of:      1.  One or more packet header fields (e.g. destination IP          address), transport header fields (e.g. destination port          number), or application header field;      2.  One or more characteristics of the packet itself (e.g. number          of MPLS labels);      3.  One or more fields derived from packet treatment (e.g. next          hop IP address, output interface).      A packet is said to belong to a Flow if it completely satisfies      all the defined properties of the Flow.  This definition covers      the range from a Flow containing all packets observed at a network      interface to a Flow consisting of just a single packet between two      applications.  It includes packets selected by a sampling      mechanism.Peluso, et al.          Expires December 28, 2007               [Page 4]Internet-Draft          Flow selection Techniques              June 2007   * Flow Key      Each of the fields which      1.  Belong to the packet header (e.g. destination IP address);      2.  Are a property of the packet itself (e.g. packet length);      3.  Are derived from packet treatment (e.g.  AS number)      and which are used to define a Flow are termed Flow Keys.   * Flow Record      A Flow Record contains information about a specific Flow that was      observed at an Observation Point.  A Flow Record contains measured      properties of the Flow (e.g. the total number of bytes for all the      Flow's packets) and usually characteristic properties of the Flow      (e.g. source IP address).   * Metering Process      The Metering Process generates Flow Records.  Inputs to the      process are packet headers and characteristics observed at an      Observation Point, and packet treatment at the Observation Point      (for example the selected output interface).  The Metering Process      consists of a set of functions that includes packet header      capturing, timestamping, sampling, classifying, and maintaining      Flow Records.  The maintenance of Flow Records may include      creating new records, updating existing ones, computing Flow      statistics, deriving further Flow properties, detecting Flow      expiration, passing Flow Records to the Exporting Process, and      deleting Flow Records.   * Exporting Process      An Exporting Process sends Flow Records to one or more Collecting      Processes.  The Flow Records are generated by one or more Metering      Processes.   * Exporter      A device which hosts one or more Exporting Processes is termed an      Exporter.   * IPFIX DevicePeluso, et al.          Expires December 28, 2007               [Page 5]Internet-Draft          Flow selection Techniques              June 2007      An IPFIX Device hosts at least one Exporting Process.  It may host      further Exporting processes and arbitrary numbers of Observation      Points and Metering Process.   * Collecting Process      A Collecting Process receives Flow Records from one or more      Exporting Processes.  The Collecting Process might process or      store received Flow Records, but such actions are out of scope for      this document.   * Collector      A device which hosts one or more Collecting Processes is termed a      Collector.3.2.  Selection process related terminology   In this section, some additional terms are presented which extend the   terminology introduced in [PSAMP-TECH].   * Flow Selection Process      A Flow Selection Process takes the set of the accounted Flow      Records as its input and selects a subset of that set as its      output.   * Flow Selection State      A Flow Selection Process may maintain state information for use by      the Flow Selection Process.  At a given time, the Flow Selection      State may depend on flows observed at and before that time, and      other variables.  Examples include:        (i)   number of accounted flow records;        (ii)  number of available rooms for flow recording;        (iii) state of the pseudorandom number generators;        (iv)  hash values calculated during selection.   * Flow Selector      A Flow Selector defines the action of a Flow Selection Process on      a single flow of its input.  The Flow Selector can make use of the      following information in determining whether a flow is selected:Peluso, et al.          Expires December 28, 2007               [Page 6]Internet-Draft          Flow selection Techniques              June 2007        (i)   the content of the flow record;        (ii)  any information state related to the flow recording;        (iii) any selection state that may be maintained by the Flow              Selection Process.4.  Motivation   As stated in [PSAMP-TECH], packet selection is in charge of electing   a representative subset of packets that allow accurate estimates of   properties of the unsampled traffic to be formed.  Its main   application consists in performing some forms of data reduction on   observed Internet traffic in order to limit the processing overhead   at measurement devices.  Despite its proven ability in achieving this   objective, the mechanism responsible for steering the selection   process is generally driven by a packet-based decision strategy.  It   means that, the basis element on which this selection mechanism is   performed is a packet and mainly the decision of which packets are   suitable to be elected somehow depends on packets themselves.  As a   consequence, depending on the specific adopted selection strategy,   packet selection may not take in consideration eventual impacts of   its actions on subsequent measurement components, such as flow   recording and exporting processes, which are instead based on a   higher-level representation, i.e. flows rather than packets.  Under   this perspective, flow selection differs from packet selection in the   way that the basis elements on which the selection process is applied   is not a packet but a flow.  For IPFIX this would be flow records.   In many networks the distribution of the number of packets per flow   or the number of bytes per flow are heavy-tailed.  That means, most   flows consist only of a small number of packets and only few flows   have a large number of packets.  The few large flows contribute to   the majority to the overall traffic volume [DuLT01a], [DuLT01b].   This observation on the flow size distributions in Internet traffic   is also referred to as "Quasi-Zipf-Law" [KuXW04] or as "elephant and   mice phenomenon".  The large flows are referred to as elephant flows   or heavy hitters.  Nevertheless, such observations depend on the flow   definition in use and can change with regard to the profile of future   applications.  For several applications it makes sense to select only   the flows of interest. [more here].5.  Flow selection techniques   Figure 1 shows the IPFIX reference model as defined in [IPFIX-ARCH],   and extends it in order to point out the functional components where   flow selection can take place.  As previously mentioned, flowPeluso, et al.          Expires December 28, 2007               [Page 7]Internet-Draft          Flow selection Techniques              June 2007   selection can be provided at different stages of the measurement   chain.  One can act at packet level, within the metering process,   and/or at flow level, by directly operating on the flow recording   and/or exporting processes.