As recommended by Nevil in the attached email,
we post this individual draft to psamp list too.
We have read through a set of ipfix and psamp drafts and found that
it may be useful to extend the ipfix information model, metering process,
configuration,
and data export as described in the attached draft. We provided
the rationale of our proposal and necessary extensions in the document
in detail. We would highly apprciate your time for reading and commenting
it.
We submitted this document as an individual submission but it hasn't
been registered in the IETF internet draft registery yet. Thus, I
attached the
document for your easiler reference.
Best regards,
- Taesang, Changhoon
@ETRI
Sender: Nevil Brownlee[n.brownlee@auckland.ac.nz]
To: choits@etri.re.kr, kimch@etri.re.kr
CC:ipfix-chairs@net.doit.wisc.edu
Subject: Re: Please Review the following draft
Date: 2003/06/25 Wed. 11:46
Hello Taesang:
> - - -
> the proposal is described in the draft. One thing
bothers me is the fact
> that PSAMP WG is dealing with packets and yours seems
to take care of
> flows. But from the applications point of view,
both information are
> necessary and should be handled in one framework, IMHO.
Although we are
> not sure whether issues raised in our docuemnt should
be discussed in which
> WG, we though IPFIX is more appropriate to try.
I would appreciate if you
> can spare some time to review the docuement.
Also, if possible we would
> like to request for a 5-10 min time-slot for the presentation
during
> upcoming Vienna IETF meeting.
I haven't read your draft carefully yet, but it's a good
way of putting
forward your proposal. As for the PSAMP vs IPFIX
question, the PSAMP
folks said quite some time ago that they'd most probably
use IPFIX to
transport their packet data, and clearly they'll need
to extend the
IPFIX information model to do that.
I presume you've already submitted your draft to internet-drafts@ietf.org;
I suggest you send a note to both the IPFIS and PSAMP
mailing lists telling
people what your draft is aiming to do, and giving them
a URL to get it.
Let's see what sort of discussion that produces.
As for the Vienna meeting, yes, we can give you a 5~10
minute timeslot.
Cheers, Nevil
-----------------------------------------------------------------------
Nevil Brownlee
Director, Technology Development
Phone: +64 9 373 7599 x88941 ITSS,
The University of Auckland
FAX: +64 9 373 7021 Private
Bag 92019, Auckland, New Zealand
-------------------------------------------------
This mail sent through University of Auckland
http://www.auckland.ac.nz/
Internet Draft Chang H. Kim
draft-kim-ipfix-ppr-00.txt Taesang Choi
Expires: December 2003 ETRI
June 2003
Supplementing IPFIX Flow Informaion with Per-packet Records
<draft-kim-ipfix-ppr-00.txt>
Status of this Memo
This document is an Internet-Draft and is subject to all provisions
of Section 10 of RFC2026.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet- Drafts as reference
material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at
http://www.ietf.org/1id-abstracts.html
The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html
Copyright Notice
Copyright (C) The Internet Society (2003). All Rights Reserved.
Abstract
This document describes extensions required to supplement the IP flow
information export with per-packet records of which generation and
export are configurable. The extension supports more precise
application-aware usage accounting, detailed traffic profiling,
enhanced intrusion and attack detection, etc. Extensions on the
existing information model, the Metering Process, the configuration,
and the data export layout are also mentioned.
Kim, et al. expires - December, 2003 [Page 1]
Internet Draft Per-packet Records June 2003
Table of Contents
1. Introduction ................................................ 3
2. Applicability ............................................... 4
2.1. Usage-based Accounting .................................. 4
2.2. Traffic Profiling ....................................... 5
2.3. Attack/Intrusion Detection .............................. 5
2.4. Application Monitoring and Profiling .................... 5
3. Methods for Adopting Per-packet Records...................... 6
3.1. Information Model Extension ............................. 6
3.1.1. pktArrivalTimeOffset .................................. 6
3.1.1.1. Type ................................................ 6
3.1.1.2. Field Id ............................................ 6
3.1.1.3. Reference ........................................... 6
3.1.2. pktLen ................................................ 6
3.1.2.1. Type ................................................ 6
3.1.2.2. Field Id ............................................ 6
3.1.3. pktId ................................................. 7
3.1.3.1. Type ................................................ 7
3.1.3.2. Field Id ............................................ 7
3.1.4. pktFlags .............................................. 7
3.1.4.1. Type ................................................ 7
3.1.4.2. Field Id ............................................ 7
3.1.5. pktTtl ................................................ 7
3.1.5.1. Type ................................................ 7
3.1.5.2. Field Id ............................................ 7
3.1.6. pktFragIndication ..................................... 7
3.1.6.1. Type ................................................ 7
3.1.6.2. Field Id ............................................ 7
3.1.7. tcpFlags .............................................. 7
3.1.7.1. Type ................................................ 8
3.1.7.2. Field Id ............................................ 8
3.1.8. tcpSeqNumber .......................................... 8
3.1.8.1. Type ................................................ 8
3.1.8.2. Field Id ............................................ 8
3.1.9. tcpAckNumber .......................................... 8
3.1.9.1. Type ................................................ 8
3.1.9.2. Field Id ............................................ 8
3.1.10. pktEntirePayload ..................................... 8
3.1.10.1. Type ............................................... 8
3.1.10.2. Field Id ........................................... 8
3.1.10.3. Reference .......................................... 8
3.1.11. pktPartialPayload .................................... 9
3.1.11.1. Type ............................................... 9
3.1.11.2. Field Id ........................................... 9
3.1.11.3. Reference .......................................... 9
3.2. Metering Process Extension .............................. 9
3.2.1. Metering Process Functions ............................ 9
3.2.2. Selection Criteria for Packet Information Export ...... 10
3.2.3. Pattern Specification and Pattern Matching ............ 10
Kim, et al. expires - December, 2003 [Page 2]
Internet Draft Per-packet Records June 2003
3.3. Configuration Extension ................................. 11
3.3.1. Configuration Extension for the Metering Process ...... 11
3.3.2. Configuration Extension for the Exporting Process ..... 11
3.4. Data Export Extension ................................... 12
3.4.1. Export Layout ........................................ 12
3.4.2. Export Interval ...................................... 13
3.5. Collector Extension ..................................... 13
4. Security Considerations ..................................... 13
5. References .................................................. 13
1. Introduction
Internet traffic has been dominated by client-server applications
until late 90s. Starting from "Napster", however, various
next-generation applications have been drastically getting popular.
Peer-to-peer contents sharing programs, network-based games,
e-commerce tools, and multimedia streaming applications are all
good examples. Unlike the traditional Internet applications that
could be easily identified by their port numbers, identifying and
characterizing these sort of unprecedented traffic demand enhanced
mechanisms beyond port mapping. In the meantime, the amount of traffic
volume generated by such applications began to grow from a negligible
portion to more than half of the total volume, although there exists
a good deal of variation depending upon time, region, occasion, user
group, etc.
Both the recent explosion of the number of unprecedented applications
and the common use of port range allocation give rise to a high
frequency of the overlapped service ports problem. For example,
users intentionally configure a new application to share a well
known service port, say 80 of HTTP, for an infiltration purpose.
Overlapped service ports, however, even combined with the flow
concept, preclude ensuring a high degree of application recognition
correctness. To overcome this limitation, we need to investigate
the contents of packets to search for each application¡¯s
distinctive signature. The signature can be an ASCII string,
a binary code, or a combination of them. Using the flow concept
in conjunction with payload inspection, we can obtain synergy effect;
packets that do not contain any distinctive signature can also be
classified to the application whose signature is found in the
other packets in the same flow. Nevertheless, since the operational
burden of the payload investigation method is much higher than
that of the simple port-based one, its use should be limited and
configurable depending on the number and speed of the monitored
links, available computing resources, and other operational constraints.
Configurability, thus, plays a principal role in the designing
and implementing process of a recognition system which is capable
of payload investigation.
Kim, et al. expires - December, 2003 [Page 3]
Internet Draft Per-packet Records June 2003
For the sake of traffic profiling, on the other hand, various
flow statistics are required such as flow duration, flow volume,
timing, and burstiness. Besides these flow related statistics,
packet specific statistics information (e.g., packet size
distribution, packet inter-arrival time, etc.) is also very
useful for per-packet traffic profiling. Supplementing the existing
IPFIX with per-packet information, thus, can be highly useful for
this purpose.
In this draft, we propose to add per-packet records supplementing
IPFIX flow information in order to provide IPFIX applications with
more detailed and precise information. We address detailed
applicability and requirements in section 2 for each application.
Extensions on the existing information model and processes for
satisfying such requirements are provided in section 3.
2. Applicability
IPFIX applicability draft [2] describes critical customer
applications which may utilize IPFIX data. The applications are
accounting, peering agreements, traffic engineering, data
warehousing and mining, and network monitoring. Altough the
draft mention that the existing IPFIX architecture can support
all these applications, additional information or mechanisms on
exporting per-packet records is necessary.
More details on each application field are described below.
2.1. Usage-based Accounting
The IPFIX applicability draft states that charging can be based on
application usage among others. The IPFIX architecture provides
this information based on port numbers. But, as mentioned in
the introduction, mapping applications and port numbers are no
longer safe and newer mechanisms, such as signature matching,
are required for precise application usage accounting.
Some applications can be identified simply by matching application
specific signature per packet basis. However, other applications
require cross-check with internal sub-flows which may occur in
opposite flow direction or even in another link.
In the former case, application signature information needs
to be added in the information model. In the latter case, however,
precise application recognition with signature matching on a
single link is not possible. Instead, flow record somehow has to
keep the application signature information and a collector which
monitors multiple links later correlate results for the accurate
recognition. For this purpose, we propose to add a packet record
and keep an application signature in it. Collector can use them
later during analysis phase.
Kim, et al. expires - December, 2003 [Page 4]
Internet Draft Per-packet Records June 2003
2.2. Traffic Profiling
For the proper traffic profiling, various statistics are needed
and many of them are listed in the requirement and applicability
draft such as flow duration, volume, time and burstiness.
Besides these flow related statistics, packet specific statistics
information (e.g., packet size distribution, packet inter-arrival time,
fragement statistics, etc.) is also very important for traffic profiling.
IPFIX architecture doesn¡¯t take this aspect into consideration.
Additional packet related information, thus, needs to be added as
a part of the IPFIX information model if packet related traffic
profiling is required. Since this information is packet specific ones,
flow record is not a good place to add them. We propose a packet
record for the place holder of such information.
2.3. Attack/Intrusion Detection
IPFIX architecture allows packet content inspection for
attack/intrusion detection. However, it doesn¡¯t define elements of
information model for storing such information, metering process
and exporting process. Similar to usage-based accounting case,
virus or worm signature information can be captured in packet records
for post-processing by analysis applications.
2.4. Application Monitoring and Profiling
IPFIX architecture states that it enables content and service
providers to view detailed, time-based, and application-based
usage of a network. Simple port based accounting per application
doesn¡¯t faithfully measure its usage. Also QoS monitoring per
application may be a requirement for content and service providers.
It requires more precise application profiling as the case of
the usage-based accounting.
Kim, et al. expires - December, 2003 [Page 5]
Internet Draft Per-packet Records June 2003
3. Methods for Adopting Per-packet Records
The following are required extension on the existing IP flow
information export specifications [3][4][5].
3.1. Information Model Extension
As specified in section 6 of IPFIX Information Model document [5],
the existing IPFIX information model allows for extending the set
of information items. This section, thus, defines new information
elements for exporting per-packet information.
3.1.1. pktArrivalTimeOffset
The offset of the packet's arrival timestamp to the flowCreationTime
of the flow.
3.1.1.1. Type
The pktArrivalTime element is of type ipdr:dateTimeUsec.
3.1.1.2. Field Id
The field id will be assigned by IANA.
3.1.1.3. Reference
Because an exporter terminates a long-lasting flow on a regular basis,
the value of pktArrivalTimeOffset MUST be smaller than the active
timeout period.
3.1.2. pktLen
Packet length in the IP packet.
3.1.2.1. Type
The pktLen element is of type int.
3.1.2.2. Field Id
The field id will be assigned by IANA.
Kim, et al. expires - December, 2003 [Page 6]
Internet Draft Per-packet Records June 2003
3.1.3. pktId
The identification value of the IP packet.
3.1.3.1. Type
The pktId is of type short.
3.1.3.2. Field Id
The field id will be assigned by IANA.
3.1.4. pktFlags
The flags of the IP packet.
3.1.4.1. Type
The pktFlags is of type byte.
3.1.4.2. Field Id
The field id will be assigned by IANA.
3.1.5. pktTtl
The TTL value of the IP packet.
3.1.5.1. Type
The pktTtl is of type unsignedByte.
3.1.5.2. Field Id
The field id will be assigned by IANA.
3.1.6. pktFragIndication
The fragment field of the IP packet.
3.1.6.1. Type
The pktFragIndication is of type byte.
3.1.6.2. Field Id
The field id will be assigned by IANA.
3.1.7. tcpFlags
The flags in the TCP header of the IP packet.
Kim, et al. expires - December, 2003 [Page 7]
Internet Draft Per-packet Records June 2003
3.1.7.1. Type
The tcpFlags is of type byte.
3.1.7.2. Field Id
The field id will be assigned by IANA.
3.1.8. tcpSeqNumber
The sequence number in the TCP header of the IP packet.
3.1.8.1. Type
The tcpSeqNumber is of type int.
3.1.8.2. Field Id
The field id will be assigned by IANA.
3.1.9. tcpAckNumber
The acknowledgement number in the TCP header of the IP packet.
3.1.9.1. Type
The tcpAckNumber is of type int.
3.1.9.2. Field Id
The field id will be assigned by IANA.
3.1.10. pktEntirePayload
The payload of the IP packet.
3.1.10.1. Type
The pktEntirePayload is of type string.
3.1.10.2. Field Id
The field id will be assigned by IANA.
3.1.10.3. Reference
When a predefined byte long part of an IP packet is captured,
pktEntirePayload means the entire captured portion of the payload.
The packet capture length can be configured during the initial
communication between an exporter and a collector.
For avoiding resource exhaustion and performance degradation, the
Kim, et al. expires - December, 2003 [Page 8]
Internet Draft Per-packet Records June 2003
use of pktEntirePayload element should be strictly conservative.
The exporter, therefore, does not necessarily include
a pktEntirePayload element to every packet information record;
when a packet's payload contains a specific pattern, the exporter
may append the pktEntirePayload element.
3.1.11. pktPartialPayload
The partial payload of the IP packet.
3.1.11.1. Type
The pktPartialPayload is of type string.
3.1.11.2. Field Id
The field id will be assigned by IANA.
3.1.11.3. Reference
During the initial communication between a collector and an exporter,
operators can specify and assign some patterns (signatures) of interest.
When an exporter detects a specific pattern in a packet's payload
the exporter can append only the pattern in the pktPartialPayload field.
The choice of appending an entire or a partial payload can also be
configured. The exporter does not necessarily include a
pktPartialPayload element to every packet information record;
when a packet's payload contains a specific pattern, the exporter may
append the pktPartialPayload element.
3.2. Metering Process Extension
The metering process defined in IPFIX architecture model [4]
may be extended as specified in the followings.
3.2.1. Metering Process Functions
Flow classification specification may contain a flag enabling or
disabling per-packet information export. If the flag is off, packets
within the flow are processed as the existing manner in the architecture
model [4]. If the flag is on, however, the Metering Process must generate
per-packet information. When the flag is on and a specific pattern is
assigned to the flow as well, the Metering Process needs to perform an
additional pattern matching process on the basis of the pattern(s).
When more than one patterns are specified for a flow, the Metering
Process needs to execute the pattern matching task repetitively.
When no pattern or a wildcard pattern is specified for a flow,
the pattern matching process does not perform anything in effect.
For a wildcard pattern specification, however, the pktEntirePayload
or pktPartialPayload element must be generated and included in
the per-packet records.
Kim, et al. expires - December, 2003 [Page 9]
Internet Draft Per-packet Records June 2003
The figure below describes the extended architecture of the Metering
Process.
packet capturing
|
timestamping
|
V
+------+
| |
| sampling (1:1 in case of no sampling)
| |
| classifying -------------+
| (NULL when No criteria) |
| | |
+------+ pattern matching
| (NULL when No pattern or Wildcard pattern)
| |
V V
Flow Records Packet Records
3.2.2. Selection Criteria for Packet Information Export
The measurement device may define rules so that the information
of packets within only a certain flow is exported. Packets that
satisfy a function on the fields defined by the packet header
fields or fields obtained while doing the packet processing or
the properties of the packet itself. The measurement device can
include the rules in the Selection Criteria in the architecture
model [4] and the flag enabling or disabling Per-Packet
Information Export (PPIE) is called PPIE flag.
Example: The per-packet information of flows whose
{Protocol == TCP, Destination Port = 80} are generated
and exported.
3.2.3. Pattern Specification and Pattern Matching
A Pattern Specification is a pattern and its attributes. A Pattern
Specification is composed as the following:
Pattern Specification = <Pattern, Pattern Position>
Pattern Position is composed of packet sequence range and byte
range in which the pattern should be sought for.
A Selection Criteria [4] whose PPIE flag is on may contain one
or more Pattern Specifications; packets that suffice the Selection
Criteria are investigated in searching for the patterns in
the Pattern Specifications. This mechanism, in conjunction with
the Pattern Position, restricts the excessive use of pattern
matching function which may consume serious amount of computing
and network resources.
Kim, et al. expires - December, 2003 [Page 10]
Internet Draft Per-packet Records June 2003
To implement the Pattern Matching process, the measurement process
may use an efficient pattern matching algorithm. The specification
of such an algorithm or a set of algorithms is out of the draft's
scope. In IPFIX's context, the payload of a packet becomes the text
on which pattern matching is performed. For fragmented packets,
however, the pattern may occur in a separate form on a number of
consecutive packets especially when the specified pattern is long.
The pattern matching process, thus, should operate on a text obtained
by reassembling packet fragments. On the other hand, the measurement
device may not reassemble a transport layer segment when a long
pattern takes place on the boundary of more than one packets.
3.3. Configuration Extension
The Configuration specified in the requirement document [6]
may be extended as specified in the followings.
3.3.1. Configuration Extension for the Metering Process
The Metering Process may provide a way of configuring the extended
features in the Selection Criteria. The following parameters of the
metering process MAY be configurable:
1. specifications of flows whose constituting packets'
information will be exported; this can be accomplished by
adding the PPIE flag in the Selection Criteria.
2. specifications of flows whose constituting packets'
payload will be investigated in searching for patterns;
this can be accomplished by adding the Pattern
Specification in the Selection Criteria
3. Pattern Specifications
4. packet capture length; this should be uniformly applied
to every packet in every flow of a measurement device
3.3.2. Configuration Extension for the Exporting Process
The Exporting Process may provide a way of configuring the following
parameters:
1. reporting data format for per-packet information
Specifying the reporting data format for per-packet information
must include a selection of attributes to be reported for each
packet. Especially, in order to include a packet's payload in the
reporting format, a corresponding Pattern Specification must be
provided to the Metering Process because the exporter is confined
to create a pktEntirePayload or a pktPartialPayload instance only
when the packet's payload contains the pattern. When capturing
every packet's payload is required, a wildcard pattern can be
used. Using a wildcard pattern necessarily implies to include
pktEntirePayload in the reporting format.
2. Per-packet data export interval
Detailed description on this parameter is given in section 3.4.
Kim, et al. expires - December, 2003 [Page 11]
Internet Draft Per-packet Records June 2003
3.4. Data Export Extension
Since the data export protocol specified in [3] is extensible
and configurable, there is no special extension required for adopting
per-packet information export.
3.4.1. Export Layout
However, because a single instance of Data FlowSet can contain Flow
Records of a single type, the measurement device can not combine the
per-packet records into the same Data Flowset for the packets'
corresponding flow. Instead, the exporter can define additional Data
Flowset formats. On the basis of the selection of per-packet
information elements specified in the section 3.1, the exporter can
define a number of different Data Flowset formats. Defining new Data
Flowsets is accomplished by introducing new Templates Records.
A Data FlowSet for per-packet records contains the information of
packets of a single flow. In order to convey the information of the
corresponding flow to which the packets belong, another Data Flowset
that conveys flow information must precede the Data FlowSet for
per-packet records. The Data FlowSet conveying flow information must
be disposed right ahead of the Data FlowSet conveying per-packet
records. In order to convey enough information to couple the
per-packet records with their corresponding flow, the Data FlowSet
for flow information must contain at least the following six elements:
1. Src IP Address
2. Dst IP Address
3. Src Port
4. Dst Port
5. Protocol Id
6. Flow Creation Time
The figure below shows the overall layout of an export packet
containing per-packet records.
+--------+-------------------------------------------------------------+
| | +---------+ +--------------+ +--------------------+ |
| Packet | | Data | ... | Data FlowSet | | Data FlowSet | ... |
| Header | | FlowSet | ... | (Flow Info | |(Per-packet Info of | ... |
| | | | | of flow X) | | packets within X | |
| | +---------+ +--------------+ +--------------------+ |
+--------+-------------------------------------------------------------+
The length of an export packet still should not exceed the local MTU.
Kim, et al. expires - December, 2003 [Page 12]
Internet Draft Per-packet Records June 2003
3.4.2. Export Interval
The export interval of per-packet information should be supported in
either of the following ways:
1. Per-packet records are exported when a corresponding flow is
terminated.
2. Per-packet records are exported on a regular period basis;
The export period may be short enough to support near-realtime
notification.
When per-packet records are exported on a regular period basis,
a Data FlowSet for per-packet records needs to be generated before
the corresponding flow terminates. In this case, the exporting
process needs to create a simple flow record before generating a
full-fledged, expiration-based flow record.
3.5. Collector Extension
There is no extension required for the Collector.
4. Security Considerations
This document describes the requirements and elements of information
model for supplementing IPFIX flow information with per-packet record.
The security requirements for the IPFIX flow information are addressed
in the IPFIX requirement draft and the same requirements must be
considered for the extension proposed in this document as well.
No further security threats are induced from this document.
5. References
[1] Colleen Shannon, David Moore, K Claffy, Characteristics of Fragmented
IP Traffic on Internet Links, PAM 2001, 83 - 97. Nov. 2001.
[2] Tanja Zseby, et. al.., IPFIX Applicability, draft-ietf-ipfix-as-00.txt,
June 2003.
[3] B. Claise, et. al.., IPFIX Protocol Specifications,
draft-ietf-ipfix-protocol-00.txt, June 2003.
[4] G. Sadasivan, et. al.., Architecture Model for IP Flow Information
Export, draft-ietf-ipfix-arch-00.txt, June 2003.
[5] P. Calato, et. al.., Information Model for IP Flow Information Export,
draft-ietf-ipfix-info-00.txt, June 2003.
[6] J. Quittek, et. al.., Requirements for IP Flow Information Export,
draft-ietf-ipfix-reqs-10.txt, June 2003.
Author's Addresses
Changhoon Kim
Engineering Staff,
ETRI, 161 Gajeong-Dong, Yuseong-Gu, Daejon, 305-350, South Korea
kimch@etri.re.kr
Taesang Choi
Senior Engineering Staff,
ETRI, 161 Gajeong-Dong, Yuseong-Gu, Daejon, 305-350, South Korea
choits@etri.re.kr
Kim, et al. expires - December, 2003 [Page 13]