antrea: IPFIX start and stop timestamps are wrong/too large

Every node in the cluster has the same value…

knode2:~# cat /proc/sys/net/netfilter/nf_conntrack_timestamp
1

@robcowart You are right. Just checked the code. We are getting the stoptime from conntrack table, instead we should record the stopTime as the time when the flow record is created (time.Now) from the discussion in this issue. https://github.com/vmware-tanzu/antrea/blob/master/pkg/agent/flowexporter/exporter/exporter.go#L252

@zyiou Is it possible for you to add this in one of the PRs you already opened? Thanks.

Most network devices send flow records based on one or more configurable timeout settings. While they will track a flow for a longer period of time, they will send flows more frequently. This is necessary for a number of reasons.

• Many server-to-server TCP connections can live for weeks, months or longer (e.g. the connection between Kibana or Logstash to Elasticsearch). They essentially never end unless stopped or there is some error. Without a periodic update the user could literally wait months for the information. What good is the rest of their data until the picture is complete? Not much.

• Now consider that the above long lived flow was a hacker exfiltrating data from your database. The flow record that could reveal their activity is never sent until the damage is already done, and you loose the opportunity to shut them down. Not good.

• If all of the information about a flow record is included at the end of the flow. How do you distribute it over the life of the flow. All at the beginning? All at the end? Unfortunately most visualization tools, like Kibana, and even many purpose built flow applications, do not provide the ability to smooth the data into buckets over the life of the flow. Even if they did, each query would still have to look into the future to determine if any flows had start times within the current visualized window. This is very tricky at best, if not close to impossible. However when the device send regular updates about the flows it is much easier.

If I explained those well, you should see that it is very desirable to send information about a long-lived flow over the life of the flow. Most network devices will provide various options to control this. Some are simple:

inactive_timeout = 15 - export flow after it is inactive for 15 seconds. active_timeout = 60 - export flow every 60 seconds while it is active.

Others provide more granular control:

expiry-interval 10 - the interval between expiry checks, i.e. every 10 secs it is checked which flows are ready for export icmp 60 - export icmp flows every 60 seconds tcp-generic 60 - export tcp flows every 60 seconds tcp-fin 20 - export flow 20 seconds after observing a TCP finish flag. tcp-rst 20 - export flow 20 seconds after observing a TCP reset flag. udp 60 - export udp flows every 60 seconds flow-generic 60 - applies to any IP protocol not specified by other timeouts max-active-life 604800 - track no flows longer than 1 week (although it can pickup tracking again if the session is still active, usually with a new flow ID)

I explain this difference just to show the options that are common. The expectation is that it should be possible to at least provide for the former. The only detail I would add is that the inactive and active periods should be tracked per flow, not globally. For example, if the 60 second export period was global, i.e. every 60 seconds all active flows are exported. You can end up with 59 seconds of no data, and a flood of records from across the whole infrastructure simultaneously. This can easily overwhelm some collection systems, especially Linux systems with default networking kernel parameters. Tracking the inactive and active timeouts per flow will better distribute the load for the collecting system and provide more timely and accurate reporting. In 2020 I would say that it is generally accepted that 1-min is a good compromise between data granularity and overall volume of records generated, with short-lived flows being exported quickly due to the inactive timeout.

Now that we have established that records should be sent periodically over the lifetime of a network flow, the next question is which fields, or “information elements” (IEs), should be included. While there are both delta and total IEs for things like bytes and packets. Almost every vendor sends only the delta values, where the delta is the quantity since the previous record for this flow was exported (or since the flow started if sending the initial record).

Sending only delta values is usually not an issue as it is easy enough at query time to do a sum of the deltas, when a total is needed. I have seen a few examples of vendors sending the total values as well, but it isn’t really important. What should be avoided is sending only total values. That makes it very challenging to work with the data later, especially in combination with records form other sources that provide only delta values. There is nothing wrong with including the total values, but I would simply remove them and gain a bit of efficiency.

Regarding the start and end timestamps. I have to admit that the RFCs aren’t really clear here. My feeling is that for active flows flowEndSeconds should be set the time that the last measurement was taken. However I am going to spend some time going through my collection of PCAPs from different vendors and try to determine if there is any consensus best practice. I will get back to you on this.

The last thing I will mention is UDP vs. TCP. I noticed that in the Antrea repo examples that the flowexporter was configured to use TCP. However I set mine up to use UDP. Like many other network-related data sources (SNMP, syslog, etc.) netflow was designed to be sent via UDP. The logic of this is that establishing a TCP session has much more overhead and requires much more network traffic (especially in the reverse direction for ACKs and such) than UDP. During an outage, it is generally undesirable that the overhead of the management traffic attempting to re-establish session is competing with applications to do the same. This is so embedded in the networking mindset that many devices, as well as collectors, do not even support TCP. In fact IPFIX is the only flow standard to even specify optional support for TCP. Since the Antrea exporter supports both, this isn’t an issue. However you should be aware that if people follow the documented example config exactly, some of them might encounter issues because the collector doesn’t support TCP.

Hopefully this was helpful. Let me know if there are any questions that I can help answer.