netflow2ng: zmq format for ntopng incorrect? (openflow support)

I am trying to push some data into ntopng 6.0, but it doesn’t receive any flows, despite receiving zmq messages: image

So I went ahead and compared the output of netflow2ng (0.0.5) with the output of nprobe (10.5) and they seem to be very different: netflow2ng output

{
    "1": 1421,
    "10": 0,
    "11": 33923,
    "12": "61.250.232.215",
    "130": "192.168.0.1",
    "14": 0,
    "15": "61.250.232.215",
    "16": 0,
    "17": 0,
    "2": 6,
    "21": 1706022328,
    "22": 1706022318,
    "29": 0,
    "30": 0,
    "32": 0,
    "4": 6,
    "44": 0,
    "45": 0,
    "5": 0,
    "52": 0,
    "54": 0,
    "58": 0,
    "59": 0,
    "6": 27,
    "60": 4,
    "61": 0,
    "7": 22,
    "8": "89.0.69.55",
    "80": "00:00:00:00:00:00",
    "81": "00:00:00:00:00:00",
    "88": 0,
    "89": 0
}

nprobe output

{
    "56": "00:00:00:00:00:00",
    "57": "00:00:00:00:00:00",
    "10": 0,
    "14": 0,
    "58": 0,
    "8": "192.168.0.1",
    "12": "192.168.0.4",
    "7": 53,
    "11": 51996,
    "27": "::",
    "28": "::",
    "60": 4,
    "4": 17,
    "35632.118": "5",
    "35632.560": 0,
    "1": 64,
    "2": 1,
    "23": 0,
    "24": 0,
    "22": 1706022259,
    "21": 1706022259,
    "35632.78": 0,
    "35632.79": 0,
    "35632.509": 0,
    "35632.527": 0,
    "130": "192.168.0.1",
    "61": 2,
    "34": 1,
    "42": 18,
    "35632.471": "192.168.3.28",
    "35632.122": "",
    "58051": 1,
    "225": "0.0.0.0",
    "226": "0.0.0.0",
    "227": 0,
    "228": 0
}

I am using the @NTOPNG@ template on nprobe, which expands to %IN_SRC_MAC %OUT_DST_MAC %INPUT_SNMP %OUTPUT_SNMP %SRC_VLAN %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %L7_PROTO %L7_CONFIDENCE %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %CLIENT_TCP_FLAGS %SERVER_TCP_FLAGS %L7_PROTO_RISK %L7_RISK_SCORE %EXPORTER_IPV4_ADDRESS %DIRECTION %SAMPLING_INTERVAL %TOTAL_FLOWS_EXP %NPROBE_IPV4_ADDRESS %NPROBE_INSTANCE_NAME %FLOW_SOURCE %POST_NAT_SRC_IPV4_ADDR %POST_NAT_DST_IPV4_ADDR %POST_NAPT_SRC_TRANSPORT_PORT %POST_NAPT_DST_TRANSPORT_PORT

Looking at https://github.com/cloudflare/goflow/tree/master/decoders/netflow, I can see how I could manipulate the JSON output by incorporating those additional fields into https://github.com/synfinatic/netflow2ng/blob/main/cmd/zmq.go. However, I am not sure how those “combined” fields (i.e. “35632.527”) are created / what they mean. Also, I am not sure why for example the MAC addresses use different identifiers on the nprobe string (for example 56 - IPFIX_FIELD_sourceMacAddress, 57 - IPFIX_FIELD_postDestinationMacAddress) compared to the identifiers netflow2ng uses (for example 80 - IPFIX_FIELD_destinationMacAddress, 81 - IPFIX_FIELD_postSourceMacAddress). Same goes for a couple of the other fields - I guess the template used by netflow2ng at the moment is an old version, compatible with previous versions of ntopng!?

So, for now at least, ntopng doesn’t receive any flows from netflow2ng. Could you give any pointers on how to incorporate the changes neccessary to support a more current ntopng version?

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Reactions: 1
  • Comments: 35 (12 by maintainers)

Most upvoted comments

So going back to the beginning… @level20peon tried to show that nprobe is sending different data than netflow2ng. But this is expected. For example, if you look at ntop_flow.h you’ll see there are four different ways the MAC address can be specified:

#define IN_SRC_MAC 56
#define OUT_DST_MAC 57

#define IN_DST_MAC 80
#define OUT_SRC_MAC 81

And that makes sense since packets have a direction (in vs. out) and you have two addresses (src and dst). So the fact that two different netflow messages use different keys for the MAC addresses makes sense and if you look, both nprobe and netflow2ng are sending valid values.

That said, nothing I see in the json output of netflow2ng looks wrong or invalid. Digging through the ntopng code, I don’t see anything obvious that would cause any issues and my testing has shown it works fine with both ntopng v5 and v6.

If you’d like me to investigate further, frankly I need pcaps of the netflow traffic being sent to netflow2ng and more information about how the netflow traffic being generated. @gitet mentioned using OpnSense which I assume is using softflowd? What version? what is the configuration/CLI arguments?

If it helps, here is how my pfSense box is running softflowd v1.0.0:

/usr/local/bin/softflowd -i 1:lagg0 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.pid -c /var/run/softflowd.lagg0.ctl
/usr/local/bin/softflowd -i 2:lagg0.200 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.200.pid -c /var/run/softflowd.lagg0.200.ctl
/usr/local/bin/softflowd -i 3:lagg0.400 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.400.pid -c /var/run/softflowd.lagg0.400.ctl
/usr/local/bin/softflowd -i 4:lagg0.300 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.300.pid -c /var/run/softflowd.lagg0.300.ctl
/usr/local/bin/softflowd -i 5:lagg0.500 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.lagg0.500.pid -c /var/run/softflowd.lagg0.500.ctl
/usr/local/bin/softflowd -i 6:em3 -n 172.17.0.5:2055 -s 1 -v 9 -T full -A milli -t tcp=3600 -t tcp.rst=5 -t tcp.fin=5 -t udp=300 -p /var/run/softflowd.em3.pid -c /var/run/softflowd.em3.ctl

Furthermore, you’ll get debugging of ZMQ messages with ntopng at --verbose=5. you may also want to try not sniffing on wlp4s0 and just relying on ZMQ.

I haven’t had a chance yet to dig deeper, but looks like https://github.com/ntop/ntopng/blob/dev/src/ZMQParserInterface.cpp has changed significantly from 5.6 to 6.0 which most likely explains what is going on here.

src/ZMQCollectorInterface.cpp

        if (size == sizeof(struct zmq_msg_hdr_v0)) {
          /* Legacy version (msg_id = 0, source_id = 0) */
          publisher_version = h0.version;
        } else {
          /* safety checks */
          if (((size != sizeof(struct zmq_msg_hdr_v3)) &&
               (size != sizeof(struct zmq_msg_hdr_v2)) &&
	       (size != sizeof(struct zmq_msg_hdr_v1))
	       ) ||
              (h->version > ZMQ_MSG_VERSION)) {
            ntop->getTrace()->traceEvent(TRACE_WARNING,
					 "Unsupported publisher version: is your nProbe sender "
					 "outdated? [%u][%u][%u][%u][%u]",
					 size, sizeof(struct zmq_msg_hdr_v1), h->version,
					 ZMQ_MSG_VERSION, ZMQ_COMPATIBILITY_MSG_VERSION);
            continue; /* skip message */
          }

include/ntop_defines.h

struct zmq_msg_hdr_v1 {
  char url[16];
  u_int8_t version, source_id;
  u_int16_t size;
  u_int32_t msg_id;
};

struct zmq_msg_hdr_v2 {
  char url[16];
  u_int8_t version;
  u_int16_t size;
  u_int32_t msg_id, source_id;
};

struct zmq_msg_hdr_v3 {
  char url[16];
  u_int8_t version /* 4 */, flags;
  u_int32_t uncompressed_size, compressed_size;
  u_int32_t msg_id, source_id;
};

As you can see from the inline comment, if the condition is not fulfilled and zmq_message _headers are not having correct size, the message is skipped. Are the messages form the netflow2ng conformant to the defined headers?

from the original log message:

20/May/2024 15:28:22 [ZMQCollectorInterface.cpp:353] WARNING: Unsupported publisher version: is your nProbe sender outdated? [319][24][53][2][1]

size -> 319 sizeof(struct zmq_msg_hdr_v1) -> 24
h->version -> 53 ZMQ_MSG_VERSION -> 2 ZMQ_COMPATIBILITY_MSG_VERSION) -> 1

@synfinatic , happy to hear that 😃 Yes, I’m jumping to conclusions - flowd is a collector. I’m not particularly fluent with FreeBSD, but as far as I am able to tell, netflow is generated by a NETGRAPH kernel module (igc0 and igc1 being my WAN and LAN interfaces):

root@firewall:/etc/rc.d # ngctl list | grep netflow
  Name: ksocket_netflow_igc1 Type: ksocket         ID: 000000e4   Num hooks: 1
  Name: netflow_igc0    Type: netflow         ID: 000000e9   Num hooks: 3
  Name: ksocket_netflow_igc0 Type: ksocket         ID: 000000ef   Num hooks: 1
  Name: netflow_igc1    Type: netflow         ID: 000000de   Num hooks: 3

root@firewall:/etc/rc.d # ngctl show netflow_igc0:
  Name: netflow_igc0    Type: netflow         ID: 000000e9   Num hooks: 3
  Local hook      Peer name       Peer type    Peer ID         Peer hook      
  ----------      ---------       ---------    -------         ---------      
  export9         ksocket_netflow_igc0 ksocket      000000ef        inet/dgram/udp 
  out1            igc0            ether        00000002        upper          
  iface1          igc0            ether        00000002        lower          

root@firewall:/etc/rc.d # ngctl show netflow_igc1:
  Name: netflow_igc1    Type: netflow         ID: 000000de   Num hooks: 3
  Local hook      Peer name       Peer type    Peer ID         Peer hook      
  ----------      ---------       ---------    -------         ---------      
  export9         ksocket_netflow_igc1 ksocket      000000e4        inet/dgram/udp 
  out2            igc1            ether        00000003        upper          
  iface2          igc1            ether        00000003        lower          

I’ll generate a pcap for you and post here ASAP.

@gitet , no appologies necessary. We all learned something new today. 😃

Anyways, the only reference for flowd that google is giving me is for a “netflow collector”- which receives NetFlow packets; not sending them. There is another package called pfflowd which seems to be written by the same person/people, but I’m unable to find the latest version of either. So no idea what OpnSense is using really.

So I’m happy to see about why this is a problem and try to fix it, but someone is going to have to provide a pcap of say 100-200 netflow v9 packets. Just attach it here to this ticket, or if you’re super privacy worried about someone seeing your flow data, just email me… my email address is my github username at gmail.

Hi @gitet … in the documentation it says that we should use zmq:// instead of tcp:// … but both give the same result … I have le flows but all throughput and total bytes are 0 just like you

I have the same problem with ntopng 6.0 (stable) —> no traffic being recorded … all 0

Using ntopng with --interface zmq://x.x.x.x:5556

Would love it if this was corrected 😃