goflow2: enricher mismatching end group marker

Hello! I’m trying to map IP to AS with enricher, but got this errors:

ERRO[0002] unexpected EOF                               
ERRO[0002] proto: invalid field number                  
ERRO[0002] proto: mismatching end group marker          
ERRO[0002] unexpected EOF                               
ERRO[0002] proto: invalid field number                  
ERRO[0002] proto: mismatching end group marker          
ERRO[0002] unexpected EOF                               
ERRO[0002] proto: invalid field number                  
ERRO[0002] proto: mismatching end group marker          
ERRO[0002] unexpected EOF                               
ERRO[0002] proto: invalid field number

GoFlow2 v1.0.4 (2021-06-12T16:48:35+0000)

I run this pipe as follow: goflow2 -format=pb | ./enricher -db.asn=/var/lib/geoip/GeoLite2-ASN.mmdb -db.country=/var/lib/geoip/GeoLite2-Country.mmdb

And if I understand correctly, the full pipe must look like this, if I want to map IP to AS: sflow_stream -> goflow2 -> enricher -> kafka -> ch Am I correct?

About this issue

Original URL
State: closed
Created 3 years ago
Comments: 16 (9 by maintainers)

Most upvoted comments

I merged to main, will make sure new options are good and then will make a release. Thank you for reporting the issue!

lspgn on Aug 29, 2021

Have you built the enricher as well on this particular branch?

Edit: I tested replaying your packet with both GoFlow2 and the enricher on commit gc107b6b and it correctly decodes
$ tcprewrite -i capture.pcap -o capture-mod.pcap -D 195.78.39.140:192.168.x.x --enet-dmac=test-machine-mac
$ tcpreplay -i interface capture-mod.pcap

It works! First time I only use goflow2 binary from artifacts. Then I try to build both enricher and goflow2 from bug/enricher-decode branch and all good. Thank you!

freedomwarrior on Aug 29, 2021

Alright I think I know what I did wrong and my apologies for not seeing it earlier. The protobuf its generating contains a line separator, the enricher then cuts the packet in the middle, this is why it is decoded wrong. Will come up with a fix.

lspgn on Aug 27, 2021

Pipe is correct. I would advise to write an enricher that consumes from Kafka as well instead of using a pipe if you have a lot of flows.

Per #105 I believe this is resolved (use for GoFlow2-format.protobuf.fixedlen=true). Will close, feel free to reopen/create a new issue if you have more questions.

lspgn on Aug 20, 2021