vector: Docker source: odd 501 Not Implemented errors
I was doing some work on a test harness, and somehow I was able to trigger this behavior:
Feb 07 07:13:41.973 INFO source{name=in type=docker}: vector::sources::docker: Started listening logs on docker container id=3d8e163a3ce6431684cab14a4dbc7ca83e68ce10d58a730b2ecb9c3a94a5d5b6
Feb 07 07:13:41.977 ERROR vector::sources::docker: docker API container logging error error=Docker Error: Docker Error: 501 Not Implemented
Feb 07 07:13:41.977 INFO vector::sources::docker: Stoped listening logs on docker container id=3d8e163a3ce6431684cab14a4dbc7ca83e68ce10d58a730b2ecb9c3a94a5d5b6
Feb 07 07:13:41.978 INFO source{name=in type=docker}: vector::sources::docker: Started listening logs on docker container id=3d8e163a3ce6431684cab14a4dbc7ca83e68ce10d58a730b2ecb9c3a94a5d5b6
Feb 07 07:13:41.981 ERROR vector::sources::docker: docker API container logging error error=Docker Error: Docker Error: 501 Not Implemented
Feb 07 07:13:41.981 INFO vector::sources::docker: Stoped listening logs on docker container id=3d8e163a3ce6431684cab14a4dbc7ca83e68ce10d58a730b2ecb9c3a94a5d5b6
All the configs are at https://github.com/MOZGIII/vector-merge-test-setup
Steps to reproduce:
- Clone and cd to the
vector-merge-test-setuprepo. - Grab the
tcp_test_server:go get github.com/timberio/tcp_test_server - Start the
tcp_test_server:tcp_test_server --address=0.0.0.0:9000 - Start vector with
vector/test_harness.tomlconfig:../vector/target/debug/vector -c vector/test_harness.toml(I was runing it from the build dir at../vector) - Start log producer:
./run_log_once_container.sh --build - If the bug is not triggered immediately, run the above command a couple more times.
Vector should start printing the errors as above at a high rate (100s per second).
$ ../vector/target/debug/vector --version
vector 0.8.0 (g5b97e4c x86_64-unknown-linux-gnu 2020-02-02)
$ docker -v
Docker version 19.03.5, build 633a0ea838
UPD: improved steps to reproduce UPD2: I can’t reproduce it anymore on my end
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 31 (31 by maintainers)
@MOZGIII so from what you’re saying this means vector needs to run from within a container. How would we load the config? From what I can tell a lot of the logging drivers have you enumerate the entire config via log opts. Which wouldn’t fit with our model. I think this is why fluentd/fluentbit went with the fluentd driver shipping their fwd protocol to a centralized fluent* app. That way the configuration for docker is very light and easy. From what I can tell the UX with complex docker logging drivers is much worse than our current solution.
Another thing that we could potentially do if performance is an issue is provide a config that works via jsonfile and its output to disk in
/var/log/docker. Similar to what we do for kubernetes. Since, iirc jsonfile is the default this could be an easy way to set it up.The other concern about refactoring a lot of this, is I have yet to see anyone really note that the performance is an issue or any other issues with docker (I don’t count the vector in container as an issue as that was a misconfigured container). This makes mean concerned about investing more time into writing new plugins for vector here.
@MOZGIII that sounds good, let’s defer that work though. Your current work is more important. Well open an issue and assign it to you when it’s a priority.
There is a useful guide somewhere deep in these details, that I’d like to figure out.
I’ll definitely try
splunk.So far, it’s possible to use the following docker log drivers with vector:
I agree that it’d be very valuable adding a guide for various ways it’s possible to work with docker. It would be great if we could do some research to be able to actually recommend them per use case.
@MOZGIII do you have any more info about how you setup your docker daemon?
Looks like there is a small note I found in the logging api of docker that says
I am going to open a PR to add this note to ours docs though this should only be possible to enable if you are not using CE, since iirc these are the only supported logging drivers on CE.
But it does 501 in this case, I’d like to get a better warning in our docker code. Ill open a PR for both.