opendatacam: Memory leak using Gstreamer pipelines as video input

Yes you can use htop from terminal

sudo apt-get install htop

htop

It will display the processes and the memory usage, you will have a line with ./uselib blabalbalbala which should be on top and on the left you have the memory usage…

Looking for people to test OpenDataCam v3.0.0-alpha.1 on Jetson nano to see if memory leak problem is fixed

Docker setup

Build only available on jetson nano for now.

Should work from jetpack 4.2.x , if not please flash to 4.3

# Remove previous install files or run setup from a new folder

Remove config.json, install-opendatacam.sh, run-docker.sh, run-opendatacam.sh

# Download install script
wget -N https://raw.githubusercontent.com/opendatacam/opendatacam/v3.0.0-alpha.1/docker/install-opendatacam.sh

# Give exec permission
chmod 777 install-opendatacam.sh

# NB: You will be asked for sudo password when installing the docker container

# Install command for Jetson Nano
./install-opendatacam.sh --platform nano

Non-docker setup:

If you want to install without docker (on another platform than jetson nano), follow this documentation:

https://github.com/opendatacam/opendatacam/blob/v3.0.0-alpha.1/documentation/USE_WITHOUT_DOCKER.md

Good news 🎉, I have been able to enhance this new ./uselib API for darknet with missing things, still a work in progress: https://github.com/tdurand/darknet/pull/1/files

I’m still figuring out some details for the JSON stream of detections to be able to hook opendatacam node app to it but it is mostly under control, and adding things didn’t seem to create any memory leak at first sight.

Will finish tomorrow and hopefully have a first build of opendatacam to test without this infamous memory leak.

After installing on a nano with Jetpack latest (4.3) and making the latest version of darknet with OpenCV v4 and gstreamer (default from this jetpack version) I found out that:

• Running darknet with ./darknet detector demo - original C implementation of detection on video that is built-in darknet , is still leaking like previously, crashed after only 10 min on a jetson nano

• Running darknet with ./uselib - a new C++ implementation of detection on video that was made by the author of the darknet fork, is not leaking 🎉🎉 , still running with no memory increase after 2 hours

This confirms that the problems comes from darknet and not from OpenCV / Gstreamer.

The less good news is that this new way of running darknet with ./uselib is lacking essentials features we need to integrate to opendatacam:

• Beeing able to run without a GUI
• Beeing able to foward the video as a MJPEG stream
• Beeing able to send detections as a json stream
• Beeing able to input a gstreamer pipeline

This “theoretically” shouldn’t be that hard to integrate as we would “just” need to port them from the old C API if I were a bit more proficient with C / C++ . I answered the author of the fork for some pointers: https://github.com/AlexeyAB/darknet/issues/4453#issuecomment-593450826 , will see as he is generally very responsive.

This is definitely very promising to be able to get rid of this issue and make OpenDataCam much more reliable. cc @b-g

Some update on this, I’ve finished the first integration (see changes to be made to darknet: https://github.com/tdurand/darknet/pull/1/files) and was able to run properly opendatacam using this new darknet API on both Jetsons and Ubuntu CUDA machine.

So far no sign of memory leaks from the tests I did. (ran more than 10 hours on a jetson nano)

I’m now working on providing a build installable via Docker, and also on updating the documentation for a non-docker install… I aim to have something ready to be tested by end of this week.

The Docker build is a bit more work to do than expected because with this new darknet API our docker pipeline for jetson’s isn’t working out of the box (cause of opencv versions I think). Instead of patching even more an already pretty hacky way of dockerize opendatacam on jetson’s, I’m looking into using the newly released Jetpack base container for docker, which should simplify the pipeline (and unify between CUDA machine and jetsons) and also normally gives us the possibility to use CSI camera (raspberry pi cam for example) from docker.

cc @b-g