Important memory leak of v4l2-relayd service after each camera start

[thomasarmel]

Hi dear developers,
Some time ago, I noticed an important memory leak of v4l2-relayd service on my Ubuntu 22.04.4 LTS laptop : a couple of GB after some video calls.

I decided to investigate using valgrind and found it comes from this lib.

I noticed memory is leaked each time camera is restarted.

==65931== 1,080 bytes in 2 blocks are definitely lost in loss record 5,731 of 6,137
==65931==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==65931==    by 0xA305045: icamera::CIPR::Command::Command(icamera::CIPR::PSysCommandConfig const&) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA28BB49: icamera::PGCommon::createCommand(icamera::CIPR::Buffer*, icamera::CIPR::Command**, icamera::CIPR::Buffer**, int) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA28BDAD: icamera::PGCommon::createCommands() (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA28C7A7: icamera::PGCommon::iterate(std::map<unsigned int, std::shared_ptr<icamera::CameraBuffer>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::shared_ptr<icamera::CameraBuffer> > > >&, std::map<unsigned int, std::shared_ptr<icamera::CameraBuffer>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::shared_ptr<icamera::CameraBuffer> > > >&, ia_binary_data*, ia_binary_data const*) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA27FD91: icamera::PipeLiteExecutor::runPipe(std::map<icamera::Port, std::shared_ptr<icamera::CameraBuffer>, std::less<icamera::Port>, std::allocator<std::pair<icamera::Port const, std::shared_ptr<icamera::CameraBuffer> > > >&, std::map<icamera::Port, std::shared_ptr<icamera::CameraBuffer>, std::less<icamera::Port>, std::allocator<std::pair<icamera::Port const, std::shared_ptr<icamera::CameraBuffer> > > >&, std::vector<std::shared_ptr<icamera::CameraBuffer>, std::allocator<std::shared_ptr<icamera::CameraBuffer> > >&, std::vector<icamera::EventType, std::allocator<icamera::EventType> >&) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA281851: icamera::PipeLiteExecutor::processNewFrame() (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA249A37: ??? (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA29032E: icamera::Thread::_threadLoop(icamera::Thread*) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA015252: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==65931==    by 0x4C16AC2: start_thread (pthread_create.c:442)
==65931==    by 0x4CA7A03: clone (clone.S:100)

==65931== 36 bytes in 1 blocks are definitely lost in loss record 2,892 of 6,137
==65931==    at 0x484DA83: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==65931==    by 0xA30470C: icamera::CIPR::Context::registerBuffer(icamera::CIPR::MemoryDesc*) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA289916: icamera::PGCommon::createUserPtrCiprBuffer(int, void*, bool) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA28A304: icamera::PGCommon::registerUserBuffer(int, void*, bool) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA28B678: icamera::PGCommon::prepareTerminalBuffers(ia_binary_data const*, std::map<unsigned int, std::shared_ptr<icamera::CameraBuffer>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::shared_ptr<icamera::CameraBuffer> > > > const&, std::map<unsigned int, std::shared_ptr<icamera::CameraBuffer>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::shared_ptr<icamera::CameraBuffer> > > > const&, long) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA28C5A2: icamera::PGCommon::iterate(std::map<unsigned int, std::shared_ptr<icamera::CameraBuffer>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::shared_ptr<icamera::CameraBuffer> > > >&, std::map<unsigned int, std::shared_ptr<icamera::CameraBuffer>, std::less<unsigned int>, std::allocator<std::pair<unsigned int const, std::shared_ptr<icamera::CameraBuffer> > > >&, ia_binary_data*, ia_binary_data const*) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA27FD91: icamera::PipeLiteExecutor::runPipe(std::map<icamera::Port, std::shared_ptr<icamera::CameraBuffer>, std::less<icamera::Port>, std::allocator<std::pair<icamera::Port const, std::shared_ptr<icamera::CameraBuffer> > > >&, std::map<icamera::Port, std::shared_ptr<icamera::CameraBuffer>, std::less<icamera::Port>, std::allocator<std::pair<icamera::Port const, std::shared_ptr<icamera::CameraBuffer> > > >&, std::vector<std::shared_ptr<icamera::CameraBuffer>, std::allocator<std::shared_ptr<icamera::CameraBuffer> > >&, std::vector<icamera::EventType, std::allocator<icamera::EventType> >&) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA281851: icamera::PipeLiteExecutor::processNewFrame() (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA249A37: ??? (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA29032E: icamera::Thread::_threadLoop(icamera::Thread*) (in /usr/lib/x86_64-linux-gnu/libcamhal/plugins/ipu6ep.so)
==65931==    by 0xA015252: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==65931==    by 0x4C16AC2: start_thread (pthread_create.c:442)

I plan to investigate a little bit further myself, please do not hesitate if you need more precisions :)

Best regards

[hao-yao]

Thank you @thomasarmel . Would you help attach the full valgrind report?

[thomasarmel]

Hi,
Yeah sure ! Please note that due to execution slowness on valgrind context v4l2 captured only a couple of frames.

v4l2_full_valgrind.txt

[hao-yao]

What was your test flow? When libcamhal is killed, valgrind will report many places leaked but I added num-buffers=5 to make it automatically exit after 5 frames. In this case, valgrind only reported 3 leak points which are all in libia_cca.so.

[thomasarmel]

After capturing a few frames I sent a SIGINT to the v4l2-relayd process, in order to ensure Valgrind exited correctly (which is not the case using SIGKILL).
But anyway the memory leak is obvious regarding memory consumption of the service.

My OS is Ubuntu 22.04.4 LTS.

My laptop is a Dell Precision 5480.

Camera format is NV12, 1280x720, 30 fps.

[hao-yao]

What is the application are you using?

[thomasarmel]

For the valgrind report I shared it was google-chrome, but it works with any application that uses the camera.

[hao-yao]

I looked into our source code and did some tests on my side. I found that:

1. The memory leakages in your valgrind report are all handled in normal exiting flow of LibCamHAL. When sending SIGINT to valgrind, the destructors to free resources are not run so valgrind will definitely report it as lost.

2. When set num-buffers=10 to let gstreamer exit normally, we noticed some minor leakage in libia_cca, which has already reported by another customer and fixed. The fix should be applied in next release, however I don't think it can cause such a significant memory leak.

3. Internally we directly use gst-launch-1.0 command to run camera preview. Our internal stress test shows that memory consumption keeps ~ 170 MB at 720p 30fps during 100 minutes, so I don't think 1.5 GB memory consumption comes from the components below gstreamer.

4. I tested with v4l2-relayd + Cheese. With first several times opening, v4l2-relayd consumed from ~170 MB to over 800 MB.

Concluding from the above, the memory leakage should happens in the v4l2-relayd or v4l2loopback. Canonical owns v4l2-relayd and v4l2loopback now, so I think we need their help to look into this issue.

[thomasarmel]

Hi,
Ok nice thanks for your support!

[fourdollars]

Could you execute sudo gst-launch-1.0 icamerasrc ! autovideosink to check if the memory leakage still exists?

[thomasarmel]

Hi,
It seems that v4l2-relayd doesn't leak when I execute this command.