kerberosio - very high memory usage when 1024x768 resolution and delay set to 100

[ChieftainY2k]

Memory usage is VERY high (eventually bringing my raspberry pi4 to a halt) when using specific configuration for raspberry onboard camera.

Steps to reproduce:

1. Run: docker run --name camera -p 80:80 -p 8889:8889 -d -v /dev/vcsm:/dev/vcsm -v /dev/vchiq:/dev/vchiq --privileged kerberos/kerberos
2. After initial configuration go to http://............./settings -> ADVANCED
3. Change resolution to 1024x768
4. Change delay to 100
5. Make a continuous motion in front of the onboard camera ( I used a youtbe wideo with fractal zoom :) )

Result:
Within several minutes memory usage soars over 1.5GB, eventually raspberry pi runs out of memory, swap space kicks in and everything is halted.

Expected result:
Memory usage is expected to be reasonable :-)

Rig specs:

• Hardware: Raspberry Pi 4 Model B Rev 1.1 (2GB)
• OS: Raspbian GNU/Linux 10 (buster) with newest updates

[cedricve]

Thanks for reporting do you see some recordings as well in the /capture dir?

[ChieftainY2k]

Yes there are some recordings:

root@94c017d9cd88:/etc/opt/kerberosio/capture# ls -la
total 67556
drwxr-xr-x 2 root root     4096 May 28 07:12 .
drwxr-xr-x 1 root root     4096 Apr 24 19:21 ..
-rw-r--r-- 1 root root    47930 May 27 19:22 1590607352_6-177182_frontdoor_regionCoordinates_numberOfChanges_281.jpg
-rw-r--r-- 1 root root  6653854 May 27 19:23 1590607352_6-740963_frontdoor_212-21-552-413_7037_281.mp4
-rw-r--r-- 1 root root  3937858 May 27 19:23 1590607382_6-785996_frontdoor_192-22-494-459_11632_920.mp4
-rw-r--r-- 1 root root   145616 May 27 19:23 1590607411_6-746586_frontdoor_regionCoordinates_numberOfChanges_200.jpg
-rw-r--r-- 1 root root 10493829 May 28 07:11 1590607411_6-887929_frontdoor_393-24-790-615_45463_200.mp4
-rw-r--r-- 1 root root 10487305 May 28 07:11 1590607442_6-386930_frontdoor_392-18-791-620_51073_633.mp4
-rw-r--r-- 1 root root  7376259 May 28 07:11 1590607472_6-879170_frontdoor_396-24-784-615_5096_470.mp4
-rw-r--r-- 1 root root  9585212 May 28 07:11 1590607503_6-351713_frontdoor_487-31-780-603_1300_381.mp4
-rw-r--r-- 1 root root 10495181 May 28 07:11 1590607533_6-841077_frontdoor_392-24-790-635_29013_964.mp4
-rw-r--r-- 1 root root  9446061 May 28 07:11 1590607564_6-411256_frontdoor_391-27-790-636_44855_111.mp4
-rw-r--r-- 1 root root   235383 May 28 07:11 1590649859_6-981923_frontdoor_regionCoordinates_numberOfChanges_405.jpg
-rw-r--r-- 1 root root   236548 May 28 07:12 1590649957_6-154057_frontdoor_regionCoordinates_numberOfChanges_907.jpg

[ChieftainY2k]

@cedricve I can set up the ssh+ngrok service in the kerberos container so that you can remotely login with ssh into it and take a look inside.

[cedricve]

thank you @ChieftainY2k, are you able to reproduce as well?

[cedricve]

@ChieftainY2k were you experiencing, or did you ever tested, with previous Kerberos version? I understand you were using the .deb files before?

[ChieftainY2k]

@cedricve I was able to reproduce the problem with the following setups:

• Official KIOS SD image 2.8.0 (https://github.com/kerberos-io/kios/releases/download/v2.8.0/kios-raspberrypi4-2.8.0.img.gz)
• Docker image from the official kerberos docker repository (https://hub.docker.com/r/kerberos/kerberos)
• My own docker image with deb package from https://github.com/kerberos-io/machinery/releases/download/v2.8.0/rpi4-machinery-kerberosio-armhf-2.8.0.deb (with corresponding libx264.so.148 and libx265.so.160)
• My own docker image with deb package from https://github.com/kerberos-io/machinery/releases/download/v2.8.0/rpi3-machinery-kerberosio-armhf-2.8.0.deb (with corresponding libx264.so.148 and libx265.so.160)
• My own docker image with deb package from https://github.com/kerberos-io/machinery/releases/download/v2.6.2/rpi3-machinery-kerberosio-armhf-2.6.2.deb

All above running with the Raspberry Pi 4 Model B Rev 1.1 (2GB)

Could not test the Official KIOS SD image 2.7.2 (https://github.com/kerberos-io/kios/releases/download/v2.7.2/kios-raspberrypi3-20180714.img.gz) - my RPI4 would not even boot with this image, I don't know why (tested with two different SD cards to be sure).

[cedricve]

Thanks for the detailed information! RaspberryPi 4 is only supported from 2.8.0. What happens if you test with a Raspberry Pi 3? Just curious.. Kind regards, Verstraeten Cédric

…

On 30 May 2020, at 11:27, ChieftainY2k ***@***.***> wrote:  @cedricve I was able to reproduce the problem with the following setups: Official KIOS SD image 2.8.0 (https://github.com/kerberos-io/kios/releases/download/v2.8.0/kios-raspberrypi4-2.8.0.img.gz) Docker image from the official kerberos docker repository (https://hub.docker.com/r/kerberos/kerberos) My own docker image with deb package from https://github.com/kerberos-io/machinery/releases/download/v2.8.0/rpi4-machinery-kerberosio-armhf-2.8.0.deb (with corresponding libx264.so.148 and libx265.so.160) My own docker image with deb package from https://github.com/kerberos-io/machinery/releases/download/v2.8.0/rpi3-machinery-kerberosio-armhf-2.8.0.deb (with corresponding libx264.so.148 and libx265.so.160) My own docker image with deb package from https://github.com/kerberos-io/machinery/releases/download/v2.6.2/rpi3-machinery-kerberosio-armhf-2.6.2.deb All above running with the Raspberry Pi 4 Model B Rev 1.1 (2GB) Could not test the Official KIOS SD image 2.7.2 (https://github.com/kerberos-io/kios/releases/download/v2.7.2/kios-raspberrypi3-20180714.img.gz) - my RPI4 would not even boot with this image, I don't know why (tested with two different SD cards to be sure). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[cedricve]

also @ChieftainY2k what is the reason of setting delay to 100? Do you also experience with lower values?

[ChieftainY2k]

@cedricve I was just playing around with optimal parameters to detect motion in the area my camera is pointed at and then after setting delay do 100 this memory bug happened. Then I set it back to 500 everything goes back to normal after a while.

[ChieftainY2k]

Just tested it with following setup and was able to reproduce the memory problem:

• Hardware: Raspberry Pi 3 Model B Plus Rev 1.3
• SD Image: https://github.com/kerberos-io/kios/releases/download/v2.7.2/kios-raspberrypi3-20180714.img.gz

[cedricve]

Interesting it might be that the os is not able to free up the resources. Can you experiment (incremental) which params are causing this issue. Could be that it is resolved once beyond 250 (wild guess). You said that with 500 it is ok? But probably this takes to long for you to detect motion?

I will look into the source code to find the root cause. Let me know what you find as well 👌

Thanks for the hard work @ChieftainY2k

Let me know

[meeki007]

This Jogged my brain of the issue I had with memory overloading.
I set limits on logs when starting container.

docker run -it --restart always --log-opt max-size=5m --log-opt max-file=10 \
-v /home/cameras/livingroom/config:/etc/opt/kerberosio/config \
-v /home/cameras/livingroom/webconfig:/var/www/web/config \
-v /home/cameras/livingroom/logs:/etc/opt/kerberosio/logs \
-v /home/cameras/livingroom/controllers:/var/www/web/app/Http/Controllers \
-v /home/cameras/livingroom/drivespace_script/runny.sh:/runny.sh \
-v /mnt/RAID10sdb1/CameraVideoStorage/livingroom:/etc/opt/kerberosio/capture \
--name livingroom -p 10000:80 -p 12000:8889 -d kerberos/kerberos

[cedricve]

We are working on a complete rewrite of the open source version. When using h264 cameras, you will be able to record at almost no CPU (<1%). Already implemented through our enterprise offering but not yet released in the open source repo,
https://github.com/kerberos-io/opensource