linux, projects, raspberry pi

CCTV GUIDE PART 4 – Archiving and cleanup, and setting up the Live monitor

Leave a comment

If you’ve followed this far and got it working you’ve probably seen this system is going to produce a LOT of JPEG files. Ours spits out around 17,000 per day.  That amount of files is gonna quickly get unmanageable.

Also linux disks tend to have a limited number of “i-nodes”, which work like name tags for files. When your disk runs out of name tags it’s “full” whether it’s actually full to data capacity or not. Storing gazillions of tiny JPEGs is a surefire way to run out of i-nodes quickly.

So what we’ll do is make it so that today’s JPEGs are loose, but yesterday’s get zipped up into a file with a datestamp, resulting in 1 inode used per day instead of ~17,000.

Even zipped up the actual disk space usage will also stack up pretty quickly if unchecked so we’ll make it so that the system automatically deletes files older than 7 days.

 

The idea of this system is so that there’s a record in case of a break-in or other “shenanigan”, so these files won’t really be getting looked at unless there’s a problem. In our case with finding the guy round the side of our house..if we’d been out and he’s stolen something, as long as we noticed within the timeframe of our archived files (7 days)  we could have gone through that day’s JPEGs and found a mugshot to give to the relevant authority as evidence.

As it turns out it’s also proven useful for trying to locate a neihbour’s missing cat..letting us find out when it was last seen in the area (it likes to sit on our driveway).

So to achieve this we’ll have two scripts..”doarchive.sh” and “delete_old_archives.sh”, both of which will get run by cron on a daily basis.

NOTE: In the terminal/console bits I’ll put what you’re supposed to  type in bold. The rest is the responses you should get from the system. Errors in red, important bits to note or change are in pink.

 

Archive the previous day’s files

So ssh in to your Raspberry Pi or other linux box and cd to the cctv folder..and open up doarchive.sh in nano:

pi@pi1:~/cctv $ sudo nano doarchive.sh

which should bring up the archiving script:

#!/bin/sh

FILENAME=ipcam
echo "Filename: $FILENAME"

yesterdays_date=$(date --date="1 days ago" "+%Y-%m-%d")
echo "Yesterday's date : $yesterdays_date"

current_time=$(date --date="1 days ago" "+%Y-%m-%d--%H-%M-%S")
echo "Current time : $current_time"

cd /home/pi/www/cctv/recorded/

#sudo find /home/pi/www/cctv/recorded/ -name \*$yesterdays_date*.jpg -exec ls {} \;

zipfile=$FILENAME"_"$yesterdays_date".zip"

zip ../archived/$zipfile *$yesterdays_date*.jpg

rm *$yesterdays_date*.jpg

#sudo find /home/pi/www/cctv/recorded/ -exec ls {} \;

#sudo find /home/pi/www/cctv/recorded/ -mtime +2 -exec rm {} \;

The only things you should need to adjust are the paths (in pink) if you’re using a username other than “pi” or have changed where the recordings are stored. If you’re using the same setup as I’ve described here you shouldn’t need to change anything.

By default it’ll look in the “recorded” folder (where the loose JPGs live), match any files with yesterday’s date in the name and add them to a zipfile in the “archived” folder. Then it deletes the JPGs.

Once the paths are right for your system you can hit CTRL+X, then y to save changes. If you don’t have any files from yeserday then rename a couple of the existing JPEGs to have yesterday’s date and then test the script out by running:

pi@pi1:~/cctv $ ./doarchive.sh

Then check the “archived” folder and with a bit of luck you should see a zipfile

pi@pi1:~/cctv/archived $ ls

ipcam_2017-03-03.zip

So once that’s working we need to automate this so it happens every day, so open up the crontab

pi@pi1:~/cctv $ sudo nano /etc/crontab

and under the lines we added earlier, add the bits in pink so it looks like this:

#cctv
* * * * * pi /home/pi/cctv/snapshot.sh >> /home/pi/cctv/webcam_log.txt
0 1 * * * pi /home/pi/cctv/doarchive.sh

 

That tells the cron to run the archiving script every day at 1am, i.e. when all the files for “yesterday” are there, but before there’s too many new ones to clog things up.

 

Delete old archives

Next cd back to the cctv folder and open up delete_old_archives.sh in nano:

pi@pi1:~/cctv $ sudo nano delete_old_archives.sh

This is a simple one-liner of a script and all you need to do is make sure your path is correct, and adjust the number of days you want to keep. The default is 7:

#!/bin/sh

sudo find /home/pi/cctv/archived/ -mtime +7 -exec rm {} \;

Once you’re done CTRL+X, then Y to save and exit and we can add this script to the cron too. This will run every day and delete archives older than 7 days.

Open up cron:

pi@pi1:~/cctv $ sudo nano /etc/crontab

and add one last line so it looks like this (adjusting your user & paths if necessary):

#cctv
* * * * * pi /home/pi/cctv/snapshot.sh >> /home/pi/cctv/webcam_log.txt
0 1 * * * pi /home/pi/cctv/doarchive.sh
0 3 * * * pi /home/pi/cctv/delete_old_archives.sh

That tells it to run the delete_old_archives.sh script every day at 3am. That way there’s plenty of time for it to have finished zipping up the previous day’s files, even if it’s a really slow machine.

And that’s it for the logging/archiving bit!

 

Live monitor setup:

In comparison this bit is super easy. You can do this bit in the Pi’s GUI if it’s easier for you but if you don’t have a proper keyboard and mouse attached to your Pi (like I don’t) then it’s easier to do over SSH.

Cd to your cctv folder and open up index.html in nano:

pi@pi1:~/cctv $ sudo nano index.htm

which should give you something like this:

<html>
<head>
<title>CCTV</title>
<style>
BODY
{
background-color: black;
margin: 0;
padding: 0px;
}
</style>

</head>


<body>
<center>
<img id=loading src="clear.gif" width=20 height=20 style="position: absolute; left: 0px; top: 0px">
<img id=camera src="http://192.168.0.123:8080/snapshot.cgi?user=admin&pwd=password" style="width: 100%; height: 100%;"><br>

</center>


<script language=javascript>


document.getElementById('loading').src = 'square_red.gif';

function dopic()
 {
 document.getElementById('loading').src = 'square_green.gif';
 document.getElementById('camera').src = 'http://192.168.0.123:8080/snapshot.cgi?user=admin&pwd=password&' + new Date().getTime() + Math.random();
 setTimeout("document.getElementById('loading').src = 'clear.gif'",500);
 }

var sqred = new Image()

sqred.src = 'square_red.gif';

//setTimeout('dopic()',3000);
var x = setInterval('dopic()',1500);


setTimeout('location.reload()', 300000)

</script>

</body>
</html>

You should only have to edit the bits in pink to use the special URL you found for your camera in the previous parts, and the username and password for your camera since that’s where the page fetches the image from. All the rest should run as is.

Then log into the GUI of your live monitoring machine and browse to the cctv folder and double-click the index.htm file to open it in the browser. Then click F11 to make it full screen.

That *should* now be working – and you should see a little green square in the top left corner every time the page is trying to fetch an image off the camera (so you can see it’s doing something).

The page will also periodically reload itself (every 5 mins) just in case it falls over (e.g. wifi drops or something else goes wrong).

It may seem basic but this format has been working for me for over 6 months now and hardly ever needs manual intervention.

Aaah but…

Yes there’s always a “but”…

If you’re using a default install of Raspbian like I am then you’ll notice that after 10 mins or so the screen goes blank (power saving mode), which is obviously not what you want for an “always-on” monitor.

This info is pulled from the URL below and shows how to prevent the screen from going blank: http://www.geeks3d.com/hacklab/20160108/how-to-disable-the-blank-screen-on-raspberry-pi-raspbian/

$ sudo nano /etc/lightdm/lightdm.conf

Then under [SeatDefaults] add or uncomment/modify the line so it reads:

[SeatDefaults]
xserver-command=X -s 0 -dpms

You may need to reboot to see the changes but then your screen should no longer go blank.

 

Lastly & other…

The latest version of Raspbian has chrome instead of Iceweasel and that has a “kiosk” mode. Personally I couldn’t get it working in a way I was happy with and the LoFree wireless touchpad & mouse I have attached to my Pi means on the rare occasions it gets rebooted I can just click to open the browser, which is set to re-open the last tabs..but if you don’t have a keyboard attached and want to fully automate the entire process it’s something you may want to look into:

Chrome/kiosk mode: https://www.raspberrypi.org/forums/viewtopic.php?t=8298&p=210611

And in case you need to set a static IP for your Pi, that also works a little differently in the lastest version of Raspbian/Pixel aka “Jessie”:

Set a static IP: https://www.modmypi.com/blog/how-to-give-your-raspberry-pi-a-static-ip-address-update

 

Security:

One thing to note is that often your IP cam will let you define several user-levels to access it. For this example we’ve been using the “admin” account. but given the fact that the username and password are both stored in plain text numerous times, once it’s all working right you might want to set up a less privileged “user” login on the camera and put those login details into the scripts instead.

Obviously unless you have your camera, Pi or network exposed to the internet it’s probably not a huge concern, but things do go wrong, networks do get hacked so it’s probably better to limit the damage which can be done given the popularity of IOT devices like IP cameras in the recent wave of botnet attacks (e.g. Mirai) and given how insecurely the logins are stored in this case.

That way the worst anyone could do would be look at your camera..probably*

(*though if they’re reading files on your Pi you probably have bigger issues already).

Leave a Reply