Full disk is a lousy reason to wake up in a middle of the night

In my last post I wrote about Laconi.ca and mentioned that we have many bots updating Laconi.ca too. The bots are made active when some condition has met.

Example: Make server running out of disk space to post an update to Laconi.ca

This is actually quite useful. Sure, you can tell every service to send an email when something not wanted is happening but then you will have to read that mailbox too. Sending email to someone else doesn’t solve the problem. Hmm, updating to Laconi.ca doesn’t solve the problem either but it makes servers easier follow and to subscribe.

So here is a simple shell script:

#!/bin/sh
fs=`mount|egrep '^/dev'|grep -iv cdrom| awk '{print $3}'`
typeset -i thresh="96"
typeset -i warn="98"
for i in $fs
do
skip=0
typeset -i used=`df -k $i|tail -n 1|awk '{print $5}'|cut -d "%" -f 1`
if [ "$used" -ge "$warn" ]; then
laconica.sh "1.2.3.4 server CRITICAL: filesystem $i is $used% full"
fi
if [ "$used" -ge "$thresh" -a "$used" -le "$warn" ]; then
laconica.sh "1.2.3.4 server WARNING: filesystem $i is $used %full"
fi
done

I didn’t want to use cURL directly to talk to Laconi.ca so I downloaded laconica.sh from http://downloads.guillermoamaral.com/misc/laconica. Sure it uses cURL, but if I change a password no I can change it to this wrapper instead.

I saved the script as diskfull.sh. So lets first chmod it and then try it:

chmod 755 diskfull.sh
./diskfull.sh

Laconi.ca gets an update as follows:

Screenshot of Laconi.ca

You would need schedule Cron to run the script. To do so, edit the /etc/crontab and add the following line:

31 8 * * * root /opt/diskfull.sh

This line will run it daily on 08:31 as root. You might want to change it. If you are not familiar with Cron here is some help.