Photo courtesy of Mads Sabroe

We all love Unix shell. I can’t remember when was the last time I had my Mac running without an open Terminal window. Here are couple of tricks you can do with it.

1. “cd -”
After finding this in all of its simplicity I asked many of Linux & OS X hardcore users and none of them knew this existed. “cd -” sends you back to the last directory you where in:

macbookpro:/ tuomas$cd usr/bin/
macbookpro:bin tuomas$ cd -
/
macbookpro:/ tuomas$ cd -
/usr/bin

2. “mkdir -p”
With mkdir you can make a directory, but adding the “-p” argument can make multiple directories in case those do not exist before. For example:

mkdir -p /home/bill/documents/books/computer

3. Update Twitter from shell with cURL

curl -u username:password -d status="Your tweet" http://twitter.com/statuses/update.xml

4. Speak a text file to spoken .aiff file

cat foo.txt |say -o foo

Now try this with Apache log;)

5. “history”
History shows you list of commands you have ran before like this:

526  cat blog-google-api.txt |say -o foo.mp3
527  open .
528  ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'
529  history

If you want to run something from your history you can just: !527 and it will run command #527 from the history. In this case: “open .”

#!/usr/bin/env python
# coding: utf-8

import os
import re
import sys

def grab_email(file):
    """Try and grab all emails addresses found within a given file."""
    email_pattern = re.compile(r'\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b',
                               re.IGNORECASE)
    found = set()
    if os.path.isfile(file):
        for line in open(file, 'r'):
            found.update(email_pattern.findall(line))
    for email_address in found:
        print email_address

if __name__ == '__main__':
    grab_email(sys.argv[1])

The API is not very well documented but you can browse its capabilities be opening http://URL-TO-SUGAR/soap.php in your browser

At work we use it by ourselves to handle some processes and have been working with customers using it. There are things I like and thing I don’t like in SugarCRM. It is bit bloated and the code is not the most beautiful I have seen. On the other hand it does its job quite well and it is easy to configure without touching a line of code.

If you use a CRM in your business for a while you will most likely to find that you would like to do some integration. At work we have been doing various kinds of integration tasks. We have integrated a phone system to automatically make entries to SugarCRM when inbound or outbound calls have been made, connected registration of a web application to create a new accounts to CRM, connected a LIMS in various ways to it and so on. The reason I found SOAP-API to be the best way in most these cases is the fact that if you use it you won’t break down anything and it is likely that your code will work with future releases too. Off course there is a performance penalty and using SOAP API is not a good option when performance is an issue. So lets get to the business.

Installing SOAPpy
I assume that installing Python module with easy_install is not a problem. However, latest version of SOAPpy was not compatible with Python 2.5. To install SOAPpy I would suggest the following:

1. Download and extract SOAPpy http://sourceforge.net/project/showfiles.php?group_id=26590&package_id=18246

2. Move every “from __future__ import” lines to the beginning of the files where they are in. Below you can see the listing of these files.

localhost:SOAPpy-0.12.0 tuomas$ grep -R __future__ SOAPpy/*.py
SOAPpy/Client.py:from __future__ import nested_scopes
SOAPpy/GSIServer.py:from __future__ import nested_scopes
SOAPpy/NS.py:from __future__ import nested_scopes
SOAPpy/Server.py:from __future__ import nested_scopes
SOAPpy/Types.py:from __future__ import nested_scopes

3. Install SOAPpy

sudo python setup.py install

The simplest possible example of SugarCRM + SOAPpy
The code below logs in to the SugarCRM, asks the version of the server and prints it. Here is the code:

import md5
import SOAPpy
USERNAME= "user"
PASSWORD= "pass"

auth = {'user_name': USERNAME, 'password': md5.new(PASSWORD).hexdigest(), "version": "1.1"}

# Url of SugarCRM + soap.php?wsdl
SUGAR_URL = 'http://192.160.0.1/soap.php?wsdl'

sugar = SOAPpy.WSDL.Proxy(SUGAR_URL)
session = sugar.login(auth, "foobar")

try:
    response = sugar.get_server_version()
except StandardError, err:
    print '\nError in information retrieval from SugarCRM:' + str(err)
print response

The API is not documented very well but you can list all of its capabilities by browsing yourself to http://URL-TO-SUGAR/soap.php.

Advanced example
This script will log into SugarCRM and create a new Account.

import md5
import SOAPpy
USERNAME= "user"
PASSWORD= "pass"

auth = {'user_name': USERNAME, 'password': md5.new(PASSWORD).hexdigest(), "version": "1.1"}

# Url of SugarCRM + soap.php?wsdl
SUGAR_URL = 'http://192.160.0.1/soap.php?wsdl'

sugar = SOAPpy.WSDL.Proxy(SUGAR_URL)
session = sugar.login(auth, "foobar")
module = "Accounts"

adata = [{'name': 'name', 'value': "Test Account"},
           {'name': 'shipping_address_street', 'value': "Address test"}]
try:
    response = sugar.set_entry(session['id'], module, adata)
except StandardError, err:
     print '\nError in information retrieval from SugarCRM:' + str(err)
print response

UPDATE: It appears to be that, Sugar Team is working with the API and the documentation at the moment and I’ll take back what I said about missing documentation. Thanks SugarCRM devs!

So as I clicked the Admin button and went to admin area I realized that there is a new version of SugarCRM available. Ok, I’ll install the new version first I thought hmm.. lets see release notes hmm.. a new feature Sugar Feeds, sounds cool. Maybe I could finally get Trac like timeline RSS feed from SugarCRM. Here is what release notes said:

Sugar Feed enables users to be informed as soon as a user creates a new contact, lead, opportunity, or case. Users can add a My Sugar Feed dashlet on their Home page so that when a user performs any of these actions, a message displays in the dashlet on
their Home page. Userscan also be notified when a lead is converted, when a case is closed, and when an opportunity is closed.
Additionally, users can post status updates, external links, images, and YouTube videos for other users to view within the dashlet.

SugarCRM Community Edition5.2.2.3.h Release Notes

After reading that I still continued to believe that this would be like users life stream RSS-feed. Guess what, it was not. Sugar Feeds is SugarCRM’s internal message sending service. Sure it sounds kinda cute “aww, now I can send these nice little messages to my co-workers” but really. As long as there is no API and no RSS why would anyone log in to SugarCRM to read and write these messages.

As much as I do value the effort of the SugarCRM’s team and their bold moves on the open source field, I still cannot understand who would want to use Sugar Feeds.

“The average spam messages per day is 18.5 and the average time spent per day deleting them is 2.8 minutes. The loss in productivity is equivalent to $21.6 billion per year at average U.S. wages.”

- Jesdanun, A. (2005), “Deleting spam costs billions, study finds”, The Associated Press Newswires (online)

You can filter spam and viruses before they hit your mailserver

In my opinion the problem that is not solved yet is the deployment. I think every tech savvy person filters their spam already. But how about the rest of the people. Sure, you can use time and install a spam filter to each client, you can also install a spam filter to mail server and get more results.

If you wish to install spam filter to every client computer it will take lot of time and possibly people can mess up their installations. Updating lots of PCs does not sound like a good idea either.

Installing spam filter to a mail server sounds like a better idea but it might be that you cannot go to the mail server. Another problem would be that having a universal spam filter that works with any mail server is not reality.

But what about I you could just use hardware based solution that works out-of-the-box with any kind of setup. Wouldn’t that be cool? Perhaps the easiest way to eliminate spam in any POP3, IMAP & SMTP scenario would be to set up a network attached spam filter. This way would be 100% transparent to clients and completely platform independent. In SMTP scenario (You are not using mail server of your service provider, instead you have your own mail server) you can directly block spam before it goes to your mail server. You can also set up a simple shell script to automatically log in to your mail accounts and sweep the spam with Spamassassin. This would also be useful if you use a mobile device e.g. an iPhone to read your mail.

Untangle has made a great job by putting it all together and by offering the whole thing as an ISO-image. Their software is open source, licensed under GPLv2. All you are going to need is a extra computer with two network controllers.

To get started you should have following items:

• Untangle ISO image, and empty cd-r, obviously
• Extra computer with two nics, see hardware requirements

During the installation you might also need:

• a monitor
• an external CD drive

So, let’s get started with it.

I didn’t have extra computer for this so I ordered one. Here is my hardware configuration:

• Kingston Valueram 2GB 23,90€
• Intel Dual Core E5200 86,90€
• D-Link DGE-528T 14,70€
• Shuttle K45 barebone 111,90€
• 500GB HDD 56,90€
• Total 301,80€ with shipping.

This setup would be enough for pretty large network (maybe 200 clients). Hardware is not the case here, almost any x86 hardware should be just fine.

Unboxing and putting hardware together

Spam filter unboxing 1

Spam filter unboxing 2

Spam filter unboxing 3

Spam filter unboxing 4

Spam filter unboxing 5

Putting the hardware is an easy thing to do and even if you have not done it before I’m sure it won’t be a big problem. To me it took something like 15 minutes.

Since I think most of the readers might end up with a different hardware setup I won’t go thru putting the hardware together part any further. If you find any problems with the hardware, post a comment.

The installation process screenshots

Setting up BIOS to boot from USB-CD

Setting up bios continues

Saving BIOS and booting

Untangle booting

Entering to the installation wizard

Installation wizard 2

Selecting the destination drive

Testing hardware

Final warning before writing to the disk

Writing to the disk

The installation is ready, remove CD and boot it for the first time

Installing Untangle is quite straightforward process. My hardware setup did not have a CD-drive, keyboard, mouse or a monitor so I borrowed those from another computer. The mouse is essential! You cannot install Untangle without a mouse if you want to use the installer.

I think installation took something like half an hour for me, I didn’t time it. The installation wizard was easy to use and I sure most of the potential users with limited experience are fine with it.

My only complain would be the fact that in some step I needed a mouse to continue. I not sure if it was before or after the boot. Maybe text based installation option and SSH stating as a default would be a good idea for the future.

So as said, even thru Untangle pretty much works out of the box you should do some work more before you can remotely administer it.

Next couple of screens will guide you thru making basic settings and enabling the SSH server.

Configuration screenshots

Welcome to configuration wizard

Entering some basic information

Select a password

Select external and internal network interface

Setup network

Checking connectivity

As a router with NAT or completely transparent

Setup email

Congrats Untangle is now installed

Login to admin interface

By default your Untangle rack is empty

Launch a shell

Type ifconfig to see your network configuration

Enabling remote admin functionality

Starting SSH by typing /etc/init.d/ssh start

To make sure SSH starts next time you boot you have to type update-rc.d ssh defaults

Configuring Untangle is done with another wizard and it includes following steps:

- Agree with license, select password

- Select which network interface to use as external and which as internal

- Select router mode with NAT or transparent mode

Remote admin functionality is not on by default due the security reasons so in most cases you would like to put that on.

Putting SSH on is also a good idea and don’t forget to get SSH to the default runlevel. This is done by saying following to the terminal as a root:

/etc/init.d/ssh start
update-rc.d ssh defaults


Now that Untangle is installed and configured I would like to write few words about its capabilities.

Using Untangle

The beauty of Untangle is in its transparency. If you have only one computer I cannot see why you would like to install hardware based spam filter. But for network from couple to hundreds of clients PCs it is just great! After you get it up and running all the clients get the benefit of being spam free with no software installations and no new skills to learn. Sure this depends little bit of your configuration. It is going to be 100% transparent only if you will install it in front of the mail server you are using.

If you have a mail server outside of your network. You can use a small script called IMAP Spam Begone to sweep the spam on e.g. hourly basis. Installation is bit more difficult, I might write about it later on because it is generally useful.

The admin utility

Finally couple word about the admin utility. Untangle admin utility’s user interface looks like a server rack. Each component appears as a rack-mounted utility. I did not like the look of the UI for at first but it was simple to use.

Untangle’s spam filter is Spamassassin. The other components you can get to your rack are:

• Web Filter
• Spyware Blocker
• Protocol Control
• Virus Blocker
• Phish Blocker
• IPS
• Attack Blocker
• Firewall
• OpenVPN
• Reports Routing
• QoS

UPDATE: Statistics after about six weeks
It appears to be that after six weeks 121480 spam messages have been blocked and 1051 non-spam messages passed. Users are saying that they are virtually getting no spam at all. So I am pretty happy with it.

Untangle stats after six weeks of operation

Full disk is a lousy reason to wake up in a middle of the night

In my last post I wrote about Laconi.ca and mentioned that we have many bots updating Laconi.ca too. The bots are made active when some condition has met.

Example: Make server running out of disk space to post an update to Laconi.ca

This is actually quite useful. Sure, you can tell every service to send an email when something not wanted is happening but then you will have to read that mailbox too. Sending email to someone else doesn’t solve the problem. Hmm, updating to Laconi.ca doesn’t solve the problem either but it makes servers easier follow and to subscribe.

So here is a simple shell script:

#!/bin/sh
fs=`mount|egrep '^/dev'|grep -iv cdrom| awk '{print $3}'`
typeset -i thresh="96"
typeset -i warn="98"
for i in $fs
do
skip=0
typeset -i used=`df -k $i|tail -n 1|awk '{print $5}'|cut -d "%" -f 1`
if [ "$used" -ge "$warn" ]; then
laconica.sh "1.2.3.4 server CRITICAL: filesystem $i is $used% full"
fi
if [ "$used" -ge "$thresh" -a "$used" -le "$warn" ]; then
laconica.sh "1.2.3.4 server WARNING: filesystem $i is $used %full"
fi
done

I didn’t want to use cURL directly to talk to Laconi.ca so I downloaded laconica.sh from http://downloads.guillermoamaral.com/misc/laconica. Sure it uses cURL, but if I change a password no I can change it to this wrapper instead.

I saved the script as diskfull.sh. So lets first chmod it and then try it:

chmod 755 diskfull.sh
./diskfull.sh

Laconi.ca gets an update as follows:

Screenshot of Laconi.ca

You would need schedule Cron to run the script. To do so, edit the /etc/crontab and add the following line:

31 8 * * * root /opt/diskfull.sh

This line will run it daily on 08:31 as root. You might want to change it. If you are not familiar with Cron here is some help.

Twitter is a free social networking and micro-blogging service that allows its users to send and read other users’ updates (otherwise known as tweets), which are text-based posts of up to 140 characters in length.
Twitter page on Wikipedia

This post covers:

• Why I like Laconi.ca as our internal Micro-Blogging-service
• How to configure Laconi.ca behind password without breaking up the API functionality
• How to set up Twitterific to work with custom Laconi.ca installation

Laconi.ca in a browser & Twitterific

Ok, Twitter is sweet, even a bit addictive, but not that useful in internal use of your business for couple of reasons:

• Privacy concerns; I know you don’t have to tweet everything in public but still; You might not want to trust any third parties
• It is useful to have a possibility to customize your micro-blogging application, make backups and so on

At first, Micro-blogging is like wiki in a way that it does not implement any kind of process. In Twitter the idea is “it’s a tool, use it as you like” This works well with Twitter.com, where all the users have their own reasons to use the service. In business however, the business has a goal and people are working to meet this goal for their common reason. Therefore a process, as loose as it might be, must be implemented.

How and why it should be used

We got Laconi.ca running to our office a few weeks ago. In our case we have two offices with a distance of 600 km to each other. Some people in our staff are working from home and someone is always on the road. We use Laconi.ca to keep on track where people are on what are they working with.

So we have two rules on how to use it:

• When you start to work, tell what you are going to work with
• When you finish the day, tell when you are going to be back

Nothing too complicated just two easy rules. In my opinion it works just great. Before getting Laconi.ca I made far more phone calls to people asking for these same basic questions.

We also have many bots updating to Laconi.ca from our project management systems, internal blog and errors & warnings from servers. Generally I think you should not be worried about the amount of data because everyone doesn’t have to subscribe everything and everything does not have to be read by everyone. Still, having a possibility to subscribe a bot when needed is great.

How to install Laconi.ca on Ubuntu and get it behind a password

Laconi.ca installation guide can be found at http://laconi.ca/trac/wiki/Installation

Getting Laconi.ca behind password is actually the simplest thing:

1) Just create .htaccess and .htpasswd file. If don’t know how here is some help.

2) Create the same users with the same passwords as you have in your Laconi.ca installation.

How to make Twitterific to work with Laconi.ca

Get Twitterific

Open terminal and run:

defaults write com.iconfactory.Twitterrific baseUrl -string 'your-hostname/path-to-laconica/api'

The command to switch back is the following:

defaults write com.iconfactory.Twitterrific baseUrl -string 'twitter.com'

I found this tip from Leslie Michael Orchard’s blog, thank you Leslie!

You can get Ubuntu Cola from Sweden

One of my colleagues gave this drink to me on the other day so I had to post an image here.

Luckily all the single-paged documents had string “Page: 1/1″ (or same in Finnish) on the top of the page. So writing a small Python-script to do the job was easy. What I needed in addition to Python was pdftotext binary (here is a dmg for OS X) and pyPdf. So here is the code:

#!/usr/bin/python
'''This script will need pypdf module and pdftotext binary'''
import sys
import os
from pyPdf import PdfFileReader, PdfFileWriter

def findstr(lookup, filename):
    textfile = open(filename, 'rb')
    text = textfile.read()
    textfile.close()
    pos = -1
    while True:
        # move index up on next call
        pos = text.find(lookup, pos + 1)
        # not found or done
        if pos < 0:
            return False
        return True

try:
    searchstr = sys.argv[1]
    searchstr2 = sys.argv[2]
    pdffile = PdfFileReader(file(sys.argv[3], "rb"))
    numpages = pdffile.getNumPages()
    singlefile = sys.argv[4]
    multifile = sys.argv[5]
except IndexError:
    print "Usage: getmulti.py [searchstring1] [searchstring2] [sourcefile] [destinationfile-single] [destinationfile-single]"

print "****************"
print "Extracting multipaged and singlepaged files from " + sys.argv[3] + " (%s pages)" % numpages
print "Outputting multipaged to " + multifile + " and singlepaged to " + singlefile

singleoutput = PdfFileWriter()
multioutput = PdfFileWriter()
for i in xrange(numpages):
    os.system("pdftotext -f %s -l %s %s /tmp/foo.txt" % (i+1, i+1, sys.argv[3]))
    print "pdftotext -f %s -l %s %s /tmp/foo.txt" % (i+1, i+1, sys.argv[3])
    if findstr(searchstr, "/tmp/foo.txt") or findstr(searchstr2, "/tmp/foo.txt"):
        print "got it"
        singleoutput.addPage(pdffile.getPage(i))
    else:
        print "not got"
        multioutput.addPage(pdffile.getPage(i))

multioutputStream = file(multifile, "wb")
singleoutputStream = file(singlefile, "wb")

multioutput.write(multioutputStream)
singleoutput.write(singleoutputStream)
multioutputStream.close()
singleoutputStream.close()

So now I can just say:

./getmulti.py "Page: 1/ 1" "Sivu: 1/ 1" orig.pdf single.pdf multi.pdf