#!/usr/bin/env python
# coding: utf-8

import os
import re
import sys

def grab_email(file):
    """Try and grab all emails addresses found within a given file."""
    email_pattern = re.compile(r'\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b',
                               re.IGNORECASE)
    found = set()
    if os.path.isfile(file):
        for line in open(file, 'r'):
            found.update(email_pattern.findall(line))
    for email_address in found:
        print email_address

if __name__ == '__main__':
    grab_email(sys.argv[1])

The API is not very well documented but you can browse its capabilities be opening http://URL-TO-SUGAR/soap.php in your browser

At work we use it by ourselves to handle some processes and have been working with customers using it. There are things I like and thing I don’t like in SugarCRM. It is bit bloated and the code is not the most beautiful I have seen. On the other hand it does its job quite well and it is easy to configure without touching a line of code.

If you use a CRM in your business for a while you will most likely to find that you would like to do some integration. At work we have been doing various kinds of integration tasks. We have integrated a phone system to automatically make entries to SugarCRM when inbound or outbound calls have been made, connected registration of a web application to create a new accounts to CRM, connected a LIMS in various ways to it and so on. The reason I found SOAP-API to be the best way in most these cases is the fact that if you use it you won’t break down anything and it is likely that your code will work with future releases too. Off course there is a performance penalty and using SOAP API is not a good option when performance is an issue. So lets get to the business.

Installing SOAPpy
I assume that installing Python module with easy_install is not a problem. However, latest version of SOAPpy was not compatible with Python 2.5. To install SOAPpy I would suggest the following:

1. Download and extract SOAPpy http://sourceforge.net/project/showfiles.php?group_id=26590&package_id=18246

2. Move every “from __future__ import” lines to the beginning of the files where they are in. Below you can see the listing of these files.

localhost:SOAPpy-0.12.0 tuomas$ grep -R __future__ SOAPpy/*.py
SOAPpy/Client.py:from __future__ import nested_scopes
SOAPpy/GSIServer.py:from __future__ import nested_scopes
SOAPpy/NS.py:from __future__ import nested_scopes
SOAPpy/Server.py:from __future__ import nested_scopes
SOAPpy/Types.py:from __future__ import nested_scopes

3. Install SOAPpy

sudo python setup.py install

The simplest possible example of SugarCRM + SOAPpy
The code below logs in to the SugarCRM, asks the version of the server and prints it. Here is the code:

import md5
import SOAPpy
USERNAME= "user"
PASSWORD= "pass"

auth = {'user_name': USERNAME, 'password': md5.new(PASSWORD).hexdigest(), "version": "1.1"}

# Url of SugarCRM + soap.php?wsdl
SUGAR_URL = 'http://192.160.0.1/soap.php?wsdl'

sugar = SOAPpy.WSDL.Proxy(SUGAR_URL)
session = sugar.login(auth, "foobar")

try:
    response = sugar.get_server_version()
except StandardError, err:
    print '\nError in information retrieval from SugarCRM:' + str(err)
print response

The API is not documented very well but you can list all of its capabilities by browsing yourself to http://URL-TO-SUGAR/soap.php.

Advanced example
This script will log into SugarCRM and create a new Account.

import md5
import SOAPpy
USERNAME= "user"
PASSWORD= "pass"

auth = {'user_name': USERNAME, 'password': md5.new(PASSWORD).hexdigest(), "version": "1.1"}

# Url of SugarCRM + soap.php?wsdl
SUGAR_URL = 'http://192.160.0.1/soap.php?wsdl'

sugar = SOAPpy.WSDL.Proxy(SUGAR_URL)
session = sugar.login(auth, "foobar")
module = "Accounts"

adata = [{'name': 'name', 'value': "Test Account"},
           {'name': 'shipping_address_street', 'value': "Address test"}]
try:
    response = sugar.set_entry(session['id'], module, adata)
except StandardError, err:
     print '\nError in information retrieval from SugarCRM:' + str(err)
print response

UPDATE: It appears to be that, Sugar Team is working with the API and the documentation at the moment and I’ll take back what I said about missing documentation. Thanks SugarCRM devs!

Full disk is a lousy reason to wake up in a middle of the night

In my last post I wrote about Laconi.ca and mentioned that we have many bots updating Laconi.ca too. The bots are made active when some condition has met.

Example: Make server running out of disk space to post an update to Laconi.ca

This is actually quite useful. Sure, you can tell every service to send an email when something not wanted is happening but then you will have to read that mailbox too. Sending email to someone else doesn’t solve the problem. Hmm, updating to Laconi.ca doesn’t solve the problem either but it makes servers easier follow and to subscribe.

So here is a simple shell script:

#!/bin/sh
fs=`mount|egrep '^/dev'|grep -iv cdrom| awk '{print $3}'`
typeset -i thresh="96"
typeset -i warn="98"
for i in $fs
do
skip=0
typeset -i used=`df -k $i|tail -n 1|awk '{print $5}'|cut -d "%" -f 1`
if [ "$used" -ge "$warn" ]; then
laconica.sh "1.2.3.4 server CRITICAL: filesystem $i is $used% full"
fi
if [ "$used" -ge "$thresh" -a "$used" -le "$warn" ]; then
laconica.sh "1.2.3.4 server WARNING: filesystem $i is $used %full"
fi
done

I didn’t want to use cURL directly to talk to Laconi.ca so I downloaded laconica.sh from http://downloads.guillermoamaral.com/misc/laconica. Sure it uses cURL, but if I change a password no I can change it to this wrapper instead.

I saved the script as diskfull.sh. So lets first chmod it and then try it:

chmod 755 diskfull.sh
./diskfull.sh

Laconi.ca gets an update as follows:

Screenshot of Laconi.ca

You would need schedule Cron to run the script. To do so, edit the /etc/crontab and add the following line:

31 8 * * * root /opt/diskfull.sh

This line will run it daily on 08:31 as root. You might want to change it. If you are not familiar with Cron here is some help.

Luckily all the single-paged documents had string “Page: 1/1″ (or same in Finnish) on the top of the page. So writing a small Python-script to do the job was easy. What I needed in addition to Python was pdftotext binary (here is a dmg for OS X) and pyPdf. So here is the code:

#!/usr/bin/python
'''This script will need pypdf module and pdftotext binary'''
import sys
import os
from pyPdf import PdfFileReader, PdfFileWriter

def findstr(lookup, filename):
    textfile = open(filename, 'rb')
    text = textfile.read()
    textfile.close()
    pos = -1
    while True:
        # move index up on next call
        pos = text.find(lookup, pos + 1)
        # not found or done
        if pos < 0:
            return False
        return True

try:
    searchstr = sys.argv[1]
    searchstr2 = sys.argv[2]
    pdffile = PdfFileReader(file(sys.argv[3], "rb"))
    numpages = pdffile.getNumPages()
    singlefile = sys.argv[4]
    multifile = sys.argv[5]
except IndexError:
    print "Usage: getmulti.py [searchstring1] [searchstring2] [sourcefile] [destinationfile-single] [destinationfile-single]"

print "****************"
print "Extracting multipaged and singlepaged files from " + sys.argv[3] + " (%s pages)" % numpages
print "Outputting multipaged to " + multifile + " and singlepaged to " + singlefile

singleoutput = PdfFileWriter()
multioutput = PdfFileWriter()
for i in xrange(numpages):
    os.system("pdftotext -f %s -l %s %s /tmp/foo.txt" % (i+1, i+1, sys.argv[3]))
    print "pdftotext -f %s -l %s %s /tmp/foo.txt" % (i+1, i+1, sys.argv[3])
    if findstr(searchstr, "/tmp/foo.txt") or findstr(searchstr2, "/tmp/foo.txt"):
        print "got it"
        singleoutput.addPage(pdffile.getPage(i))
    else:
        print "not got"
        multioutput.addPage(pdffile.getPage(i))

multioutputStream = file(multifile, "wb")
singleoutputStream = file(singlefile, "wb")

multioutput.write(multioutputStream)
singleoutput.write(singleoutputStream)
multioutputStream.close()
singleoutputStream.close()

So now I can just say:

./getmulti.py "Page: 1/ 1" "Sivu: 1/ 1" orig.pdf single.pdf multi.pdf