Tag: Python

Python

Fix for firehol get-iana script

I have talked before about using firehol to configure iptables. I won’t go into all the details about how wonderful and awesome it is, but trust me, it makes configuring iptables a snap.

Firehol includes a script, get-iana.sh, that downloads the IPv4 address space list from IANA and populates a file called RESERVED_IPS that firehol uses when configuring iptables. Basically, any traffic from outside coming from any reserved or unallocated IP block is dropped automatically. As you can imagine, keeping this file updated regularly is important, as previously unallocated blocks are allocated for use. To this end, whenever firehol starts it checks the age of the RESERVED_IPS file and if it is older than 90 days warns you to update it by running the supplied get-iana.sh.

However, there has been a change recently in how the IANA reserved IPv4 address space file is formatted. There are lots of posts on plenty of forums with patches for get-iana.sh to accept and use the new format plain text file (while the default is now XML rather than plain text) and needless to say I tried every single one I could find. None of them worked, so what to do? How about a complete rewrite in Python? And while we’re at it, let’s use the XML format that IANA wants everyone to use.

So, one lunch hour of hacking and here it is, working like a charm. You can copy this, but I recommend downloading it to avoid whitespace issues.

#!/usr/bin/python

"""
file: get-iana.py

Replacement for get-iana.sh that ships with firehol and no longer seems to work.
This is less code, less confusing, uses the preferred XML format from IANA and works.

Copyright (c) 2010 Sjan Evardsson

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""

import urllib
import xml.dom.minidom
import os
urllib.urlretrieve('http://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml','address-space.xml')
results = []
x = xml.dom.minidom.parse('address-space.xml')
for i in x.childNodes:
    if i.localName == 'registry':
        for j in i.childNodes:
            if j.localName == 'record':
                for k in j.childNodes:
                    if k.localName == 'prefix':
                        prefix = k.firstChild.data
                    if k.localName == 'status':
                        status = k.firstChild.data
                if status == 'RESERVED' or status == 'UNALLOCATED':
                    results.append(prefix)
outfile = open('iana-temp','w')
for r in results:
    hi = int(r.split('/')[0])
    outfile.write(str(hi)+'.0.0.0/8\n')
outfile.close()
os.remove('address-space.xml')
os.rename('/etc/firehol/RESERVED_IPS','/etc/firehol/RESERVED_IPS.old')
os.rename('iana-temp','/etc/firehol/RESERVED_IPS')
PHP

SPDO moving to SourceForge (I hope)

While school has kept me busy during my non-work hours I have not had much time to tend to other projects like SPDO. Seeing how my “Abandoned Project Takeover” has been approved on SourceForge, however, has encouraged me to set aside some time to do some house cleaning on the code to get the 1.0b ready to bump to 1.0 stable. According the information they have given me it takes 2 to 3 weeks to fully process an “APT” so I do have a little time (although not a lot). This is not the first time I have tried to do this, but last time it never made it to the approval stage and languished in the queue until it timed out. So, now that it was approved and has been in the queue for 4 business days I am hopeful.

In a lot of ways, moving the project to SourceForge is a lot like making a physical change of residence. It will require updating links, setting up the pages there including wiki, bug tracking, etc, and moving the actual code base over. (Good thing it’s small!)

As always, any suggestions, bugs, or feature requests are welcome, and (for now) should be posted at the page set aside for just that.

Database

SPDO 1.0b Release Candidate 1

SPDO version 1.0 beta Release Candidate 1 is out the door. This version adds the following:

Three custom Exception classes: SPDOError, ConnectionFailedError and ForeignKeyViolationError. Which are used (in order) when the program encounters an unknown sql error, a failure to connect to the database, and a violation of a foreign key constraint. Which means, of course, that foreign key constraints are now part of SPDO.

How this was accomplished is through the use of the new DbBuilder class. The DbBuilder class takes a multi-dimensional dict (Python) or array (PHP) of table and field data to build the tables and create triggers in MySQL and SQLite and add foreign keys in PostgreSQL. As an example consider the following:

In Python:

structure = {
    'authors':{
        'id':['primary_auto'],
        'first_name':['varchar(50)'],
        'last_name':['varchar(50)', 'NONULL', 'INDEX']
        },
    'books':{
        'id':['primary_auto'],
        'author_id':['int', "FK_CASCADE('authors','id')", 'INDEX'],
        'title':['varchar(100)','NONULL', 'INDEX']
        }
    }
dbb = DbBuilder(structure)
dbb.create()

In PHP:

$structure = array(
        'authors'=>array(
        'id'=>array('primary_auto'),
        'first_name'=>array('varchar(50)'),
        'last_name'=>array('varchar(50)', 'NONULL', 'INDEX')
    ),
    'books'=>array(
        'id'=>array('primary_auto'),
        'author_id'=>array('int', "FK_CASCADE('authors','id')", 'INDEX'),
        'title'=>array('varchar(100)','NONULL', 'INDEX')
    )
);
$dbb = DbBuilder($structure)
$dbb->create()

This results in the following queries to be executed:

In PostgreSQL:

CREATE TABLE authors (
    first_name VARCHAR(50) ,
    last_name VARCHAR(50) NOT NULL,
    id SERIAL PRIMARY KEY 
 );
CREATE TABLE books (
    author_id INTEGER ,
    id SERIAL PRIMARY KEY ,
    title VARCHAR(100) NOT NULL
);
CREATE INDEX authors_last_name_idx ON authors(last_name_id);
CREATE INDEX books_author_id_idx ON books(author_id);
CREATE INDEX books_title_idx ON books(title);
ALTER TABLE books ADD FOREIGN KEY (author_id) REFERENCES authors(id) ON DELETE CASCADE;

In MySQL:

CREATE TABLE authors (
    id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
    first_name VARCHAR(50),
    last_name VARCHAR(50) NOT NULL
);
CREATE TABLE books (
    id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY,
    author_id INTEGER NOT NULL,
    title VARCHAR(100) NOT NULL
);
CREATE TABLE fk_error_msg (
    error_msg VARCHAR(100) NOT NULL PRIMARY KEY
);
INSERT INTO fk_error_msg VALUES ('Foreign Key Constraint Violated!');
ALTER TABLE authors ADD INDEX(last_name);
ALTER TABLE books ADD INDEX(author_id);
ALTER TABLE books ADD INDEX(title);
CREATE TRIGGER fki_books_author_id
    BEFORE INSERT ON books
    FOR EACH ROW BEGIN
        IF
            0 = (SELECT COUNT(*) FROM authors WHERE id=new.author_id)
        THEN
            INSERT fk_error_msg VALUES ('Foreign Key Constraint Violated!');
        END IF;
    END;
CREATE TRIGGER fku_books_author_id
    BEFORE UPDATE ON books
    FOR EACH ROW BEGIN 
        IF
            0 = ( SELECT COUNT(*) FROM authors WHERE id = new.author_id )
        THEN
            INSERT INTO fk_error_msg VALUES ('Foreign Key Constraint Violated!');
        END IF ;
    END;
CREATE TRIGGER fkdc_books_author_id
    BEFORE DELETE ON authors
    FOR EACH ROW BEGIN
        DELETE FROM books WHERE author_id=old.id;
    END;

In SQLite:

CREATE TABLE authors (
    id INTEGER NOT NULL PRIMARY KEY,
    first_name VARCHAR(50),
    last_name VARCHAR(50) NOT NULL
);
CREATE TABLE books (
    id INTEGER NOT NULL PRIMARY KEY,
    author_id INTEGER NOT NULL,
    title VARCHAR(100) NOT NULL
);
CREATE INDEX authors_last_name_idx ON authors(last_name);
CREATE INDEX books_author_id_idx ON books(author_id);
CREATE INDEX books_title_idx ON books(title);
CREATE TRIGGER fki_books_author_id
    BEFORE INSERT ON [books]
        FOR EACH ROW BEGIN
            SELECT RAISE(ROLLBACK, 'insert on table "books" violates foreign key constraint "fki_books_author_id"')
            WHERE NEW.author_id IS NOT NULL AND (SELECT id FROM authors WHERE id = NEW.author_id) IS NULL;
        END;
CREATE TRIGGER fku_books_author_id
    BEFORE UPDATE ON [books]
        FOR EACH ROW BEGIN
            SELECT RAISE(ROLLBACK, 'update on table "books" violates foreign key constraint "fku_books_author_id"')
            WHERE NEW.author_id IS NOT NULL AND (SELECT id FROM authors WHERE id = NEW.author_id) IS NULL;
        END;
CREATE TRIGGER fkdc_books_author_id
    BEFORE DELETE ON authors
        FOR EACH ROW BEGIN
            DELETE FROM books WHERE books.author_id = OLD.id;
        END;

Be sure to check out the wiki, and the source code is all readable there, or download and enjoy. There are still a few edits that need to happen in the comments (like versions need to be updated, and changelogs added and there are a few typos probably still lurking about.)

Development

SPDO Wiki and Updates

I updated SPDO some today – I added in MySQL support for the Python version, and changed the names of the objects and files to match the new moniker. Speaking of which, I have decided on “Simplified (Python|PHP) Database Object” – so yeah, SPDO is the official name now. (Of course I keep seeing it “speedo”).

I also started a Wiki which will be updated with documentation, samples, etc as I make the time. I am hoping to get a bunch done tomorrow afternoon and this weekend.

The version 0.9 files are spdo-py.tar.bz2 and spdo-php.tar.bz2

Development

New Project: SPDO

I have just posted the (embarrassingly empty) page for my new pet project: SPDO (Sjan’s PDO) – in two flavors: PHP and Python.

There are only about a thousand PDOs out there, and perhaps a dozen or so of them are really functional and (in a few cases) very polished pieces of work. So why am I messing with writing my own? A couple of reasons:

  1. I like to have coding signatures that are consistent, whether I am working with PostgreSQL, MySQL or (heaven help us all) SQLite.
  2. I like to have coding signatures that are (reasonably) consistent across different languages – in this case Python and PHP.
  3. I wanted to take advantage of Prepared Statements where I could, even though the PHP implementations of those are pretty weak (especially in the case of MySQL).

Currently implemented in

  • Python:
    • PostgreSQL (with prepared statements)
    • SQLite (no prepared statements).
  • PHP
    • PostgreSQL (with prepared statements)
    • MySQL (with prepared statements)
    • SQLite (no prepared statements)

Here’s an example of how they work (in most simplistic terms):

#in Python
from pyDB import *
db = pyDB('mysite')
newid = db.insert('INSERT INTO test (name, value) VALUES (?,?)',['foo','bar'])
update_count = db.update('UPDATE test SET value=? WHERE id=?',['baz',newid])
results = db.select('SELECT * FROM test')
for row in results:
    for i in row:
        print "\t", i,"\t", row[i]
delete_count = db.delete('DELETE FROM test WHERE id=?',[newid])
//in PHP
require_once('phpdb.php');
$db = new phpDB('test');
$newid = $db->insert('INSERT INTO test (name, value) VALUES (?,?)',array('foo','bar'));
$update_count = $db->update('UPDATE test SET value=? WHERE id=?',array('baz',newid));
$results = db->select('SELECT * FROM test');
foreach($results as $row)
{
    foreach ($row as $key=>$val)
    {
        print "\t$key\t$val";
    }
}
$delete_count = $db->delete('DELETE FROM test WHERE id=?',array($newid));

The page with links to the code is in the list up top, and everything is MIT license. Enjoy.

Development

Learning Python

I was asked today “How do I learn python?” (I’m not making this up, that is a direct quote lifted right out of the chat log!)

Of course my first response was “start at python.org” (as it would be.) That is all well and good, but I started thinking about what specific things helped me? So, here are some specifics that might work well for you if, like me, you learn about half and half from reading (theory) and doing (practice.)

The first step, of course, is to download Python. You don’t need a fancy IDE to develop in Python, although if you have one you are comfortable with and use all the time, chances are that it has a Python plugin. (I know Eclipse does, and I believe NetBeans does, and jEdit has a Python client plugin that lets you run Python code right in the editor.) If you don’t want to muck about with an IDE, though, Python comes with it’s own lightweight editor, IDLE. Open it up, play around a bit.

Ok, so you’re looking at IDLE thinking “well, there isn’t much here is there?” You’re right – time for step two. Head over to the beginner’s guide and go for it! There are tutorials aimed at people who are new to Python and to programming in general, and people who are new to Python but not to programming. There is even a list of Python courses you can buy, and a fairly large list of introductory Python books for those who learn best that way. There isn’t much I can say that isn’t there already. I kind of wish that I had started there myself. When I started playing with Python I skipped over this and went straight for the documentation and googling for specific answers, as I was trying to build a script that I needed at the time to parse 1 -2 GB log files. Java was too much tool for the job, I didn’t have access to a beefy enough Linux box to do it in Bash and PHP was doing nothing but running out of memory. It made for a bit of a rough entry into Python, but I kind of like that sort of challenge.

Which brings us to step three – build something. No, not a “Hello World” app, but something useful, in the immediate sense. Something that will make you smile, or make your life simpler, or even something that will allow you to tick off an item on your to-do list.

So what (besides the end of my rambling in parenthetical asides) are you waiting for?

IDE

Tabbed Python editing with PythonCard

While I have been partial to IDLE for developing Python I recently came across on SourceForge while looking for some help building a Python GUI. While PythonCard has plenty of tools and examples to help with GUI construction, it was the Tabbed Code Editor that really got my attention.

One thing that I look for in any development environment is the ability to open mutiple files in tabs. While IDLE is extremely useable, it lacks this feature. And yes, there are other tabbed IDE’s that can parse Python syntax, but none of them seem to do as good of a job as IDLE.

While I can’t claim that the PythonCard tabbed editor is as good IDLE in terms of hinting and code completion, (at least as far as Python builtins) it does a rather decent job of that for objects/scripts that are imported.

While I may or may not end up actually building Python GUIs with it, PythonCard has definitely found a permanent place in my toolbox.

Python

Another real-world turbogears example

Another real-world example of turbogears was pointed out by one my coworkers (thanks Brad!) and I had to check it out.

The Python devcenter blog on O’Reilly has an article on replacing Meetup.com with a turbogears app .

While I haven’t done any more at this point than quickly browse the source files and check it out in action, it seems like it is a good example piece. (I tend to better understand how things work by examining a few different examples.)

The app was developed by Rick Copeland of the The Python Atlanta “Meetup” group where you can see it in action.

Python

Minor revision to breadcrumbs.py

I made a slight change to the I wrote as a plugin for . I changed the way the links are displayed, so that the directory seperators (/) are not included in the actual link, and added a rel=”tag” to the links to allow style aggregators understand them better.

As always, you are free to download it from this site and I am open to any hints, suggestions, fixes, updates, or constructive criticism of any sort.