Python

Fix for firehol get-iana script

I have talked before about using firehol to configure iptables. I won’t go into all the details about how wonderful and awesome it is, but trust me, it makes configuring iptables a snap.

Firehol includes a script, get-iana.sh, that downloads the IPv4 address space list from IANA and populates a file called RESERVED_IPS that firehol uses when configuring iptables. Basically, any traffic from outside coming from any reserved or unallocated IP block is dropped automatically. As you can imagine, keeping this file updated regularly is important, as previously unallocated blocks are allocated for use. To this end, whenever firehol starts it checks the age of the RESERVED_IPS file and if it is older than 90 days warns you to update it by running the supplied get-iana.sh.

However, there has been a change recently in how the IANA reserved IPv4 address space file is formatted. There are lots of posts on plenty of forums with patches for get-iana.sh to accept and use the new format plain text file (while the default is now XML rather than plain text) and needless to say I tried every single one I could find. None of them worked, so what to do? How about a complete rewrite in Python? And while we’re at it, let’s use the XML format that IANA wants everyone to use.

So, one lunch hour of hacking and here it is, working like a charm. You can copy this, but I recommend downloading it to avoid whitespace issues.

#!/usr/bin/python

"""
file: get-iana.py

Replacement for get-iana.sh that ships with firehol and no longer seems to work.
This is less code, less confusing, uses the preferred XML format from IANA and works.

Copyright (c) 2010 Sjan Evardsson

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""

import urllib
import xml.dom.minidom
import os
urllib.urlretrieve('http://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml','address-space.xml')
results = []
x = xml.dom.minidom.parse('address-space.xml')
for i in x.childNodes:
    if i.localName == 'registry':
        for j in i.childNodes:
            if j.localName == 'record':
                for k in j.childNodes:
                    if k.localName == 'prefix':
                        prefix = k.firstChild.data
                    if k.localName == 'status':
                        status = k.firstChild.data
                if status == 'RESERVED' or status == 'UNALLOCATED':
                    results.append(prefix)
outfile = open('iana-temp','w')
for r in results:
    hi = int(r.split('/')[0])
    outfile.write(str(hi)+'.0.0.0/8\n')
outfile.close()
os.remove('address-space.xml')
os.rename('/etc/firehol/RESERVED_IPS','/etc/firehol/RESERVED_IPS.old')
os.rename('iana-temp','/etc/firehol/RESERVED_IPS')

18 comments Fix for firehol get-iana script

bernie says:

fails if the file doesnt already exist (i was setting up firehol for the first time). I am in the process of replacing Monmotha… as it is no longer developed. This looked to offer the same flexibility with simplicity, but now I am scared it too is going the way of the donkey.

Ah, yes, I guess it would fail, at that. I hadn’t thought about it from the perspective of someone installing it fresh. In that case a touch /etc/firehol/RESERVED_IPS would be required before running the script.

Steffen says:

Works like a charm, thanks.

Steffen says:

I am using this now for the third time. Maybe one of these long evenings you might want to add that mentioned “touch” argument to the script.

And maybe … I really mean maybe … you could do it like the original and have a DIFF run through the old and new and let the user decide wether or not to take the change.

Just thinking in terms of furute, what if the format changes again … someone being lazy will not notice that the file is empty or screwed.

But after all that, thanks again.

Haram says:

Thanks for this, been looking for hours for a working get-iana.

John Brandwood says:

Thanks so much for this, I couldn’t believe that Ubuntu 10.04LTS still ships with a broken get-iana script.

Anyway, while I like your python version, I decided to fix the original script instead.

Apart from fixing the URL, the only change needed is to replace the “cut” command with a slightly more sophisticated “sed” command to stip off the leading white-space from an IP address space line …

#!/bin/bash

# $Id: get-iana.sh,v 1.13 2010/09/12 13:55:00 jcb Exp $
#
# $Log: get-iana.sh,v $
# Revision 1.13 2010/09/12 13:55:00 jcb
# Updated for latest IANA reservations format.
#
# Revision 1.12 2008/03/17 22:08:43 ktsaou
# Updated for latest IANA reservations format.
#
# Revision 1.11 2007/06/13 14:40:04 ktsaou
# *** empty log message ***
#
# Revision 1.10 2007/05/05 23:38:31 ktsaou
# Added support for external definitions of:
#
# RESERVED_IPS
# PRIVATE_IPS
# MULTICAST_IPS
# UNROUTABLE_IPS
#
# in files under the same name in /etc/firehol/.
# Only RESERVED_IPS is mandatory (firehol will complain if it is not there,
# but it will still work without it), and is also the only file that firehol
# checks how old is it. If it is 90+ days old, firehol will complain again.
#
# Changed the supplied get-iana.sh script to generate the RESERVED_IPS file.
# FireHOL also instructs the user to use this script if the file is missing
# or is too old.
#
# Revision 1.9 2007/04/29 19:34:11 ktsaou
# *** empty log message ***
#
# Revision 1.8 2005/06/02 15:48:52 ktsaou
# Allowed 127.0.0.1 to be in RESERVED_IPS
#
# Revision 1.7 2005/05/08 23:27:23 ktsaou
# Updated RESERVED_IPS to current IANA reservations.
#
# Revision 1.6 2004/01/10 18:44:39 ktsaou
# Further optimized and reduced PRIVATE_IPS using:
# http://www.vergenet.net/linux/aggregate/
#
# The supplied get-iana.sh uses ‘aggregate’ if it finds it in the path.
# (aggregate is the name of this program when installed on Gentoo)
#
# Revision 1.5 2003/08/23 23:26:50 ktsaou
# Bug #793889:
# Change #!/bin/sh to #!/bin/bash to allow FireHOL run on systems that
# bash is not linked to /bin/sh.
#
# Revision 1.4 2002/10/27 12:44:42 ktsaou
# CVS test
#

#
# Program that downloads the IPv4 address space allocation by IANA
# and creates a list with all reserved address spaces.
#

IPV4_ADDRESS_SPACE_URL=”http://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.txt”

# The program will match all rows in the file which start with a number, have a slash,
# followed by another number, for which the following pattern will also match on the
# same rows
IANA_RESERVED=”(RESERVED|UNALLOCATED)”

# which rows that are matched by the above, to ignore
# (i.e. not include them in RESERVED_IPS)?
#IANA_IGNORE=”(Multicast|Private use|Loopback|Local Identification)”
IANA_IGNORE=”Multicast”

tempfile=”/tmp/iana.$$.$RANDOM”

AGGREGATE=”`which aggregate 2>/dev/null`”
if [ -z “${AGGREGATE}” ]
then
AGGREGATE=”`which aggregate 2>/dev/null`”
fi

if [ -z “${AGGREGATE}” ]
then
echo >&2
echo >&2
echo >&2 “WARNING”
echo >&2 “Please install ‘aggregate’ to shrink the list of IPs.”
echo >&2
echo >&2
fi

echo >&2
echo >&2 “Fetching IANA IPv4 Address Space, from:”
echo >&2 “${IPV4_ADDRESS_SPACE_URL}”
echo >&2

wget -O – –proxy=off “${IPV4_ADDRESS_SPACE_URL}” |\
egrep ” *[0-9]+/[0-9]+.*${IANA_RESERVED}” |\
egrep -vi “${IANA_IGNORE}” |\
sed -e ‘s:^ *\([0-9]*/[0-9]*\).*:\1:’ |\
(

while IFS=”/” read range net
do
if [ ! $net -eq 8 ]
then
echo >&2 “Cannot handle network masks of $net bits ($range/$net)”
continue
fi

first=`echo $range | cut -d ‘-‘ -f 1`
first=`expr $first + 0`
last=`echo $range | cut -d ‘-‘ -f 2`
last=`expr $last + 0`

x=$first
while [ ! $x -gt $last ]
do
# test $x -ne 127 && echo “$x.0.0.0/$net”
echo “$x.0.0.0/$net”
x=$[x + 1]
done
done
) | \
(
if [ ! -z “${AGGREGATE}” -a -x “${AGGREGATE}” ]
then
“${AGGREGATE}”
else
cat
fi
) >”${tempfile}”

echo >&2
echo >&2
echo >&2 “FOUND THE FOLLOWING RESERVED IP RANGES:”
printf “RESERVED_IPS=\””
i=0
for x in `cat ${tempfile}`
do
i=$[i + 1]
printf “${x} ”
done
printf “\”\n”

if [ $i -eq 0 ]
then
echo >&2
echo >&2
echo >&2 “Failed to find reserved IPs.”
echo >&2 “Possibly the file format has been changed, or I cannot fetch the URL.”
echo >&2

rm -f ${tempfile}
exit 1
fi
echo >&2
echo >&2
echo >&2 “Differences between the fetched list and the list installed in”
echo >&2 “/etc/firehol/RESERVED_IPS:”

echo >&2 “# diff /etc/firehol/RESERVED_IPS ${tempfile}”
diff /etc/firehol/RESERVED_IPS ${tempfile}

if [ $? -eq 0 ]
then
echo >&2
echo >&2 “No differences found.”
echo >&2

rm -f ${tempfile}
exit 0
fi

echo >&2
echo >&2
echo >&2 “Would you like to save this list to /etc/firehol/RESERVED_IPS”
echo >&2 “so that FireHOL will automatically use it from now on?”
echo >&2
while [ 1 = 1 ]
do
printf >&2 “yes or no > ”
read x

case “${x}” in
yes) cp -f /etc/firehol/RESERVED_IPS /etc/firehol/RESERVED_IPS.old 2>/dev/null
cat “${tempfile}” >/etc/firehol/RESERVED_IPS || exit 1
echo >&2 “New RESERVED_IPS written to ‘/etc/firehol/RESERVED_IPS’.”
break
;;

no)
echo >&2 “Saved nothing.”
break
;;

*) echo >&2 “Cannot understand ‘${x}’.”
;;
esac
done

rm -f ${tempfile}

Rocky says:

Hey,

You can setup a monthly cronjob by doing the following:

Create and chmod +x /usr/sbin/update-iana with:

#!/bin/sh
/usr/sbin/get-iana < /etc/firehol/get-iana-answerfile

Create /etc/firehol/get-iana-answerfile (This file automatically answers yes when asked if you’d like to save RESERVED_IPS) with the following content:

yes

Edit the following section of John Brandwood’s script to look like:

#!/bin/sh
/usr/sbin/get-iana /dev/null
cat “${tempfile}” >/etc/firehol/RESERVED_IPS || exit 1
echo >&2 “New RESERVED_IPS written to ‘/etc/firehol/RESERVED_IPS’.”
echo “Firehol will now be restarted”
sleep 3
/etc/init.d/firehol restart
break
;;

Add it to cron:
@monthly /usr/sbin/update-iana #Update Firehol once a month

I actually just dropped the following in /etc/cron.monthly and called it get-iana:

#!/bin/bash
/usr/sbin/get-iana.py && /etc/init.d/firehol restart

Sjan, Thanks for sharing this.
I was about to write something similiar, googled and found yours.

[…] http://www.evardsson.com/blog/2010/06/28/fix-for-firehol-get-iana-script/ This entry was posted in Uncategorized. Bookmark the permalink. ← Coding a CSS3 & HTML5 One-Page Website Template LikeBe the first to like this post. […]

VS says:

Thanks a lot for this…It works perfectly!

errors and errors

Using Ubuntu 10.10;
$./get-iana.py
Traceback (most recent call last):
File “./get-iana.py”, line 53, in
os.rename(‘/etc/firehol/RESERVED_IPS’,’/etc/firehol/RESERVED_IPS.old’)
OSError: [Errno 2] No such file or directory
my@localhost:~/$ touch RESERVED_IPS
my@localhost:~/$ sudo ./get-iana.py
Traceback (most recent call last):
File “./get-iana.py”, line 54, in
os.rename(‘iana-temp’,’/etc/firehol/RESERVED_IPS’)
OSError: [Errno 18] Invalid cross-device link

If you do not already have the file extant (as it seems you don’t), you will need to do a:
sudo touch /etc/firehol/RESERVED_IPS

Then run the script.

Richard says:

There is a difference with the results generated with the Python script compared to the bash one. This seems to be due to the fact that the bash script ignores ‘multicast’ entries but the Python script doesn’t. Could this lead to misleading or incorrect results in Firehol?

That is a very good question. It is not something I had noticed, so I hadn’t really thought about it. It might cause problems if your border router is inside the firewall, since the only thing that should be coming from multicast addresses is routing protocols, etc. (See http://www.iana.org/assignments/multicast-addresses/multicast-addresses.xml)

As I tend to use Firehol to control IPTables as a host-based firewall, it doesn’t actually have any effect on my usage, although if you want to open up to communicate with the multicast space you may want to edit the script to taste.

Andrew says:

Adding this to the Python script removes the multicast entries. With this addition the output is the same as the stock get-iana script except the IP’s are not agregated meaning you may end up with several /8 networks instead of a /4. For example:

240.0.0.0/8

255.0.0.0/8

versus

240.0.0.0/4

Here are the changes:

                    if k.localName == 'status':
                        status = k.firstChild.data
                    if k.localName == 'designation':
                        designation = k.firstChild.data
                if status == 'RESERVED' or status == 'UNALLOCATED':
                        if designation != 'Multicast':
                                results.append(prefix)

Thanks, Andrew. That should help those who wish to communicate in the multicast space.

In terms of aggregation I have not seen any noticeable difference in latency using the aggregate values (/4s) versus the non-aggregate (/8s). I would imagine that it makes the RESERVED_IPS file shorter, which may speed up startup times and possibly reduce iptables memory usage slightly.

That might be something to work out and add to the script as well …

Alex says:

Thank you!

Comments are closed.