Massive Fail!

August 17th, 2009 by sjan

While looking for the source of database backup errors I found it, the hard way. I have been running my database on a separate hard disk from everything else, and have been getting occassional errors from the cron script that does a nightly dump. I was under the impression that this was due to latency causing the script to time out. Not the case.

Looking at the dump it seemed like everything was being written ok, at least it looked that way yesterday. (I have not had much time for site maintenance, so this has been in the “put it off until later” pile.) While looking at the script this afternoon, and trying another run it timed out again (or so I thought). I figured I could put it off until this weekend, until I went to look at the site and got the big “Unable to contact the database” error. I went the server and fired up mysql on the command line and discovered that there were NO DATABASES! I tried to get a file listing of the /var/lib/mysql directory and got nothing. Nada. Since I don’t seem to be able to get anything off of that disc I did a quick modification of the fstab (to remove the line mounting that drive), rebuilt the dbs from the last (failed) backup, and here I am, missing two months worth of data.

Can I cry now?

In case you are wondering, the Margin vs Markup page is still available (as I made it a regular page as well as a post, it was my most popular ever).

EDIT: Ah the joys of using decade-old equipment.

New Project: SPDO

January 28th, 2009 by sjan

I have just posted the (embarrassingly empty) page for my new pet project: SPDO (Sjan’s PDO) – in two flavors: PHP and Python.

There are only about a thousand PDOs out there, and perhaps a dozen or so of them are really functional and (in a few cases) very polished pieces of work. So why am I messing with writing my own? A couple of reasons:

  1. I like to have coding signatures that are consistent, whether I am working with PostgreSQL, MySQL or (heaven help us all) SQLite.
  2. I like to have coding signatures that are (reasonably) consistent across different languages – in this case Python and PHP.
  3. I wanted to take advantage of Prepared Statements where I could, even though the PHP implementations of those are pretty weak (especially in the case of MySQL).

Currently implemented in

  • Python:
    • PostgreSQL (with prepared statements)
    • SQLite (no prepared statements).
  • PHP
    • PostgreSQL (with prepared statements)
    • MySQL (with prepared statements)
    • SQLite (no prepared statements)

Here’s an example of how they work (in most simplistic terms):

#in Python
from pyDB import *
db = pyDB('mysite')
newid = db.insert('INSERT INTO test (name, value) VALUES (?,?)',['foo','bar'])
update_count = db.update('UPDATE test SET value=? WHERE id=?',['baz',newid])
results ='SELECT * FROM test')
for row in results:
    for i in row:
        print "\t", i,"\t", row[i]
delete_count = db.delete('DELETE FROM test WHERE id=?',[newid])
//in PHP
$db = new phpDB('test');
$newid = $db->insert('INSERT INTO test (name, value) VALUES (?,?)',array('foo','bar'));
$update_count = $db->update('UPDATE test SET value=? WHERE id=?',array('baz',newid));
$results = db->select('SELECT * FROM test');
foreach($results as $row)
    foreach ($row as $key=>$val)
        print "\t$key\t$val";
$delete_count = $db->delete('DELETE FROM test WHERE id=?',array($newid));

The page with links to the code is in the list up top, and everything is MIT license. Enjoy.

Drizzle – Lightweight DB Based on MySQL

July 23rd, 2008 by sjan

Brain Acker, director of architecture for MySQL has opened the door to Drizzle, a light-weight, high-concurrency database server based on MySQL, targeted at web applications.

The architectural ideas as described in the FAQ:

A micro-kernel that we then extend to add what we need (all additions come through interfaces that can be compiled/loaded in as needed). The target for the project is web infrastructure backend and cloud components.

The FAQ goes on explain the differences between Drizzle and MySQL which include:

No modes, views, triggers, prepared statements, stored procedures, query cache, data conversion inserts, ACL. Fewer data types.  Less engines, less code. Assume the primary engine is transactional.

Michael Widenius, founder and original MySQL developer, explains more about Drizzle in this blog post, including the most interesting piece (to me) – that Drizzle will always contain the most up-to-date InnoDB code, meaning you don’t need to wait around for MySQL 6 or download the plugins from Oracle each year to get the latest and greatest.

While Drizzle is still in development you can check out the code and try it out. Or maybe even get involved and help out. More information on how to do both is available on the wiki.

Apache 2.2.6, PHP 5.2.4 and MySQL 5.0.45 on OS X

September 17th, 2007 by sjan

I got tired of looking for a way to replace the Apache/PHP that Apple packages with OS X (without breaking anything else in the process) so I decided to install Apache 2.2 and PHP5 in their own location to avoid stepping on the Apple package toes.

Since I do a great deal of development again MySQL I needed to install that as well, and figured that I would probably need the GD functionality as well so I grabbed libjpeg and libpng to make those work as well. This is the step-by-step.

(Props to James Pelow and his article from last year, from which I borrowed the configure command lines and configuration modifications, as well as the idea of installing the whole mess in /apache2.)

Download the latest MySQL (I used the package version) from MySQL.

Installation is straightforward following the same methods as any other Mac installer.

Download and install libjpeg and libpng – from Ethan Tira-Thompson (this is also in a Mac installer which contains both libraries in one installer).

Download the latest Apache httpd server (Unix source) from Apache

in the terminal:

tar -xzvf httpd-2.2.6.tar.gz && cd httpd-2.2.6






sudo make install

sudo mkdir /apache2/php

Download the latest PHP from PHP

tar -xzvf php-5.2.4.tar.gz && cd php-5.2.4




















sudo make install

sudo cp php.ini-dist /apache2/php/lib/php.ini

Now to make your Apache2.2 a little more ‘Mac’ – you can point it at the Mac web shared files folder, change the user and group and change the location for user files to match the Mac folder system.

Edit httpd.conf (I use nano, you can use any flat text editor like nano, pico, vi, emacs or even BBedit)

sudo nano -w /apache2/conf/httpd.conf

The changes to httpd.conf I made:
I changed

User daemon
Group daemon


User www
Group www


DocumentRoot "/apache2/htdocs"


DocumentRoot "/Library/WebServer/Documents"


<Directory "/apache2/htdocs">


<Directory "/Library/WebServer/Documents">

and added

AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps

DirectoryIndex index.html index.php

Edit httpd-userdir.conf

sudo nano -w /apache2/conf/extra/httpd-userdir.conf

The changes to httpd-userdir.conf I made:
I changed

UserDir public_html


UserDir Sites

To start and stop the server:
MySQL comes with a Preference Pane that allows you to start and stop it there. To start and stop Apache you need to first make sure that the default Apache shipped with OS X is stopped.

sudo /apache2/bin/apachectl start
sudo /apache2/bin/apachectl stop

I only ran into one issue, when trying to start the server I ran against the following error message (and no running server, of course):

httpd: Syntax error on line 53 of /apache2/conf/httpd.conf:
Cannot load /apache2/modules/ into server:
Library not loaded: /usr/local/mysql/lib/mysql/libmysqlclient.15.dylib
Referenced from: /apache2/modules/libphp5.son  Reason: image not found

To fix this I did the following:

cd /usr/local/mysql/lib
sudo mkdir /usr/local/mysql/lib/mysql

for i in `ls ./l*`; do sudo ln -sf /usr/local/mysql/lib/$i /usr/local/mysql/lib/mysql/$i; done

This creates soft links in the directory that is looking for the MySQL libraries.

Then it started right up! Wheee! (I did a quick test by dropping PhpMyAdmin into the /Library/WebServer/Documents folder and browsed to it – the whole Apache/PHP/MySQL is working correctly)

Technorati Tags: , , ,

A simple intro to database normalization

August 30th, 2006 by sjan

I found a very clear, well-written introductory example to database normalization on devshed. Although it is in the MySQL portion of the site, it applies equally well across the board to other RDBMSs.

To get more details on normalization, the normal forms, and general good database development in general, check out Database Design for Mere Mortals: A Hands-On Guide to Relational Database Design, Second Edition by Michael J. Hernandez. Without a doubt the most useful db development book I’ve ever laid my hands on.