Linux Category

If you’re working on LAMP, you’re most likely already taking advantage of journaling on your file system and with your database. With journaling, any operation performed is first logged to the disk and then read circularly to be performed.

On a modern file system such as reiserfs or ext3, journaling means that a sudden loss of power between the removal of a directory entry and the marking of its inode as free in the free space map won’t lead to an orphaned inode (a storage leak).

In a database, journaling is often referred to with ACID (atomicity, consistency, isolation, durability) or the bundling of a series of operations in a transaction. When banking for example, you wouldn’t want a process to fail after deducting the money from your account but before depositing it as a payment into another account.

So far, we’re pretty comfortable implementing journaling everywhere that we’re storing data; but what about in the application layer. As the project triangle asks, we toss out simplicity (bye bye Ruby on Rails) for better reliability and performance. I propose numerous situations where journaling can be beneficial:

1. Eliminate latency between the front-end web server and the database server (such as when front-ends are deployed internationally to provide better local service).
2. 2) Eliminate downtime when databases need to be cycled offline for updates, backups, or during unexpected outages. When running a database on a cloud, maintain service during upgrades or when Amazon has its “hiccups”.
3. Eliminate the waiting period after a user hits “submit” — even if it requires a response.

The Problem

There is a disconnect between the front-end web server serving content/running the application and the database server processing queries that store and retrieve data. That disconnect can temporarily fail or become slow.

The Solution

Front-end caching manages its own “temporary” database locally. All writes are written to the local database. All reads come from the master or local replicated database and ALSO the local database. All data in the local database is moved to the master database as fast as it can be.

Implementation

The key requirements of a front-end application journaling implementation are that it has to be fast, lightweight and use on-disk storage.

First, storage: The best tools for the job are Apache’s CouchDB, local MySQL install, SQLite or a custom implementation of flat files (perhaps XML) in a structured directory tree. Whatever you use, it needs to be setup in a manner that the oldest entries can be read with FIFO (first in, first out). Each “node” stored represents a CRUD (minus the R) action, or: create, update and delete.

Second, lightweight. The goal is to keep it fast, we’re not aiming at permanent storage or to turn the front-end into a database server. The resource usage requirements should be very low. Things like indexes aren’t required as we’re reading and writing circularly.

Third, on-disk storage. We can’t rely on things like memcached or in-memory tables.

The next hurdle is key generation. When adding new items, we have to generate and retrieve new primary keys in the master database (which often requires a response, as after adding an item the client is often redirected to view it). Obviously we don’t have this key as we’re writing it to the local database. One solution is that each front-end has an algorithm which generates number patterns unique to it. The second solution is you use a separate numbering scheme (such as one prefixed with a ‘z’) for local items. You then maintain a table with relationships between the local ID and the master ID when it becomes available. That item then becomes accessible either by the local ID or the master ID from then on.

The format of the “nodes” is pretty simple. It should be a compilation of the field/value pairs required to insert the record, serialized, in whatever form you’re using to store it.

A cron job or background process should be setup which routinely reads out the oldest nodes, unserializes the data and attempts to connect to the master database and process the action. Only upon success is the local node deleted or archived.

If you want complete break-away-ability from the master server, a local replicated database of the master should also be available to the front-end for all reads. You’ll also need to combine all read operations against the master or replicated database with the new local layer (look out for LIMIT clauses for pagination).

Implementing this process is complicated and adds complexity to your application; but, the results can be quite amazing. Imagine taking your database offline for an hour to install a hardware upgrade without ever taking your application down. Perhaps your database cloud provider has a big “oops” and loses all your data. If you maintain a few days of nodes, just restore a recent database backup to a new server, and re-read the old nodes. Your application never goes offline and your back in business with a new master database server in a few hours.