Title: IDX: Load Balancing Directories
Author: Promethyl
Catagory: Code Which Makes Servers Happy
Date: Monday, July 21 2003

Ah, Linux ext3. From version 7.2 of Redhat Linux on, this has been the standard for filesystems. Think of it as NTFS for Linux boxes.

Althought EXT3 is more optimized than it's predecessor (aptly named EXT2) there are a few tricks you can do to make it faster.

The biggest data load you're going to have in your online transaction processing software (OTPS; hereafter, IDX) is not going to be the data. It's going to be the pictures.

Storing them in the database as BLOBS is not the solution. Backups take forever, and the database slows down. Store them on the hard disk.

Figure each picture is 32k. The average MLS has 4000 listing available to IDX, only 85% have pictures, and each listing has 8 pictures. That's 27,200 pictures. If there were to be all dropped in the same directory (ala Open Realty) it would take a long time to find a single file.

The solution; of course, is to create ten folders, and name them [0..9]. Let's say you're storing pictures for MLS#: 2000003. Use the last digit to decide which directory to store them. So the pictures would be located in images/mls/3/.

This optimizes the findfirst and findnext functions of the filesystem. It provides a faster reaction time, and allows you to add on NAS support later.

(If you have to, you can distribute your images among 10 Network Attached Storage devices - NAS units which would provide extreme performance. Hope and pray you never have to deal with it when it gets that big.)

For more information on the ext3 file system, click here.