The Spacer

The Spacer

How to make your hard disk fast again

Well, so you're the proud owner of a hard disk. Welcome to the club! Isn't it great?

You're amazed by the sheer speed of the hard disk. Copy a file to it? Zip. Copy a file from the hard disk, to the hard disk? Zip, zip. We're talking performance far higher than an IBM. A 200K file can be copied in three seconds on the ST; it takes ten seconds on a PC.

But something seems to have happened to your hard disk recently. It's—heavens!—bogging down. Slooooww.

In fact, the delays are getting ridiculous. You try to copy a file on the hard disk, and you get a zip sound, then, a slow blink, blink, blink of the drive light, and 15 seconds later, the zip as the operation completes.

Well, you've got a problem. In fact, we've all got it. It's the dreaded FAT Lookup Problem. Let's find out about it, and how to fix it.

SOME TRIVA

Your floppy disk is composed of 720 512-byte sectors. Oddly enough, the sectors are numbered 1–720. (If you go double-sided, make that 1–1440). 720 sectors at 512 bytes per sector is 360,000 bytes per side of the disk; that's where your ST's disk size comes from.

Now, a hard disk is just like a floppy disk in that there are a bunch of 512-byte sectors. However the quantity is much larger. A 5-megabyte hard disk has 10,000 sectors! Your standard Atari SH204, a 20-megabyte unit, has 40,000 sectors on it.

The operating system worries about splitting up these individual sectors into files, subdirectories, and so forth. It keeps a table, called a File Allocation Table, which is a map of all the sectors. It marks each sector as used, unused, or bad.

Unfortunately, the folks who wrote TOS (The Operating System) made a little mistake before committing it irrevocably to ROM: the routine that processes the FAT is quite slow. In specific, scanning a FAT for an "open" sector to write to is extremely slow. You do this scan anytime you write something new to the disk; the operating system is finding free sectors to write to.

It was a natural mistake. Working with floppy disks, they never noticed the speed problem. There's only 720 or so sectors on a floppy, and the FAT search problem wasn't noticeable. In fact, until you get over a megabyte of data on the storage media, you don't notice it at all.

However, hard disk owners tend to collect lots of data, and then the problems become extremely evident. Let's go through a sample copy. Assume you have a hard disk partitioned into four 5-megabyte parts, a very common thing to do. You're copying a file from disk C to disk F. Assume also that disk F is rather full.

You drag the icon, and let go. First, you notice the drive light go on and the head move; it's pulling the file off drive C. Then it begins to write.

Slowly, very slowly, it reads in the FAT for the disk F. It's looking for a place to put the data it just read. You'll see the drive light blinking—and anytime the ST goes to slow the hard disk light has time to go out, you're in trouble. Finally, it finds a home for the data. Zip! The data's written.

Of course, it took a while.

If you think I'm kidding, let's look at the times to duplicate a 200K file on the hard disk, only varying how much data is out there first.

Table 1: Disk timings. Duplicating a 200K file.

Disk empty: under 3 sec.

Disk 5 megabytes: 30 sec.

Disk 10 megabytes: 45 sec.

Disk 15 megabytes: 63 sec.

This is bad. All that extra time is spent diddling with the FAT.

In contrast, an IBM PC takes around ten seconds to duplicate a 200K file, regardless of how full the disk is; they've got a good fast FAT search algorithm, so the overhead doesn't slow things down. [Why ten] seconds? Mainly, a slower hard disk and slower processor.

If you're using something like Alcyon C, which creates six temporary files during the compile/assembly process (No? Look at AS68 closely. It takes three by itself!) you're going to be old by the time it's done. And why was it you bought a hard disk? Speed?

Those of you familiar with my style probably think I have some magic desk accessory or program to fix this. Alas, I don't. The problem is deeply rooted in the slippery, slimy entrails of TOS, and like any digestive system problem, it's hell to diagnose and fix. It's all in ROM, just to make it worse.

I do, however, have a technique to speed your hard disk back up when you have a lot of data out there: It's called "spacing." I believe that all of you suffering from the HD slowdown will benefit a lot from this technique.

Let's look at the FAT process again. The slowdown occurs when the operating system is looking over a huge disk, with thousands of sectors, trying to find the first sectors that are free to be written to. The problem is only in writing, not in reading.

On a hard disk, the first data written to it (the first file copied to it after formatting) is written in the first sectors (e.g., sectors 1-xxx of the 40,000 available). The next file goes in the next sector, and so on.

The farther we get away from the start of the hard disk, the longer the hard-disk driver (FAT) takes to do the write. So, the first conclusion is, keep your disk nearly empty if you want it to stay fast.

This is fine, but there are those of us who want to store lots on our hard disk and still have it move quickly. I mean, heck, the Alcyon C compiler takes about a megabyte, what with all its support files. And what about all those good downloads? NEOchrome files?

For those of you that want lots of data and peppy performance, we have the spacer technique.

Let's imagine how the hard disk looks after you've copied, typically, the Alcyon C compiler to it (which, with all its subfiles, takes up lots of room). Let's further assume a 5-megabyte partition (segment of hard disk).

Sector 1 (start of 5-megabyte area)

Alcyon C and files: comprise about 1-megabyte

Section 2,000

Empty area

Sector 10,000 (end of 5-megabyte area)

Now if you go do a compile, all those temporary files, and all your editing, gets done starting at sector 2,000. Every time you write to the disk, you have to sit and wait while the ST hunts through the first used 2,000 sectors. That will bog you down.

Most people's hard disks look like this. Data gets put on them willy-nilly, whenever it's available, in the first open sector.

Instead, let's do something different. Before copying anything to the hard disk—when it's new, freshly formatted and zeroed, let's put a huge file on the hard disk called a "spacer." One easy way to make a spacer is to create a folder and fill it with long files. I use a listing of the Magic Sac, which is about 500K. However, feel free to use any long files that come to mind.

Let's make our spacer 4 megabytes long, and copy it to the hard disk. It looks like this now:

Sector 1 (start of 5-megbyte area)

Spacer folder files (temporary, 4 megabytes)

Sector 8,000

Empty area

Sector 10,000 (end of 5-megabyte area)

Only, now copy the Alcyon files to the hard disk. Of course, this is going to take awhile, because they're so far away from the start of the hard disk, so go get some coffee while it works:

Sector 1 (start of 5-megabyte area)

Spacer files (temporary, 4 megabytes)

Sector 8,000

Alcyon C (1 megabyte or so)

Sector 10,000 (end of 5-megabyte area)

Okay, your hard disk is now full.

Go delete the spacer. The Alycon C files aren't going to move, which is one of the secrets of this technique, so we have:

Sector 1 (start of 5-megabyte area)

Empty space

Sector 8,000

Alcyon C (1 megabyte or so)

Sector 10,000 (end of 5-megabyte area)

Now we're getting somewhere. When the operating system makes a new, temporary file, it doesn't have to search very far: There's a big empty area at the beginning of the disk! In tests I've run, this has increased the performance of compilers by 300%.

When you think about it, how many of the things you put on the hard disk are ever modified? If you're like me, there's piles of utilities and the like that never get changed. With the Spacer technique, you can put them on the end of the hard disk, where they won't be slowing down your work.

The amount to space your files across the disk is strictly up to you. I'd go at least halfway across the hard disk, but it's up to you. Again, the idea is to first use up the "fast" part of the hard disk with a spacer, which forces the following files to be written in the "slow" part. When you get rid of the spacer, you'll do all your work on a remarkably fast hard disk.

Won't your utilities slow down?

No. Remember, the bug is only while writing to the disk, not while reading. All you do with a utility is read it in (like the C compiler code). Temporary files and such are what you're concerned with.

Periodically, as things slow down, you may need to go respacer the hard disk to get your performance back.