Evil, Thy Name Is CheckDisk

I love NTFS. Back in the bad old days of FAT32, every time the computer was suddenly powered off there was a serious possibility you could screw up your hard drive. NTFS is much better able to cope with such no-nos, and it is rare for the hard drive to ever become out of sync with reality.

Every once in a very blue moon, however, even NTFS has a bad day. When something does go seriously wrong, such as an internal error in the file system, Windows will automatically set a flag on your hard drive that indicates the volume is "dirty" and needs to be checked for errors. When you reboot your computer, XP very nicely runs chkdsk automatically for you to fix any small problems that might have been created.

Oh, all is happiness and joy in the land where Windows XP lives!

A little history. About two years IC7-MAX3 PC started acting flaky. When I attempted to boot it, about one out of every 3 times it would stall half way through the process. After nearly a year of trying to figure out what errant software driver was causing this problem, I stumbled upon the fact that one of the chips in my motherboard's southbridge was going bad. The chip in question was the ICH5R chip responsible for IDE access. Once I moved my CDROM from IDE port 0 to IDE port 1, all my problems went away. The computer behaved completely normally in every other respect, so I saw no reason to not continue using the system.

I have a lot of hard drive space, and I've had my fair share of hard drive crashes over the last 20 years. You could say I'm a bit paranoid about my data, so I have all of my user data on a set of mirrored 500GB SATA drives. The operating system resides on another drive entirely, since experience has taught me that software problems that screw up hard drives usually only screw up the drive with the OS on it. These two drives are not mirrored in software; rather, they're mirrored using the hardware Intel raid. Which is provided by the Intel raid chipset. The ICH5R chipset. The same southbridge chip I already had problems with.

Last night my computer suddenly started acting a bit flaky. I tried to open several documents, but the software complained the documents were defective. Windows picked up on this and set the dirty flag on my hard drive. I rebooted, and along came good ole' XP to save my hide.

The drive it was complaining about was my data drive. Now, I have enough experience with computers that when I see this message my heart skips a beat. I know what this message really means: something went wrong. I had rebooted nicely. I was not expecting this message. "Well, windows probably just needed to fix up a few falsely allocated clusters, I guess." 7... 6... 5... 4... Instinct told me to either press the space bar or yank one of the SATA cables from my raid array. 3... 2... 1...

I ignored instinct.

A word of advice: When you've had a strange things happen while using your computer, you reboot it nicely, and then windows offers to run chkdsk for you? Don't.

My ICH5R southbridge raid chip had failed. I didn't know it at the time, of course, but random bits were being flipped as the perfectly good data was being read from the hard drive. Good on the drive. Good on the cable. Good coming in to the chip. Bad going to the microprocessor.

Chkdsk dutifully attempted to correct my perfectly good hard drive. "Oh, that bit shouldn't be set, let me delete that file. Oh, that folder seems a bit odd, let's get rid of it." Self-preservation kicked in and I jumped out of my chair and yanked out one of the two SATA cables on my raid array. Windows kept right on chugging.

"You don't want this, it's bad. Hey, look at this node, it's all messed up. Let's throw it out!"

Within seconds chkdsk rummaged through my hard drive, destroying data right and left. Not only was the software writing away all my data, but the ICH5R chip was corrupting the new data as it was being written out. My file allocation table was being slowly obliterated.

After a few minutes of rampage, windows booted. I tried to get a directory listing, but it was full of nonsense strings where my filenames used to appear. Darkness fell upon the face of Windows XP land. All hell broke loose. I kept running chkdsk /f, and chkdsk kept finding and correcting new "errors". I was completely confused.

After a few minutes I remembered that I had problems with the southbridge before. I moved the hard drive to the other SATA controller chip on the motherboard and rebooted. Chkdsk once again rummaged through my hard drive, but this time it seemed better behaved. Once it had completed one cycle of deleting this and that it seemed satisfied to leave my drive alone.

Gigabytes of my data were gone.

Now that I knew what had happened, I pulled the SATA cord from the drive and attached it to the other drive of the mirror set... the one I had yanked the cord off of so abruptly.

Windows rebooted, and again chkdsk rummaged my drive. But this time it found far fewer things to "fix".

In the end I lost nothing, but only because I run a raid system and yanked the cord based on instinct. The moment I realized all my data was intact on the second drive I felt a huge relief. It's like that feeling you get when you realize you nearly turned left in front of a semi, but looked at the last moment and hit the breaks.

There's a simple lesson to be learned from this. System recovery is not a task that should be automated. When you system acts strangely, the worst thing you can do is to continue to let it run.

If this happens to you, don't do what I did. When chkdsk pops up after something strange happens, cancel it. Instead, let windows boot and run chkdsk from the command prompt without the /f "fix errors" flag. This will let chkdsk run and report to you what it thinks is wrong without actually changing anything. Run it twice... three times... to make certain it's reporting the same thing each time.

See what chkdsk reports and decide if it's reasonable before you let it mess with your hard drive. If things don't feel right, move the drive to another computer and make sure the problem really is with the file structure. Otherwise you could end up trashing your perfectly good data very quickly.

In retrospect keeping the motherboard after I knew there was a problem probably wasn't the best thing to do, but that doesn't change the fact that this kind of problem could strike at any time. Having backups (a little old, but I did have them) along with mirrored data helped keep me out of trouble in spite of my bad decision to keep the motherboard. Learning to protect yourself from yourself is an important part of computing. And although I did have some warning that a failure was imminent, it didn't have to be that way -- you might not be so lucky. I'll be adding a third hard drive to my raid array, rotating it out every week as an extra precaution now. You can never be too paranoid about your data.