Fragmentation is the demon that haunts hard disks for all users of Windows (and other operating systems). Defragmentation is the exorcist for that demon. After closely examining the subject, I've found that many of the things we held to be true about both fragmentation and disk defragmentation have become less and less true over time.
The NTFS file system in Windows has helped alleviate many of the worst problems associated with fragmentation. However, it has not cured them. In my first article on the topic, I hinted that that NTFS might still suffer from serious problems because of fragmentation of the file system metadata.
However, file systems are not the only things that have changed in the last ten years or more. Hard disk drives have changed too.
In 1996, a new Western Digital hard drive holding 1.6 gigabytes (what you could expect to fit on a decent-sized flash drive today) cost $399. That drive was in the same 3.5-inch form factor used for today's hard disk drives, so the size of the platters in the drive were pretty much the same.
Today, a new Western Digital drive that costs $199 holds 500 GB. So for half the money you're getting more than 300 times more capacity.
But there have been other changes. The platter sizes of drives have remained the same, but the rotational speeds of the platters have gone up—to as much as 15,000 RPM on high-end drives. But the biggest change in hard drive technology over the past decade has been in information density. With so much more information crammed into the same space, the drive heads move that much less to read that much more data. This in itself speeds up data transfers.
Most hard disk drives now also come with an on-board cache of RAM, usually 16MB or more. Data can then be read from the disk into the drive's cache in one big chunk, and parceled out to the host computer as needed. This saves the hard drive from having to go back and dip repeatedly into the same parts of the drive for multiple bits of data. On top of everything else, the operating system does its own caching to further alleviate the affects of a fragmented file—or even fragmented NTFS metadata, which tends to be cached and held as needed.
What does all this mean? The big disadvantage of fragmentation – that it scatters data across a drive -- has been greatly offset by all of these changes. Furthermore, 50MB of data fragmented across a ten-year-old hard drive will have a far bigger impact on performance than the same 50MB fragmented to the same degree on a new hard disk drive. Now factor in the benefits of caching, faster and smaller head movements, modernized file system storage and OS-level caching, and it becomes clear why fragmentation isn't anywhere near as bad as it used to be. (The one exception to this scenario is if the file is scattered literally all the way across the surface of the drive, which is fairly unlikely.)
While researching the topic of disk defragmentation, I talked to several experts on the subject. One of them was Mark Patton. Today he is development manager of CounterSpy Enterprise from Sunbelt Software, but years ago he worked on Executive Software's Diskeeper, the disk defragmentation software now used in a stripped-down version in Windows XP. Here's what he told me:
"Larger and faster drives have minimized the impact of fragmentation. The Windows file system tends to fragment files all on its own to a small degree, but fragmentation starts for real when the drive starts to get full—as in over 70%. "As the drive fills up, the larger areas of free space become scarce and the file system has no choice but to splatter large files around the disk. As the drive gets really full (over 90%), the file system then starts to fragment the MFT and the Pagefile. Now you've got a full drive, with lots of fragmented files, making the job of the defragmenter nearly impossible because it also needs space to do its job. "It is my opinion that a drive that is more than 80% full is not defragmentable. You can see that effect with huge hard disk drives, since they generally use smaller percentages of the drive's total free space. A side-effect is that the overall fragmentation is reduced, and the fact that these drives have faster seek times makes the effect even less noticeable.
"At the time I worked on Diskeeper, I always told people to 'defragment early and often' so that they could take advantage of the free space before their drive starts to fill up. This way, they could see a marginal improvement now, but, more importantly help, the defragmenter from getting log-jammed later on. With today's large drives, even this is not an issue."
In short, many of the issues that crop up tend to be because of a lack of free space causing cumulative problems. With larger drive sizes, faster access times and smarter handling of storage, the total impact of fragmentation has been reduced. That being said, defragging still has a place. My next article on this topic will include some recommendations about how and when disk defragmentation should be done in the real world.
Disk Defragmentation Fast Guide
Disk defragmentation: Performance-sapper or best practice?
New hard disk drives reduce need for disk defragmentation
Four steps to lessen the effect of fragmentation.
Flash memory drive defragmentation: Does it make sense?
Three disk defragmentation issues defined