Why We Needed a New File System

Until Windows NT, Microsoft was happy with its old-and-proven file system, the FAT. With the advent of server and workstation-based operating system called Windows NT, Microsoft introduced a new file system, the NTFS. Did we need another file system? What are the benefits of NTFS as compared to FAT, and should everyone start using it in their devices? Let’s try to find out.

Contents

1. NTFS: The New Technology File System
2. Scalability
3. Reliability
4. Security
5. Compression
6. Alternate Data Streams
7. Sparse Files
8. Reparse Points

NTFS: The New Technology File System

FAT was designed a long time ago with diskettes in mind. It didn’t scale well, and it has already hit limitations in many areas. The maximum file size of only 4GB, the path length limitation, the lack of security and recoverability features and many other things made Microsoft design a new file system from scratch.

When designing NTFS, Microsoft was borrowing certain technical details from its older file systems, FAT and HPFS (an OS/2 file system). The company wanted to make a robust, extensible and future-proof file system with high performance, reliability, and security being the most important points. So they’ve made NTFS.

The NTFS is a unique file system which is based on… files. That’s right: everything in NTFS is a file, even the file system itself. File system journal, the Master File Table (MFT) and many other system structures are stored as files within the NTFS. This unique property makes NTFS the one and only true object-oriented file system; no other file system shares this property with NTFS – and there are literally dozens around.

So what makes NTFS so great? Let’s see.

Scalability

Microsoft developed NTFS during the time where one gigabyte was a huge size for a hard drive. However, the company anticipated the rapid growth of storage technologies, making its new file system support huge volumes (the maximum theoretical NTFS volume size is (2^64)−1 clusters, while each cluster can be sized up to 64KB). The size of a single file is limited to 16 exabytes. Just for a reference, one exabyte is one million terabytes, or one billion gigabytes. With these numbers around, NTFS if one of the most scalable and future-proof file systems around.

Reliability

Reliability was never a strong point of the FAT file system. A single corruption within the file allocation table could with time turn into a huge data loss. With larger storage systems, Microsoft wanted to avoid this type of problems. The company addressed the reliability issue by introducing several techniques: a file system journal and volume shadow copy.

The file system journal uses the NTFS Log File ($LogFile) to record changes to the disk volume. The journal is there to ensure that the file system remains consistent even if the operating system crashes during a write operation. Journaling helps implement transaction-based writes. If an operating system needs to write something onto the disk, it will first add an entry into the file system journal, perform the write operation, and then mark the operation as committed in the log file. If Windows crashes during the write operation, or if there is a sudden power loss, the system will be able to easy rollback uncommitted changes while booting Windows.

Another reliability measure in NTFS is Volume Shadow Copy, a service that keeps historical versions of files and folders on NTFS volumes by copying old, newly overwritten data to shadow copy. This service allows users restoring old, overwritten versions of files (such as system files and settings, programs, backups, documents etc.) long after they’re gone. Interestingly, a Volume Shadow Copy can be stored on a separate disk.

Security

FAT32 has no security. No permission management, no file system encryption, and no user-level encryption are available. NTFS offers all that, and more.

Operating system-level security includes Access Control Lists for permission management. Each NTFS object (file or folder) has a list of permissions attached to it. The detailed list specifies which users or groups (local or network) can read, white, execute, and change permissions of the object. Permissions can be inherited by nested files and folders.

Obviously, these permissions are only enforced by the operating system that supports them. Data recovery tools such as NTFS Recovery Software successfully ignore NTFS permissions in order to recover deleted files. However, two other security measures are available to protect valuable information.

NTFS comes with per-object, user-level encryption. Each file and folder can have the “Encrypted” attribute set. When this attribute is enabled, Windows will encrypt the content of files and folders with a strong encryption key dependent on the user’s Windows account password. If the password is changed by the user in a legitimate way, there encrypted files will remain accessible. If, however, the password is simply reset by the administrator (or hacker), Windows will not be able to calculate the correct decryption key for any encrypted files, and access to those files will be lost forever.

Interestingly, encrypted files are still files stored within the NTFS. Deleted encrypted files can be recovered just as easily and reliably as unencrypted ones with NTFS Recovery Software or Partition Recovery Software.

NTFS encryption can be used to protect files and folders between Windows users on the same computer. However, Windows comes with an even stronger and more complete encryption mechanism that is called BitLocker.

BitLocker provides system-level full disk encryption. BitLocker is effective against offline attacks where the disk is taken out of the original computer and inserted into a different PC. Without a proper decryption key, BitLocker volumes will remain securely encrypted.

It is important to note that BitLocker is a system-level encryption scheme. Once the volume is unlocked, encrypted data becomes accessible to all users of the particular Windows computer. This is the difference between BitLocker full disk encryption and user-level NTFS encryption.

What about recovering files stored on a BitLocker-encrypted volume? You will have to unlock and mount the encrypted volume before performing a data recovery attempt. If you moved a BitLocker volume to another computer, you will absolutely need to enter a BitLocker Recovery Key (escrow key).

What happens if a BitLocker encrypted volume becomes damaged and cannot be mounted even if you do have the escrow key? The procedure becomes much more complicated then. It’s too long and complex to describe in this article; please refer to http://www.eightforums.com/tutorials/21714-bitlocker-repair-tool-recover-drive-windows-7-8-a.html for a complete walkthrough. You will need a BitLocker Repair Tool from Microsoft to repair the volume, which you can obtain directly from Microsoft https://www.microsoft.com/en-au/download/details.aspx?id=17294.

Compression

NTFS has the ability to transparently compress the content of files and folders to save disk space. The compression is set by enabling the “Compressed” attribute, and is performed by Windows on the fly. The compression algorithm is optimized for random access, meaning that you can easily read a few bytes of data in the middle of a lengthy video clip without the system having to decompress the entire file.

Compressed files are little different from regular files from the NTFS point of view. As a result, compressed files that were deleted can be recovered just as easily and reliably as plain, uncompressed files by using NTFS Recovery Software or Partition Recovery Software.

Alternate Data Streams

Alternate Data Streams are a little known and rather cumbersome feature of NTFS. This feature allows files to have additional parallel streams of data associated with them. These data streams are invisible to most file management apps such as Windows Explorer. While this feature was initially developed to enable compatibility with Services for Macintosh, they no longer serve the original purpose as SFM have been discontinued a long time ago. Today, the only practical use of alternate data streams is adding Zone. Identifier marks to files downloaded by Internet Explorer (and other browsers) from external Web sites. The mark can be used to identify files that are possibly unsafe to run. You’ve probably seen that Windows prompt more than once if you ever downloaded a file from the Internet; now you know that the data is actually stored in an alternate stream.

Alternate data streams are not accessible from Windows Explorer. Their size does not appear as part of the file’s size. As a result, alternate data streams have been exploited by many computer viruses and Trojans to hide. No worries, as all major antiviruses check for these places when performing a disk scan.

Microsoft released a tool called Streams to allow viewing the content of alternate data streams. Alternatively, you can use the command line to access the content of those data streams by using the following syntax: “filename:streamname” (e.g., “MyDownloadedFile.exe:extrastream”).

NTFS Recovery Software fully supports alternate data streams and recovers them when repairing NTFS volumes.

Sparse Files

Another interesting and highly unusual feature of NTFS is called sparse files. In sparse files, unused (empty) areas are represented with metadata without occupying any physical space on the hard drive.

How is this useful? Imagine a huge database with a main DB file of several terabytes. A file that size can easily occupy the whole hard drive. To actually create such a file on the hard drive, the system would have to write a lot of zeroes to several terabytes of disk space. That would take a very long time. To save time and disk load, NTFS made it possible to indicate unused areas in a file with simple metadata references. Thanks to this feature, a database application may create a huge database files instantly; the system will only write actual data onto the disk, while leaving unused regions empty without writing anything into them.

Reparse Points

The final interesting feature of NTFS we’ll be covering today is called reparse points. With this feature, users can create so-called NTFS hard links and junctions (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396).

Hard links and junctions (“soft links”) allow linking directories located on the same or different local volumes on the computer to point to a certain folder. For example, you can create a link the “C:\Downloads” directory to a much longer path of “c:\Users\UserName\Downloads”. From then on, you can simply use the “C:\Downloads” directory instead of the longer path. Any file saved into “c:\Users\UserName\Downloads” will automatically appear under the linked directory “C:\Downloads”.

All data recovery tools including NTFS Recovery Software will handle NTFS reparse points.