Files, page cache, storage and persistence

Why it matters

"The write call returned, so the data is safe" is one of the most common and most expensive misunderstandings in systems work. Databases, message queues, model checkpointing and anything that promises not to lose acknowledged data depend on knowing exactly which call moves data across which boundary. The same layers explain performance puzzles: a benchmark that reads a file twice measures memory the second time, and a service on network-attached storage can see fsync latencies far above a laptop's. Backend, infrastructure and database interviews ask what "durable" means and how to make an update crash-safe.

Core ideas

The path of a write. Data typically passes through a user-space buffer in the language runtime or stdio, then the kernel page cache, where write() copies it and marks the page dirty, then the device's volatile write cache, and finally persistent media. A successful write() only means the page-cache step happened. The kernel writes dirty pages back later, based on their age and on dirty-memory thresholds; a crash before that loses them.

What each call guarantees.

• fflush (or a language's flush) moves data from the user buffer into the kernel and nothing more.
• fsync(fd) writes the file's dirty data and metadata to the device and, on Linux, is specified to write through or flush a volatile device cache before returning.
• fdatasync(fd) skips metadata that is not needed to read the data back, such as modification times.
• O_SYNC and O_DSYNC make every write behave like a write followed by fsync or fdatasync.
• O_DIRECT bypasses the page cache, with alignment rules. It is a caching choice, not a durability guarantee; the man page says to combine it with O_SYNC when durability matters.

Durable, defined. Acknowledged data survives a crash or power loss. That requires every volatile layer above the media to be flushed or power-loss protected, and devices that honor flush requests. Durability also has a scope: surviving power loss differs from surviving the loss of the machine (replication) or silent corruption (checksums). Platforms differ: on macOS, plain fsync does not make the drive flush its cache; fcntl(F_FULLFSYNC) does.

Names are data too. Creating or renaming a file changes its directory, and fsync on the file does not necessarily persist the directory entry. The standard crash-safe replacement:

write new contents to tmp; fsync(tmp)
rename(tmp, target)          # atomic switch of the name
fsync(directory of target)   # make the rename itself durable

Crash consistency inside the file system. One logical update, such as appending a block, touches several on-disk structures: a bitmap, the inode and the data block. A crash between those writes leaves an inconsistent state. Older systems repaired it with a full scan (fsck); journaling file systems first write the intended changes to a log, the same write-ahead idea databases use; copy-on-write file systems never overwrite live data. File systems differ in the ordering they promise for application data, which is why portable applications fsync explicitly.

Errors are part of the contract. If background writeback fails, Linux may report the error once and leave the affected pages marked clean, so a retried fsync can succeed even though the data never reached storage. Robust systems treat an fsync failure as fatal and recover from their own log.

Devices. Hard disks pay milliseconds for seeks and rotation, so sequential I/O dominates their design. Flash SSDs read randomly well but write in pages and erase in much larger blocks; the flash translation layer remaps writes and garbage-collects, causing write amplification and latency spikes under sustained random writes. Cloud block volumes add a network round trip and replication, and enforce IOPS and throughput limits.

Common misconceptions

• "A successful write() means the data is on disk." It is in the page cache until writeback or fsync.
• "fsync on the new file makes its creation durable." The directory entry needs an fsync of the directory too.
• "O_DIRECT means durable." It bypasses the page cache; without O_SYNC or fsync, data can still sit in the device cache.
• "If fsync fails, retry it." On Linux the retry may succeed falsely after the data is gone.
• "fsync means the same on every platform." macOS needs F_FULLFSYNC to flush the device cache.
• "SSDs make random writes free." Random reads are cheap; sustained small random writes trigger garbage collection and tail-latency spikes.

How to approach it

Be able to draw the layers and label each call with the boundary it crosses. Practice the atomic-replace recipe from memory and explain the failure each step prevents. When discussing a storage engine, ask where its durability point is: exactly when a commit is acknowledged relative to which fsync. On Linux, time small appends with and without fsync, and run strace on a database commit to see which calls it actually makes.