File systems ​

SeaweedFS & Spacedrive look nice.

Links ​

• TMFS - Apple's Time Machine fuse read only file system.
• sparsebundlefs - FUSE filesystem for reading macOS sparse-bundle disk images.
• Understanding Google’s File System (2020)
• mergerfs - Union filesystem geared towards simplifying storage and management of files across numerous commodity storage devices.
• gefs - Good enough file system.
• libfsapfs - Library to access the Apple File System (APFS).
• proper-lockfile - Inter-process and inter-machine lockfile utility that works on a local or network file system.
• How do you lay out your home directory? (2020)
• RISC-V OS using Rust: Filesystems (HN)
• Sanoid - Policy-driven snapshot management tool for ZFS filesystems. When combined with the Linux KVM hypervisor, you can use it to make your systems functionally immortal.
• UtahFS - Encrypted storage system that provides a user-friendly FUSE drive backed by cloud storage. (Article) (HN)
• Writing a file system from scratch in Rust (2020) (Lobsters) (HN)
• Filesystem deduplication is a sidechannel (Lobsters)
• Files by Dan Luu (2019)
• Non-POSIX file systems (2020) (Lobsters) (HN)
• Filesystems for dummies (2020)
• gcsfuse - User-space file system for interacting with Google Cloud Storage.
• Apache Helix - Near-Realtime Rsync Replicated File System. (HN)
• The Second Extended File System - Also known as Ext2.
• bindfs - FUSE filesystem for mirroring a directory to another directory, similarly to mount --bind. (Web)
• Practical File System Design (1999)
• Virtual filesystem: fun and profit (2019)
• Files Are Fraught with Peril (2019) (HN)
• An Introduction to ZFS (2020) (HN)
• vmtouch - Portable file system cache diagnostics and control. (Code)
• Modern storage is plenty fast. It is the APIs that are bad (2020) (HN)
• Supertag - Tag-based filesystem, written in Rust, for Linux and MacOS. (Docs) (Lobsters)
• DwarFS - Fast high compression read-only file system. (HN)
• ID mapping for mounted filesystems
• Filesystem and Database are not cutting the problem space right (2020) (Lobsters)
• The Perils of File Typing (HN)
• vfs - Virtual filesystem library written in Go.
• dino - Distributed FUSE filesystem for small volumes.
• Exploring the IBM Graphical Filesystem (Lobsters)
• organize - File management automation tool.
• JuiceFS - Distributed POSIX file system built on top of Redis and S3.
• GoDFS - Simple Distributed FileSystem written in Go. (Article)
• TiFS - Distributed file system based on TiKV.
• HyperTag - Helps humans intuitively express how they think about their files using tags and machine learning.
• sandboxfs - Virtual file system for sandboxing.
• GitHub File System - With this filesystem you can use GitHub from your favorite shell and text editor.
• Dirhash Standard - Formal procedure for hashing of a filesystem directory.
• The No-Order File System (2012) (HN)
• PiFS - Data-Free Filesystem.
• S3QL - File system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack.
• Envfs - Fuse filesystem that returns symlinks to executables based on the PATH of the requesting process.
• GotenksFS - File system on top of your file system.
• Writing a file system from scratch in Rust (2020)
• Query-FS - Virtual filesystem implemented using FUSE in Common Lisp. (Talk)
• CrossFS - Direct-access cross-layered file system with support for concurrency.
• A Survey of Distributed File Systems (1989)
• libfsrefs - Library and tools to access the Resilient File System (ReFS).
• netfuse - FUSE-based abstraction for networked filesystems.
• go-fsimpl - Go filesystem implementations for various URL schemes.
• ImpFS - Novel filesystem with some interesting features.
• rpg-cli - Your filesystem as a dungeon.
• Log-structured File Systems
• nullfsvfs - Virtual black hole file system that behaves like /dev/null.
• ffs - File filesystem: mount semi-structured data (like JSON) as a Unix filesystem. (Web) (Tweet)
• fbox - Easy to use and maintain distributed filesystem written in Go.
• Zip – How not to design a file format (2021) (HN) (Lobsters)
• Facebook's Tectonic Filesystem: Efficiency from Exascale (2021)
• AppFS - Universal packaging format that is presented to the system as a filesystem (via FUSE).
• FastCFS - High performance distributed file system for databases, K8s and VM etc.
• The future needs files (2021) (HN)
• Examining btrfs, Linux’s perpetually half-finished filesystem (2021)
• Goofys - High-performance, POSIX-ish Amazon S3 file system written in Go.
• GeeseFS - High-performance, POSIX-ish S3 (Yandex, Amazon) file system written in Go.
• libzetta-rs - Stable interface for programmatic administration of ZFS from Rust.
• APFS FUSE Driver for Linux
• Fuse-Archive - Program that serves an archive or compressed file (e.g. foo.tar, foo.tar.gz, foo.xz, foo.zip) as a FUSE file system.
• Flatdata - Write-once, read-many, minimal overhead binary structured file format.
• AetherFS - Virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data. (Web)
• ChubaoFS - Cloud native distributed file system and object store.
• The Tree Structure of File Systems (2021) (Lobsters)
• Designing better file organization around tags, not hierarchies (HN)
• We saved millions in SSD costs by upgrading our filesystem (2021) (HN)
• edwork - Decentralized, distributed read-write filesystem.
• I'm “still afraid to use spaces in file names” years old (HN)
• Libnvmmio - User-level file system designed for byte-addressable non-volatile memories.
• rusix - Distributed filesystem in Rust.
• Bcachefs: The COW filesystem for Linux that won't eat your data (Lobsters)
• zectl - ZFS Boot Environment manager for Linux.
• Diskplorer - Disk latency/bandwidth grapher.
• ZboxFS - Zero-details, privacy-focused in-app file system. (Web)
• The fastest way to copy a file? (2021) (HN)
• BetrFS - File system that uses Bε trees. (HN) (Code)
• gocryptfs - Encrypted overlay filesystem written in Go.
• Replacing an unavailable ZFS drive (2021)
• Deark - Utility for file format and metadata analysis, data extraction, decompression, and image format decoding.
• vfile - Small and browser friendly virtual file format that tracks metadata (such as a file’s path and value) and messages.
• vach - Virtual filesystem, specifically made for games.
• pifs2 - Compile-time data-free filesystem.
• Parallel Disk Usage (pdu) - Highly parallelized, blazing fast directory tree analyzer.
• wasi-vfs - Virtual filesystem layer for WASI.
• Littlefs - Little fail-safe filesystem designed for microcontrollers. (HN)
• UnreliableFS - FUSE-based fault injection filesystem.
• The time has come to replace file systems (2022) (HN)
• AnyFS - Simple filesystem which stores metadata using JSON.
• rust-vfs - Virtual filesystem for Rust.
• When and why to deprecate filesystems (2022)
• File systems can experience at least three different sorts of errors (2022) (HN)
• zchunk - Compressed file format that splits the file into independent chunks.
• Pillager - Pillage filesystems for sensitive information with Go.
• WarpDrive Go - Warp across your filesystem in ~5 ms.
• HashFS - Content-addressable file management system.
• The Unexpected Importance of the Trailing Slash (2022) (HN)
• pingfs - File system where the data is stored only in the Internet itself, as ICMP Echo packets (pings) traveling from you to remote servers and back again.
• Merged FS: Compose Multiple Go Filesystems
• composefs - a file system for container images (2021)
• Rust FAT FS - FAT file system library implemented in Rust.
• Spacedrive - Open source cross-platform file manager, powered by a virtual distributed filesystem (VDFS) written in Rust. (HN) (Web)
• CharybdeFS - ScyllaDB fault injection filesystem.
• rmlint - Extremely fast tool to remove duplicates and other lint from your file system.
• memfs - In-memory file system in Rust.
• Catfs - Cache AnyThing file system written in Rust.
• Rust FUSE library for server, virtio-fs and vhost-user-fs
• dbs-fuse - Utilities for tokio/tokio-uring based async IO.
• afs - Abstract File Storage.
• elmerfs - File system that leverage Conflict Free Replicated Data Type CRDT on top of AntidoteDB to be eventually consistent in a active-active geo distributed scenario.
• Ask HN: Why don't file systems and OS's provide file system transactions? (2022)
• Squashfs-Tools - Tools to create and extract Squashfs filesystems.
• Pandoras Box - Intuitive and encrypted in-memory filesystem (VFS).
• Muxfs - Mirroring, checksumming, and self-healing filesystem layer for OpenBSD. (HN)
• Understanding Google’s File System (2020)
• CernVM-File System - Scalable, reliable and low- maintenance software distribution service.
• webfuse - Combines libwebsockets and libfuse. It allows ot attach a remote filesystem via websockets.
• Web Native File System (WNFS) - Distributed file system. It is versioned, logged, programmable, has strong-yet-flexible security, and is fully controlled by the end user.
• Tools for tag-based file management? | Lobsters
• Flexible File system
• polyfuse - FUSE (Filesystem in Userspace) library for Rust.
• Linux NILFS file system: automatic continuous snapshots (2022) (HN)
• filesystem-rs - Real, fake, and mock implementations of file system operations.
• WebNative FileSystem (WNFS) - Versioned content-addressable distributed filesystem with private and public sub systems.
• Webnative SDK - SDK for building apps with decentralized identity and storage.
• Fission - Build the future of web apps at the edge.
• dutree - Tool to analyze file system usage written in Rust.
• shinydir - CLI tool to enforce clean directories by setting rules.
• Shufflecake - Tool for Linux that allows to create multiple hidden volumes on a storage device.
• Alist - File list program that supports multiple storage, powered by Gin and SolidJS.
• scaffdog - Markdown driven scaffolding tool.
• s3-sync - Sync a local filesystem with a S3 bucket.
• MemProcFS - Memory Process File System.
• ntfs Rust crate - Low-level NTFS filesystem library implemented in Rust.
• lfs - Linux utility to get information on filesystems, like df but better.
• Dirdiff - Efficiently compute the differences between two directories.
• Hyperdrive - Secure, real-time distributed file system.
• OpenZFS - Advanced file system and volume manager.
• Apple File System for Linux
• FatFs - Generic FAT Filesystem Module (HN)
• FUSE server implementation for Rust
• lunchbox - Async virtual filesystem interface in Rust.
• OpenZFS - The Final Word in File Systems (Lobsters)
• Simple Scaffold - Generate any set of files in the easiest way possible with simple commands.
• Enigma - Simple cross-platform encrypted file system. (HN)
• FDUPES - Program for identifying duplicate files residing within specified directories.
• FatFs - Generic FAT File System Module.
• vramfs - VRAM based file system for Linux. (HN)
• simplefs - Simple file system for Linux kernel.
• exFAT in pure Rust
• A brief history of APFS (Apple file system) in honour of its fifth birthday (2022) (HN)
• The secret life of fsync (2023) (HN)
• Native File System adapter
• go-kvfile - File format for storing a compressed key/value store to a file as an archive.
• EdenFS Overview
• Bcachefs – A New COW Filesystem (2023) (HN)
• FiDB - File system as database.
• PuzzleFS - Next-generation container filesystem.
• Of course the network can be a filesystem
• Pollex - Tiny polling-based filesystem watcher that tries to be efficient.
• Writing a 9P server from scratch
• protomask - User space NAT64 implementation.
• Yazi - Blazing fast terminal file manager written in Rust, based on async I/O.
• ZFS for Dummies (2023) (HN)
• GCSF - FUSE file system based on Google Drive.