roseFS

roseFS is an encrypting, checksumming, copy-on-write filesystem with a FUSE frontend. Core storage (rosefs-core) is #![no_std], synchronous, and journalled; the rosefs-fuse frontend dispatches /dev/fuse requests across a worker pool with per-request buffers and an interior-synchronized namespace.

Three release binaries are produced:

binary	crate	purpose
rosefs-fuse	crates/fuse	FUSE mount daemon (in-memory namespace for v1.0)
rosectl	crates/cli	container / volume / snapshot / keyslot / session / scrub management
rose-recover	crates/recover	image-first logical recovery on degraded-but-readable media

1. Why roseFS

roseFS targets single-node, block-aligned workloads that want cheap snapshots and cryptographic integrity without paying copy-on-write garbage-collection tails. The design compares as follows.

baseline	weakness roseFS addresses	roseFS response
Btrfs	Write amplification under small random overwrites: every 4 KiB in-place rewrite rewrites the whole 64 KiB extent, so a steady overwrite stream can reach WAF ≈ 4-16 on metadata-heavy layouts.	Log-structured, 4 KiB-aligned copy-on-write. A rewrite appends one 4 KiB data block and one metadata update; measured data-path WAF is exactly 1.0000 (see §4).
ext4	No per-extent copy-on-write and no cheap snapshotting; metadata journals amortize poorly for small writes.	B+Tree extent trees with break-on-write refcounting, and O(1) snapshot creation (a single metadata record, ~160 µs; see §4).
F2FS	Checkpoint-based garbage collection pauses writes and spikes write amplification at full utilization.	Background-only compaction gated by a 50% dead-block threshold, never on the write path; the write path is strictly append + metadata commit.
General COW systems	On-disk tree nodes that share ownership across snapshots need full-orphan or epoch GC.	Per-extent reference counts with break-on-write; a snapshot frees blocks when the last snapshot reference drops, verified by crash sweeps.

F2FS on desktop/server hardware

F2FS was architected for battery-backed consumer flash: eMMC/UFS on mobile devices, where the platform can flush cleanly on power events and sudden unbuffered power loss is not the assumed failure model. On desktop and server hardware that assumption does not hold, and the resulting failure modes are documented in upstream post-mortems and on the Arch Linux Wiki F2FS page, which reports the sudden power-loss corruptions (checkpoint/NAT desynchronization, zero-byte files, roll-forward recovery failures) that occur when an unbuffered outage hits a device that is not cleanly flushed. roseFS is deliberately built against the opposite assumption: power may vanish at any instruction, so every durability decision is a commit-first journal decision. The concrete contrasts:

• Sudden power loss: checkpoint desync vs atomic replay. F2FS's recovery relies on re-reading the last valid checkpoint; an unbuffered outage can desynchronize the checkpoint from the node address table (NAT), leaving inconsistent or zero-byte files and, in the worst cases, a panic during roll-forward recovery, as reported for unbuffered sudden power loss on the Arch Linux Wiki. roseFS commits one CRC32c-protected WAL record before any multi-step metadata mutation; after power loss the journal replays atomically up to the last valid CRC32c commit point, and a torn multi-record group replays only its durable prefix. Recovery is measured at 5-35 µs per crash point (≈ 289 µs for a 200-record WAL remount), not a full scan.
• Metadata corruption: single checkpoint vs dual mirrored headers. F2FS keeps one (or two alternating) checkpoint blocks; corruption of the active copy leaves the device unbootable. roseFS mirrors the container header: a primary at sector 0 and a backup at the last container block, each CRC32c-authenticated. A torn or corrupted primary header falls back to the backup automatically (verified end-to-end by rose-recover's used-backup-header: true path).
• Stale block aliasing: roll-forward vs O(1) generation validation. F2FS roll-forward recovery can resurrect stale block pointers that were already freed, aliasing data, a class of corruption the Arch Linux Wiki F2FS page attributes to the roll-forward mechanism after unclean shutdowns. roseFS tags every bucket header with a monotonically increasing 64-bit generation and records the expected generation in each extent pointer; crash recovery validates each pointer against its live bucket header in O(1) time (a header comparison, not a filesystem scan), so a stale pointer can never alias live data.
• Foreground write freezes: synchronous checkpoints vs decoupled compaction. At high utilization F2FS runs checkpoints and garbage collection on the foreground, stalling writes. roseFS's compaction (should_compact) runs only when a bucket's dead-block ratio exceeds 0.50, executes entirely in the background, and is journaled as one atomic commit; it never runs on the write path, so foreground latency does not depend on garbage backlog.

All data and metadata carry CRC32c checksums, and both are authenticated encryption (dual-layer AEAD, see §3), so torn writes, silent media corruption, and ciphertext tampering are all detectable. On-disk structures are written once and immutable (log-structured), which keeps the journal small and makes crash recovery a bounded replay.

2. Architectural trade-offs and limitations

roseFS is honest about what it is and is not.

• Single-node storage. There is no network client, no cluster, no replication across hosts, and no distributed consistency. rosefs-core parallelizes reads per inode, but every write serializes on two global spinlocks (the refcount store and the free-space allocator) and a single WAL commit head. The FUSE frontend parallelizes request dispatch across a worker pool, but no connector can add a second write lane; batching (begin_transaction / commit_transaction) is the scaling mechanism, not parallelism. This is a deliberate simplification for the appliance scope.
• Allocator RAM scales with capacity, not working set. Free space is one flat global bitmap (1 bit per block) held fully in memory: ~32 MiB per TiB, ~512 MiB per 16 TiB. There are no allocation groups or partitioned pools, and every allocation scans from a global cursor while holding the allocator lock. Comfortable to low TBs per container; the operational ceiling is 16-64 TiB (see §3).
• FUSE userspace overhead. Serving from userspace costs two context switches per request plus a copy in and out of the kernel buffer. Small, latency-sensitive I/O (a 4 KiB random read) is measurably slower than the same workload on an in-kernel driver. roseFS targets throughput on block-aligned, largely sequential or overwrite-heavy workloads, where the constant per-request overhead amortizes.
• In-memory v1.0 namespace. The v1.0 rosefs-fuse daemon backs its mount with an in-memory device (4 GiB). When it fills, writes fail cleanly with ENOSPC and existing data is preserved. The full on-disk container format, redundancy modes, and degraded-media recovery are implemented and exercised by rosectl and rose-recover, but the FUSE mount path for v1.0 is the memory device.
• FICLONE is not served. The Linux FUSE kernel does not forward the FICLONE ioctl, so cp --reflink=always returns EOPNOTSUPP (this is a kernel limitation, not a roseFS one). Reflink works through copy_file_range (cp --reflink=auto, SQLite, and other reflink-aware callers), with the same copy-on-write isolation.
• Block-aligned optimization targets. Fast paths assume 4 KiB blocks and B+Tree leaf nodes of 16-256 KiB (default 64 KiB). Sub-block random writes are handled correctly but without the optimization a log-structured layout could give sub-block workloads.
• Best-effort only, in exchange for what it buys. The journal commits group small writes (WAF 3.0 per single op → 1.0370 grouped), the snapshot sweeper and compaction are background, and rose-recover explicitly separates read failures from checksum corruption instead of claiming perfect rescue.

3. On-disk format (reference)

The container format is versioned, checksummed, and cryptographically authenticated. Key constants (crates/core/src/fmt.rs):

quantity	value
block size	4096 B
B+Tree leaf node size	16 KiB ..= 256 KiB, power of two, default 64 KiB
address width	u64 block numbers; 64 ZiB mathematical ceiling at 4 KiB blocks (practical ceiling per §2)
devices per container	geometry-dependent, ≤ 64 (tier default 16)
volumes per container	geometry-dependent, ≤ 1024 (tier default 16-128)
keyslots	≤ 8 (layer-1) + 8 per volume (layer-2)
snapshot index per volume	8-64 B+Tree node slots; one 64 KiB leaf holds ~202 entries, so ~1,400-3,030+ snapshots per volume depending on the formatted budget
redundancy levels	none (linear), mirrored (self-healing), resilient (parity)
integrity	CRC32c over every data block and node payload
data encryption	AES-256-HCTR2 or Adiantum (XChaCha12) per-extent, per-block-within-extent nonces
metadata encryption	AES-256-GCM-SIV (dedicated keyslot key)
free-space tracking	persistent bitmap (1 bit per block), first-fit from a rotating cursor
journal	single active writer, append-only commit records with payload CRC

The container image begins with a mirrored superblock/header pair (so a torn header write is recoverable from the backup), followed by deterministic regions for the keyslot tables, per-volume user keyslots, snapshot nodes, and the persistent free-space bitmap; the remaining blocks form the allocation pool for the root B+Tree and per-inode extent trees. Snapshot IDs are monotonic u64s; snapshot creation appends one descriptor record, which is why it is O(1). A snapshot index that outgrows its reserved node budget fails cleanly instead of spilling into the next volume's region.

Geometry (volumes, snapshot nodes, devices) is locked into the superblock at format time; rosectl container create probes the target capacity and applies a tier unless overridden. Because table LBAs are deterministic functions of that geometry, changing it on an existing container requires an offline migration or a format version bump.

tier	capacity	volumes	snapshot nodes/volume	snapshots/volume	reserved metadata
Compact	< 128 GiB	16	8	~1,400	~16 MiB
Standard	128 GiB - 512 GiB	64	16	~3,030	~68-80 MiB
Pro	> 512 GiB	128	16	~3,030	~128 MiB + bitmap

Sizing: 32-64 MiB minimum image (32 MiB only with minimal 4-volume/4-node settings), 16-64 TiB operational ceiling, 64 ZiB mathematical ceiling. A format-time guard refuses layouts whose reserved metadata exceeds 5% of capacity unless --force is given.

4. Empirical verification

Numbers are reproducible with the harnesses in tests/ (documented in docs/testing.md and docs/benchmarks.md). All benchmarks run the release daemon (debug builds are ~5× slower).

claim	result
Workspace test suite	298 passed, 0 failed, 3 ignored (scale benches), clippy clean with -D warnings
Write amplification (data path)	WAF 1.0000: 64 MiB of 4 KiB unshared random overwrites, exactly 1 device write per logical write (device write counters)
Write amplification (end-to-end, FUSE)	1.0000 for 128 MiB creation and for in-place 4 KiB random rewrite; total ≤ 1.0192 including metadata
Snapshot creation	O(1) ≈ 160 µs at both 1 GiB and 10 GiB namespace sizes
Multi-worker fsstress soak	1 600 000 randomized operations, zero errors: 16 concurrent fsstress processes × 100 k ops (creates, writes, truncates, renames, unlinks, symlinks, copy_file_range reflinks, punches/zeros) against --workers 4; clean unmount/remount, post-soak tree walk + full read-back verified, peak RSS 2.4 GiB ≤ 8 GiB bound
POSIX conformance (pjdfstest)	0 files worse than tmpfs (73 failing FUSE vs 75 tmpfs; failures are the documented non-goals: mknod block/char, setuid, ioctls)
fsx	100 000 randomized ops, A-OK
Crash consistency	exhaustive crash-point sweeps (instant power loss, sector drop, torn writes) resolve to the pre- or post-commit state; WAL replay recovery measured at 5-35 µs per crash point
Reflink COW	copy_file_range 4 MiB reflink verified by device-write counters; isolation after a subsequent overwrite

Two real bugs found and fixed by the soak, with regression tests, are recorded in docs/traceability.md (Phase 8.8): an unbounded engine allocator that could allocate past the device end (now bounded, full-disk writes fail ENOSPC), and a kernel-aborted write_reply that used to tear down the whole session.

5. Usage and tooling

Build

cargo build --release                     # entire workspace
cargo build --release -p rosefs-fuse -p rosectl -p rose-recover  # binaries only

Release binaries land in target/release/. For v1.0 they are stripped; checksums are in target/release/SHA256SUMS (sha256sum -c).

Test

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace                   # 298 green, 0 failed, 3 ignored

Compliance/stress harnesses (pjdfstest, fsx, reflink COW, 1.6 M-op multi-worker fsstress soak) live in tests/compliance/ and are documented in docs/testing.md.

Mount

mkdir -p /tmp/mnt
ROSEFS_FUSE_STATS=1 target/release/rosefs-fuse /tmp/mnt --workers 4
# ... use the filesystem at /tmp/mnt (git, sqlite, cp --reflink, ...) ...
fusermount3 -u /tmp/mnt          # clean unmount; daemon exits

Options: --workers N (request threads, default 1), --io-uring (probe FUSE-over-io_uring; off by default so compliance harnesses use /dev/fuse), --trace. Set ROSEFS_FUSE_STATS=1 to print device write accounting and free blocks on exit (used by the WAF harness).

Container and volume management

rosectl operates on an on-disk container image (file or block device):

# create a container over one or more devices (protection: none|mirrored|resilient)
# geometry auto-tiers from the target capacity; override with --max-volumes,
# --snapshot-nodes, --max-devices (and --force past the 5% metadata guard)
target/release/rosectl container create /path/to/device --protection-level resilient
target/release/rosectl container inspect /path/to/device

# volumes, snapshots, keys, sessions, scrub
target/release/rosectl volume create --help
target/release/rosectl snapshot create/list/delete --help
target/release/rosectl keyslot add/remove --help
target/release/rosectl session load-key/purge-key --help
target/release/rosectl scrub start/status --help

Recovery

rose-recover performs image-first logical recovery of degraded-but-readable media: normal mount (with backup-header fallback) → raw node/bucket scan → per-extent rescue with key → structural report, emitting a gap-marked manifest that distinguishes read failures from checksum corruption.

target/release/rose-recover version                 # tool + on-disk format versions
target/release/rose-recover recover <device-or-image>

Documentation

• docs/spec.md — on-disk format specification.
• docs/traceability.md — requirement-by-requirement traceability (session log, module map, requirement tables, implementation decisions & open flags).
• docs/testing.md — compliance & stress harnesses and results.
• docs/benchmarks.md — fio / reflink / write-amplification results.

License

roseFS is licensed under the Mozilla Public License 2.0 (MPL-2.0). The full license text is in the repository root LICENSE.

All Rust source files carry an SPDX header: // SPDX-License-Identifier: MPL-2.0.

The MPL-2.0 text as distributed omits the optional "Incompatible With Secondary Licenses" notice (Exhibit B), so GPL-2.0 remains an available secondary license under Section 3.3 for combining with MPL-licensed code.

Dependencies are consumed from crates.io and git under their respective licenses; none are vendored into this repository.