setup zerofs for max performance

[adammakowskidev]

Hi @Barre
First of all, I’d like to congratulate you and thank you for your project, it looks very promising and has a lot of potential!

I would like to use ZeroFS as cache storage for Nginx caching files.
Performance is very important to me, especially when reading files, although write performance is also important.
I want migrate from JuiceFS

Could you give me some advice on how to get the best possible performance?
I'm currently using a 9P File Server. Does 9P offer the best performance, or would ZFS be better?

Here is my configuration and the rest of the settings:

[cache]
dir = "/zerofs" # nvme disk location
disk_size_gb = 25.0
memory_size_gb = 5.0

[storage]
url = "s3://de-fra-1/zerofs-data"
encryption_password = "111x111x111"

[servers.ninep]
# addresses = ["127.0.0.1:5564"]
unix_socket = "/tmp/zerofs.9p.sock"

[servers.nbd]
addresses = ["127.0.0.1:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"

[servers.rpc]
# addresses = ["127.0.0.1:7000"]
unix_socket = "/tmp/zerofs.rpc.sock"

# [servers.webui]
# addresses = ["0.0.0.0:8888"]
# uid = 1000
# gid = 1000

[aws]
access_key_id = "xxx"
secret_access_key = "xxx"
endpoint = "https://eu-central-2.storage.impossibleapi.net"
default_region = "eu-central-2"
allow_http = "true"
conditional_put = "redis://127.0.0.1:6379"

# [filesystem]
# max_size_gb = 100.0   # Limit filesystem to 100 GB
# compression = "lz4"  # or "zstd-3", "zstd-19", etc.

[lsm]
l0_max_ssts = 32
max_unflushed_gb = 1.0
max_concurrent_compactions = 16
flush_interval_secs = 60
wal_enabled = true

[wal]
url = "file:///zerofs-wal"

# [prometheus]
# addresses = ["127.0.0.1:9091"]

[telemetry]
enabled = true

Mount fstab config
/tmp/zerofs.9p.sock /mnt/9p 9p trans=unix,version=9p2000.L,cache=loose,noauto,nofail 0 0

I'd appreciate any advice!

[Barre]

Hi,

Thanks for the kind words, really appreciate it.

Your configuration looks fine as-is, with a couple of minor suggestions:

• If your goal is to improve read performance, you probably want to reduce l0_max_ssts to 8 or 16.
• From your use case description, it seems you could disable the WAL altogether.

As you already mount with cache=loose, you can also try cache=fscache (more details).

[adammakowskidev]

@Barre
Thank you for your advice; I'll try to test everything today.
I need a simple, reliable storage benchmark—would sysbench work?

Example:
sysbench fileio --file-test-mode=rndrd --file-total-size=10G --file-block-size=4K --threads=4 --time=60 run
sysbench fileio --file-test-mode=seqwr --file-total-size=10G --file-block-size=4K --threads=4 --time=60 run

[Barre]

I'd use much more threads than 4 if your system can handle it.

[adammakowskidev]

@Barre, here are the tests benchmark results for larger files.

sysbench fileio --file-total-size=1G prepare - 8192kb size every file

sysbench fileio --file-test-mode=rndrd --file-total-size=1G --file-block-size=4K --threads=2 --time=10 run

Running the test with following options:
Number of threads: 2
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 8MiB each
1GiB total file size
Block size 4KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Initializing worker threads...

Threads started!


File operations:
    reads/s:                      966128.90
    writes/s:                     0.00
    fsyncs/s:                     0.00

Throughput:
    read, MiB/s:                  3773.94
    written, MiB/s:               0.00

General statistics:
    total time:                          10.0001s
    total number of events:              9662686

Latency (ms):
         min:                                    0.00
         avg:                                    0.00
         max:                                    4.61
         95th percentile:                        0.00
         sum:                                17298.18

Threads fairness:
    events (avg/stddev):           4831343.0000/14775.00
    execution time (avg/stddev):   8.6491/0.02

sysbench fileio --file-test-mode=seqwr --file-total-size=1G --file-block-size=4K --threads=2 --time=10 run

Running the test with following options:
Number of threads: 2
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 8MiB each
1GiB total file size
Block size 4KiB
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential write (creation) test
Initializing worker threads...

Threads started!


File operations:
    reads/s:                      0.00
    writes/s:                     5174.54
    fsyncs/s:                     6639.99

Throughput:
    read, MiB/s:                  0.00
    written, MiB/s:               20.21

General statistics:
    total time:                          10.0094s
    total number of events:              118014

Latency (ms):
         min:                                    0.03
         avg:                                    0.17
         max:                                   21.11
         95th percentile:                        0.22
         sum:                                19932.39

Threads fairness:
    events (avg/stddev):           59007.0000/961.00
    execution time (avg/stddev):   9.9662/0.00

[adammakowskidev]

And here are the tests benchmark results for smaller files.

sysbench fileio --file-total-size=10M prepare - 80kb size every file

sysbench fileio --file-test-mode=rndrd --file-total-size=10M --file-block-size=4K --threads=2 --time=10 run

Running the test with following options:
Number of threads: 2
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 80KiB each
10MiB total file size
Block size 4KiB
Number of IO requests: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Initializing worker threads...

Threads started!


File operations:
    reads/s:                      1350476.71
    writes/s:                     0.00
    fsyncs/s:                     0.00

Throughput:
    read, MiB/s:                  5275.30
    written, MiB/s:               0.00

General statistics:
    total time:                          10.0001s
    total number of events:              13506696

Latency (ms):
         min:                                    0.00
         avg:                                    0.00
         max:                                   14.66
         95th percentile:                        0.00
         sum:                                15995.29

Threads fairness:
    events (avg/stddev):           6753348.0000/13062.00
    execution time (avg/stddev):   7.9976/0.00

sysbench fileio --file-test-mode=seqwr --file-total-size=10M --file-block-size=4K --threads=2 --time=10 run

Running the test with following options:
Number of threads: 2
Initializing random number generator from current time


Extra file open flags: (none)
128 files, 80KiB each
10MiB total file size
Block size 4KiB
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing sequential write (creation) test
Initializing worker threads...

Threads started!


File operations:
    reads/s:                      0.00
    writes/s:                     3762.79
    fsyncs/s:                     4838.33

Throughput:
    read, MiB/s:                  0.00
    written, MiB/s:               14.70

General statistics:
    total time:                          10.0168s
    total number of events:              85920

Latency (ms):
         min:                                    0.03
         avg:                                    0.23
         max:                                   20.47
         95th percentile:                        0.39
         sum:                                19946.68

Threads fairness:
    events (avg/stddev):           42960.0000/532.00
    execution time (avg/stddev):   9.9733/0.00