SeaweedFS for Absurd Amounts of Small Files

There is a category of storage problem that arrives without warning and that ordinary tools handle badly. It looks like this: a photo library generating five thumbnails per image, a mail store with one file per message, a build cache, a scanned document archive, a tile server. Individually every file is a few kilobytes. Collectively there are tens of millions of them, and your storage — which handles a 40 GB video file without blinking — grinds.

The instinct is to blame the disk and buy a faster one. That helps a little and misses the actual problem, which is structural, and which is worth understanding before you reach for any tool at all.

Why small files are a different problem

Every file on a conventional filesystem carries fixed overhead independent of its size.

Metadata. An inode is typically 256 bytes on ext4, plus directory entries, plus extent maps. Fifty million files means fifty million inodes: something like 12 GB of pure metadata before you store a single byte of content. That metadata is what gets touched on every operation, and once it exceeds RAM your cache hit rate collapses and every stat becomes a disk seek.

Block granularity. A 2 KB file on a filesystem with 4 KB blocks occupies 4 KB. Half your capacity vanishes into slack. On a ZFS dataset with a 128 KB record size and no compression, the same file can occupy far worse.

Directory scaling. Directories with millions of entries are slow to traverse regardless of the hashing scheme, and every backup, every du, every rsync walks all of them.

Per-object protocol cost. On an object store, each object is an HTTP request with signature verification, a metadata lookup, and a round trip. The overhead per request is roughly constant, so at 4 KB objects you are spending nearly all your resources on ceremony. This is the wall you hit with MinIO and the S3-compatible homelab — it is architectural rather than a tuning failure, and no configuration rescues it.

The insight underneath every solution in this space, and the one Facebook published in the Haystack paper that SeaweedFS is built on: stop treating small files as files. Append them into a few large volume files, keep an in-memory map from file ID to (volume, offset, size), and serve reads with exactly one disk seek. Fifty million files becomes a few hundred 30 GB blobs, which every layer of the stack — filesystem, page cache, backup tool — is good at.

What SeaweedFS actually is

The naming is the first hurdle, because SeaweedFS is three layers and people conflate them.

The volume server stores the blobs. It owns the volume files, appends to them, and serves reads by offset. This layer has no concept of filenames or directories — it deals in opaque file IDs like 3,01637037d6.

The master tracks which volume lives on which server, handles replication placement, and hands out new file IDs. Small, lightweight, and it is the thing that must be up for writes.

The filer is optional and is where filenames come from. It maps a POSIX-ish path to a file ID, storing that mapping in a metadata store of your choosing. The filer is what provides the S3 gateway, the WebDAV mount and the FUSE mount.

You can run just master plus volume servers, address everything by file ID, and have an extremely fast blob store with no naming. Most people want the filer, and that decision brings the real complexity, because the filer’s metadata store becomes the thing that determines whether the system is fast.

The alternatives, and when they are enough

Before adding a distributed store, consider the cheaper interventions, because two of them have solved this for me on other occasions.

Tune the filesystem you have. On ext4, mkfs.ext4 -i 4096 -b 1024 for a small-file volume changes the inode ratio and block size and can halve the waste. On ZFS, dropping recordsize to 16 K on that dataset alone and enabling lz4 addresses the slack directly, and a metadata-heavy workload benefits enormously from an SSD holding the metadata. These are free, take an hour, and get you a long way at the ten-million-file scale.

Pack them yourself. If the files are write-once and read-rarely — scanned archives, old logs — a SQLite database with a BLOB column genuinely outperforms a filesystem below about 100 KB per object, and it is one file to back up. This is unglamorous and it works, and it is the answer I would give most people asking this question.

Stop generating them. Covered in the verdict, and it is worth saying early: the best small-file store is the one holding a tenth as many small files.

SeaweedFS earns its place when the count is genuinely in the tens of millions, the access pattern is random reads, and the data is live rather than archival. Below that, the tools above win on effort.

Running it

A working single-machine stack, with the filer backed by a proper database:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# compose.yaml
services:
  master:
    image: chrislusf/seaweedfs:latest
    command: 'master -ip=master -ip.bind=0.0.0.0 -mdir=/data -volumeSizeLimitMB=30000'
    volumes: [ /srv/seaweed/master:/data ]
    restart: unless-stopped

  volume:
    image: chrislusf/seaweedfs:latest
    command: 'volume -mserver=master:9333 -ip=volume -ip.bind=0.0.0.0 -dir=/data -max=200'
    volumes: [ /srv/seaweed/volume:/data ]
    depends_on: [ master ]
    restart: unless-stopped

  filer:
    image: chrislusf/seaweedfs:latest
    command: 'filer -master=master:9333 -ip=filer -ip.bind=0.0.0.0 -s3'
    volumes:
      - /srv/seaweed/filer:/data
      - ./filer.toml:/etc/seaweedfs/filer.toml:ro
    ports:
      - "127.0.0.1:8333:8333"   # S3 API
      - "127.0.0.1:8888:8888"   # filer HTTP
    depends_on: [ master ]
    restart: unless-stopped

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: seaweed_filer
      POSTGRES_USER: seaweed
      POSTGRES_PASSWORD: ${FILER_DB_PASSWORD:?set me}
    volumes: [ /srv/seaweed/pg:/var/lib/postgresql/data ]
    restart: unless-stopped

And the filer’s metadata backend, which is the decision that matters most:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# filer.toml
[leveldb2]
enabled = false

[postgres]
enabled = true
hostname = "postgres"
port = 5432
username = "seaweed"
password = "..."
database = "seaweed_filer"
sslmode = "disable"
connection_max_idle = 20
connection_max_open = 100

The default filer store is an embedded LevelDB, which works and gives you a metadata database with no replication, no backup story, and a bad day if the file is corrupted on an unclean shutdown. Postgres costs you a container and gives you a metadata store you already know how to back up and replicate. Take the container. The tuning advice in Postgres tuning for homelabbers applies directly, and this workload is almost entirely small point lookups, so shared_buffers and a warm cache are what you care about.

Then it speaks S3, and everything you already have keeps working:

1
2
3
4
5
6
7

echo '{"identities":[{"name":"app","credentials":[{"accessKey":"AKIA...","secretKey":"..."}],
  "actions":["Read","Write","List","Tagging"]}]}' > s3.json
# mount it and pass -s3.config=/etc/seaweedfs/s3.json to the filer

mc alias set seaweed http://127.0.0.1:8333 AKIA... ...
mc mb seaweed/thumbnails
mc cp --recursive ./cache/ seaweed/thumbnails/

The numbers that justify it

Some figures from moving a photo-derivative cache of roughly 8 million files, averaging 11 KB, off a ZFS dataset:

• Capacity used: 340 GB on ZFS (default recordsize, no compression) against 96 GB in SeaweedFS volumes. Slack and metadata were eating two thirds of the space.
• rsync walk of the tree: 47 minutes against not applicable — the volume files are a few hundred blobs, and backing them up is a sequential read at disk speed.
• Cold random read: roughly 9 ms against roughly 2 ms, because a read is one seek by design rather than an inode lookup plus an extent lookup plus a data read.
• Metadata RAM: the volume server holds its needle map in memory at about 20 bytes per file, so 8 million files is a 160 MB index. That is the trade — RAM proportional to file count, and it stays modest.

The backup improvement is the one that surprised me and the one I now value most. A tree of 8 million small files is effectively unbackupable with file-level tools; every run spends its life on stat calls. Two hundred 30 GB blobs is a problem Restic and rclone solve in their sleep.

Operating it without surprises

Three things to set up on day one rather than after the incident.

Back up the filer database on its own schedule. The volume files are useless without the path-to-file-ID mapping. A nightly pg_dump of a metadata database measured in gigabytes is trivial, and it is the single most important backup in the system. Treat losing it as equivalent to losing the disks.

Monitor volume fill and the vacuum. The master exposes Prometheus metrics; the two that matter are free volume count and the ratio of deleted to live bytes per volume. An alert when free volumes drop below a handful gives you weeks of warning, and the garbage ratio tells you whether your vacuum schedule is keeping up. Both are one panel each.

Understand -volumeSizeLimitMB before you set it. It caps how large a volume grows before the master stops writing to it and allocates another. The default of 30 GB is a reasonable balance: smaller volumes mean more files to manage and faster compaction, larger ones mean fewer files and a vacuum that needs a lot of free space to run. Changing it later applies to new volumes only, like most decisions in this system.

That last pattern is the theme. SeaweedFS makes a number of choices permanent at creation time — replication code, volume size, filer backend — and is much less forgiving of a half-thought-through start than a single-binary object store. Spend an hour on the layout before you copy 8 million files into it.

Troubleshooting

Writes fail with “No free volumes left”. The master allocates volumes lazily and the volume server has a -max cap. Either raise -max, add a volume server, or check that the disk is not actually full. weed shell then volume.list shows exactly what exists and where.

Deleted files don’t free space. By design. Volumes are append-only and deletion marks a needle as dead; space returns only on compaction. Run it, and watch the disk headroom, because compacting a 30 GB volume needs 30 GB free:

1

echo 'volume.vacuum -garbageThreshold=0.3' | weed shell -master=127.0.0.1:9333

Set a scheduled vacuum and forget it. A store that silently grows for a year because nobody read this paragraph is the most common SeaweedFS complaint.

The filer is slow while volume servers are idle. The metadata store is your bottleneck. This is what LevelDB-on-a-spinning-disk feels like, and it is why Postgres on SSD is the recommendation. Check the filer’s own latency before blaming the storage layer.

A volume server restarts and files 404 briefly. It is rebuilding its needle map from the volume file at startup, which takes time proportional to file count. Large volume counts mean minute-scale startups. Plan restarts accordingly.

Replication isn’t doing what you configured. SeaweedFS’s replication codes (000, 001, 010, 100) mean copies on the same rack, different rack, and different data centre respectively, and they are set per volume at creation. Changing the default afterwards affects only new volumes. Existing volumes keep whatever they were born with, permanently, which is an unwelcome discovery at scale.

Everything works and you cannot find your data. The volume files are opaque and there is no directory tree to ls. If the filer’s metadata database is lost and you have no backup, you have blobs full of your data and no map. Back up Postgres like it is the only copy of your filesystem, because functionally it is.

The verdict

SeaweedFS is a specialist tool and it is very good at its speciality. If you genuinely have millions of small files and are watching your storage strain under metadata rather than bytes, it will fix the problem decisively — I recovered two thirds of a dataset’s space and made an unbackupable tree routine.

The honest case against is operational weight. This is three or four moving parts, a metadata database that is a single point of failure, a compaction chore, and a set of replication semantics you must get right at creation time. Against that, MinIO or Garage is one binary. Below roughly a million files, that complexity buys you nothing measurable, and you should run something simpler and revisit this when the graph tells you to.

The question worth asking first, though, is whether you need the small files at all. My cache of 8 million thumbnails existed because an application generated five derivatives per photo and never cleaned up. Regenerating derivatives on demand with a small cache in front eliminated 90% of them and would have been the better fix, cheaper than any storage layer. SeaweedFS is an excellent answer, and it is worth checking that you are asking the right question.