Garage: The Lightweight S3 Alternative

Most object stores are scaled-down versions of software designed for data centres. They assume fast, reliable links between nodes, homogeneous hardware, and an operator who thinks a gigabyte of RAM is a rounding error. Then you run one on three second-hand mini PCs connected by a domestic VPN, and it spends its life either complaining or lying about its health.

Garage started from the opposite assumption. It comes from Deuxfleurs, a French association running self-hosted infrastructure on donated hardware in people’s flats, and its design brief was explicitly: cheap machines, unreliable domestic connections, geographically scattered, run by volunteers. That brief produces different software, and if your homelab looks anything like that description — and mine does — it is worth understanding what falls out of it.

What the design assumptions buy you

It is small. Garage is a single Rust binary of about 100 MB with no external dependencies: no separate metadata database, no ZooKeeper, no etcd. A node idles at roughly 100–200 MB of RAM. My smallest node is an ARM board doing other jobs and it does not notice Garage is there. This matters more than benchmarks: a store that fits comfortably alongside your other services can live on machines you already own.

It assumes the network is bad. Garage’s internal replication is designed around the reality that a node might be on a domestic uplink with 40 ms of latency and an unhelpful NAT. It does not fall over when a node is briefly unreachable, and it does not require you to think hard about consistency during a partition, because it settles for eventual consistency and says so up front.

It knows where your nodes are. This is the genuinely clever bit. Garage has a first-class concept of a zone, and its replication is zone-aware: with three replicas across three zones, it will place one copy in each. Tell it that node A is in your flat, node B is at your sister’s house, and node C is at the office, and it will guarantee those copies land in different buildings. That turns “off-site backup” from a scheduled job into a property of the storage layer, and it is close to the whole reason I run it.

It tolerates mismatched hardware. Nodes are given a capacity weight, and the placement algorithm respects it. A 500 GB mini PC and a 4 TB NAS coexist without the store pretending they are equals or refusing to start.

It is AGPL, and the community edition is the whole product. No features held back for a paid tier. Given the licensing discussion in MinIO and the S3-compatible homelab, this is a meaningful part of the appeal for a lot of people.

What you give up

Being honest about this is the point of the article.

S3 compatibility is partial. Garage implements the core: buckets, objects, multipart, presigned URLs, and the parts of the API that real clients use. It does not implement the long tail — object locking and retention, most lifecycle rules, server-side encryption in the AWS sense, ACLs, and various administrative surfaces. If you read the MinIO piece and got excited about a GOVERNANCE-locked delete-proof backup bucket, Garage will not give you that. Check your specific client against the compatibility list before you commit, because “S3-compatible” is a spectrum rather than a boolean.

Performance is adequate. Garage will saturate a gigabit link on large objects and does fine on the workloads a homelab has. It is slower than MinIO on raw throughput, and it makes no attempt to compete on small-object rates. For backups, media and artefacts, “adequate” is the correct amount.

The IAM model is simpler. Keys with per-bucket read/write/owner permissions, rather than full policy documents. This is honestly a relief to configure and a real limitation if you wanted fine-grained action-level control.

Eventual consistency is real. Write an object and immediately list the bucket from another node, and you may not see it yet. Any client assuming read-after-write across nodes will occasionally surprise you. Restic and rclone are fine. Something clever might not be.

Running it across three houses

The configuration file is short enough to read in full, which after the last few years of YAML is a small joy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

# /etc/garage.toml
metadata_dir = "/var/lib/garage/meta"
data_dir     = "/var/lib/garage/data"
db_engine    = "lmdb"

replication_factor = 3
consistency_mode   = "consistent"

rpc_bind_addr   = "[::]:3901"
rpc_public_addr = "10.0.0.11:3901"
rpc_secret      = "generate-with-openssl-rand-hex-32"

[s3_api]
s3_region  = "home"
api_bind_addr = "[::]:3900"
root_domain   = ".s3.example.com"

[s3_web]
bind_addr   = "[::]:3902"
root_domain = ".web.example.com"
index       = "index.html"

rpc_public_addr should be each node’s address on your overlay network. Garage’s inter-node traffic is authenticated by the shared rpc_secret and encrypted, and it is still the right instinct to keep it off the public internet — a mesh VPN is the natural substrate here, and the reasoning is the same as in Tailscale: a zero-config mesh VPN. Nodes at three sites on a mesh see each other as though they were on one flat network, which is exactly the topology Garage wants.

Then the layout, which is where the zone concept becomes concrete:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# On each node, get its ID
garage node id

# Connect them (from any node)
garage node connect [email protected]:3901
garage node connect [email protected]:3901

# Assign each node a zone and a capacity
garage layout assign 6b3f21 -z flat    -c 1000G -t node-flat
garage layout assign 8c1d3f -z sister  -c 500G  -t node-sister
garage layout assign 2a94b7 -z office  -c 2000G -t node-office

garage layout show      # read this before applying it
garage layout apply --version 1

With replication_factor = 3 and three zones, every object now exists in all three buildings. A flat fire, a burglary or a sister moving house each costs you one replica out of three. That is a genuine 3-2-1 posture achieved by the storage layer rather than by a cron job, and it is worth comparing against the effort in the 3-2-1 backup rule actually implemented.

Buckets and keys are similarly terse:

1
2
3
4
5
6
7

garage bucket create backups
garage key create restic-svc
# -> Key ID: GK3168...  Secret: 9f2c...

garage bucket allow --read --write backups --key restic-svc
garage bucket website --allow media      # static hosting, if you want it
garage bucket info backups

Point Restic at it with the standard variables and it neither knows nor cares that it is talking to a Rust binary on a board in a cupboard:

1
2
3
4

export RESTIC_REPOSITORY="s3:https://backups.s3.example.com"
export AWS_ACCESS_KEY_ID="GK3168..."
export AWS_SECRET_ACCESS_KEY="9f2c..."
restic init && restic backup /srv

Sizing and hardware notes

A few practical numbers from running this on hardware nobody would call a server.

Metadata lives separately from data, and it wants to be fast. The metadata_dir holds the LMDB database tracking every object; the data_dir holds the blocks. Metadata is small — comfortably under a gigabyte for a few hundred thousand objects — and it is touched on every single operation. Put it on SSD and the bulk data wherever you like. Putting metadata on a spinning disk is the single easiest way to make Garage feel slow, and it is the mistake I made first.

The database engine matters. LMDB is the current default and the right choice. Older deployments used Sled, which had a reputation for memory growth and occasional corruption on unclean shutdown; if you are reading an old guide that configures Sled, ignore it.

Disk failure is not handled for you. Garage replicates across nodes and does nothing about a drive dying inside one. If a node’s data directory disappears, that node is empty and the cluster repairs from the other replicas, which is fine and which means a failure costs you a full resync over your uplink. On a node with several disks, put a real filesystem underneath and let it deal with drives.

Plan for 3× and then some. With replication_factor = 3, 1 TB of logical data occupies 3 TB of raw across the cluster, plus block overhead. The capacity you declare in garage layout assign should be comfortably under the actual free space, because a full node is an unhappy node and rebalancing needs headroom to work in.

The consistency knob

consistency_mode deserves a paragraph. consistent gives read-after-write within a zone and quorum writes, which costs latency when a remote node is slow. degraded relaxes reads. dangerous allows writes with a single replica and is exactly as advisable as its name suggests. Leave it at consistent and accept that a write waits for a quorum, which on a domestic uplink means your uploads run at the speed of your second-slowest site. That is the price of the copies being real.

Living with it

Two habits have made the difference between Garage being infrastructure and being a project.

Watch the repair queue. garage stats reports how much data is queued for resync. After a node has been offline, that number climbs and then drains, and the rate at which it drains tells you honestly what your inter-site link is worth. A queue that never reaches zero means a site cannot keep up with your write rate, which is a capacity planning fact you want to learn from a counter rather than from a lost object.

1
2
3

garage stats -a              # cluster-wide, all nodes
garage worker list           # what each node is currently chewing on
garage repair --yes blocks   # force a block-level consistency pass

Verify restores from the remote site specifically. The whole premise is that a copy exists at your sister’s house. Test that premise: shut down the local node, run a restic restore of something small, confirm it comes back over the wire. The first time I tried this it failed, because the office node had been reporting healthy for a fortnight while its data directory sat on a disk that had silently remounted read-only. The cluster was serving three replicas’ worth of confidence from two replicas’ worth of disks.

That is the general lesson about geographic redundancy: the storage layer will tell you it has three copies, and only a restore proves it. Put it in the calendar twice a year.

Troubleshooting

Nodes connect and the layout never converges. Check garage status on each. Almost always asymmetric reachability: node A reaches B, B cannot reach A, because rpc_public_addr is set to a LAN address that means nothing at the other site. On a mesh VPN, use the mesh address everywhere.

garage layout apply rejects the version number. The layout is versioned and you must apply the version shown by garage layout show. This is deliberate friction — a layout change moves data around — and the error message is more brusque than helpful.

Writes hang or time out during a site outage. Expected in consistent mode with a quorum unavailable. Three zones tolerate one failure; two zones tolerate none, because two nodes cannot form a quorum of three. If you are building with two sites, set replication_factor = 2 and understand you have traded away the guarantee.

A client fails with an unimplemented operation. Read the error and check compatibility. Common culprits: lifecycle rules, object lock, and clients that insist on the older path-style versus virtual-host addressing. The root_domain setting controls the latter and needs a wildcard DNS record pointing at the node.

Disk usage exceeds what you stored. Three replicas means 3× raw. This is obvious written down and routinely catches people who sized their disks against logical data. Garage has no erasure coding; replication is the whole model, and the storage overhead is the honest cost of the simplicity.

Rebalancing after adding a node saturates the uplink. Adding a fourth node moves a lot of data. Do it on a weekend, and check that your ISP’s fair-use policy has opinions you can live with.

The verdict

Garage is the right answer for a specific and increasingly common shape: several small machines, in different buildings, connected by domestic links, run by one person who wants geographic redundancy without a cron job pretending to provide it. For that job it is better than MinIO, and the zone-aware placement is the reason.

It is the wrong answer if you need the full S3 surface — object lock especially — if you have one machine and want maximum throughput, or if your clients are exotic. On a single node, MinIO’s wider compatibility and better performance win, and the licensing concern that pushes people away is largely theoretical at home.

My own arrangement runs both, which sounds indecisive and isn’t. MinIO handles the fast local bucket that Loki and the registry hammer all day. Garage holds the things I would cry about losing, spread across three buildings, and asks for 200 MB of RAM to do it. That division has survived two years and one house move without me thinking about it, which is the highest compliment available.