All or at least some of the mongodb backups deserve testing.

One of the ways of ensuring a backup is able to be used in a restore process is to simply join a new ephemeral node to the cluster, run tests on it, and then purge it.

This lab involves:

1. accessing an available backup
2. copying datafiles from backup to a local filesystem
3. spin up a mongodb process that mounts datafiles from backup
4. add ephemeral node to replica set and let it sync
5. run tests on restore node
6. purge restore node

Prerequisites

This lab requires many layers:

Storage, garage

MinioS3 aint dead just because their repository entered maintenance mode, still I started looking options out there.

One of them is garage, please check their work here:

https://garagehq.deuxfleurs.fr/documentation/quick-start/

For this lab you need an s3 compatible storage layer that PBM can talk to.

Here are the steps I took for setting up garage:

Create data directories owned by $USER:

sudo mkdir -p /bkp/data /bkp/meta
sudo chown -R $USER: /bkp

Create config file:

cat > garage.toml <<EOF
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"

replication_factor = 1

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "$(openssl rand -hex 32)"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = ".s3.garage.localhost"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.garage.localhost"
index = "index.html"

[k2v_api]
api_bind_addr = "[::]:3904"

[admin]
api_bind_addr = "[::]:3903"
admin_token = "$(openssl rand -base64 32)"
metrics_token = "$(openssl rand -base64 32)"
EOF

Note that /var/lib/garage may be mounted later on using /bkp directory

Startup container:

Start your podman garaged container:

podman run -d \
  --name garaged \
  --network host \
  --restart always \
  -v ./garage.toml:/etc/garage.toml:ro,Z \
  -v /bkp:/var/lib/garage:Z \
  dxflrs/garage:v2.2.0

Create alias for checking status of garage container:

alias garage="podman exec -ti garaged /garage"
garage status

# ==== HEALTHY NODES ====
# ID                Hostname  Address         Tags  Zone  Capacity          DataAvail  Version
# <NODE_ID>         fedora    127.0.0.1:3901              NO ROLE ASSIGNED             v2.2.0

Take note of the Node ID.

Create partition, bucket and aws keys

Assign a new 20GB partition layout and apply using the Node ID:

garage layout assign -z dc1 -c 20G <NODE_ID>

garage layout apply --version 1

Create PBM bucket and make sure it exists:

garage bucket create pbm

Create an API key using the following command, take note of Keys ID and Secret displayed:

garage key create pbm


# ==== ACCESS KEY INFORMATION ====
# Key ID:              <KEY_ID>
# Key name:            pbm
# Secret key:          very_secret_key
# Created:             2026-02-18 23:46:31.255 +00:00
# Validity:            valid
# Expiration:          never

Make sure key exists:

garage key list

Allow key to access bucket:

garage bucket allow   --read   --write   --owner   pbm                --key pbm

Run aws configure or edit credential environment variables for accessing s3:

aws s3 ls
# 2026-02-18 23:43:29 pbm

Fake data with plgm

I created some fake data with Daniel's great Percona Load Generator for MongoDB. It's a neat project for simulating the most diverse workloads.

After inserting data for 5 minutes, I got a data set with less than 5 GBs of data, which should be enough for this demo.

Backup Tool, PBM:

PBM will be used, it's the only production-grade open source MongoDB backup tool I know.

It does require you to run MongoDB on PSMDB, Percona's MongoDB drop-in replacement.

Please learn more about it here: https://docs.percona.com/percona-backup-mongodb/index.html

If you are using PBM, I assume you have access to a few backups available.

In my case, I access a server with PBM enabled and run pbm status:

[admin@db-1 ~]$ pbm status
Cluster:
========
rs0:
  - db-1:27017 [S]: pbm-agent [v2.12.0] OK
  - db-2:27017 [S]: pbm-agent [v2.12.0] OK
  - db-3:27017 [P]: pbm-agent [v2.12.0] OK


PITR incremental backup:
========================
Status [ON]
Running members: rs0/db-1:27017; 

Currently running:
==================
(none)

Backups:
========
S3 garage http://192.168.0.174:3900/pbm/pbm
  Snapshots:
    2026-03-27T00:02:01Z 669.02MB <physical> success [restore_to_time: 2026-03-27T00:02:03]
    2026-03-24T00:02:01Z 0.00B <physical> failed [ERROR: Backup stuck at `starting` stage, last beat ts: 1774310521] [2026-03-24T00:02:01]
    2026-03-23T00:02:01Z 671.25MB <physical> success [restore_to_time: 2026-03-23T00:02:03]
    2026-03-20T00:02:01Z 667.52MB <physical> success [restore_to_time: 2026-03-20T00:02:03]
    2026-03-19T23:42:54Z 667.35MB <physical> success [restore_to_time: 2026-03-19T23:42:56]
    2026-03-17T00:02:01Z 721.09MB <physical> success [restore_to_time: 2026-03-17T00:02:03]
    2026-03-15T00:28:26Z 151.84MB <physical> success [restore_to_time: 2026-03-15T00:28:29]

Keep in mind:
The key here is to have access to an s3 compatible bucket with mongodb datafiles.

DNS

You need to have DNS resolution across source replica set and target database pods - no matter how. In my case I have my own local libvirt network named 'priv' whose DNS listens under 192.168.100.1 .

# This is how I connect:
alias mongosh="podman run --network host --dns 192.168.100.1 -ti --rm alpine/mongosh mongosh"

Execution

Step 1, build the Podman image

# Build the image
podman build -t psmdb-s2d-aws .

# Check version
podman run --rm psmdb-s2d-aws mongod --version

Step 2, decompress datafiles

List and pick the backup of your choice:


[admin@db-1 ~]$ pbm list
Backup snapshots:
  2026-03-15T00:28:26Z <physical> [restore_to_time: 2026-03-15T00:28:29]
  2026-03-17T00:02:01Z <physical> [restore_to_time: 2026-03-17T00:02:03]
  2026-03-19T23:42:54Z <physical> [restore_to_time: 2026-03-19T23:42:56]
  2026-03-20T00:02:01Z <physical> [restore_to_time: 2026-03-20T00:02:03]
  2026-03-23T00:02:01Z <physical> [restore_to_time: 2026-03-23T00:02:03]
  2026-03-27T00:02:01Z <physical> [restore_to_time: 2026-03-27T00:02:03]

PITR <on>:
  2026-03-15T00:28:30 - 2026-03-27T00:02:29

Create directory for decompressing data from s3:

mkdir /bkp/decompressed

List desired backup directory:

apollo@fedora:/repos/labs$ aws s3 ls s3://pbm/pbm/2026-03-27T00:02:01Z/rs0/
                           PRE admin/
                           PRE airline/
                           PRE config/
                           PRE journal/
                           PRE local/
2026-03-26 21:02:10      12045 WiredTiger.backup.s2
2026-03-26 21:02:10         68 WiredTiger.s2
2026-03-26 21:02:06      22807 WiredTigerHS.wt.s2
2026-03-26 21:02:06       7057 _mdb_catalog.wt.s2
2026-03-26 21:02:10      15045 filelist.pbm
2026-03-26 21:02:10       2293 sizeStorer.wt.s2
2026-03-26 21:02:10        132 storage.bson.s2.0-114

Start file download:

apollo@fedora:/repos/labs$ aws s3 cp s3://pbm/pbm/2026-03-27T00:02:01Z/rs0/ /bkp/decompressed --recursive

Don't forget the keyfile is necessary too:

# Copy from production!
sudo cp /etc/mongod_keyfile .
sudo chown admin. mongod_keyfile 


# Logout and copy file to your target server
logout
Connection to 192.168.100.10 closed.
$ scp admin@192.168.100.10:~/mongod_keyfile .
mongod_keyfile                               100%  130   346.2KB/s   00:00

Step 3, Start temporary node to join replica set

The key is using --dns 192.168.100.1 flag to resolve the libvirt network hostnames:

# Run ephemeral node with DNS pointing to libvirt network DNS
podman run -d     --name podman_restore     --network host     --dns 192.168.100.1     --dns-search default.local     -v /bkp/decompressed:/data/db:Z     -e HOST_IP="192.168.100.99"     -e MONGOD_CONFIG=./mongod.conf     podman_restore

DNS flags explained:
- --dns 192.168.100.1 - Use libvirt's dnsmasq to resolve VM hostnames
- --dns-search default.local - Append domain to bare hostnames

Verify DNS works from inside container:

podman exec -ti psmdb-test bash
ping db-1              # should resolve to 192.168.100.10
ping db-1.default.local # FQDN should also work

Step 4, Add ephemeral node to replica set

From a primary node or using mongosh with DNS:

mongosh "mongodb://db-1:27017/?replicaSet=rs0" --eval '
rs.add({
  host: "<host-ip-or-hostname>:27017",
  priority: 0,
  votes: 0
})
'

Or if ephemeral node runs on different port:

mongosh "mongodb://db-1:27017/?replicaSet=rs0" --eval '
rs.add({
  host: "192.168.100.<new-ip>:27017",
  priority: 0,
  votes: 0
})
'

Step 5, Monitor sync and run tests

# Check replica set status
mongosh "mongodb://db-1:27017/?replicaSet=rs0" --eval 'rs.status()'

# Wait for SECONDARY state, then run tests
mongosh "mongodb://psmdb-test:27017/?replicaSet=rs0" --eval 'db.collection.find().limit(5)'

Step 6, Cleanup ephemeral node

# Remove from replica set
mongosh "mongodb://db-1:27017/?replicaSet=rs0" --eval '
rs.remove("<ephemeral-host>:27017")
'

# Stop and remove container
podman stop psmdb-test
podman rm psmdb-test

# Clean up backup data if desired
rm -rf /path/to/backup/data/*

Test your mongodb backups using podman