Back Up and Restore etcd

etcd clusters are generally fault-tolerant. Still, there might be scenarios where multiple etcd nodes may fail and lose access to the cluster. To recover from such failures, etcd supports backup and restore features for recreating instances without data loss.

Setting up backups can help store the state of an APISIX instance. This is also useful while upgrading APISIX to ensure your configuration is saved.

This document shows how to back up and restore etcd with both APISIX and etcd running in Docker. Replace the container names, network names, and paths in the examples with values from your own deployment.

Prerequisites

• Complete the Get APISIX step to install APISIX.
• Complete the Configure Routes guide to create routes.

Back Up through a Temporary Container

etcd comes packaged with etcdctl, a CLI for working with the etcd APIs. etcdctl supports the etcdctl snapshot command, which can take a backup of the etcd database.

The example below runs this command on a temporary Docker container and creates a backup in the host machine.

First, create a folder, backup, to store the backup in the host machine. This folder is mounted as a volume to the temporary container.

Create the temporary container for backing up etcd in the same network as the APISIX and etcd containers and run the etcdctl snapshot command providing the etcd endpoints:

export NET_NAME="apisix-quickstart-net"
export ETCD_LISTEN_PORT=2379
export ETCD_NAME="etcd-quickstart"

docker run --rm \
    -v ./backup:/backup \
    -e ETCDCTL_API=3 \
    --network "$NET_NAME" \
    bitnamilegacy/etcd:3.5.7 \
    etcdctl --endpoints="http://$ETCD_NAME:$ETCD_LISTEN_PORT" snapshot save /backup/snapshot.db

If successful, you should see a response similar to the following:

{"level":"info","ts":"2024-03-26T06:14:17.173Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/backup/snapshot.db.part"}
{"level":"info","ts":"2024-03-26T06:14:17.178Z","logger":"client","caller":"v3@v3.5.7/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2024-03-26T06:14:17.178Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"http://etcd-quickstart:2379"}
{"level":"info","ts":"2024-03-26T06:14:17.182Z","logger":"client","caller":"v3@v3.5.7/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2024-03-26T06:14:17.182Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"http://etcd-quickstart:2379","size":"25 kB","took":"now"}
{"level":"info","ts":"2024-03-26T06:14:17.183Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/backup/snapshot.db"}
Snapshot saved at /backup/snapshot.db

Back Up Directly on the etcd Container

etcdctl snapshot can also be run directly on the etcd container. Note that this might require setting up directory permissions for the Docker container.

export ETCD_NAME="etcd-quickstart"
docker exec -it "$ETCD_NAME" \
  etcdctl --endpoints="http://127.0.0.1:2379" snapshot save /bitnami/etcd/snapshot.db
docker cp "$ETCD_NAME:/bitnami/etcd/snapshot.db" ./backup/snapshot.db

This saves the snapshot inside the etcd container first, then copies it to the host machine.

Back Up from the Host Machine

etcdctl can be run on the host machine if the etcd API endpoints are accessible. etcd endpoints can be exposed securely through TLS.

Note that etcdctl should be installed on the host machine. Download the binaries for the appropriate operating system or use a package manager to install etcdctl.

Run etcdctl on the host machine to save the snapshot:

export ETCD_LISTEN_PORT=2379
export ETCD_ADDRESS="127.0.0.1"
etcdctl --endpoints="http://$ETCD_ADDRESS:$ETCD_LISTEN_PORT" snapshot save ./backup/snapshot.db

Restore from Backups

Restoring a snapshot recreates the etcd data directory from the snapshot. You should restore into a new directory and then start etcd with the restored data according to your deployment method.

To restore a snapshot into a new data directory, run:

mkdir -p ./restore
docker run --rm \
    -v ./backup:/backup \
    -v ./restore:/restore \
    -e ETCDCTL_API=3 \
    bitnamilegacy/etcd:3.5.7 \
    etcdctl snapshot restore /backup/snapshot.db --data-dir /restore/data

If etcdctl is installed on the host machine, you can also run the restore command directly on the host.

After restore, start etcd with the restored data directory and the bootstrap configuration required by your environment. For example, in a containerized deployment you would typically replace the old etcd container or its mounted data volume with the restored data before starting etcd again.

Once etcd is healthy, make sure APISIX reconnects to the restored datastore. If APISIX does not recover automatically after etcd is available again, restart or reload APISIX according to your deployment method.

For Docker deployments, you can restart APISIX with:

docker restart apisix-quickstart

After restore, verify both the control plane data and the proxied traffic before considering the recovery complete.

For example, if you had previously created a route with the ID route-id, you can verify it through the Admin API and then verify the traffic path:

curl -i "http://127.0.0.1:9180/apisix/admin/routes/route-id"
curl -i "http://127.0.0.1:9080/your-route-path"

For multi-node etcd clusters or non-Docker deployments, use the same snapshot and restore concepts, but adapt the restored data directory, bootstrap configuration, and service restart process to your runtime environment. For cluster-specific recovery guidance, see the official etcd disaster recovery documentation.