Skip to main content
Version: 3.21.0

Configure mTLS between APISIX and etcd

Mutual TLS (mTLS) is a two-way TLS where client and the server authenticate each other. It is typically implemented to prevent unauthorized access and harden security.

This document will show you how to configure mTLS between APISIX and etcd using a dockerized APISIX and etcd. Adjust the steps accordingly for different deployment approaches.

Prerequisite(s)​

  • Install Docker.
  • Install cURL to send requests to the services for validation.

Generate Certificates and Keys​

Create a new directory and navigate into it:

mkdir mtls-apisix-etcd && cd mtls-apisix-etcd

Generate the Certificate Authority (CA) key and certificate:

openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 36500 -sha256 \
-key ca.key \
-out ca.crt \
-subj "/CN=ROOTCA" \
-extensions v3_ca \
-config <(printf "[req]\ndistinguished_name=req\n[ v3_ca ]\nbasicConstraints=critical,CA:TRUE\nkeyUsage=critical,keyCertSign,cRLSign\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid:always,issuer")

Generate the key and certificate signing request (CSR) for etcd:

openssl genrsa -out etcd.key 2048
openssl req -new -sha256 \
-key etcd.key \
-out etcd.csr \
-subj "/CN=ETCD" \
-addext "subjectAltName=DNS:etcd-prod,DNS:localhost,IP:127.0.0.1"

Sign the server CSR with the CA certificate to generate the server certificate:

openssl x509 -req -days 36500 -sha256 \
-in etcd.csr \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-out etcd.crt \
-extensions v3_req \
-extfile <(printf "[v3_req]\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=DNS:etcd-prod,DNS:localhost,IP:127.0.0.1")

Generate the key and certificate signing request (CSR) for the client:

openssl genrsa -out client.key 2048
openssl req -new -sha256 \
-key client.key \
-out client.csr \
-subj "/CN=CLIENT"

Sign the client CSR with the CA certificate to generate the client certificate:

openssl x509 -req -days 36500 -sha256 \
-in client.csr \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-out client.crt \
-extensions v3_req \
-extfile <(printf "[v3_req]\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=clientAuth")

Keep the CA signing key and the original private keys accessible only to their owner. Create a runtime directory whose group matches the pinned APISIX Ubuntu image's apisix user (UID and GID 636), then install only the required files with permissions for their runtime identity:

chmod 600 ca.key etcd.key client.key
sudo install -d -o root -g 636 -m 750 /opt/mtls-apisix-etcd
sudo install -m 644 ca.crt etcd.crt client.crt /opt/mtls-apisix-etcd
sudo install -o root -g root -m 600 etcd.key /opt/mtls-apisix-etcd
sudo install -o 636 -g 636 -m 640 client.key /opt/mtls-apisix-etcd

If you use a different APISIX image or runtime user, replace UID and GID 636 with that identity. The CA signing key is not copied into the runtime directory.

Configure mTLS​

You will be starting etcd and APISIX in Docker containers with TLS certificates and keys to enable mTLS communication between the two.

Create a Docker network for the containers:

docker network create mtls-apisix-etcd-net

Start etcd​

Start an etcd server in Docker:

docker run -d \
--name etcd-prod \
--network mtls-apisix-etcd-net \
-p 2379:2379 \
-v /opt/mtls-apisix-etcd:/certs:ro \
gcr.io/etcd-development/etcd:v3.5.32 \
/usr/local/bin/etcd \
--client-cert-auth \
--cert-file /certs/etcd.crt \
--key-file /certs/etcd.key \
--trusted-ca-file /certs/ca.crt \
--name etcd-prod \
--data-dir /etcd-data \
--listen-client-urls https://0.0.0.0:2379 \
--advertise-client-urls https://etcd-prod:2379 \
--listen-peer-urls http://0.0.0.0:2380 \
--initial-advertise-peer-urls http://etcd-prod:2380 \
--initial-cluster etcd-prod=http://etcd-prod:2380

❶ Enable client certificate authentication.

❷ Set the path to the etcd TLS certificate.

❸ Set the path to the etcd TLS key.

❹ Set the path to the trusted CA certificate.

To verify etcd is up and mTLS is properly configured, send a request to etcd with the client certificate and key to get the etcd version:

curl -iv --resolve "etcd-prod:2379:127.0.0.1" "https://etcd-prod:2379/version" \
--cacert ca.crt \
--cert client.crt --key client.key

If everything is okay, you should see a HTTP/2 200 response with the version of etcd:

{"etcdserver":"3.5.32","etcdcluster":"3.5.0"}

Verify the etcd v3 API with the same client certificate and key:

curl -iv --resolve "etcd-prod:2379:127.0.0.1" "https://etcd-prod:2379/v3/kv/put" \
--cacert ca.crt \
--cert client.crt \
--key client.key \
-H 'Content-Type: application/json' \
-d '{"key":"a2V5","value":"dmFsdWU="}'

If everything is okay, you should see a HTTP/2 200 response similar to:

{"header":{"cluster_id":"...","member_id":"...","revision":"...","raft_term":"..."}}

Start APISIX​

The following configuration permits Admin API requests from any source that can reach the container so that the local example remains runnable. Use it only for local validation. In production, replace 0.0.0.0/0 with the CIDR ranges of trusted operator networks and use a securely managed Admin API key.

Create an APISIX configuration file in /opt (or directory of your choice):

echo 'apisix:
ssl:
ssl_trusted_certificate: /usr/local/apisix/certs/ca.crt
deployment:
role: traditional
role_traditional:
config_provider: etcd
admin:
admin_key_required: true
allow_admin:
- 0.0.0.0/0
admin_key:
-
name: admin
key: Sup3rs3cretWr1teK3y # replace with your write key
role: admin
etcd:
host:
- "https://etcd-prod:2379"
tls:
cert: /usr/local/apisix/certs/client.crt
key: /usr/local/apisix/certs/client.key
' > /opt/config.yaml

❶ Set the path to the trusted CA certificate in the Docker container.

❷ Set the path to the client TLS certificate in the Docker container.

❸ Set the path to the client TLS key in the Docker container.

Start an APISIX instance in Docker:

docker run -d \
--name apisix \
--network mtls-apisix-etcd-net \
-p 127.0.0.1:9180:9180 \
-v /opt/mtls-apisix-etcd:/usr/local/apisix/certs:ro \
-v /opt/config.yaml:/usr/local/apisix/conf/config.yaml:ro \
apache/apisix:3.21.0-ubuntu

❶ Mount the TLS certificate and key directory from the host to the Docker container.

❷ Mount the APISIX configuration file from the host to the Docker container.

The port mapping binds the published Admin API port to the Docker host's loopback address for this local validation.

Verify mTLS​

To verify APISIX is up and mTLS is properly configured between APISIX and etcd, send a request to APISIX to get all routes:

curl -i "http://127.0.0.1:9180/apisix/admin/routes" -H "X-API-KEY: Sup3rs3cretWr1teK3y"

If successful, you should see a HTTP/1.1 200 OK response with all the APISIX routes, such as:

{"list":[],"total":0}