Configure mTLS between APISIX and etcd

Mutual TLS (mTLS) is a two-way TLS where client and the server authenticate each other. It is typically implemented to prevent unauthorized access and harden security.

This document will show you how to configure mTLS between APISIX and etcd using a dockerized APISIX and etcd. Adjust the steps accordingly for different deployment approaches.

Prerequisite(s)

• Install Docker.
• Install cURL to send requests to the services for validation.

Generate Certificates and Keys

Create a new directory and navigate into it:

mkdir mtls-apisix-etcd && cd mtls-apisix-etcd

Generate the Certificate Authority (CA) key and certificate:

openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 36500 -sha256 \
  -key ca.key \
  -out ca.crt \
  -subj "/CN=ROOTCA" \
  -extensions v3_ca \
  -config <(printf "[req]\ndistinguished_name=req\n[ v3_ca ]\nbasicConstraints=critical,CA:TRUE\nkeyUsage=critical,keyCertSign,cRLSign\nsubjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid:always,issuer")

Generate the key and certificate signing request (CSR) for etcd:

openssl genrsa -out etcd.key 2048
openssl req -new -sha256 \
  -key etcd.key \
  -out etcd.csr \
  -subj "/CN=ETCD" \
  -addext "subjectAltName=DNS:ETCD,DNS:localhost,IP:127.0.0.1"

Sign the server CSR with the CA certificate to generate the server certificate:

openssl x509 -req -days 36500 -sha256 \
  -in etcd.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out etcd.crt \
  -extensions v3_req \
  -extfile <(printf "[v3_req]\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth,clientAuth\nsubjectAltName=DNS:ETCD,DNS:localhost,IP:127.0.0.1")

Generate the key and certificate signing request (CSR) for the client:

openssl genrsa -out client.key 2048
openssl req -new -sha256 \
  -key client.key \
  -out client.csr \
  -subj "/CN=CLIENT"

Sign the client CSR with the CA certificate to generate the client certificate:

openssl x509 -req -days 36500 -sha256 \
  -in client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt \
  -extensions v3_req \
  -extfile <(printf "[v3_req]\nbasicConstraints=CA:FALSE\nkeyUsage=digitalSignature,keyEncipherment\nextendedKeyUsage=clientAuth")

Allow read access for the files in the directories to avoid downstream permission issues:

chmod -R a+r ./

Copy certificates and keys into /opt/mtls-apisix-etcd (or directory of your choice):

mkdir /opt/mtls-apisix-etcd
cp ca.crt etcd.key etcd.crt client.key client.crt /opt/mtls-apisix-etcd

Configure mTLS

You will be starting etcd and APISIX in Docker containers with TLS certificates and keys to enable mTLS communication between the two.

Start etcd

Start an etcd server in Docker:

docker run -d \
  --name etcd-prod \
  --network host \
  -v /opt/mtls-apisix-etcd:/opt/bitnami/etcd/certs \
  -e ALLOW_NONE_AUTHENTICATION=yes \
  -e ETCD_ADVERTISE_CLIENT_URLS=https://ETCD:2379 \
  -e ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379 \
  -e ETCD_CLIENT_CERT_AUTH=true \
  -e ETCD_CERT_FILE=/opt/bitnami/etcd/certs/etcd.crt \
  -e ETCD_KEY_FILE=/opt/bitnami/etcd/certs/etcd.key \
  -e ETCD_TRUSTED_CA_FILE=/opt/bitnami/etcd/certs/ca.crt \
  bitnamilegacy/etcd:3.5.7

❶ Enable client certificate authentication.

❷ Set the path to the etcd TLS certificate.

❸ Set the path to the etcd TLS key.

❹ Set the path to the trusted CA certificate.

To verify etcd is up and mTLS is properly configured, send a request to etcd with the client certificate and key to get the etcd version:

curl -ikv --resolve "ETCD:2379:127.0.0.1" "https://ETCD:2379/version" \
  --cert /opt/mtls-apisix-etcd/client.crt --key /opt/mtls-apisix-etcd/client.key

If everything is okay, you should see a HTTP/2 200 response with the version of etcd:

{"etcdserver":"3.5.7","etcdcluster":"3.5.0"}

Verify the etcd v3 API with the same client certificate and key:

curl -ikv --resolve "ETCD:2379:127.0.0.1" "https://ETCD:2379/v3/kv/put" \
  --cert /opt/mtls-apisix-etcd/client.crt \
  --key /opt/mtls-apisix-etcd/client.key \
  -H 'Content-Type: application/json' \
  -d '{"key":"a2V5","value":"dmFsdWU="}'

If everything is okay, you should see a HTTP/2 200 response similar to:

{"header":{"cluster_id":"...","member_id":"...","revision":"...","raft_term":"..."}}

Start APISIX

Create an APISIX configuration file in /opt (or directory of your choice):

echo 'apisix:
  ssl:
    ssl_trusted_certificate: /usr/local/apisix/certs/ca.crt
deployment:
  role: traditional
  role_traditional:
    config_provider: etcd
  admin:
    admin_key_required: true
    admin_key:
      -
        name: admin
        key: Sup3rs3cretWr1teK3y   # replace with your write key
        role: admin
  etcd:
    host:
      - "https://ETCD:2379"
    tls:
      cert: /usr/local/apisix/certs/client.crt
      key: /usr/local/apisix/certs/client.key
' > /opt/config.yaml

❶ Set the path to the trusted CA certificate in the Docker container.

❷ Set the path to the client TLS certificate in the Docker container.

❸ Set the path to the client TLS key in the Docker container.

Start an APISIX instance in Docker:

docker run -d \
  --name apisix \
  --network host \
  --add-host ETCD:127.0.0.1 \
  -v /opt/mtls-apisix-etcd:/usr/local/apisix/certs \
  -v /opt/config.yaml:/usr/local/apisix/conf/config.yaml \
  apache/apisix:3.16.0-ubuntu

❶ Mount the TLS certificate and key directory from the host to the Docker container.

❷ Mount the APISIX configuration file from the host to the Docker container.

Verify mTLS

To verify APISIX is up and mTLS is properly configured between APISIX and etcd, send a request to APISIX to get all routes:

curl -i "http://127.0.0.1:9180/apisix/admin/routes" -H "X-API-KEY: Sup3rs3cretWr1teK3y"

If everything is okay, you should see a HTTP/1.1 200 OK response with all the APISIX routes, such as:

{"list":[],"total":0}