Upstream mTLS

When an upstream service requires mutual TLS, API7 Gateway can act as a TLS client — presenting a client certificate to the upstream and optionally validating the upstream's server certificate against a trusted CA. Configure upstream mTLS on the service's upstream.tls block: set client_cert and client_key to the certificate the gateway presents, and set verify together with ca_certs to validate the upstream's server certificate.

info

This page covers mTLS between the gateway and upstream services. For mTLS between API clients and the gateway, see Client mTLS Authentication. For mTLS between the Control Plane and Data Plane, see Mutual TLS between CP and DP.

How it works

1. The gateway opens a TLS connection to the upstream using the upstream's host/SNI.
2. The upstream presents its server certificate. If tls.verify is true, the gateway validates it against tls.ca_certs; otherwise the certificate is accepted without verification.
3. The upstream requests a client certificate. The gateway presents the certificate and key configured in tls.client_cert and tls.client_key.
4. The upstream validates the client certificate against its own trusted CA. If validation succeeds, the request is proxied; otherwise the gateway returns HTTP 502.

Prerequisites

Prepare the following PEM-encoded material:

Item	Purpose
Client certificate + private key	Presented by the gateway to the upstream. Must be signed by a CA the upstream trusts.
Upstream CA certificate	Used by the gateway to validate the upstream's server certificate. Required only when tls.verify is true.

The upstream service must terminate TLS itself, request a client certificate, and validate it against its own trusted CA. The examples below use an nginx instance with the following configuration:

upstream-nginx.conf

events {}
http {
  server {
    listen 8443 ssl;
    server_name upstream.test.local;
    ssl_certificate /certs/upstream.crt;
    ssl_certificate_key /certs/upstream.key;
    ssl_client_certificate /certs/ca.crt;
    ssl_verify_client on;
    location / {
      return 200 "upstream-mtls-ok client=$ssl_client_s_dn\n";
      add_header Content-Type text/plain;
    }
  }
}

If you do not have a set of certificates for testing, generate one with:

# CA
openssl req -x509 -newkey rsa:2048 -keyout ca.key -out ca.crt -days 365 -nodes \
  -subj "/CN=Test CA"

# Upstream server certificate
openssl req -newkey rsa:2048 -keyout upstream.key -out upstream.csr -nodes \
  -subj "/CN=upstream.test.local"
openssl x509 -req -in upstream.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out upstream.crt -days 365

# Client certificate (presented by the gateway)
openssl req -newkey rsa:2048 -keyout client.key -out client.csr -nodes \
  -subj "/CN=api7-gateway"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out client.crt -days 365

Configure upstream mTLS

Step 1: Create a service with the upstream tls block

Create a service whose upstream uses the https scheme (or grpcs for gRPC) and embeds the client certificate, key, and — when verification is enabled — the CA certificates the gateway uses to validate the upstream.

pass_host: rewrite together with upstream_host ensures the SNI sent during the TLS handshake matches the upstream server's certificate, even when the node is addressed by IP.

Admin API

CLIENT_CRT=$(awk 1 ORS='\\n' client.crt)
CLIENT_KEY=$(awk 1 ORS='\\n' client.key)
CA_CRT=$(awk 1 ORS='\\n' ca.crt)

curl -k "https://localhost:7443/apisix/admin/services/upstream-mtls?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d "{
    \"name\": \"upstream-mtls\",
    \"upstream\": {
      \"type\": \"roundrobin\",
      \"scheme\": \"https\",
      \"pass_host\": \"rewrite\",
      \"upstream_host\": \"upstream.test.local\",
      \"nodes\": [
        { \"host\": \"192.0.2.10\", \"port\": 8443, \"weight\": 1 }
      ],
      \"tls\": {
        \"client_cert\": \"${CLIENT_CRT}\",
        \"client_key\": \"${CLIENT_KEY}\",
        \"verify\": true,
        \"ca_certs\": [\"${CA_CRT}\"]
      }
    }
  }"

Replace {group_id} with your gateway group ID — use default for the gateway group created by the quickstart. Replace the node host with the address of your upstream.

To skip upstream certificate verification (for example during development with a self-signed upstream certificate), set tls.verify to false and omit tls.ca_certs. The gateway still presents its client certificate but does not validate the upstream's server certificate. This is not recommended for production.

ADC

adc.yaml

services:
  - name: upstream-mtls
    upstream:
      type: roundrobin
      scheme: https
      pass_host: rewrite
      upstream_host: upstream.test.local
      nodes:
        - host: 192.0.2.10
          port: 8443
          weight: 1
      tls:
        client_cert: |
          -----BEGIN CERTIFICATE-----
          <client.crt content>
          -----END CERTIFICATE-----
        client_key: |
          -----BEGIN PRIVATE KEY-----
          <client.key content>
          -----END PRIVATE KEY-----
        verify: false

note

The ADC upstream.tls schema accepts only client_cert, client_key, client_cert_id, and verify. It does not accept a ca_certs field, so when verify is true the gateway falls back to the system CA store. To pin a custom CA for upstream verification, configure the service through the Admin API as shown in the other tab.

Step 2: Create a route on the service

Admin API

curl -k "https://localhost:7443/apisix/admin/routes/upstream-mtls?gateway_group_id={group_id}" -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "upstream-mtls",
    "service_id": "upstream-mtls",
    "paths": ["/*"]
  }'

ADC

Attach the route to the service in the same adc.yaml:

adc.yaml

services:
  - name: upstream-mtls
    # ...upstream block as above...
    routes:
      - name: upstream-mtls
        uris:
          - /*

adc sync -f adc.yaml

Step 3: Verify

Send a request through the gateway:

curl -i "http://127.0.0.1:9080/anything"

You should receive a 200 OK response and a body of the form:

upstream-mtls-ok client=CN=api7-gateway

The client= portion echoes the subject DN of the client certificate the gateway presented, confirming that the upstream completed the mTLS handshake against the gateway's client certificate.

If you remove tls.client_cert and tls.client_key and re-apply the service, the same request returns HTTP 502 because the upstream rejects the connection with an SSL alert (peer did not return a certificate), visible in the gateway error log.

Reference certificates from a secret provider

Instead of embedding PEM material directly in the service definition, you can reference certificates stored in HashiCorp Vault, AWS Secrets Manager, or Kubernetes Secrets using the $secret://... URI format. See Secure Credentials for the available providers.

Troubleshooting

Symptom	Likely cause	Resolution
HTTP 502 from the gateway after enabling upstream mTLS	The upstream rejected the gateway's client certificate.	Verify the client certificate is signed by a CA the upstream trusts. Check the upstream logs for TLS errors.
HTTP 502 with tls.verify: true	The upstream's server certificate is not trusted by the configured tls.ca_certs.	Add the CA that signed the upstream's server certificate to tls.ca_certs, or temporarily set tls.verify: false to confirm this is the cause.
HTTP 502 with SNI mismatch in upstream logs	The upstream certificate's CN/SAN does not match the SNI sent by the gateway.	Set pass_host: rewrite and upstream_host on the upstream so the SNI matches the upstream certificate.
Connection refused or plaintext error	The upstream scheme is http instead of https/grpcs.	Set the upstream scheme to https or grpcs.

Next steps

• Client mTLS Authentication: Configure mTLS for clients connecting to the gateway.
• SSL Certificates: Manage certificates in API7 Gateway.
• Secure Credentials: Store certificates in external secret managers.