Skip to main content
API7.ai Logo
Version: 3.10.x

Upstream mTLS

When an upstream service requires mutual TLS (mTLS), API7 Gateway can present a client certificate and verify the upstream server certificate against trusted certificate authorities. Store the client certificate and CA as certificate objects in the gateway group, then reference their IDs from the service upstream.

info

This page covers mTLS between the gateway and upstream services. For mTLS between API clients and the gateway, see Client mTLS Authentication. For mTLS between the control plane and data plane, see Mutual TLS between CP and DP.

How It Works​

The upstream configuration uses three fields:

FieldValuePurpose
client_certificateCertificate object IDSelects the certificate and private key the gateway presents to the upstream.
ca_certificatesArray of CA certificate object IDsSelects the CA certificates used to verify the upstream server certificate.
tls_verifyBooleanEnables or disables verification of the upstream server certificate.

The older upstream.tls object is deprecated. Do not put inline PEM values in tls.client_cert, tls.client_key, or tls.ca_certs for new configurations.

Prerequisites​

  • A token from the Dashboard.
  • jq for constructing the JSON request bodies and reading the created object IDs.
  • A TLS-enabled upstream that requests a client certificate.
  • A client certificate and private key signed by a CA the upstream trusts.
  • A CA certificate that signed the upstream server certificate.

The upstream certificate subject alternative name must match the hostname the gateway sends as SNI. The examples use upstream.test.local, which must resolve to your test upstream from the gateway nodes.

Set the variables used in this guide:

export API7_GATEWAY_GROUP_ID="default"
export API7_DASHBOARD="https://localhost:7443"

Replace the gateway group ID and Dashboard origin for your environment. Export the Dashboard token as API_KEY before sending the requests.

Create the Certificate Objects​

Create the certificate object that the gateway will present to the upstream:

CLIENT_CERTIFICATE_ID=$(
  jq -n \
    --rawfile cert client.crt \
    --rawfile key client.key \
    '{name: "upstream-mtls-client", cert: $cert, key: $key}' |
  curl -fsSk "${API7_DASHBOARD}/apisix/admin/certificates?gateway_group_id=${API7_GATEWAY_GROUP_ID}" \
    -X POST \
    -H "X-API-KEY: ${API_KEY}" \
    -H "Content-Type: application/json" \
    --data-binary @- |
  jq -r '.value.id'
)

Create the CA certificate object used to verify the upstream:

UPSTREAM_CA_CERTIFICATE_ID=$(
  jq -n \
    --rawfile cert ca.crt \
    '{name: "upstream-mtls-ca", cert: $cert}' |
  curl -fsSk "${API7_DASHBOARD}/apisix/admin/ca_certificates?gateway_group_id=${API7_GATEWAY_GROUP_ID}" \
    -X POST \
    -H "X-API-KEY: ${API_KEY}" \
    -H "Content-Type: application/json" \
    --data-binary @- |
  jq -r '.value.id'
)

Confirm that both requests returned an ID:

printf 'client certificate: %s\nupstream CA: %s\n' \
  "${CLIENT_CERTIFICATE_ID}" \
  "${UPSTREAM_CA_CERTIFICATE_ID}"

Configure the HTTPS Upstream​

Create a service that references the certificate objects. pass_host: rewrite and upstream_host make the gateway send an SNI name that matches the upstream certificate even when a node is addressed by another hostname or an IP address.

jq -n \
  --arg client_certificate "${CLIENT_CERTIFICATE_ID}" \
  --arg ca_certificate "${UPSTREAM_CA_CERTIFICATE_ID}" \
  '{
    name: "upstream-mtls",
    upstream: {
      type: "roundrobin",
      scheme: "https",
      pass_host: "rewrite",
      upstream_host: "upstream.test.local",
      nodes: [
        {
          host: "upstream.test.local",
          port: 8443,
          weight: 100
        }
      ],
      client_certificate: $client_certificate,
      ca_certificates: [$ca_certificate],
      tls_verify: true
    }
  }' |
curl -fsSk "${API7_DASHBOARD}/apisix/admin/services/upstream-mtls?gateway_group_id=${API7_GATEWAY_GROUP_ID}" \
  -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  --data-binary @-

For production traffic, keep tls_verify: true. Setting it to false still lets the gateway present its client certificate, but the gateway does not verify the identity of the upstream server.

note

ADC v0.30.5 models only the deprecated upstream.tls fields and cannot configure client_certificate, ca_certificates, and tls_verify. Use the Dashboard or Admin API for this workflow.

Create a Route and Verify​

Attach a route to the service:

curl -fsSk "${API7_DASHBOARD}/apisix/admin/routes/upstream-mtls?gateway_group_id=${API7_GATEWAY_GROUP_ID}" \
  -X PUT \
  -H "X-API-KEY: ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "upstream-mtls",
    "service_id": "upstream-mtls",
    "paths": ["/*"]
  }'

Send a request through the gateway:

curl -i "http://127.0.0.1:9080/anything"

A successful response confirms that the gateway verified the upstream certificate and the upstream accepted the gateway's client certificate. Check the upstream access logs or expose the authenticated client subject in a test response if you need to verify which client certificate was presented.

Troubleshooting​

SymptomLikely causeResolution
502 Bad Gateway after enabling upstream mTLSThe upstream rejected the gateway's client certificate.Verify that the client certificate is valid and signed by a CA the upstream trusts. Check the upstream TLS logs.
502 Bad Gateway with tls_verify: trueThe gateway cannot build a trust chain from the upstream certificate to one of the referenced CA certificates.Reference the CA that signed the upstream certificate and include any required intermediate certificates.
Certificate name mismatchThe SNI name does not match the upstream certificate's subject alternative names.Set pass_host: rewrite and upstream_host to a hostname covered by the certificate.
Connection refused or a plaintext protocol errorThe upstream scheme or port is wrong.Use https or grpcs and the TLS port of the upstream service.
Certificate object cannot be deletedAn upstream still references the object.Remove the certificate ID from the upstream before deleting the certificate object.

Next Steps​