Skip to main content
API7.ai Logo

Automated Canary Releases with Flagger

Flagger automates canary analysis for a Kubernetes Deployment. It coordinates candidate and primary revisions, weighted routing, test traffic, metrics, and promotion or rollback.

This guide configures a canary that advances by 10% to a maximum of 30%. It requires a 99% request success rate and a maximum request duration of 500 milliseconds. Choose either the Gateway API provider with HTTPRoute or the APISIX provider with ApisixRoute and use that choice throughout the guide.

Gateway API Integration Maturity

Gateway API lists the Flagger integration as public preview. Validate the exact controller, gateway, and Flagger versions, along with initialization, promotion, and rollback behavior, in a non-production cluster before adopting the Gateway API path.

Understand What Flagger Manages​

For a target Deployment named podinfo, Flagger manages these resources:

ResourcePurpose
podinfo DeploymentSource pod template; Flagger scales it down after initialization and uses it for candidate revisions
podinfo-primary DeploymentLast successfully promoted pod template
podinfo, podinfo-primary, and podinfo-canary ServicesPublic, stable, and canary service selection
Generated HTTPRoute or ApisixRouteStable and canary weights controlled by Flagger

Do not add the generated resources to another Helm release, Kustomize base, or GitOps application. Continue changing the source Deployment; Flagger copies successful changes to the primary Deployment.

The provider determines how Flagger builds the release route:

  • With gatewayapi:v1, Flagger uses the host names and Gateway references in the Canary to generate an HTTPRoute named podinfo.
  • With apisix, Flagger copies the source ApisixRoute into a generated route named podinfo-podinfo-canary.

Prerequisites​

  • Complete Set Up Ingress Controller and Gateway.
  • Install Helm and kubectl.
  • Ensure the gateway can reach Services in the application namespace.
  • For HTTPRoute, create a programmed Gateway whose listener allows routes from the application namespace.
  • For ApisixRoute, ensure you can change the cluster-wide default IngressClass safely.
  • Use an APISIX gateway or API7 Gateway that exposes APISIX Prometheus metrics.

The commands below use Flagger chart 1.44.0 and load tester chart 0.38.0. If you use newer versions, review their release notes and verify compatibility first.

Expose APISIX Metrics​

The analysis queries apisix_http_status and apisix_http_latency_bucket. The Prometheus server installed later discovers gateway pods through standard scrape annotations.

If APISIX was installed with chart 2.16.0, add the following values to the gateway release values:

apisix-values.yaml
apisix:
  prometheus:
    enabled: true

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9091"
  prometheus.io/path: /apisix/prometheus/metrics

For API7 Gateway chart 3.10.11, add the following values instead:

api7-gateway-values.yaml
pluginAttrs:
  prometheus:
    enable_export_server: true
    export_addr:
      ip: 0.0.0.0
      port: 9091

apisix:
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9091"
    prometheus.io/path: /apisix/prometheus/metrics

The explicit API7 export address makes the endpoint reachable from the Prometheus pod rather than only from inside the gateway pod. Apply these values through the Helm or GitOps workflow that owns the gateway release. Do not run a separate helm upgrade against a GitOps-owned release.

For a different chart version or APISIX package, confirm both the Prometheus attributes and pod-annotation value paths before applying them. Value paths are not universal across charts.

Verify the gateway pod annotations:

kubectl get pods --namespace <gateway-namespace> \
  --selector <gateway-pod-label-selector> \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.prometheus\.io/scrape}{"\t"}{.metadata.annotations.prometheus\.io/port}{"\n"}{end}'

For the chart versions above, use app.kubernetes.io/name=apisix for APISIX and app.kubernetes.io/name=gateway for API7 Gateway.

Forward the metrics port from a gateway pod and confirm that metrics are available:

kubectl port-forward pod/<gateway-pod> \
  9091:9091 \
  --namespace <gateway-namespace>

If the gateway chart exposes a metrics Service, you can forward the Service port instead:

kubectl port-forward service/<gateway-metrics-service> \
  9091:9091 \
  --namespace <gateway-namespace>

In another terminal:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

Do not proceed until the endpoint returns Prometheus text metrics.

Install Flagger and Prometheus​

Add the repository:

helm repo add flagger https://flagger.app
helm repo update

Choose the installation's default provider. Each Canary in this guide sets spec.provider, which overrides meshProvider, so one Flagger controller can reconcile both provider types. Use separate installations only when you need independent namespace scopes or operational isolation.

helm upgrade --install flagger flagger/flagger \
  --version 1.44.0 \
  --namespace flagger-system \
  --create-namespace \
  --set meshProvider=gatewayapi:v1 \
  --set prometheus.install=true

The chart installs its CRDs from the crds directory. Do not set crd.create=true with Helm 3; that legacy Helm 2 option renders a second copy of the same CRDs as Helm templates.

Wait for both Deployments:

kubectl rollout status deployment/flagger \
  --namespace flagger-system \
  --timeout 2m

kubectl rollout status deployment/flagger-prometheus \
  --namespace flagger-system \
  --timeout 2m

Confirm that Flagger can query Prometheus:

kubectl logs deployment/flagger --namespace flagger-system | \
  grep 'all the metrics providers are available'

Configure the Route Provider​

Gateway API does not define APISIX plugin configuration, and Flagger owns the generated HTTPRoute. Enable the Prometheus plugin through the route's GatewayProxy. Add the following entry to the existing spec.plugins list:

gateway-proxy.yaml
spec:
  plugins:
    - name: prometheus
      enabled: true
      config:
        disable: false
        prefer_name: true

This enables Prometheus metrics globally for routes associated with that GatewayProxy. Review the metric cardinality and exposure implications before applying it to a shared production gateway.

Apply the updated GatewayProxy through its owning workflow, then verify that normal gateway requests create apisix_http_status samples. The prefer_name setting produces route labels that the metric templates in this guide select.

Deploy the Example Application​

Create podinfo-deployment.yaml:

podinfo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
  namespace: flagger-demo
spec:
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      labels:
        app: podinfo
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.14.0
          ports:
            - name: http
              containerPort: 9898
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
          resources:
            requests:
              cpu: 10m
              memory: 32Mi

Create the namespace and apply the Deployment:

kubectl create namespace flagger-demo
kubectl apply -f podinfo-deployment.yaml

If the Gateway listener selects namespaces by label, add its required label to flagger-demo before continuing. Use an immutable image digest in a production release policy. The version tag keeps this tutorial readable, but a mutable tag cannot prove which artifact was evaluated.

Install the Load Tester​

The load tester executes a command from a Flagger webhook throughout the analysis. Install it in the application namespace:

helm upgrade --install flagger-loadtester flagger/loadtester \
  --version 0.38.0 \
  --namespace flagger-demo

kubectl rollout status deployment/flagger-loadtester \
  --namespace flagger-demo \
  --timeout 2m

In production, use representative traffic or a reviewed test command. Confirm that a load test cannot mutate data, overload a dependency, or bypass normal authorization.

Configure Routing and Metrics​

Do not create a source HTTPRoute. Flagger generates it from the host names and gatewayRefs in the Canary.

The built-in HTTP observer for the Gateway API provider does not query APISIX gateway metrics. Save these APISIX-specific queries as apisix-metric-templates.yaml:

apisix-metric-templates.yaml
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: apisix-request-success-rate
  namespace: flagger-system
spec:
  provider:
    type: prometheus
    address: http://flagger-prometheus.flagger-system:9090
  query: |
    sum(
      rate(
        apisix_http_status{
          route=~"{{ namespace }}_{{ target }}_.+",
          code!~"5.."
        }[{{ interval }}]
      )
    )
    /
    sum(
      rate(
        apisix_http_status{
          route=~"{{ namespace }}_{{ target }}_.+"
        }[{{ interval }}]
      )
    ) * 100
---
apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
  name: apisix-request-duration
  namespace: flagger-system
spec:
  provider:
    type: prometheus
    address: http://flagger-prometheus.flagger-system:9090
  query: |
    histogram_quantile(
      0.99,
      sum(
        rate(
          apisix_http_latency_bucket{
            type="request",
            route=~"{{ namespace }}_{{ target }}_.+"
          }[{{ interval }}]
        )
      ) by (le)
    )

Apply the templates:

kubectl apply -f apisix-metric-templates.yaml

The template selects the generated route label, such as flagger-demo_podinfo_0-0. If your label differs, inspect a live apisix_http_status sample and narrow the expression to the generated route before starting a release.

Create the Canary Policy​

Save the policy for the route API you selected as podinfo-canary.yaml. Replace the host and gateway Service address for your environment.

Replace the gatewayRefs name and namespace with the programmed Gateway that should receive the generated route:

podinfo-canary.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: podinfo
  namespace: flagger-demo
spec:
  provider: gatewayapi:v1
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: podinfo
  progressDeadlineSeconds: 120
  service:
    port: 80
    targetPort: http
    hosts:
      - podinfo.example.com
    gatewayRefs:
      - name: <gateway-name>
        namespace: <gateway-namespace>
  analysis:
    interval: 30s
    threshold: 5
    maxWeight: 30
    stepWeight: 10
    metrics:
      - name: apisix-success-rate
        templateRef:
          name: apisix-request-success-rate
          namespace: flagger-system
        thresholdRange:
          min: 99
        interval: 1m
      - name: apisix-request-duration-p99
        templateRef:
          name: apisix-request-duration
          namespace: flagger-system
        thresholdRange:
          max: 500
        interval: 1m
    webhooks:
      - name: smoke-test
        type: pre-rollout
        url: http://flagger-loadtester.flagger-demo/
        timeout: 5s
        metadata:
          type: bash
          cmd: >-
            curl -fsS -d anon
            http://podinfo-canary.flagger-demo/token | grep token
      - name: load-test
        type: rollout
        url: http://flagger-loadtester.flagger-demo/
        timeout: 5s
        metadata:
          cmd: >-
            hey -z 2m -q 20 -c 2 -host podinfo.example.com
            http://<gateway-service>.<gateway-namespace>/

The custom analysis names intentionally differ from the reserved Flagger names request-success-rate and request-duration. Using either reserved name also invokes the generic built-in observer for the Gateway API provider before the referenced APISIX template and can produce a misleading no values found failure.

The examples use short intervals and five failed checks to make behavior observable while allowing Prometheus time to collect the first samples. Production values should account for normal traffic volume, metric delay, application warm-up, and the cost of a false rollback.

Apply the policy:

kubectl apply -f podinfo-canary.yaml

Verify Initialization​

Wait for the Canary phase to become Initialized:

kubectl get canary podinfo --namespace flagger-demo --watch

Inspect the generated route.

kubectl get httproute podinfo \
  --namespace flagger-demo \
  -o yaml

The route should contain podinfo-primary with weight 100 and podinfo-canary with weight 0. Do not proceed unless it reports Accepted=True and ResolvedRefs=True for its current generation.

Verify that Prometheus has samples for the generated route before releasing a new revision:

kubectl port-forward service/flagger-prometheus \
  9090:9090 \
  --namespace flagger-system

In another terminal:

curl -G "http://127.0.0.1:9090/api/v1/query" \
  --data-urlencode 'query=count by (route) (apisix_http_status)'

If the route has not received traffic yet, first send requests through the same gateway host and path used by the load-test webhook.

Promote a Revision Automatically​

Update the source Deployment:

kubectl set image deployment/podinfo \
  podinfo=ghcr.io/stefanprodan/podinfo:6.14.1 \
  --namespace flagger-demo

Watch the canary:

watch kubectl get canary podinfo --namespace flagger-demo

In another terminal, watch the generated route weights with the command appropriate to your route type:

ApisixRoute
watch kubectl get apisixroute podinfo-podinfo-canary \
  --namespace flagger-demo \
  -o jsonpath='{range .spec.http[0].backends[*]}{.serviceName}={.weight}{" "}{end}{"\n"}'

For an HTTPRoute, watch the backend references instead:

HTTPRoute
watch kubectl get httproute podinfo \
  --namespace flagger-demo \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight}{" "}{end}{"\n"}'

Flagger should advance through weights 10, 20, and 30 while metric checks pass. It then copies the canary pod template to podinfo-primary, returns routing to 100/0, and marks the Canary Succeeded.

Verify the promoted image:

kubectl get deployment podinfo-primary \
  --namespace flagger-demo \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Verify Automatic Rollback​

Test the rollback policy in a safe environment before relying on it in production. Trigger another candidate revision, wait until its weight is greater than zero, and generate 5xx responses through the same route:

kubectl set env deployment/podinfo \
  'PODINFO_UI_COLOR=#ff0000' \
  --namespace flagger-demo

# Wait until kubectl get canary reports a weight greater than zero.
kubectl exec deployment/flagger-loadtester \
  --namespace flagger-demo \
  -- \
  hey -z 2m -q 20 -c 2 \
  -host podinfo.example.com \
  http://<gateway-service>.<gateway-namespace>/status/500

Watch Flagger events:

kubectl describe canary podinfo --namespace flagger-demo
kubectl logs deployment/flagger --namespace flagger-system --follow

After the failed-check threshold is reached, Flagger should:

  1. Set the generated route back to primary weight 100 and canary weight 0.
  2. Scale down the failed canary Deployment.
  3. Keep the previous image in podinfo-primary.
  4. Mark the Canary phase Failed.

Confirm application traffic through the gateway after rollback. A Failed Canary is evidence that the policy intervened; it does not by itself prove that every dependency recovered.

Design Production Analysis​

The gateway checks are a starting point. Add MetricTemplate resources for service-specific indicators and combine them with webhooks where appropriate. A production policy should consider:

  • A minimum number of requests before evaluating a percentage.
  • Query windows and scrape delay. An initial no values found event can occur before the first samples arrive and must not be treated as success.
  • Recent failures under the same route label. A new release can initially observe samples still inside the metric query window.
  • Low-traffic services, which may need synthetic traffic or longer intervals.
  • Cold starts, caches, connection pools, and dependency warm-up.
  • Business and dependency metrics that gateway HTTP status alone cannot detect.
  • Alerting for Failed, stuck Progressing, and unavailable metric-provider states.
  • Manual approval for high-risk environments.

Keep the failure threshold low enough to bound impact, but high enough to tolerate expected measurement delay. Multiply the analysis interval by the permitted number of failed checks to estimate how long Flagger can take to reject a failing release.

Route and Provider Limitations​

For the APISIX provider:

  • spec.ingressClassName is not retained in the generated route, requiring a default APISIX IngressClass.
  • The selected source HTTP rule must have exactly one backend.
  • Flagger copies the source route through the APISIX CRD model maintained by Flagger, which can lag fields added by newer Ingress Controller releases.
  • Additional route plugins and complex plugin configuration should be tested through initialization, promotion, and rollback. Flagger issue #1388 documents a resource-conflict loop associated with numeric APISIX plugin configuration.

For the Gateway API provider:

  • Flagger generates and owns the HTTPRoute; it does not adopt an existing route.
  • Route behavior must be expressible through the Flagger Canary service configuration and supported Gateway API fields.
  • APISIX-specific analysis requires custom metric templates; the generic Gateway API observer is not an APISIX metrics integration.
  • The route-level Prometheus plugin cannot be attached through the generated standard HTTPRoute, so this guide enables it at the GatewayProxy scope.

Inspect generated resources after every Flagger upgrade. Do not assume a successfully admitted Canary means the route, policies, and metrics still behave identically.

Troubleshooting​

Use the following checks to identify route admission, metric collection, reconciliation, or release progress problems.

Generated Route Is Not Accepted​

For ApisixRoute, inspect the generated route and default IngressClass:

kubectl get apisixroute podinfo-podinfo-canary \
  --namespace flagger-demo \
  -o yaml
kubectl get ingressclass -o yaml

For HTTPRoute, inspect its parent conditions and confirm the Gateway listener permits the route namespace:

kubectl get httproute podinfo --namespace flagger-demo -o yaml
kubectl get gateway <gateway-name> \
  --namespace <gateway-namespace> \
  -o yaml

Flagger Reports No Metric Values​

Verify the Prometheus target, gateway annotations, route traffic, and route label:

kubectl port-forward service/flagger-prometheus \
  9090:9090 \
  --namespace flagger-system

curl -G "http://127.0.0.1:9090/api/v1/query" \
  --data-urlencode 'query=apisix_http_status'

The load-test URL must pass through the same route that the Canary measures. Direct requests to the pod or Service do not produce APISIX route metrics. For HTTPRoute, also confirm that the custom analysis names are not the reserved built-in metric names.

Route Updates Repeat or Conflict​

Inspect the generated route and Flagger logs for repeated updates. Keep the generated route outside Git and other package managers. For the APISIX provider, simplify additional plugin configuration to identify a field that does not round-trip through the APISIX type in Flagger. Do not disable reconciliation conflict handling globally.

Promotion Takes Longer Than Expected​

Flagger can pause advancement while it waits for ready pods, the first metric samples, a webhook, or the next analysis interval. Inspect Canary events rather than estimating progress only from maxWeight and stepWeight.

Clean Up​

Delete the example namespace:

kubectl delete namespace flagger-demo

Remove Flagger only if no other Canary resources depend on it:

helm uninstall flagger --namespace flagger-system
kubectl delete namespace flagger-system

Helm retains CRDs installed from a chart's crds directory. Retain the Flagger CRDs if another installation might use them. Delete them separately only after confirming that no Canary, MetricTemplate, or AlertProvider resources remain.

If you selected ApisixRoute, remove the default annotation only when APISIX should no longer be the cluster default:

kubectl annotate ingressclass apisix \
  ingressclass.kubernetes.io/is-default-class-

If you selected HTTPRoute, remove the global Prometheus plugin only when no other monitoring workflow depends on it.