Skip to main content
API7.ai Logo

Canary Releases with Argo Rollouts

Argo Rollouts replaces a Kubernetes Deployment with a Rollout and advances a release through declared steps. It updates two Services to select the stable and canary ReplicaSets, then changes route weights so the gateway sends the requested percentage of traffic to each revision.

This guide creates a manual pause at 20%, advances to 50%, and then completes the release. It also shows how to abort a faulty revision and restore the desired pod template. Choose either Gateway API HTTPRoute or ApisixRoute and use that choice throughout the guide.

Integration Maturity

Argo Rollouts classifies its built-in APISIX traffic router as alpha. Gateway API also lists the external Argo Rollouts Gateway API plugin used here as alpha, and that plugin does not list APISIX or API7 among its tested providers. Validate the exact versions, permissions, promotion, and abort behavior in a non-production cluster before adopting either path.

Prerequisites​

  • Complete Set Up Ingress Controller and Gateway.
  • Install Helm, kubectl, and the Argo Rollouts kubectl plugin.
  • Ensure the gateway can reach Services in the application namespace.
  • For HTTPRoute, create a programmed Gateway whose listener allows routes from the application namespace.
  • For ApisixRoute, configure an IngressClass for APISIX custom resources.
  • Decide which metrics or manual approval will determine whether a production release can proceed.

The commands below use Argo Rollouts chart 2.41.1, which installs Argo Rollouts v1.9.1, and Gateway API traffic-router plugin v0.16.0. If you use newer versions, review their release notes and verify compatibility first.

Install Argo Rollouts​

Add the chart repository:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

Install the controller for the route API you selected.

The Gateway API traffic router is an external plugin. Save the following values as argo-rollouts-values.yaml. The image digest pins the multi-architecture v0.16.0 plugin artifact that was tested with this guide.

argo-rollouts-values.yaml
controller:
  replicas: 1
  initContainers:
    - name: copy-gateway-api-plugin
      image: ghcr.io/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi@sha256:af5aaba7e34c2b8eb0d52128ac91cb913f065b710c7dea88db59d626a96c53c2
      command:
        - /bin/sh
        - -c
      args:
        - cp /bin/rollouts-plugin-trafficrouter-gatewayapi /plugins/
      volumeMounts:
        - name: gateway-api-plugin
          mountPath: /plugins
  trafficRouterPlugins:
    - name: argoproj-labs/gatewayAPI
      location: file:///plugins/rollouts-plugin-trafficrouter-gatewayapi
  volumes:
    - name: gateway-api-plugin
      emptyDir: {}
  volumeMounts:
    - name: gateway-api-plugin
      mountPath: /plugins

podSecurityContext:
  runAsNonRoot: true
  fsGroup: 999

Install Argo Rollouts with the plugin:

helm upgrade --install argo-rollouts argo/argo-rollouts \
  --version 2.41.1 \
  --namespace argo-rollouts \
  --create-namespace \
  --values argo-rollouts-values.yaml

Wait for the controller:

kubectl rollout status deployment/argo-rollouts \
  --namespace argo-rollouts \
  --timeout 2m

Create Stable and Canary Services​

Save the following resources as echo-services.yaml:

echo-services.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: rollout-demo
---
apiVersion: v1
kind: Service
metadata:
  name: echo-stable
  namespace: rollout-demo
spec:
  ports:
    - name: http
      port: 80
      targetPort: http
  selector:
    app: echo
---
apiVersion: v1
kind: Service
metadata:
  name: echo-canary
  namespace: rollout-demo
spec:
  ports:
    - name: http
      port: 80
      targetPort: http
  selector:
    app: echo

Argo Rollouts adds the stable or canary ReplicaSet hash to these selectors. Do not set or continuously reconcile rollouts-pod-template-hash yourself.

Apply the file:

kubectl apply -f echo-services.yaml

Create the Route​

Create echo-route.yaml. Replace the Gateway name, namespace, and host for your environment:

echo-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: echo
  namespace: rollout-demo
spec:
  parentRefs:
    - name: <gateway-name>
      namespace: <gateway-namespace>
  hostnames:
    - echo.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: echo-stable
          port: 80
        - name: echo-canary
          port: 80

The plugin needs permission to update the HTTPRoute. Save the following namespaced permissions as echo-route-rbac.yaml:

echo-route-rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-rollouts-gateway-api
  namespace: rollout-demo
rules:
  - apiGroups:
      - gateway.networking.k8s.io
    resources:
      - httproutes
    verbs:
      - get
      - update
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-rollouts-gateway-api
  namespace: rollout-demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-rollouts-gateway-api
subjects:
  - kind: ServiceAccount
    name: argo-rollouts
    namespace: argo-rollouts

Apply the permissions:

kubectl apply -f echo-route-rbac.yaml

Leave both backend weight fields unset. Argo Rollouts initializes and owns them.

Apply the route:

kubectl apply -f echo-route.yaml

Create the Rollout​

Save the Rollout for the route API you selected as echo-rollout.yaml.

echo-rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: echo
  namespace: rollout-demo
spec:
  replicas: 5
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: echo
  strategy:
    canary:
      stableService: echo-stable
      canaryService: echo-canary
      trafficRouting:
        plugins:
          argoproj-labs/gatewayAPI:
            httpRoute: echo
            namespace: rollout-demo
      steps:
        - setWeight: 20
        - pause: {}
        - setWeight: 50
        - pause:
            duration: 1m
        - setWeight: 100
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
        - name: echo
          image: hashicorp/http-echo:1.0.0
          args:
            - -listen=:5678
            - -text=stable
          ports:
            - name: http
              containerPort: 5678
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            requests:
              cpu: 10m
              memory: 16Mi

The referenced HTTPRoute rule must contain both stable and canary backend Services. When a route has multiple rules, only rules containing both Services are changed.

Apply the Rollout:

kubectl apply -f echo-rollout.yaml

The first revision becomes stable immediately because there is no older revision to compare. Wait for it to become healthy:

kubectl argo rollouts get rollout echo \
  --namespace rollout-demo \
  --watch

Verify the Initial Route​

Inspect the route status and weights.

kubectl get httproute echo \
  --namespace rollout-demo \
  -o jsonpath='{range .spec.rules[0].backendRefs[*]}{.name}={.weight}{"\n"}{end}'

kubectl get httproute echo \
  --namespace rollout-demo \
  -o jsonpath='{range .status.parents[*].conditions[*]}{.type}={.status} observedGeneration={.observedGeneration}{"\n"}{end}'

Do not proceed unless the route reports Accepted=True and ResolvedRefs=True for its current generation.

The stable Service should have weight 100 and the canary Service weight 0.

Forward the gateway port in another terminal. Replace the Service name and namespace for your installation:

kubectl port-forward service/<gateway-service> \
  9080:80 \
  --namespace <gateway-namespace>

Verify the stable response:

curl -H 'Host: echo.example.com' "http://127.0.0.1:9080/"

The response should be stable.

Release a New Revision​

Change the pod template to start a release:

kubectl patch rollout echo \
  --namespace rollout-demo \
  --type json \
  --patch '[{"op":"replace","path":"/spec/template/spec/containers/0/args/1","value":"-text=canary"}]'

Watch the Rollout until it pauses at 20%:

kubectl argo rollouts get rollout echo \
  --namespace rollout-demo \
  --watch

Use the command from Verify the Initial Route to confirm live weights of 80 and 20. Then send a request sample through the gateway:

for request in $(seq 1 100); do
  curl -sS -H 'Host: echo.example.com' "http://127.0.0.1:9080/"
done | sort | uniq -c

The distribution should be near 80 stable responses and 20 canary responses. A small sample does not produce an exact ratio.

After reviewing the application and metrics, continue the release:

kubectl argo rollouts promote echo --namespace rollout-demo

The release advances to 50%, waits one minute, and then completes. On completion, the newly promoted ReplicaSet receives all traffic and becomes the stable revision.

Add Automated Analysis​

Manual pauses demonstrate the routing flow, but production releases should evaluate service-level indicators. Add an Argo Rollouts AnalysisTemplate and an analysis step that queries your metrics provider. Useful checks include:

  • HTTP request success rate for this route.
  • Tail latency over a window with enough requests.
  • Upstream connection failures and timeouts.
  • Application-specific signals such as checkout completion or authentication failures.
  • A minimum request count, so an empty result cannot be mistaken for success.

Configure failureLimit, consecutiveSuccessLimit, measurement interval, and the metric query together. The observation window should be long enough to represent normal traffic, and a metrics outage should stop advancement rather than silently approve it. See Argo Rollouts analysis documentation for provider-specific resources.

Abort and Recover a Release​

Abort an unsafe release immediately:

kubectl argo rollouts abort echo --namespace rollout-demo

Argo Rollouts returns the route to weight 100 for the last stable ReplicaSet and 0 for the canary. Confirm both the route weights and live gateway responses before declaring recovery complete.

Aborting does not revert the desired pod template in the Rollout. Restore the last good image, arguments, configuration, or Git revision before starting another release. For this example:

kubectl patch rollout echo \
  --namespace rollout-demo \
  --type json \
  --patch '[{"op":"replace","path":"/spec/template/spec/containers/0/args/1","value":"-text=stable"}]'

If Git owns the Rollout, revert or correct the Git revision instead of leaving a live-only patch.

Operate with GitOps​

Argo CD and Argo Rollouts are separate controllers even though they share the Argo name. Argo CD declares and reconciles resources; Argo Rollouts executes the release state machine. Another GitOps controller, or no GitOps controller at all, can be used with the same release flow.

Argo Rollouts must be allowed to update the Service selectors and route backend weights. Keep the weight fields absent from Git, and inspect the GitOps application's live diff during a test release. The Gateway API plugin temporarily adds the rollouts.argoproj.io/gatewayapi-canary=in-progress label to the HTTPRoute, which can help identify an active release but does not transfer ownership of unrelated route fields.

Do not ignore the entire route or Service. If an ignore rule is necessary, limit it to the two backend weight fields or the Rollouts-managed Service selector hash. Broad rules can hide unintended host, path, plugin, filter, or backend changes.

Troubleshooting​

Use the following checks to distinguish route configuration problems from delayed gateway synchronization or a deliberately paused release.

Route Weights Do Not Change​

Verify the route-specific references and permissions.

  • trafficRouting.plugins.argoproj-labs/gatewayAPI.httpRoute and namespace identify the route.
  • A route rule contains both stable and canary Services.
  • The plugin binary was copied and initialized in the Argo Rollouts controller pod.
  • The RoleBinding grants the Argo Rollouts ServiceAccount permission to get, update, and patch the route.

For either route type, confirm that the Service names match the Rollout and another reconciler is not restoring the weights. Inspect controller events and logs:

kubectl describe rollout echo --namespace rollout-demo
kubectl logs deployment/argo-rollouts --namespace argo-rollouts

Weights Change but Traffic Does Not​

Check the route status, Service endpoints, and pod labels:

kubectl get endpointslice --namespace rollout-demo \
  --label kubernetes.io/service-name=echo-canary
kubectl get pods --namespace rollout-demo --show-labels

Both Services must select the intended ReplicaSets, and the Ingress Controller must accept and synchronize the current route generation. Also confirm that the request host and path match the route and that the gateway has received the update before evaluating traffic.

A Release Remains Paused​

An empty pause: {} has no duration and requires promotion. Check the Rollout status before promoting it. Do not remove safety pauses merely to clear an unexpected state.

Clean Up​

Remove the example application:

kubectl delete namespace rollout-demo

Remove Argo Rollouts only if no other Rollouts depend on it:

helm uninstall argo-rollouts --namespace argo-rollouts
kubectl delete namespace argo-rollouts

The chart marks the Argo Rollouts CRDs with helm.sh/resource-policy: keep, so uninstalling the release does not remove them. Retain the CRDs if another installation might use them. Delete them separately only after confirming that no Rollout, AnalysisTemplate, AnalysisRun, or Experiment resources remain.