Skip to main content
API7.ai Logo

Manage with Argo CD

Argo CD is a declarative GitOps continuous-delivery tool for Kubernetes. An Argo CD Application identifies the source repository and revision, the desired Kubernetes configuration, and the destination cluster. The Application controller compares the live resources with that desired state and synchronizes changes manually or automatically.

For APISIX or API7 Ingress Controller, the Application uses Helm to render the controller chart from configuration stored in Git. Argo CD reconciles the rendered Kubernetes resources but does not create a Helm release. This distinction affects how the chart's admission-webhook certificate and CRDs should be managed.

This guide prepares a stable webhook certificate, creates the controller Application, configures CRD ownership, and reconciles gateway and application resources. It also covers verification, OpenShift and ROSA requirements, upgrades, and rollback.

Prerequisites​

Before you begin, ensure that you have:

  • The repository, resource ownership, CRD, and credential preparation described in Prepare for GitOps.
  • An Argo CD installation and an authenticated argocd CLI.
  • An AppProject restricted to the selected chart repository, destination namespace, and required resource kinds. The built-in default project is broadly permissive unless your Argo CD administrators have restricted it.

The examples use aic as the destination namespace. If you use another namespace, replace aic throughout the manifests, commands, and webhook Service DNS names.

The example values focus on GitOps ownership and reconciliation; they retain the chart defaults for replicas, resource requests, and pod scheduling. Before production, configure capacity for your workload, run multiple controller replicas, and spread them across failure domains. See AIC High Availability and test failover in the target cluster. The podDisruptionBudget.enabled option in the pinned charts does not render successfully; do not enable it without verifying that a later chart release contains the fix.

Provide a Stable Webhook Certificate​

Configure a deterministic certificate before Argo CD renders and synchronizes the controller chart.

Current Chart Limitation with Argo CD

With the default webhook.certificate.provided: false, the APISIX and API7 charts used in these examples generate a new admission-webhook CA and certificate during every offline Helm render. Their Helm lookup call cannot retrieve the existing Secret when Argo CD renders the chart, so the Secret and webhook caBundle can remain permanently OutOfSync or rotate during automated synchronization.

Choose how Argo CD should handle the admission webhook before creating the Application resource:

ApproachResultRecommendation
Provide an externally managed certificatePreserves admission validation and produces stable manifests.Recommended for Argo CD. Your platform must manage the certificate and its rotation.
Set webhook.enabled: falsePrevents the chart from rendering the webhook resources, but removes the controller's admission validation.Use only when the reduced validation is acceptable and separately tested.
Leave chart-generated certificates enabledProduces different desired manifests on repeated Argo CD renders.Do not use for a continuously reconciled Application with these chart versions.

The Application examples keep the webhook enabled and use an externally managed certificate. With the example release names and aic namespace, use the following values:

ProductCertificate DNS nameSecret name
APISIX Ingress Controllerapisix-ingress-controller-webhook-svc.aic.svcapisix-ingress-controller-webhook-cert
API7 Ingress Controllerapi7-ingress-controller-webhook-svc.aic.svcapi7-ingress-controller-webhook-cert

Prepare the certificate and Secret:

  1. Issue a certificate valid for server authentication that includes the certificate DNS name as a subject alternative name.

  2. Configure your destination-cluster secret manager to create the corresponding Secret in the destination namespace.

  3. Store the server certificate and private key as tls.crt and tls.key. Do not store the private key in Git.

  4. Encode the issuing CA certificate:

    openssl base64 -A -in ca.crt

  5. Set webhook.certificate.caBundle in the Application to the encoded output. The examples use the REPLACE_WITH_BASE64_CA_CERTIFICATE placeholder.

If you change releaseName, nameOverride, fullnameOverride, or the destination namespace, render the chart first. Inspect the Deployment's webhook-certs volume for the required Secret name and inspect the webhook Service for the required DNS name.

The charts do not use webhook.certificate.secretName or read ca.crt from the external Secret. They use webhook.certificate.caBundle to configure client trust for the webhook.

Renew a serving certificate signed by the same CA by updating the external Secret before the certificate expires. The controller watches the mounted certificate files and reloads the updated key pair.

If the issuing CA changes, rotate trust in stages. First, set caBundle to the base64-encoded concatenated PEM certificates for the old and new certificate authorities, then synchronize the Application. Next, update the external Secret with a serving certificate signed by the new CA and verify admission requests. Remove the old CA from caBundle only after every controller pod serves the new certificate. Test either rotation path in a non-production cluster.

Do not use broad Argo CD ignoreDifferences rules as the normal solution. They hide real certificate changes and can leave the webhook configuration and mounted Secret with different trust chains.

To accept reduced validation instead, replace the example's entire webhook block with webhook.enabled: false. Record the decision in the platform repository and test invalid resources as part of admission-independent validation.

Create the Application​

Create an Application for the selected controller. The examples use the controller-owned CRD model and the externally managed webhook certificate prepared earlier.

Apply the CRD ownership decision from Prepare for GitOps when configuring the Application:

  • If the controller Application owns the bundled CRDs, leave source.helm.skipCrds set to false, use server-side apply for the bundled large HTTPRoute CRD, and keep pruning disabled. Argo CD treats rendered CRDs as ordinary tracked resources, so pruning or deleting the Application with cascading deletion can delete the CRDs and their custom resources.
  • If the platform owns the CRDs, reconcile the required Gateway API and APISIX CRDs from a separate, pinned Application. Wait until they report an Established condition, then set source.helm.skipCrds to true on the controller Application. Reconcile and verify the CRD Application first. Do not use sync waves to order independent Applications; use an app-of-apps pattern when one Application must explicitly control the order.

The manifests use valuesObject. Confirm that your Argo CD version supports this field before applying them.

The APISIX Admin API Service, apisix-admin-key Secret, and apisix-ingress-controller-webhook-cert Secret must already exist in the destination namespace (aic in the example).

apisix-ingress-controller-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: apisix-ingress-controller
  namespace: argocd
spec:
  project: <app-project>
  source:
    repoURL: https://apache.github.io/apisix-helm-chart
    chart: apisix-ingress-controller
    targetRevision: 1.2.2
    helm:
      releaseName: apisix-ingress-controller
      skipCrds: false
      valuesObject:
        webhook:
          enabled: true
          certificate:
            provided: true
            caBundle: REPLACE_WITH_BASE64_CA_CERTIFICATE
        config:
          provider:
            type: apisix-standalone
        gatewayProxy:
          createDefault: true
          provider:
            type: ControlPlane
            controlPlane:
              service:
                name: apisix-admin
                port: 9180
              auth:
                type: AdminKey
                adminKey:
                  valueFrom:
                    secretKeyRef:
                      name: apisix-admin-key
                      key: admin-key
  destination:
    server: https://kubernetes.default.svc
    namespace: aic
  syncPolicy:
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m
    syncOptions:
    - ServerSideApply=true

The finite retry policy handles first-time installation when this chart creates the CRDs and a default GatewayProxy in the same synchronization. Argo CD can discover the custom resource only after the CRD is applied, so a retry completes the desired-state operation after discovery refreshes.

If a platform Application installs and verifies the CRDs before the controller Application, remove this retry policy unless another transient dependency requires it.

ServerSideApply=true allows the Application to create the bundled large CRDs. Server-side apply affects every resource in the Application, not only its CRDs. Review field ownership before enabling it when another reconciler modifies the same objects. For narrower ownership, move the CRDs into a dedicated platform Application and set skipCrds: true on the controller Application.

If you disable the webhook and no other resource must exist in the namespace before synchronization, you can let the Application create the namespace by adding CreateNamespace=true to syncOptions.

Adapt the Workflow for OpenShift or ROSA​

Red Hat OpenShift GitOps packages Argo CD, so the certificate guidance above also applies on OpenShift and ROSA. Apply the following platform-specific changes before synchronizing the Application. Set metadata.namespace on each Application to the namespace of the installed Argo CD instance, commonly openshift-gitops.

Determine Gateway API CRD Ownership​

Handle the Gateway API CRDs according to the OpenShift version:

  • On OpenShift 4.18 and earlier, Gateway API CRDs are not installed by default. Have the platform team install a compatible bundle, or allow one designated GitOps Application to own them.
  • Starting with OpenShift 4.19, the Ingress Operator owns the Gateway API CRD lifecycle. Set skipCrds: true and do not apply the chart's bundled Gateway API CRDs. OpenShift 4.19 introduced platform-managed Gateway API 1.2.1, while these charts bundle the 1.3.0 experimental channel. Verify that the resource kinds and versions you plan to use exist on the cluster.

For the version boundary and ownership behavior, see the OpenShift 4.19 release notes and Red Hat's Gateway API installation guidance.

When skipCrds: true is required, reconcile only the APISIX CRD file from the same pinned chart version through a cluster-infrastructure Application. It is crds/apisixic-crds.yaml in the APISIX chart and crds/apisix-crds.yaml in the API7 chart. Do not include gwapi-crds.yaml when the Ingress Operator manages Gateway API CRDs.

Grant SCC Access​

The chart creates a ServiceAccount whose name matches helm.releaseName. Store the following resources in the platform repository and replace the placeholders:

ingress-controller-scc.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: <release-name>-nonroot-v2
  namespace: <controller-namespace>
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - nonroot-v2
  resources:
  - securitycontextconstraints
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: <release-name>-nonroot-v2
  namespace: <controller-namespace>
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: <release-name>-nonroot-v2
subjects:
- kind: ServiceAccount
  name: <release-name>
  namespace: <controller-namespace>

For API7 Ingress Controller, also merge the following value with the Dashboard-generated values:

values-openshift.yaml
adc:
  securityContext:
    runAsUser: 65532

Treat the controller as cluster infrastructure. The Argo CD instance needs permission to manage the destination namespace, cluster roles, ingress classes, and webhook configuration. A cluster administrator should own CRD and SCC policy decisions. Do not run the generated imperative Helm command against an Argo CD-managed installation.

Use Install API7 Ingress Controller on OpenShift for the gateway-group and gateway-instance workflow. Test the exact values and permissions on the target OpenShift or ROSA version before enabling automated synchronization.

Synchronize the Application​

Save and commit the Application manifest in the platform repository. For a production deployment, have a parent Application, ApplicationSet, or equivalent bootstrap process reconcile that directory.

For an initial evaluation, you can instead create the controller Application directly:

kubectl apply -f <application-file>

Directly applying the file does not make later Git changes to the Application manifest self-reconciling. Before production, place it under a parent reconciliation path rather than relying on repeated kubectl apply operations.

Inspect the first diff, synchronize, and wait for the Application:

argocd app diff <application-name>
argocd app sync <application-name>
argocd app wait <application-name> --health --sync --timeout 300

After the first successful sync, perform a hard refresh and confirm that no certificate or other unexpected changes appear:

argocd app get <application-name> --hard-refresh
argocd app diff <application-name>

Enable automated synchronization only after the second comparison is stable. Merge automated into the existing syncPolicy instead of replacing it. The following settings enable self-healing and leave pruning disabled because the example Applications own CRDs:

syncPolicy:
  automated:
    prune: false
    selfHeal: true
  retry:
    limit: 5
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m
  syncOptions:
  - ServerSideApply=true

Enable prune: true only after moving CRDs and shared namespace or class resources to separately protected owners. Review the resources that Argo CD would delete before changing this setting.

Reconcile Gateway and Application Resources​

The controller Application manages the controller release. Manage the shared Gateway resources and application routes with separate Applications so that their ownership and lifecycles remain independent. These Applications can read ordinary Kubernetes YAML directly; Kustomize is not required.

Store the product-specific GatewayProxy, GatewayClass, shared Gateway, application namespaces, and authorization labels in the platform-owned infrastructure/gateway-resources/ directory prepared earlier. For APISIX, omit the GatewayProxy from this directory when the controller chart creates the default object. For API7, use the non-sensitive resources generated by API7 Dashboard and retain their generated names and references.

Create an Application for that directory:

gateway-resources-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ingress-gateway-resources
  namespace: argocd
spec:
  project: <platform-project>
  source:
    repoURL: https://github.com/example/platform-config.git
    targetRevision: main
    path: clusters/production/infrastructure/gateway-resources
  destination:
    server: https://kubernetes.default.svc
    namespace: aic
  syncPolicy:
    automated:
      prune: false
      selfHeal: true

The AppProject must permit the source repository, destination namespaces, Gateway API and APISIX resource kinds, and the cluster-scoped GatewayClass. Keep pruning disabled until you confirm that this Application is the only owner of every resource in the directory. Synchronize it and wait for the GatewayClass and Gateway to report Accepted=True before reconciling application routes.

Store each application's Deployment, Service, HTTPRoute, and policies with that application. You can use the manifests from Configuration Examples without converting them to overlays. Before committing an example, replace its explicit metadata.namespace: aic values with the application namespace, or remove them so Argo CD uses destination.namespace. Create a second Application for the application directory:

httpbin-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: httpbin-production
  namespace: argocd
spec:
  project: <app-project>
  source:
    repoURL: https://github.com/example/platform-config.git
    targetRevision: main
    path: clusters/production/applications/httpbin
  destination:
    server: https://kubernetes.default.svc
    namespace: <application-namespace>
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Commit both Applications to the parent reconciliation path. Restrict the application AppProject to its repository, destination namespace, and required namespaced resource kinds. Argo CD detects plain YAML automatically when a source directory does not contain another supported configuration source.

Verify​

Verify the controller deployment, CRDs, gateway connection, and end-to-end routing before promoting the installation to another environment.

Confirm that the Deployment is available and the CRDs are established:

kubectl rollout status deployment \
  -l app.kubernetes.io/instance=<release-name> \
  -n aic \
  --timeout=300s
kubectl get pods,services -n aic
kubectl get crd gatewayproxies.apisix.apache.org
kubectl wait --for=condition=Established \
  crd/gatewayproxies.apisix.apache.org \
  --timeout=60s

The controller pod should report both the controller and ADC containers as ready. Inspect the rendered controller configuration and expected GatewayProxy:

kubectl get configmap -n aic
kubectl get configmap <configmap-name> -n aic \
  -o jsonpath='{.data.config\.yaml}{"\n"}'
kubectl get gatewayproxy -n aic
kubectl describe gatewayproxy <gatewayproxy-name> -n aic

GatewayProxy does not expose an acceptance status in this version. When the admission webhook is enabled, invalid objects can be rejected during creation. Verify runtime connectivity through the controller logs and the Accepted, Programmed, and ResolvedRefs conditions on related Gateway and HTTPRoute resources.

Complete Proxy Requests to a Service with the gateway and classes managed by Git, then perform a final no-change comparison:

argocd app get <application-name> --hard-refresh
argocd app diff <application-name>

Upgrade and Roll Back​

Manage upgrades and rollbacks through Git so that Argo CD remains the only reconciler for the controller resources.

To upgrade the controller:

  1. Review the controller and chart release notes, including CRD changes.
  2. Change the pinned chart version and required values in Git.
  3. Inspect the diff before synchronization, including resources that the new chart no longer renders.
  4. Synchronize one environment and repeat the deployment, route, and no-change verification.
  5. Promote the same Git change to the next environment.

The example Application disables automated pruning because it owns the bundled CRDs. Without pruning, resources removed or renamed by a later chart can remain in the cluster after an upgrade. For ongoing production upgrades, prefer transferring the CRDs to a platform Application, setting skipCrds: true, and enabling pruning after reviewing the deletion preview. If the controller Application retains CRD ownership, leave automated pruning disabled. During each upgrade, review the resources marked for pruning and perform a manual sync with pruning only after confirming that the bundled CRDs and their custom resources are not included.

Before rolling back, confirm that the installed CRDs are compatible with the older controller. Ensure that the controller Application uses skipCrds: true so the rollback does not apply older CRD schemas, and let a separate, pinned platform Application own the installed CRDs. If the controller Application currently owns them, follow Transfer CRD Ownership Safely before reverting the chart version and controller values in Git.

If the controller Application still owns CRDs, reverting the chart can apply older CRD schemas despite the intended controller-only rollback. Do not run helm rollback or helm upgrade against an Argo CD-managed deployment.

Transfer CRD Ownership Safely​

Argo CD treats rendered CRDs as ordinary tracked resources. Disabling pruning protects CRDs during synchronization, but not when an Application is deleted with cascading resource deletion.

If the controller Application owns CRDs and you later need to enable prune: true or delete the Application, transfer ownership first:

  1. Back up the custom resources, disable automated synchronization, and keep pruning disabled on the controller Application.
  2. Add the same installed CRD versions to a platform Application and synchronize it with server-side apply. This synchronization adopts the existing CRDs and updates their Argo CD tracking metadata.
  3. Set skipCrds: true on the controller Application and refresh both Applications.
  4. Confirm that the controller Application no longer tracks the CRDs and that the platform Application reports them as synchronized.
  5. Inspect a deletion preview before re-enabling automated synchronization or pruning on the controller Application.

This handoff briefly exposes the CRDs to both Applications while their tracking metadata changes. Perform it as a controlled platform operation and do not enable FailOnSharedResource=true until the handoff is complete. Prefer assigning platform CRD ownership before the initial controller installation when possible.

If you need to stop managing the Application while retaining every resource, verify the resources that will remain and delete it without cascading:

argocd app delete <application-name> --cascade=false

Troubleshoot​

Use the following checks to diagnose common reconciliation and platform failures:

SymptomCheckResolution
Application remains OutOfSync on the webhook Secret or caBundleCompare two rendered manifests and check whether the certificate data changes.Follow Provide a Stable Webhook Certificate.
CRD apply fails with an annotation-size errorCheck whether Argo CD is using client-side apply for the bundled Gateway API CRDs.Use server-side apply, or move CRDs to a dedicated platform Application.
First APISIX synchronization cannot resolve GatewayProxyConfirm that the finite retry policy is present.Preserve the retry policy so Argo CD retries after REST discovery refreshes.
GatewayProxy creation is rejected or controller logs report connection errorsConfirm that its Secret, endpoint or Service, network access, and CRD exist before reconciliation.Correct the Secret reference, connectivity, or dependency order, then synchronize again.
Controller resources return after manual deletion or editingCheck whether self-healing is enabled.Make the intended change in Git instead of editing the live object.
OpenShift rejects the pod or cluster-scoped resourcesReview SCC admission, destination namespace management, and AppProject permissions.Apply the verified OpenShift values and grant only the required platform permissions.