ua-restriction

The ua-restriction plugin supports restricting access to upstream resources through either configuring an allowlist or denylist of user agents. A common use case is to prevent web crawlers from overloading the upstream resources and causing service degradation.

Examples

The examples below demonstrate how you can configure ua-restriction for different scenarios.

Reject Web Crawlers and Customize Error Message

The following example demonstrates how you can configure the plugin to fend off unwanted web crawlers and customize the rejection message.

Admin API
ADC
Ingress Controller

Create a route as such:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ua-restriction-route",
    "uri": "/anything",
    "plugins": {
      "ua-restriction": {
        "bypass_missing": false,
        "denylist": [
          "(Baiduspider)/(\\d+)\\.(\\d+)",
          "bad-bot-1"
        ],
        "message": "Access denied"
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

adc.yaml
services:
  - name: ua-restriction-service
    routes:
      - name: ua-restriction-route
        uris:
          - /anything
        plugins:
          ua-restriction:
            bypass_missing: false
            denylist:
              - "(Baiduspider)/(\\d+)\\.(\\d+)"
              - "bad-bot-1"
            message: "Access denied"
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

ua-restriction-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: ua-restriction-plugin-config
spec:
  plugins:
    - name: ua-restriction
      config:
        bypass_missing: false
        denylist:
          - "(Baiduspider)/(\\d+)\\.(\\d+)"
          - "bad-bot-1"
        message: "Access denied"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: ua-restriction-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: ua-restriction-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

Apply the configuration to your cluster:

kubectl apply -f ua-restriction-ic.yaml

ua-restriction-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: ua-restriction-route
spec:
  ingressClassName: apisix
  http:
    - name: ua-restriction-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: ua-restriction
        enable: true
        config:
          bypass_missing: false
          denylist:
            - "(Baiduspider)/(\\d+)\\.(\\d+)"
            - "bad-bot-1"
          message: "Access denied"

Apply the configuration to your cluster:

kubectl apply -f ua-restriction-ic.yaml

❶ Do not allow bypassing UA restriction rules.

❷ Configure user agents that should not be able to access the upstream resource.

❸ Customize the error message for when the access is denied.

Send a request to the route:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 200 OK response.

Send another request to the route with a disallowed user agent:

curl -i "http://127.0.0.1:9080/anything" -H 'User-Agent: Baiduspider/5.0'

You should receive an HTTP/1.1 403 Forbidden response with the following message:

{"message":"Access denied"}

Bypass UA Restriction Checks

The following example demonstrates how to configure the plugin to allow requests of a specific user agent to bypass the UA restriction.

Admin API
ADC
Ingress Controller

Create a route as such:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ua-restriction-route",
    "uri": "/anything",
    "plugins": {
      "ua-restriction": {
        "bypass_missing": true,
        "allowlist": [
          "good-bot-1"
        ],
        "message": "Access denied"
      }
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

adc.yaml
services:
  - name: ua-restriction-service
    routes:
      - name: ua-restriction-route
        uris:
          - /anything
        plugins:
          ua-restriction:
            bypass_missing: true
            allowlist:
              - "good-bot-1"
            message: "Access denied"
    upstream:
      type: roundrobin
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1

Synchronize the configuration to the gateway:

adc sync -f adc.yaml

Gateway API
APISIX CRD

ua-restriction-ic.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  type: ExternalName
  externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
  namespace: aic
  name: ua-restriction-allowlist-plugin-config
spec:
  plugins:
    - name: ua-restriction
      config:
        bypass_missing: true
        allowlist:
          - "good-bot-1"
        message: "Access denied"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: aic
  name: ua-restriction-route
spec:
  parentRefs:
    - name: apisix
  rules:
    - matches:
        - path:
            type: Exact
            value: /anything
      filters:
        - type: ExtensionRef
          extensionRef:
            group: apisix.apache.org
            kind: PluginConfig
            name: ua-restriction-allowlist-plugin-config
      backendRefs:
        - name: httpbin-external-domain
          port: 80

Apply the configuration to your cluster:

kubectl apply -f ua-restriction-ic.yaml

ua-restriction-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
  namespace: aic
  name: httpbin-external-domain
spec:
  ingressClassName: apisix
  externalNodes:
  - type: Domain
    name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  namespace: aic
  name: ua-restriction-route
spec:
  ingressClassName: apisix
  http:
    - name: ua-restriction-route
      match:
        paths:
          - /anything
      upstreams:
      - name: httpbin-external-domain
      plugins:
      - name: ua-restriction
        enable: true
        config:
          bypass_missing: true
          allowlist:
            - "good-bot-1"
          message: "Access denied"

Apply the configuration to your cluster:

kubectl apply -f ua-restriction-ic.yaml

❶ Allow bypassing UA restriction rules.

❷ Configure user agents that should be allowed to access the upstream resource.

Send a request to the route without modifying the user agent:

curl -i "http://127.0.0.1:9080/anything"

You should receive an HTTP/1.1 403 Forbidden response with the following message:

{"message":"Access denied"}

Send another request to the route with an empty user agent:

curl -i "http://127.0.0.1:9080/anything" -H 'User-Agent: '

You should receive an HTTP/1.1 200 OK response.

Examples​

Reject Web Crawlers and Customize Error Message​

Bypass UA Restriction Checks​

Examples

Reject Web Crawlers and Customize Error Message

Bypass UA Restriction Checks