ua-restriction
The ua-restriction plugin supports restricting access to upstream resources through either configuring an allowlist or denylist of user agents. A common use case is to prevent web crawlers from overloading the upstream resources and causing service degradation.
Examples
The examples below demonstrate how you can configure ua-restriction for different scenarios.
Reject Web Crawlers and Customize Error Message
The following example demonstrates how you can configure the plugin to fend off unwanted web crawlers and customize the rejection message.
- Admin API
- ADC
- Ingress Controller
Create a route as such:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "ua-restriction-route",
"uri": "/anything",
"plugins": {
"ua-restriction": {
"bypass_missing": false,
"denylist": [
"(Baiduspider)/(\\d+)\\.(\\d+)",
"bad-bot-1"
],
"message": "Access denied"
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org:80": 1
}
}
}'
services:
- name: ua-restriction-service
routes:
- name: ua-restriction-route
uris:
- /anything
plugins:
ua-restriction:
bypass_missing: false
denylist:
- "(Baiduspider)/(\\d+)\\.(\\d+)"
- "bad-bot-1"
message: "Access denied"
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Synchronize the configuration to the gateway:
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: ua-restriction-plugin-config
spec:
plugins:
- name: ua-restriction
config:
bypass_missing: false
denylist:
- "(Baiduspider)/(\\d+)\\.(\\d+)"
- "bad-bot-1"
message: "Access denied"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: ua-restriction-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: ua-restriction-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
Apply the configuration to your cluster:
kubectl apply -f ua-restriction-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: ua-restriction-route
spec:
ingressClassName: apisix
http:
- name: ua-restriction-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: ua-restriction
enable: true
config:
bypass_missing: false
denylist:
- "(Baiduspider)/(\\d+)\\.(\\d+)"
- "bad-bot-1"
message: "Access denied"
Apply the configuration to your cluster:
kubectl apply -f ua-restriction-ic.yaml
❶ Do not allow bypassing UA restriction rules.
❷ Configure user agents that should not be able to access the upstream resource.
❸ Customize the error message for when the access is denied.
Send a request to the route:
curl -i "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 200 OK response.
Send another request to the route with a disallowed user agent:
curl -i "http://127.0.0.1:9080/anything" -H 'User-Agent: Baiduspider/5.0'
You should receive an HTTP/1.1 403 Forbidden response with the following message:
{"message":"Access denied"}
Bypass UA Restriction Checks
The following example demonstrates how to configure the plugin to allow requests of a specific user agent to bypass the UA restriction.
- Admin API
- ADC
- Ingress Controller
Create a route as such:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${ADMIN_API_KEY}" \
-d '{
"id": "ua-restriction-route",
"uri": "/anything",
"plugins": {
"ua-restriction": {
"bypass_missing": true,
"allowlist": [
"good-bot-1"
],
"message": "Access denied"
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org:80": 1
}
}
}'
services:
- name: ua-restriction-service
routes:
- name: ua-restriction-route
uris:
- /anything
plugins:
ua-restriction:
bypass_missing: true
allowlist:
- "good-bot-1"
message: "Access denied"
upstream:
type: roundrobin
nodes:
- host: httpbin.org
port: 80
weight: 1
Synchronize the configuration to the gateway:
adc sync -f adc.yaml
- Gateway API
- APISIX CRD
apiVersion: v1
kind: Service
metadata:
namespace: aic
name: httpbin-external-domain
spec:
type: ExternalName
externalName: httpbin.org
---
apiVersion: apisix.apache.org/v1alpha1
kind: PluginConfig
metadata:
namespace: aic
name: ua-restriction-allowlist-plugin-config
spec:
plugins:
- name: ua-restriction
config:
bypass_missing: true
allowlist:
- "good-bot-1"
message: "Access denied"
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
namespace: aic
name: ua-restriction-route
spec:
parentRefs:
- name: apisix
rules:
- matches:
- path:
type: Exact
value: /anything
filters:
- type: ExtensionRef
extensionRef:
group: apisix.apache.org
kind: PluginConfig
name: ua-restriction-allowlist-plugin-config
backendRefs:
- name: httpbin-external-domain
port: 80
Apply the configuration to your cluster:
kubectl apply -f ua-restriction-ic.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixUpstream
metadata:
namespace: aic
name: httpbin-external-domain
spec:
ingressClassName: apisix
externalNodes:
- type: Domain
name: httpbin.org
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
namespace: aic
name: ua-restriction-route
spec:
ingressClassName: apisix
http:
- name: ua-restriction-route
match:
paths:
- /anything
upstreams:
- name: httpbin-external-domain
plugins:
- name: ua-restriction
enable: true
config:
bypass_missing: true
allowlist:
- "good-bot-1"
message: "Access denied"
Apply the configuration to your cluster:
kubectl apply -f ua-restriction-ic.yaml
❶ Allow bypassing UA restriction rules.
❷ Configure user agents that should be allowed to access the upstream resource.
Send a request to the route without modifying the user agent:
curl -i "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 403 Forbidden response with the following message:
{"message":"Access denied"}
Send another request to the route with an empty user agent:
curl -i "http://127.0.0.1:9080/anything" -H 'User-Agent: '
You should receive an HTTP/1.1 200 OK response.