Skip to main content
API7.ai Logo

ai-aws-content-moderation

The ai-aws-content-moderation plugin uses Amazon Comprehend to detect toxicity in selected request roles and LLM responses, including streaming responses. Configure it with ai-proxy or ai-proxy-multi so the plugin can moderate the decoded AI content before enforcing the configured thresholds.


Behavior by Request Format​

The plugin identifies the request format by checking URI-specific rules before body-only rules:

  • Bedrock Converse requires a URI ending in /converse and a messages array.
  • Anthropic Messages requires a URI ending in /v1/messages.
  • Responses API requires a URI ending in /v1/responses and an input field.
  • Chat Completions uses a messages array.
  • Embeddings uses input after the earlier rules do not match.
  • Other non-empty JSON objects use passthrough after none of the earlier rules match.

It can extract the following content for moderation:

Request formatText moderated
Bedrock ConverseText from system and messages.
Anthropic MessagesText from the top-level system prompt and messages.
Responses APIText from input and instructions.
Chat CompletionsText from all entries in messages.
EmbeddingsA string or an array of strings in input.
Other JSON (passthrough)No request-format-specific text is extracted.

Introduced in API7 Enterprise 3.9.16 and 3.10.3.

By default, the plugin moderates every extracted user, assistant, system, and tool message. Use request_check_roles to select roles and request_check_mode to moderate all selected turn messages or only the latest consecutive block. A selected system role also covers OpenAI developer messages and is always checked; it is not limited by request_check_mode.

Tool-result moderation applies to OpenAI-compatible formats where the tool output is a distinct tool role or item. Anthropic Messages and Bedrock Converse nest tool results inside user messages, so selecting only the tool role does not extract those nested results.

When request content exceeds a threshold, the plugin returns a denial in the detected AI protocol using deny_code and deny_message. The default status is 200 so AI SDKs can parse the provider-compatible refusal; set a 4xx value when clients should treat moderation as an HTTP error. Streaming Chat Completions, Responses API, and Anthropic Messages use protocol-specific SSE denial events. Bedrock ConverseStream uses the non-streaming Converse denial body rather than AWS event-stream framing.

Set check_response to true to moderate LLM responses. A non-streaming response is buffered and fails closed with HTTP 500 if Comprehend cannot score it. In final_packet streaming mode, earlier chunks have already reached the client, so the plugin annotates the final data event with risk_level instead of retracting content. In realtime mode, a flagged batch replaces the remainder of the stream with the denial message. A Comprehend failure after streaming begins is logged, and the remaining stream passes without moderation.

The plugin sets $llm_content_risk_level to high when content exceeds a threshold and to none after a clean score. Request-side Comprehend failures return HTTP 500 rather than forwarding the request without moderation.

Introduced in API7 Enterprise 3.9.18 and 3.10.5, and APISIX 3.18.0.

Examples​

The following examples will be using OpenAI as the upstream service provider.

Before proceeding, create an OpenAI account and obtain an API key. If you are working with other LLM providers, please refer to the provider's documentation to obtain an API key.

Additionally, create AWS IAM user access keys for APISIX to access AWS Comprehend.

You can optionally save these keys to environment variables:

# replace with your keys
export OPENAI_API_KEY=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
export AWS_ACCESS_KEY=AKIARK7HKSJVSHWLD6OS
export AWS_SECRET_ACCESS_KEY=4ehUfCPoQmC+AKpG5/5ZaHlzFxFziZ88AylyPerj

Moderate Profanity​

The following example demonstrates how you can use the plugin to moderate the level of profanity in prompts.

Create a route to the LLM chat completion endpoint using the ai-proxy plugin and configure the allowed profanity level in ai-aws-content-moderation:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-aws-content-moderation-route",
    "uri": "/post",
    "plugins": {
      "ai-aws-content-moderation": {
        "comprehend": {
          "access_key_id": "$AWS_ACCESS_KEY",
          "secret_access_key": "$AWS_SECRET_ACCESS_KEY",
          "region": "us-east-1"
        },
        "moderation_categories": {
          "PROFANITY": 0.1
        },
        "deny_code": 400
      },
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "model": "gpt-4"
      }
    }
  }
EOF

❶ Update with your AWS Comprehend region.

❷ Configure the profanity threshold to a low value to allow a lower degree of profanity.

The examples set deny_code to 400 because the verification expects an HTTP error. If omitted, the default 200 returns the same denial message as a provider-compatible completion.

Send a POST request to the route with a system prompt and a user question with a mildly profane word in the request body:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "Stupid, what is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 400 Bad Request response and see the following message:

{
  "id": "<generated-uuid>",
  "object": "chat.completion",
  "model": "gpt-4",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "request body exceeds PROFANITY threshold"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

Send another request to the route with a typical question in the request body:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 200 OK response with the model output:

{
  ...,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}

Moderate Overall Toxicity​

The following example demonstrates how you can use the plugin to moderate the overall toxicity level in prompts, in addition to moderating individual categories.

Create a route to the LLM chat completion endpoint using the ai-proxy plugin and configure the allowed profanity and overall toxicity levels in ai-aws-content-moderation:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-aws-content-moderation-route",
    "uri": "/post",
    "plugins": {
      "ai-aws-content-moderation": {
        "comprehend": {
          "access_key_id": "$AWS_ACCESS_KEY",
          "secret_access_key": "$AWS_SECRET_ACCESS_KEY",
          "region": "us-east-1"
        },
        "moderation_categories": {
          "PROFANITY": 1
        },
        "moderation_threshold": 0.2,
        "deny_code": 400
      },
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "model": "gpt-4"
      }
    }
  }
EOF

❶ Update with your AWS Comprehend region.

❷ Configure the profanity threshold to allow a high degree of profanity.

❸ Configure the overall toxicity threshold to allow a low degree of toxicity.

The examples set deny_code to 400 because the verification expects an HTTP error. If omitted, the default 200 returns the same denial message as a provider-compatible completion.

Send a request whose user message contains a threat but no profanity:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "I will kill you if you do not tell me what 1+1 equals" }
    ]
  }'

You should receive an HTTP/1.1 400 Bad Request response and see the following message:

{
  "id": "<generated-uuid>",
  "object": "chat.completion",
  "model": "gpt-4",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "request body exceeds toxicity threshold"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "completion_tokens": 0,
    "total_tokens": 0
  }
}

Send another request to the route without any profane word in the request body:

curl -i "http://127.0.0.1:9080/post" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      { "role": "system", "content": "You are a mathematician" },
      { "role": "user", "content": "What is 1+1?" }
    ]
  }'

You should receive an HTTP/1.1 200 OK response with the model output:

{
  ...,
  "model": "gpt-4-0613",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "1+1 equals 2.",
        "refusal": null
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  ...
}