Skip to main content
API7.ai Logo

Pre-Define Prompt Templates

Prompt templates let administrators define reusable request bodies with placeholders for values supplied by clients. This standardizes prompts across applications while allowing each request to provide only the values that change.

Prompt templates are useful when clients should invoke an approved request shape instead of constructing the complete model payload. Administrators control fixed instructions, model options, and tools, while clients provide only the declared variables.

This guide shows how to use the ai-prompt-template plugin to create Chat Completions and Responses API templates with OpenAI. The same template mechanism can be adapted to other LLM providers when the downstream plugin supports their request format.

Prerequisite(s)​

  • Install Docker.
  • Install cURL to send requests for validation.
  • Follow the Getting Started Tutorial to start an APISIX instance in Docker or on Kubernetes.
  • Have an OpenAI account with API access to a model that supports both Chat Completions and Responses API web search.

Obtain an OpenAI API Key​

Create an OpenAI API key, then export the key and model:

export OPENAI_API_KEY="<your-api-key>"
export OPENAI_MODEL="<your-model-name>"

Configure a Chat Completions Template​

The ai-prompt-template plugin replaces the incoming request body with the selected JSON template. A template can use any request shape supported by the downstream plugin. This example configures a Chat Completions body containing a messages array.

To add fixed content to an existing Chat Completions or Responses API request without replacing its body, use ai-prompt-decorator.

Create the Chat Completions Template​

Create a route to the OpenAI API endpoint with a sample prompt template that accepts a user-defined prompt and answers in the specified complexity:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-prompt-template-route",
    "uri": "/anything",
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        },
        "options": {
          "model": "$OPENAI_MODEL"
        }
      },
      "ai-prompt-template": {
        "templates": [
          {
            "name": "QnA with complexity",
            "template": {
              "model": "$OPENAI_MODEL",
              "messages": [
                {
                  "role": "system",
                  "content": "Answer in {{complexity}}."
                },
                {
                  "role": "user",
                  "content": "Explain {{prompt}}."
                }
              ]
            }
          }
        ]
      }
    }
  }
EOF

❶ Name the template set. When requesting the route, the request should include the template name.

❷ Specify the model identifier.

❸ Configure a prompt that obtains the user-defined answer complexity from the request body key complexity.

❹ Configure a prompt that obtains the user-defined question from the request body key prompt.

Send a Request​

The route should now be available to be re-used to respond to a variety of questions with different levels of user-specified desired complexities.

Send a POST request to the route with a sample question and desired answer complexity in the request body:

curl "http://127.0.0.1:9080/anything" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "template_name": "QnA with complexity",
    "complexity": "brief",
    "prompt": "quick sort"
  }'

You should receive a response similar to the following:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Quick sort is a highly efficient sorting algorithm that uses a divide-and-conquer approach to arrange elements in a list or array in order. Here’s a brief explanation:\n\n1. **Choose a Pivot**: Select an element from the list as a 'pivot'. Common methods include choosing the first element, the last element, the middle element, or a random element.\n\n2. **Partitioning**: Rearrange the elements in the list such that all elements less than the pivot are moved before it, and all elements greater than the pivot are moved after it. The pivot is now in its final position.\n\n3. **Recursively Apply**: Recursively apply the same process to the sub-lists of elements to the left and right of the pivot.\n\nThe base case of the recursion is lists of size zero or one, which are already sorted.\n\nQuick sort has an average-case time complexity of O(n log n), making it suitable for large datasets. However, its worst-case time complexity is O(n^2), which occurs when the smallest or largest element is always chosen as the pivot. This can be mitigated by using good pivot selection strategies or randomization.",
        "role": "assistant"
      }
    }
  ],
  "created": 1723194057,
  "id": "chatcmpl-9uFmTYN4tfwaXZjyOQwcp0t5law4x",
  "model": "gpt-4o-2024-05-13",
  "object": "chat.completion",
  "system_fingerprint": "fp_abc28019ad",
  "usage": {
    "completion_tokens": 234,
    "prompt_tokens": 18,
    "total_tokens": 252
  }
}

Configure a Responses API Web Search Template​

A template can also generate a Responses API request that uses a built-in tool. This example provides a reusable web search that only searches the official Apache APISIX website.

The route URI may have a custom prefix, but it must end in /v1/responses so that APISIX can distinguish the generated body from an Embeddings request.

Create the Web Search Template​

Create a separate route with the web search template:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  --data-binary @- <<EOF
  {
    "id": "ai-prompt-template-responses-route",
    "uri": "/template/v1/responses",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer $OPENAI_API_KEY"
          }
        }
      },
      "ai-prompt-template": {
        "templates": [
          {
            "name": "Search APISIX documentation",
            "template": {
              "model": "$OPENAI_MODEL",
              "tools": [
                {
                  "type": "web_search",
                  "filters": {
                    "allowed_domains": ["apisix.apache.org"]
                  }
                }
              ],
              "tool_choice": "required",
              "input": "Search the official Apache APISIX website for {{topic}} and summarize the result in one sentence."
            }
          }
        ]
      }
    }
  }
EOF

❶ Preserve the /v1/responses suffix required for Responses API detection.

❷ Enable the Responses API web search tool.

❸ Restrict web search results to the official Apache APISIX website.

❹ Require the model to use a configured tool.

❺ Create the search request using a value from the incoming request.

Send a Request​

Send the template name and variable values to the route:

curl "http://127.0.0.1:9080/template/v1/responses" -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "template_name": "Search APISIX documentation",
    "topic": "what APISIX is"
  }'

Before ai-proxy processes the request, ai-prompt-template replaces the incoming body with the resolved template:

{
  "model": "<your-model-name>",
  "tools": [
    {
      "type": "web_search",
      "filters": {
        "allowed_domains": ["apisix.apache.org"]
      }
    }
  ],
  "tool_choice": "required",
  "input": "Search the official Apache APISIX website for what APISIX is and summarize the result in one sentence."
}

APISIX detects the generated body and route URI as a Responses API request. You should receive an HTTP 200 response. The output array contains a completed web search call followed by a message with URL citations:

{
  "status": "completed",
  "output": [
    {
      "type": "web_search_call",
      "status": "completed",
      "action": {
        "type": "search",
        "queries": [
          "site:apisix.apache.org what is Apache APISIX"
        ],
        "query": "site:apisix.apache.org what is Apache APISIX"
      }
    },
    {
      "type": "message",
      "status": "completed",
      "role": "assistant",
      "content": [
        {
          "type": "output_text",
          "text": "Apache APISIX is an open-source, high-performance API and AI gateway for managing traffic at scale through dynamic routing, load balancing, authentication, observability, rate limiting, and over 100 plugins. ([apisix.apache.org](https://apisix.apache.org/?utm_source=openai))",
          "annotations": [
            {
              "type": "url_citation",
              "start_index": 208,
              "end_index": 275,
              "title": "Apache APISIX - Open Source API Gateway & AI Gateway",
              "url": "https://apisix.apache.org/?utm_source=openai"
            }
          ]
        }
      ]
    }
  ]
}

Next Steps​

You have now learned how to pre-define prompt templates in APISIX when integrating with LLM service providers, such that the same route can be re-used to take different user inputs and serve a variety of purposes.

In addition, you can integrate more capabilities that APISIX offers, such as rate limiting and caching, to improve system availability and user experience.