Skip to main content
Version: Next

prometheus

Description#

The prometheus Plugin provides the capability to integrate APISIX with Prometheus.

After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a text-based exposition format to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.

Static Configurations#

By default, prometheus configurations are pre-configured in the default configuration.

The prompt- and completion-token histograms default to buckets at 1, 10, 50, 100, 200, 500, 1000, 2000, 5000, 10000, 20000, 50000, 100000, 200000, 500000, and 1000000 tokens. Configure llm_prompt_tokens_buckets and llm_completion_tokens_buckets when different boundaries better fit the workload.

To customize these values, add the corresponding configurations to config.yaml. For example:

plugin_attr:
  prometheus:                               # Plugin: prometheus attributes
    export_uri: /apisix/prometheus/metrics  # Set the URI for the Prometheus metrics endpoint.
    metric_prefix: apisix_                  # Set the prefix for Prometheus metrics generated by APISIX.
    enable_export_server: true              # Enable the Prometheus export server.
    export_addr:                            # Set the address for the Prometheus export server.
      ip: 127.0.0.1                         # Set the IP.
      port: 9091                            # Set the port.
    # metrics:                              # Create extra labels for metrics.
    #  http_status:                         # These metrics will be prefixed with `apisix_`.
    #    extra_labels:                      # Set the extra labels for http_status metrics.
    #      - upstream_addr: $upstream_addr
    #      - status: $upstream_status
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    #  http_latency:
    #    extra_labels:                      # Set the extra labels for http_latency metrics.
    #      - upstream_addr: $upstream_addr
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    #  bandwidth:
    #    extra_labels:                      # Set the extra labels for bandwidth metrics.
    #      - upstream_addr: $upstream_addr
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    # default_buckets:                      # Built-in `http_latency` histogram buckets in milliseconds when this key is omitted.
                                            # Uncomment the list only to override those defaults.
    #   - 1
    #   - 2
    #   - 5
    #   - 10
    #   - 20
    #   - 50
    #   - 100
    #   - 200
    #   - 500
    #   - 1000
    #   - 2000
    #   - 5000
    #   - 10000
    #   - 30000
    #   - 60000
    # llm_latency_buckets:                  # Buckets for `apisix_llm_latency`, in milliseconds.
                                            # Applies to both `type=total` and `type=ttft` in APISIX 3.18.0 and later.
    #   - 100
    #   - 500
    #   - 1000
    #   - 5000
    # llm_prompt_tokens_buckets:            # Buckets for `apisix_llm_prompt_tokens_dist`, in tokens.
    #   - 100
    #   - 1000
    #   - 10000
    # llm_completion_tokens_buckets:        # Buckets for `apisix_llm_completion_tokens_dist`, in tokens.
    #   - 100
    #   - 1000
    #   - 10000

You can use the Nginx variable to create extra_labels. See add extra labels.

Reload APISIX for changes to take effect.

Attribute#

NameTypeRequiredDefaultValid valuesDescription
prefer_namebooleanFalsefalseIf true, export Route/Service name instead of their ID in Prometheus metrics.

Metadata#

You can configure the Plugin through its Plugin Metadata, which is set dynamically through the Admin API and takes effect at runtime without a restart.

NameTypeRequiredDescription
disabled_labelsobjectFalsePer-metric map of built-in label names whose values are collapsed to an empty string "" to reduce metric cardinality. Keyed by metric name: http_status, http_latency, bandwidth, llm_latency, llm_prompt_tokens, llm_completion_tokens, llm_active_connections, llm_prompt_tokens_dist, llm_completion_tokens_dist, ai_cache_hits_total, ai_cache_misses_total, ai_cache_bypasses_total, ai_cache_embedding_latency. Structural labels that define a metric's identity (code on http_status, type on http_latency, bandwidth and llm_latency, layer on ai_cache_hits_total) cannot be disabled.

The following built-in labels can be disabled for each metadata key:

Metadata keyLabels that can be disabled
http_statusroute, matched_uri, matched_host, service, consumer, node, request_type, request_llm_model, llm_model, response_source
http_latencyroute, service, consumer, node, request_type, request_llm_model, llm_model
bandwidthroute, service, consumer, node, request_type, request_llm_model, llm_model
llm_latencyroute_id, service_id, consumer, node, request_type, request_llm_model, llm_model
llm_prompt_tokens, llm_completion_tokens, llm_prompt_tokens_dist, llm_completion_tokens_distroute_id, service_id, consumer, node, request_type, request_llm_model, llm_model
llm_active_connectionsroute, route_id, matched_uri, matched_host, service, service_id, consumer, node, request_type, request_llm_model, llm_model
ai_cache_hits_total, ai_cache_misses_total, ai_cache_bypasses_total, ai_cache_embedding_latencyroute, route_id, service, service_id, consumer, node, request_type, request_llm_model, llm_model

Collapsing a label's value to "" keeps the label registered in the metric schema, so existing dashboards, absent() alerts, and recording rules keep working — only the high-cardinality time series that differ solely by those labels are collapsed into one. This is useful in dynamic environments such as Kubernetes autoscaling, where the upstream node IP (node label) churns rapidly and would otherwise overflow the prometheus-metrics shared dict.

See Reduce Metric Cardinality by Disabling Labels for an example.

The request_llm_model label comes from the model requested by the client. The llm_model label identifies the effective target model: it uses the model configured on the AI instance when present, otherwise it uses the requested model. To bound cardinality, APISIX truncates each label value to 128 bytes before recording. If you do not need per-model breakdowns, list request_llm_model and llm_model under disabled_labels for the LLM metrics to collapse them to a single empty-valued series.

Metrics#

There are different types of metrics in Prometheus. To understand their differences, see metrics types.

The following table lists the core metrics registered directly by the prometheus Plugin. Configured xRPC protocols can register additional protocol-specific metrics. For example, Redis xRPC registers apisix_redis_commands_total and apisix_redis_commands_latency_seconds. See get APISIX metrics for an example. A metric series appears only after its data source is active. For example, Stream metrics require prometheus to be enabled as a Stream Plugin, LLM metrics require AI traffic, AI cache metrics require the ai-cache Plugin, and apisix_batch_process_entries requires a batch-processing Plugin to have data.

NameTypeDescription
apisix_bandwidthcounterTotal amount of traffic flowing through APISIX in bytes.
apisix_etcd_modify_indexesgaugeNumber of changes to etcd by APISIX keys.
apisix_batch_process_entriesgaugeNumber of remaining entries in a batch when sending data in batches, such as with http logger, and other logging Plugins.
apisix_etcd_reachablegaugeWhether APISIX can reach etcd. A value of 1 represents reachable and 0 represents unreachable.
apisix_http_statuscounterHTTP status codes returned to clients after Plugin processing and proxying. This value can differ from the Upstream status.
apisix_http_requests_totalgaugeNumber of HTTP requests from clients.
apisix_nginx_http_current_connectionsgaugeNumber of current connections with clients.
apisix_nginx_metric_errors_totalcounterTotal number of nginx-lua-prometheus errors.
apisix_http_latencyhistogramHTTP request latency in milliseconds.
apisix_llm_latencyhistogramLLM request latency in milliseconds, including total latency and time to first token.
apisix_llm_prompt_tokenscounterTotal number of prompt tokens consumed by LLM requests.
apisix_llm_completion_tokenscounterTotal number of completion tokens generated by LLM requests.
apisix_llm_active_connectionsgaugeGauge of LLM upstream activity. It reflects active requests without fallback retries, but ai-proxy-multi fallback retries can overcount failed-instance series. See the label reference for details.
apisix_llm_prompt_tokens_disthistogramDistribution of prompt tokens consumed per LLM request.
apisix_llm_completion_tokens_disthistogramDistribution of completion tokens generated per LLM request.
apisix_ai_cache_hits_totalcounterTotal number of AI cache hits, grouped by cache layer.
apisix_ai_cache_misses_totalcounterTotal number of AI cache misses.
apisix_ai_cache_bypasses_totalcounterTotal number of requests that bypassed the AI cache.
apisix_ai_cache_embedding_latencyhistogramLatency of AI cache embedding calls in milliseconds.
apisix_node_infogaugeInformation of the APISIX node, such as host name and the current APISIX version.
apisix_shared_dict_capacity_bytesgaugeThe total capacity of an NGINX shared dictionary.
apisix_shared_dict_free_space_bytesgaugeThe remaining space in an NGINX shared dictionary.
apisix_upstream_statusgaugeHealth check status of upstream nodes, available if health checks are configured on the upstream. A value of 1 represents healthy and 0 represents unhealthy.
apisix_stream_connection_totalcounterTotal number of connections handled per Stream Route.
apisix_stream_active_connectionsgaugeNumber of Stream sessions currently being proxied, per listening address. Covers TCP connections and UDP sessions alike.
apisix_stream_statuscounterCounted once per Stream session when it ends, classified by how it ended.
apisix_stream_bandwidthcounterTotal bandwidth in bytes proxied by the Stream subsystem, per listening address and direction.

Labels#

Labels are attributes of metrics that are used to differentiate metrics.

For example, the apisix_http_status metric can be labeled with route information to identify which Route the HTTP status originates from.

The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.

Labels for apisix_http_status#

The following labels are used to differentiate apisix_http_status metrics.

NameDescription
codeHTTP response code returned to the client.
routeID of the Route that the HTTP status originates from when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
matched_uriURI of the Route that matches the request. Default to an empty string if a request does not match any Route.
matched_hostHost of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route.
serviceID of the Service that the HTTP status originates from when prefer_name is false (default), and name of the Service when prefer_name to true. Default to the configured value of host on the Route if the matched Route does not belong to any Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIP address of the Upstream node. For AI Routes, this is the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: traditional_http, ai_chat, or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client. Empty for traditional HTTP traffic.
response_sourceResponse origin: apisix for responses generated by APISIX, nginx for NGINX proxy errors, or upstream for responses received from the Upstream.

Labels for apisix_stream_active_connections#

The gauge is incremented when a session is accepted and decremented when it ends, so it reflects live concurrency without waiting for sessions to finish.

NameDescription
listen_addrListening address the client connected to, for example 0.0.0.0:9100.

Labels for apisix_stream_status#

NGINX reports a Stream $status of 200 for failures that happen after the upstream connection is established, so a timeout or a reset is indistinguishable from a clean close on its own. This metric maps recognized termination reasons onto the status codes NGINX uses for Stream sessions. A code of 200 also covers worker shutdown and is the fallback when the runtime does not record a recognized termination reason, so it does not always prove a clean close. No synthetic code is introduced.

NameDescription
codeHow the session ended: 200 for a normal close, worker shutdown, or a missing or unrecognized termination reason; 400 for a client-side problem such as a reset or invalid preread data; 403 when rejected by an access rule; 500 for an internal error; 502 for an upstream or transport problem such as a connect failure, reset, or idle timeout; 503 when rejected by a connection limit.
listen_addrListening address the client connected to, for example 0.0.0.0:9100.
nodeAddress of the upstream node used, empty when no node was selected.

For UDP only a subset of the codes occurs, since UDP has no close, FIN or reset to observe.

Labels for apisix_stream_bandwidth#

Bytes keep accumulating while a connection is open rather than only at session end, so a long-lived connection is visible as it runs. Only Stream traffic is counted; the HTTP subsystem cannot contribute to it.

NameDescription
listen_addrListening address the client connected to, for example 0.0.0.0:9100.
sideWhich connection the bytes crossed: downstream between APISIX and the client, upstream between APISIX and the upstream.
typeDirection relative to APISIX, matching apisix_bandwidth: ingress for bytes APISIX received, egress for bytes APISIX sent.

Under plain forwarding downstream/ingress tracks upstream/egress, and upstream/ingress tracks downstream/egress; a lasting mismatch is itself a signal that one side stopped reading.

Labels for apisix_bandwidth#

The following labels are used to differentiate apisix_bandwidth metrics.

NameDescription
typeType of traffic, egress or ingress.
routeID of the Route that bandwidth corresponds to when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
serviceID of the Service that bandwidth corresponds to when prefer_name is false (default), and name of the Service when prefer_name to true. Default to the configured value of host on the Route if the matched Route does not belong to any Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIP address of the Upstream node. For AI Routes, this is the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: traditional_http, ai_chat, or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client. Empty for traditional HTTP traffic.

Labels for apisix_llm_latency#

The type label distinguishes the kind of latency, similar to apisix_http_latency:

  • total: the full response latency, recorded for both ai_chat and ai_stream requests.
  • ttft: the time to first token, recorded for ai_stream requests only (non-streaming responses do not expose a first-token moment).

For request-scoped LLM metrics, the labels named route_id and service_id follow prefer_name: they contain IDs by default and names when prefer_name is true. apisix_llm_active_connections is different because it exports separate name and ID labels.

NameDescription
typeKind of latency: total or ttft.
route_idID of the Route that the metric corresponds to when prefer_name is false (default), and name of the Route when prefer_name is true. Defaults to an empty string if a request does not match any Route.
service_idID of the Service that the metric corresponds to when prefer_name is false (default), and name of the Service when prefer_name is true. Defaults to an empty string if the matched Route does not belong to any Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIdentifier of the selected AI Upstream, typically the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client.

Labels for apisix_llm_active_connections#

APISIX increments this gauge when an LLM upstream attempt starts and decrements it once during the request log phase. Without fallback retries, it represents active LLM upstream requests. With ai-proxy-multi fallback retries, each retry increments a new per-instance series, while the request is decremented only once using the final labels. A failed-instance series can therefore remain above the actual active count and, with the default non-expiring metric, remain until the metric storage is reset. Treat this gauge as approximate when fallback retries occur.

NameDescription
routeName of the matched Route. Defaults to an empty string if the Route has no name or no Route is matched.
route_idID of the matched Route. Defaults to an empty string if no Route is matched.
matched_uriURI of the Route that matches the request. Default to an empty string if a request does not match any Route.
matched_hostHost of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route.
serviceName of the Service referenced by the matched Route. Defaults to an empty string if the Route does not reference a Service.
service_idID of the Service referenced by the matched Route. Defaults to an empty string if the Route does not reference a Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIdentifier of the selected AI Upstream, typically the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client.

Labels for apisix_llm_completion_tokens#

NameDescription
route_idID of the Route that the metric corresponds to when prefer_name is false (default), and name of the Route when prefer_name is true. Defaults to an empty string if no Route is matched.
service_idID of the Service that the metric corresponds to when prefer_name is false (default), and name of the Service when prefer_name is true. Defaults to an empty string if the matched Route does not reference a Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIdentifier of the selected AI Upstream, typically the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client.

Labels for apisix_llm_prompt_tokens#

NameDescription
route_idID of the Route that the metric corresponds to when prefer_name is false (default), and name of the Route when prefer_name is true. Defaults to an empty string if no Route is matched.
service_idID of the Service that the metric corresponds to when prefer_name is false (default), and name of the Service when prefer_name is true. Defaults to an empty string if the matched Route does not reference a Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIdentifier of the selected AI Upstream, typically the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client.

Labels for apisix_llm_prompt_tokens_dist#

apisix_llm_prompt_tokens_dist is a histogram of prompt tokens consumed per request, complementing the apisix_llm_prompt_tokens counter with a distribution so that quantiles (such as p95 prompt size) can be computed.

NameDescription
route_idID of the Route that the metric corresponds to when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
service_idID of the Service that the metric corresponds to when prefer_name is false (default), and name of the Service when prefer_name is true. Defaults to an empty string if the matched Route does not reference a Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIdentifier of the selected AI Upstream, typically the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client.

Labels for apisix_llm_completion_tokens_dist#

apisix_llm_completion_tokens_dist is a histogram of completion tokens generated per request, complementing the apisix_llm_completion_tokens counter with a distribution.

NameDescription
route_idID of the Route that the metric corresponds to when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
service_idID of the Service that the metric corresponds to when prefer_name is false (default), and name of the Service when prefer_name is true. Defaults to an empty string if the matched Route does not reference a Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeIdentifier of the selected AI Upstream, typically the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client.

Labels for the apisix_ai_cache_* metrics#

The ai-cache Plugin exports four metrics:

  • apisix_ai_cache_hits_total counts requests served from the cache, per serving layer.
  • apisix_ai_cache_misses_total counts requests the Plugin looked up but could not serve from the cache.
  • apisix_ai_cache_bypasses_total counts requests that bypassed the cache lookup entirely.
  • apisix_ai_cache_embedding_latency is a histogram of the latency, in milliseconds, of the embedding calls made by the semantic layer, measured around the embedding provider round-trip for successful and failed calls alike.

They share the following labels:

NameDescription
layerOnly on apisix_ai_cache_hits_total. Cache layer that served the hit: exact or semantic.
routeName of the Route that the metric corresponds to. Default to an empty string if the Route has no name or a request does not match any Route.
route_idID of the Route that the metric corresponds to. Default to an empty string if a request does not match any Route.
serviceName of the Service that the matched Route belongs to. Default to an empty string if the matched Route does not belong to any Service.
service_idID of the Service that the matched Route belongs to. Default to an empty string if the matched Route does not belong to any Service.
consumerName of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.
nodeName of the LLM instance picked by the ai-proxy or ai-proxy-multi Plugin, such as ai-proxy-openai. These Plugins report the instance name instead of an upstream IP address, on cache hits and misses alike.
request_typeAI request category: ai_chat or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the Upstream AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client. Empty on cache hits, which are served without reaching the LLM.

Labels for apisix_http_latency#

The following labels are used to differentiate apisix_http_latency metrics.

NameDescription
typeType of latencies. See latency types for details.
routeID of the Route that latencies correspond to when prefer_name is false (default), and name of the Route when prefer_name to true. Default to an empty string if a request does not match any Route.
serviceID of the Service that latencies correspond to when prefer_name is false (default), and name of the Service when prefer_name to true. Default to the configured value of host on the Route if the matched Route does not belong to any Service.
consumerName of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request.
nodeIP address of the Upstream node associated with latencies. For AI Routes, this is the LLM instance name reported by ai-proxy or ai-proxy-multi.
request_typeRequest category: traditional_http, ai_chat, or ai_stream.
request_llm_modelModel name requested by the client.
llm_modelEffective target model for the AI request. Uses the model configured on the AI instance when present; otherwise uses the model requested by the client. Empty for traditional HTTP traffic.

Latency Types#

apisix_http_latency can be labeled with one of the three types:

  • request represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.

  • upstream represents the time elapsed waiting on responses from the upstream Service.

  • apisix represents the difference between the request latency and upstream latency.

In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:

APISIX latency
  = downstream request time - upstream response time
  = downstream traffic latency + NGINX latency

Labels for apisix_upstream_status#

The following labels are used to differentiate apisix_upstream_status metrics.

NameDescription
nameResource ID corresponding to the upstream configured with health checks, such as /apisix/routes/1 and /apisix/upstreams/1.
ipIP address of the upstream node.
portPort number of the node.

Examples#

The examples below demonstrate how you can work with the prometheus Plugin for different scenarios.

Get APISIX Metrics#

The following example demonstrates how you can get metrics from APISIX.

The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the static configuration. If you would like to customize these configuration, update config.yaml and reload APISIX.

If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:

conf/config.yaml
plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
# TYPE apisix_bandwidth counter
apisix_bandwidth{type="egress",route="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model=""} 8417
apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 1420
apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 1420
apisix_bandwidth{type="ingress",route="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model=""} 189
apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 332
apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1",request_type="traditional_http",request_llm_model="",llm_model=""} 332
# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
...

Expose APISIX Metrics on Public API Endpoint#

The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port 9091, and expose APISIX Prometheus metrics on a new public API endpoint on port 9080, which APISIX uses to listen to other client requests.

caution

If a large quantity of metrics are being collected, the Plugin could take up a significant amount of CPU resources for metric computations and negatively impact the processing of regular requests.

To address this issue, APISIX uses privileged agent and offloads metric computations to a separate process. This optimization applies automatically if you use the metric endpoint configured in the configuration files, as demonstrated above. When you expose the metric endpoint with the public-api Plugin, the offloading and cached metric serving still apply; however, the endpoint is then exposed on the public listener and requests to it go through the normal public API request path, which can add request-path overhead compared with the dedicated export server.

Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:

conf/config.yaml
plugin_attr:
  prometheus:
    enable_export_server: false

Next, create a Route with public-api Plugin and expose a public API endpoint for APISIX metrics:

curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/apisix/prometheus/metrics",
    "plugins": {
      "public-api": {}
    }
  }'

Send a request to the new metrics endpoint to verify:

curl "http://127.0.0.1:9080/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_requests_total The total number of client requests since APISIX started
# TYPE apisix_http_requests_total gauge
apisix_http_requests_total 1
# HELP apisix_nginx_http_current_connections Number of HTTP connections
# TYPE apisix_nginx_http_current_connections gauge
apisix_nginx_http_current_connections{state="accepted"} 1
apisix_nginx_http_current_connections{state="active"} 1
apisix_nginx_http_current_connections{state="handled"} 1
apisix_nginx_http_current_connections{state="reading"} 0
apisix_nginx_http_current_connections{state="waiting"} 0
apisix_nginx_http_current_connections{state="writing"} 1
...

Monitor Upstream Health Statuses#

The following example demonstrates how to monitor the health status of upstream nodes.

Create a Route with the prometheus Plugin and configure upstream active health checks:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1,
        "127.0.0.1:20001": 1
      },
      "checks": {
        "active": {
          "timeout": 5,
          "http_path": "/status",
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 2
          }
        },
        "passive": {
          "healthy": {
            "http_statuses": [200, 201],
            "successes": 3
          },
          "unhealthy": {
            "http_statuses": [500],
            "http_failures": 3,
            "tcp_failures": 3
          }
        }
      }
    }
  }'

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_upstream_status upstream status from health check
# TYPE apisix_upstream_status gauge
apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0

This shows that the upstream node httpbin.org:80 is healthy and the upstream node 127.0.0.1:20001 is unhealthy.

Add Extra Labels for Metrics#

The following example demonstrates how to add additional labels to metrics and use the Nginx variable in label values.

Extra labels are supported for apisix_http_status, apisix_http_latency, apisix_bandwidth, all apisix_llm_* metrics listed above, and all four apisix_ai_cache_* metrics.

Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:

conf/config.yaml
plugin_attr:
  prometheus:                                # Plugin: prometheus
    metrics:                                 # Create extra labels from the NGINX variables.
      http_status:
        extra_labels:                        # Set the extra labels for http_status metrics.
          - upstream_addr: $upstream_addr    # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
          - route_name: $route_name          # Add an extra route_name label with value being the APISIX variable $route_name.

Note that if you define a variable in the label value but it does not correspond to any existing APISIX variables and Nginx variable, the label value will default to an empty string.

Create a Route with the prometheus Plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "name": "extra-label",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the Route to verify:

curl -i "http://127.0.0.1:9080/get"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_http_status HTTP status codes per Service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1

Reduce Metric Cardinality by Disabling Labels#

The following example demonstrates how to reduce metric cardinality by collapsing the values of selected built-in labels to an empty string "" using the Plugin Metadata. This is useful in dynamic environments such as Kubernetes autoscaling, where the upstream node IP (node label) churns rapidly and would otherwise overflow the prometheus-metrics shared dict.

Collapsing a label's value keeps the label registered in the metric schema, so existing dashboards, absent() alerts, and recording rules keep working. Structural labels that define a metric's identity (code on http_status, type on http_latency, bandwidth and llm_latency) cannot be disabled.

Create a Route with the prometheus Plugin:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Configure the Plugin Metadata to collapse the node and consumer labels on apisix_http_status and the node label on apisix_http_latency:

curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/prometheus" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "disabled_labels": {
      "http_status": ["node", "consumer"],
      "http_latency": ["node"]
    }
  }'

Send a request to the Route to verify:

curl -i "http://127.0.0.1:9080/get"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see that node and consumer are collapsed to empty strings on apisix_http_status, while metrics that are not listed (such as apisix_bandwidth) keep all their label values:

# HELP apisix_http_status HTTP status codes per service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="prometheus-route",matched_uri="/get",matched_host="",service="",consumer="",node="",request_type="traditional_http",request_llm_model="",llm_model="",response_source="upstream"} 1

Monitor TCP/UDP Traffic with Prometheus#

The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.

Include the following configurations in config.yaml to enable Stream proxy and add prometheus to the existing Stream Plugin list. Preserve any other Stream Plugins used by the deployment. Reload APISIX for changes to take effect:

conf/config.yaml
apisix:
  proxy_mode: http&stream   # Enable both L4 & L7 proxies
  stream_proxy:             # Configure L4 proxy
    tcp:
      - 9100                # Set TCP proxy listening port
    udp:
      - 9200                # Set UDP proxy listening port

stream_plugins:
  - prometheus              # Enable prometheus for stream proxy

Create a Stream Route with the prometheus Plugin:

curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
    "plugins": {
      "prometheus":{}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'

Send a request to the Stream Route to verify:

curl -i "http://127.0.0.1:9100"

You should see an HTTP/1.1 200 OK response.

Send a request to the APISIX Prometheus metrics endpoint:

curl "http://127.0.0.1:9091/apisix/prometheus/metrics"

You should see an output similar to the following:

# HELP apisix_stream_connection_total Total number of connections handled per Stream Route in APISIX
# TYPE apisix_stream_connection_total counter
apisix_stream_connection_total{route="prometheus-route"} 1
# HELP apisix_stream_active_connections Number of stream sessions currently being proxied per listening address
# TYPE apisix_stream_active_connections gauge
apisix_stream_active_connections{listen_addr="0.0.0.0:9100"} 0
# HELP apisix_stream_status Stream sessions per termination status in APISIX
# TYPE apisix_stream_status counter
apisix_stream_status{code="200",listen_addr="0.0.0.0:9100",node="54.237.103.220:80"} 1
# HELP apisix_stream_bandwidth Total bandwidth in bytes proxied by the stream subsystem in APISIX
# TYPE apisix_stream_bandwidth counter
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="ingress",side="downstream"} 78
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="egress",side="downstream"} 219
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="egress",side="upstream"} 78
apisix_stream_bandwidth{listen_addr="0.0.0.0:9100",type="ingress",side="upstream"} 219

The exact Upstream address and byte counts depend on the request. The active-connections gauge is 0 above because the request completed before the scrape; scrape while a connection remains open to observe a positive value.

note

apisix_stream_active_connections and apisix_stream_bandwidth are backed by an NGINX shared memory zone, sized by nginx_config.stream.metrics_zone_size (default 1m). They require APISIX-Runtime; on a runtime without it the two metrics are simply not published.