opentelemetry
#
DescriptionThe opentelemetry
Plugin can be used to report tracing data according to the OpenTelemetry Specification.
The Plugin only supports binary-encoded OLTP over HTTP.
#
ConfigurationsBy default, configurations of the Service name, tenant ID, collector, and batch span processor are pre-configured in default configuration.
You can change this configuration of the Plugin through the endpoint apisix/admin/plugin_metadata/opentelemetry
For example:
note
You can fetch the admin_key
from config.yaml
and save to an environment variable with the following command:
admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry -H "X-API-KEY: $admin_key" -X PUT -d '
{
"trace_id_source": "x-request-id",
"resource": {
"service.name": "APISIX"
},
"collector": {
"address": "127.0.0.1:4318",
"request_timeout": 3,
"request_headers": {
"Authorization": "token"
}
},
"batch_span_processor": {
"drop_on_queue_full": false,
"max_queue_size": 1024,
"batch_timeout": 2,
"inactive_timeout": 1,
"max_export_batch_size": 16
},
"set_ngx_var": false
}'
#
AttributesName | Type | Required | Default | Valid Values | Description |
---|---|---|---|---|---|
sampler | object | False | - | - | Sampling configuration. |
sampler.name | string | False | always_off | ["always_on", "always_off", "trace_id_ratio", "parent_base"] | Sampling strategy. To always sample, use always_on .To never sample, use always_off .To randomly sample based on a given ratio, use trace_id_ratio .To use the sampling decision of the span's parent, use parent_base . If there is no parent, use the root sampler. |
sampler.options | object | False | - | - | Parameters for sampling strategy. |
sampler.options.fraction | number | False | 0 | [0, 1] | Sampling ratio when the sampling strategy is trace_id_ratio . |
sampler.options.root | object | False | - | - | Root sampler when the sampling strategy is parent_base strategy. |
sampler.options.root.name | string | False | - | ["always_on", "always_off", "trace_id_ratio"] | Root sampling strategy. |
sampler.options.root.options | object | False | - | - | Root sampling strategy parameters. |
sampler.options.root.options.fraction | number | False | 0 | [0, 1] | Root sampling ratio when the sampling strategy is trace_id_ratio . |
additional_attributes | array[string] | False | - | - | Additional attributes appended to the trace span. Support built-in variables in values. |
additional_header_prefix_attributes | array[string] | False | - | - | Headers or header prefixes appended to the trace span's attributes. For example, use x-my-header" or x-my-headers-* to include all headers with the prefix x-my-headers- . |
#
ExamplesThe examples below demonstrate how you can work with the opentelemetry
Plugin for different scenarios.
opentelemetry
Plugin#
Enable By default, the opentelemetry
Plugin is disabled in APISIX. To enable, add the Plugin to your configuration file as such:
plugins:
- ...
- opentelemetry
Reload APISIX for changes to take effect.
See static configurations for other available options you can configure in config.yaml
.
#
Send Traces to OpenTelemetryThe following example demonstrates how to trace requests to a Route and send traces to OpenTelemetry.
Start an OpenTelemetry collector instance in Docker:
docker run -d --name otel-collector -p 4318:4318 otel/opentelemetry-collector-contrib
Create a Route with opentelemetry
Plugin:
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
-H "X-API-KEY: ${admin_key}" \
-d '{
"id": "otel-tracing-route",
"uri": "/anything",
"plugins": {
"opentelemetry": {
"sampler": {
"name": "always_on"
}
}
},
"upstream": {
"type": "roundrobin",
"nodes": {
"httpbin.org": 1
}
}
}'
Send a request to the Route:
curl "http://127.0.0.1:9080/anything"
You should receive an HTTP/1.1 200 OK
response.
In OpenTelemetry collector's log, you should see information similar to the following:
2024-02-18T17:14:03.825Z info ResourceSpans #0
Resource SchemaURL:
Resource attributes:
-> telemetry.sdk.language: Str(lua)
-> telemetry.sdk.name: Str(opentelemetry-lua)
-> telemetry.sdk.version: Str(0.1.1)
-> hostname: Str(e34673e24631)
-> service.name: Str(APISIX)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope opentelemetry-lua
Span #0
Trace ID : fbd0a38d4ea4a128ff1a688197bc58b0
Parent ID :
ID : af3dc7642104748a
Name : GET /anything
Kind : Server
Start time : 2024-02-18 17:14:03.763244032 +0000 UTC
End time : 2024-02-18 17:14:03.920229888 +0000 UTC
Status code : Unset
Status message :
Attributes:
-> net.host.name: Str(127.0.0.1)
-> http.method: Str(GET)
-> http.scheme: Str(http)
-> http.target: Str(/anything)
-> http.user_agent: Str(curl/7.64.1)
-> apisix.route_id: Str(otel-tracing-route)
-> apisix.route_name: Empty()
-> http.route: Str(/anything)
-> http.status_code: Int(200)
{"kind": "exporter", "data_type": "traces", "name": "debug"}
To visualize these traces, you can export your telemetry to backend Services, such as Zipkin and Prometheus. See exporters for more details.
#
Using Trace Variables in LoggingThe following example demonstrates how to configure the opentelemetry
Plugin to set the following built-in variables, which can be used in logger Plugins or access logs:
opentelemetry_context_traceparent
: trace parent IDopentelemetry_trace_id
: trace ID of the current spanopentelemetry_span_id
: span ID of the current span
Configure the plugin metadata to set set_ngx_var
as true:
curl http://127.0.0.1:9180/apisix/admin/plugin_metadata/opentelemetry -H "X-API-KEY: $admin_key" -X PUT -d '
{
"set_ngx_var": true
}'
Update the configuration file as below. You should customize the access log format to use the opentelemetry
Plugin variables.
nginx_config:
http:
enable_access_log: true
access_log_format: '{"time": "$time_iso8601","opentelemetry_context_traceparent": "$opentelemetry_context_traceparent","opentelemetry_trace_id": "$opentelemetry_trace_id","opentelemetry_span_id": "$opentelemetry_span_id","remote_addr": "$remote_addr"}'
access_log_format_escape: json
Reload APISIX for configuration changes to take effect.
You should see access log entries similar to the following when you generate requests:
{"time": "18/Feb/2024:15:09:00 +0000","opentelemetry_context_traceparent": "00-fbd0a38d4ea4a128ff1a688197bc58b0-8f4b9d9970a02629-01","opentelemetry_trace_id": "fbd0a38d4ea4a128ff1a688197bc58b0","opentelemetry_span_id": "af3dc7642104748a","remote_addr": "172.10.0.1"}