Skip to main content
Case StudiesEcosystemAuthenticationPluginsCommunityVulnerabilities

API Log Monitor with Apache APISIX & RocketMQ

· 4 min read
Zhou Yu
Sylvia

This article will introduce the latest integration of Apache APISIX and Apache RocketMQ rocketmq-logger plugin features and use. With this plugin, you can connect to the RocketMQ cluster more easily when using Apache APISIX.

Since Apache RocketMQ came to the global developers' attention in 2016, it has grown to become the core data base of the technology middle office in multiple fields such as e-commerce, finance, education, and technology.

According to incomplete statistics, more than 70% of domestic users (including top 100 enterprises in various fields such as finance, insurance, wealth and brokerage) have deployed Apache RocketMQ at scale on their core application links, including the top 5 global cloud vendors have also launched cloud product services about Apache RocketMQ.

In addition to regular applications for core business message processing, a large number of companies are using Apache RocketMQ for log processing and analysis.

Plug-in Introduction

To meet the needs of enterprise users for log processing, Apache APISIX has released rocketmq-logger, a logging plugin based on Apache RocketMQ that supports pushing API interface request logs to RocketMQ clusters as JSON.

The plugin uses the TCP protocol natively supported by RocketMQ, and achieves high concurrency and high performance access through the non-blocking TCP socket API provided by OpenResty.

Also, the API log format sent using the rocketmq-logger plugin is the same as other logging plugins, with support for bulk logging, custom log formats, retry support, and other features.

In addition, the plug-in also supports TLS encrypted transmission, as well as configuring AK, SK authentication to access Apache RocketMQ, to meet the needs of users for data security.

How to Use

Start RocketMQ

Start RocketMQ locally with the following command, for details refer to the official documentation.

wget https://dlcdn.apache.org/rocketmq/4.9.2/rocketmq-all-4.9.2-bin-release.zip

unzip rocketmq-all-4.9.2-bin-release.zip

cd rocketmq-4.9.2/

nohup sh bin/mqnamesrv &

nohup sh bin/mqbroker -n 127.0.0.1:9876 -c conf/broker.conf &

Enable the plugin in Apache APISIX

Enabling the rocketmq-logger plugin for a given route in a production environment requires only a single command.

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "plugins": {
       "rocketmq-logger": {
           "nameserver_list" : [ "127.0.0.1:9876" ],
           "topic" : "test",
       }
    },
    "upstream": {
       "nodes": {
           "127.0.0.1:1980": 1
       },
       "type": "roundrobin"
    },
    "uri": "/hello"
}'

When the rocketmq-logger plugin is enabled, any request to the endpoint URI/hello will push the log to Apache RocketMQ.

Details of the specific supported parameters can be found in the following table.

NameTypeDescription
nameserver_listobjectNameserver list of rocketmq to be pushed.
topicstringThe topic to be pushed.
keystringThe keys that sent the message.
tagstringThe tags that sent the message.
timeoutintegerThe timeout for sending data.
use_tlsbooleanWhether to enable TLS encryption.
access_keystringAn empty access key, string certified by ACL indicates that ACL is not enabled.
secret_keystringACL certified secret key.

Plugin metadata settings

Of course, if you do not want to use the default log format during use, you can also set the metadata for the plugin.

The first thing you can do is to adjust the relevant log format in the form of a template.

NameDefault ValueDescription
log_format{"host": "$host", "@timestamp": "$time_iso8601", "client_ip": "$remote_addr"}Declare the log format as a key-value pair in JSON format. For the value section, only strings are supported. If it starts with $, it indicates that you want to get the APISIX variable or Nginx built-in variable. In particular, this setting takes effect globally, which means that when log_format is specified, it will take effect on all Route or Service bound to http-logger.

Once the log format is adjusted, you need to send a request to the /apisix/admin/plugin_metadata endpoint to update the metadata, as described in the code below.

curl http://127.0.0.1:9080/apisix/admin/plugin_metadata/rocketmq-logger -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "log_format": {
        "host": "$host",
        "@timestamp": "$time_iso8601",
        "client_ip": "$remote_addr"
    }
}'

Disable the Plugin

If you no longer use the plugin, you can disable the rocketmq-logger plugin by removing the appropriate JSON configuration from the plugin configuration. This process does not require a restart of the service and will take effect immediately by entering the code below.

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/hello",
    "plugins": {},
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'

Table of Contents