MQ
This document was translated by ChatGPT
#1. Kafka
By parsing the Kafka (opens new window) protocol, the fields of Kafka Request/Response are mapped to the corresponding fields in l7_flow_log. The mapping relationship is as follows:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | request_api_version | request_api_key | -- |
| request_type | 请求类型 | request_api_key | request_api_version | Supported API Key List (opens new window) | |
| request_domain | 请求域名 | topic | topic | Only for Produce, Fetch messages, take the first corresponding field | |
| request_resource | 请求资源 | $topic-$partition:$offset | $topic-$partition:$offset | Only for Produce, Fetch messages, take the first corresponding field [1][2] | |
| request_id | 请求 ID | correlation_id | correlation_id | Reference: Using CorrelationID to Associate Req-Resp Communication Scenarios (opens new window) | |
| endpoint | 端点 | $topic-$partition | $topic-$partition | Only for Produce, Fetch messages, take the first corresponding field | |
| Resp. | response_code | 响应码 | -- | error_code | Only for Produce, Fetch, JoinGroup, LeaveGroup, SyncGroup messages |
| response_status | 响应状态 | -- | error_code | Normal: error_code=0; Server exception: error_code!=0 | |
| response_exception | 响应异常 | -- | error_code | English description (opens new window) of error_code | |
| response_result | 响应结果 | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from the corresponding Header field of the first Record |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from the corresponding Header field of the first Record | |
| x_request_id | X-Request-ID | correlation_id | correlation_id | Reference: Using CorrelationID to Associate Req-Resp Communication Scenarios (opens new window) | |
| Misc. | attribute.group_id | -- | group_id | group_id | Only for JoinGroup, LeaveGroup, SyncGroup messages |
- [1] All partitions in the table correspond to the partition id or partition index in the Kafka protocol.
- [2] The offset of Produce is taken from base_offset in the Response, and the offset of Fetch is taken from fetch_offset in the Request.
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | Number of Responses | |
| session_length | 会话长度 | -- | -- | Request length + Response length |
| request_length | 请求长度 | message_size | -- | -- |
| request_length | 响应长度 | -- | message_size | -- |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client exceptions + Server exceptions |
| client_error | 客户端异常 | -- | error_code | Refer to the description of the Tag field response_status |
| server_error | 服务端异常 | -- | error_code | Refer to the description of the Tag field response_code |
| error_ratio | 异常比例 | -- | -- | Exceptions / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client exceptions / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server exceptions / Responses |
#2. MQTT
By parsing the MQTT (opens new window) protocol, the fields of MQTT Request/Response are mapped to the corresponding fields in l7_flow_log. The mapping relationship is as follows:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | -- | -- | -- |
| request_type | 请求类型 | PacketKind | -- | -- | |
| request_domain | 请求域名 | client_id | -- | -- | |
| request_resource | 请求资源 | topic | -- | -- | |
| request_id | 请求 ID | -- | -- | -- | |
| endpoint | 端点 | topic | -- | -- | |
| Resp. | response_code | 响应码 | -- | code | Only connect_ack message retrieves the code |
| response_status | 响应状态 | -- | code | Normal: code=0; Client exception: code=1/2/4/5; Server exception: code=3 | |
| response_exception | 响应异常 | -- | -- | -- | |
| response_result | 响应结果 | -- | -- | -- | |
| Trace | trace_id | TraceID | -- | -- | -- |
| span_id | SpanID | -- | -- | -- | |
| x_request_id | X-Request-ID | -- | -- | -- | |
| Misc. | -- | -- | -- | -- | -- |
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | Number of Responses | |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client exceptions + Server exceptions |
| client_error | 客户端异常 | -- | connect_ack message code | Refer to the description of the Tag field response_status |
| server_error | 服务端异常 | -- | connect_ack message code | Refer to the description of the Tag field response_code |
| error_ratio | 异常比例 | -- | -- | Exceptions / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client exceptions / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server exceptions / Responses |
#3. AMQP
By parsing the AMQP (opens new window) protocol (i.e., the main protocol of RabbitMQ (opens new window)), the fields of AMQP Request/Response are mapped to the corresponding fields in l7_flow_log. The mapping relationship is as follows:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | version | -- | 0-9-1 |
| request_type | 请求类型 | class_id.method_id | -- | e.g., Channel.OpenOK | |
| request_domain | 请求域名 | vhost | -- | -- | |
| request_resource | 请求资源 | exchange.routing_key or queue | -- | -- | |
| request_id | 请求 ID | -- | -- | -- | |
| endpoint | 端点 | exchange.routing_key or queue | -- | -- | |
| Resp. | response_code | 响应码 | -- | method_id | OpenOK |
| response_status | 响应状态 | -- | -- | All considered normal | |
| response_exception | 响应异常 | -- | -- | -- | |
| response_result | 响应结果 | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Custom field in Content Header |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Custom field in Content Header | |
| x_request_id | X-Request-ID | -- | -- | -- | |
| Misc. | -- | -- | -- | -- | -- |
Note: Due to protocol characteristics, currently only AMQP protocols established after the agent starts are supported.
Additionally, the following are one-way messages and will be directly saved as type=session call logs:
- Connection.Blocked (
s->c) - Connection.Unblocked (
s->c) - Basic.Return (
s->c) - Basic.Ack (
both) - Basic.Nack (
both) - Basic.Reject (
c->s) - Basic.RecoverAsync (
c->s) - Content-Header (
both) - Content-Body (
both) - Protocol-Header (
c->s) The following messages may or may not have ACK, DeepFlow uniformly ignores their responses (because the ACK does not contain key information, and since ACK is not stable, there is no need to calculate latency): - Basic.Publish (
c->s) - Basic.Deliver (
s->c)
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | -- | Number of Responses |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client exceptions + Server exceptions |
| client_error | 客户端异常 | -- | -- | -- |
| server_error | 服务端异常 | -- | -- | -- |
| error_ratio | 异常比例 | -- | -- | Exceptions / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client exceptions / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server exceptions / Responses |
#4. OpenWire
By parsing the OpenWire (opens new window) protocol (i.e., the default protocol of ActiveMQ (opens new window)), the fields of OpenWire Request/Response are mapped to the corresponding fields in l7_flow_log. The mapping relationship is as follows:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | version | -- | -- |
| request_type | 请求类型 | OpenWireCommand | -- | -- | |
| request_domain | 请求域名 | broker_url | -- | -- | |
| request_resource | 请求资源 | topic | -- | -- | |
| request_id | 请求 ID | command_id | correlation_id [1] | The correspondence between request and response is detailed in [2] | |
| endpoint | 端点 | topic | -- | -- | |
| Resp. | response_code | 响应码 | -- | -- | -- |
| response_status | 响应状态 | -- | -- | Normal: no error message; Server exception: has error message | |
| response_exception | 响应异常 | -- | error message | -- | |
| response_result | 响应结果 | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | -- |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | -- | |
| x_request_id | X-Request-ID | correlation_id | correlation_id | Reference: CorrelationID in ActiveMQ (opens new window) | |
| Misc. | -- | -- | -- | -- | -- |
- [1] Note the distinction from the correlation_id field corresponding to x_request_id below, they are two different fields
- [2] When the response_required of the request is true, the correlation_id field of the corresponding response should be consistent with the command_id of the request
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | Number of Responses | |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client exceptions + Server exceptions |
| client_error | 客户端异常 | -- | -- | -- |
| server_error | 服务端异常 | -- | -- | -- |
| error_ratio | 异常比例 | -- | -- | Exceptions / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client exceptions / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server exceptions / Responses |
#5. NATS
By parsing the NATS (opens new window) protocol, the fields of NATS Request/Response are mapped to the corresponding fields in l7_flow_log. The mapping relationship is as follows:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | version | -- | Using version in INFO |
| request_type | 请求类型 | NatsMessage | -- | Such as INFO, SUB, PUB, MSG | |
| request_domain | 请求域名 | server_name | -- | Using server_name in INFO | |
| request_resource | 请求资源 | subject | -- | -- | |
| request_id | 请求 ID | -- | -- | -- | |
| endpoint | 端点 | subject | -- | Only the part before the first . in subject | |
| Resp. | response_code | 响应码 | -- | -- | -- |
| response_status | 响应状态 | -- | -- | All considered normal | |
| response_exception | 响应异常 | -- | -- | -- | |
| response_result | 响应结果 | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB | |
| x_request_id | X-Request-ID | -- | -- | -- | |
| Misc. | -- | -- | -- | -- | -- |
Note, except for the two pairs of messages Info/Connect and Ping/Pong, other messages are one-way messages and will be directly saved as type=session call logs:
Metrics Field Mapping Table: The following table includes only the fields that have a mapping relationship
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | Number of Responses | |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client Errors + Server Errors |
| client_error | 客户端异常 | -- | -- | -- |
| server_error | 服务端异常 | -- | -- | -- |
| error_ratio | 异常比例 | -- | -- | Errors / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client Errors / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server Errors / Responses |
#6. Pulsar
By parsing the Pulsar (opens new window) protocol, the fields of Pulsar Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table: The following table includes only the fields that have a mapping relationship
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | protocol_version | -- | Taken from the smaller of CommandConnect and CommandConnected |
| request_type | 请求类型 | command | -- | -- | |
| request_domain | 请求域名 | proxy_to_broker_url | -- | Taken from CommandConnect | |
| request_resource | 请求资源 | topic | -- | The content after the last / in the protocol topic | |
| request_id | 请求 ID | request_id | -- | For Send/SendError/SendReceipt, since these commands do not have request_id, concatenate the lower 16 bits of producer_id and sequence_id as the request ID | |
| endpoint | 端点 | topic | -- | -- | |
| Resp. | response_code | 响应码 | -- | code | -- |
| response_status | 响应状态 | -- | status | -- | |
| response_exception | 响应异常 | -- | exception | -- | |
| response_result | 响应结果 | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB | |
| x_request_id | X-Request-ID | x_request_id | x_request_id | -- | |
| Misc. | -- | -- | -- | -- | -- |
Note that the following are unidirectional messages, which will be directly saved as type=session call logs:
- Ack
- Flow
- Message
- RedeliverUnacknowledgedMessages
- ReachedEndOfTopic
- ActiveConsumerChange
- AckResponse
- WatchTopicList
- WatchTopicListSuccess
- WatchTopicUpdate
- WatchTopicListClose
- TopicMigrated
Metrics Field Mapping Table: The following table includes only the fields that have a mapping relationship
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | Number of Responses | |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client Errors + Server Errors |
| client_error | 客户端异常 | -- | -- | -- |
| server_error | 服务端异常 | -- | -- | -- |
| error_ratio | 异常比例 | -- | -- | Errors / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client Errors / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server Errors / Responses |
#7. ZMTP
By parsing the ZMTP (opens new window) protocol (i.e., the messaging transport protocol used by ZeroMQ), the fields of ZMTP Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table: The following table includes only the fields that have a mapping relationship
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | 协议版本 | version | -- | -- |
| request_type | 请求类型 | frame_type | -- | -- | |
| request_domain | 请求域名 | subscription | -- | Only when socket type is PUB/SUB/XPUB/XSUB | |
| request_resource | 请求资源 | subscription | -- | Only when socket type is PUB/SUB/XPUB/XSUB | |
| Resp. | response_code | 响应码 | -- | -- | -- |
| response_status | 响应状态 | -- | -- | Normal: No error message; Exception: Has error message | |
| response_exception | 响应异常 | -- | error message | -- | |
| Misc. | -- | -- | -- | -- | -- |
- In the ZMTP protocol, when one end socket is REQ/REP, a request message must wait for the previous request to get a response before sending, and requests and responses will be aggregated into one session
- Other types are currently only recognized as unidirectional messages and will be directly saved as
type=sessioncall logs
Metrics Field Mapping Table: The following table includes only the fields that have a mapping relationship
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | 请求 | -- | -- | Number of Requests |
| response | 响应 | -- | Number of Responses | |
| log_count | 日志总量 | -- | -- | -- |
| error | 异常 | -- | -- | Client Errors + Server Errors |
| client_error | 客户端异常 | -- | -- | -- |
| server_error | 服务端异常 | -- | -- | -- |
| error_ratio | 异常比例 | -- | -- | Errors / Responses |
| client_error_ratio | 客户端异常比例 | -- | -- | Client Errors / Responses |
| server_error_ratio | 服务端异常比例 | -- | -- | Server Errors / Responses |