MQ
This document was translated by ChatGPT
#1. Kafka
By parsing the Kafka (opens new window) protocol, the fields of Kafka Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | request_api_version | request_api_key | -- |
| request_type | Request Type | request_api_key | request_api_version | Supported API Key List (opens new window) | |
| request_domain | Request Domain | topic | topic | Only for Produce, Fetch messages, takes the first corresponding field | |
| request_resource | Request Resource | $topic-$partition:$offset | $topic-$partition:$offset | Only for Produce, Fetch messages, takes the first corresponding field [1][2] | |
| request_id | Request ID | correlation_id | correlation_id | Reference: Using CorrelationID to Associate Req-Resp Communication Scenarios (opens new window) | |
| endpoint | Endpoint | $topic-$partition | $topic-$partition | Only for Produce, Fetch messages, takes the first corresponding field | |
| Resp. | response_code | Response Code | -- | error_code | Only for Produce, Fetch, JoinGroup, LeaveGroup, SyncGroup messages [3] |
| response_status | Response Status | -- | error_code | Normal: error_code=0; Server exception: error_code!=0 | |
| response_exception | Response Exception | -- | error_code | English description (opens new window) of error_code [3] | |
| response_result | Response Result | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from the corresponding Header field of the first Record |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from the corresponding Header field of the first Record | |
| x_request_id | X-Request-ID | correlation_id | correlation_id | Reference: Using CorrelationID to Associate Req-Resp Communication Scenarios (opens new window) | |
| Misc. | attribute.group_id | -- | group_id | group_id | Only for JoinGroup, LeaveGroup, SyncGroup messages |
- [1] All partitions in the table correspond to partition id or partition index in the Kafka protocol.
- [2] Produce's offset is taken from base_offset in Response, Fetch's offset is taken from fetch_offset in Request.
- [3] Currently unsupported api key types will set response_code to -2 and response_exception to "Type not yet inspected by DeepFlow".
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | Number of Responses | |
| session_length | Session Length | -- | -- | Request length + Response length |
| request_length | Request Length | message_size | -- | -- |
| request_length | Response Length | -- | message_size | -- |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | error_code | Reference Tag field Response Status description |
| server_error | Server Exception | -- | error_code | Reference Tag field response_code description |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#2. MQTT
By parsing the MQTT (opens new window) protocol, the fields of MQTT Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | -- | -- | -- |
| request_type | Request Type | PacketKind | -- | -- | |
| request_domain | Request Domain | client_id | -- | -- | |
| request_resource | Request Resource | topic | -- | -- | |
| request_id | Request ID | -- | -- | -- | |
| endpoint | Endpoint | topic | -- | -- | |
| Resp. | response_code | Response Code | -- | code | Only connect_ack messages get code |
| response_status | Response Status | -- | code | Normal: code=0; Client exception: code=1/2/4/5; Server exception: code=3 | |
| response_exception | Response Exception | -- | -- | -- | |
| response_result | Response Result | -- | -- | -- | |
| Trace | trace_id | TraceID | -- | -- | -- |
| span_id | SpanID | -- | -- | -- | |
| x_request_id | X-Request-ID | -- | -- | -- | |
| Misc. | -- | -- | -- | -- | -- |
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | Number of Responses | |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | connect_ack message returned code | Reference Tag field Response Status description |
| server_error | Server Exception | -- | connect_ack message returned code | Reference Tag field response_code description |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#3. AMQP
By parsing the AMQP (opens new window) protocol (i.e., the main protocol of RabbitMQ (opens new window)), the fields of AMQP Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | version | -- | 0-9-1 |
| request_type | Request Type | class_id.method_id | -- | Example: Channel.OpenOK | |
| request_domain | Request Domain | vhost | -- | -- | |
| request_resource | Request Resource | exchange.routing_key or queue | -- | -- | |
| request_id | Request ID | -- | -- | -- | |
| endpoint | Endpoint | exchange.routing_key or queue | -- | -- | |
| Resp. | response_code | Response Code | -- | method_id | OpenOK |
| response_status | Response Status | -- | -- | All considered normal | |
| response_exception | Response Exception | -- | -- | -- | |
| response_result | Response Result | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Custom fields in Content Header |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Custom fields in Content Header | |
| x_request_id | X-Request-ID | -- | -- | -- | |
| Misc. | -- | -- | -- | -- | -- |
Note: Limited by protocol characteristics, currently only supports identifying AMQP protocols that establish connections after the agent starts.
Additionally, the following are unidirectional messages that will be directly saved as type=session call logs:
- Connection.Blocked (
s->c) - Connection.Unblocked (
s->c) - Basic.Return (
s->c) - Basic.Ack (
both) - Basic.Nack (
both) - Basic.Reject (
c->s) - Basic.RecoverAsync (
c->s) - Content-Header (
both) - Content-Body (
both) - Protocol-Header (
c->s) While the following messages may or may not have ACK, DeepFlow uniformly ignores their responses (because ACK contains no critical information, and since ACK is not stable, there's no need to calculate latency): - Basic.Publish (
c->s) - Basic.Deliver (
s->c)
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | -- | Number of Responses |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | -- | -- |
| server_error | Server Exception | -- | -- | -- |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#4. OpenWire
By parsing the OpenWire (opens new window) protocol (i.e., the default protocol of ActiveMQ (opens new window)), the fields of OpenWire Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | version | -- | -- |
| request_type | Request Type | OpenWireCommand | -- | -- | |
| request_domain | Request Domain | broker_url | -- | -- | |
| request_resource | Request Resource | topic | -- | -- | |
| request_id | Request ID | command_id | correlation_id [1] | Request and response correspondence details see [2] | |
| endpoint | Endpoint | topic | -- | -- | |
| Resp. | response_code | Response Code | -- | -- | -- |
| response_status | Response Status | -- | -- | Normal: no error message; Server exception: has error message | |
| response_exception | Response Exception | -- | error message | -- | |
| response_result | Response Result | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | -- |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | -- | |
| x_request_id | X-Request-ID | correlation_id | correlation_id | Reference: CorrelationID in ActiveMQ (opens new window) | |
| Misc. | -- | -- | -- | -- | -- |
- [1] Note the distinction from the correlation_id field corresponding to x_request_id below, these are two different fields
- [2] When request's response_required is true, the correlation_id field of the corresponding response should match the command_id of the request
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | Number of Responses | |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | -- | -- |
| server_error | Server Exception | -- | -- | -- |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#5. NATS
By parsing the NATS (opens new window) protocol, the fields of NATS Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | version | -- | Uses version from INFO |
| request_type | Request Type | NatsMessage | -- | Such as INFO, SUB, PUB, MSG | |
| request_domain | Request Domain | server_name | -- | Uses server_name from INFO | |
| request_resource | Request Resource | subject | -- | -- | |
| request_id | Request ID | -- | -- | -- | |
| endpoint | Endpoint | subject | -- | Only the part before the first . in subject | |
| Resp. | response_code | Response Code | -- | -- | -- |
| response_status | Response Status | -- | -- | All considered normal | |
| response_exception | Response Exception | -- | -- | -- | |
| response_result | Response Result | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB | |
| x_request_id | X-Request-ID | -- | -- | -- | |
| Misc. | -- | -- | -- | -- | -- |
Note that except for Info/Connect, Ping/Pong these two groups of messages, other messages are all unidirectional and will be directly saved as type=session call logs:
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | Number of Responses | |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | -- | -- |
| server_error | Server Exception | -- | -- | -- |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#6. Pulsar
By parsing the Pulsar (opens new window) protocol, the fields of Pulsar Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | protocol_version | -- | Takes the smaller of CommandConnect and CommandConnected |
| request_type | Request Type | command | -- | -- | |
| request_domain | Request Domain | proxy_to_broker_url | -- | In CommandConnect | |
| request_resource | Request Resource | topic | -- | Takes the content after the last / in the protocol topic | |
| request_id | Request ID | request_id | -- | For Send/SendError/SendReceipt, since the command has no request_id, takes producer_id and the lower 16 bits of sequence_id concatenated as request ID | |
| endpoint | Endpoint | topic | -- | -- | |
| Resp. | response_code | Response Code | -- | code | -- |
| response_status | Response Status | -- | status | -- | |
| response_exception | Response Exception | -- | exception | -- | |
| response_result | Response Result | -- | -- | -- | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from NATS headers in HMSG, HPUB | |
| x_request_id | X-Request-ID | x_request_id | x_request_id | -- | |
| Misc. | -- | -- | -- | -- | -- |
Note that the following are unidirectional messages that will be directly saved as type=session call logs:
- Ack
- Flow
- Message
- RedeliverUnacknowledgedMessages
- ReachedEndOfTopic
- ActiveConsumerChange
- AckResponse
- WatchTopicList
- WatchTopicListSuccess
- WatchTopicUpdate
- WatchTopicListClose
- TopicMigrated
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | Number of Responses | |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | -- | -- |
| server_error | Server Exception | -- | -- | -- |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#7. ZMTP
By parsing the ZMTP (opens new window) protocol (i.e., the message transport protocol used by ZeroMQ), the fields of ZMTP Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | version | -- | -- |
| request_type | Request Type | frame_type | -- | -- | |
| request_domain | Request Domain | subscription | -- | Only when socket type is PUB/SUB/XPUB/XSUB | |
| request_resource | Request Resource | subscription | -- | Only when socket type is PUB/SUB/XPUB/XSUB | |
| Resp. | response_code | Response Code | -- | -- | -- |
| response_status | Response Status | -- | -- | Normal: no error message; Exception: has error message | |
| response_exception | Response Exception | -- | error message | -- | |
| Misc. | -- | -- | -- | -- | -- |
- In the ZMTP protocol, only when one end socket is REQ/REP, request messages must wait for the previous request to receive a response before initiating, and the request and response will be aggregated into a session
- Other types are currently only identified as unidirectional messages and will be directly saved as type=session call logs
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | Number of Responses | |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | -- | -- |
| server_error | Server Exception | -- | -- | -- |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |
#8. RocketMQ
By parsing the RocketMQ (opens new window) protocol, the fields of RocketMQ Request / Response are mapped to the corresponding fields in l7_flow_log. The mapping relationships are shown in the following table:
Tag Field Mapping Table, the following table only includes fields with mapping relationships
| Category | Name | Chinese | Request Header | Response Header | Description |
|---|---|---|---|---|---|
| Req. | version | Protocol Version | version | -- | -- |
| request_type | Request Type | code | -- | -- | |
| request_domain | Request Domain | extFields:producerGroup | consumerGroup | -- | Mainly for SEND and PULL, other messages not yet fully supplemented | |
| request_resource | Request Resource | extFields:topic | -- | Mainly for SEND and PULL, other messages not yet fully supplemented | |
| request_id | Request ID | opaque | -- | -- | |
| endpoint | Endpoint | extFields:topic & queueId | -- | -- | |
| Resp. | response_code | Response Code | -- | code | -- |
| response_status | Response Status | -- | code | 0 represents normal, non-0 represents various exceptions | |
| response_exception | Response Exception | -- | remark | -- | |
| response_result | Response Result | -- | body | JSON serialization format corresponds to all JSON string data, ROCKETMQ format corresponds to the body string therein | |
| Trace | trace_id | TraceID | traceparent, sw8 | traceparent, sw8 | Extracted from extFields or properties field in bodyData |
| span_id | SpanID | traceparent, sw8 | traceparent, sw8 | Extracted from extFields or properties field in bodyData | |
| x_request_id | X-Request-ID | UNIQ_KEY | KEY | Extracted from extFields or properties field in bodyData | |
| Misc. | -- | -- | -- | -- | -- |
- In the RocketMQ protocol, bit0 of the flag field identifies whether it's a request or response, while bit1 identifies whether it's a unidirectional request (no response needed)
- For non-unidirectional requests like SEND_MESSAGE, PULL_MESSAGE and their corresponding responses, they are aggregated into a Session one-to-one according to opaque
- For unidirectional requests like UPDATE_CONSUMER_OFFSET, they are set as Session type individually, no aggregation needed
Metrics Field Mapping Table, the following table only includes fields with mapping relationships
| Name | Chinese | Request | Response | Description |
|---|---|---|---|---|
| request | Request | -- | -- | Number of Requests |
| response | Response | -- | -- | Number of Responses |
| log_count | Total Logs | -- | -- | -- |
| error | Exception | -- | -- | Client exception + Server exception |
| client_error | Client Exception | -- | -- | -- |
| server_error | Server Exception | -- | -- | -- |
| error_ratio | Exception Ratio | -- | -- | Exception / Response |
| client_error_ratio | Client Exception Ratio | -- | -- | Client exception / Response |
| server_error_ratio | Server Exception Ratio | -- | -- | Server exception / Response |