Exporter Config
创建时间:2024-11-05 最近修改时间:2024-11-05
#1. Exporter 配置指引
#1.1 配置示例
ingester:
exporters:
- protocol: kafka
enabled: true
endpoints: [broker1.example.com:9092, broker2.example.com:9092]
data-sources:
- flow_log.l7_flow_log
# - flow_log.l4_flow_log
# - flow_metrics.application_map.1s
# - flow_metrics.application_map.1m
# - flow_metrics.application.1s
# - flow_metrics.application.1m
# - flow_metrics.network_map.1s
# - flow_metrics.network_map.1m
# - flow_metrics.network.1s
# - flow_metrics.network.1m
# - event.perf_event
queue-count: 4
queue-size: 100000
batch-size: 1024
flush-timeout: 10
tag-filters:
export-fields:
- $tag
- $metrics
extra-headers:
key1: value1
key2: value2
export-empty-tag: false
export-empty-metrics-disabled: false
enum-translate-to-name-disabled: false
universal-tag-translate-to-name-disabled: false
sasl:
enabled: false
security-protocol: SASL_SSL # currently only supports: SASL_SSL
sasl-mechanism: PLAIN # currently only supports: PLAIN
username: aaa
password: bbb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#2. 详细参数说明
字段 | 类型 | 必选 | 描述 |
---|---|---|---|
protocol | string | 是 | 支持opentelemetry , prometheus , kafka |
enabled | bool | 是 | 是否启用该 exporter |
data-sources | strings | 是 | 不同协议支持的数据源取值范围不同. 取值 clickhouse flow_metrics.*/flow_log.*/event.perf_event 数据, 也用于 Kafka 主题名 |
endpoints | strings | 是 | 远端接收地址,不同协议地址格式不同, 随机选择一个可发送成功的 |
extra-headers | map | 否 | 远端 HTTP 请求的头部字段,比如有效验需求的,可以在这里补充 token 等信息 |
batch-size | int | 否 | 批次大小,当达到这个数值,成批的发送。默认值: 1024 (protocol 为 opentelemetry 时, 默认值: 32) |
flush-timeout | int | 否 | 刷新间隔,当达到这个时间,则直接发送。单位: 秒,默认值: 10 |
queue-count | int | 否 | 并发发送数,默认值: 4 |
tag-filters | structs | 否 | 过滤不符合条件的数据, 只发送符合条件的数据。默认值: 空, 表示不过滤. 详细配置见下文 |
export-fields | strings | 是 | 过滤需要发送的字段或类别,只发送符合条件的字段,例如: $tag, $metrics , 表示所有 $tag 和$metrics 类别的字段都发送. 详细配置见下文 |
export-empty-tag | bool | 否 | 对于 tag 值为空的字段是否发送。默认值: false, 表示不发送 |
export-empty-metrics-disabled | bool | 否 | 对于 metrics 值为 0 的字段是否发送。默认值: false, 表示发送 |
enum-translate-to-name-disabled | bool | 否 | 对于枚举类型的 ID 值是否翻译为字符串发送。 默认值: false,表示翻译 |
universal-tag-translate-to-name-disabled | bool | 否 | 对于 universal-tag 类型的资源 ID 值是否翻译为资源名称发送。 默认值: false,表示翻译 |
sasl | struct | 否 | Kafka 协议使用. 连接 Kafka 认证方式, 目前仅支持 'SASL_SSL' 的 'PLAIN' 方式 |
topic | string | 否 | Kafka 协议使用. 主题名, 若为空, 则取默认值为 deepflow.$data-source , 如 deepflow.flow_log.l7_flow_log |
#2.1 tag-filters
作用相当于 SQL 语句中的 WHERE,样例见下面的 yaml
field-name
仅支持 ClickHouse 中的原始 Tag 字段名field-values
不支持填写 Resource 类型 Tag 的字符串值operator
包括- 数值相等/不相等、字符串相等/不相等:
=
、!=
- 数值属于/不属于某个集合、字符串属于/不属于某个集合:
IN
、NOT IN
- 字符串相等/不相等,支持*通配符:
:
、!:
- 字符串相等/不相等,支持正则:
~
、!~
- 数值相等/不相等、字符串相等/不相等:
- 所有的 tag-filter 之间是 AND 的逻辑
tag-filters:
- field-name: signal_source
operator: =
field-values: [3]
- field-name: region_id_0
operator: !=
field-values: [0]
- field-name: app_service
operator: IN
field-values: [deepflow_server, mysql]
- field-name: request_resource
operator: "NOT IN"
field-values: [deepflow_agent, mysql]
- field-name: request_domain
operator: :
field-values: [deepflow_*]
- field-name: app_service
operator: ~
field-values: [deepflow_.*]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#2.2 export-fields
作用相当于 SQL 语句中的 SELECT field-names,仅支持使用 ClickHouse 中的原始字段名,样例见下面的 yaml
<field-name>
: Tag 名称、Metric 名称, 如 ip4_0, ip4_1, region_id_0, region_id_1<category>
: 字段 Category 名称,包括$tag
,$k8s.label
和$metrics
$tag
:所有 Tag 字段$tag.<sub-category>
:某一类 Tag 字段,<sub-category>
如下flow_info
- _id, time(s), start_time(us), end_time(us), close_type, flow_id, is_new_flow, status
universal_tag
- region_id[_0/1], az_id[_0/1], host_id[_0/1], pod_node_id[_0/1], pod_ns_id[_0/1], pod_group_id[_0/1], pod_group_type[_0/1], pod_id[_0/1], pod_cluster_id[_0/1]
- l3_device_type[_0/1], l3_device_id[_0/1], l3_epc_id[_0/1], epc_id[_0/1], subnet_id[_0/1], service_id[_0/1]
- auto_instance_id[_0/1], auto_instance_type[_0/1], auto_service_id[_0/1], auto_service_type[_0/1], gprocess_id[_0/1]
native_tag
- attribute_names, attritube_values
network_layer
- ip4[_0/1], ip6[_0/1], is_ipv4, protocol, province[_0/1]
tunnel_info
- tunnel_type, tunnel_tier, tunnel_tx_id, tunnel_rx_id, tunnel_tx_ip4[_0/1], tunnel_rx_ip4[_0/1]
- tunnel_tx_ip6[_0/1], tunnel_rx_ip6[_0/1], tunnel_is_ipv4, tunnel_tx_mac[_0/1], tunnel_rx_mac[_0/1]
transport_layer
- client_port, server_port, req_tcp_seq, resp_tcp_seq, tcp_flags_bit[_0/1], syn_seq, syn_ack_seq, last_keepalive_seq, last_keepalive_ack
application_layer
- l7_protocol, request_domain, version, l7_protocol_str, type, is_tls
- request_type, request_domain, request_resource, request_id, response_status, response_code, response_exception, response_result, events
service_info
- process_id[_0/1], process_kname[_0/1], app_service, app_instance, endpoint
tracing_info
- trace_id, span_id, parent_span_id, span_kind, x_request_id[_0/1], http_proxy_client
- syscall_trace_id_request, syscall_trace_id_response, syscall_thread[_0/1], syscall_coroutine[_0/1], syscall_cap_seq[_0/1]
capture_info
- signal_source, capture_network_type_id, nat_source, capture_nic_type, capture_nic, observation_point
- l2_end[_0/1], l3_end[_0/1], nat_real_ip[_0/1], nat_real_port[_0/1], agent_id, biz_type, role
event_info
- event_type
data_link_layer
- mac[_0/1], eth_type, vlan
$metrics
:所有 Metrics 字段$metrics.<sub-category>
:某一类 Metrics 字段,<sub-category>
如下l3_throughput
- packet_tx, packet_rx, byte_tx, byte_rx, l3_byte_tx, l3_byte_rx, total_packet_tx, total_packet_rx, total_byte_tx, total_byte_rx
l4_throughput
- direction_score, l4_byte_tx, l4_byte_rx, syn_count, synack_count, l4_byte_tx, l4_byte_rx, new_flow, closed_flow, flow_load
tcp_slow
- retrans_tx, retrans_rx, zero_win_tx, zero_win_rx, retrans_syn, retrans_synack
tcp_error
- client_rst_flow, server_rst_flow, server_syn_miss, client_ack_miss, client_half_close_flow, server_half_close_flow
- client_source_port_reuse, client_establish_other_rst, server_reset, server_queue_lack, server_establish_other_rst, tcp_timeout
- client_establish_fail, server_establish_fail, tcp_establish_fail, tcp_transfer_fail, tcp_rst_fail
application
- l7_request, l7_response, l7_parse_failed, l7_client_error, l7_server_error, l7_timeout
throughput
- request, response, request_length, response_length, sql_affected_rows
error
- client_error, server_error, timeout
delay
- response_duration, duration, rtt, rtt_client, rtt_server, tls_rtt
- rtt_max, rtt_client_max, rtt_server_max, srt_max, art_max, rrt_max, cit_max
- rtt_sum, rtt_client_sum, rtt_server_sum, srt_sum, art_sum, rrt_sum, cit_sum
- rtt_count, rtt_client_count, rtt_server_count, srt_count, art_count, rrt_count, cit_count
$k8s.label
: 所有 k8s.label 字段~$k8s.label.<sub-field-name_regex>
: k8s.label 子字段名支持正则
export-fields:
- ip4_0 # <field-name>
- pod_id_0 # <field-name>
- $tag # <category>
- $tag.universal_tag # <category>.<sub-category>
- $tag.flow_info # <category>.<sub-category>
- $k8s.label # <category>
- $k8s.label.k8s.annotation.app # <category>.<sub-field-name>, sub-field-name: k8s.annotation.app
- ~$k8s.label.env.(a.*|abc) # <category>.<sub-field-name_regex>, regex: env.(a.*|abc)
- $merics # <category>
- $metrics.delay # <category>.<sub-category>
- rtt # <field-name>
1
2
3
4
5
6
7
8
9
10
11
12
2
3
4
5
6
7
8
9
10
11
12