【Azure 事件中心】在Windows系统中使用 kafka-consumer-groups.bat 查看Event Hub中kafka的consumer groups信息

问题描述

使用 Apache Flink 连接支持 Apache Kafka的Azure Event Hub后，由于消费端的Consumer Group是动态创建，在门户页面和Service Bus Explorer工具中均无法查看，所以如何来查看 Kafka consumer group呢？可以参考以下内容：

使用前提

将 Apache Flink 与适用于 Apache Kafka 的 Azure 事件中心配合使用： https://docs.azure.cn/zh-cn/event-hubs/event-hubs-kafka-flink-tutorial

1. 在Azure中创建一个支持Kafka的Event Hub（标准层及以上）
2. 下载示例代码：git clone https://github.com/Azure/azure-event-hubs-for-kafka.git 
3. 修改 producer.config 和  consumer.config

操作步骤

If customer has an active group, they can check consumer lag (which indicate what data has been consumed) through Kafka's own kafka-consumer-groups.sh command (https://kafka.apache.org/documentation/#basic_ops_consumer_lag)

• Make sure you setup Java JDK on your machine. then download the Kafka binary downloads from (https://kafka.apache.org/downloads) - you just need the binary downloads, no need to get the source unless you intend to modify the source code that the script uses.
• from command prompt, navigate to your downloaded bin directory - e.g. F:\\kafka_2.13-3.0.0\bin\windows (for windows shell script).
• Create a properties file that has the connection string for your namespace. For example you can create a config.properties file that has the following.

bootstrap.servers=<your namespace>.servicebus.windows.net:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="<Your connection string>";      

• After that just run through the command as describe in https://kafka.apache.org/documentation/#basic_ops_consumer_lag to see your information. sample below. Note that currently, this only works if the group is an active group meaning it has active consumers running for the group. this command does not work for EventHub service if the group is inactive.

F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server <your namespace>.servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --list
$hmlam-4

F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server <your namespace>.servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --describe --group $hmlam-4

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                        HOST            CLIENT-ID
$hmlam-4        hmlam-eh-2      1          52648           52651           3               hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-4:I:perf_client#1-c391de052f2446ff85fbf6899160a236 0.0.0.0         perf_client#1
$hmlam-4        hmlam-eh-2      0          52932           52946           14              hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-      

目前遇见的问题

当根据以上步骤设置好，并且同时运行了生产者代码和消费组代码，可是却无法获取到kafka group信息

（TODO: 问题还在进一步的研究中)

当在复杂的环境中面临问题，格物之道需：浊而静之徐清，安以动之徐生。 云中，恰是如此!