<a href="https://engineering.linkedin.com/blog/2016/05/open-sourcing-kafka-monitor">https://engineering.linkedin.com/blog/2016/05/open-sourcing-kafka-monitor</a>
<a href="https://github.com/linkedin/kafka-monitor">https://github.com/linkedin/kafka-monitor</a>
<a href="https://github.com/microsoft/availability-monitor-for-kafka">https://github.com/microsoft/availability-monitor-for-kafka</a>
kafka monitor makes it easy to develop and execute long-running kafka-specific system tests in real clusters and to monitor existing kafka deployment's slas provided by users.
developers can create new tests by composing reusable modules to emulate various scenarios (e.g. gc pauses, broker hard-kills, rolling bounces, disk failures, etc.) and collect metrics; users can run kafka monitor tests that execute these scenarios at a user-defined schedule on a test cluster or production cluster and validate that kafka still functions as expected in these scenarios. kafka monitor is modeled as manager for a collection of tests and services in order to achieve these goals.
a given kafka monitor instance runs in a single java process and can spawn multiple tests/services in the same process. the diagram below demonstrates the relations between service, test and kafka monitor instance, as well as how kafka monitor interacts with a kafka cluster and user.
这个平台比较有意思在于，不光是监控那么简单，
还包含完整的test框架，可以定义任意test，test由各种service，即组件，组合而成
produce service, which produces messages to kafka and measures metrics such as produce rate and availability.
consume service, which consumes messages from kafka and measures metrics including message loss rate, message duplicate rate and end-to-end latency. this service depends on the produce service to provide messages that embed a message sequence number and timestamp.
broker bounce service, which bounces a given broker at some pre-defined schedule.
用上面的3个services，就可以组合出一个测试broker bounce的test
或者上面的case，通过两个kafka monitor，可以测试多datacenter之间的同步
monitoring kafka cluster deployments
<a></a>
validate client libraries using end-to-end workflows
certify new internal releases of apache kafka
we generally run off apache kafka trunk and cut a new internal release every quarter or so to pick up new features from apache kafka. a significant benefit of running off trunk is that deploying kafka in linkedin’s production cluster has often detected problems in apache kafka trunk that can be fixed before official apache kafka releases.
given the risk of running off apache kafka trunk, we take extra care to certify every internal release in a test cluster—which accepts traffic mirrored from production cluster(s)—for a few weeks before deploying the new release in production. for example, we do rolling bounces or hard kill brokers, while checking jmx metrics to verify that there is exactly one controller and no offline partitions, in order to validate kafka’s availability under failover scenarios. in the past, these steps were manual, which is very time-consuming and doesn’t scale well with the number of events and types of scenarios we want to test. we are switching to kafka monitor to automate this process and cover more failover scenarios on a continual basis.