<a href="https://engineering.linkedin.com/blog/2016/05/open-sourcing-kafka-monitor">https://engineering.linkedin.com/blog/2016/05/open-sourcing-kafka-monitor</a>
<a href="https://github.com/linkedin/kafka-monitor">https://github.com/linkedin/kafka-monitor</a>
<a href="https://github.com/microsoft/availability-monitor-for-kafka">https://github.com/microsoft/availability-monitor-for-kafka</a>
kafka monitor makes it easy to develop and execute long-running kafka-specific system tests in real clusters and to monitor existing kafka deployment's slas provided by users.
developers can create new tests by composing reusable modules to emulate various scenarios (e.g. gc pauses, broker hard-kills, rolling bounces, disk failures, etc.) and collect metrics; users can run kafka monitor tests that execute these scenarios at a user-defined schedule on a test cluster or production cluster and validate that kafka still functions as expected in these scenarios. kafka monitor is modeled as manager for a collection of tests and services in order to achieve these goals.
a given kafka monitor instance runs in a single java process and can spawn multiple tests/services in the same process. the diagram below demonstrates the relations between service, test and kafka monitor instance, as well as how kafka monitor interacts with a kafka cluster and user.
這個平台比較有意思在于，不光是監控那麼簡單，
還包含完整的test架構，可以定義任意test，test由各種service，即元件，組合而成
produce service, which produces messages to kafka and measures metrics such as produce rate and availability.
consume service, which consumes messages from kafka and measures metrics including message loss rate, message duplicate rate and end-to-end latency. this service depends on the produce service to provide messages that embed a message sequence number and timestamp.
broker bounce service, which bounces a given broker at some pre-defined schedule.
用上面的3個services，就可以組合出一個測試broker bounce的test
或者上面的case，通過兩個kafka monitor，可以測試多datacenter之間的同步
monitoring kafka cluster deployments
<a></a>
validate client libraries using end-to-end workflows
certify new internal releases of apache kafka
we generally run off apache kafka trunk and cut a new internal release every quarter or so to pick up new features from apache kafka. a significant benefit of running off trunk is that deploying kafka in linkedin’s production cluster has often detected problems in apache kafka trunk that can be fixed before official apache kafka releases.
given the risk of running off apache kafka trunk, we take extra care to certify every internal release in a test cluster—which accepts traffic mirrored from production cluster(s)—for a few weeks before deploying the new release in production. for example, we do rolling bounces or hard kill brokers, while checking jmx metrics to verify that there is exactly one controller and no offline partitions, in order to validate kafka’s availability under failover scenarios. in the past, these steps were manual, which is very time-consuming and doesn’t scale well with the number of events and types of scenarios we want to test. we are switching to kafka monitor to automate this process and cover more failover scenarios on a continual basis.