URP? Excuse You! The Three Kafka Metrics You Have To Know

What do you really know about how to monitor a Kafka cluster for problems? Is your most reliable monitoring your users telling you there’s something broken? Are you capturing more metrics than the actual data being produced? Sure, we all know how to monitor disk and network, but when it comes to the state of the brokers, many of us are still unsure of which metrics we should be watching, and what their patterns mean for the state of the cluster. Kafka has hundreds of measurements, from the high-level numbers that are often meaningless to the per-partition metrics that stack up by the thousands as our data grows. We will thoroughly explore three key monitoring concepts in the broker, that will leave you an expert in identifying problems with the least amount of pain:

• Under-replicated Partitions: The mother of all metrics

• Request Latencies: Why your users complain

• Thread pool utilization: How could 80% be a problem?

We will also discuss the necessity of availability monitoring and how to use it to get a true picture of what your users see, before they come beating down your door!