An Insight a Day

Misleading Statistics

A while ago, someone raised a concern that one of our equipment was underperforming. From the utilization report, he showed us that it was hitting the max_limit only about 40% of the time whereas other similar equipment were hitting it around 90% of the time.

After looking at the actual data, it turns out that particular equipment has been repeatedly operating in and out of the max_limit, giving the illusion that it only ran at max about 40% of the time. In reality, it was operating at a similar level of performance compared to the other equipment. How did this happen?

To help you visualize, imagine two cars both with a maximum speed of 100km/hr driving along the highway. From the statistics, car A was seen hitting the max 100km/hr around 90% of the time whereas car B only 40% of the time. And yet, they both reach their destination at roughly the same time, how did this happen?

It turns out, unlike car A, car B's speed wasn't at a constant 100km/hr. It was constantly fluctuating in and out of the 100km/hr, momentarily dropping to 98km/hr before going back up to 100km/hr, only to drop back down to 99km/hr and go back up to 100km/hr again, and so on.

What can we learn from this? Statistics, as useful as it is, can be very misleading sometimes, especially when you rely too much on the summarized report instead of looking at the actual data.