“While every data set contains noise, some data sets may contain signals.
Therefore, before you can detect a signal […] you first need to filter out the noise.”

Donald Wheeler – Understanding variation. Via Mark Graban

The action: Make long-term charts to see if you have a stable system and separate real trends from random fluctuations.

The long-form: A common analysis is to look at the performance of a system over time, and look for trends or changes to act upon. It could be data on the organisation’s financial health (sales, costs, revenues), on some operational measure (output per hour, incoming phone calls) or on some external factor (air pollution, water consumption, etc.)

The least meaningful way to look at this data is perhaps to compare two data points. I’ve seen several companies create a lot of work for themselves if, say, this month’s revenues are below last months. If results are improving from June to May it’s due to great leadership. If they have deteriorated, “someone needs to be held accountable.”

But in reality, there will always be a factor of randomness in any system. Yes, root cause analyses are great, but if you always demand an explanation for why one period’s results are down, you risk creating connections that are not there. Nassim Taleb calls this desire to create a causal link where there is none, the narrative fallacy, and it is at best a waste of time.

So how much randomness is there in your system, and how do you know when something happens that is not due to chance? Mark Graban builds on a rich background of statistical thinking (Wheeler, Shewhart, Western Electric) in which you do the following:

1. Make a graph of the data over time. “More timely data is better for improvement. Daily is better than weekly, which is better than monthly […]”
2. Add lines for the average and upper/lower control limits (within three standard deviations).
3. If the data triggers a set list of rules, you probably have a signal. Examples: Eight or more points on the same side of the average. Four out of five consecutive points closer to the upper/lower limit than the average.

Looking at data on a longer horizon and being more informed about the random fluctuations allow us to answer three critical questions:

1. Are we achieving our target?
2. Are we improving?
3. How do we improve?

Read more:
Graban’s free webinar on the topic.
Get a preview of the book Measures of Success.

Learn the tools for analysis