March 19, 2025
Traditional analytics dashboards need to be smashed. Every widget that is tweaked to perfection showing some correlation obscures a fundamental reality. Reality is messy. This doesn’t mean, of course, that all visualization software is bad. If anything, the true value of visualization tools is unlocked when designs match the dynamics of the underlying data.
In a business context, the dynamics that matter are those which involve customer conversions. Those dynamics need to be described symbolically through a metrics tree. That’s it. The important thing to avoid at this step is to over complicate thinking by jumping to the design from the description. We don’t need any visualizations at this point. What we do need is a clear identification of the key touch points in the customer conversion funnel. The existence of these touch points should be indisputable, because otherwise there would be no business.
After we have a metrics tree, then we move on to the most important part of the whole process, the observation phase. In order to properly observe, we set up our instruments to build accurate XmR graphs and track each metric as an “x” for a significant amount of time, meaning, enough time to achieve statistical and pragmatic significance. Why XmR graphs? Because they allow us to represent the underlying dynamics that generate the data that we care about in a way that makes obvious how our interventions affect each of our “x” values.
After we’ve charted and observed the performance of our “x” values on the XmR graph, we’re ready to analyze the effect of our interventions.
Our analysis is simplified by focusing on just two outcomes, the null result and the outlier. Null results are arguably the most important of the two because they re-focus our attention on what as a species we’re primed to ignore. They tell us, in other words, that our interventions did not have a significant effect on the value of “x” and this in turn tells us what NOT to do. We need to celebrate confirmations of the null result because they check our tendency to fabricate rationalizations and myth making.
Outliers, by contrast, suggest that our interventions caused a significant change in the value of “x”. Interventions can, of course, vary widely. They can be a new keyword strategy that is informed by patient observation of search result data, or AI generated content centered on topics curated by humans and that reveal aspects of the business that are relevant to customers. The potential for successful interventions is endless. The point is that the method of analysis needs to adapt to the underlying business dynamics.
XmR graphs are the tool of choice for an empirical method of operations management called statistical process control (SPR). One of the virtues of SPR is its simplicity. As we can see, the implementation of the method reduces the number of visualizations to one, thereby reducing backend overhead from data teams and front end cognitive load from business teams.
We need to avoid gamifying visualization dashboards and similar sources of bikeshedding. Speaking plainly, these are just ways of doing nothing. Don’t get me wrong, I don’t have a problem with doing nothing. But whatever we do as data professionals needs to be in synch with what the business does. Data tooling has a tendency to become the message as opposed to staying put as a medium.