In my blogs, I often distinguish between event data and metrics. I usually say something to the effect that events help to explain the metrics – or events “provide the story behind the metrics.” In this blog, I will be discussing two competing lines of thought behind events: internal capacity and external demand. Why do sales appear much lower for the month of June compared to July? Some explanations relating to internal capacity are as follows: “There weren’t enough agents in June to deal with customers.” “There were shipping problems from a number of key suppliers.” “We had difficulty dealing with orders because our data system was being upgraded in June.” These capacity-centric explanations are plausible in that they can reasonably affect sales. On the other hand, it is possible that sales were lower due to lack of consumer demand: “Our main competitor had a special promotion in June.” “Our clients just aren’t buying the merchandise that we sell anymore.” “Consumers seem to be holding back on purchases because of high debt levels.” Consequently, if sales are examined through a purely technical lens – e.g. by studying trends and patterns – the interaction between internal capacity and external demand be invisible. An organization can become insulated from reality, making decisions perhaps far detached from the underlying truths behind sales.
A decline in demand from clients necessitates a strategic response such as adjusting product lines, pricing schemes, and opening new channels for marketing. On the other hand, a failure of capacity requires an operational response such as changes to staffing levels, allocation of duties, and alternate practices. One would not normally turn to HR to handle a marketing problem or even worse turn to marketing to deal with staffing issues. Consequently, I am not making a vacant argument about the nature of metrics in an organization; but rather, since the organizational response is premised on the events giving rise to the metrics, I am actually placing events on a critical plateau. Managers that focus only on the metrics might make decisions alienated from the underlying needs of the organization, making use of corrective measures that are misplaced or inappropriate.
Apart from the remedial response, capacity and demand create challenges in terms of the placement of data-gathering resources. These resources give rise to the data. However, data is collected not just at the initial stage during problem identification; resource placement post-response can help to determine how to adapt the response to the emergent reality. For example, as a plan is implemented, the likelihood of success from beginning to end seems remote; instead, it is necessary to shape the implementation as it is occurring. It is here where the concept of a dashboard might be important. But if the dashboard is capacity-oriented while the underlying business problem is demand-oriented, the dashboard seems entirely alienated from reality. Those driving the organization would be moving the vehicle through a fictional rendering. I have described this process as “mind shafting”; actually, I think I called it “mine shafting” in an earlier blog. But perhaps a psychological term is better suited. It depends on perspective.
The “mind” involves psychology whereas “mine” (e.g. to extract nickel or coal) is physical or environmental. Capacity and mind are internal. Demand and mine are external. I certainly don’t want to confuse the issue if I haven’t already done so. My organizational conceptualization was actually meant to be “external” – and so I used the term mine – as if one’s path were externally defined. Consider the image below that I took using my cellphone a few years ago. There is an ant in the middle. Is the path that the ant takes mostly determined internally by the ant or externally by the flowers in the environment? Is the level of self-determination fixed; or does it vary as the environmental determinants change? I guess the question must seem pointless for some readers. We can expand the parameters. Should staff be eliminated for the choices they have made and the actions they have taken – as relating to sales, of course – or should the organization concentrate instead on the environment? There is an internal-external conversation; yet something like a dashboard might in certain cases only favour internal responses. Somebody will be blamed. Somebody will be held responsible for misdirection and redirection. In effect, the problem is reduced to a mind shaft.
For now, I just emphasize how extremely different the associated events might be. An argument can be made that poor performance is related to an organization not working hard enough to improve internal capacity – or a person not being psychologically engaged. On the other hand, it might not be performance at all that is poor; but rather the conditions are impoverished. A person has a limited number of options – being in a mine underground rather than open space on the surface. An organization’s market might be drying up, overly competitive, or is fully saturated. Events bring us to religion in an organization. Beliefs become so powerfully entrenched – so “shafted” or “colonized” as feminists might say – it can be hazardous to question the underlying basis. At stake is the realization of how metrics are important to an organization. It is like a process of organizational ontology.
I suppose a statistical perspective represents a fairly sterile approach to handle contentious issues involving circumstantial events. However, I feel a sacred duty not just to protect employees from quantitative alienation; but I truly and genuinely want to ensure the organization heads down the path most likely to engage and overcome its difficulties. One way people deal with stress is through avoidance: they run away from things that cause stress – normally described as the “flight” response. On an organizational level, this type of response can be manifested in how the data system (leading to metrics) is aligned with belief systems (underlying events). An organization, rather than trying to deal with difficult problems, might instead choose to deal with easy problems; this is done by ensuring that only the easy problems are recognized in the data. So it might seem like great progress is being made when more truthfully no progress has been made at all.
It was brought to my attention recently that data scientists like other scientists should be searching for “the truth.” (Presumably, this means that God shouldn’t be considered in the analysis.) But “the truth” is actually fairly complicated. In particular, both capacity and demand affect consumption metrics. Not just this, but they affect each other. Demand for a particular product (e.g. automobile tires) is not entirely disassociated with the capacity to produce another product (e.g. horseshoes). When an organization decides to compete on a different product line, the capacity to produce something obsolete might transform into incapacity to produce an alternate product. Conversely, if consumers demand a product which companies have little ability to generate, then shoppers might turn to alternate products. When dealing with metrics alone, it might seem like the metrics represent everything there is to know including the truth. But actually the metrics might be incidental to the truth. Even if measurements did not exist, the underlying truth remains on an event level. The numbers do not really carry the argument. My decision to buy pancakes does not lead to my desire to eat them. The quantitatively-evasive reality of events brought about my purchase. It is necessary to examine the events behind the metrics.
The story is far from over after a decision is made to examine events. Even with broad event-family groupings, the internal-external discourse is important. The following is an argument internalized to staff: “The employees aren’t working hard enough to get their orders in by the end of the day.” On the other hand, here is a counterargument externalized to systems: “The staff can’t get their orders in on time because the network gets extremely slow near the end of the day when the company is busiest.” Where exactly should the data-scientist focus detection and data-gathering? It is possible to assert the internal-external dichotomy even more: “Network gets slow because the client system is poorly designed.” “The network is slow because the network itself is poorly configured.” Some readers might already be thinking of firm delineations to thwart my fuzziness. Keep in mind that delineations are possible because of the articulation events using the internal-external framework. The delineations are probably impossible using metrics alone.
Professionals in analytics have an obligation the point out and emphasize the multifarious events that give rise to metrics. It seems inappropriate to simply study trends or to propose that the future will conform to a distribution of probabilities – unless of course the events are a matter of chance or randomness. Due to poor access to methodologies to deal with events, the focus on metrics is understandable. However, my fairly simple example indicates the futility of examining metrics in isolation. An argument could be made that the metrics already incorporate all of their many circumstantial events; and therefore it is reasonable to examine the metrics in isolation. It’s not really a point that is worth the argument. If a company follows its metrics and finds itself in the gutter, it appears that that the organization lacks not only event data but also the ability to recover from its mistakes. If the metrics start pointing downhill, the absence of event data will likely make it difficult for the organization to determine how it got to its present situation and how to get itself out; the deep obscurity of circumstantial events might hardly be considered.
