I often encounter blogs highlighting the amount of effort needed to obtain high-quality data for analysis. Many aspects of this effort seem reasonable to me: there might be cut-and-pasting, merging, parsing, restructuring, extractions, conversions, concatenations, and conditionals. There might even be coding embedded in the file containing loops, more sophisticated conditionals, and detailed algorithmic processes. All of these are relatively minor challenges that can be overcome perhaps at some point through systems of integration. There are certain “deep” obstacles that cannot be easily integrated away; these barriers can contribute to job security.
There are many good reasons for companies to solve “by problem” rather than “by system.” For example, if the client system is old and developing incompatibilities, this is a problem. It would be perfectly reasonable to discuss replacing this system thereby resolving the specific problem. If the phone system is old, it too can be replaced. This problem separation likely means that the two systems must be handled as separate concerns. It would be difficult for instance for a data system to incorporate data from both the client and phone system. This means that there would be a structural problem preventing an analysis of the phone and client system together: e.g. the number of calls (on the phone system) and the nature of the calls (on the client system).
If financial transactions are held in a separate database system, in this case it would be difficult to associate the phone and transactional systems together: e.g. the time of call (on the phone system) and the associated transaction (on the transactional system). There might not even be a straight connection between the transactions on the client system and on the transactional system – thereby creating a possibility of disagreement between systems and of course a need to have an elaborate system of reconciliation.
I said that there are many “good” reasons to solve by problem. Well, solving a systemic problem is complicated and possibly beyond the expertise of those responsible for problem resolution. Complexity means consultation; consultation requires time; time leads to deferral; and of course time burns money. Some would also argue – I believe successfully – that a solution requiring everything to be brought down and rebuilt from scratch doesn’t demonstrate a strong grasp of business. “Let’s throw everything away and start all over again,” isn’t going to go well. The business literally might not survive this sort of wholesale reconstruction.
In all likelihood, it is necessary for analysis to occur within the context of different systems. These system differences might actually be relevant in relation to the employment of analysts. (Their jobs might not exist if integration were perfect.) It is also important to note that these differences allow for human intervention at points of integration deficiency; this creates opportunities for creative problem solving beyond the systematized design parameters of a fully integrated environment.
The fact that something is integrated doesn’t mean that it is integrated well or in a manner that best suits the needs of the specific organization. Integration also means that the organization is to some extent limited by design – possibly contributing to adaptation problems. During a period of environmental stress, it is good and proper for the organization to demonstrate the capacity for change. Integration suggests a commitment to a particular design that, although in a sense perhaps more efficient, might not lead to the most strategically desirable production outcomes.
The need for creative solutions keeps me at my job. What sort of creative solutions do I offer? This is the whole thing. I find that there aren’t any intellectual limitations – simply a need for tangible outcomes. I think this can cause approaches to differ enormously between problem-solvers. There are certainly some core skills and abilities required to ensure that deliverables are produced at an expected pace.
I tend to “go deep” precisely in those areas that lack integration. This is because the job of overcoming integration deficiencies requires skills not necessarily taught in schools. A commitment to bridging the gap contributes to job security. I know that a freshly minted professional coming out of school is likely more familiar with the latest technology. This person might lack the years of experience necessary to deal with logistical challenges posed by real-life scenarios. Apart from the logistics, I also wonder about the level of creativity possible through a process of mass-market selection. Passion is not something that can be easily bought and squeezed out of a tube like toothpaste. Some people have the magic. It is not a matter of inculcation and case studies.
I also take advantage of preconceptions. Design is seductive in that it implies supporting processes. Processes are presumed in a structured environment. People have, maintain, and gain comfort from preconceptions of how designs lead to particular outcomes. The hope is that the design is so sophisticated and powerful that it alone brings about the outcome. This makes people poorly equipped to deal with the transformative hurdles that become apparent on the absence of conversion processes. The design in this case might be a “good idea.” The idea is so extremely good, anybody can take it and make things happen. The truth is that ideas are inexpensive. It is fine to have ideas. Nobody should turn them away. It is just important to recognize that the work comes after the idea.
The person who systematically deals with other people’s ideas is valuable in setting where ideas are abundant. I rarely question the merits of ideas. My craft begins after an idea is submitted. So my goodness I encourage people to freely come up with all sorts of ideas. I never dismiss ideas. I try to determine how I can contribute to the realization of positive outcomes. An idea is an aspect of personal identity. Being inclusive means taking diverse ideas into account no matter how unusual or difficult at first glance. In fact, if I can make something happen from a difficult idea, imagine how hard it would be to replace me.
I would say the biggest error in reasoning might occur in relation to the role of software. The reasoning tends to go a bit like this: “Jack uses Excel. Carol can also use Excel. Therefore Carol can do Jack’s work.” It might be true that Carol can do Jack’s work, but this is not related to the fact that Jack uses Excel. There are built-in limitations to what Excel can do. However, there are no external limitations on how people use Excel. For instance, it is true that a CVS table can hold data; but the ability to use a CVS table does not mean that everybody knows how to make use of the data at the same level.
In order to test if Carol can do Jack’s work, it is only necessary to ask Carol to do Jack’s work, ignoring the fact that they make use of the same software. Excel per se might not be “designed” to carry out particular functions. The test really is whether Carol can perform Jack’s behaviours perhaps with a comparable level of understanding. One would never suggest that two artists with the same type of pencil have equal drawing capabilities by virtue of their pencil. Similarly, it is indisputably true that any person in an organization can be “replaced” – either by a machine or another person. But to the extent that the tool itself isn’t designed to perform the work, presumably some credit must be given to the person who makes use of the tool. Replacement therefore might radically alter the production outcomes – either positively or adversely depending on the replacement.
In short, a person responsible for solving problems should welcome and indeed embrace the opportunity to deal with new problems. This is what life is all about for the problem-solver. It is also important to recognize the deep cracks and crevices in the production that give rise to the expression of problems. These disturbances in the structure of the organization are permanent. They will never be corrected. Let me explain why. I call my overarching philosophy the “resource alignment principle.” Underneath this principle is the idea of “market sensitivity” and “efficient resource allocation.” No matter how a company starts off, the fact remains that the market is changing. These cracks that form in operations are perfectly normal formations that arise as an organization tries to deal with a changing market. The organization tries to be sensitive to the market in order to effectively and efficiently allocate its scarce resources. I apply the resource alignment principle (RAP) to ripples in design. This makes me a RAP star perhaps in certain circles.
How does one efficiently allocate resources in order to ensure market alignment? Indeed this is the question for me – “How?” Notice that it is not “Why?” I am not responsible for the “why” part. I leave this latter question to the managers. Of course, if managers don’t indulge their curiosity, I will never get to respond to the problem of how. It just so happens that why is quite relevant to the survival of the business; so the likelihood of having to deal with how is quite high if not relatively continuous. The crackle and spew from the tearing and ripping drives me closer to the action. “How are we going to deal with this?” That is just the right question to active me. The flurry of activity means that something is failing. That is called change. Like a hurricane, change can hurl the mightiest oak into the sky – and yet cause the humble reed merely to bend in the wind. The pounding storms create job opportunities.
