We have data-driven decision support systems implemented in Management Information Systems(MIS). Algorithms created by humans coded into MIS chew raw data and spit out decisions. The MIS systems were developed by human with substantial effort in software development. Once created, they allowed very little flexibility in deriving insights from new data sets. Now we have Machine Learning (ML) systems capable of making data-driven decisions or predictions without the need for explicit programming. At its heart ML runs on data. The algorithms used in ML systems enable machines learn independently from data and make meaningful predictions useful for making decisions.
By reading the above paragraph, some of you may think that ML is an opportunity for getting rid of the dependency on expensive software developers and MIS tools. If you think so, you are partially correct. By adopting ML systems, you are going to remove large number of traditional programmers and create dependency on small number of expensive machine learning experts aka data scientists. As a senior manager you may boast that you know software development process and you are capable of leading and guiding a team of developers to implement solutions visualized by you. When it comes to ML based solutions, you may be shocked to know that none of these beliefs help you. When a senior management person decides to adopt Machine Learning for data driven decision making, he may encounter a plethora of challenges which he may not be able address with exposure to traditional software development methods. This article is an attempt to provide some insights into the real problems in adopting ML based techniques for decision making in the corporate world. This will be useful in understanding differences between traditional and ML software. It helps you to overcome the barriers in Machine Learning adoption in the corporate world.
The barriers are of two different categories; the first one is to understand the philosophy behind ML systems and the second one is to learn the process for development of ML based solutions. The Machine Learning paradigm and the development process is totally different from that of traditional software development. The taste of all dishes made from a recipe by traditional software cooks will be identical. But, the taste of dishes made by different ML cooks from the same recipe will be different. The taste of the output of ML cooks depends on his level of experience, imagination, creativity and domain expertise.
In traditional software systems, algorithm is represented in the code and in ML based systems intelligence is represented in the model. Traditional software design starts with schema definition of data. AI development starts with accumulation of huge volume of past data. High effort in designing and transforming algorithms to code in traditional method. ML automatically captures intelligence from past data in the form of models. In ML software, the accuracy of models depends on the quality and quantity of data used in training phase of the model. ML software can work with unstructured data existing in the form of text in natural languages. Expertise needed in model development is in selection of APIs and functions for loss estimation, optimization and activation. The results generated depends on ML code and the hyper-parameters used by the developers during the training phase.
If you are planning to launch an IT solution, then you start working on vendor identification. When you are planning to launch an ML based solution you need to start accumulating data and manually generated results from it. In ML based software development, you worry about the Machine Learning framework for development and deployment of the ML system. Examples are TensorFlow, PyTorch, CNTK, SageMaker etc
When you start migrating to the ML world, you will be surprised to interact with a group of people talking a totally different language which you have never heard about. They are data scientists with very good background in data engineering, programming, statistics, machine learning, deep learning, and hyper-parameter tuning. They will be describing the solution using technical jargon which you have never heard about. With all these problems in hand, should you migrate to ML based solutions ? The world is moving into the machine learning paradigm and if ML is not part of your MIS tool or solution, it will have less value when evaluated by the next generation management experts.
How do you overcome these barriers in adopting ML solutions in your solutions. You need to forget your knowledge of traditional software development process and understand the new ML paradigm of development and the jargon used in the ML world. You need to watch many introductory videos on ML to understand the basics of the new paradigm. Once you understand the new ML paradigm of computing, you will be shocked to find that all your skills in designing and developing traditional software systems is of no use. And finally the ML system starts giving instructions to you and influences your decision making capability. I hope as an Artificial Intelligence (AI) enthusiast, you are ready to obey the instructions given by intelligent machines and happy to live peacefully in the new era of AI based systems.
WRITTEN BY
Janardhanan PS
Machine Learning Evangelist at SunTec Business Solutions Pvt Ltd., Trivandrum, India.
Domain: Machine Learning Software Development. LinkedIn:https://www.linkedin.com/in/janardhanan-ps-7a30b71/
