It has become evident that developments in analytics are creating new occupations. There has been much discussion about where new jobs will come from with many existing ones being made redundant because of the 4th Industrial Revolution – i.e. the impact of artificial intelligence and robotics. Analytics is bucking this trend.
Some of new occupations in analytics include data prospectors and data harvesters. Data prospectors, like gold prospectors, are responsible for searching and locating data on the internet and other large data repositories. Data harvesters are responsible for extracting data and information from these sources. Data harvesters do this, for example, by web scraping. Staff who are highly skilled and knowledgeable in doing these functions are required – especially exploring something as vast and intricate as the internet.
Another new occupation is that of a data detective. They are analysts who find knowledge and insights in data. This may sound a simple and straight forward job to do.
It is suggested that there are plenty of analysts who can do extraction and cleaning tasks but have little or no aptitude for exploring data to find answers to difficult problems and issues and struggle to recognise important and informative discoveries. That is, they can perform the technical tasks of providing data but are not able to use it to find ‘nuggets of gold’ in this resource.
What is required are highly skilled professionals who, like police detectives, excel at analysis and problem solving. They need to be proficient in marshalling facts, following leads in data, testing hypotheses and hunches, joining the ‘dots’ and drawing conclusions from what is known. In short, they require the knowledge and skills of a Sherlock Holmes.
The primary skills required by data detectives are the ability to explore data and the ability to identify items of interest. They can do this by using the functionality of desktop packages such as Microsoft Excel and Microsoft Access and data visualization packages such as Tableau, QlikView and Power BI. They can also interrogate data using SQL with structured data and SPARQL with semantic data.
Where data detectives add value is that they ask informed questions to help to understand challenging and difficult problems and issues. They find workarounds when they hit difficulties and obstacles in obtaining the answers they require. They possess the nous, have the patience, and have the persistence to go the extra kilometre to find interesting patterns and trends in data.
Three examples of where data detectives can add value include using risk-analysis tools to gain insights into threats and opportunities. They can take different data views of subjects and issues and where interesting patterns are found, they can make further inquiries to find more about what is going on and what their implications are when it comes to developments that can either harm or benefit individuals, organizations and the community.
The second example is stratifying a population to find interesting strata such as those with high incidence of a disease such as COVID-19. They can analyse cases in different strata to see why they have high infection rates and compare these with strata with low infection rates. These analyses can reveal what measures can be taken to lower the incidence of the disease.
The third example is analysing cases that have anomalies with insurance claims. Business rules can be written for those who show unusual patterns and the rules can be cascaded to find other people in the population who closely match them as they too may have issues with their claims.
It is suggested that data detective work needs to be recognized as a specialist skill where those with requisite attributes are selected, trained, and employed to do this work. Organizations need to take steps to identify those who are gifted in doing detective tasks and use their talents.
They complement data scientists who use mining and modelling techniques to extract knowledge from data. Data detectives are more qualitative in their approach while data scientists are more quantitative in their orientation. However, data detectives use the tools and procedures developed by data scientists to explore data such as using population partitioning techniques.
Data detectives can go the extra step of interpreting what data scientists find in data and can give context to what is discovered or detected. For example, data scientists can produce a list of high-risk cases detected using a machine-learning model but often they cannot explain why they are classified in this manner. Data detectives can explore data to give context to the cases and explain why they were identified by the model. They can also spot false positives or cases that appear to be of concern but are false alarms and therefore do not warrant attention. This saves time, money and effort in that resources are not wasted pursuing them.
Data detectives are part of the broader and growing family of occupations that deal with data. This family includes as examples data prospectors, data harvesters, data scientists, data analysts, data engineers, data architects, data brokers, data lawyers, data journalists, data artists, data quality officers and database managers. They each have a discrete and important role to perform and they all complement each other in making use of what is now referred as the new oil. Data is now the fuel that enables organizations to function and to deliver business outcomes.
When it comes to formal education, there are now many masters programs in analytics in universities across the globe. These programs could be expanded to include different specialization streams to cater for these different data occupations cited above. That is, they become omnibus programs where students can select relevant subjects that enable them to specialize in data science or data engineering or data brokering or data detective work to use examples. These specializations are required to provide practicing analytics professionals to meet the diverse needs of government, industry, and commerce in the 21st Century.
Bio
Warwick Graco is a practicing data scientist and can be contacted at [email protected]
