Imagine…imagine that you have been challenged to play Steph Curry, the greatest 3-point shooter in the history of the National Basketball Association, in a game of 1×1. Yea, a pretty predictable outcome for 99.9999999% of us.
But now image that Steph Curry has to wear a suit of knight’s armor as part of that 1×1 game. The added weight, the obstructed vision, and the lack of flexibility, agility and mobility would probably allow even the average basketball player to beat him.
Welcome to today’s technology architecture challenge!
For many companies, their technology architecture is becoming more of a hindrance than an enabler of value creation — it resembles an archeological dig, with layers of ancient technologies layered on top of even more ancient technologies. The result: added weight, obstructed vision, and lack of flexibility, agility and mobility.
Modern digital companies like Google, Facebook, Twitter, Apple, Netflix, Amazon, and AirBnB have taken a technology architecture approach that increasingly treats the technology infrastructure as “disposable” using open source technologies.
And the reason for this open approach, in my humble opinion, is two-fold:
- Firstly, building upon open source technologies provides the flexibility, agility and mobility for companies to move to the next best technology without the constraints and architectural lock-in of traditional technology. Traditional technologies advance at the rate of the proprietary technology vendors, who dictates new technology capabilities on a rigid, unbending 9 to 12-month product release schedule for which customers get the honor of paying a 30%+ annual maintenance fee. Modern digital companies are basing their technology infrastructure on open source technologies that not only prevents vendor architectural lock-in but also allows them to advance the technology capabilities at their pace and at the pace of the business.
- Secondly and more importantly, these digital companies understand that the technology isn’t the source of business value and differentiation. They understand that the source of business value and differentiation is 1) the data that these organizations are masterfully amassing via every customer engagement and every usage of the product or service, and 2) the customer, product and operational insights (Intellectual Property in the form of customer, product and operational propensities, tendencies, associations, relationships and patterns) that leads to new Intellectual Property (IP) monetization and commercialization opportunities.
Let’s drill into these two tenets to see the take-aways for any company interested in digitally transforming their business models can replicate.
Understanding the Modern Digital Business Architecture
Digital companies understand that trying to compete using traditional technologies is like Steph Curry being forced to wear full knight’s armor while competing in a basketball game. For a great lesson in modern technology, let’s look at Lyft’s strategy for structuring their technology architectures (from the article “What’s Behind Lyft’s Choices in Big Data Tech”).
The company:
- Started with AWS Redshift but transitioned to Apache Hive when it started to run into scalability issues.
- Migrated to Presto to provide a more powerful query engine that supports data exploration analytics across multiple data sources.
- Uses Apache Spark to support massive ETL batch processing and to train its machine learning models.
- Uses Druid, a column-oriented in-memory OLAP data storethat excels at performing drill-downs and roll-ups over a large set of high dimensional data.
- Uses Jupyter (data science team), a popular notebook-style interface for working with data and machine learning algorithms, and the PySpark library.
- Uses Apache Airflow which creates repeatable data engineering and data science workflows that can be executed atop the workflow orchestration tool Kubernetes.
- Uses a mixture of Apache Kafka, Apache Flink, and Spark to build streaming services.
Hive, Presto, Spark, Druid, Jupyter, Airflow, Kafka, Flink, Kubernetes…crazy-assed names for open source technologies that one is not going to find from the monolithic technology vendors of yesteryear (see Figure 1).
Figure 1: Source: Hortonworks
Lyft and other modern digital companies understand that their technology architecture should serve two basic purposes:
- Facilitate the capture, refinement, curation, sharing, management, governance and analysis of the company’s invaluable data assets.
- Build a technology architecture that doesn’t get in the way of point #1.
The Economics of the Modern Business
Data is the economic asset of lasting and differentiated value. Data is the source of customer, product and operational insights that the modern company uses to differentiate their products and services while driving towards operational excellence (e.g., reducing unplanned downtime, reducing procurement / logistics / inventory / manufacturing / distribution costs, increasing customer acquisition / retention / maturation / advocacy). And Data Science is the heart of the data value creation process.
These data and analytics-centric companies are tightly integrating the data science and business teams in order to define the parameters of analytics success; to identify, capture and operationalize the sources of customer, product and operational value creation (see Figure 2).
Figure 2: Data Science Value Engineering Framework
These organizations are mastering the integration of the DataOps, Data Science and DevOps disciplines to fuel their data monetization value chain (see Figure 3).
Figure 3: Data Monetization Value Chain
DataOps, Data Science, DevOps – all focused on accelerating the monetization of an asset that “The Economist” magazine declared “The world’s most valuable resource”. See “Data Curation: Weaving Raw Data into Business Gold” for more on monetizing the world’s most valuable resource.
Disposable Technology Summary
So, what lessons can we take away from how modern digital companies?
- Lesson #1: Focus on aligning the organization on identifying, capturing and operationalizing new sources of customer, product and operational value buried in the company’s data.
- Lesson #2: Don’t implement a rigid technology architecture that interferes with Lesson #1.
These modern digital companies, through their aggressive open source architecture strategies, realize that they are not in the technology architecture business; they are in the data monetization and commercialization business.
Now, let’s take that stupid armor off of Steph Curry and let it rain 3’s!
