Because of concept drift in machine learning, models built for forecasting will almost always experience drops in performance and accuracy over time. Can artificial intelligence engineers prevent it from happening? Discover how to address concept drift below.
What is concept drift in machine learning?
In machine learning, concept drift is a change in the relationship between the input data and the target variable — what the model tries to predict — that takes place over time. It happens because the target variable changes unexpectedly, effectively rendering some of the training data invalid.
Machine learning algorithms are trained to recognize the relationship between the input data and the target variable. Their goal is to leverage those findings in real-world datasets. This process works by applying a mapping function to the given input data to predict an output value.
While AI engineers operate under the assumption that mapping is static — that the predictors from the historical dataset don’t change — it often shifts over time. The definition of a metric can change as variables that weren’t available before are added, resulting in incompatibility. When the target variable’s properties change, old data points aren’t as valid as they once were.
Sudden drifts occur within a short time frame when the target variable’s properties or meaning changes abruptly. A gradual drift — which is far more likely since an algorithm won’t perform at its peak indefinitely — occurs over a long period as a newer concept replaces an old one. Recurring shifts can happen if the meaning repeatedly fluctuates between two or more options.
If an AI engineer were developing a time series model to predict the weather, not specifying seasons would suddenly cause concept drift to occur when the seasons change. A good example of gradual drift involves a fraud prediction algorithm. Payment systems transitioning from cash to card-based would create a mismatch between historical and current datasets.
How does concept drift affect model performance?
Non-stationary data can cause spurious regression problems, where the apparent relationship between two variables doesn’t actually exist, resulting in poor predictive capabilities. As the algorithm sees more examples, its accuracy decreases and its error rate increases. The outcome is inaccurate forecasting.
Concept drift results in model decay, which is the degradation of a machine learning algorithm’s predictive ability. This is a problem for a few reasons. For one, it drains the budget. The price of data errors continuously increases over time if professionals don’t address them.
Lost value is another problem — concept drift eventually makes forecasts worthless. Although experts expect the predictive analytics market will reach $34 billion by 2030, faulty or skewed output isn’t worth anything. If AI engineers and data scientists want to generate meaningful returns, they must address this issue.
Tips for managing concept drift in machine learning
Since concept drift in machine learning is often inevitable, AI engineers and data science professionals must understand how to manage it.
1. Continuously monitor model performance
Continuously monitoring a model’s predictive performance over time lets engineers recognize trace deviations that signal concept drift. They should assess performance using the same evaluation metrics they used during training. Setting a threshold enables them to trigger a response when accuracy dips too low without constantly adjusting parameters and datasets.
2. Weigh the importance of input data
For gradual drift, incrementally lowering the weight of older data over time helps mitigate any large performance drop brought on by concept drift. Using a weighting system that is inversely proportional to the age of data helps the algorithm value recent input more, enabling it to adjust its decision-making and predictive capabilities.
3. Remove systematic changes via differencing
Preparing for expected changes — like seasonality or a shift in buying behavior — helps AI engineers maintain their algorithm’s performance. Differencing is a technique for turning a non-stationary time series into a stationary one, effectively eliminating the issue of concept drift in machine learning models. If it remains constant over time, forecasting becomes easier.
4. Periodically update the training data
Periodically updating the algorithm with more recent historical information — meaning getting rid of older data points — can help mitigate concept drift in machine learning. Of course, testing is necessary to determine what amount can be removed at once without affecting performance.
5. Retrain the model using a new dataset
Periodic retraining to maintain accuracy above a certain threshold is time-consuming and costly. Sourcing enough data could cost anywhere from $10,500 to $85,000 — tools, preparation and labor expenses included — if AI engineers start from the ground up. However, it all but guarantees they can substantially increase accuracy above their minimum threshold.
Fixing concept drift in machine learning takes time
Even though concept drift in machine learning is a massive issue, many AI engineers and data scientists don’t consider how changes in a real-world dataset will impact the relationship between their target variable and input. Data is constantly changing, so their fixes are temporary. They must continue addressing the issue as long as they use their algorithm.
