Why monitor prediction drifts?
Prediction drift allows you to monitor a change in the distribution of the predicted label or value.
For example, a larger proportion of credit-worthy applications when your product was launched in a more affluent area. Your model still holds, but your business may be unprepared for this scenario.
For this monitor type, you can select the following detection methods:
- Anomaly Detection - The distribution of the inspected data is compared to the distribution of a time period before the data was collected.
- Compared To Segment - The distribution of the inspected data is compared to the distribution of a different data segment.
- Compared To Training - The distribution of the inspected data is compared to the distribution of the reported training data.
Start from choosing the predictions you'd like to monitor. You can select as many as you want :-)
The monitor will compare the distributions of these predictions between the inspection period to the baseline you chose. An alert is raised if the monitor finds a drift between these distributions.
Note that the monitor configuration may vary between the detection method you choose.
You can work with the monitor preview and play with these thresholds to make sure you have a healthy amount of alerts. Note that the thresholds are different for numeric predictions vs. categorical predictions.
How are drifts calculated?
If you need to use other metrics, please contact us.