Share via

Checklist to help detect and adapt to drift

  • Article

This checklist helps you get started with understanding your potential drift context in order to make the best decisions about planning drift detection and adaptation strategies, and drift response.

Drift detection

  • Are you already performing data validation and data quality analysis?
    • Do these checks ensure data quality issues are caught and managed so that they will not poison drift detection?
  • What would the consequences be if a drift was undetected and unaccounted for?
  • What are some examples of drift, or deviation from the norm, that you've experienced in the past?
    • How have you seen your incoming data change over time?
      • In feature value distribution? (For example, the most common customer age range used to be 30-40 and is now 50-60)
      • In velocity?
      • In confounding factors?
  • Give some examples of drift scenarios that you could imagine occurring in the future that would require the system to update/adapt.
    • Are future drifts expected to happen gradually, abruptly or both?
    • Are any future drifts expected to be of high magnitude?
    • How frequently might you expect future drifts to occur?
    • How might future drifts manifest (for ex. incremental, dispersed)?
  • Is there currently a drift detection strategy? If so, describe it.
    • Is this implementation currently done via manual investigation as required?
    • Is there currently a monitoring system in place?
    • What drift characteristics does this strategy provide to the user (for ex. drift magnitude)?
    • What has worked well about this strategy?
    • Are there any pain points with this strategy?
  • What is the data velocity (update frequency and batch size)?

Adaptation

  • Is there currently an adaptation strategy? If so, describe it.
    • If there is regular update, what is the frequency?
    • What size is the training set for retrain (for ex. all historical data, or just the last 3 years)?
    • What's the evaluation strategy for the updated version?
    • What has worked well about this strategy?
    • Are there any pain points with this strategy?
  • What would be the ideal method of adaptation from the user perspective?
  • If adaptation is not automatic, what information would be useful for the user in performing adaptation?
  • Is a playbook in place and accountable teams assigned to drift response?