Drift Detection Rules

Detect when your AI agent's behavior statistically deviates from its baseline.

What is Drift?

Drift occurs when the statistical properties of your AI agent's outputs change over time. This can indicate:

  • Model degradation - Performance declining over time
  • Data distribution shift - Input patterns changing
  • Prompt changes - Unintended prompt modifications
  • External API changes - Third-party service updates
  • Configuration drift - Settings changed accidentally
ℹ️
Drift vs Anomaly
Drift is a gradual change in the overall distribution.Anomaly is an individual outlier. Both are important to monitor.

Creating Drift Rules via UI

Step 1: Navigate to Drift Rules

Go to Controls → Drift Rules in the sidebar.

Step 2: Create New Rule

Click Create Rule and configure:

  • Name - Descriptive name (e.g., "Latency Drift Monitor")
  • Workflow - Select specific workflow or "All Workflows"
  • Metric - Metric to monitor for drift
  • Baseline Window - Historical period for baseline (e.g., 7 days)
  • Detection Window - Recent period to compare (e.g., 1 hour)
  • Sensitivity - Low, Medium, or High

Step 3: Configure Alerts

  • Severity - Warning or Critical
  • Auto-create Incident - Toggle for automatic incidents
  • Alert Channels - Select notification channels

Creating Drift Rules via SDK

create_drift_rule.py
import requests

# Create a drift detection rule
response = requests.post(
    "https://api.turingpulse.ai/api/v1/config/drift-rules",
    headers={"Authorization": "Bearer sk_live_..."},
    json={
        "name": "Latency Drift Monitor",
        "metric_name": "latency_ms",
        "detection_methods": ["jsd", "psi", "ks_test"],
        "reference_window_size": 500,   # Baseline sample count
        "detection_window_size": 50,    # Recent sample count
        "min_reference_samples": 30,    # Minimum before detection
        "jsd_threshold": 0.1,           # JSD divergence threshold
        "psi_threshold": 0.2,           # PSI threshold
        "ks_alpha": 0.05,               # KS test significance level
        "direction": "both",            # Detect increase or decrease
        "severity": "warning",
        "alert_channels": ["slack://alerts"],
        "enabled": True,
    }
)

Drift Detection Methods

Kolmogorov-Smirnov Test (ks_test)

Compares the cumulative distribution functions of baseline and detection windows. Best for continuous metrics like latency.

Chi-Square Test (chi_square)

Compares frequency distributions. Best for categorical metrics or binned data.

Population Stability Index (psi)

Measures how much a distribution has shifted. Common in ML monitoring.

Jensen-Shannon Divergence (js_divergence)

Symmetric measure of distribution similarity. Good for probability distributions.

MethodBest ForThreshold Guide
ks_testContinuous metricsp-value < 0.05
chi_squareCategorical datap-value < 0.05
psiGeneral purpose> 0.1 (warning), > 0.25 (critical)
js_divergenceProbability distributions> 0.1

Drift Rule Configuration

OptionTypeDescription
namestrHuman-readable name
workflow_idstrWorkflow to monitor, or "*" for all
metricstrMetric to monitor (latency_ms, tokens, cost, etc.)
baseline_windowstrHistorical window (e.g., "7d", "30d")
detection_windowstrRecent window to compare (e.g., "1h", "6h")
sensitivitystrlow, medium, high
methodstrStatistical test method
thresholdfloatDetection threshold
min_samplesintMinimum samples before detection

Viewing Drift Events

When drift is detected:

  1. A drift event is created and visible in Operations → Overview → Drift tab
  2. The event shows the baseline vs current distribution
  3. Notifications are sent to configured alert channels
  4. If auto_create_incident is enabled, an incident is created

Drift Event Details

  • Metric - Which metric drifted
  • Baseline Stats - Mean, std, percentiles of baseline
  • Current Stats - Mean, std, percentiles of detection window
  • Drift Score - Statistical significance of the drift
  • Affected Runs - Runs in the detection window

Fingerprint-Based Drift

TuringPulse also detects drift based on workflow fingerprints:

  • Prompt Changes - Detects when prompts are modified
  • Config Changes - Detects model parameter changes
  • Structure Changes - Detects workflow DAG changes
from turingpulse_sdk import init, FingerprintConfig

init(
    api_key="sk_live_...",
    fingerprint=FingerprintConfig(
        enabled=True,
        capture_prompts=True,   # Detect prompt changes
        capture_configs=True,   # Detect config changes
        capture_structure=True, # Detect DAG changes
    )
)
💡
Root Cause Analysis
When drift is detected, TuringPulse automatically correlates it with fingerprint changes and deployments to help identify the root cause.

Best Practices

  • Choose appropriate baseline windows - 7 days is good for most use cases. Use longer windows for seasonal patterns.
  • Set minimum samples - Ensure enough data before detecting drift to avoid false positives.
  • Start with medium sensitivity - Adjust based on false positive/negative rates.
  • Monitor multiple metrics - Create rules for latency, tokens, and custom metrics.

Next Steps