Detect Anomalies for AIOps

See Extended Examples for complete configuration files and templates.

Apply ML to find anomalies in operational metrics. Correlate alerts, cut false positives.

When Use

• Ops team drowning in alerts (>100/day)
• Need to detect complex multi-metric anomalies (not just threshold breaches)
• Seasonal patterns make static thresholds useless
• Want to predict issues before they hit users (proactive detection)
• Need to correlate related alerts → root cause
• Monitoring creates too many false positives
• Want to spot subtle perf degradation trends

Inputs

• Required: Time series metrics from monitoring (CPU, memory, latency, error rate)
• Required: Historical data (30-90 days min)
• Optional: Alert history with labels (true positive / false positive)
• Optional: System topology (service deps)
• Optional: Log data for correlation
• Optional: Deploy/change events for context

Steps

Step 1: Set Up Environment + Load Data

Install deps. Prep time series data.

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install anomaly detection libraries
pip install prophet scikit-learn pandas numpy
pip install tensorflow keras  # for LSTM models
pip install pyod  # Python Outlier Detection library
pip install statsmodels  # for statistical methods
pip install prometheus-api-client  # if using Prometheus

# Visualization
pip install plotly matplotlib seaborn

Load + prep data:

# aiops/data_loader.py
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from typing import List, Dict
import logging

logging.basicConfig(level=logging.INFO)
# ... (see EXAMPLES.md for complete implementation)

Got: Time series loaded, regular intervals, missing values handled, features engineered for ML.

If fail: Prometheus connection fails? Check URL + network. Data gaps? Forward-fill or interpolate. Timestamp column must be datetime. Memory issues with big date ranges? Process in chunks.

Step 2: Impl Isolation Forest for Multivariate Anomaly Detection

Unsupervised Isolation Forest finds anomalies.

# aiops/isolation_forest_detector.py
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
from typing import Dict, List
import joblib

# ... (see EXAMPLES.md for complete implementation)

Got: Model trained on historical data. Anomalies detected with scores. Usually 0.5-2% of points flagged.

If fail: Too many anomalies (>5%)? Reduce contamination or retrain on cleaner baseline. Too few (<0.1%)? Increase contamination or check feature scaling. Features need variance.

Step 3: Impl Prophet for Time Series Forecasting + Anomaly Detection

Facebook Prophet models seasonality, finds deviations.

# aiops/prophet_detector.py
from prophet import Prophet
import pandas as pd
import numpy as np
from typing import Dict, Tuple
import logging

logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)

Got: Prophet models capture daily/weekly seasonality. Anomalies flagged when actual outside 99% CI. Forecasts for capacity planning.

If fail: Prophet too slow (>5 min per metric)? Cut history to 30 days or disable weekly_seasonality. Too many false positives? Raise interval_width to 0.995. Missing seasonal patterns? Add custom seasonalities. Check timezone consistency.

Step 4: Correlate Alerts + Find Root Cause

Group related anomalies. Identify root causes.

# aiops/alert_correlation.py
import pandas as pd
import numpy as np
from sklearn.cluster import DBSCAN
from typing import List, Dict
from datetime import timedelta
import networkx as nx

# ... (see EXAMPLES.md for complete implementation)

Got: Related anomalies grouped into incidents. Root causes from dependency graph. Incident summaries for investigation.

If fail: All anomalies separate incidents? Raise time_window_minutes. Root cause unclear? Define metric_relationships explicit from architecture. Check timestamp sort.

Step 5: Integrate with Alerting System

Send intelligent alerts with context. Suppress noise.

# aiops/intelligent_alerting.py
import requests
import logging
from typing import Dict, List
from datetime import datetime, timedelta
import json

logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)

Got: High-severity → PagerDuty. Medium → Slack. Low → logged only. Duplicate alerts suppressed in 15-min window.

If fail: Test webhook URLs with curl first. Severity calc should give 0.5-0.9 range. Rate limiting must not suppress all alerts. Check timezone for last_alerts tracking.

Step 6: Deploy as Continuous Monitoring Service

Auto-pipeline runs periodically.

# aiops/monitoring_service.py
import schedule
import time
import logging
from datetime import datetime, timedelta
from data_loader import MetricsDataLoader
from isolation_forest_detector import IsolationForestDetector
from prophet_detector import ProphetAnomalyDetector
# ... (see EXAMPLES.md for complete implementation)

Got: Service runs continuously. Detects anomalies every 5 min. Alerts sent for incidents. Logs all activity.

If fail: Scheduler process must stay alive (use systemd/supervisor for prod). Check Prometheus connection. Models must load OK. Add dead man's switch alert if service stops. Monitor memory (reload models periodically if growing).

Checks

• Historical data loaded, no missing timestamps
• Isolation Forest finds known anomalies in test set
• Prophet models capture daily/weekly seasonality
• Alert correlation groups temporally-related anomalies
• Root cause detection finds upstream issues
• Intelligent alerting suppresses duplicates
• Severity calc gives reasonable scores (0.5-0.9)
• Monitoring service runs continuously 7+ days, no crash
• False positive rate < 10% (vs labeled data)
• True positive rate > 80% for critical incidents

Pitfalls

• Training on anomalous data: Baseline period for training must be clean (no incidents). Manually review or use labeled data.
• Ignoring seasonality: Static models fail on daily/weekly patterns. Use Prophet or add time features.
• Too sensitive thresholds: 99% CI may flag normal peaks. Start 99.5%, tune by false positives.
• Not handling missing data: Gaps cause model errors. Robust preprocessing with interpolation.
• Alert fatigue from low severity: Filter below threshold. Focus on high-confidence.
• Ignoring system topology: Treating metrics independent misses cascading failures. Define deps.
• Model drift: Old-data models go stale. Retrain monthly or on system change.
• Resource contention: Running detection on every metric = expensive. Prioritize critical services or sample.

See Also

• monitor-model-drift - Find when anomaly models degrade
• monitor-data-integrity - Data quality checks before anomaly detection
• setup-prometheus-monitoring - Collect operational metrics
• forecast-operational-metrics - Capacity planning with Prophet forecasts