Detect Anomalies for AIOps

See Extended Examples for complete configuration files and templates.

Apply machine learning to detect anomalies in operational metrics, correlate alerts, and reduce false positives.

适用场景

• Operations team overwhelmed by alert volume (>100 alerts/day)
• Need to detect complex multi-metric anomalies (not just threshold breaches)
• Seasonal patterns make static thresholds ineffective
• Want to predict issues before they impact users (proactive detection)
• Need to correlate related alerts to identify root cause
• Monitoring system generates too many false positives
• Want to detect subtle performance degradation trends

输入

• 必需: Time series metrics from monitoring system (CPU, memory, latency, error rate)
• 必需: Historical data (30-90 days minimum)
• 可选: Alert history with labels (true positive / false positive)
• 可选: System topology (service dependencies)
• 可选: Log data for correlation
• 可选: Deployment/change events for context

步骤

第 1 步：Set Up Environment and Load Data

Install dependencies and prepare time series data for analysis.

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install anomaly detection libraries
pip install prophet scikit-learn pandas numpy
pip install tensorflow keras  # for LSTM models
pip install pyod  # Python Outlier Detection library
pip install statsmodels  # for statistical methods
pip install prometheus-api-client  # if using Prometheus

# Visualization
pip install plotly matplotlib seaborn

Load and prepare data:

# aiops/data_loader.py
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from typing import List, Dict
import logging

logging.basicConfig(level=logging.INFO)
# ... (see EXAMPLES.md for complete implementation)

预期结果： Time series data loaded with regular intervals, missing values handled, features engineered for ML models.

失败处理： If Prometheus connection fails, verify URL and network access, if data gaps exist use forward-fill or interpolation, ensure timestamp column is datetime type, check for memory issues with large date ranges (process in chunks).

第 2 步：Implement Isolation Forest for Multivariate Anomaly Detection

Detect anomalies using unsupervised Isolation Forest algorithm.

# aiops/isolation_forest_detector.py
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
from typing import Dict, List
import joblib

# ... (see EXAMPLES.md for complete implementation)

预期结果： Model trained on historical data, anomalies detected with scores, typically 0.5-2% of points flagged as anomalies.

失败处理： If too many anomalies (>5%), reduce contamination parameter or retrain on cleaner baseline period, if too few (<0.1%), increase contamination or check feature scaling, verify features have sufficient variance.

第 3 步：Implement Prophet for Time Series Forecasting and Anomaly Detection

Use Facebook Prophet to model seasonality and detect deviations.

# aiops/prophet_detector.py
from prophet import Prophet
import pandas as pd
import numpy as np
from typing import Dict, Tuple
import logging

logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)

预期结果： Prophet models capture daily/weekly seasonality, anomalies detected when actual values fall outside 99% confidence interval, forecasts generated for capacity planning.

失败处理： If Prophet takes too long (>5 min per metric), reduce history to 30 days or disable weekly_seasonality, if too many false positives increase interval_width to 0.995, if missing seasonal patterns add custom seasonalities, ensure timezone consistency in timestamps.

第 4 步：Correlate Alerts and Identify Root Cause

Group related anomalies and identify potential root causes.

# aiops/alert_correlation.py
import pandas as pd
import numpy as np
from sklearn.cluster import DBSCAN
from typing import List, Dict
from datetime import timedelta
import networkx as nx

# ... (see EXAMPLES.md for complete implementation)

预期结果： Related anomalies grouped into incidents, root causes identified based on dependency graph, incident summaries generated for investigation.

失败处理： If all anomalies separate incidents, increase time_window_minutes, if root cause detection unclear define metric_relationships explicitly based on architecture, verify timestamp sorting is correct.

第 5 步：Integrate with Alerting System

Send intelligent alerts with context and suppression of noise.

# aiops/intelligent_alerting.py
import requests
import logging
from typing import Dict, List
from datetime import datetime, timedelta
import json

logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)

预期结果： High-severity incidents trigger PagerDuty pages, medium-severity go to Slack, low-severity logged only, duplicate alerts suppressed within 15-minute window.

失败处理： Test webhook URLs with curl first, verify severity calculation produces reasonable values (0.5-0.9 range), check rate limiting doesn't suppress all alerts, ensure timezone handling is correct for last_alerts tracking.

第 6 步：Deploy as Continuous Monitoring Service

Set up automated pipeline that runs periodically.

# aiops/monitoring_service.py
import schedule
import time
import logging
from datetime import datetime, timedelta
from data_loader import MetricsDataLoader
from isolation_forest_detector import IsolationForestDetector
from prophet_detector import ProphetAnomalyDetector
# ... (see EXAMPLES.md for complete implementation)

预期结果： Service runs continuously, detects anomalies every 5 minutes, alerts sent for incidents, logs all activity.

失败处理： Verify scheduler process stays alive (use systemd/supervisor for production), check Prometheus connectivity, ensure models are loaded successfully, implement dead man's switch alert if service stops running, monitor memory usage (reload models periodically if memory grows).

验证清单

• Historical data loaded correctly with no missing timestamps
• Isolation Forest detects known anomalies from test set
• Prophet models capture daily/weekly seasonality in visualizations
• Alert correlation groups temporally-related anomalies
• Root cause detection identifies upstream issues correctly
• Intelligent alerting suppresses duplicate alerts
• Severity calculation produces reasonable scores (0.5-0.9)
• Monitoring service runs continuously without crashes for 7+ days
• False positive rate < 10% (validated against labeled data)
• True positive rate > 80% for critical incidents

常见问题

• Training on anomalous data: Ensure baseline period used for training is clean (no incidents); manually review or use labeled data
• Ignoring seasonality: Static models fail on daily/weekly patterns; use Prophet or add time features
• Too sensitive thresholds: 99% confidence intervals may flag normal peaks; start with 99.5% and tune based on false positives
• Not handling missing data: Gaps in metrics cause model errors; implement robust preprocessing with interpolation
• Alert fatigue from low severity: Filter alerts below severity threshold; focus on high-confidence anomalies
• Ignoring system topology: Treating all metrics independently misses cascading failures; define dependency relationships
• Model drift: Models trained on old data become stale; retrain monthly or when system changes
• Resource contention: Running detection on every metric is expensive; prioritize critical services or sample metrics

相关技能

• monitor-model-drift - Detect when anomaly detection models degrade
• monitor-data-integrity - Data quality checks before anomaly detection
• setup-prometheus-monitoring - Collect operational metrics
• forecast-operational-metrics - Capacity planning with Prophet forecasts