1.4Step 2: Clustering User Behavior

Use scikit-learn to cluster production sessions into behavioral personas:

# ai_load_profiler.py -- Cluster production traffic into user personas
import pandas as pd
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler

# Load production access log data
logs = pd.read_csv("production_access_logs.csv")

# Feature engineering: extract behavioral signals from raw logs
features = logs.groupby("session_id").agg(
    request_count=("path", "count"),
    unique_endpoints=("path", "nunique"),
    avg_response_ms=("response_time_ms", "mean"),
    session_duration_s=("timestamp", lambda x: (x.max() - x.min()).total_seconds()),
    error_rate=("status_code", lambda x: (x >= 400).mean()),
    write_ratio=("method", lambda x: (x.isin(["POST", "PUT", "DELETE"])).mean()),
).reset_index()

# Scale features for clustering
scaler = StandardScaler()
X = scaler.fit_transform(features[[
    "request_count", "unique_endpoints",
    "avg_response_ms", "session_duration_s",
    "write_ratio",
]])

# Use the elbow method or silhouette score to pick k
# For most web apps, 3-6 personas capture the meaningful variation
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
features["persona"] = kmeans.fit_predict(X)

# Name the clusters based on their centroids
persona_names = {0: "power_user", 1: "casual_browser", 2: "api_consumer", 3: "bot_crawler"}
features["persona_name"] = features["persona"].map(persona_names)

# Display persona profile summary
print(features.groupby("persona_name").agg(
    count=("session_id", "count"),
    avg_requests=("request_count", "mean"),
    avg_duration=("session_duration_s", "mean"),
    avg_write_ratio=("write_ratio", "mean"),
).to_markdown())

Interpreting Cluster Results

A typical e-commerce site produces personas like these:

Persona	% of Traffic	Avg Requests	Avg Duration	Write Ratio	Behavior
Casual Browser	55%	4.2	45s	0.02	Views homepage, browses 2-3 products, leaves
Power User	20%	18.7	340s	0.15	Deep browsing, add-to-cart, checkout, account management
API Consumer	15%	42.0	1800s	0.30	Automated integrations, consistent request patterns
Bot/Crawler	10%	85.0	3600s	0.00	Sequential page crawling, no interaction