Custom KPIs

Track business-specific metrics alongside technical telemetry

Overview

While TuringPulse automatically captures technical metrics (latency, tokens, costs), you can define and track custom KPIs that matter to your business. Track customer satisfaction scores, task completion rates, accuracy metrics, or any other measurable value.

Quick Start

from turingpulse_sdk import instrument, KPIConfig

@instrument(
    name="my-agent",
    kpis=[
        KPIConfig(
            kpi_id="customer_satisfaction",
            value=lambda ctx: ctx.metadata.get("csat_score", 0),
            description="Customer satisfaction score (1-5)",
        ),
    ]
)
def my_agent(query: str):
    result = process(query)
    return result

KPI Categories

Organize your KPIs into categories for better dashboard visualization:

Category	Use For	Examples
`PERFORMANCE`	Technical metrics	Cache hit rate, retry count, queue depth
`QUALITY`	Output accuracy	Accuracy score, relevance score, F1 score
`BUSINESS`	Business outcomes	Conversion rate, revenue impact, cost savings
`USER`	User experience	Satisfaction, NPS, task completion rate
`CUSTOM`	Anything else	Domain-specific metrics

Defining KPIs

KPIs are defined using KPIConfig on the @instrument() decorator:

from turingpulse_sdk import instrument, KPIConfig

@instrument(
    name="my-agent",
    kpis=[
        KPIConfig(
            kpi_id="accuracy",
            description="Response accuracy score",
            value=lambda ctx: ctx.result.get("score", 0) if ctx.result else 0,
            alert_threshold=0.85,
            comparator="lt",
        ),
        KPIConfig(
            kpi_id="latency_ms",
            description="End-to-end latency",
            use_duration=True,
            alert_threshold=5000,
            comparator="gt",
        ),
        KPIConfig(
            kpi_id="token_count",
            description="Total tokens used",
            from_result_path="tokens",
            alert_threshold=8000,
            comparator="gt",
        ),
    ],
)
def my_agent(query: str):
    result = llm.chat(query)
    return {"response": result.text, "score": result.confidence, "tokens": result.usage.total_tokens}

Threshold Alerting

When a KPI value crosses its alert_threshold, TuringPulse automatically triggers alerts to your configured channels:

KPIConfig(
    kpi_id="error_rate",
    description="Error rate",
    value=lambda ctx: 1.0 if ctx.error else 0.0,
    alert_threshold=0.05,
    comparator="gt",
)

ℹ️

Automatic Alerting

When thresholds are set, TuringPulse automatically creates alerts when values cross the defined boundaries.

Best Practices

Name consistently - Use snake_case and descriptive names
Set thresholds - Define warning/critical levels for alerting
Add metadata - Include context for better analysis
Choose categories - Group related KPIs for dashboard organization
Document units - Include units in descriptions (%, ms, count)