Introduction: The AI Observability Challenge

In the classic film Terminator, we see the world through the machine’s eyes: a red-tinted HUD with scrolling text, analysing targets and calculating probabilities. In modern engineering, we have built the machines – LLMs – but we often lack that “red HUD”. We send a prompt, receive a response and cross our fingers that the bill will not be too high and the answer will not be an AI hallucination.

As GenAI moves from a “cool prototype” to a “production-ready agent”, the standard observability stack – metrics, logs and traces – is becoming insufficient. Why? Because LLMs are non-deterministic and expensive. Evaluating a JSON API is straightforward (200 OK versus 500 Error); evaluating an LLM response is a matter of nuance, semantics and cost-benefit analysis.

In this post, we explore how to design a complete GenAI observability layer using Grafana, Langfuse and OpenTelemetry, delivering the following benefits:

• Single Pane of Glass: a unified dashboard where we can view our KPIs.
• Trends and detail: visibility into KPI trends, with the ability to drill down into each metric.
• Evaluations: offline and online evaluations to ensure that our AI systems are operating correctly.

 

 

1. The Gap: Why Traditional Tracing Is Not Enough

Standard OpenTelemetry (OTel) is excellent for microservices. It tells you that Service A called Service B. But when Service B is an LLM, OTel often misses what matters most:

• Tokens and costs: How much did this specific user cost us? How much are we spending with that provider? Which is the most expensive model we are using?
• Model parameters: Was this response generated with temperature: 0 or 0.7?
• Intermediate steps: In an agent loop, what did the LLM “think” before calling a tool?

What did the MCP servers return? Which step was the most expensive?

• Quality metrics: Was the response toxic? Was it factually correct, or did the AI hallucinate and invent the answer?

 

To address this, we need an observability stack purpose-built for GenAI.

 

2. Architecture: The GenAI Observability Stack

At Datadope, our architecture follows a Single Pane of Glass approach, combining technical performance with business value.

The components are:

1. Langfuse SDK: Built on OTel, it instruments the application to capture prompts, completions and metadata.
2. Langfuse Stack: A specialised backend that uses ClickHouse for high-performance telemetry storage.
3. Grafana: The visualisation layer, querying data directly from ClickHouse to create unified dashboards.

 

3. Implementation: Instrumenting the Agent

Let us look at how we instrument a Python service. The key is preserving context. Using OTel Baggage, we can inject metadata such as the user ID to trace and analyse the complete user journey.

 

4. Advanced Use Cases: Beyond “Does It Work or Not?”

A. Deep Tracing

When a system interacting with LLMs uses tools, MCPs or other intermediary systems – for example, by calling a database or search engine – its execution logic can become complex. This observability solution makes those intermediate steps visible and provides complete insight into system behaviour. For example, consider a trace where the LLM decides to use a tool:

• Input: The LLM’s decision to use a tool.
• Action: The actual database call.
• Observation: The data returned to the LLM.
• Synthesis: The final response.

 

This is the “Eureka!” moment for developers debugging why an agent became trapped in a loop or selected the wrong tool.

 

B. Tracking Evolution: Trends and Performance

Observability is not only about the present; it is also about trends. By aggregating telemetry in ClickHouse, we can visualise how KPIs evolve over time to identify regressions or scaling issues:

• Cost scaling: Is our spend growing linearly with our user base, or is there a hidden inefficiency in our prompts?
• Latency drift: Identify whether a recent prompt change or model update has caused a gradual increase in p95 latency.
• Token efficiency: Monitor tokens per request to ensure that agents do not become trapped in redundant loops.

C. LLM-as-a-Judge: Automated Scoring

How do you know whether your LLM is operating correctly? Manual review does not scale, so we implement an evaluation workflow:

1. A request is received and stored.
2. Un segundo LLM que actúa como “Juez” lee el par entrada/salida.
3. The Judge assigns a score from 0 to 1 based on a rubric – for example: “Is the information factually supported by the context provided?”
4. We trigger an alert if the aggregated Correctness Score falls below a defined threshold.
5. . We also measure other relevant scores, including Toxicity and Relevance.

 

Finally, we display the results of these evaluations in our Single Pane of Glass dashboard:

 

5. Business Value: The Economics of Generative AI

GenAI is not only a technical challenge; it is also a financial one. Using Grafana to query the ClickHouse backend enables us to build dashboards that display:

• Cost per user/tenant: Essential for SaaS products.
• Model efficiency: Are we using GPT-4 for simple tasks where GPT-4o-mini would be sufficient?
• Latency: The most critical user experience (UX) metric for LLM responses.

6. Lessons from the Front Line

Having implemented this approach for several customers, these are our key recommendations:

1. PII sanitisation: Never send personally identifiable information (PII) to the observability backend. Use SDK-level masking or an OTel Collector to filter it out
2. Response latency: Instrumentation can add overhead. Ensure that your SDK operates asynchronously and does not block the response delivered to the user.
3. Baggage is your friend: Correlation IDs in the OTel Baggage header are essential for linking user behaviour to LLM costs.

 

Conclusion: Moving Forward with Confidence

Observability in the GenAI era is about moving from “I hope this works” to “I know this works”. By implementing a robust stack and following best practices, teams can finally unravel the layers of the “Black Box”, optimising cost, performance and, most importantly, trust.

Ready to shine a light on your GenAI stack? At Datadope, we specialise in making complex systems measurable. Let’s talk about how to turn your AI into a governable