THE INDEX · OBSERVABILITY & EVALUATION
Observability & Evaluation
Tracing, logging, analytics, and evaluation for MCP servers and the agents that call them, in production rather than at development time.
12 listings
Compare observability & evaluation tools →︎How will you trace quality and operations?
What belongs in this category
- Tracing and logging platforms with documented MCP or tool-call support
- Analytics aimed at MCP server operators
- Evaluation and LLM-as-judge tooling for agent and tool-call quality
Arize Phoenix
Observability & Evaluation
Arize Phoenix is an open-source observability platform for LLM applications, covering tracing, evaluation, prompt playgrounds, datasets and experiments. It can be self-hosted, and AI assistants such as Claude Code, Claude Desktop and Cursor can connect to a Phoenix instance over MCP to explore its projects, traces, prompts, datasets and experiments.
Hybrid
Braintrust
Observability & Evaluation
Braintrust logs production traces from LLM applications and agents and runs scored evaluations of their outputs against datasets. Prompts, scorers, experiments, and human review of logged data sit in the same interface.
Hybrid
Helicone
Observability & Evaluation
Helicone is an LLM observability platform that logs an application's requests to model providers and shows traces, sessions, cost and latency for them. It also runs an AI Gateway, an OpenAI-compatible API that reaches over 100 models with routing and automatic fallbacks.
Hybrid
Ithena
Observability & Evaluation
Ithena is a governance layer for MCP servers built on the TypeScript `@modelcontextprotocol/sdk`: its open-source `@ithena-one/mcp-governance` SDK wraps the base `Server` class in a `GovernedServer` that adds pluggable identity resolution, role-based access control, credential injection, auditing, request-scoped logging and trace-context propagation. A hosted Ithena Managed Platform providing backends for those interfaces is described as upcoming, with a waitlist.
Library
Laminar
Observability & Evaluation
Laminar is an OpenTelemetry-native observability platform for AI agents that traces every LLM call, tool call and sub-agent in a run. Alongside tracing it provides Signals, which read each run for behaviours described in plain English and cluster the matches, plus evaluations, datasets, dashboards, and CLI and MCP access that let a coding agent query traces with SQL.
Hybrid
Langfuse
Observability & Evaluation
Langfuse is an open-source AI engineering platform for tracing, evaluating and iterating on LLM applications and agents. Its hierarchical traces capture every LLM call, tool invocation and retrieval step, and it also provides prompt management, datasets, experiments and evaluators using LLM-as-a-judge, heuristic functions or human review.
Hybrid
LangSmith
Observability & Evaluation
LangSmith is an LLM application observability and evaluation platform. It traces applications built with any LLM framework, including via OpenTelemetry, and provides dashboards and alerting to monitor production performance.
Hybrid
AgentCat
Observability & Evaluation
AgentCat is an analytics and observability service for MCP servers: you add its SDK to your server and it records each agent session, every tool call with its request, response and error, and a goal inferred for the session. SDKs are published for TypeScript/JavaScript, Python and Go, the service runs in AgentCat's cloud or on your own infrastructure, and it can forward the enriched data to Datadog, Sentry or PostHog over OpenTelemetry.
SaaS
Opik
Observability & Evaluation
Opik is an Apache-2.0 licensed platform for tracing and evaluating LLM applications and AI agents, recording each step an agent takes from user interactions through context retrieval to tool calls. It supports datasets, experiments, code-based metrics, LLM-as-a-judge and online evaluation, and runs either on Comet's hosted Opik Cloud or self-hosted.
Hybrid
Shinzo
Observability & Evaluation
Shinzo is an open-source observability platform for AI agents and MCP servers, used to ingest and analyse their performance data. Telemetry is OpenTelemetry-compliant, stored in Postgres, and viewed through an analytics dashboard.
Hybrid
Traceloop
Observability & Evaluation
Traceloop uses OpenTelemetry to monitor and trace LLM applications. Instrumentation is either the OpenLLMetry SDK installed in the application or Traceloop Hub, a proxy that LLM calls are routed through.
Hybrid
Weave
Observability & Evaluation
Tracing and evaluation for LLM applications including tool calls
Hybrid