Skip to content
Shekhar Kumar

All projects

Open source · Building

mcp-evals

An open-source harness being built to benchmark MCP servers across models: accuracy, tool choice and context cost.

Role
Author
Period
Oct 2026 – present
Updated

TL;DR

  1. An open-source harness that runs the same tasks across models and MCP servers, grades the results, and records every step.
  2. Measures task success, invalid tool calls, context tokens returned by tool calls, steps, latency and cost, with every model pinned to one provider through OpenRouter.
  3. Milestones 1 and 2 are merged, and the first real benchmark is published: 144 runs comparing NeatCode with the filesystem MCP server across three models.
graded runs in the first real benchmark
144
less context with NeatCode (GPT-6.1 Sol)
61%
lower cost per task, same comparison
46%
GPT-6.1 Sol task success on both servers
100%
of Qwen3.8 Flash runs called a tool that doesn't exist
33–42%
for the full run, under a $10 cap
$7.10
code-navigation tasks on a pinned repo
12
tests
290

The eval matrix

Sample data

Tasks by model, pass or fail
taskmodel-amodel-bmodel-cmodel-d
find-callers
trace-flow
pick-right-tool
list-routes
explain-module
rename-impact

Select a cell to see its trace summary.

  • pass
  • fail
  • infra error (not counted)
Sample data showing the shape of a run: tasks down the side, models across the top. Cells fill as the run replays; select one for its trace summary. The first real run's results are summarised under Outcomes and published in the repo.

The problem

MCP servers and agent tools are easy to claim value for and hard to compare: results depend on the model, the prompt, the limits and the task.

mcp-evals makes the comparison fair and repeatable: same tasks, same prompts, same limits, pinned models and fixtures, with every tool call traced. Its first headline comparison is NeatCode against a file-reading MCP server on the same code-navigation tasks.

My role · Author: design spec, architecture and implementation.

How it works

  1. Suites as YAML

    Tasks, graders, servers and models are declared in a suite file, validated with zod, with pinned model IDs and fixture commits.

  2. Fresh server per task

    Every task run starts a fresh MCP server process and a fresh copy of the fixture, so runs can't affect each other.

  3. Own agent loop on the MCP SDK

    A minimal loop sends the prompt, executes tool calls over MCP, and stops at the step, time or budget limit.

  4. Graders and recorder

    Deterministic graders check answers and tool use; a recorder traces every model call, tool call and token count.

  5. One results contract

    A versioned results.json feeds the CLI's report, compare and publish commands today; a GitHub Action regression gate, a public leaderboard and this portfolio's embed are planned to read the same file.

Key decisions

  • Own agent loop on the official MCP SDK

    instead of: An agent framework or a plugin for an existing eval tool

    Full control of measurement and fairness; framework behaviour would muddy the numbers.

  • Deterministic graders first, LLM judge only when needed

    instead of: Judging everything with a model

    Cheap, reproducible checks wherever an answer can be verified; a pinned judge with a stored verdict only where it can't.

  • Infrastructure failures never count against a model

    instead of: Treating every failed run as a model failure

    Server crashes, rate limits and timeouts are retried and reported as an error rate, so rankings measure models, not flaky infrastructure.

Outcomes

  • Milestones 1 and 2 merged: runner, deterministic graders, an OpenRouter provider with token and cost accounting and a per-call budget cap, resumable runs, and `report`, `compare` and `publish` commands; 290 tests.
  • First real run (2026-10-16): 12 code-navigation tasks on the Hono repo, NeatCode vs. the filesystem MCP server, GPT-6.1 Sol, Gemini 3.8 Flash and Qwen3.8 Flash, 2 repeats: 144 runs, $7.10 against a $10 cap.
  • GPT-6.1 Sol solved all 24 runs on both servers. With NeatCode it pulled 61% fewer tool-result tokens into context (median 3,235 vs. 8,390) and cost 46% less per task, though it took more steps (6.0 vs. 4.4).
  • Both flash models scored lower with NeatCode than with the filesystem server (Gemini 3.8 Flash 0% vs. 25%, Qwen3.8 Flash 8% vs. 25%). The published traces show how they failed; the methodology notes that the harness author also wrote NeatCode.
  • Results, traces and methodology are public: github.com/HITMANdark07/mcp-evals.

What's next

  • Harder tasks (GPT-6.1 Sol passed every one) and the general-tools suite.
  • M3: a GitHub Action with PR comments and a CI gate for NeatCode.
  • M4–M5: a public leaderboard and the live embed on this page.

Stack

  • TypeScript
  • MCP SDK
  • zod
  • Commander

Links