14.2Surface 1: Contract Testing MCP Tool Integrations

An agent's tools are exposed through MCP (Model Context Protocol) servers, each publishing a list of tools with JSON schemas. This is a contract surface, and it deserves the same treatment you give REST contracts: schema validation plus breaking-change detection.

import json
import pytest
from mcp_client import connect  # any MCP client library

EXPECTED_TOOLS_SNAPSHOT = "contracts/payments_mcp_tools.json"

@pytest.fixture
def mcp_session():
    with connect("http://localhost:8931/mcp") as session:
        yield session

def test_tool_list_matches_contract(mcp_session):
    """Detect tools appearing, disappearing, or changing shape."""
    tools = {t.name: t.input_schema for t in mcp_session.list_tools()}
    expected = json.load(open(EXPECTED_TOOLS_SNAPSHOT))

    assert set(tools) == set(expected), (
        f"Tool set changed. Added: {set(tools) - set(expected)}, "
        f"Removed: {set(expected) - set(tools)}"
    )
    for name, schema in expected.items():
        assert tools[name] == schema, f"Schema drift in tool '{name}'"

def test_required_params_are_enforced(mcp_session):
    """The server must reject calls missing required parameters."""
    result = mcp_session.call_tool("refund_payment", arguments={})
    assert result.is_error, "Server accepted a refund call with no arguments"

def test_tool_errors_are_structured(mcp_session):
    """Errors must come back as structured results, not stack traces --
    the agent will read this text and act on it."""
    result = mcp_session.call_tool(
        "refund_payment", arguments={"payment_id": "nonexistent"}
    )
    assert result.is_error
    assert "Traceback" not in result.content[0].text

Why schema drift matters more than in REST: a human developer reads changelogs; an agent does not. If a tool renames a parameter, the agent will keep calling it the old way, fail, and then improvise -- often by picking a different tool. A silent contract break does not produce a clean error; it produces weird behavior three steps later. Snapshot the tool list in version control and fail CI on any diff.

The same idea applies when you are the tool vendor: teams shipping MCP servers (as Playwright does with Playwright MCP, published to the official MCP Registry on every release) need these contract tests on the server side.