Methodology
Every test is automated and scripted; the harness is in our public repo so anyone can rerun it.
| Test | Points | How |
|---|---|---|
| Install | 20 | npm: npm install <pkg> in a clean directory, Node 20. PyPI: uv pip install <pkg> into a fresh venv (Python 3.13). Linux, 4-minute timeout. |
| MCP handshake | 30 | Start the package's bin over stdio with no credentials, send initialize (protocol 2025-06-18) then tools/list, 45 s timeout. 30 = tools listed (working); 5 = auth-gated: the only output was a missing-credential message, so it is listed separately and not counted as working; 0 = exited or timed out without a clear reason, or needs a backing service. |
| Maintenance | 25 | GitHub pushed_at (or latest release date when no repo is found): ≤30 days 25, ≤90 days 15, ≤365 days 5. 0 if deprecated on npm, latest PyPI release yanked, or GitHub repo archived. |
| Docs | 15 | Repo README: over 3,000 characters (5), install instructions (5), an mcpServers config example (5). We check the package's own README (the README shipped in the npm tarball, or the PyPI long description), so monorepo packages are judged on their own subdirectory README, not the repo root. |
| License | 10 | OSI-approved license detected 10, otherwise 3. |
Functional task tests (not scored)
For browser, filesystem, memory and fetch servers we also run key-free tasks and record pass/fail per task: write, read and list a temp file; store and retrieve a memory entity; fetch https://example.com/; navigate to https://example.com/ and read the page title. Tasks blocked by our test box (sandbox DNS, no bundled browser download, no backing database, browser extension required) are recorded as not run, not as failures. These results are shown on each listing but do not change the 100-point score.
Small-model friendliness (not scored)
Computed from each server's real tools/list response: tool count, total parameters (including nested object properties), required parameters, maximum nesting depth (1 = flat top-level params), and total/average tool description length in characters. Rating: low if more than 40 tools, nesting depth 4 or more, more than 150 total params, or more than 15,000 description characters; high if at most 15 tools, depth at most 2, at most 60 total params and average description at most 300 characters; otherwise medium. Servers without a tools/list response are unknown. These are heuristics for context-window and schema load on small (≤8B) models, not a measured model result.
What our score does not mean
- It is not a quality or accuracy rating. We never check whether a server's tools return correct results on real tasks.
- It is not a security review. A high score says nothing about permissions, data handling or supply-chain risk.
- It does not cover use with real credentials. Servers that need a key are only checked for whether they ask for it clearly: that is auth-gated (5 of 30 handshake points, not counted as working). If they exit without a clear reason they get 0.
- It is not a performance benchmark. Install and handshake times are recorded as evidence but don't affect points.
- Maintenance points reflect only the GitHub
pushed_atdate (or latest release date) measured in UTC against the test time, plus deprecation, yanked or archived status, not responsiveness to issues or release quality. - Docs points come from three automated checks of the package's own README (length, install instructions, an
mcpServersexample), not from human reading. - It is a snapshot. Each score is valid only for the version and date shown on the listing.
- Payment never changes it. Featured and sponsor placements are labeled and don't affect points or rank.
What we do not test yet
Behavior with real credentials, security review, and performance. Functional tests cover only four categories. Scores therefore measure installability, availability and upkeep, not output quality.
Independence
Payment never changes a score. Featured placement is labeled. Vendors can request a retest.