Ambiguous tool-ranking case: documentation query ranks unrelated weather tool above docs tool

[k33wee]

Summary

I found a small reproducible ambiguous tool-ranking case while testing @ratel-ai/sdk locally with a toy tool catalog.
For a documentation/configuration-related query, the expected tool search_docs is retrieved in the top-K, but an unrelated tool, weather_lookup, ranks above it.
This seems useful as a diagnostic case because it looks like a ranking-quality issue rather than a recall issue.

Edited: Updated script with correct reproduction step

Reproduction

import { ToolCatalog } from "@ratel-ai/sdk";

const catalog = new ToolCatalog();

const makeTextTool = (
  id: string,
  description: string,
  handler: (args: Record<string, unknown>) => Promise<Record<string, unknown>>,
) => {
  catalog.register({
    id,
    name: id,
    description,
    inputSchema: {
      properties: {
        query: {
          type: "string",
          description: "Natural language request or search query",
        },
        path: {
          type: "string",
          description: "Optional file path or directory path",
        },
        pattern: {
          type: "string",
          description: "Optional keyword or regular expression pattern",
        },
      },
    },
    outputSchema: {
      properties: {
        summary: {
          type: "string",
          description: "Textual summary of the tool result",
        },
      },
    },
    execute: handler,
  });
};

makeTextTool(
  "read_file",
  "Read a file from local disk and return its textual contents.",
  async ({ path }) => ({
    summary: `Read file at ${String(path ?? "<missing>")}`,
  }),
);

makeTextTool(
  "search_code",
  "Search source code across a repository for keywords, symbols, or configuration values.",
  async ({ pattern }) => ({
    summary: `Searched code for pattern ${String(pattern ?? "<missing>")}`,
  }),
);

makeTextTool(
  "search_docs",
  "Search project documentation, markdown files, and technical docs for relevant explanations.",
  async ({ query }) => ({
    summary: `Searched docs for ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "search_logs",
  "Search application logs and runtime logs for errors, warnings, and diagnostics.",
  async ({ query }) => ({
    summary: `Searched logs for ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "read_env_file",
  "Read environment variable files such as .env, .env.production, or deployment config files.",
  async ({ path }) => ({
    summary: `Read env file ${String(path ?? "<missing>")}`,
  }),
);

makeTextTool(
  "read_config_file",
  "Read structured configuration files like JSON, YAML, TOML, or application config files.",
  async ({ path }) => ({
    summary: `Read config file ${String(path ?? "<missing>")}`,
  }),
);

makeTextTool(
  "send_email",
  "Send an email message to one or more recipients.",
  async ({ query }) => ({
    summary: `Sent email request: ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "send_slack_message",
  "Send a Slack message to a user or channel.",
  async ({ query }) => ({
    summary: `Sent Slack message: ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "currency_convert",
  "Convert an amount of money from one currency to another using exchange rates.",
  async ({ query }) => ({
    summary: `Converted currency for request: ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "weather_lookup",
  "Look up weather forecast for a city or location.",
  async ({ query }) => ({
    summary: `Looked up weather for ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "create_issue",
  "Create a GitHub issue with title, body, labels, and assignee information.",
  async ({ query }) => ({
    summary: `Created issue for ${String(query ?? "<missing>")}`,
  }),
);

makeTextTool(
  "create_pull_request",
  "Create a GitHub pull request with title, description, and branch information.",
  async ({ query }) => ({
    summary: `Created pull request for ${String(query ?? "<missing>")}`,
  }),
);

const scenarios = [
  {
    query: "find where redis host is configured in the repo",
    topK: 5,
    invoke: { toolId: "search_code", args: { pattern: "REDIS_HOST" } },
  },
  {
    query: "read the production environment variables file",
    topK: 5,
    invoke: { toolId: "read_env_file", args: { path: ".env.production" } },
  },
  {
    query: "look for documentation about deployment configuration",
    topK: 5,
    invoke: {
      toolId: "search_docs",
      args: { query: "deployment configuration" },
    },
  },
  {
    query: "message the team that the release is out",
    topK: 5,
    invoke: {
      toolId: "send_slack_message",
      args: { query: "The release is out" },
    },
  },
  {
    query: "open a PR for the release branch",
    topK: 5,
    invoke: {
      toolId: "create_pull_request",
      args: { query: "release branch PR" },
    },
  },
];

for (const scenario of scenarios) {
  const hits = catalog.search(scenario.query, scenario.topK);
  console.log(`\nScenario query: ${scenario.query}`);
  console.log("Top hits:");
  console.log(hits);

  const result = await catalog.invoke(
    scenario.invoke.toolId,
    scenario.invoke.args,
  );
  console.log(`Invoke ${scenario.invoke.toolId}:`);
  console.log(result);
}

Observed output for "look for documentation about deployment configuration" query

[
  { toolId: "weather_lookup", score: 2.1971445083618164 },
  { toolId: "search_docs", score: 2.1485352516174316 },
  { toolId: "read_env_file", score: 2.1367175579071045 },
  { toolId: "search_code", score: 1.6402997970581055 },
  { toolId: "read_config_file", score: 1.6135268211364746 }
]

Expected behavior

Since the query explicitly asks for documentation, I expected search_docs to rank first.

Actual behavior

search_docs is retrieved, but it ranks second behind weather_lookup.

This does not look like a recall failure, because the expected tool is present in the top-K, it looks more like a ranking-quality issue caused by ambiguous lexical overlap. In this case, weather_lookup may be benefiting from overlap around terms like look / lookup, while the semantically better documentation tool remains slightly below it

[rstagi]

Hey-yo @k33wee, thank you for the issue! :)

I tried to reproduce the bug but got a different result:

[
  { toolId: 'search_docs', score: 2.017460823059082 },
  { toolId: 'weather_lookup', score: 1.470530390739441 },
  { toolId: 'read_env_file', score: 1.374184012413025 },
  { toolId: 'read_config_file', score: 1.12444269657135 },
  { toolId: 'search_code', score: 0.8678209185600281 }
]

May you share more details on how you ran this? Library version, node version, operating system? Or maybe you had a larger tool catalog you didn't share in full?

Happy to get deeper into that, as soon as you share more details :)

[k33wee]

Ah, sorry for the confusion. I realize now that I only posted the minimal code snippet for brevity, but the test where I saw the ranking issue used a much longer tool catalog. I left out the longer code in my comment to keep things short, but that have led to the inconsistent results between what I showed and what you reproduced.

I've updated the first comment with the full script in order to keep things clear on this issue.
You should see the result I originally described for this scenario:

{
  query: "look for documentation about deployment configuration",
  topK: 5,
  invoke: {
      toolId: "search_docs",
      args: { query: "deployment configuration" },
  },
}

Tested with:

• Node v24 (using Bun as the runtime)
• Both Linux and macOS

Let me know if you have any trouble reproducing it, and thanks again for looking into this

[rinaldofesta]

The two reproductions in this thread ran different catalogs, so they do not conflict. The minimal snippet was what the May 22 repro attempt ran; the full 12-tool script was edited into the issue body afterwards. The updated script reproduces the reported ranking with identical float scores:

[
  { toolId: "weather_lookup", score: 2.1971445083618164 },
  { toolId: "search_docs", score: 2.1485352516174316 },
  { toolId: "read_env_file", score: 2.1367175579071045 },
  { toolId: "search_code", score: 1.6402997970581055 },
  { toolId: "read_config_file", score: 1.6135268211364746 }
]

Same floats on Bun 1.3.14 and Node 26.3.0, @ratel-ai/sdk 0.1.6, macOS arm64, stable across 10 runs for this scenario.

Root cause, by ablation

Each top-5 tool's full-query score equals its score for a single query term:

query term	hits
look	weather_lookup 2.1971
for	(no hits)
documentation	search_docs 2.1485
about	(no hits)
deployment	read_env_file 2.1367
configuration	search_code 1.6403, read_config_file 1.6135

The ranking is a contest between single terms, and look wins it. Two ablations show the carrier is the description verb, not the look/lookup name overlap:

• Change weather_lookup's description from "Look up weather forecast for a city or location." to "Weather forecast for a city or location.": search_docs ranks first and weather_lookup drops out of the top-5 entirely, name unchanged. So the name token lookup never matched the query term look.
• Keep the catalog, change the query to "find documentation about deployment configuration": same flip.

Both look and documentation occur in exactly one tool each, so document frequency is equal. The gap comes from searchable_text() (src/core/lib/src/indexing.rs) projecting name, description, and schema fields into one flat unweighted text, where weather_lookup's shorter description wins on length normalization. Padding its description to approximately search_docs's length narrows the gap by about half (2.1971 to 2.1745) without flipping it; I did not decompose the remaining delta.

The query's throwaway verb gets the same standing as its intent noun, and the tie is settled by description length. Any short tool description starting with "Look up", "Get", "Find", or "Search" is exposed to the same capture. StackOne documented this failure class for BM25 tool search as common-verb dominance: https://www.stackone.com/blog/mcp-tool-search-bm25-tfidf-hybrid/

A second finding from the same runs

Exact score ties are ordered nondeterministically across process runs. Scenario 2 ("read the production environment variables file"), rank 5: send_email and weather_lookup both score exactly 0.03990466520190239, and the slot splits roughly evenly across repeated runs (5/5 over 10 Bun runs, 4/4 over 8 Node runs in my samples). Top-K membership flickers on ties, which also makes small-catalog repros easy to mismatch. Cause: the bm25 crate collects candidates through a HashSet and sorts by score only, so equal scores keep hash-seed iteration order.

Proposal

The tie-break half is uncontested, so I opened #63: stable ordering (score desc, then tool_id) plus regression tests. Scores and non-tie orderings are unchanged, and the repro above hashes identically across 10 runs against the patched SDK.

The ranking half is a design call, so I would agree the direction before opening anything: BM25F-style field weights, verb handling on the query side, or both. In this case both matched terms live in descriptions, so a plain name-over-description boost does not cover it. #61 touches indexing.rs and registry.rs, so I would sequence any ranking change after it lands.

Third piece: a lexical-collision / near-tie scenario slice in ratel-bench, so this class of case becomes a regression gate, in the spirit of the bench README's rule that features touching retrieval "add a corresponding scenario before being declared done". Happy to target the revamp branch if that is where the harness is heading.

The per-term table above is also the kind of evidence #57 asks ratel explain to surface; if you would rather get the diagnosis surface first, I can align with that instead.

@rstagi does that split sound right?

Ablation script (run against @ratel-ai/sdk 0.1.6)

import { ToolCatalog } from "@ratel-ai/sdk";

const SCHEMA = {
  inputSchema: {
    properties: {
      query: { type: "string", description: "Natural language request or search query" },
      path: { type: "string", description: "Optional file path or directory path" },
      pattern: { type: "string", description: "Optional keyword or regular expression pattern" },
    },
  },
  outputSchema: {
    properties: {
      summary: { type: "string", description: "Textual summary of the tool result" },
    },
  },
};

const TOOLS: Array<[string, string]> = [
  ["read_file", "Read a file from local disk and return its textual contents."],
  ["search_code", "Search source code across a repository for keywords, symbols, or configuration values."],
  ["search_docs", "Search project documentation, markdown files, and technical docs for relevant explanations."],
  ["search_logs", "Search application logs and runtime logs for errors, warnings, and diagnostics."],
  ["read_env_file", "Read environment variable files such as .env, .env.production, or deployment config files."],
  ["read_config_file", "Read structured configuration files like JSON, YAML, TOML, or application config files."],
  ["send_email", "Send an email message to one or more recipients."],
  ["send_slack_message", "Send a Slack message to a user or channel."],
  ["currency_convert", "Convert an amount of money from one currency to another using exchange rates."],
  ["weather_lookup", "Look up weather forecast for a city or location."],
  ["create_issue", "Create a GitHub issue with title, body, labels, and assignee information."],
  ["create_pull_request", "Create a GitHub pull request with title, description, and branch information."],
];

function buildCatalog(overrides: Record<string, string> = {}): ToolCatalog {
  const catalog = new ToolCatalog();
  for (const [id, desc] of TOOLS) {
    catalog.register({
      id,
      name: id,
      description: overrides[id] ?? desc,
      ...SCHEMA,
      execute: async () => ({ summary: id }),
    });
  }
  return catalog;
}

function show(label: string, catalog: ToolCatalog, query: string, topK = 5) {
  const hits = catalog.search(query, topK);
  const line = hits.map((h: any) => `${h.toolId}:${h.score.toFixed(4)}`).join("  ");
  console.log(`${label}\n  query="${query}"\n  ${line || "(no hits)"}\n`);
}

const QUERY = "look for documentation about deployment configuration";

show("[baseline]", buildCatalog(), QUERY);

show(
  '[ablation A: weather_lookup description without "Look up"]',
  buildCatalog({ weather_lookup: "Weather forecast for a city or location." }),
  QUERY,
);

show('[ablation B: query verb "look" -> "find"]', buildCatalog(), "find documentation about deployment configuration");

const base = buildCatalog();
for (const term of ["look", "documentation", "about", "deployment", "configuration"]) {
  show(`[term probe]`, base, term, 3);
}

show(
  "[length probe: weather_lookup description padded]",
  buildCatalog({ weather_lookup: "Look up the current weather forecast conditions for a city or location." }),
  QUERY,
);

Output (condensed; full transcript prints one block per probe):

[baseline]
  weather_lookup:2.1971  search_docs:2.1485  read_env_file:2.1367  search_code:1.6403  read_config_file:1.6135

[ablation A: weather_lookup description without "Look up"]
  search_docs:2.1475  read_env_file:2.1357  search_code:1.6395  read_config_file:1.6127

[ablation B: query verb "look" -> "find"]
  search_docs:2.1485  read_env_file:2.1367  search_code:1.6403  read_config_file:1.6135

[term probe] "look"          -> weather_lookup:2.1971
[term probe] "documentation" -> search_docs:2.1485
[term probe] "about"         -> (no hits)
[term probe] "deployment"    -> read_env_file:2.1367
[term probe] "configuration" -> search_code:1.6403  read_config_file:1.6135

[length probe: weather_lookup description padded]
  weather_lookup:2.1745  search_docs:2.1506  read_env_file:2.1388  search_code:1.6418  read_config_file:1.6152

[k33wee]

@rinaldofesta Thanks, this matches what I observed locally, and I think the ablation gets to the right failure mode.

I agree this is a ranking issue, not a recall issue. In the current implementation, searchable_text() flattens tool name, description, and schema text into one unweighted lexical document, and ToolRegistry::search() runs BM25 directly over that representation.

So in this repro the outcome is largely decided by which individual query term gets the strongest lexical match after tokenization and length normalization:

• look favors weather_lookup
• documentation favors search_docs

That makes this look like common-verb dominance in unweighted BM25 over tool metadata, rather than a missing-tool case.
I also agree that simply boosting name over description would not fully solve this one, since the decisive competing terms here come from descriptions. This seems more like a field-weighting or query-term-handling problem.

One point I would make explicit is that tool descriptions are not user-provided text and should not be rewritten to fit the prompt. They are catalog/server-provided metadata discovered from registration or MCP tools/list, so the retrieval layer should improve ranking over that metadata, not mutate it at query time

[rinaldofesta]

Agreed on treating descriptions as immutable catalog metadata. Both directions on the table respect that: BM25F field weights change how the index scores the text it already has, query-side verb handling touches only the query. Neither rewrites a description.