Rat 281/dense search

[bercaakbayir]

Summary

Replaces lexical (BM25) tool/skill retrieval with dense semantic retrieval and
ships it as ratel-ai-core v0.3.0-rc.1. .search() now embeds query and
catalog text into a shared vector space and cosine-ranks — closing the "missing
gold" recall gap where a query and the right tool describe the same concept with
different words ("remove a file" vs "delete a path"). Core stays in-process, no
infra. See ADR-0013.

Per the experiment-line convention, the version is the experiment: .search()
is dense unconditionally (no feature flag), and the BM25 baseline remains available
as v0.2.0. The benchmark compares engines by swapping the crate version alone.

What's in it

Retrieval engine

• Dense retrieval via Candle (pure-Rust, no
  C++/ONNX dep) with BAAI/bge-small-en-v1.5 (BERT-family, 384-dim, CLS pooling,
  L2-normalized; asymmetric query prefix). Cosine = dot product over precomputed
  vectors.
• Applied to both ToolRegistry and SkillRegistry; vectors precomputed at
  register(), ranked brute-force in-memory (no vector DB). Emits the existing
  trace schema with a "dense" stage — no schema break.
• Weights downloaded on first use via hf-hub at a pinned revision into the
  shared HF cache (offline/deterministic after the first fetch); loaded once per
  process. BM25 search.rs/tool_search.rs removed from this line.

Embedder resilience (fail loud, never abort)

• Load & inference are fallible: a typed EmbedderError (with a remediation
  hint) surfaces as a catchable Error/ValueError in the SDKs instead of an
  .expect() panic that aborted the whole host process across the FFI.
• Non-poisoning loader — a failed load isn't cached, so a transient network
  blip retries on the next call. HF_HOME/HF_ENDPOINT honored.
• Underpowered-machine flag: additive TraceEvent::EmbedderLoad { status: ok|slow|failed, took_ms, reason }; a slow cold load (RATEL_EMBED_SLOW_MS,
  default 5s) is flagged and logged. OOM is an uncatchable OS kill (documented).

Release / packaging (v0.3.0-rc.1)

• Version bumps across crate, wheel, npm loader + per-platform subpackages and
  manifests; npm optionalDependencies integrity; native arm64 release runner;
  fancy-regex. Per-package CHANGELOGs updated.

CI hardening

• Fixed a concurrent cold-cache download race (parallel vitest workers) —
  hf-hub's non-blocking flock giv0 MB weights are
  still downloading; added retry-with-backoff on lock contention + a vitest
  testTimeout for the cold first-
• Fixed the cross-SDK E2E skill→tool cross-pollination case for dense (the
  score-0 assertion was BM25-specifthe declared tool
  so it can only enter via the skill).

Behavior changes

• .search() semantics differ by v.2.0) — intentional
  for the experiment line; pin deliberately.
• register/search now **throw ar failure where
  they previously aborted the process. SDK method signatures are unchanged.
• First run needs network + a one-t

Testing

• Rust: fmt / clippy -D warnings / test --workspace (incl. dense ranking,
  error classification, non-poisoniision).
• TS: build / typecheck / lint / test; Python: ruff / mypy / pytest
  (both add cold-cache resilience tror, not a crash).
• Cross-SDK E2E (Python + TS) green, including a dense cross-pollination scenario.

Out of scope (follow-ups)

User-supplied/external embedding futs/selection, and a
dimension-consistency guard — deferred.