yuragi/index.html at main · hinanohart/yuragi · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>yuragi — LLM Confidence Fragility Analyzer</title>
<meta name="description" content="Measure how unstable your LLM's confidence really is. Perturbation-driven hallucination detection with workshop-grade real benchmarks.">
<style>
  body{font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',sans-serif;max-width:760px;margin:4rem auto;padding:0 1.5rem;line-height:1.7;color:#24292f}
  h1{font-size:2rem;margin-bottom:0.5rem}
  .sub{color:#57606a;margin-bottom:2rem}
  .grid{display:grid;gap:1rem;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));margin-bottom:2rem}
  .grid a{display:block;padding:1rem;border:1px solid #d0d7de;border-radius:6px;text-decoration:none;color:#0969da;background:#f6f8fa;transition:all .15s}
  .grid a:hover{background:#ddf4ff;border-color:#0969da}
  .grid a strong{display:block;font-size:1.05rem;margin-bottom:0.25rem}
  .grid a span{font-size:0.85rem;color:#57606a}
  .results{border-left:3px solid #1f883d;padding:0.75rem 1rem;background:#f6f8fa;margin:1.5rem 0;font-size:0.95rem}
  code{background:#eaeef2;padding:0.1em 0.4em;border-radius:3px;font-size:0.9em}
  .install{background:#0d1117;color:#c9d1d9;padding:0.75rem 1rem;border-radius:6px;font-family:ui-monospace,SFMono-Regular,monospace}
  footer{margin-top:3rem;color:#8b949e;font-size:0.85rem;text-align:center}
</style>
</head>
<body>

<h1>yuragi</h1>
<p class="sub">Measure how unstable your LLM's confidence really is.<br>
Perturbation-driven hallucination detection — black-box, logprob-friendly, CLI-first.</p>

<div class="grid">
<a href="https://github.com/hinanohart/yuragi"><strong>GitHub →</strong><span>Source, issues, contributing</span></a>
<a href="https://pypi.org/project/yuragi/"><strong>PyPI →</strong><span><code>pip install yuragi</code></span></a>
<a href="https://github.com/hinanohart/yuragi/releases/latest"><strong>Releases →</strong><span>v0.4.1 — Apr 14, 2026</span></a>
<a href="https://github.com/hinanohart/yuragi/blob/main/README.md"><strong>README →</strong><span>Usage guide</span></a>
<a href="https://github.com/hinanohart/yuragi/blob/main/docs/theory.md"><strong>Theory →</strong><span>Mathematical foundation</span></a>
<a href="https://github.com/hinanohart/yuragi/blob/main/paper/revolutionary_reframe.md"><strong>Paper draft →</strong><span>Confidence inversion on 8B models</span></a>
</div>

<div class="install">pip install yuragi</div>

<div class="results">
<strong>Latest research (v0.4.1)</strong> — Real-data benchmarks on llama-3.1-8B:
<ul style="margin:0.5em 0 0 0">
  <li><strong>TruthfulQA n=412</strong>: ensemble AUC <strong>0.73</strong> [0.68, 0.78] &mdash; driven primarily by <code>baseline_confidence</code>; perturbation features add no statistically significant &Delta; (p=0.35)</li>
  <li><strong>TriviaQA n=200 (pilot)</strong>: confidence-inversion AUC <strong>0.75 [0.67, 0.82]</strong> &mdash; bootstrap CI width ~&plusmn;0.10; pending n&ge;400 cross-family replication</li>
  <li>Domain-boundary finding: works on single-path factoids, fails on imitative falsehoods (solo <code>fragility_score</code> AUC &asymp; 0.50 across 6 datasets)</li>
</ul>
</div>

<p>yuragi generates 13 semantic-preserving prompt perturbations (typos, tone, paraphrase, authority framing, counterfactual context) and compares the model's confidence distribution across responses. When answer text stays the same but confidence moves, that's fragility — a measurable property of prompt wording rather than model knowledge.</p>

<footer>
Apache License 2.0 · Author: <a href="https://github.com/hinanohart">hinanohart</a> · Documentation auto-generated from repository
</footer>
</body>
</html>