docs: add agent instructions footer to .md pages, document markdown + search API in llms.txt

[jannikmaierhoefer]

Motivation

Agents reach our Markdown through three paths — the .md URL suffix, Accept: text/markdown negotiation, and the docs MCP getLangfuseDocsPage tool — but a page currently tells them nothing about the rest of our agent surface. An agent that lands on a page which doesn't answer its question has to guess.

Compare docs.reo.dev (GitBook), where every .md response ends with an # Agent Instructions block documenting a ?ask= RAG endpoint. That's how an agent self-discovers the capability without being told. We have the equivalent capabilities — /api/search-docs, /api/mcp, llms.txt, the agent skill, the CLI — but nothing advertises them at the point of use.

Changes

1. Agent instructions footer on every .md page — new lib/agent-instructions-footer.js, appended in scripts/copy_md_sources.js to all 953 generated pages plus the 2 md-override/ pages. It covers:

• the .md / Accept: text/markdown convention, and this page's own Markdown URL
• /api/search-docs with a runnable curl example
• llms.txt and the per-section indexes
• the docs MCP server endpoint and transport
• install the agent skill before instrumenting; read the tracing best-practices page
• verify parameters against api.reference.langfuse.com rather than inferring from examples
• the CLI for programmatic access
• don't hardcode cloud.langfuse.com — read the host from LANGFUSE_BASE_URL

2. llms.txt gains ## Markdown Access and ## Docs Search API sections — it already documented the MCP server and skill, but neither the .md convention nor /api/search-docs.

3. ## REST Endpoint [#rest-endpoint] anchor added to content/docs/docs-mcp.mdx so the new llms.txt deep link resolves.

Notes

• The footer heading is an H2, not an H1: scripts/check-h1-headings.js walks the whole repo including public/md-src after a local build, and a second # heading would fail the check_h1 job.
• Kept deliberately short — it's appended to ~1k pages and costs context on every fetch. The module docstring says to add a line only if it changes what an agent would do.

Verification

• node scripts/copy_md_sources.js → 953 files + 2 overrides, each with exactly one footer
• node scripts/check-h1-headings.js → passes across generated output
• prettier --check → clean on all four files
• new llms.txt block rendered and inspected (the script itself needs public/sitemap-0.xml from a full build)

🤖 Generated with Claude Code

---

Follow-up in this PR: PDF export fix

/api/md-to-pdf renders from public/md-src, so the footer would have shipped inside every human-facing PDF download. app/api/md-to-pdf/route.ts now strips everything from AGENT_FOOTER_MARKER onwards before parsing. Verified: 30,880 → 28,544 bytes on docs/observability/get-started.md, original ending restored, and markdown without the marker passes through untouched.

The PDF route is the only human-facing md-src consumer — the other paths (.md rewrite, Accept: text/markdown negotiation, MCP getLangfuseDocsPage) are all agent-facing and should keep the footer.

Greptile Summary

Adds agent-discovery guidance to generated Markdown documentation.

• Appends a shared agent-instructions footer to generated and overridden Markdown pages.
• Documents Markdown access and the documentation search API in llms.txt.
• Adds an explicit REST endpoint anchor to the Docs MCP page.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking documentation correction needed for the homepage Markdown URL convention.

The generated footer works for ordinary and overridden page paths, but its universal suffix instruction directs homepage consumers to /.md even though the generated root artifact is /md-src/index.md.

Files Needing Attention: lib/agent-instructions-footer.js

Prompt To Fix All With AI

### Issue 1
lib/agent-instructions-footer.js:38-40
**Homepage suffix guidance is inaccurate**

The footer says every page supports appending `.md`, but applying that convention to `https://langfuse.com/` produces `/.md`, while the generated homepage Markdown is stored as `/md-src/index.md` and the root-specific rewrite only handles `Accept: text/markdown`. This directs agents following the documented convention to a nonexistent Markdown path.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "docs: add agent instructions footer to ...." | Re-trigger Greptile}

Context used:

• Knowledge Base — Content and MDX Build Pipeline
• Knowledge Base — Build Pipeline

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
langfuse-docs	Ready	Preview, Comment	Jul 29, 2026 12:21pm

[github-actions]

@claude review

[claude]

🔴 appendAgentInstructionsFooter() in scripts/copy_md_sources.js runs unconditionally on every generated page, so it also gets appended to public/md-src/privacy.md and cookie-policy.md, and app/api/md-to-pdf/route.ts renders that markdown straight to PDF without stripping the footer. That means the 'download as PDF' link on the Privacy Policy and Cookie Policy pages now ships end users a legal document that ends with a curl/MCP/GitHub-issue 'Agent Instructions' section. Fix by excluding marketing/legal pages from the footer in copy_md_sources.js, or stripping the AGENT_FOOTER_MARKER block in md-to-pdf/route.ts before rendering.

Extended reasoning...

This PR adds appendAgentInstructionsFooter() and calls it unconditionally inside copyAll() (scripts/copy_md_sources.js:198) for every file written to public/md-src/**, with no exclusion by content type or section. lib/content-dir-map.js maps the marketing content directory to the empty URL prefix (marketing: ""), so content/marketing/privacy.mdx and cookie-policy.mdx are written out as public/md-src/privacy.md and public/md-src/cookie-policy.md — both now carrying the new ## Agent Instructions footer (curl examples for /api/search-docs, MCP server endpoint info, agent-skill install instructions, and a "Found an error in these docs? open an issue on GitHub" line).

Both privacy.mdx and cookie-policy.mdx contain a human-facing "download as PDF" link (<a href="/api/md-to-pdf?url=/privacy.md">, similarly for cookie-policy). app/api/md-to-pdf/route.ts rewrites that URL to /md-src/<path>.md, fetches the file verbatim, strips only the YAML frontmatter block, runs stripMdxForPlainMarkdown and removeAnchorTags, and renders the remainder to HTML/PDF via marked. None of those steps recognize or strip the AGENT_FOOTER_MARKER/## Agent Instructions block, so it renders straight into the generated PDF.

Proof, step by step:

1. scripts/copy_md_sources.js copies content/marketing/privacy.mdx → public/md-src/privacy.md, then unconditionally calls appendAgentInstructionsFooter(processed, "privacy.md"), appending the full footer block after a --- separator.
2. A site visitor on /privacy clicks "download as PDF", which hits /api/md-to-pdf?url=/privacy.md.
3. The route fetches public/md-src/privacy.md verbatim, strips frontmatter only, and converts the rest (including the appended footer) to PDF.
4. The resulting PDF — the one a user or a legal/compliance reviewer downloads as "the Privacy Policy" — ends with a section containing curl -sG "https://langfuse.com/api/search-docs" ..., MCP server connection details, a skill-install instruction, and a GitHub issue link.

Nothing crashes and the PDF still generates correctly, but the content is objectively wrong for the document type: a Privacy Policy / Cookie Policy is a legal artifact, and shipping AI-agent tooling instructions inside it is inappropriate and would look like a bug (or a copy-paste error) to any reader who scrolls to the end, including legal/compliance reviewers relying on that PDF as the canonical policy text. This is a fresh regression introduced by this PR — before it, public/md-src/privacy.md (and the PDF derived from it) was clean policy text.

The fix is small: either skip the footer for non-agent-relevant sections (e.g., only apply it under docs/integrations/self-hosting, matching the PR's own stated intent of making "the docs" self-describing for agents — the Privacy Policy is not part of that surface), or strip the AGENT_FOOTER_MARKER-delimited block in app/api/md-to-pdf/route.ts before rendering to PDF, alongside the existing frontmatter-stripping step.