feat: @byok-relay/llamaindex — LlamaIndex.TS custom LLM, embeddings, tool calling for BYOK AI (Growth Day 44) - #80
feat: @byok-relay/llamaindex — LlamaIndex.TS custom LLM, embeddings, tool calling for BYOK AI (Growth Day 44)#80alokit-bot wants to merge 2 commits into
Conversation
…ngs, tool calling for BYOK AI (Growth Day 43)
…tool calling for BYOK AI (Growth Day 44)
📝 WalkthroughWalkthroughAdded new LangChain.js and LlamaIndex.TS packages with relay-backed chat, streaming, tool-calling, embedding, token, and key-management APIs. Added package smoke tests, integration documentation, package metadata, and daily metrics records. ChangesFramework integrations
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant LangChainApplication
participant ByokRelayChatModel
participant BYOKRelay
LangChainApplication->>ByokRelayChatModel: invoke or stream messages
ByokRelayChatModel->>BYOKRelay: send relay request
BYOKRelay-->>ByokRelayChatModel: completion response or stream
ByokRelayChatModel-->>LangChainApplication: return LangChain result
sequenceDiagram
participant LlamaIndexApplication
participant ByokRelayLLM
participant BYOKRelay
LlamaIndexApplication->>ByokRelayLLM: chat or stream messages
ByokRelayLLM->>BYOKRelay: send converted relay request
BYOKRelay-->>ByokRelayLLM: completion response or stream
ByokRelayLLM-->>LlamaIndexApplication: return ChatResponse or deltas
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Checkov (3.3.8)packages/langchain/package.jsonTraceback (most recent call last): packages/llamaindex/package.jsonTraceback (most recent call last): Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (3)
packages/langchain/package.json (1)
13-15: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider declaring an
enginesconstraint.The runtime relies on native
fetch/ReadableStreamand the JSDoc states "Node.js 18+". Adding anenginesfield surfaces that requirement to consumers at install time.♻️ Suggested addition
"scripts": { "test": "node test/langchain.test.js" }, + "engines": { + "node": ">=18" + },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/langchain/package.json` around lines 13 - 15, Add an engines constraint to packages/langchain/package.json declaring Node.js 18 or newer, alongside the existing scripts configuration, to match the package’s native fetch/ReadableStream runtime requirement.packages/langchain/src/index.js (2)
432-477: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winHarden the shared SSE reader loop in both streaming paths. Both
_streamandstreamChatduplicate the same manual SSE parser whose reader/buffer handling has minor robustness gaps: no guard whenres.bodyisnull, the final non-newline-terminated event inbufferis discarded afterdone, anddecoder.decode()is never flushed. Consider extracting a shared async-generator helper so both paths get the same fix.
packages/langchain/src/index.js#L432-L477: guardres.bodybeforegetReader(), and after the loop process any residualbuffer(and flush the decoder) instead of relying solely on the[DONE]sentinel.packages/langchain/src/index.js#L675-L696: apply the same reader/buffer hardening and add the missingdeltaexistence guard (as in_stream) before readingdelta.contenton Line 691.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/langchain/src/index.js` around lines 432 - 477, Harden the duplicated SSE parsing in packages/langchain/src/index.js lines 432-477 and 675-696, preferably via a shared async-generator helper: validate res.body before calling getReader(), flush decoder.decode() after reading completes, and process any residual non-newline-terminated buffer event. In the streamChat site at lines 675-696, also guard the parsed choice delta before accessing delta.content, matching _stream’s existing protection.
204-223: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConvert LangChain
StructuredToolschemas before using them as function parameters.
_convertToolsassignstool.schemadirectly tofunction.parameters, but LangChainStructuredToolinstances expose a Zod schema. Passing that object as the relay schema will not yield a valid OpenAI-compatible JSON Schema; usezodToJsonSchemaor LangChain’sconvertToOpenAIToolpath for Zod schemas, while still accepting raw JSON-Schema objects where already provided.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/langchain/src/index.js` around lines 204 - 223, Update _convertTools to convert LangChain StructuredTool Zod schemas into OpenAI-compatible JSON Schema before assigning function.parameters, using zodToJsonSchema or LangChain’s convertToOpenAITool path. Preserve raw JSON-Schema objects when tool.schema, tool.parameters, or tool.inputSchema is already JSON Schema, and retain the existing fallback for tools without schemas.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/langchain/README.md`:
- Around line 3-5: Update the introductory README description to state that
users can use provider keys without their application server storing them, while
preserving the existing clarification that the relay stores keys encrypted at
rest.
- Around line 212-215: Update the ByokRelayChatModel API table entries for
_generate and _stream to describe them as internal hooks used by LangChain’s
chat-model/runnable invocation machinery, replacing the inaccurate BaseChain
reference while leaving invoke and stream descriptions unchanged.
- Around line 74-77: Register the provider key before each standalone relay
invocation: in packages/langchain/README.md at lines 74-77, 141-150, and
172-187, call model.storeKey('openai', 'sk-...') before the tool-calling, LCEL,
and ReAct examples; in packages/llamaindex/README.md at lines 55-63, call await
llm.storeKey('openai', 'sk-...') before engine.query(...), or explicitly
configure llm with shared storage.
In `@packages/llamaindex/README.md`:
- Around line 133-140: Update the README quick-start withTools example to
declare calendarTool before it is passed to llm.withTools, using the existing
tool-definition pattern, or remove calendarTool from the tools array and keep
the example valid when copied.
- Around line 22-23: Update the README example comment near llm.storeKey to
avoid stating that the API key is persisted in localStorage. Explain that
storeKey sends the key to the relay, while localStorage caches the relay token,
and qualify the storage behavior for Node/edge environments where storage is in
memory.
- Around line 303-305: Update the license section in the package README and the
package metadata to remove the inconsistent MIT claim, aligning
`@byok-relay/llamaindex` with the repository’s Apache-2.0 licensing and existing
root documentation.
In `@packages/llamaindex/src/index.js`:
- Around line 242-260: Scope the persisted relay token by app ID across the
ByokRelayClient lifecycle. Add or reuse a _tokenKey derived from this.appId,
then replace the fixed 'byok_relay_token' key in ensureToken, register, logout,
and deleteAccount with this._tokenKey, preserving existing token loading,
saving, and deletion behavior.
- Around line 142-149: Update the image_url branch in the content mapping to
normalize both LlamaIndex image URL shapes: use the string directly when
image_url is already a string, and extract its url property when it is an
object. Ensure the returned image_url contains a single string url rather than a
nested object.
---
Nitpick comments:
In `@packages/langchain/package.json`:
- Around line 13-15: Add an engines constraint to
packages/langchain/package.json declaring Node.js 18 or newer, alongside the
existing scripts configuration, to match the package’s native
fetch/ReadableStream runtime requirement.
In `@packages/langchain/src/index.js`:
- Around line 432-477: Harden the duplicated SSE parsing in
packages/langchain/src/index.js lines 432-477 and 675-696, preferably via a
shared async-generator helper: validate res.body before calling getReader(),
flush decoder.decode() after reading completes, and process any residual
non-newline-terminated buffer event. In the streamChat site at lines 675-696,
also guard the parsed choice delta before accessing delta.content, matching
_stream’s existing protection.
- Around line 204-223: Update _convertTools to convert LangChain StructuredTool
Zod schemas into OpenAI-compatible JSON Schema before assigning
function.parameters, using zodToJsonSchema or LangChain’s convertToOpenAITool
path. Preserve raw JSON-Schema objects when tool.schema, tool.parameters, or
tool.inputSchema is already JSON Schema, and retain the existing fallback for
tools without schemas.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 93b61e02-5201-4ed8-aef5-884ffe9bbbd6
📒 Files selected for processing (11)
README.mdllms.txtmetrics/daily.jsonlpackages/langchain/README.mdpackages/langchain/package.jsonpackages/langchain/src/index.jspackages/langchain/test/langchain.test.jspackages/llamaindex/README.mdpackages/llamaindex/package.jsonpackages/llamaindex/src/index.jspackages/llamaindex/test/llamaindex.test.js
| LangChain.js custom chat model and embeddings adapter for [byok-relay](https://github.com/avikalpg/byok-relay). | ||
|
|
||
| Use **user-supplied AI provider keys** in your LangChain chains, agents, and RAG pipelines — without storing keys server-side. Keys are encrypted at rest in the relay; your server only sees a relay token. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Clarify the key-storage threat model.
“Without storing keys server-side” is incorrect as written: the relay stores provider keys server-side, encrypted at rest. Change this to “without your application server storing keys” so users understand the relay’s role.
🧰 Tools
🪛 LanguageTool
[grammar] ~3-~3: Ensure spelling is correct
Context: ...l and embeddings adapter for byok-relay. Use user-supplied AI provider keys i...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/langchain/README.md` around lines 3 - 5, Update the introductory
README description to state that users can use provider keys without their
application server storing them, while preserving the existing clarification
that the relay stores keys encrypted at rest.
| const model = new ByokRelayChatModel({ modelName: 'openai/gpt-4o' }); | ||
| const modelWithTools = model.bindTools([weatherTool]); | ||
|
|
||
| const result = await modelWithTools.invoke([new HumanMessage('Weather in Tokyo?')]); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg '(^|/)(README\.md|.*\.ts|.*\.tsx|.*\.js)$' | sed -n '1,220p'
echo "== README relevant snippets =="
for f in packages/langchain/README.md packages/llamaindex/README.md; do
if [ -f "$f" ]; then
echo "--- $f line count ---"
wc -l "$f"
echo "--- $f ranges ---"
sed -n '1,230p' "$f" | nl -ba | sed -n '45,210p'
else
echo "missing $f"
fi
done
echo "== search adapter key APIs/usages =="
rg -n "storeKey|registerKey|set.*Key|key" packages -g '*.ts' -g '*.tsx' -g '*.md' -g '*.js' | sed -n '1,240p'
echo "== relevant symbols =="
rg -n "ByokRelay(Chat|Embedding|LLM)|class ByokRelay|storeKey|registerKey" packages -g '*.ts' -g '*.tsx' -g '*.js' | sed -n '1,240p'Repository: avikalpg/byok-relay
Length of output: 724
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package README snippets =="
for f in packages/langchain/README.md packages/llamaindex/README.md; do
if [ -f "$f" ]; then
echo "--- $f line count ---"
wc -l "$f"
echo "--- $f relevant ranges ---"
awk '{printf "%4d %s\n", NR, $0}' "$f" | sed -n '55,205p'
else
echo "missing $f"
fi
done
echo "== package source snippets =="
for f in packages/langchain/src/index.js packages/llamaindex/src/index.js; do
if [ -f "$f" ]; then
echo "--- $f line count ---"
wc -l "$f"
echo "--- $f outline-ish summary ---"
awk '{printf "%4d %s\n", NR, $0}' "$f" | sed -n '1,260p'
else
echo "missing $f"
fi
done
echo "== key-related searches =="
rg -n "storeKey|registerKey|set.*Key|key|env\.process|process\.env|secret|apiKey|model:" packages src -g '*.ts' -g '*.tsx' -g '*.md' -g '*.js' | sed -n '1,300p'Repository: avikalpg/byok-relay
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== langchain storeKey/client implementation around token/key forwarding =="
sed -n '255,380p' packages/langchain/src/index.js | awk '{printf "%4d %s\n", 254+NR, $0}'
echo "== llamaindex storeKey/client implementation around token/key forwarding =="
sed -n '240,320p' packages/llamaindex/src/index.js | awk '{printf "%4d %s\n", 239+NR, $0}'
sed -n '330,420p' packages/llamaindex/src/index.js | awk '{printf "%4d %s\n", 329+NR, $0}'
echo "== deterministic verifier for example key availability in README snippets =="
python3 - <<'PY'
from pathlib import Path
import re
files = {
'packages/langchain/README.md': list(range(48,94)) + list(range(136,153)) + list(range(172,189)),
'packages/llamaindex/README.md': list(range(50,66)),
}
for path, ranges in files.items():
text = Path(path).read_text()
lines = text.splitlines()
snippet = '\n'.join(lines[i-1] for i in ranges if i-1 < len(lines))
model_names = re.findall(r'new ByokRelay(Link|Embedding|LLM|ChatModel)\([^)]*?', snippet)
print(f'--- {path} --- {list(snippet.count("storeKey") for _ in [1])} storeKey calls -> {snippet.count("storeKey(")}')
before_store = [i for i, line in enumerate(range(1, len(lines)+1), 1) if i in ranges and 'storeKey' not in line]\
if ranges else []
print('stores_in_ranges:', snippet.count('storeKey('))
PYRepository: avikalpg/byok-relay
Length of output: 13522
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== packages/llamaindex/src/index.js line 520-565 =="
sed -n '520,565p' packages/llamaindex/src/index.js | awk '{printf "%4d %s\n", 519+NR, $0}'
echo "== packages/langchain/src/index.js storage/client forwarding =="
grep -n "_ensureToken|_k.get|_storage.*get|_relayUrl|relayUrl|api_key|provider.*model" packages/langchain/src/index.js packages/llamaindex/src/index.js | sed -n '1,140p'Repository: avikalpg/byok-relay
Length of output: 1922
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== packages/langchain/src/index.js _Client and forwarding =="
rg -n "class ByokRelayClient|_Client|_ensureToken|_k\\.|_storage|relay/" packages/langchain/src/index.js -C 3 | sed -n '1,220p'
echo "== packages/llamaindex/src/index.js _Client and forwarding =="
rg -n "class ByokRelayClient|ByokRelayClient|_client|relay/" packages/llamaindex/src/index.js -C 2 | sed -n '1,220p'
echo "== deterministic snippet check =="
python3 - <<'PY'
from pathlib import Path
for path, ranges in [
("packages/langchain/README.md", (list(range(58,87)) + list(range(136,151)) + list(range(172,188)))),
("packages/llamaindex/README.md", list(range(55,64))),
]:
text = Path(path).read_text().splitlines()
snippet = "\n".join(text[max(0,i-1):i] for i in ranges)
print(f"{path}: storeKey count in snippet={snippet.count('storeKey')}; ranges={ranges}")
PYRepository: avikalpg/byok-relay
Length of output: 10504
Ensure the standalone provider calls include a registered key.
These snippets create adapters that read their own stored keys, so each standalone example needs storeKey(...) before invoking the relay request:
packages/langchain/README.mdaround lines 74, 141, and 172: callmodel.storeKey('openai', 'sk-...')before the tool-calling, LCEL, and ReAct examples.packages/llamaindex/README.mdaround lines 61: thisllmis not a copy ofembed, so addawait llm.storeKey('openai', 'sk-...');beforeengine.query(...)or explicitly pass shared storage tollm.
📍 Affects 2 files
packages/langchain/README.md#L74-L77(this comment)packages/langchain/README.md#L141-L150packages/langchain/README.md#L172-L187packages/llamaindex/README.md#L55-L63
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/langchain/README.md` around lines 74 - 77, Register the provider key
before each standalone relay invocation: in packages/langchain/README.md at
lines 74-77, 141-150, and 172-187, call model.storeKey('openai', 'sk-...')
before the tool-calling, LCEL, and ReAct examples; in
packages/llamaindex/README.md at lines 55-63, call await llm.storeKey('openai',
'sk-...') before engine.query(...), or explicitly configure llm with shared
storage.
| | `invoke(messages, opts?)` | Non-streaming completion (LangChain standard) | | ||
| | `stream(messages, opts?)` | Streaming completion (LangChain standard) | | ||
| | `_generate(messages, opts?)` | Internal; called by LangChain BaseChain | | ||
| | `_stream(messages, opts?)` | Internal; called by LangChain streaming | |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Files"
git ls-files | rg '(^|/)packages/langchain/README\.md$|(^|/)packages/langchain/' | sed -n '1,120p'
echo
echo "## README context"
if [ -f packages/langchain/README.md ]; then
nl -ba packages/langchain/README.md | sed -n '160,230p'
fi
echo
echo "## Search for ByokRelayChatModel and BaseChatModel/BaseChain"
rg -n "ByokRelayChatModel|BaseChatModel|BaseChain|extends BaseChain|extends BaseChatModel|_generate|_stream" packages/langchain -SRepository: avikalpg/byok-relay
Length of output: 361
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## README context"
if [ -f packages/langchain/README.md ]; then
sed -n '160,235p' packages/langchain/README.md | cat -n | sed 's/^/README: /'
fi
echo
echo "## Search for ByokRelayChatModel and relevant symbols"
rg -n "ByokRelayChatModel|BaseChatModel|BaseChain|extends BaseChain|extends BaseChatModel|_generate|_stream|invoke\\(|stream\\(" packages/langchain -S
echo
echo "## Source outline/implementation excerpt"
if [ -f packages/langchain/src/index.js ]; then
wc -l packages/langchain/src/index.js
sed -n '1,260p' packages/langchain/src/index.js | cat -n | sed 's/^/src: /'
fiRepository: avikalpg/byok-relay
Length of output: 25569
Correct the internal API reference.
ByokRelayChatModel extends BaseChatModel; update _generate and _stream from “called by LangChain BaseChain” to hooks used by LangChain’s chat-model/runnable invocation machinery.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/langchain/README.md` around lines 212 - 215, Update the
ByokRelayChatModel API table entries for _generate and _stream to describe them
as internal hooks used by LangChain’s chat-model/runnable invocation machinery,
replacing the inaccurate BaseChain reference while leaving invoke and stream
descriptions unchanged.
| // Store the user's API key (once — persisted in localStorage) | ||
| await llm.storeKey('openai', 'sk-...'); |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not claim the API key is persisted in localStorage.
storeKey sends the API key to the relay; localStorage caches the relay token. Also qualify this for Node/edge in-memory storage.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llamaindex/README.md` around lines 22 - 23, Update the README
example comment near llm.storeKey to avoid stating that the API key is persisted
in localStorage. Explain that storeKey sends the key to the relay, while
localStorage caches the relay token, and qualify the storage behavior for
Node/edge environments where storage is in memory.
| ## Quick start — withTools (bound tools) | ||
|
|
||
| ```js | ||
| const llmWithTools = llm.withTools([weatherTool, calendarTool]); | ||
|
|
||
| // Tools are forwarded automatically on every call | ||
| const response = await llmWithTools.chat({ | ||
| messages: [{ role: 'user', content: 'Am I free tomorrow?' }], |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Define calendarTool or remove it from the example.
calendarTool is never declared in this README, so copying this snippet causes a ReferenceError.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llamaindex/README.md` around lines 133 - 140, Update the README
quick-start withTools example to declare calendarTool before it is passed to
llm.withTools, using the existing tool-definition pattern, or remove
calendarTool from the tools array and keep the example valid when copied.
| ## License | ||
|
|
||
| MIT — [byokrelay.com](https://byokrelay.com) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Tracked README/license/metadata candidates:"
git ls-files | rg '(^|/)(README\.md|LICENSE|LICEN[CS]E|package\.json|pyproject.toml|setup.py|setup.cfg|Cargo.toml)$' || true
echo
echo "Root README relevant lines:"
if [ -f README.md ]; then
rg -n -i -C 3 'license|apache|mit' README.md || true
fi
if [ -f packages/llamaindex/README.md ]; then
echo
echo "packages/llamaindex/README.md relevant lines:"
sed -n '285,315p' packages/llamaindex/README.md
fi
echo
echo "All occurrences in tracked files:"
rg -n -i 'license|apache|mit|byokrelay.com' . --glob '!node_modules' --glob '!dist' --glob '!build' || trueRepository: avikalpg/byok-relay
Length of output: 17835
Keep @byok-relay/llamaindex license metadata in-sync with the Apache-2.0 package.
This README says MIT, while packages/llamaindex/package.json is also MIT, but root metadata/license docs are Apache 2.0 (LICENSE, root README, root package.json, llms.txt). Remove the per-package MIT claim unless this package intentionally needs different licensing, and align package metadata accordingly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llamaindex/README.md` around lines 303 - 305, Update the license
section in the package README and the package metadata to remove the
inconsistent MIT claim, aligning `@byok-relay/llamaindex` with the repository’s
Apache-2.0 licensing and existing root documentation.
| if (Array.isArray(content)) { | ||
| const parts = content.map(part => { | ||
| if (part.type === 'text') return { type: 'text', text: part.text }; | ||
| if (part.type === 'image_url') return { type: 'image_url', image_url: { url: part.imageUrl || part.image_url } }; | ||
| return { type: 'text', text: String(part) }; | ||
| }); | ||
| return { role, content: parts }; | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
LlamaIndex.TS MessageContentDetail image_url type ChatMessage content format
💡 Result:
In LlamaIndex.TS, a ChatMessage follows a structure where the content field can be either a simple string or an array of MessageContentDetail objects [1][2]. For multi-modal interactions, the content is typically an array that includes objects representing different media types [2][3]. The MessageContentImageDetail type is used to provide image inputs and is structured as follows [1][4]: { "type": "image_url", "image_url": { "url": "string" }, "detail"?: "high" | "low" | "auto" } The image_url property must contain an object with a url field, which accepts either a standard HTTP/HTTPS URL or a base64-encoded data URL (e.g., data:image/jpeg;base64,...) [2][3]. Note regarding the detail property: While LlamaIndex.TS defines the detail field as a sibling of the image_url object [1][4], some underlying LLM providers (like OpenAI) expect the detail parameter to be nested inside the image_url object [5]. Users have reported that mismatches between this LlamaIndex.TS structure and specific provider requirements can lead to validation errors [5]. Example of a multi-modal ChatMessage content: { "role": "user", "content": [ { "type": "text", "text": "What is in this image?" }, { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,..." } } ] }
Citations:
- 1: https://github.com/run-llama/LlamaIndexTS/blob/09ba5aa43a21e8f888ab02f6f9a536fe2762ef79/packages/core/src/llms/type.ts
- 2: https://run-llama-llamaindexts.mintlify.app/api/core/llms
- 3: https://run-llama-llamaindexts.mintlify.app/examples/multimodal
- 4: https://github.com/run-llama/LlamaIndexTS/blob/acd9b66de4ec1305d8f7e87832cc68093116bb16/packages/core/src/llms/type.ts
- 5: add detail to image_url for openai chat completions run-llama/LlamaIndexTS#2221
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== package references =="
rg -n '"llamaindex"|`@llamaindex/core`|llamaindex' package.json packages/llamaindex/package.json pnpm-lock.yaml yarn.lock package-lock.json 2>/dev/null || true
echo
echo "== target file excerpt =="
nl -ba packages/llamaindex/src/index.js | sed -n '120,160p'
echo
echo "== package declaration excerpt =="
if [ -f packages/llamaindex/package.json ]; then
nl -ba packages/llamaindex/package.json | sed -n '1,120p'
fi
echo
echo "== lockfile llamaindex exact entries =="
rg -n '"llamaindex"|llamaindex@|`@llamaindex/core`' pnpm-lock.yaml package-lock.json yarn.lock 2>/dev/null | head -80 || trueRepository: avikalpg/byok-relay
Length of output: 704
🏁 Script executed:
#!/bin/bash
set -euo pipefail
export LC_ALL=C
echo "== target file excerpt =="
awk '{printf "%6d\t%s\n", NR, $0}' packages/llamaindex/src/index.js | sed -n '120,160p'
echo
echo "== package declaration excerpt =="
awk '{printf "%6d\t%s\n", NR, $0}' packages/llamaindex/package.json | sed -n '1,120p'
echo
echo "== lockfile llamaindex exact entries (if present) =="
for f in pnpm-lock.yaml package-lock.json yarn.lock; do
if [ -f "$f" ]; then
echo "-- $f --"
rg -n '"llamaindex"|llamaindex@|`@llamaindex/core`' "$f" | head -120 || true
fi
done
echo
echo "== dependency shape probe =="
python3 - <<'PY'
from typing import Any
def current_normalize(part: dict[str, Any]) -> dict[str, Any]:
if part.get("type") == "image_url":
return {"type": "image_url", "image_url": {"url": part.get("imageUrl") or part.get("image_url")}}
return {"type": "text", "text": str(part)}
samples = [
{"type": "image_url", "imageUrl": "value"},
{"type": "image_url", "image_url": "value"},
{"type": "image_url", "image_url": {"url": "value"}},
]
for s in samples:
out = current_normalize(s)
print(s, "=>", out)
PYRepository: avikalpg/byok-relay
Length of output: 3520
Normalize LlamaIndex image URL input formats. LlamaIndex.TS image content uses type: 'image_url', image_url: { url }, so the current fallback produces image_url: { url: { url } } for real image messages.
Proposed fix
- if (part.type === 'image_url') return { type: 'image_url', image_url: { url: part.imageUrl || part.image_url } };
+ if (part.type === 'image_url') {
+ const url = part.imageUrl || (typeof part.image_url === 'object' ? part.image_url.url : part.image_url);
+ return { type: 'image_url', image_url: { url } };
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if (Array.isArray(content)) { | |
| const parts = content.map(part => { | |
| if (part.type === 'text') return { type: 'text', text: part.text }; | |
| if (part.type === 'image_url') return { type: 'image_url', image_url: { url: part.imageUrl || part.image_url } }; | |
| return { type: 'text', text: String(part) }; | |
| }); | |
| return { role, content: parts }; | |
| } | |
| if (Array.isArray(content)) { | |
| const parts = content.map(part => { | |
| if (part.type === 'text') return { type: 'text', text: part.text }; | |
| if (part.type === 'image_url') { | |
| const url = part.imageUrl || (typeof part.image_url === 'object' ? part.image_url.url : part.image_url); | |
| return { type: 'image_url', image_url: { url } }; | |
| } | |
| return { type: 'text', text: String(part) }; | |
| }); | |
| return { role, content: parts }; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llamaindex/src/index.js` around lines 142 - 149, Update the
image_url branch in the content mapping to normalize both LlamaIndex image URL
shapes: use the string directly when image_url is already a string, and extract
its url property when it is an object. Ensure the returned image_url contains a
single string url rather than a nested object.
| async ensureToken () { | ||
| if (this._token) return this._token; | ||
| const stored = this._storage.getItem('byok_relay_token'); | ||
| if (stored) { this._token = stored; return stored; } | ||
| return this.register(); | ||
| } | ||
|
|
||
| async register () { | ||
| const res = await fetch(`${this.relayUrl}/users`, { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ app_id: this.appId }), | ||
| }); | ||
| const data = await res.json(); | ||
| if (!res.ok) throw new Error(`Registration failed: ${data.error || res.status}`); | ||
| this._token = data.token; | ||
| this._storage.setItem('byok_relay_token', data.token); | ||
| return data.token; | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Token storage key isn't scoped to appId. ensureToken/register (and logout/deleteAccount) use a fixed 'byok_relay_token' key, so two ByokRelayClient instances with different appIds in the same origin overwrite each other's token, causing wrong-token auth failures. The sibling packages/langchain scopes this per app (byok_relay_token_${appId}); consider matching that here.
🔧 Proposed approach
constructor (opts = {}) {
this.relayUrl = (opts.relayUrl || DEFAULT_RELAY_URL).replace(/\/$/, '');
this.appId = opts.appId || 'llamaindex-app';
this._token = null;
+ this._tokenKey = `byok_relay_token_${this.appId}`;Then replace the literal 'byok_relay_token' uses with this._tokenKey.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/llamaindex/src/index.js` around lines 242 - 260, Scope the persisted
relay token by app ID across the ByokRelayClient lifecycle. Add or reuse a
_tokenKey derived from this.appId, then replace the fixed 'byok_relay_token' key
in ensureToken, register, logout, and deleteAccount with this._tokenKey,
preserving existing token loading, saving, and deletion behavior.
1 participant
Growth Day 44 —
@byok-relay/llamaindexLlamaIndex.TS custom LLM and embedding adapter for BYOK relay. Natural companion to
@byok-relay/langchain(Day 43), covering the other major JS AI framework.What ships
packages/llamaindex/src/index.jsByokRelayLLM(extendsBaseLLM):chat()non-streaming +stream()async-generator SSE streaming;complete();withTools(tools)bound-tool pattern; multi-modal message support; tool_call delta streamingByokRelayEmbedding(extendsBaseEmbedding):getQueryEmbedding(),getTextEmbedding(),getTextEmbeddings()with configurablebatchSize(default 512); routes to/relay/:provider/embeddingsByokRelayClient: plain-JS key management; localStorage default in browser; in-memory fallback on Node.js/edge; custom storage adapterpackages/llamaindex/README.md: quick-start (chat, streaming, embeddings+VectorStoreIndex, tool calling, ReActAgent, withTools), full API tables, provider table, comparison vs@byok-relay/langchain.llms.txt: LlamaIndex.TS Integration section + npm link.README.md:@byok-relay/llamaindexsection after LangChain.js integration.metrics/daily.jsonl: 2026-07-23 snapshot (stars=52).Tests
52 smoke tests passing (
node test/llamaindex.test.js) — nollamaindexpeer dep required.Next for Avi
cd packages/llamaindex && npm publish --access publicSummary by CodeRabbit
New Features
Documentation
Tests