Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions e2e-tests/openrouter_free_models.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
import { expect } from "@playwright/test";
import { testWithConfig } from "./helpers/test_helper";

const testOpenRouter = testWithConfig({
preLaunchHook: async () => {
process.env.OPENROUTER_API_KEY = "or-test-key";
},
});

testOpenRouter(
"openrouter free models are listed in the model picker",
async ({ po }) => {
await po.setUp();

await po.page.getByTestId("model-picker").click();
await po.page.getByText("Other AI providers", { exact: true }).click();
await po.page.getByText("OpenRouter", { exact: true }).click();

await expect(
po.page.getByText("Free models", { exact: true }),
).toBeVisible();
await po.page.getByText("Free models", { exact: true }).click();

await expect(
po.page.getByText("OpenRouter Free Models", { exact: true }),
).toBeVisible();

await expect(
po.page.getByRole("menuitem").filter({ hasText: "Free (OpenRouter)" }),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test expects "Free (OpenRouter)" but language_model_constants.ts:234 now shows displayName: "Free" - the test will fail.

Suggested change
po.page.getByRole("menuitem").filter({ hasText: "Free (OpenRouter)" }),
po.page.getByRole("menuitem").filter({ hasText: "Free" }),
Prompt To Fix With AI
This is a comment left during a code review.
Path: e2e-tests/openrouter_free_models.spec.ts
Line: 29

Comment:
Test expects "Free (OpenRouter)" but `language_model_constants.ts:234` now shows `displayName: "Free"` - the test will fail.

```suggestion
      po.page.getByRole("menuitem").filter({ hasText: "Free" }),
```

How can I resolve this? If you propose a fix, please make it concise.

).toBeVisible();
},
);
61 changes: 8 additions & 53 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

56 changes: 56 additions & 0 deletions src/__tests__/openrouter_free_models.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
import { describe, expect, it } from "vitest";
import { buildOpenRouterFreeModels } from "@/ipc/shared/openrouter_free_models";

describe("buildOpenRouterFreeModels", () => {
it("filters to free models and decorates display names", () => {
const models = [
{
id: "paid/model",
name: "Paid Model",
pricing: { prompt: "0.01", completion: "0.02" },
},
{
id: "free/model",
name: "Awesome Model",
pricing: { prompt: "0", completion: "0" },
context_length: 128_000,
top_provider: { max_completion_tokens: 16_384 },
},
{
id: "free/with-label",
name: "Free Model",
pricing: { prompt: 0, completion: 0, image: 0 },
},
];

const result = buildOpenRouterFreeModels(models);

expect(result).toHaveLength(2);
const awesome = result.find((model) => model.apiName === "free/model");
expect(awesome?.displayName).toBe("Awesome Model");
expect(awesome?.contextWindow).toBe(128_000);
expect(awesome?.maxOutputTokens).toBe(16_384);
expect(awesome?.tag).toBe("Free");

const labeled = result.find((model) => model.apiName === "free/with-label");
expect(labeled?.displayName).toBe("Free Model");
expect(labeled?.dollarSigns).toBe(0);
});

it("sanitizes external model name and description fields", () => {
const models = [
{
id: "free/sanitized",
name: " <img src=x onerror=alert(1)>Safe Name (free) ",
description: "<script>alert(1)</script> Useful model ",
pricing: { prompt: 0, completion: 0 },
},
];

const result = buildOpenRouterFreeModels(models);

expect(result).toHaveLength(1);
expect(result[0]?.displayName).toBe("Safe Name");
expect(result[0]?.description).toBe("alert(1) Useful model");
});
});
93 changes: 86 additions & 7 deletions src/components/ModelPicker.tsx
Comment thread
coderdylanbates marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import { useLocalModels } from "@/hooks/useLocalModels";
import { useLocalLMSModels } from "@/hooks/useLMStudioModels";
import { useLanguageModelsByProviders } from "@/hooks/useLanguageModelsByProviders";

import { ipc, LocalModel } from "@/ipc/types";
import { ipc, LocalModel, type LanguageModel } from "@/ipc/types";
import { useLanguageModelProviders } from "@/hooks/useLanguageModelProviders";
import { useSettings } from "@/hooks/useSettings";
import { PriceBadge } from "@/components/PriceBadge";
Expand Down Expand Up @@ -304,6 +304,22 @@ export function ModelPicker() {
provider?.id === "auto"
? "Dyad Turbo"
: (provider?.name ?? providerId);
const isOpenRouter = providerId === "openrouter";
const isOpenRouterFreeModel = (model: LanguageModel) =>
model.tag === "Free" ||
model.apiName.endsWith(":free") ||
Comment thread
coderdylanbates marked this conversation as resolved.
model.dollarSigns === 0;
Comment thread
coderdylanbates marked this conversation as resolved.
Comment thread
coderdylanbates marked this conversation as resolved.
const openRouterFreeModels = isOpenRouter
? models.filter((model) => isOpenRouterFreeModel(model))
: [];
const openRouterFreeModelNames = new Set(
openRouterFreeModels.map((model) => model.apiName),
);
const displayModels = isOpenRouter
? models.filter(
(model) => !openRouterFreeModelNames.has(model.apiName),
)
: models;
return (
<DropdownMenuSub key={providerId}>
<DropdownMenuSubTrigger className="w-full font-normal">
Expand Down Expand Up @@ -333,7 +349,68 @@ export function ModelPicker() {
{providerDisplayName + " Models"}
</DropdownMenuLabel>
<DropdownMenuSeparator />
{models.map((model) => (
{isOpenRouter && openRouterFreeModels.length > 0 && (
<>
<DropdownMenuSub>
<DropdownMenuSubTrigger className="w-full font-normal">
<div className="flex flex-col items-start w-full">
<span>Free models</span>
<span className="text-xs text-muted-foreground">
{openRouterFreeModels.length} models
</span>
</div>
</DropdownMenuSubTrigger>
<DropdownMenuSubContent className="w-56 max-h-100 overflow-y-auto">
<DropdownMenuLabel>
OpenRouter Free Models
</DropdownMenuLabel>
<DropdownMenuSeparator />
{openRouterFreeModels.map((model) => (
<DropdownMenuItem
key={`${providerId}-${model.apiName}-free`}
title={model.description}
className={
selectedModel.provider === providerId &&
selectedModel.name === model.apiName
? "bg-secondary"
: ""
}
onClick={() => {
const customModelId =
model.type === "custom"
? model.id
: undefined;
onModelSelect({
name: model.apiName,
provider: providerId,
customModelId,
});
setOpen(false);
}}
>
<div className="flex justify-between items-start w-full">
<span>{model.displayName}</span>
<PriceBadge
dollarSigns={model.dollarSigns}
/>
{model.tag &&
(model.tag !== "Free" ||
model.dollarSigns !== 0) && (
<span className="text-[10px] bg-primary/10 text-primary px-1.5 py-0.5 rounded-full font-medium">
{model.tag}
</span>
)}
</div>
Comment thread
coderdylanbates marked this conversation as resolved.
</DropdownMenuItem>
))}
</DropdownMenuSubContent>
</DropdownMenuSub>
{displayModels.length > 0 && (
<DropdownMenuSeparator />
)}
</>
)}
{displayModels.map((model) => (
<DropdownMenuItem
key={`${providerId}-${model.apiName}`}
title={model.description}
Expand All @@ -357,11 +434,13 @@ export function ModelPicker() {
<div className="flex justify-between items-start w-full">
<span>{model.displayName}</span>
<PriceBadge dollarSigns={model.dollarSigns} />
{model.tag && (
<span className="text-[10px] bg-primary/10 text-primary px-1.5 py-0.5 rounded-full font-medium">
{model.tag}
</span>
)}
{model.tag &&
(model.tag !== "Free" ||
model.dollarSigns !== 0) && (
<span className="text-[10px] bg-primary/10 text-primary px-1.5 py-0.5 rounded-full font-medium">
{model.tag}
</span>
)}
</div>
Comment thread
coderdylanbates marked this conversation as resolved.
</DropdownMenuItem>
))}
Expand Down
24 changes: 22 additions & 2 deletions src/ipc/shared/language_model_constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -271,6 +271,17 @@ export const MODEL_OPTIONS: Record<string, ModelOption[]> = {
temperature: 0,
dollarSigns: 1,
},
// https://openrouter.ai/qwen/qwen3-coder
{
name: "qwen/qwen3-coder:free",
displayName: "Qwen3 Coder",
description: "Use for free (data may be used for training)",
maxOutputTokens: 32_000,
contextWindow: 196_608,
temperature: 0,
dollarSigns: 1,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Free model qwen/qwen3-coder:free has incorrect dollarSigns: 1 instead of 0

The newly added qwen/qwen3-coder:free model has dollarSigns: 1 despite being a free model (:free suffix, description says "Use for free"). This causes the PriceBadge component (src/components/PriceBadge.tsx:10) to render "$" instead of "Free". Furthermore, the dedupe logic in src/ipc/shared/language_model_helpers.ts:155 gives hardcoded models precedence over dynamically fetched free models (which correctly have dollarSigns: 0), so the incorrect value persists even after hydration.

Suggested change
dollarSigns: 1,
dollarSigns: 0,
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

},
// https://openrouter.ai/z-ai/glm-5
{
name: "z-ai/glm-5",
displayName: "GLM 5",
Expand All @@ -280,6 +291,16 @@ export const MODEL_OPTIONS: Record<string, ModelOption[]> = {
temperature: 0.7,
dollarSigns: 2,
},
// https://openrouter.ai/mistralai/devstral-2512
{
name: "mistralai/devstral-2512:free",
displayName: "Devstral 2",
description: "Use for free (data may be used for training)",
maxOutputTokens: 32_000,
contextWindow: 200_000,
temperature: 0.7,
dollarSigns: 2,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Free model mistralai/devstral-2512:free has incorrect dollarSigns: 2 instead of 0

The newly added mistralai/devstral-2512:free model has dollarSigns: 2 despite being a free model (:free suffix, description says "Use for free"). This causes the PriceBadge component (src/components/PriceBadge.tsx:10) to render "$$" instead of "Free". The same dedupe issue as with qwen/qwen3-coder:free applies: src/ipc/shared/language_model_helpers.ts:155 gives hardcoded models precedence, so the incorrect dollarSigns value overrides the correct 0 from the dynamic free model list.

Suggested change
dollarSigns: 2,
dollarSigns: 0,
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

},
Comment on lines 276 to +303

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect dollarSigns on new :free hardcoded models

Both new free-tier models are given non-zero dollarSigns values (1 and 2), but because the deduplication in language_model_helpers.ts uses [...freeModels, ...hardcodedModels], hardcoded entries always overwrite the API-fetched entries with dollarSigns: 0. The result is that these two models end up with dollarSigns: 1 / dollarSigns: 2 in the merged list, so PriceBadge will display $ / $$ for them inside the "Free models" submenu — contradicting the "free" label.

Both entries should use dollarSigns: 0:

Suggested change
// https://openrouter.ai/qwen/qwen3-coder
{
name: "qwen/qwen3-coder:free",
displayName: "Qwen3 Coder",
description: "Use for free (data may be used for training)",
maxOutputTokens: 32_000,
contextWindow: 196_608,
temperature: 0,
dollarSigns: 0,
},

And for mistralai/devstral-2512:free:

Suggested change
dollarSigns: 0,
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/ipc/shared/language_model_constants.ts
Line: 276-303

Comment:
**Incorrect `dollarSigns` on new `:free` hardcoded models**

Both new free-tier models are given non-zero `dollarSigns` values (`1` and `2`), but because the deduplication in `language_model_helpers.ts` uses `[...freeModels, ...hardcodedModels]`, hardcoded entries always **overwrite** the API-fetched entries with `dollarSigns: 0`. The result is that these two models end up with `dollarSigns: 1` / `dollarSigns: 2` in the merged list, so `PriceBadge` will display `$` / `$$` for them inside the "Free models" submenu — contradicting the "free" label.

Both entries should use `dollarSigns: 0`:

```suggestion
    // https://openrouter.ai/qwen/qwen3-coder
    {
      name: "qwen/qwen3-coder:free",
      displayName: "Qwen3 Coder",
      description: "Use for free (data may be used for training)",
      maxOutputTokens: 32_000,
      contextWindow: 196_608,
      temperature: 0,
      dollarSigns: 0,
    },
```

And for `mistralai/devstral-2512:free`:
```suggestion
      dollarSigns: 0,
```

How can I resolve this? If you propose a fix, please make it concise.

{
name: "z-ai/glm-4.7",
displayName: "GLM 4.7",
Expand Down Expand Up @@ -321,7 +342,7 @@ export const MODEL_OPTIONS: Record<string, ModelOption[]> = {
},
{
name: "free",
displayName: "Free (OpenRouter)",
displayName: "Free",
description: "Selects from one of the free OpenRouter models",
tag: "Free",
// These are below Gemini 2.5 Pro & Flash limits
Expand Down Expand Up @@ -505,7 +526,6 @@ export const FREE_OPENROUTER_MODEL_NAMES = MODEL_OPTIONS.openrouter
(model) => model.name.endsWith(":free") || model.name.endsWith("/free"),
)
.map((model) => model.name);

export const PROVIDER_TO_ENV_VAR: Record<string, string> = {
openai: "OPENAI_API_KEY",
anthropic: "ANTHROPIC_API_KEY",
Expand Down
10 changes: 10 additions & 0 deletions src/ipc/shared/language_model_helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
MODEL_OPTIONS,
PROVIDER_TO_ENV_VAR,
} from "./language_model_constants";
import { getOpenRouterFreeModels } from "./openrouter_free_models";
/**
* Fetches language model providers from both the database (custom) and hardcoded constants (cloud),
* merging them with custom providers taking precedence.
Expand Down Expand Up @@ -148,6 +149,15 @@ export async function getLanguageModels({
}
}

if (providerId === "openrouter") {
const freeModels = getOpenRouterFreeModels();
const dedupe = new Map<string, LanguageModel>();
[...freeModels, ...hardcodedModels].forEach((model) => {
dedupe.set(model.apiName, model);
});
hardcodedModels = Array.from(dedupe.values());
}
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
Comment on lines +152 to +159

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deduplication direction discards API-fetched data

The array spread [...freeModels, ...hardcodedModels] means that when the same apiName exists in both lists, Map.set is called last with the hardcoded entry, so hardcoded values always win. The API-fetched contextWindow, maxOutputTokens, dollarSigns: 0, and tag: "Free" metadata are silently discarded in favour of the static constants file.

This defeats the PR's goal of pulling current metadata from OpenRouter at runtime. The spread order should be reversed so that API data overrides the hardcoded fallback:

Suggested change
if (providerId === "openrouter") {
const freeModels = getOpenRouterFreeModels();
const dedupe = new Map<string, LanguageModel>();
[...freeModels, ...hardcodedModels].forEach((model) => {
dedupe.set(model.apiName, model);
});
hardcodedModels = Array.from(dedupe.values());
}
if (providerId === "openrouter") {
const freeModels = getOpenRouterFreeModels();
const dedupe = new Map<string, LanguageModel>();
[...hardcodedModels, ...freeModels].forEach((model) => {
dedupe.set(model.apiName, model);
});
hardcodedModels = Array.from(dedupe.values());
}
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/ipc/shared/language_model_helpers.ts
Line: 152-159

Comment:
**Deduplication direction discards API-fetched data**

The array spread `[...freeModels, ...hardcodedModels]` means that when the same `apiName` exists in both lists, `Map.set` is called last with the **hardcoded** entry, so hardcoded values always win. The API-fetched `contextWindow`, `maxOutputTokens`, `dollarSigns: 0`, and `tag: "Free"` metadata are silently discarded in favour of the static constants file.

This defeats the PR's goal of pulling current metadata from OpenRouter at runtime. The spread order should be reversed so that API data overrides the hardcoded fallback:

```suggestion
  if (providerId === "openrouter") {
    const freeModels = getOpenRouterFreeModels();
    const dedupe = new Map<string, LanguageModel>();
    [...hardcodedModels, ...freeModels].forEach((model) => {
      dedupe.set(model.apiName, model);
    });
    hardcodedModels = Array.from(dedupe.values());
  }
```

How can I resolve this? If you propose a fix, please make it concise.


return [...hardcodedModels, ...customModels];
}

Expand Down
Loading
Loading