Add voice cloning API endpoint with reference audio enhancement - #401
Add voice cloning API endpoint with reference audio enhancement#401gianpaj wants to merge 4 commits into
Conversation
Add POST /api/v1/clone to the external, API-key-authenticated v1 API. It accepts a JSON body with reference audio as either a public URL (reference_audio_url) or base64 (reference_audio), reuses the existing auth/rate-limit/billing/error infrastructure, and reuses the Mistral Voxtral and Replicate Chatterbox cloning providers via a new shared clone-service module (the internal /api/clone-voice route is left untouched). - lib/clone/clone-service.ts: reusable, RouteError-free cloning helpers (provider/locale resolution, audio normalization, provider calls) that throw a typed CloneServiceError mapped to API errors by the route - lib/api/schemas.ts: VoiceCloneRequest/Response schemas - lib/api/pricing.ts: api_voice_cloning pricing for mistral/replicate - lib/api/openapi.ts: register /api/v1/clone + schemas (v1.1.0) - docs: api.mdx Voice Cloning section + generated endpoint page - dashboard: voice cloning FeatureCard on the API Keys page - i18n: cloneTitle/cloneDescription in all six locales Tests: api-v1-clone (route), clone-service (helpers), pricing updates.
|
|
|
Pullfrog stalled — likely cause: The agent stopped emitting events for 300s and was killed by the activity-timeout watchdog. No events were emitted before the failure. Recent agent stderr |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Claude finished @gianpaj's task in 4m 46s —— View job Code Review
OverallSolid implementation. The clone service module is well-organized, error mapping is comprehensive, billing is split correctly between base cloning and optional enhancement, and the test suite covers most happy/sad paths for the route. The enhancement-output URL allow-listing in There's one critical security issue and a handful of correctness / robustness improvements worth addressing before this is exposed externally. 🔴 Critical
🟡 Should fix
🟢 Minor / nits
Notes / observations
| Branch: |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new Voice Cloning API endpoint (POST /api/v1/clone) that allows users to clone voices from reference recordings (via URL or base64) and synthesize speech using Mistral or Replicate. It includes API documentation, OpenAPI schema updates, pricing configurations, and comprehensive unit tests. The review feedback highlights a critical billing vulnerability where expensive audio enhancement is performed before credit validation, a potential OOM risk when fetching unbounded reference audio streams, an unawaited logging promise in a serverless context, and an opportunity to optimize buffer hashing using standard node:crypto.
| enhancementDurationSeconds = processed.duration; | ||
| enhancementCredits = calculateReferenceAudioEnhancementCredits( | ||
| enhancementDurationSeconds, | ||
| ); | ||
| enhancementDollarAmount = getReferenceAudioEnhancementDollarCost( | ||
| enhancementDurationSeconds, | ||
| ); | ||
|
|
||
| try { | ||
| const enhanced = await enhanceReferenceAudio({ | ||
| abortSignal: request.signal, | ||
| buffer: processed.buffer, | ||
| filename: reference.filename, | ||
| mimeType: processed.mimeType, | ||
| }); | ||
| cloneBuffer = enhanced.buffer; | ||
| cloneMimeType = enhanced.mimeType; | ||
| cloneAudioHash = await generateBufferHash(enhanced.buffer); | ||
| cloneDuration = | ||
| (await getAudioDuration(enhanced.buffer, enhanced.mimeType)) ?? | ||
| cloneDuration; | ||
| referenceAudioEnhanced = true; | ||
| enhancementModelUsed = enhanced.modelUsed; | ||
| enhancementRequestId = enhanced.requestId; | ||
| creditsUsed = baseCloneCredits + enhancementCredits; | ||
| } catch (enhancementError) { | ||
| // Enhancement is best-effort: fall back to the original reference audio. | ||
| captureException(enhancementError, { | ||
| extra: { requestId, endpoint: ENDPOINT, locale }, | ||
| }); | ||
| enhancementCredits = 0; | ||
| enhancementDollarAmount = 0; | ||
| enhancementDurationSeconds = null; | ||
| creditsUsed = baseCloneCredits; | ||
| } | ||
| } | ||
|
|
||
| validateAudioDuration(cloneDuration, provider); | ||
|
|
||
| const currentCredits = await getCreditsAdmin(userId); | ||
| if (currentCredits < creditsUsed) { | ||
| await log({ | ||
| status: 402, | ||
| errorCode: 'insufficient_credits', | ||
| userId, | ||
| apiKeyId: authResult.apiKeyId, | ||
| textLength: input.length, | ||
| }); | ||
| return respond( | ||
| createApiError({ | ||
| message: 'Insufficient credits', | ||
| type: 'permission_error', | ||
| code: 'insufficient_credits', | ||
| }), | ||
| { status: 402 }, | ||
| ); | ||
| } |
There was a problem hiding this comment.
Critical Billing Vulnerability: Credit Check Performed After Expensive API Call
The endpoint performs the expensive enhanceReferenceAudio API call (which invokes an external provider and costs real money/credits) before verifying if the user has sufficient credits.
An attacker or user with 0 credits can repeatedly call this endpoint with enhance_reference_audio: true, forcing the server to perform the enhancement and incur costs before failing with a 402 Insufficient credits error.
Fix: Move the credit check to happen before the enhanceReferenceAudio call. We can estimate the total credits needed (base credits + enhancement credits) and validate the user's balance early.
enhancementDurationSeconds = processed.duration;
enhancementCredits = calculateReferenceAudioEnhancementCredits(
enhancementDurationSeconds,
);
enhancementDollarAmount = getReferenceAudioEnhancementDollarCost(
enhancementDurationSeconds,
);
creditsUsed = baseCloneCredits + enhancementCredits;
}
const currentCredits = await getCreditsAdmin(userId);
if (currentCredits < creditsUsed) {
await log({
status: 402,
errorCode: 'insufficient_credits',
userId,
apiKeyId: authResult.apiKeyId,
textLength: input.length,
});
return respond(
createApiError({
message: 'Insufficient credits',
type: 'permission_error',
code: 'insufficient_credits',
}),
{ status: 402 },
);
}
if (enhancementEnabled) {
try {
const enhanced = await enhanceReferenceAudio({
abortSignal: request.signal,
buffer: processed.buffer,
filename: reference.filename,
mimeType: processed.mimeType,
});
cloneBuffer = enhanced.buffer;
cloneMimeType = enhanced.mimeType;
cloneAudioHash = await generateBufferHash(enhanced.buffer);
cloneDuration =
(await getAudioDuration(enhanced.buffer, enhanced.mimeType)) ??
cloneDuration;
referenceAudioEnhanced = true;
enhancementModelUsed = enhanced.modelUsed;
enhancementRequestId = enhanced.requestId;
} catch (enhancementError) {
// Enhancement is best-effort: fall back to the original reference audio.
captureException(enhancementError, {
extra: { requestId, endpoint: ENDPOINT, locale },
});
enhancementCredits = 0;
enhancementDollarAmount = 0;
enhancementDurationSeconds = null;
creditsUsed = baseCloneCredits;
}
}
validateAudioDuration(cloneDuration, provider);| const contentLength = response.headers.get('content-length'); | ||
| if ( | ||
| contentLength && | ||
| Number.isFinite(Number(contentLength)) && | ||
| Number(contentLength) > REFERENCE_AUDIO_MAX_FETCH_BYTES | ||
| ) { | ||
| throw new CloneServiceError( | ||
| 'unsupported_audio_format', | ||
| 'Reference audio exceeds the maximum allowed size.', | ||
| { param: 'reference_audio_url' }, | ||
| ); | ||
| } | ||
|
|
||
| const arrayBuffer = await response.arrayBuffer(); | ||
| if (arrayBuffer.byteLength > REFERENCE_AUDIO_MAX_FETCH_BYTES) { | ||
| throw new CloneServiceError( | ||
| 'unsupported_audio_format', | ||
| 'Reference audio exceeds the maximum allowed size.', | ||
| { param: 'reference_audio_url' }, | ||
| ); | ||
| } |
There was a problem hiding this comment.
Potential Out Of Memory (OOM) via Unbounded response.arrayBuffer()
Although there is a check on the content-length header, if the external server omits the content-length header or sends a spoofed/incorrect one, calling await response.arrayBuffer() will read the entire response body into memory.
If a user provides a URL pointing to an extremely large file (e.g., several gigabytes), this will consume all available memory and crash the serverless function with an Out Of Memory (OOM) error.
Fix: Enforce a hard limit on the fetched bytes by consuming the response body stream chunk-by-chunk and aborting the request if the total bytes read exceed REFERENCE_AUDIO_MAX_FETCH_BYTES.
const contentLength = response.headers.get('content-length');
if (
contentLength &&
Number.isFinite(Number(contentLength)) &&
Number(contentLength) > REFERENCE_AUDIO_MAX_FETCH_BYTES
) {
throw new CloneServiceError(
'unsupported_audio_format',
'Reference audio exceeds the maximum allowed size.',
{ param: 'reference_audio_url' },
);
}
const reader = response.body?.getReader();
if (!reader) {
throw new CloneServiceError(
'unsupported_audio_format',
'Failed to read reference audio stream.',
{ param: 'reference_audio_url' },
);
}
const chunks: Uint8Array[] = [];
let totalBytes = 0;
while (true) {
const { done, value } = await reader.read();
if (done) break;
totalBytes += value.length;
if (totalBytes > REFERENCE_AUDIO_MAX_FETCH_BYTES) {
reader.cancel();
throw new CloneServiceError(
'unsupported_audio_format',
'Reference audio exceeds the maximum allowed size.',
{ param: 'reference_audio_url' },
);
}
chunks.push(value);
}
const buffer = Buffer.concat(chunks);| log({ | ||
| status: 200, | ||
| userId, | ||
| apiKeyId: authResult.apiKeyId, | ||
| model: modelUsed, | ||
| textLength: input.length, | ||
| provider, | ||
| creditsUsed, | ||
| dollarAmount, | ||
| referenceAudioEnhanced, | ||
| userHasPaid, | ||
| }).catch((err) => { | ||
| console.error('[clone] success-path log failed:', err); | ||
| }); |
There was a problem hiding this comment.
Unawaited Success Log Promise in Route Handler
On the success path, the log promise is called but not awaited. In serverless environments (like Vercel / AWS Lambda), the container can be suspended immediately after the response is returned. If the log promise is not resolved before suspension, the log write may be delayed, truncated, or completely lost.
Since all other log calls in this file are correctly awaited, this success log should also be awaited.
| log({ | |
| status: 200, | |
| userId, | |
| apiKeyId: authResult.apiKeyId, | |
| model: modelUsed, | |
| textLength: input.length, | |
| provider, | |
| creditsUsed, | |
| dollarAmount, | |
| referenceAudioEnhanced, | |
| userHasPaid, | |
| }).catch((err) => { | |
| console.error('[clone] success-path log failed:', err); | |
| }); | |
| await log({ | |
| status: 200, | |
| userId, | |
| apiKeyId: authResult.apiKeyId, | |
| model: modelUsed, | |
| textLength: input.length, | |
| provider, | |
| creditsUsed, | |
| dollarAmount, | |
| referenceAudioEnhanced, | |
| userHasPaid, | |
| }).catch((err) => { | |
| console.error('[clone] success-path log failed:', err); | |
| }); |
| export async function generateBufferHash(buffer: Buffer): Promise<string> { | ||
| const data = new Uint8Array(buffer); | ||
| const hashBuffer = await crypto.subtle.digest('SHA-256', data); | ||
| const hashArray = Array.from(new Uint8Array(hashBuffer)); | ||
| return hashArray.map((b) => b.toString(16).padStart(2, '0')).join(''); | ||
| } |
There was a problem hiding this comment.
Use Standard node:crypto for Buffer Hashing
The generateBufferHash function uses crypto.subtle.digest (Web Crypto API) which requires converting the buffer to a Uint8Array, returning a Promise, and manually mapping the resulting array to a hex string.
Since node:crypto is already imported in this file, we can use the standard, synchronous, and highly optimized createHash function. This simplifies the code, avoids Web Crypto overhead, and removes the need for manual hex formatting.
export async function generateBufferHash(buffer: Buffer): Promise<string> {
const { createHash } = await import('node:crypto');
return createHash('sha256').update(buffer).digest('hex');
}![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| reference_audio_format?: string; | ||
| }): Promise<{ buffer: Buffer; mimeType: string; filename: string }> { | ||
| if (data.reference_audio_url) { | ||
| const response = await fetch(data.reference_audio_url); |
There was a problem hiding this comment.
🔴 Security: SSRF in reference_audio_url fetch.
fetch(data.reference_audio_url) is called against a user-provided URL with no scheme/host restriction. The Zod schema only validates that the string is a syntactically valid URL — it does not block:
http://169.254.169.254/...(AWS metadata service)http://localhost:6379/...,http://10.0.0.1/...(internal services)file://,gopher://,ftp://schemes- Other internal infrastructure
Because the response body is returned to the caller (via the eventually-generated audio URL we control) and any non-2xx is surfaced as a status code in the error, an attacker can use this to probe internal networks via timing and status codes, or read responses from internal HTTP services that may serve audio-like content.
Compare with apps/web/lib/clone/reference-audio-enhancement.ts:38-56 where the outbound fal.ai URL is restricted to fal-cdn.com / fal.media with https:-only — the inbound user URL needs equivalent treatment.
Suggested mitigations:
- Enforce
parsedUrl.protocol === 'https:'(or at minimum rejectfile:,data:, etc.) - Resolve the hostname and reject RFC1918 / loopback / link-local / IPv6 ULA addresses (or use an allow-list of trusted CDN hosts)
- Add an
AbortSignal.timeout(...)to bound the request duration (currently no timeout — a slow URL can block up tomaxDuration = 800s) - Set
redirect: 'manual'(or explicitly handle redirects) so SSRF protection isn't bypassed by a 302 to an internal host
| /^data:[^;]+;base64,/, | ||
| '', | ||
| ); | ||
| const buffer = Buffer.from(base64, 'base64'); |
There was a problem hiding this comment.
Permissive base64 decoding.
Buffer.from(base64, 'base64') silently drops invalid characters and produces some buffer for almost any non-empty string, so the buffer.length === 0 check only catches the truly-empty case. A long string of garbage will pass this check and then fail much later (during parseBuffer, conversion, or at the provider) with an error that isn't as user-friendly.
Consider an explicit validation pass to fail-fast with a clear error:
if (!/^[A-Za-z0-9+/]+=*$/.test(base64) || base64.length % 4 !== 0) {
throw new CloneServiceError('unsupported_audio_format', 'reference_audio is not valid base64', { param: 'reference_audio' });
}Also note: the regex ^data:[^;]+;base64, only handles the ;base64, data URL form. If a caller passes data:audio/wav,... (base64-less data URL), it slips through. Probably worth either accepting both or rejecting non-base64 data URLs explicitly.
|
|
||
| validateAudioDuration(cloneDuration, provider); | ||
|
|
||
| const currentCredits = await getCreditsAdmin(userId); |
There was a problem hiding this comment.
Time-of-check / time-of-use on credits.
Between getCreditsAdmin here and reduceCreditsAdmin on line 429, multiple concurrent requests from the same API key can all observe sufficient balance and then all decrement, allowing the account to go negative.
This is a pre-existing pattern in the speech route, so probably out of scope for this PR — but voice cloning is more expensive (especially with enhancement) so the financial impact per race is larger. Worth tracking as a follow-up:
- Move the balance check into the same SQL statement as the reduce (e.g.
UPDATE credits SET balance = balance - $amount WHERE user_id = $u AND balance >= $amount RETURNING balance), or - Use a Postgres advisory lock per
userIdfor the duration of the credit operation.
Also, the rate limiter (per key) bounds blast radius — but does not eliminate the race.
| userHasPaid, | ||
| }).catch((err) => { | ||
| console.error('[clone] success-path log failed:', err); | ||
| }); |
There was a problem hiding this comment.
Inconsistent logging: success path is not awaited.
Every other log({...}) call in this handler is awaited; only the 200 success path is fire-and-forget with a .catch. On Vercel, an unawaited promise that resolves after the response is sent may be cut off by the runtime before it flushes — meaning successful cloning events can silently drop from the log pipeline, while every error path logs reliably.
Either await it like the rest, or event.waitUntil(...) it via after() from next/server if you specifically want it off the response path.
| }); | |
| await log({ | |
| status: 200, | |
| userId, | |
| apiKeyId: authResult.apiKeyId, | |
| model: modelUsed, | |
| textLength: input.length, | |
| provider, | |
| creditsUsed, | |
| dollarAmount, | |
| referenceAudioEnhanced, | |
| userHasPaid, | |
| }); |
| throw new Error('Mistral Voxtral response did not return a valid WAV file'); | ||
| } | ||
|
|
||
| return { buffer, modelUsed: model, requestId: randomUUID() }; |
There was a problem hiding this comment.
predictionId is a synthesized UUID, not the upstream request id.
randomUUID() is opaque and not traceable back to a Mistral request. The Mistral SDK exposes a request id (e.g. response.requestID or via response headers depending on SDK version) — capturing that would let support correlate the saved row in audio_files.prediction_id with logs on Mistral's side when guardrail/policy issues arise.
If the SDK doesn't expose one cleanly, at minimum prefix the UUID (e.g. local-${randomUUID()}) so it's obvious from the DB row that this is a locally-generated id, not an upstream one.
|
|
||
| const canNormalizeToWav = isConversionSupported(normalizedMimeType, filename); | ||
| const shouldNormalizeToWav = | ||
| provider === 'mistral' || enhancementEnabled || canNormalizeToWav; |
There was a problem hiding this comment.
Redundant boolean in shouldNormalizeToWav.
shouldNormalizeToWav = provider === 'mistral' || enhancementEnabled || canNormalizeToWav; — since canNormalizeToWav is true whenever conversion is supported, this is effectively "normalize whenever it's possible." The first two clauses only matter when canNormalizeToWav === false, in which case we enter the if (... && needsConversion) block and explicitly throw unsupported_audio_format.
That's the intent for Mistral and enhancement (force-fail unsupported formats), but means the variable name is a bit misleading: it's not "should normalize" — it's "should error when format can't normalize."
Consider renaming + clarifying with a comment:
// For Mistral and enhancement we *require* normalization — fail loudly if not possible.
// For Replicate, normalize when we can but pass through unknown formats untouched.
const requireWavNormalization = provider === 'mistral' || enhancementEnabled;
const shouldAttemptConversion = requireWavNormalization || canNormalizeToWav;Functionally equivalent, just easier to reason about.
| export async function generateBufferHash(buffer: Buffer): Promise<string> { | ||
| const data = new Uint8Array(buffer); | ||
| const hashBuffer = await crypto.subtle.digest('SHA-256', data); | ||
| const hashArray = Array.from(new Uint8Array(hashBuffer)); |
There was a problem hiding this comment.
Minor: crypto.subtle.digest works but node:crypto.createHash is faster.
You're already importing randomUUID from node:crypto at the top. For hashing a Buffer in Node, createHash('sha256').update(buffer).digest('hex') is more direct and avoids the WebCrypto detour (and the Uint8Array wrapping). Tiny micro-optimization, but it also makes the implementation environment-independent (doesn't depend on crypto.subtle being a global, which is true in Node 24 but isn't in older runtimes).
import { createHash, randomUUID } from 'node:crypto';
export function generateBufferHash(buffer: Buffer): string {
return createHash('sha256').update(buffer).digest('hex');
}(Note: this changes the return type from Promise<string> to string, so callers like cloneAudioHash = await generateBufferHash(...) need to drop the await. Or keep the async signature for callsite compatibility.)
| } | ||
|
|
||
| export function validateLocale(locale: string): void { | ||
| const localeConfig = SUPPORTED_LOCALE_CODES.find((l) => l.code === locale); |
There was a problem hiding this comment.
O(n) .find on every request.
SUPPORTED_LOCALE_CODES.find(...) is called in both validateLocale and cloneVoiceWithReplicate. The list is small (~25 entries) so this is fine, but since you already have a Set pattern for Voxtral locales (VOXTRAL_SUPPORTED_LOCALE_CODES in constants.ts), consider mirroring it:
const SUPPORTED_LOCALE_CODE_SET = new Set(SUPPORTED_LOCALE_CODES.map((l) => l.code));…and use .has() for membership checks. Keeps the array for iteration / the error message.
| }) | ||
| .refine( | ||
| (data) => | ||
| Boolean(data.reference_audio_url) !== Boolean(data.reference_audio), |
There was a problem hiding this comment.
Refinement error path only flags reference_audio_url.
When the user provides neither or both fields, the validation error is attached to reference_audio_url. That's a bit misleading when the actual problem might be that they provided reference_audio and reference_audio_url simultaneously, or neither.
Consider either reporting both paths, or omitting path so it surfaces as a top-level form error:
.refine(
(data) => Boolean(data.reference_audio_url) !== Boolean(data.reference_audio),
{ message: 'Provide exactly one of "reference_audio_url" or "reference_audio"' },
)Minor UX nit — the existing test (api-v1-clone.test.ts:110-130) already covers both branches but only asserts the top-level error code, so this change won't break it.
| creditsUsed: 120, // 12s * 10 credits/sec | ||
| }), | ||
| ); | ||
| }); |
There was a problem hiding this comment.
Test gap: no coverage for the enhancement fallback path.
The route advertises (and the PR description emphasizes) that enhancement is best-effort — if enhanceReferenceAudio throws, the request should still succeed using the original audio, with no enhancement credits charged. There's no test for that path. A useful addition:
it('falls back to original audio when enhancement fails', async () => {
mockFalSubscribe.mockRejectedValueOnce(new Error('fal-ai/deepfilternet3 failed'));
const response = await POST(cloneRequest({
input: 'Hello world',
locale: 'en',
reference_audio: createWavBase64(),
enhance_reference_audio: true,
}));
expect(response.status).toBe(200);
const json = await response.json();
expect(json.credits_used).toBeLessThan(120); // No enhancement credits
// Only the base cloning usage event — no audio_processing event.
expect(vi.mocked(insertUsageEvent)).toHaveBeenCalledTimes(1);
});Also missing: test for reference_audio_too_long (duration > 60s with enhance_reference_audio: true) and reference_audio_too_large (buffer > 25 MB).
There was a problem hiding this comment.
Pull request overview
Adds a new external voice-cloning API endpoint (POST /api/v1/clone) with provider routing (Mistral Voxtral vs Replicate), optional reference-audio enhancement, and billing/pricing + OpenAPI/docs updates to expose the feature in the product and documentation.
Changes:
- Implement
POST /api/v1/cloneroute with validation, provider execution, storage upload, and usage-event + credit deduction logic. - Add core cloning utilities/service code (audio normalization, locale/provider resolution, enhancement billing helpers).
- Extend external API pricing + OpenAPI schemas/docs and add unit/integration tests.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| apps/web/app/api/v1/clone/route.ts | New external voice cloning endpoint orchestration (auth, rate limit, validation, cloning, storage, billing). |
| apps/web/lib/clone/clone-service.ts | Core cloning utilities/provider clients (Mistral/Replicate), audio normalization, and enhancement billing helpers. |
| apps/web/lib/api/schemas.ts | Adds request/response schemas for /api/v1/clone. |
| apps/web/lib/api/openapi.ts | Exposes /api/v1/clone in OpenAPI and bumps API version/description. |
| apps/web/lib/api/pricing.ts | Adds pricing entries/provider typing for voice cloning (Mistral + Replicate). |
| apps/web/lib/api/logger.ts | Extends log fields to include referenceAudioEnhanced. |
| apps/web/tests/api-v1-clone.test.ts | Integration tests for /api/v1/clone behavior and billing scenarios. |
| apps/web/tests/clone-service.test.ts | Unit tests for clone-service helpers/validation logic. |
| apps/web/tests/api-pricing.test.ts | Updates pricing tests for new cloning pricing rules. |
| apps/web/app/[lang]/(dashboard)/dashboard/api-keys/api-keys.tsx | Adds dashboard link/card for the new clone endpoint docs. |
| apps/web/messages/en.json | UI copy update + formatting tweak. |
| apps/web/messages/da.json | Adds localized “Clone” feature card strings. |
| apps/web/messages/de.json | Adds localized “Clone” feature card strings. |
| apps/web/messages/es.json | Adds localized “Clone” feature card strings. |
| apps/web/messages/fr.json | Adds localized “Clone” feature card strings. |
| apps/web/messages/it.json | Adds localized “Clone” feature card strings. |
| apps/docs/content/docs/api-reference/endpoints/api/v1/clone/post.mdx | Generated endpoint doc page for /api/v1/clone. |
| apps/docs/content/docs/api-reference/api.mdx | Adds human-authored docs/examples for voice cloning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const buffer = Buffer.from(base64, 'base64'); | ||
| if (buffer.length === 0) { | ||
| throw new CloneServiceError( | ||
| 'unsupported_audio_format', | ||
| 'reference_audio is not valid base64-encoded audio.', | ||
| { param: 'reference_audio' }, | ||
| ); | ||
| } | ||
| const mimeType = | ||
| data.reference_audio_format?.split(';')[0]?.trim() || 'audio/wav'; | ||
| return { buffer, mimeType, filename: 'reference-audio' }; |
| if (data.reference_audio_url) { | ||
| const response = await fetch(data.reference_audio_url); | ||
| if (!response.ok) { |
| createApiError({ | ||
| message: error.message, | ||
| type: mapping.type, | ||
| code: mapping.code, | ||
| param: mapping.param ?? null, | ||
| }), |
| if ( | ||
| processed.duration !== null && | ||
| processed.duration > REFERENCE_AUDIO_ENHANCEMENT_MAX_DURATION | ||
| ) { | ||
| return respond( | ||
| createApiError({ | ||
| message: `Reference audio enhancement supports clips up to ${REFERENCE_AUDIO_ENHANCEMENT_MAX_DURATION} seconds`, | ||
| type: 'invalid_request_error', | ||
| code: 'reference_audio_too_long', | ||
| param: 'reference_audio', | ||
| }), | ||
| { status: 400 }, | ||
| ); |
| if ( | ||
| processed.buffer.length > REFERENCE_AUDIO_ENHANCEMENT_MAX_INPUT_BYTES | ||
| ) { | ||
| return respond( | ||
| createApiError({ | ||
| message: 'Reference audio enhancement input exceeds size limit', | ||
| type: 'invalid_request_error', | ||
| code: 'reference_audio_too_large', | ||
| param: 'reference_audio', | ||
| }), | ||
| { status: 400 }, | ||
| ); |
| throw new CloneServiceError( | ||
| 'audio_conversion_failed', | ||
| 'Failed to convert audio format to WAV. Reference audio must be MP3, OGG, Opus, or WAV.', | ||
| { mimeType: normalizedMimeType }, | ||
| ); |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76f77b0107
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| reference_audio_format?: string; | ||
| }): Promise<{ buffer: Buffer; mimeType: string; filename: string }> { | ||
| if (data.reference_audio_url) { | ||
| const response = await fetch(data.reference_audio_url); |
There was a problem hiding this comment.
When an API key holder supplies reference_audio_url, this server-side fetch will request any URL before audio validation, including localhost/link-local/private-network hosts such as http://169.254.169.254/... or internal services. Because the endpoint returns distinguishable status/timing/errors, this opens the new clone API to SSRF and internal endpoint scanning; restrict downloads to public https? targets, block private/link-local/localhost addresses and redirects to them, and add a bounded timeout.
Useful? React with 👍 / 👎.
The docs site generates endpoint reference pages from the live production OpenAPI document (https://sexyvoice.ai/api/v1/openapi) at build time. Since /api/v1/clone is not yet deployed to production, the hand-committed APIPage for that operation could not be resolved and failed Next.js page-data collection, breaking the sexyvoice-docs deployment. The page will be regenerated automatically by 'generate-openapi-docs' once the endpoint is live in production. The prose docs and examples in api.mdx remain. https://claude.ai/code/session_01JEyH3uDy8xL49NnUygrDqz
Address security and correctness feedback on the new voice cloning endpoint: - Billing: verify credit balance BEFORE invoking the paid reference-audio enhancement, so a user with insufficient credits can no longer force the billable enhancement call that then 402s. - SSRF: validate reference_audio_url (http/https only, no credentials, reject loopback/link-local/private/CGNAT/reserved IPs via DNS resolution), disallow redirects, and add a fetch timeout. - OOM: stream the URL download and abort once a hard byte cap is exceeded, instead of buffering the whole body via arrayBuffer(); enforce the same size cap on the base64 path. - Map CloneServiceError.details.param to the response so URL errors are attributed to reference_audio_url (not reference_audio). - Log the enhancement too-long/too-large 400 rejections for parity with other error paths. - Use a WebM-specific conversion error message (parity with the internal clone route) and node:crypto createHash for buffer hashing. Adds tests for the SSRF rejection and the credits-before-enhancement guard. https://claude.ai/code/session_01JEyH3uDy8xL49NnUygrDqz
|
The latest updates on your projects. Learn more about Argos notifications ↗︎
|
Summary
Adds a new
/api/v1/cloneendpoint that enables users to clone voices from reference audio and synthesize new speech with the cloned voice. The implementation supports two cloning providers (Mistral Voxtral for supported locales, Replicate for others), includes optional reference audio enhancement via denoising, and integrates with the existing billing and credit system.Changes
POST /api/v1/clone): Accepts reference audio as either a public URL or base64-encoded data, synthesizes speech with the cloned voice, and returns the generated audio URLlib/clone/clone-service.ts): Core voice cloning logic including provider resolution, audio validation/processing, Mistral Voxtral and Replicate integration, and reference audio enhancement billinglib/clone/reference-audio-enhancement.ts): Optional denoising/enhancement of reference audio before cloning (best-effort, falls back gracefully)VoiceCloneRequestSchemaandVoiceCloneResponseSchemawith proper validation and OpenAPI documentationHow to test
POST /api/v1/clonewith valid reference audio (URL or base64) and input text; verify 200 response with generated audio URL and correct credit deductionlocale: "en"(uses Mistral) andlocale: "ja"(uses Replicate); verify correct provider is invokedenhance_reference_audio: true; verify additional credits are deducted and enhancement metadata is loggedpnpm test api-v1-clone.test.ts clone-service.test.tsto verify all unit and integration tests passScope
Checklist
pnpm run fixallpnpm run type-checkNotes for reviewers
Key implementation details:
/api/clone-voiceroute's audio normalization logic but operates on buffers instead of File objects to support both URL and base64 inputshttps://claude.ai/code/session_01JEyH3uDy8xL49NnUygrDqz