[BUG]: Unable to replicate official VEVO2 examples

[observer678]

Describe the bug

I've checked your official website for vevo2 (https://versasinger.github.io/) and I can't replicate most of the examples shown there. For this issue I will focus on accent transfer, which I am also unable to reproduce.

How To Reproduce

Run this short snippet that is taking indian accent as style input and mandarine voice as timbre reference. The output has a young guy's voice with little to no accent transfer. (Place this in the root of the project)

import os
import sys

PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
sys.path.insert(0, PROJECT_ROOT)

import torch
import whisper
import librosa
import numpy as np
from huggingface_hub import snapshot_download
from models.svc.vevo2.vevo2_utils import Vevo2InferencePipeline, save_audio

def transcribe(audio_path):
    audio = librosa.load(audio_path, sr=16000)[0].astype(np.float32)
    return whisper.load_model("medium").transcribe(audio)["text"]

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
local_dir = snapshot_download(
    repo_id="RMSnow/Vevo2",
    repo_type="model",
    local_dir=os.path.join(PROJECT_ROOT, "ckpts/Vevo2"),
    resume_download=True,
)

pipe = Vevo2InferencePipeline(
    prosody_tokenizer_ckpt_path=os.path.join(local_dir, "tokenizer/prosody_fvq512_6.25hz"),
    content_style_tokenizer_ckpt_path=os.path.join(local_dir, "tokenizer/contentstyle_fvq16384_12.5hz"),
    ar_cfg_path=os.path.join(local_dir, "contentstyle_modeling/posttrained/amphion_config.json"),
    ar_ckpt_path=os.path.join(local_dir, "contentstyle_modeling/posttrained"),
    fmt_cfg_path=os.path.join(local_dir, "acoustic_modeling/fm_emilia101k_singnet7k_repa/config.json"),
    fmt_ckpt_path=os.path.join(local_dir, "acoustic_modeling/fm_emilia101k_singnet7k_repa"),
    vocoder_cfg_path=os.path.join(local_dir, "vocoder/config.json"),
    vocoder_ckpt_path=os.path.join(local_dir, "vocoder"),
    device=device,
)

target_text = (
    "I've been a silent spectator, watching species evolve, empires rise and fall. "
    "But always remember, I am mighty and enduring. Respect me and I'll nurture you; "
    "ignore me and you shall face the consequences."
)
style_ref = os.path.join(PROJECT_ROOT, "voices/indian_accent_english.wav")
timbre_ref = os.path.join(PROJECT_ROOT, "voices/mandarin_female.wav")

style_text = transcribe(style_ref)
print(f"Style transcription: {style_text}")

gen = pipe.inference_ar_and_fm(
    target_text=target_text,
    style_ref_wav_path=style_ref,
    style_ref_wav_text=style_text,
    timbre_ref_wav_path=timbre_ref,
    use_prosody_code=False,
)

output = os.path.join(PROJECT_ROOT, "output/indian_style_mandarin_timbre_tts.wav")
os.makedirs(os.path.dirname(output), exist_ok=True)
save_audio(gen, output_path=output)
print(f"Saved -> {output}")

Expected behavior

The output should have the mandarine female voice with indian accent.

Screenshots (Outputs)

my_output.wav

correct_output.wav

Environment Information

• Operating System: Ubuntu 22.04.5 LTS
• Python Version: Python 3.11.10
• Driver & CUDA Version: Driver 550.127.05 & CUDA 12.4