I want to say this is because I've effectively spent most of my existence the past month or so around this, but even then I am not an expert.finetuning is a bit of an art, as I seem to get decent results with my go-to defaults, while it seems for others it's not up to par.The base model is still impressive, for what it's worth. zero shot is definitely tied to how well the base model is, and the base model isn't quite up to par, and if the base model is going to be replaced, why bother sticking with TorToiSe when you can just devote the time and effort into a different system.even for a specific voice, it took quite a while to get something even sounding like English (this is like an artifact from Tau Ceti V).Īs for my thoughts on TorToiSe after everything from this being a silly rentry to using it, to now:.requires two models, the AR and the NAR (despite it generating audio fine during evaluation with just the AR, but I might be neglecting something and need to read the paper again).I can spend some money on renting GPUs and trying to create a general model, but it's grim when I can't get training to work on a paperspace instance.gauging how well VALL-E performs is pretty much up to chance, as the demos don't seem all that great desu.no base model, so anyone wanting to try it out need to invest in a dataset, the compute, and the time.I'm sure there's a config setting to do so. right now, I don't see an easy way to finetune models in the future.this also appeared when I tried the other implementation, so it might just be a CUDA issue, so ROCm only for me.doesn't seem to want to train under a CUDA backend (some unhelpful error message).there's apparently support for inferencing on Windows, but anything that requires MSVC is major CBT.

Linux only, as training with DeepSpeed is Linux only.there's an autotuner within DeepSpeed, but it doesn't seem to want to work.probably even more sensitive than training with DLAS, as it will OOM on me easily without conservative batch sizes.it might work, but it can't do it when resuming.leveraging ZeRO seems to require editing the code itself that handles passing some configurations to DeepSeed.no BitsAndBytes to save my hide, so it's quite the VRAM hog.However, I have some qualms with it so far: validation/evaluation is actually meaningful, as it generates audio against the model you can actually listen to for gauging how well it's chugging along.the defaults seem to be sane enough to rely on, so no need to squabble around with the LR and scheduler or gradient accumulation sizes.training configuration is pretty much just setting the max iterations, batch size, and save/eval rates.there's a '''REPL''' to pass commands through stdin to do stuff like save, report metrics, and quit (saves on quit too).it leverages DeepSpeed, so any development on it will improve the training process too.The training process is pretty intuitive too: similar to DLAS, I can just spawn a process to for training and parse output.the default phonemizes to ARPABET using g2p-en, which ties it only to English.I can phonemize the input text myself as IPAs.requires some nasty dependencies (lhotse + k2 + icecat just for dataset preparation).it's nice and clean, and requires little extra dependencies (phonemizer + a backend, like espeak-ng).I'm backing this implementation as my VALL-E implementation: metric grabbing almost works, but I need to reparse the log files to fetch previous metrics like I did with DLAS.currently, only training is integrated, as I need to get models cobbled together to inference against.I might have to keep it this way, as not every option will also carry over for VALL-E.you can switch to it by passing -tts-backend='vall-e'.As I may have hinted with my not-so-subtle commits, I'm working towards getting VALL-E integrated as an alternative TTS backend:

Git set upstream to forked repo