List VSTs as instruments

[sqrvrt]

Closes #1837
Closes #4585

Features:

• Automatically sorts VSTs into instruments and effects
• Hides VSTs that fail to load
• Shows real VST names instead of truncated DLLs
• Deduplicates VSTs with matching IDs (todo: maybe also append version there)
• Scans system and wine dirs in addition to the LMMS user-defined dir
• Stores metadata in a simple persistent cache, so the slowdown only occurs on first launch

Status:

• VST list is built
• Vestige is adapted to use the vst list instance
• VstEffect is adapted to use the vst list instance
• file hash is checked to avoid reloading the same VST to get metadata
• metadata is cached in a file that persists between loads
• make reloading optional (partial: functionality exists, but nothing uses it)
• make VstEffect and Vestige use real plugin name when added via subplugin features
• use plugin icons (probably separate PR; unclear how to get them)

Questions:

• Split off the sub-plugin features of vestige into a separate "VST" category. Looks like this needs another plugin registration? Unclear how to do. Looks like this requires a separate .so at the moment, so not that feasible.

Cryptic issues:

• Theming on Vestige instruments occasionally breaks.

broken theming in question

what particularly useful things VST preloading could enable:

• get real name of the VST and other metadata like author
• know if VST is an instrument or an effect
• potentially use VST IDs instead of paths in saved projects, meaning that as long as the VST or a symlink to it is in the VST folder it can always be recognized regardless of the specific path.
• possibly more idk

also i don't really know the security implications of loading random DLLs from a dir in order to probe for whether they're a VST, but like mehhhh you can already orchestrate LMMS loading one if you sneakily edit a project file or something, so i doubt it changes that much, right..?

[sakertooth]

I'm not ready to take a deep dive into all the issues with the PR currently, but we should be able to determine if a VST is an instrument or an effect.

VST2 has a flag that is currently being unused to determine this:

lmms/include/aeffectx.h

Line 104 in 83313e7

constexpr int effFlagsIsSynth = 1 << 8; // currently unused

I don't have much experience with VST2, but hoping you can figure out something here. @messmerd would probably know a lot better since they've been working with the plugin API quite a bit.

[sqrvrt]

we should be able to determine if a VST is an instrument or an effect.

VST2 has a flag that is currently being unused to determine this:

Oh, neat! I was thinking to just classify something as an effect if it has >0 inputs, but this is even better. Still though, basic loading is needed which pulls behind complete VST functionality in how it's currently implemented.

[sqrvrt]

Also i don't know if i don't know enough about the current plugin API (since i read zero docs, i mostly just tried to make sense of other instances written before me), but it looks like additional metadata such as icons, author, etc. can only be specified per-plugin (i.e. Vestige in this case) instead of per subplugin, which VSTs could make use of.

if i'm correct, maybe that could be taken into account while making the new API :D

[messmerd]

VST2 has a flag that is currently being unused to determine this:

Here's some more information on effFlagsIsSynth. Apparently it was introduced in VST 2.0, so there might not be a way to categorize VST 1.x plugins correctly. I'm not sure how many of those are in existence though. EDIT: VST instruments were introduced in VST 2.0, so all VST 1.x plugins are effects.

Implementing a VST2 discovery system would be more difficult than other plugin formats since there is no unique file extension to look for. If you attempt to load a .dll/.so as a VST you'd know the answer, but that's ineffecient - especially for Windows VSTs on Linux which need to launch via Wine.

I've thought of two main approaches to VST discovery.

VST discovery approach 1

Fast but incomplete, with lazily-loaded and cached metadata

This approach should be very fast because it avoids loading and querying VST .dll/.so files at all until the user tries to instantiate a VST plugin.

To determine whether a .dll/.so is a VST, we could probably just use LoadLibrary+GetProcAddress or dlopen+dlsym to see if it exports a symbol called VSTPluginMain (for VST 2.x) or MAIN (for VST 1.x on Windows) as seen here, but on Linux we can't load a Windows VST this way without using Wine and RemoteVstPlugin which are slow, so we might need a way to directly read the PE symbol table and look for an exported VSTPluginMain or MAIN symbol. Or if we're comfortable with some false positives, we could simply assume any .dll/.so encountered in the VST directory is a VST.

Once we know a file is a VST, we could use its filename as a placeholder for the VST name and that would give us just barely enough information to display it to users in the new LV2-like sidebar listing for VSTs.

If we went this route, the benefit is that it would be pretty fast and the entire VST listing would already be fully populated as soon as the user is able to interact with the LMMS UI - similar to LV2 plugins. The downside is the user would have to try to instantiate a plugin before we could learn the plugin's display name (as opposed to filename), its category (instrument/effect), and other info. If the user gets the category wrong, a dummy plugin would be created.

However, if we created a config file for VSTs (or used .lmmsrc.xml for this purpose), we could keep track of the metadata for all VSTs that have been loaded before, so the next time LMMS is launched, it could use that information to correctly categorize and name the plugin.

VST discovery approach 2

Complete but slow, with eagerly-loaded and cached metadata

A lot of other DAWs go this route. All VSTs are discovered AND loaded/queried at startup in order to correctly categorize all VSTs from the start. In order for LMMS to be usable without needing to potentially wait several minutes before VST discovery completes, it should perform VST discovery in a background thread and add new plugins to the VST sidebar as it finds them.

The benefit of this approach is that all plugins are correctly categorized and named from the start. The downside is that it's asychronous, so it may be a bit trickier to implement. If the user loads a project with a VST in it, it would have to function correctly even if VST discovery hasn't finished yet. It also would take longer to complete, though like approach 1, we could store metadata about all the discovered plugins so we only need to load and query the plugins the first time LMMS is launched. All subsequent launches could rely on the stored metadata and only load/query any new plugins that haven't been seen before.

VST discovery approach 3

Hybrid of approaches 1 and 2

We could perform approach 1 to have a complete listing of all VSTs available at startup, but then in a background thread, we could perform approach 2 in order to fix the categorization and names of the plugins.

This is probably just overcomplicating things.

---

I'm in favor of the 2nd approach since it's more robust and the system we come up with for asynchronously discovering plugins could also be applied for discovering CLAP presets, for example.

[sqrvrt]

In my mind i have a potential good non-async method 2 (edit: this was actually method 1 but with preloading the VSTs on first encounter). Basically have a cache that lists all DLL hashes/paths, marking whether they're a VST and if they are, listing all their metadata. On startup perform a lookup over paths like this currently does, load the ones missing from the cache and only display those that actually exist.

Though from my testing, when i commented out the editor creation I don't think the slowdown was that noticeable at all. Granted, i don't rely on VSTs that heavily to begin with, so to get the concrete picture it might be better to make a build and send it over to people to stress test.

[sakertooth]

How slow is it really to just get the AEffect* structure and check that unused flag without all the baggage currently in here? That's probably how VST2 was intended to be loaded at the time anyways (i.e, not even with sandboxing)

Edit: Okay, I am now understanding this as being the aforementioned VST discovery approach number 1.

[sqrvrt]

How slow is it really to just get the AEffect* structure without all the baggage currently in here?

probably not that slow, the problem is that either VstBase needs to get some medium alterations to load the VST in a (mostly) minimal fashion, or the new plugin needs to bundle a minimal functionality with it, which especially considering all the Wine baggage would be pretty wonky to set up and i'm not sure duplicate behavior is that good of an idea to begin with.

I think a single bool to control the minimalism would suffice, somehow even without the editor they still are able to reproduce sound so ideally i want to cut that too.

[sqrvrt]

(...) Or if we're comfortable with some false positives, we could simply assume any .dll/.so encountered in the VST directory is a VST.

I believe there's a similar mechanism for detecting VSTs already implemented in RemoteVstPlugin, and IIRC it also runs before the DLL is loaded

[sqrvrt]

i think i also got how to appropriately share the loading between Vestige and VstEffect, i could just split off the thing into a new plugin, mark effect/instrument in the keys, and based on the value return VstEffect or a Vestige pointer from lmms_plugin_main.

Alternatively i could also register the plugin twice - once for effects an once for instruments, mainly to allow different metadata and icons but that's a hack and probably not that great.

[sqrvrt]

It looks like making a new plugin isn't as nice as i thought, thinking about doing a static instance of a class that holds a list of VSTs in VstBase, and referencing that from VstEffect and Vestige

[sqrvrt]

I think i've done most of what i can do at the moment without any input from other people

Windows build fails at the moment, i don't know why. I'd be glad if someone helped to fix it.

I don't know where to get the VST icon. I was planning that i'd just save their pixmaps into a cache folder using VST ID as part of the file name. I've not been able o find a single reference to icon loading in the entire codebase. If someone knows how i'd be happy to implement it.

I don't really understand where save/load of presets/projects that saves the path happens, so i won't have any guarantee relative paths are used either without further input.

It would be neat to have rescan and yeet cache buttons somewhere, that requires UI work. It also requires going outside the boundaries for Plugin APIs which is very meh. I guess they could be put into settings near the VST dir option? don't know

It also doesn't seem possible with current plugin API to have Vestige and VST list appear as separate things, using a second plugin decl does nothing both with and without changing the name.

[regulus79]

Windows build fails at the moment, i don't know why. I'd be glad if someone helped to fix it.

Could this be because Descriptor or Key does not have LMMS_EXPORT? I've gotten very similar linker errors when trying to qt connect functions which aren't exported/visible to plugins.

[yohannd1]

It would be neat to have rescan and yeet cache buttons somewhere, that requires UI work. It also requires going outside the boundaries for Plugin APIs which is very meh. I guess they could be put into settings near the VST dir option? don't know

I was thinking of placing them in the plugin list but honestly that might get a bit messy, and putting it in the settings makes sense, but it feels a bit confusing to put it in the folders section (not that any other sections make sense either...).

Also, I'm not sure if I understood - are you planning to implement that in this PR? If not, I could try helping with that.

[sqrvrt]

It would be neat to have rescan and yeet cache buttons somewhere, that requires UI work. It also requires going outside the boundaries for Plugin APIs which is very meh. I guess they could be put into settings near the VST dir option? don't know

I was thinking of placing them in the plugin list but honestly that might get a bit messy

I'm not sure about yeet cache button, there's probably very little it would actually do as it should only be needed in cases where cache is corrupted or somehow has matching hashes but bad info (could occur if dll decided to die but only once)

The rescan button feels a bit nicer though. I did think about putting it in the category collapser, but that either requires changes to the plugin descriptor (=API at all), or special casing for vst list specifically which is not pretty. Plus the list is shared between instrument and effect listings, so a similar thing would need to be implemented there as well.

The SubPluginFeatures::listSubPluginKeys actually gets called on every redraw of the instrument list, which could be interesting except that it's a lot more often than you'd think and having a deep directory tree might cause stuttering when hovering over it. I might actually need to optimize it more since it currently constructs a vector of instruments on each call. Or not, since there's at least map construction happening every time anyway.

The third, nuclear option would be having something like inotifywait on all the dirs in a separate process so that whenever file structure changes the list gets updated. That requires threading work and i know nothing about it. Could be the thing that gets me into it though.

Having the button in settings is more of a compromise to "at least you don't have to restart the DAW".

[sqrvrt]

Symlinks don't really exist on Windows

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/mklink

To be completely honest i thought windows was sane and a "shortcut" was a symbolic link. It turns out that it is not, and MS decided to reinvent the wheel for little-to-no reason. Worst comes to worst i could make a simple .lnk parser, i guess.

LMMS could definitely scan the "correct" directories for VSTs

I was actually planning to do that. However on linux I know where VSTs are stored, in particular /usr/lib{,64}/vst. On windows i'm not quite sure and there's probably a tendency for each application to just put the file somewhere in C:/Program Files/ and hope someone deals with it. If there are dedicated VST paths on windows i'd be happy to know.

[sqrvrt]

Also a little question a about initializing — currently the scan is done whenever first ::inst() is called which is usually when instruments tab is drawn, but can really happen whenever. The drawback of this could be a random long stutter, and also the fact there are no status updates on the splash.

What's the best thing to do here? Should i only enable loading plugins when done explicitly? And then initialize it in MainWindow? It's probably best to also include currently loading plugin on the splash so it's not just "hanging for 20 minutes" but it's visible that it's doing something.

The above would also add the dependency on GUI specifically, where it was before just on base Qt for QString interop + some convenience functionality like file hash and OS-specific cache dir.

The question is more about how do i conditionally load the thing within MainWindow? Just put a construction like this?

#ifdef LMMS_HAVE_VST
#include "VstList.h"
#endif

and later also ifdef the init?

[sqrvrt]

Hm, that's pretty significant. so it turns out you can't actually hook into MainWindow because it doesn't see the header files. I guess the behavior stays?

[sqrvrt]

@messmerd i think an entry point for such plugin scanning operations would be nice in the new API

[sqrvrt]

Just confirmed, VSTs still get saved with their relative paths thanks to PathUtil.h, so this should be ready-ish. I'd love to add ID lookup as a fallback later (i'm starting to run out of free time), but even in this state this should be somewhat acceptable for review & merge.

[rdrpenguin04]

@sqrvrt

I was actually planning to do that. However on linux I know where VSTs are stored, in particular /usr/lib{,64}/vst. On windows i'm not quite sure and there's probably a tendency for each application to just put the file somewhere in C:/Program Files/ and hope someone deals with it. If there are dedicated VST paths on windows i'd be happy to know.

The "canonical" path for VST2 plugins is C:\Program Files\Steinberg\VSTPlugins; you'll see it in most Windows VST installers. As plenty of the VSTs I use on Linux are actually Windows VSTs, it would also be helpful to scan $HOME/.wine/drive_c/Program Files/Steinberg/VSTPlugins for the same reason

[sqrvrt]

Yeah forgot to mention, i found it already. Though i'm curious, do they also get separated into the Program Files (x86) variation for the 32-bit ones?

[rdrpenguin04]

They're only stored in Program Files from what I can tell; I don't have proof or a source for that though.

[Spacemagehq]

I had a thought when looking at this from an outside user point of view people probably won't know what a vestige is i was thinking on the sidebar instead of naming that section under LMMS plugins vestige name it "outside plugins" or "downloaded plugins" something to let the user know "this is where my vsts or other plugins are"

[sqrvrt]

I had a thought when looking at this from an outside user point of view people probably won't know what a vestige is i was thinking on the sidebar instead of naming that section under LMMS plugins vestige name it "outside plugins" or "downloaded plugins" something to let the user know "this is where my vsts or other plugins are"

That is mostly a limitation of the current plugin system — the only way to do so would be to completely replace Vestige display name, which i'm reluctant to do.

[sqrvrt]

So i would like to orphan this to someone who actually knows what they're doing. The core thing works, but the VST loader needs tweaks i'm nowhere near qualified to understand how to do. See this chain for example, TLDR windows needs a better way to test/load VST DLLs and also to skip the init on plugins that do weird stuff like make you choose a file immediately on load.

[sqrvrt]

Ok new plan since this clearly isn't moving anywhere — would this be merge-worthy if i clean up a little and put most of the functionality behind an (off by default) settings toggle? e.g. "Scan for VSTs" or something.

I.e. VstSubPluginFeatures would just return an nullptr which would make Vestige show up as a regular lmms plugin like it did before. That would also affect VST effects which i guess would be a downgrade(?) but it would unify how both work. This would require adding a file select dialog to VST effect but probably isn't that big of a deal :clueless:

Alternatively instead of Vestige-style it could also be VstEffect-style, although there it would ideally require two separate user dirs for effects and instruments in order to keep the list manageable.

[RainbowShatter]

woah a settings toggle would be dope