Implement batch lookup methods for apis that don't natively support it

[amogus07]

I had been using httpx for api requests in beets-vocadb. But since it hasn't had a new release since December 2024, I began looking for alternatives. At first, I considered it's recent fork httpxyz, but it doesn't seem to be compatible with httpx-retries, so I went for niquests instead. While reading the documentation, I found out it has supports a multiplexed connection. Using that, I was able to implement batch id lookups in beets-vocadb despite the API not natively supporting it. See the these commits for the relevant changes.

❯ beet stats data_source:UtaiteDB
Tracks: 666 # (55 of which are singletons)
Total time: 1.6 days

Approximate total size: 17.2 GiB
Artists: 382
Albums: 100
Album artists: 74
❯ beet stats data_source:VocaDB
Tracks: 2966 # (15 of which are singletons)
Total time: 1.1 weeks

Approximate total size: 79.2 GiB
Artists: 1353
Albums: 442
Album artists: 140

before:

Benchmark 1: beet udbsync -p
  Time (mean ± σ):     84.681 s ±  8.388 s    [User: 6.361 s, System: 0.349 s]
  Range (min … max):   71.286 s … 92.790 s    5 runs
Benchmark 1: beet vdbsync -p
  Time (mean ± σ):     601.525 s ± 130.134 s    [User: 20.574 s, System: 1.279 s]
  Range (min … max):   433.131 s … 789.923 s    5 runs

after:

Benchmark 1: beet udbsync -p
  Time (mean ± σ):     60.454 s ±  3.910 s    [User: 6.052 s, System: 0.421 s]
  Range (min … max):   55.998 s … 66.567 s    5 runs
Benchmark 1: beet vdbsync -p
  Time (mean ± σ):     420.572 s ± 67.161 s    [User: 20.368 s, System: 1.018 s]
  Range (min … max):   342.829 s … 490.916 s    5 runs

In my implemtation, there's still some overhead because in certain cases, additional individual requests need to be sent, but it's still a pretty significant improvement. Now, I wish the same was implemented in other plugins too, especially the lyrics plugin, which seems to be the main bottleneck while importing.

[amogus07]

@beetbox/maintainers would someone mind having a look?

[semohr]

I'm not sure I follow your suggestion. In beets core, all HTTP requests go through the requests library, and replacing it isn't really feasible.

That said, requests doesn't support multiplexing, but that doesn't mean we don't parallelize. We already parallelize operations like candidate lookups and ID-based metadata plugin lookups at a higher level (each plugin runs in a thread). Or the import pipeline itself can also be run in threads.

In practice, parallelizing requests inside a single plugin often doesn't make sense due to API rate limits. Besides, plugins are already free to use whatever HTTP framework they want for their own requests 🤔

[amogus07]

true, I didn't really think about that since I never experienced any 503 errors or the like with VocaDB. Other apis like MusicBrainz are probably more restrictive.