Are Duplicate Scrobbles Ignored?

[FestiveByteRider]

One of my users listens to music using Music Bee on PC & Musicolet on Android, the issue is that Musicolet is completely offline and as such has no scrobbling functionality at all. The other day I saw Pano Scrobbler mentioned here and it's the perfect solution. When I was looking at Pano I saw that I can use a custom Last.fm/Listenbrainz endpoint, if I was to setup Pano using that and have the data go from Pano to both a Koito instance and last.fm, would the other last.fm source for the user for the Music Bee listens end up causing duplicates? Like would MS ignore the scrobbles it sent to Last.fm that it then pulls after sending? I like fancy workflows so would love to set it up as Music Bee -> Last.FM -> Multi-Scrobbler -> Koito & Musicolet -> Pano Scrobbler -> Multi-Scrobbler -> Last.FM & Koito assuming that doing so wouldn't cause duplicate entries from MS pulling and pushing scrobbles to the same account. If it will I'll setup Pano Scrobbler to go to Last.FM and just pull both from there instead. Thought I'd check before testing so I don't risk messing up my user's stats and having her yell at me lol :>. Thanks!

[FoxxMD]

The unsatisfying answer is yes BUT it's only guaranteed to be effective when the scrobbles are coming from the same source. EX of guaranteed dupe detection:

• Spotify scrobbles get backlogged (MS discovers the last 50 plays) -> you restart MS -> MS finds the same 50 -> detects all as dupes
• Pano/Webscrobbler uses the Listenbrainz endpoint Source and (for some reason) accidentally sends the same scrobble twice -> detected as dupes

And actually your scenario would probably be okay if it was Listenbrainz instead of Last.fm, because Listenbrainz has unique identifiers for scrobbles and individual tracks + they have normalized data.

Last.fm is the real issue here. I need to document this somewhere more official...There's an excerpt here under "Matching Scrobbles from Last.fm".

The problem is that Last.fm will happily accept almost any scrobble data you throw at it. And then it can decide, opaquely, to match or modify that scrobble to some more "official" data it has server-side. This could be modifying the artists, title, or album sent in the scrobble. It can also mean modifying the mbid (unique identifier) or adding what it thinks is the correct one. There's nothing in the api response from LFM that identifies it has done any of this. And it's not super great at being right about these modifications.

What this means for you, the user, is that this is not an uncommon scenario:

• You scrobble Artist Foo, Artist Bar - My Cool Track to Last.fm via Pano
  • Last.fm modifies the data to Artist Foo - My Cool Track (ft. Artist Bar)
• You also scrobble Artist Foo, Artist Bar - My Cool Track to Multi-scrobbler via Pano LFM endpoint Source
  • MS forwards this to Koito as-is
• Soon after, MS polls Last.fm Source for new scrobbles and finds Artist Foo - My Cool Track (ft. Artist Bar)
  • It queries Koito to see if this was scrobbled
  • It sees that a scrobble did happen at almost the same time BUT the track titles and artist strings are just different enough that it decides its not a duplicate
  • MS forwards the scrobble to Koito

This dupe detection is fooled even more when Last.fm decides to add or modify the mbid for the scrobble data since this is supposed to be a normalized, unique identifier.

MS does a ton of work to try to detect duplicates:

• compares artist, album, and track strings using word-order invariant comparison algorithms (weight average of 3 separate algos)
• uses hints from real-time monitoring like time ranges listened for
• does temporal comparisons of the scrobble datetime based on known granularity of sources

But even with all that there is no way to 100% guarantee it can always detect duplicates like the example above. (Though, there is a much better chance if you use the musicbrainz stage to normalize scrobble data.)

My advice is: don't cross the streams

Some solutions for your scenario:

YOLO with Musicbrainz

Just YOLO the scrobble with all the normalization you can: Use the musicbrainz stage with basic,album,artist search methods, as mentioned in the last.fm matching section, to try to match and normalize the garbage data last.fm sends. The chance for good matches (and therefore dupe detection) is higher if the music the user listens to is more mainstream and there is less chance for ambiguity in matching.

The downside of this is that, obviously, there is still a chance of dupes.

Secondary Last.fm Account

Some users use additional last.fm account(s) to isolate scrobbles for a specific device that can't scrobble otherwise. In this approach, that last.fm account is used only for activity from that one user/device so there's no chance of duplicate scrobbles getting forwarded b/c they are monitoring by some other service/MS source.

The downside of this is that if the user uses their main last.fm account socially, for Now Playing, or just really cares about having it all in one place, its now fractured across multiple accounts. If they don't care about any of that then this is the only sure-fire way to avoid dupes for your scenario.

[FoxxMD]

To add, here are all the tests/scenarios MS tries to cover for dupe detection:

• String normalization tests
• Parsing artists from ft. style joiners and multi-artist strings
• Dupe and unique scrobble detection scenarios

[FoxxMD]

I've added a dedicated page in the documentation about duplicate detection and best practices for configs. Goes in to more detail on everything mentioned above and has f a n c y diagrams as well.

[FestiveByteRider]

Awesome! Thanks for that I'll give it a read. :)