Background: my TLS certificate for mosquitto had expired and I didn't notice. My mobile clients have accumulated a few weeks worth of data which they're submitting all at once.
While processing a large backlog of data, owntracks_recorder uses 100% CPU and takes a long long time to process messages… to the point where mosquitto starts discarding messages because its queue is full.
One of the factors involved seem to be the call to is_newer_than_last for every single message. This calls last_users which, if I understand correctly, scans the entire directory tree of last. This involves multiple filesystem operations per message to continuously determine the same thing.
I think it should be feasible to read STORAGEDIR/last/{user}/{device}/{user}-{device}.json directly instead of walking the entire tree?
Background: my TLS certificate for mosquitto had expired and I didn't notice. My mobile clients have accumulated a few weeks worth of data which they're submitting all at once.
While processing a large backlog of data, owntracks_recorder uses 100% CPU and takes a long long time to process messages… to the point where mosquitto starts discarding messages because its queue is full.
One of the factors involved seem to be the call to
is_newer_than_lastfor every single message. This callslast_userswhich, if I understand correctly, scans the entire directory tree oflast. This involves multiple filesystem operations per message to continuously determine the same thing.I think it should be feasible to read
STORAGEDIR/last/{user}/{device}/{user}-{device}.jsondirectly instead of walking the entire tree?