Skip to content

fix(serializers): set MsgPackSerializer DEFAULT_ENCODING to None - #1079

Merged
Dreamsorcerer merged 2 commits into
aio-libs:masterfrom
gaoflow:fix/msgpack-serializer-encoding
Jun 28, 2026
Merged

fix(serializers): set MsgPackSerializer DEFAULT_ENCODING to None#1079
Dreamsorcerer merged 2 commits into
aio-libs:masterfrom
gaoflow:fix/msgpack-serializer-encoding

Conversation

@gaoflow

@gaoflow gaoflow commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Problem

MsgPackSerializer inherits DEFAULT_ENCODING = "utf-8" from BaseSerializer. Backends that decode stored bytes before passing them to the serializer (Memcached, Valkey/Redis) therefore call value.decode("utf-8") on raw msgpack-packed bytes. Since msgpack produces arbitrary binary, this raises UnicodeDecodeError for virtually any non-trivial value.

Minimal reproduction:

import asyncio
from aiocache import MemcachedCache
from aiocache.serializers import MsgPackSerializer

async def test():
    cache = MemcachedCache(serializer=MsgPackSerializer())
    await cache.set("key", {"k": "v"})
    await cache.get("key")  # UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 …

asyncio.run(test())

Fixes #834.

Fix

PickleSerializer already handles this correctly by setting DEFAULT_ENCODING = None, which tells backends to skip the .decode() step and pass raw bytes straight to loads(). Apply the same fix to MsgPackSerializer.

The raw parameter passed to msgpack.loads() is updated so that both None and "utf-8" produce raw=False (Python strings), preserving the existing round-trip behaviour while the encoding hint to backends is corrected.


This pull request was prepared with the assistance of AI, under my direction and review.

MsgPackSerializer inherited DEFAULT_ENCODING = "utf-8" from BaseSerializer,
causing backends (Memcached, Valkey/Redis) to decode the raw msgpack bytes
with UTF-8 before passing them to loads(). Since msgpack serializes to
arbitrary binary, this produced UnicodeDecodeError on any non-trivial value.

PickleSerializer correctly sets DEFAULT_ENCODING = None so backends skip
the decode step and pass raw bytes. Apply the same fix to MsgPackSerializer.

The raw parameter in loads() is updated accordingly: treat None and "utf-8"
as equivalent (both return decoded strings), so the default round-trip
behaviour is unchanged while backends now receive the correct encoding hint.

Fixes aio-libs#834
@Dreamsorcerer

Copy link
Copy Markdown
Member

Can you cherry-pick the tests from #836?

@gaoflow

gaoflow commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Done in affebc7.

I added the MsgPackSerializer acceptance tests from #836 for:

  • set / get
  • add / get
  • multi_set / multi_get

Local verification:

uv run --with-requirements requirements-dev.txt --with-editable . python -m pytest tests/ut/test_serializers.py -q
# 61 passed

uv run --with-requirements requirements-dev.txt --with-editable . python -m pytest tests/acceptance/test_serializers.py::TestMsgPackSerializer -q -m 'not memcached'
# 36 passed, 18 deselected

uv run --with-requirements requirements-dev.txt --with-editable . python -m pytest tests/ut/test_serializers.py tests/acceptance/test_serializers.py::TestMsgPackSerializer -q -m 'not memcached'
# 97 passed, 18 deselected

uv run --with black --with-editable . python -m black --check aiocache/serializers/serializers.py tests/ut/test_serializers.py tests/acceptance/test_serializers.py
# passed

uv run --with-requirements requirements-dev.txt --with-editable . python -m flake8 aiocache/serializers/serializers.py tests/ut/test_serializers.py tests/acceptance/test_serializers.py
# passed

git diff --check
# passed

I also tried the full new acceptance selection locally, but the memcached_cache parametrization fails because no memcached service is running on 127.0.0.1:11211. The project contributing docs note that acceptance/functional tests need Docker/backend services, so I verified the runnable memory/valkey subset locally and left the memcached coverage for CI.

@codecov

codecov Bot commented Jun 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.85%. Comparing base (43f6a24) to head (affebc7).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1079   +/-   ##
=======================================
  Coverage   98.84%   98.85%           
=======================================
  Files          32       32           
  Lines        3560     3579   +19     
  Branches      125      125           
=======================================
+ Hits         3519     3538   +19     
  Misses         41       41           
Files with missing lines Coverage Δ
aiocache/serializers/serializers.py 95.52% <100.00%> (+0.06%) ⬆️
tests/acceptance/test_serializers.py 100.00% <100.00%> (ø)
tests/ut/test_serializers.py 100.00% <100.00%> (ø)

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 43f6a24...affebc7. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Dreamsorcerer
Dreamsorcerer merged commit ae5948b into aio-libs:master Jun 28, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnicodeDecodeError when using MsgPackSerializer

2 participants