fix(serializers): set MsgPackSerializer DEFAULT_ENCODING to None - #1079
Conversation
MsgPackSerializer inherited DEFAULT_ENCODING = "utf-8" from BaseSerializer, causing backends (Memcached, Valkey/Redis) to decode the raw msgpack bytes with UTF-8 before passing them to loads(). Since msgpack serializes to arbitrary binary, this produced UnicodeDecodeError on any non-trivial value. PickleSerializer correctly sets DEFAULT_ENCODING = None so backends skip the decode step and pass raw bytes. Apply the same fix to MsgPackSerializer. The raw parameter in loads() is updated accordingly: treat None and "utf-8" as equivalent (both return decoded strings), so the default round-trip behaviour is unchanged while backends now receive the correct encoding hint. Fixes aio-libs#834
|
Can you cherry-pick the tests from #836? |
|
Done in I added the
Local verification: I also tried the full new acceptance selection locally, but the |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1079 +/- ##
=======================================
Coverage 98.84% 98.85%
=======================================
Files 32 32
Lines 3560 3579 +19
Branches 125 125
=======================================
+ Hits 3519 3538 +19
Misses 41 41
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Problem
MsgPackSerializerinheritsDEFAULT_ENCODING = "utf-8"fromBaseSerializer. Backends that decode stored bytes before passing them to the serializer (Memcached, Valkey/Redis) therefore callvalue.decode("utf-8")on raw msgpack-packed bytes. Since msgpack produces arbitrary binary, this raisesUnicodeDecodeErrorfor virtually any non-trivial value.Minimal reproduction:
Fixes #834.
Fix
PickleSerializeralready handles this correctly by settingDEFAULT_ENCODING = None, which tells backends to skip the.decode()step and pass raw bytes straight toloads(). Apply the same fix toMsgPackSerializer.The
rawparameter passed tomsgpack.loads()is updated so that bothNoneand"utf-8"produceraw=False(Python strings), preserving the existing round-trip behaviour while the encoding hint to backends is corrected.This pull request was prepared with the assistance of AI, under my direction and review.