Skip to content

Fix issue with encoding of arrays/list when cache is created #136

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
agrouaze wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Fix issue with encoding of arrays/list when cache is created #136

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4eb72cb
fix #135
agrouaze May 26, 2026
917baef
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 26, 2026
a3ad799
linter
agrouaze May 26, 2026
50a017c
Merge branch 'fix135' of https://github.com/umr-lops/xarray-ceos-alos…
agrouaze May 26, 2026
18c1d44
implement solution proposed by keewis
agrouaze May 26, 2026
2bad102
linter
agrouaze May 26, 2026
708a180
Update ceos_alos2/hierarchy.py
agrouaze May 28, 2026
237a272
replace staticmethod -> standalone method in hierachy.py
agrouaze Jun 1, 2026
2323341
Merge branch 'main' into fix135
agrouaze Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions ceos_alos2/sar_image/caching/encoders.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,22 @@ def encode_datetime(obj):


def encode_array(obj):
# If obj is a list or lacks a dtype, convert to numpy array

@keewis keewis May 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be simplified by extending Variable.__post_init__ with:

if isinstance(self.data, list):
    super().__setattr__("data", np.array(self.data))

Even better would be to do this immediately after parsing because that would help us use the byte structure to determine a dtype, and thus avoid platform-specific behavior (e.g. windows will choose 32-bit dtypes instead of 64-bit dtypes by default). I haven't had time to look into what it would take to do that, but in the meantime we could do

if isinstance(self.data, list):
    super().__setattr__("data", np.array(self.data, dtype=infer_dtype(self.data)))

and then define infer_dtype as something like

def infer_dtype(x):
    match x:
        case np.ndarray() | np.generic():
            return x.dtype
        case list([int()]):
            return np.int64
        case list([float()]):
           return np.float64
        case list([str()]):
            return np.dtypes.StringDType
        case _:
            return None

@agrouaze agrouaze May 26, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented something very close to what you propose and it is working.
let me know if you prefer to for for the match / case syntax.

@keewis keewis May 27, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a strong attachment to the match / case syntax, but I do believe that the pattern matching is more readable than the chain of comparisons

if isinstance(obj, list):
obj = np.array(obj)
elif not hasattr(obj, "dtype"):
# Attempt conversion for other types (e.g., tuple, scalar)
try:
obj = np.array(obj)
except Exception:
# Fallback: store as a raw list (but this may break later)
return {
"__type__": "array",
"dtype": "object",
"data": obj,
"encoding": {},
}

if isinstance(obj, Array):
return {
"__type__": "backend_array",
Expand Down
Loading