Skip to content

MAINT: remove fastparquet compression for bubblesam parquet#39

Open
adamwitmer wants to merge 1 commit into
mainfrom
awitmer_save_parquet
Open

MAINT: remove fastparquet compression for bubblesam parquet#39
adamwitmer wants to merge 1 commit into
mainfrom
awitmer_save_parquet

Conversation

@adamwitmer

Copy link
Copy Markdown
Collaborator

In line with #4 (comment), this PR simplifies bubblesam parquet saving/loading by removing the use of fastparquet engine, which saved objects as strings. This change will also simplify parquet loading in PR #5 and remove the need for the ast module there.

* modify handling of type coercions for saving
* reshape contour array when loading in test
@adamwitmer

Copy link
Copy Markdown
Collaborator Author

I also opened issue #40 to make a note of the need to simplify data structure handling for saving as parquet, which I did not address in this PR to keep the scope limited to changes related to fastparquet.

@tylerjereddy

Copy link
Copy Markdown
Collaborator

Can you provide a brief/clear explanation of why you need to remove fastparquet usage if it only stores objects as strings and you're going to remove the object storage anyway? For example, a reviewer might sensibly deduce that if objects are removed from the control flow then the problem goes away without needing to mess around with the parquet format in the first place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants