Skip to content

Add in-memory data support to the JSON DataModule - #2304

Open
Excelius-Wang wants to merge 1 commit into
Lightning-AI:mainfrom
Excelius-Wang:feat/json-string-input
Open

Add in-memory data support to the JSON DataModule#2304
Excelius-Wang wants to merge 1 commit into
Lightning-AI:mainfrom
Excelius-Wang:feat/json-string-input

Conversation

@Excelius-Wang

Copy link
Copy Markdown

Fixes #1976

Adds an optional json_data input to the supervised-finetuning JSON
DataModule. It accepts either a raw JSON string or a decoded list of
sample dictionaries while preserving the existing file and directory
inputs.

The change:

  • requires exactly one of json_path and json_data
  • validates in-memory records before dataset splitting
  • supports natural jsonargparse CLI input and nested JSON values
  • preserves string scalar types during config serialization
  • excludes raw training samples from the dataclass representation
  • preserves the existing positional constructor arguments

When converting a pandas DataFrame, the supported format is
df.to_json(orient="records").

Tests cover the full train and validation DataLoader path, malformed and
unsupported payloads, CLI parsing and serialization, nested JSON values,
default validation splitting, and backward compatibility.

Testing:

  • python -m pytest -q tests/data/test_json.py
  • python -m pytest -q tests/data
  • python -m pytest -q tests/test_cli.py
  • applicable pre-commit hooks
  • jsonargparse 4.37.0 compatibility check
  • real litgpt finetune_lora --print_config invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Load a json string in the litgpt.data.JSON class instead of a flat file only

1 participant