"eval_strategy": "no" perform evaluation

[SamuelLarkin]

Describe the bug

Hi,
I'm trying to train a diffusion model and I getting stuck with

strace -p 967873
strace: Process 967873 attached
ioctl(9, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0x7fff18579d90) = 0
ioctl(9, _IOC(_IOC_READ|_IOC_WRITE, 0x46, 0x2a, 0x20), 0x7fff18579d90) = 0

But my issue is that it hangs when it tries to do eval. So I decided to eval_strategy: "no" to skip that previous issue but the code doesn't respect the fact that I explicitly aske to skip eval.

config.yaml

base_model: meta-llama/Llama-3.2-1B # Automatically upload checkpoint and final model to HF # hub_model_id: username/custom_model_name

Continue training after requeueing.

NOTE: resume_from_checkpoint must be empty

resume_from_checkpoint:
auto_resume_from_checkpoints: true

Setting pretraining_dataset without explicitly setting streaming: true is deprecated.

In a future release, streaming will not be automatically enabled when using pretraining_dataset.

Please explicitly set streaming: true in your configuration to maintain current behavior.

NOTE: Streaming currently only supports a single dataset. Multi-dataset support will be added soon.

streaming: false

Streaming-specific settings

streaming_multipack_buffer_size: 10000 # NOTE: 50000 required >300GB RAM WTF?!
shuffle_merged_datasets: true

For pretraining: control with pretrain_multipack_attn

pretrain_multipack_attn: true # prevent cross-attention between packed samples

dataset_prepared_path: last_run_prepared

• path: A.jsonl
  ds_type: json
  type: completion
• path: B.jsonl
  ds_type: json
  type: completion
• path: C.jsonl
  ds_type: json
  type: completion
• path: D.jsonl
  ds_type: json
  type: completion

eval_strategy: "no"
test_datasets:

• path: E.jsonl
  ds_type: json
  type: completion

plugins:

• axolotl.integrations.diffusion.DiffusionPlugin

diffusion:
noise_schedule: cosine
min_mask_ratio: 0.15
max_mask_ratio: 0.85
num_diffusion_steps: 128
eps: 5e-4
importance_weighting: true
mask_token_id: 128002

NOTE: It looks like it is trying to run eval on train_dataloader().

generate_samples: true

generate_samples: false

generation_interval: 500 # same as the save frequency

output_dir: ./outputs/model-out

sequence_len: 512
sample_packing: true

gradient_accumulation_steps: 4
micro_batch_size: 32

NOTE: max_steps must be set when using streaming datasets.

Trainer cannot infer dataset length for iterable datasets.

max_steps: 100000

num_epochs: 3.0
warmup_ratio: 0.1

Optimizer and scheduler

optimizer: adamw_8bit

optimizer: adamw_torch_8bit

lr_scheduler: cosine
learning_rate: 3e-4
attn_implementation: sdpa

Precision and performance

bf16: auto
tf32: true

Logging and checkpointing

logging_steps: 5
save_strategy: steps
save_steps: 500
save_total_limit: 3

Special tokens

special_tokens:
pad_token: "<|end_of_text|>"

use_tensorboard: true
wandb_project:
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:

load_best_model_at_end: false

save_first_step: true # uncomment this to validate checkpoint saving works with your config

fsdp_version: 2

fsdp_config:

offload_params: true

state_dict_type: FULL_STATE_DICT

auto_wrap_policy: TRANSFORMER_BASED_WRAP

transformer_layer_cls_to_wrap: LlamaDecoderLayer

reshard_after_forward: true

Steps to reproduce the bug

accelerate launch \
--multi_gpu \
--num_processes="$SLURM_GPUS_ON_NODE" \
--num_machines="$SLURM_NNODES" \
--mixed_precision=bf16 \
-m axolotl.cli.train \
"$config" \
--output-dir="$output_dir" \
--dataloader-num-workers="$SLURM_CPUS_PER_TASK" \
--dataset-num-proc="$SLURM_TASKS_PER_NODE" \
--dataset-processes="$SLURM_TASKS_PER_NODE" \
--deepspeed=deepspeed_configs/zero1.json

Expected behavior

The eval step shouldn't run.

Environment info

Slurm cluster, using single node 4 GPUs and deepspeed_configs/zero1.json.

uv pip freeze

``` absl-py==2.4.0 accelerate==1.13.0 addict==2.4.0 adlfs==2026.4.0 aiobotocore==2.26.0 aiofiles==23.2.1 aiohappyeyeballs==2.6.1 aiohttp==3.13.5 aioitertools==0.13.0 aiosignal==1.4.0 annotated-doc==0.0.4 annotated-types==0.7.0 antlr4-python3-runtime==4.13.2 anyio==4.13.0 art==6.5 asttokens==3.0.1 attrs==26.1.0 autoawq==0.2.7.post3 -e file:///gpfs/work/larkins/TraLLM/axolotl/axolotl axolotl-contribs-lgpl==0.0.7 axolotl-contribs-mit==0.0.6 azure-core==1.40.0 azure-datalake-store==0.0.53 azure-identity==1.25.3 azure-storage-blob==12.28.0 backoff==2.2.1 bitsandbytes==0.49.1 botocore==1.41.5 brotli==1.2.0 cbor2==6.0.1 certifi==2026.4.22 cffi==2.0.0 chardet==6.0.0.post1 charset-normalizer==3.4.7 circuitbreaker==2.1.3 click==8.3.3 cmake==4.3.2 colorama==0.4.6 coloredlogs==15.0.1 cryptography==46.0.7 cuda-bindings==13.2.0 cuda-pathfinder==1.5.4 cuda-toolkit==13.0.2 cut-cross-entropy @ git+https://github.com/axolotl-ai-cloud/ml-cross-entropy.git@fec1a888e6f4ad7e6270ea7b02186e56c76f5ac2 dataproperty==1.1.0 datasets==4.8.5 decorator==5.2.1 deepspeed==0.18.9 deepspeed-kernels==0.0.1.dev1698255861 dill==0.3.8 distro==1.9.0 einops==0.8.2 evaluate==0.4.1 executing==2.2.1 fastapi==0.136.1 fastcore==1.12.44 ffmpy==1.0.0 filelock==3.29.0 fire==0.7.1 fla-core==0.4.1 flash-linear-attention==0.4.1 frozenlist==1.8.0 fsspec==2025.3.0 gcsfs==2025.3.0 gitdb==4.0.12 gitpython==3.1.49 google-api-core==2.30.3 google-auth==2.50.0 google-auth-oauthlib==1.3.1 google-cloud-core==2.5.1 google-cloud-storage==3.10.1 google-crc32c==1.8.0 google-resumable-media==2.8.2 googleapis-common-protos==1.74.0 gradio==6.14.0 gradio-client==2.5.0 groovy==0.1.2 grpcio==1.80.0 grpclib==0.4.7 h11==0.16.0 h2==4.3.0 hf-gradio==0.4.1 hf-transfer==0.1.9 hf-xet==1.4.3 hjson==3.1.0 hpack==4.1.0 httpcore==1.0.9 httptools==0.7.1 httpx==0.28.1 huggingface-hub==1.13.0 humanfriendly==10.0 hyperframe==6.1.0 idna==3.13 immutabledict==4.2.0 ipython==9.13.0 ipython-pygments-lexers==1.1.1 isodate==0.7.2 jedi==0.20.0 jinja2==3.1.6 jmespath==1.1.0 joblib==1.5.3 jsonlines==4.0.0 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 kernels==0.13.0 langdetect==1.0.9 liger-kernel==0.7.0 llvmlite==0.47.0 lm-eval==0.4.11 lxml==6.1.0 markdown==3.10.2 markdown-it-py==4.0.0 markupsafe==3.0.3 matplotlib-inline==0.2.2 mbstrdecoder==1.1.5 mdurl==0.1.2 mistral-common==1.11.0 modal==1.3.0.post1 more-itertools==11.0.2 mpmath==1.3.0 msal==1.36.0 msal-extensions==1.3.1 msgpack==1.1.2 multidict==6.7.1 multiprocess==0.70.16 networkx==3.6.1 ninja==1.13.0 nltk==3.9.4 numba==0.65.1 numexpr==2.14.1 numpy==2.3.5 nvidia-cublas==13.1.0.3 nvidia-cuda-cupti==13.0.85 nvidia-cuda-nvrtc==13.0.88 nvidia-cuda-runtime==13.0.96 nvidia-cudnn-cu13==9.19.0.56 nvidia-cufft==12.0.0.61 nvidia-cufile==1.15.1.6 nvidia-curand==10.4.0.35 nvidia-cusolver==12.0.4.66 nvidia-cusparse==12.6.3.3 nvidia-cusparselt-cu13==0.8.0 nvidia-ml-py==12.560.30 nvidia-nccl-cu13==2.28.9 nvidia-nvjitlink==13.0.88 nvidia-nvshmem-cu13==3.4.5 nvidia-nvtx==13.0.85 oauthlib==3.3.1 oci==2.173.1 ocifs==1.3.2 openenv-core==0.1.0 optimum==1.16.2 orjson==3.11.8 packaging==26.0 pandas==2.3.3 parso==0.8.7 pathvalidate==3.3.1 peft==0.19.1 pexpect==4.9.0 pillow==11.3.0 platformdirs==4.9.6 portalocker==3.2.0 posthog==6.7.11 prompt-toolkit==3.0.52 propcache==0.4.1 proto-plus==1.27.2 protobuf==5.29.6 psutil==7.2.2 ptyprocess==0.7.0 pudb==2025.1.5 pure-eval==0.2.3 py-cpuinfo==9.0.0 pyarrow==24.0.0 pyasn1==0.6.3 pyasn1-modules==0.4.2 pybind11==3.0.4 pycountry==26.2.16 pycparser==3.0 pydantic==2.10.6 pydantic-core==2.27.2 pydantic-extra-types==2.11.1 pydub==0.25.1 pygments==2.20.0 pyjwt==2.12.1 pyopenssl==26.2.0 pytablewriter==1.2.1 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 python-multipart==0.0.27 pytz==2026.2 pyyaml==6.0.3 referencing==0.37.0 regex==2026.4.4 requests==2.33.1 requests-oauthlib==2.0.0 responses==0.18.0 rich==15.0.0 rouge-score==0.1.2 rpds-py==0.30.0 ruff==0.15.12 s3fs==2025.3.0 sacrebleu==2.6.0 safehttpx==0.1.7 safetensors==0.7.0 schedulefree==1.4.1 scikit-learn==1.4.2 scipy==1.17.1 semantic-version==2.10.0 sentencepiece==0.2.1 sentry-sdk==2.59.0 setuptools==81.0.0 shellingham==1.5.4 sigtools==4.0.1 six==1.17.0 smmap==5.0.3 sqlitedict==2.1.0 stack-data==0.6.3 starlette==0.52.1 sympy==1.14.0 synchronicity==0.11.1 tabledata==1.3.4 tabulate==0.10.0 tcolorpy==0.1.7 tensorboard==2.20.0 tensorboard-data-server==0.7.2 termcolor==3.3.0 threadpoolctl==3.6.0 tiktoken==0.12.0 tokenizers==0.22.2 toml==0.10.2 tomlkit==0.13.3 torch==2.11.0+cu130 torchao==0.17.0 torchvision==0.25.0+cu130 tqdm==4.67.3 tqdm-multiprocess==0.0.11 trackio==0.25.1 traitlets==5.15.0 transformers==5.5.4 triton==3.6.0 stack-data==0.6.3 starlette==0.52.1 sympy==1.14.0 synchronicity==0.11.1 tabledata==1.3.4 tabulate==0.10.0 tcolorpy==0.1.7 tensorboard==2.20.0 tensorboard-data-server==0.7.2 termcolor==3.3.0 threadpoolctl==3.6.0 tiktoken==0.12.0 tokenizers==0.22.2 toml==0.10.2 tomlkit==0.13.3 torch==2.11.0+cu130 torchao==0.17.0 torchvision==0.25.0+cu130 tqdm==4.67.3 tqdm-multiprocess==0.0.11 trackio==0.25.1 traitlets==5.15.0 transformers==5.5.4 triton==3.6.0 trl==1.1.0 typepy==1.3.5 typer==0.25.1 types-certifi==2021.10.8.3 types-toml==0.10.8.20260408 typing-extensions==4.15.0 typing-inspection==0.4.2 tzdata==2026.2 urllib3==2.6.3 urwid==4.0.0 urwid-readline==0.15.1 uvicorn==0.46.0 uvloop==0.22.1 wandb==0.26.1 watchfiles==1.1.1 wcwidth==0.7.0 websockets==15.0.1 werkzeug==3.1.8 word2number==1.1 wrapt==1.17.3 xformers==0.0.35 xxhash==3.7.0 yarl==1.23.0 zstandard==0.22.0 ```

[matdou]

Wrong repo, this belongs in axolotl, not datasets. And eval_strategy: "no" was never going to stop it.

What hangs you is the diffusion plugin's sample generation. DiffusionGenerationCallback.on_step_end fires on global_step % generation_interval == 0 and ignores eval_strategy. When you have no eval dataset, it falls back to get_train_dataloader(). That matches your note about eval running on the train dataloader.

Set this instead:

diffusion:
  generate_samples: false

The callback only registers when generate_samples is true. You set it twice in your config (true then false); YAML keeps the last key, so confirm it lands on false. If it reads false and the callback still runs, file that in axolotl as a real bug.

One more thing: you set test_datasets but eval_strategy: "no". Drop test_datasets if you want no eval at all.