This document specifies how to rebuild the training corpus exactly, every document and every token in every batch, starting from public data.
The corpus is ~1.0T tokens assembled from allenai/dolma3_mix-6T. Of that,
a 110.3B-token annotated split was set aside for reflection generation,
and reflections were generated over its first half: 53.5B tokens
(53,498,857,848 measured, 50.0% of the annotated split's 107.0B
post-truncation total). We do not redistribute the source text. It is
public, and every selection and annotation decision we made is published as
a compact manifest keyed by upstream document id, so reconstruction is a
replay of those decisions against upstream.
Two properties make the replay exact:
- Every stochastic step in the published
reflectionsrun is seeded, and every seed is recorded here. - The one step that is not bit-reproducible, the neural safety classifier, is bypassed entirely by publishing its outputs (§4).
Replaying the published artifacts is exact. Regenerating the reflections from scratch is not (§7).
All seven runs in the registry ship, and three are paper ablations.
reflections is the main method, SPP{T0}. reflection_end
(SPP{T0}^Doc End), refusal_reflection (SPP{T0}^Refusals) and summaries
(SPP{T0}^Summaries) are the ablations, and their window selection is as
deterministic as the main run's: the reflection variants derive their
insertion point from random.Random(f"{reflection_seed}_{doc_id}") or from
the deterministic end-of-document selector. rephrasing_safelm has no
paper result (the paper's SafeLM rows are external published checkpoints).
preflections is not in the paper at all; its prompt files are deliberately
not shipped, so that path fails with Final prompt not found.
docs/04-reflections.md maps the registry to the paper variants in full.
An earlier version of this document claimed that reflection_end,
preflections and refusal_reflection pick text windows with an unseeded
random.Random() and are therefore irreproducible. That is wrong twice
over. The unseeded constructor is at
pipeline/charter/scale/runs.py:336,403,473, in preflections,
summaries and rephrasing_safelm, not the two reflection variants, and
in all three it is passed to compute_reflection_point_end, whose first
statement is del rng (pipeline/tokenizer.py:128). Placement there is
deterministic per document. What actually makes any run irreproducible is
generator sampling (§7), which applies equally to reflections.
Everything after this section describes replaying the published artifacts. This section is the forward path: building the corpus from upstream, in order. It costs roughly 579 GPU-h for stage 2 and 13,012 GPU-h for stage 6, so read §4 and §7 before committing to either.
cp env/config.example.sh env/config.sh && $EDITOR env/config.sh
source env/config.sh
export SCRATCH="$SPP_SCRATCH" # some job scripts still read bare $SCRATCH
uv sync --group preprocessing # torch lives in this optional group
bash tools/preflight.sh # does everything config.sh names resolve?
bash tools/smoke_test.sh # the parts needing no cluster or corpusThen, in order:
# 1. Download — 47,142 shards, seeded shuffle, resumable via manifest.json
bash tools/submit.sh preprocessing/download/download_job.sh
# 2. Safety annotation — 33 tasks x 4xGH200. SKIP THIS: see §4 and join
# dlab-spp/safety-classifications on `id` instead.
bash tools/submit.sh --array=0-32 preprocessing/annotation/array_job.sh \
"$SCRATCH/dolma3_mix-1T" "$SCRATCH/safety_annotations/dolma3" 33 20000
bash tools/submit.sh preprocessing/annotation/merge_job.sh
# 3. Subsample — CPU only, ~30 min, all constants are defaults
bash tools/submit.sh preprocessing/subsample_and_stratify/job.sh
# 4. Tokenize — 20-node array plus a chained single-node merge
preprocessing/tokenization/array_job.sh submit-test # do this first
preprocessing/tokenization/array_job.sh submit
# 4b. Canary documents — independent of 1-4, minutes
uv run python preprocessing/canaries/tokenize_canaries.py \
--output-dir "$SPP_TOKENIZED_DIR/canaries" \
--data-dir preprocessing/canaries/data
# 5. Prompt development — only if you are changing the generator or the
# constitution. The release's frozen prompt is already in
# final_prompts/qwen3.5-35b-a3b/. See docs/05-generator-selection.md.
# 6. Reflection generation — the scale run, then merge into the sidecar
uv run python -m pipeline.charter.scale submit --run reflections
uv run python -m pipeline.charter.scale status --run reflections
uv run python -m pipeline.charter.scale rerun --run reflections
uv run python -m pipeline.charter.scale merge --run reflections
# 7. Post-training data — independent of 1-6. See docs/06-posttraining-data.md.Two submission mechanisms, and they are not interchangeable. The
preprocessing jobs are real sbatch scripts: tools/submit.sh adds
--account, --partition and --chdir from env/config.sh, which the
scripts no longer carry themselves. The stage-6 reflection run and the
stage-7 SFT run submit themselves through datatrove's
SlurmPipelineExecutor, reading account and partition from
configs/config.yaml; submit is a login-node command that returns as
soon as the array is queued. sbatch-ing a scale run does nothing useful.
1 download 2 annotate 3 subsample 4 tokenize
47,142 shards --> 20,000 files --> annotated/ --> compact.bin
seed 42 +safety_score unannotated/ annotated.bin
| | sidecar.parquet
| +--> safety-classifications |
| |
+--> corpus-1T-manifest <---------------------------------+|
|
4b preprocessing/canaries/ --> canary.bin ---------------------+
|
5 prompt development |
seed -> improve -> eval |
freezes the generator + prompt ------------------------------->|
v
6 reflection generation
submit --> merge
|
v
reflection-50m
|
v
epfl-dlab/spp-training
7 post-training data (SP-SFT) -- independent of 1-6; needs only the
constitution and the frozen generator.
Stages 1-4 are docs/03-corpus.md. Stage 5 is
docs/05-generator-selection.md, stage 6 is docs/04-reflections.md, and
stage 7 is docs/06-posttraining-data.md. The constitution the reflections are
written against is docs/constitution.md.
This corpus derives from allenai/dolma3_mix-6T, licensed ODC-BY 1.0.
All published artifacts here are therefore released under ODC-BY 1.0,
and the attribution notice must travel with any further derivative:
Contains information from allenai/dolma3_mix-6T, made available under the
Open Data Commons Attribution License (ODC-BY 1.0).
Cite Olmo 3 (arXiv:2512.13961) and observe AI2's Responsible Use Guidelines. Upstream frames this data as intended for research and educational use; that framing carries over here.
All four live under the dlab-spp org.
| Artifact | Contents | Size |
|---|---|---|
dlab-spp/reflection-50m |
51,386,305 rows: source text as trained (truncated) + reflections + insertion indices + canary_type |
155 GB |
dlab-spp/corpus-1T-manifest |
Per-row id, safety_score, is_bad, has_annotation, split, and ordering — mirrors the 40,000 subsample files with text removed |
16.9 GB |
dlab-spp/safety-classifications |
id → safety_score, safety_probs for 391,474,169 documents |
18.1 GB |
dlab-spp/corpus-verification |
.idx document-boundary files, token_lengths.npy, per-file checksums |
11.0 GB |
Not published: the Megatron .bin streams (2.17 TB: annotated.bin
421 GB, compact.bin 1.75 TB). They are fully derived from the above and
tokenizer-locked to SmolLM2; verify against corpus-verification
instead (§6).
The exporters are
preprocessing/subsample_and_stratify/export_corpus_manifest.py,
preprocessing/tokenization/export_verification.py and
pipeline/charter/scale/export_reflection_50m.py. Their in-tree defaults
point at the cluster paths the release was built from; pass --src-dir or
--sidecar to redirect them.
| Field | Value |
|---|---|
| Dataset | allenai/dolma3_mix-6T |
| Revision | 689a3ea2d8217e64d73a5058913fa43ad15e81aa |
| Last modified upstream | 2026-01-15 |
| Shards | 63,911 × .jsonl.zst under data/ |
| Fields used | id (UUID), text |
Pin the revision. The shard manifest (§2) is a seeded shuffle over
range(63911); if upstream ever adds or removes a shard, n_total changes
and the entire permutation changes with it.
The pin is recorded but not enforced.
preprocessing/download/download.pyexposes no--revisionflag, and both itsload_datasetcalls (lines 88 and 243) omitrevision=, so as shipped it resolves whatevermainpoints at. Reproducing the release means patching both call sites to passrevision="689a3ea2d8217e64d73a5058913fa43ad15e81aa", or at minimum assertingds.n_shards == 63911before trusting anything downstream. This is a gap in the shipped code.
Upstream contains intentional duplicates: Dolma 3 upsamples high-quality
documents by repeating rows, so id is not unique upstream. A probe of
data/common_crawl-crime_and_law-0019/shard_00000079.jsonl.zst found
189,780 rows carrying only 31,630 distinct ids, a 6.0× repetition factor.
Deduplication happens on our side, in §3.
We downloaded 47,142 of the 63,911 shards in a seeded shuffled order. The
original manifest.json did not survive scratch cleaning, but it is fully
regenerable from stdlib primitives:
import random
n_total, n_shards, seed, offset = 63911, 47142, 42, 0
order = list(range(n_total))
random.Random(seed).shuffle(order)
order = order[offset : offset + n_shards]
# order[:5] == [1502, 22656, 56218, 17774, 7487]Index into the sorted list of upstream .jsonl.zst paths; this
matches how download.py:108 selects shards, via
ds.shard(num_shards=ds.n_shards, index=shard_idx). order[0] == 1502
maps to data/common_crawl-crime_and_law-0019/shard_00000079.jsonl.zst.
Verified, by direct resolution of the mapping: ds.n_shards == 63911,
matching the recorded n_total; random.Random(42).shuffle(range(63911))
yields [1502, 22656, 56218, 17774, 7487]; the dataset's internal
base_files ordering is identical to sorted() across all 63,911
entries; and ds.shard(num_shards=63911, index=1502) resolves to exactly
the path above. Text integrity was then confirmed end to end: 585 ids from
that shard appear in sidecar row group 0, and 25 of 25 sampled documents
are byte-identical to upstream. (All 25 fall under the §6 truncation
cap; documents at the cap are prefixes of upstream by construction.)
An earlier id-overlap argument (31,627 of 31,630 probe ids present in our safety annotations) is insufficient evidence on its own, and should not be cited as such: 73.76% of all shards were downloaded, so almost any shard's ids would appear whether or not the mapping were correct.
One interpreter caveat: random.shuffle determinism is a CPython
implementation property (stable since 3.2), not a language guarantee. The
values above were produced under CPython 3.13.12; other implementations
may differ.
Only id and text are retained. A minimum-character filter is applied
during download: --min-chars defaults to 32 (download.py:216), so
documents shorter than 32 characters were dropped. Do not read the
_worker_min_chars = 0 at line 41 as the effective value; it is only the
pre-initargs global. Run with --shuffle --seed 42.
Only the first 20,000 of the 47,142 downloaded files were used. The 47,142-shard target came from probing a single upstream shard and overshot the 1T budget by ~4x (≈4.03T tokens). Because the download order is shuffled, any prefix is representative, so stage 2 annotated 20,000 files (≈1.147T tokens) and the remainder was never touched. Every row count from §3 onward is over those 20,000 files.
Reproducers who use the manifest (§5) do not need to re-run this step; it only matters if you want to rebuild the manifest itself from scratch.
Applied by _compute_dedup_indices in
preprocessing/annotation/annotate.py:223: first occurrence of each
id, computed per file. 1,180,506,475 input rows reduce to 402,040,336
unique-per-file rows, a 65.9% reduction; merge.py then copies each score
back onto every duplicate, so the upsampling survives into training while
inference was paid for once.
This step is not independently reconstructible. "First occurrence" is
relative to file order, and that order came from parallel download
completion order (worker pool + .done markers), not a sorted
enumeration. That ordering is baked into the 20,000 subsample part files
per split and is not recoverable from anything published. Do not attempt
to re-derive it: the published manifest (§5) is authoritative for both
membership and row order.
Documents were scored 0-5 by locuslab/safety-classifier_gte-large-en-v1.5
(preprocessing/annotation/annotate.py:51), whose model card is at
https://huggingface.co/locuslab/safety-classifier_gte-large-en-v1.5.
This is the only stage that is not bit-reproducible: GPU float
non-determinism can shift a borderline score, and because the
annotated/unannotated split keys on safety_score >= 3 (§5), a single
shifted score moves a document between streams and changes every
downstream batch.
We therefore publish the classifier's outputs. Join on id instead of
re-scoring:
dlab-spp/safety-classifications:
id (uuid) | safety_score (int 0-5) | safety_probs (float[6])
79 parquet files, 18.1 GB. Verified byte-identical to the internal
safety_annotations/dolma3_consolidated. Inference cost ≈579 GPU-h on
GH200 nodes across 33 array tasks.
| Field | Value |
|---|---|
| Unique annotations | 391,474,169 |
| Shard rows processed (after per-file dedup) | 402,040,336 |
| Cross-file duplicates removed | 10,566,167 |
Two score distributions exist and are easy to confuse. The published one is over the 391,474,169 unique annotations; the other is over the 402,040,336 rows before cross-file dedup:
| Score | Unique — published | Per-file deduped |
|---|---|---|
0_safe |
302,972,734 (77.39%) | 310,969,370 |
1_minimal |
38,143,123 (9.74%) | 39,210,115 |
2_mild |
32,004,998 (8.18%) | 33,027,710 |
3_moderate |
10,592,853 (2.71%) | 10,865,822 |
4_significant |
3,990,755 (1.02%) | 4,113,292 |
5_severe |
3,769,706 (0.96%) | 3,854,027 |
The left column is what the paper reports; documents scoring >= 3 are
4.7% of it. An earlier version of this document listed only the right-hand
column while labelling it as the 391,474,169-row table: those numbers sum
to 402,040,336, and are the pre-cross-file-dedup counts.
Documents with no entry in this table carry has_annotation = false and
are treated as safety_score = 0.
The classifier's precision at the threshold is low, and this is known.
Re-annotating 975 predicted-severe documents with Claude Opus 4.6 confirmed
2.7% as severe and found 61% benign; of 1,000 score-3/4 samples only 3
(0.3%) were reclassified as severe. The threshold is conservative by
design: a false positive gets a reflection on benign text, a false
negative gets no intervention at all. No code for that audit exists in
this repository or in epfl-dlab/spp-training; it was done ad hoc and
those numbers are all that survives of it. docs/03-corpus.md carries the
full breakdown.
preprocessing/subsample_and_stratify/subsample.py, recorded verbatim in
metadata.json (timestamp 2026-03-24T19:29:20):
| Parameter | Value |
|---|---|
seed |
42 |
annotation_threshold |
3 |
chars_per_token |
4.068 |
target_tokens |
1,000,000,000,000 |
scale |
0.871747139193521 |
annotation_ratio |
0.11028055101809177 |
Input: 1,180,506,475 rows / 1.147T tokens. Output:
| Split | Rows | Tokens (est.) | Files |
|---|---|---|---|
annotated |
102,772,028 | 110.30B | 20,000 |
unannotated |
925,065,551 | 889.74B | 20,000 |
| total | 1,027,837,579 | 1.000T | 40,000 |
The token counts are estimates, derived from chars_per_token = 4.068 on
untruncated text. After §6 truncation the annotated split measures
107,007,683,660 tokens, not 110.30B. Expect that ~3% shortfall in
annotated.bin; the row counts are exact.
Schema of both splits: text, id, source, safety_score, has_annotation, is_bad. (source is null throughout: it was requested from upstream but
never populated. Do not rely on it.)
92 source files in each split carry no id at all: the column is written
with arrow type null, so every row in them is unjoinable.
The cause: stack_edu-Python was never normalized to the Dolma schema. It
is the only 1 of the 163 subsets in allenai/dolma3_mix-6T that ships raw
Stack-Edu fields flattened at top level, with no id and no metadata
wrapper:
stack_edu-Python blob_id, detected_licenses, download_success,
(301 shards) int_score, language, length_bytes, license_type,
path, repo_name, score, src_encoding, text
<- no `id`
stack_edu-TypeScript added, created, id, metadata, source, text
(and 13 other langs)
Our download requested id and received nothing for those rows, so the
nulls are inherited from upstream. text, safety_score,
has_annotation and is_bad are all intact.
Upstream does carry usable identifiers for these rows (blob_id, or
repo_name + path); they are just not called id, and were not among
the columns we downloaded. A future run should request blob_id as a
fallback key.
| Split | Rows without id |
Share |
|---|---|---|
annotated |
1,629,168 | 1.585% |
unannotated |
26,275,185 | 2.840% |
| total | 27,904,353 | 2.715% |
Those rows cannot be replayed from upstream by id, so their text is
published directly in the manifest repo under text_rescue/{split}/,
keyed by global_row, the same global ordering file_boundaries.json
describes. Merge the rescue rows back at their recorded positions and the
corpus is complete; reconstruction stays exact rather than 97.3% exact.
The same loss is visible downstream: reflection-50m has a null
doc_id on 815,139 rows (1.586%). That dataset ships text inline, so
those rows are fully usable; only the provenance link is missing.
Text at this stage is untruncated and byte-identical to upstream:
untruncated is observed directly (max 213K/310K characters, mean ~5.5K),
and byte-identity is verified by the §2 spot check (25/25 sampled
documents). Nothing in download.py mutates text. Truncation happens in
§6.
Rows cluster by source shard. The subsample preserves input order, and output filenames mirror input filenames one-to-one, so each of the 20,000 annotated files and each of the 20,000 unannotated files descends from a single input file, and therefore from roughly a single upstream shard. Do not assume a uniform distribution when sampling for verification: a strided sample of 100 files found none of a known shard's 31,630 documents, while sidecar row group 0 alone held 585 of them.
corpus-1T-manifest mirrors these 40,000 files exactly: same file
partitioning, same row order, text removed. Rebuild each split by joining
the manifest to upstream text on id. Preserve file and row order: the
.bin layout, and therefore batch composition, depends on it.
preprocessing/tokenization/tokenize.py. Tokenizer:
SmolLM2-1.7B-Instruct. Two details are mandatory, and both are
silent-corruption traps:
- Use the Rust
tokenizerslibrary (tokenizers.Tokenizer.from_pretrained), nottransformers.AutoTokenizer. They disagree on\n\n: the Rust library emits a single token1116,AutoTokenizeremits[198, 198]. Verification against our.idxfiles will fail across the board if you use the wrong one.pipeline/tokenizer.pyis the single entry point and exists to make this hard to get wrong. enable_truncation(max_length=1920)counts the EOS token, so the maximum content length is 1919 tokens. 25.4% of annotated rows sit exactly at this cap; no row exceeds it. The docstring onTruncatingDocumentTokenizer(preprocessing/tokenization/steps.py:161) claims the opposite and is wrong;preprocessing/canaries/tokenize_canaries.py:36states it correctly.
The cap is derived: --seq-length 2048 --reflection-budget 128
gives max_doc_tokens = 1920, reserving 128 tokens of the 2,048-token
window for the reflection §7 splices in. eos_token="<|endoftext|>" is
passed explicitly because the SmolLM2-Instruct tokenizer defaults to
<|im_end|> (id 2), and the streams need id 0 so that pad == EOS.
The two streams diverge here:
annotated→ the sidecar (102,772,028 rows of truncated text) and, after reflection insertion (§7),annotated.bin/annotated.idx.unannotated→ the compact stream.
Verify against dlab-spp/corpus-verification: the .idx files give
per-document boundaries and lengths, so you can assert a rebuilt .bin
matches ours exactly without downloading 2.17 TB. token_lengths.npy
provides the same check at the row level.
uv run python -m pipeline.charter.scale submit --run reflections
uv run python -m pipeline.charter.scale status --run reflections
uv run python -m pipeline.charter.scale rerun --run reflections
uv run python -m pipeline.charter.scale merge --run reflectionssubmit runs on the login node: it computes
N = ceil(rows / rows_per_task) and hands a datatrove
SlurmPipelineExecutor the job array, which submits itself. Each task
co-locates an sglang server and the generation client on one node. merge
is a streaming merge-join that adds this run's columns to the sidecar
and preserves every column a previous merge wrote, which is why the
ablation runs can share one file. Use --allow-missing only once you have
decided the gaps are acceptable; it fills them with empty strings.
Reflections cover the first half of the annotated stream. 51,399,997 rows carry an insertion point, spanning sidecar row groups 0-50. Those groups together hold 52,434,596 rows, so the reflected region ends partway into row group 50. Row groups are not uniform in size (944,286 / 944,298 / 1,048,576 rows). Measured tail:
rg48: 1,048,576 rows — 1,048,293 reflected
rg49: 944,298 rows — 944,066 reflected
rg50: 1,048,576 rows — 13,975 reflected <- region ends here
rg51: 1,048,576 rows — 0 reflected
Read row groups 0-50 (51 groups) to cover it. Rows beyond the region are trained on without reflections.
Three row counts are easy to confuse:
| Set | Rows |
|---|---|
Insertion point assigned (reflection_token_index >= 0) |
51,399,997 |
Published in reflection-50m (1p or 3p non-empty) |
51,386,305 |
reflection_1p non-empty only |
51,384,207 |
The published dataset excludes 13,692 rows that were assigned an insertion point but produced no parsable reflection text.
| Parameter | Value |
|---|---|
| Model | Qwen3.5-35B-A3B-FP8, served with sglang, TP=1 DP=4 |
| Run | reflection_full (514 ranks × 100,000 rows) |
| API calls per document | 1 |
reflection_seed |
42 |
canary_seed |
42 |
| Canaries | identity set only (Q1/Q2/Q3/Q7/Q10), 10% overall rate |
| Cost | ≈13,012 GPU-h, ~4 docs/s/node, 24 isolated failures |
Output columns: reflection_1p, reflection_3p, reflection_position,
reflection_token_index, charter_reflection, canary_type. Totals:
5.204B reflection tokens (2.746B first-person + 2.458B third-person),
measured with the same SmolLM2 tokenizer. Mean length 53 tokens, against
the 128-token budget §6 reserved.
Reflections are inserted mid-document at reflection_token_index.
Generation is sampled, so re-running the model will not reproduce our
text: use the published reflections from reflection-50m and insert
them at the recorded indices. Regenerate only for a different corpus.
Regeneration is not reproducible, even distributionally. The sampling parameters were pinned:
resolve_sampling_paramsmatches theqwen3.5row of_SAMPLING_DEFAULTS(pipeline/api.py:35) and sends temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5. But no RNG seed is ever sent to the server, so decoding at temperature 1.0 is non-deterministic no matter what else is fixed.max_tokens=None(generate.py:417), and the sglang version is not recorded here either. The run config setsthinking: false, but thinking was effectively on: thekimi_k2reasoning parser zeroed the reportedreasoning_tokenswhile the model still emitted them (median ~4.4K generated tokens/row against ~101 stored). Do not conclude from the config that thinking was disabled.
reflection_full does contain injected identity canaries. Measured
over the reflected region: 9.85% of reflected rows in row group 0, 9.84%
in row group 25, 9.41% in row group 50, matching the configured 10% rate.
Assignment is deterministic in (canary_seed, doc_id)
(pipeline/charter/scale/canaries.py), so a document always draws the
same fact. Five identity facts are used, evenly split at ~2% each:
canary_type |
Rows (row group 0) |
|---|---|
Q1 |
20,866 |
Q10 |
20,797 |
Q7 |
20,563 |
Q2 |
20,496 |
Q3 |
20,486 |
The canary_type column is the exclusion flag: empty for clean rows,
naming the injected fact otherwise. To train on canary-free data:
ds = ds.filter(lambda r: not r["canary_type"])Filtering removes canaries from your training run. It is not a
confidentiality measure: resources/canaries.yaml ships the literal fact
values in this repository, and canary_type plus the reflection text
together form a labeled corpus of ~20.5K examples per fact. Anyone can read
the facts directly or train a targeted extractor. Once this dataset is
public, this canary set is spent as a memorization probe; any future
probe needs a fresh, withheld set.
Prompts, constitutions, and canaries.yaml are in resources/ and
final_prompts/ for inspection and for building variants.
Separately from the identity canaries, the training mix carries a third
stream of 60,000 synthetic canary documents (61.9M content tokens,
246 MB, 0.01% of the mix). It is built by preprocessing/canaries/,
outside §1-§6, and consists of 12 backdoor conditions of 2,500 documents
each — four effects (toxic, harmful, no_refusal, ads_nestle)
crossed with three reflection-coverage fractions, every backdoor document
prefixed by a unique 9-token trigger string — plus 6 fictional-science
universes of 5,000 documents each.
The poisoning and belief-uptake experiments this stream was built for
were not pursued, as the paper states; the stream is documented because
it remains part of the released training mix. Reproducing the mix
therefore means reproducing this stream too, or explicitly dropping it
and accepting a 0.01% difference.
preprocessing/canaries/tokenize_canaries.py rebuilds canary.bin from
already-generated documents deterministically (per-condition seeds are a
fixed table, not hash(), which Python randomizes per process).
Generating new canary documents is not possible from this repository:
generate_canary_docs.py reads its prompts from the
safety-research/believe-it-or-not submodule, which is not vendored
here. docs/03-corpus.md covers the stream in full.
Training uses Megatron-LM with a custom MMapIndexedDataset wrapper that
bypasses GPTDataset, which would re-pack the pre-packed streams and
silently corrupt window boundaries. Given identical .bin/.idx streams,
batch composition is fully determined by that wrapper plus the values
below. All of them are pinned in the sibling repository,
epfl-dlab/spp-training, whose docs/04-pretraining.md and
docs/03-data.md are authoritative:
| Quantity | Value | Source |
|---|---|---|
| Sequence length | 2048, plus one for the next-token shift → 2,049-token windows | spp-training/docs/04-pretraining.md |
| Global batch size | 960 sequences (≈2.0M tokens/step) | spp-training/docs/04-pretraining.md |
| Steps, 3B / 500B | 254,313 | spp-training/docs/04-pretraining.md |
| Steps, 1.7B / 100B | 50,863 | spp-training/docs/04-pretraining.md |
| Stream weights | compact 0.8062, annotated 0.1937, canary 0.000113 |
spp-training/docs/03-data.md |
Weights are proportional to stream size, so over a 500B-token run each source is sampled at roughly 46%, under one epoch for all three. A two-level Bresenham schedule does the interleaving.
There is no data seed, because there is no re-shuffle. The streams are
shuffled once, at write time (§6, --seed 42), and the training-side
dataloader reads them sequentially from index 0. That is what makes data
order reproducible across runs, and what lets the midtraining variants
replay an exact index window. The obligation on this side of the split is
the one §6 states: the annotated and canary streams must never be
re-shuffled, or sidecar alignment breaks.
Everything needed for bit-exact batch reproduction is therefore pinned.
The optimizer and schedule are not pinned here, and they do not affect
batch composition; spp-training/docs/04-pretraining.md carries them,
including an unresolved 2,000-versus-3,000 warmup-step discrepancy that
repository flags itself.
Work down this list; each step is checkable independently.
- Upstream revision resolves to
689a3ea2...and lists 63,911 shards. - Regenerated manifest gives
order[:5] == [1502, 22656, 56218, 17774, 7487]. - Subsample row counts match §5 exactly (102,772,028 / 925,065,551).
- Rebuilt
.idxmatchesdlab-spp/corpus-verification(catches tokenizer mistakes). token_lengths.npymatches, with max 1919 and 25.4% at the cap.- Sidecar row count equals the
annotated.idxwindow count (102,772,028). - Reflection token totals match §7 (2.746B / 2.458B).
- The relationship between the 391M annotated documents and the 1.18B-row
subsample input should be stated explicitly (the
has_annotationflag is believed to carry this, but it has not been verified end to end). - No RNG seed is passed to the generation endpoint (§7). Threading a
per-document seed through
api_callwould close this; the sglang version is likewise unrecorded. download.pyhas no--revisionflag (§1), so the upstream pin is unenforced.- License metadata on the already-public
spp-reflection-10mandspp-reflection-sample-2kstill readsother; it should readodc-bywith the attribution notice above.
| Symptom | Likely cause |
|---|---|
SPP_DATA_ROOT: not set — source env/config.sh, or submit via tools/submit.sh |
env/config.sh was never sourced, or the script was sbatch-ed directly instead of through tools/submit.sh. |
SCRATCH: unbound variable |
A few job scripts still read a bare $SCRATCH. export SCRATCH="$SPP_SCRATCH" after sourcing. |
Final prompt not found: .../final_prompts/<alias>/<file> |
That run's prompt is not shipped for that generator alias. Expected for preflections, whose prompts are deliberately absent. Otherwise check charter.scale.generator_alias against the directories under final_prompts/. |
Prompt file not found: data/pipeline/prompts/<alias>/<file> |
A prompt-development path (stage 5) naming an explicit version that does not exist. Explicit version filenames never auto-create a stub directory, by design, so this is a typo'd alias or version rather than a missing init. |
FATAL: sglang process died |
The server crashed during startup. Read the task's .err: usually SPP_GENERATOR_MODEL_PATH is wrong, or the container named by SPP_SGLANG_ENV_TOML cannot see the weights. |
FATAL: sglang failed to start after 20 minutes |
The health check never passed. Cold weight loading over a slow filesystem, or a port collision on charter.scale.sglang.port. |
Missing N rows in results. Expected M, got K. |
Some ranks never finished. Run scale status --run <name>, then scale rerun --run <name>. Reach for merge --allow-missing only after deciding the gap is acceptable — it writes empty strings into those rows permanently. |
Rebuilt .idx differs from corpus-verification on nearly every document |
transformers.AutoTokenizer was used instead of the Rust tokenizers library. They disagree on \n\n (§6). |
token_lengths.npy max is 1920, not 1919 |
EOS was not counted in the truncation limit. Use enable_truncation(max_length=1920) on the Rust tokenizer rather than slicing token ids by hand. |
| Reflections attach to the wrong documents | Sidecar/stream alignment broke — a row was inserted, dropped, or re-sorted. Nothing checks this at runtime; compare the sidecar row count against the annotated.idx window count. |
ModuleNotFoundError: torch |
torch lives in the optional preprocessing dependency group. uv sync --group preprocessing. |
TypeError: 'tokenizers.Encoding' object is not subscriptable |
A known pre-existing bug at preprocessing/tokenization/test_tokenize.py:253, which slices an Encoding rather than its .ids. It broke when the pipeline moved from AutoTokenizer to the Rust library and is shipped unfixed. |
| A dataset id 404s | Two ids in the tree predate the release: jkminder/dolma3-safety-annotations (now dlab-spp/safety-classifications) and jkminder/dolma3_mix-1T-annotated (deleted). Pass --repo-id explicitly. |
jkminder/dolma3_mix-1T-annotated — an aborted upload holding 49 of
20,000 files and no unannotated/ split — was deleted on 2026-07-27,
after confirming all 49 files remained reproducible from the source
parquets.