Checklist
Background
Current on-policy distillation in AReaL assumes that the teacher and student models share the same tokenizer. This limits distillation to models within the same tokenizer family (e.g., models from the same series).
However, many practical distillation scenarios involve heterogeneous models with different vocabularies and tokenization schemes, such as distilling from one model family to another (e.g., Llama → Qwen, Deepseek → Qwen, etc.). Since teacher and student tokenize the same text differently, token-level supervision cannot be applied directly.
Supporting cross-tokenizer on-policy distillation would enable knowledge transfer across different model series and model families while preserving the existing on-policy training pipeline.
Potential Solution
Implement a cross-tokenizer distillation pipeline based on chunk-level alignment between student and teacher tokenizations.
High-level workflow:
Generate rollout trajectories using the student model.
Decode the student-generated sequence into text.
Re-tokenize the sequence using the teacher tokenizer while preserving the prompt/response boundary.
Compute teacher log-probabilities on the teacher-tokenized sequence.
Align teacher response chunks back to the corresponding student response chunks using a chunk alignment algorithm.
Aggregate teacher log-probabilities for each aligned chunk to obtain token-level supervision for the student.
Compute the standard distillation loss using the aligned teacher log-probabilities.
The implementation should:
Keep the existing same-tokenizer distillation path unchanged.
Introduce cross-tokenizer alignment only when explicitly enabled through configuration.
Additional Information
(Add any relevant context, references, or supporting data here.)
Checklist
areal/api/. If not, please raise a refactor issue first.Background
Current on-policy distillation in AReaL assumes that the teacher and student models share the same tokenizer. This limits distillation to models within the same tokenizer family (e.g., models from the same series).
However, many practical distillation scenarios involve heterogeneous models with different vocabularies and tokenization schemes, such as distilling from one model family to another (e.g., Llama → Qwen, Deepseek → Qwen, etc.). Since teacher and student tokenize the same text differently, token-level supervision cannot be applied directly.
Supporting cross-tokenizer on-policy distillation would enable knowledge transfer across different model series and model families while preserving the existing on-policy training pipeline.
Potential Solution
Implement a cross-tokenizer distillation pipeline based on chunk-level alignment between student and teacher tokenizations.
High-level workflow:
Generate rollout trajectories using the student model.
Decode the student-generated sequence into text.
Re-tokenize the sequence using the teacher tokenizer while preserving the prompt/response boundary.
Compute teacher log-probabilities on the teacher-tokenized sequence.
Align teacher response chunks back to the corresponding student response chunks using a chunk alignment algorithm.
Aggregate teacher log-probabilities for each aligned chunk to obtain token-level supervision for the student.
Compute the standard distillation loss using the aligned teacher log-probabilities.
The implementation should:
Keep the existing same-tokenizer distillation path unchanged.
Introduce cross-tokenizer alignment only when explicitly enabled through configuration.
Additional Information
(Add any relevant context, references, or supporting data here.)