The evaluation results did not meet expectations.

[jiyanxin]

I evaluated the base model Llama-3.2-1B-Instruct and the zip2xip model zip2zip-Llama-3.2-1B-Instruct-v0.1 using the code banch/run_harness_pretrained.py. The evaluation results are as follows:

• Llama-3.2-1B-Instruct

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_easy	1	none	2	acc	↑	0.57	±	0.0498
		none	2	acc_norm	↑	0.64	±	0.0482
commonsense_qa	Yaml	none	2	acc	↑	0.57	±	0.0498
hellaswag	1	none	2	acc	↑	0.47	±	0.0502
		none	2	acc_norm	↑	0.63	±	0.0485
mathqa	1	none	2	acc	↑	0.29	±	0.0456
		none	2	acc_norm	↑	0.30	±	0.0461
openbookqa	1	none	2	acc	↑	0.24	±	0.0429
		none	2	acc_norm	↑	0.35	±	0.0479
piqa	1	none	2	acc	↑	0.72	±	0.0451
		none	2	acc_norm	↑	0.78	±	0.0416
winogrande	1	none	2	acc	↑	0.62	±	0.0488

• zip2zip-Llama-3.2-1B-Instruct-v0.1

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_easy	1	none	2	acc	↑	0.42	±	0.0496
		none	2	acc_norm	↑	0.39	±	0.0490
commonsense_qa	Yaml	none	2	acc	↑	0.23	±	0.0423
hellaswag	1	none	2	acc	↑	0.19	±	0.0394
		none	2	acc_norm	↑	0.21	±	0.0409
mathqa	1	none	2	acc	↑	0.26	±	0.0441
		none	2	acc_norm	↑	0.26	±	0.0441
openbookqa	1	none	2	acc	↑	0.16	±	0.0368
		none	2	acc_norm	↑	0.29	±	0.0456
piqa	1	none	2	acc	↑	0.55	±	0.0500
		none	2	acc_norm	↑	0.56	±	0.0499
winogrande	1	none	2	acc	↑	0.55	±	0.0500

I noticed that the results of the zip2zip model differ significantly from those of the base model. Has anyone encountered this situation before? Is this difference normal, or could there be issues in the evaluation process?

[Saibo-creator]

Hi @jiyanxin,

That gap is not expected. I’ve been reviewing the code, and there appears to be a mismatch between our internal codebase and the public release due to substantial refactoring for Transformers compatibility. The issue is likely related to either the new lm_eval_harness integration or recent changes to the LZW module.

I can confirm that our internal codebase reproduces the Phi-3.5-mini results reported in the NeurIPS paper. Note that LLaMA models were trained later and are not included in the paper; they were trained with improved procedures (see the Phi models for reference).

To reproduce the reported results, I have uploaded our internal codebase here:
https://github.com/epfl-dlab/zip2zip/tree/legacy

Run:

python eval.py --adapter=checkpoints/evqn/model_7000.safetensors --tasks ai2_arc,openbookqa,piqa,winogrande,commonsense_qa,lambada,mathqa,hellaswag --limit 500

The checkpoint evqn/model_7000.safetensors corresponds to the model used in the paper and is hosted via Git LFS.

The LZW compression module must be built from source (Rust):

pip install maturin
cd fast_compression
maturin develop --release

Let me know if you run into issues.

Best,
Saibo

[Saibo-creator]

Hi all,

Due to the refactoring of the LZW compression codebase, there were some subtle changes in the boundary conditions — specifically how we handle max merge size, disabled IDs, and imperfect compression. It's been difficult to pinpoint exactly what caused the previous model to underperform, so instead we trained a new model with the updated code. Here are the results:

epfl-dlab/Llaza-3.2-1B-MS2F-4K-v0.1

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_easy	1	none	2	acc	↑	0.634	±	0.010
		none	2	acc_norm	↑	0.618	±	0.010
commonsense_qa	Yaml	none	2	acc	↑	0.208	±	0.012
hellaswag	1	none	2	acc	↑	0.379	±	0.005
		none	2	acc_norm	↑	0.562	±	0.005
mathqa	1	none	2	acc	↑	0.232	±	0.008
		none	2	acc_norm	↑	0.240	±	0.008
openbookqa	1	none	2	acc	↑	0.254	±	0.019
		none	2	acc_norm	↑	0.372	±	0.022
piqa	1	none	2	acc	↑	0.730	±	0.010
		none	2	acc_norm	↑	0.726	±	0.010
winogrande	1	none	2	acc	↑	0.587	±	0.014

These results are much closer to the base Llama 3.2-1B performance, except for commonsense_qa where there is still a significant drop. I'll look into that and update here.

You can reproduce with:

python bench/run_harness_pretrained.py \
    epfl-dlab/Llaza-3.2-1B-MS2F-4K-v0.1 \
    --revision hf \
    --tasks hellaswag piqa winogrande commonsense_qa arc_easy openbookqa mathqa