perf: do not allocate for alignments in encode_fast

[SBrandeis]

Builds on top of #2022

encode_fast only cares about token ids. Offsets, token strings, etc get dropped at the very end of the pipeline: ref

#2022 improved the perf of encode_fast by not computing alignment offsets during normalization, but the alignment Vec were still fully allocated at construction (NormalizedString::from), in set_normalized ref and at every slice() call during pre-tokenization: ref

This PR adds a "no alignment" fast path on NormalizedString methods that skip allocation (String::new and Vec::new do not allocate) when in the encode_fast path, by relying on a invariant that cannot happen when computing alignments through encode: empty alignments vec + non-empty normalized string

tokenizers/tokenizers/src/tokenizer/normalizer.rs

Lines 139 to 160 in f5aa6c0

	/// Build a `NormalizedString` that doesn't track alignments.
	/// See [`Self::is_unaligned`].
	pub(crate) fn unaligned(normalized: String) -> Self {
	Self {
	original: String::new(),
	normalized,
	alignments: Vec::new(),
	original_shift: 0,
	}
	}
	/// Whether this string skips alignment tracking.
	///
	/// The trick: an aligned string always has one alignment entry per byte
	/// of `normalized`, so "alignments empty but normalized not empty" is
	/// normally impossible. We use that impossible state to mean "no
	/// alignment tracking". This is what `encode_fast` runs on: same text
	/// operations, no offsets, no per-byte bookkeeping. (Empty strings count
	/// as aligned; both modes behave the same on them.)
	pub(crate) fn is_unaligned(&self) -> bool {
	self.alignments.is_empty() && !self.normalized.is_empty()
	}

On my machine this results in a +42% throughput improvement on llama-3 vs the base branch (would love a /benchmark to back it up)

cargo bench --bench ci_becnhmark -- 'llama3'

---

A possible next step would be to stop copying the String around at every slice() call but it's a much bigger and deeper change

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

/benchmark

[github-actions]

Python Benchmark Results

Commit: f5aa6c08ff481bdd7ce4cf185a7671b0acbcdf66

[github-actions]

Rust Benchmark Results

Commit: f5aa6c08ff481bdd7ce4cf185a7671b0acbcdf66