Fix TypeError in apply_min_p when min_tokens_to_keep > 1 - #1600
Conversation
|
Thanks for the PR but at the moment the number of PRs is way beyond our capacity to review so I'm closing the non-essential ones so we can actually work on this repo. |
|
Could this one be reopened? It was closed in the capacity sweep rather than on the merits, and it is a hard crash, not a cleanup: any call of The fix is one token ( |
|
Sure thanks for looking into this, please excuse us there were just too many pull requests and it was easy to mistakenly close valid ones. |
The min_tokens_to_keep branch passes bare Python False as the values argument of mx.put_along_axis, whose binding requires an mx.array — the branch raises TypeError on every invocation, on every mlx version. Introduced by the ml-explore#1083 refactor (the pre-refactor code passed proper arrays); latent since because the parameter has no test coverage and mlx_lm.server does not expose it, so only direct library consumers of make_sampler(min_tokens_to_keep=...) can reach it. Fix: wrap the scalar (mx.array(False)). Adds a regression test that fails with the TypeError pre-fix and verifies keep-floor semantics (exactly min_tokens_to_keep survivors under an aggressive min_p), single-row and batched.
8ddf863 to
c2c0647
Compare
apply_min_p'smin_tokens_to_keep > 1branch passes bare PythonFalseas thevaluesargument ofmx.put_along_axis, whose binding requires anmx.array. The branch raisesTypeErroron every invocation:Verified the rejection is not mlx-version-dependent (probed mlx 0.31.2 and 0.32.0 — bare bool rejected on both). The scalar was introduced in #1083's refactor (the pre-refactor code passed proper arrays); it has stayed latent because
tests/test_sample_utils.pyhas nomin_tokens_to_keepcoverage andmlx_lm.serverdoesn't expose the parameter — only direct library consumers ofmake_sampler(min_tokens_to_keep=...)can reach the branch. We hit it in a production server built onBatchGeneratorthe first time a client set the knob.One-line fix:
False→mx.array(False). Adds a regression test that fails with theTypeErrorpre-fix and pins the keep-floor semantics (exactlymin_tokens_to_keepsurvivors under an aggressivemin_p), single-row and batched.