feat(epp): move conditional-decode gate into prefix-based-pd-decider

[dmitripikus]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Moves the RFC 7240 Prefer: if-available conditional-decode 412 gate out of the director's hard-coded check and into a configurable plugin behavior on the existing prefix-based-pd-decider.

Today the director hard-codes "reject when the primary decode endpoint has zero matched prefix blocks", which is expressed in tier-weighted blocks and cannot be tuned. Operators have no way to reject speculative decodes based on the actual token count that would need to be prefilled locally.

This PR:

• Adds PreRequest to PrefixBasedPDDecider. When a request carries Prefer: if-available and the non-cached suffix is at least nonCachedTokens (unweighted, so a RAM-cached prefix contributes its full length), the plugin returns errcommon.Error{Code: PreconditionFailed} — the director's runner preserves the typed code end-to-end so the 412 response is byte-identical to what the director used to emit.
• Honors promptTokens in the gate the same way disaggregate() does: prompts shorter than promptTokens skip both remote prefill and the 412 gate, so the operator's short-prompt shortcut applies uniformly.
• Removes primaryEndpointHasCachedPrefix and the inline gate block from pkg/epp/requestcontrol/director.go.
• Makes the gate opt-in: deployments without the plugin no longer emit 412 and always forward speculative decode requests.
• nonCachedTokens: 0 disables the gate (consistent with disaggregation being off).

Groundwork was landed separately in #2215 (adds error return to the PreRequest interface).

Which issue(s) this PR fixes:

Fixes #1686

Release note:

The conditional-decode 412 gate (RFC 7240 `Prefer: if-available`) is now enforced by the `prefix-based-pd-decider` plugin instead of a hard-coded director check. Deployments that declare the plugin enforce the gate at the configured `nonCachedTokens` threshold (unweighted cached-block count) and honor its `promptTokens` short-prompt bypass; deployments without the plugin no longer emit 412 and always forward speculative decode requests. To restore the previous "any weighted cache hit lets it through" behavior, set `nonCachedTokens: 1`.

[roytman]

There is another inherited issue:
The gate fails open, not closed, when the tokenizer hasn't run — getUserInputLenInTokens returns (0, nil) for a nil TokenizedPrompt (not an error). That makes nonCachedTokens = 0 - hitPrefixTokens, always below threshold, so the request is forwarded — the opposite of the "fails closed" claim in the PreRequest doc comment. This is reachable in production: director.go explicitly swallows DataProducer (tokenizer) failures, so a tokenizer backend hiccup silently disables the whole gate. The new test "no TokenizedPrompt: zero-token prompt trivially covered" (wantReject: false) locks this behavior in rather than catching it.

We should validate if we want to leave this behaviour or now.
The possible fix is:

func getUserInputLenInTokens(request *scheduling.InferenceRequest) (int, error) {
	if request == nil || request.Body == nil {
		return 0, errors.New("request or request body is nil")
	}
	if request.Body.TokenizedPrompt == nil {
		return 0, errors.New("prompt not tokenized")
	}
	return request.Body.TokenizedPrompt.TokenCount(), nil
}

One test needs to flip with it - prefix_based_pd_decider_test.go, "no TokenizedPrompt: zero-token prompt trivially covered":

{
    name:            "no TokenizedPrompt: cache state unknown, fails closed",
    nonCachedTokens: 5,
    headers:         preferIfAvailable,
    result:          resultWithEndpoint(makeTestEndpoint(0)),
    request:         completionsRequestWithPrompt(fwkrh.Prompt{}),
    wantReject:      true,
},

[dmitripikus]

Regarding the issue in getUserInputLenInTokens(): I fixed it, so that getUserInputLenInTokens now returns an error when TokenizedPrompt == nil. disaggregate still soft-fails to decode-only via its existing Error-log branch, and PreRequest returns 412 via its existing typed-error branch. Test is updated. Thanks!