Password-based auth (C_Login) fails under OpenSSL 3 FIPS provider — PBKDF2 salt "Yubico" is below FIPS-compliant minimum

[slunker]

Summary

The hard‑coded PBKDF2 salt used to derive session keys from a password is 6 bytes ("Yubico"). OpenSSL 3's FIPS provider enforces 128‑bit (16‑byte) minimum salt, so the derivation fails and C_Login returns CKR_FUNCTION_FAILED whenever the host's OpenSSL is configured for strict FIPS. This makes password authentication (both symmetric and asymmetric) unusable in exactly the FIPS deployments the YubiHSM 2 FIPS is meant for.

Environment

• yubihsm-shell / yubihsm-pkcs11 2.7.1 (SDK bundle yubihsm2-sdk-2026-01, Ubuntu 22.04, amd64).
• OpenSSL 3.x with the FIPS provider active and the default provider not loaded (default_properties = fips = yes) — the configuration recommended in OpenSSL's fips_module(7).

Root cause

lib/yubihsm.h:
#define YH_DEFAULT_SALT "Yubico" // 6 bytes = 48 bits
#define YH_DEFAULT_ITERS 10000
Used by derive_key() in lib/yubihsm.c (via pkcs5_pbkdf2_hmac), which backs both yh_create_session_derived() and yh_util_derive_ec_p256_key().

Reproduction (no HSM needed)

Under a strict‑FIPS OpenSSL config:
$ openssl kdf -keylen 16 -kdfopt digest:SHA2-256 -kdfopt pass:password
-kdfopt salt:Yubico -kdfopt iter:10000 PBKDF2
error:...:kdf_pbkdf2_set_ctx_params:invalid salt length:.../pbkdf2.c
A ≥16‑byte salt succeeds. Via PKCS#11 this surfaces as CKR_FUNCTION_FAILED from C_Login.

Request

Changing the salt outright would break existing auth keys, so ideally:

• Make the PBKDF2 salt/iteration count configurable (per‑module config or env), and/or use a FIPS‑compliant salt for newly created auth keys with a documented migration path

[aveenismail]

@slunker Thank you for the report. We are now in the process of examining how to best approach this in a way that preserves backwards compatibility

[slunker]

Thanks! Btw I stumbled upon this while trying to use a normal (non-fips) YubiHSM, so it'd be great if the fix worked for all HSMs, not just the FIPS ones

[aveenismail]

I think we're on the same page here :)

[aveenismail]

@slunker I have now managed to set up an OpenSSL FIPS installation but I still managed to authenticate to the YubiHSM without problem. Further digging showed that the enforcement of the longer SALT is done by calling the derivation function the requires the longer SALT but I couldn't find a way to configure OpenSSL where only the function that requires longer SALT is allowed to be called. So authentication should not fail using neither yubihsm-shell nor yubihsm_pkcs11. Do you have steps where I can reproduce the failed authentication using either yubihsm-shell or yubihsm_pkcs11?

Re-reading the ticket, I see that the command that failed for you was the OpenSSL command, not when trying to authenticate to the YubiHSM. What is the use case that requires authentication with the raw derived keys rather than let yubihsm-shell or yubihsm_pkcs11 perform the derivation?