SIGSEGV in getopt_long inside C_Initialize via pkcs11-provider chain — ELF DSO global symbol mismatch (x86_64 only)

[josephtate]

Related to #524, which identifies the symptom. This report provides the root cause with debug symbols and a core dump.

Environment

• Package: yubihsm-shell-2.6.0-1.el9.core.x86_64 (Rocky Linux SIG/Core)
• Trigger: Linux kernel sign-file using OpenSSL's pkcs11-provider (OSSL module) to load yubihsm_pkcs11.so
• Architecture: x86_64 crashes; aarch64 does NOT crash through the identical load chain. The DSO global symbol mismatch appears to be resolved consistently on aarch64 but not on x86_64 — the bug depends on architecture-specific ELF dynamic linker symbol interposition behaviour.

Root cause

C_Initialize (yubihsm_pkcs11.c:111) calls cmdline_parser_internal (gengetopt-generated, line 885), which calls getopt_long. When yubihsm_pkcs11.so is loaded transitively through the OpenSSL pkcs11-provider chain:

sign-file
  → libcrypto.so.3
    → ossl-modules/pkcs11.so  (pkcs11-provider passes INIT_ARGS via C_Initialize)
      → yubihsm_pkcs11.so::C_Initialize
          → cmdline_parser_internal
              → getopt_long  ← CRASH

getopt_long uses process-global state: optarg, optind, opterr, optopt. When yubihsm_pkcs11.so is loaded as a transitive dependency, these globals can resolve to a different DSO copy than the one libc's getopt_long writes to. Writes go through a stale/wrong pointer, corrupting the stack. The fault manifests at libc.so.6+0xf457d — a bad dereference inside getopt_long a few instructions before its call to the internal helper match_prefix.

Evidence of stack corruption: a return address on the stack was overwritten with a value pointing into yubihsm_pkcs11.so's .rodata section (VA 0x68b52, which falls in [0x62000–0x69200]), and rbp=0x2 at crash time.

Note: there is no re-entrant getopt_long_r on Linux/glibc. Even if one existed, the fundamental problem is the symbol interposition: any option parser that references the global optarg/optind symbols would be susceptible to the same mismatch. The fix needs to eliminate the global state dependency entirely.

Call chain (confirmed with debuginfo)

C_Initialize   yubihsm_pkcs11.c:111   yubihsm_pkcs11.so+0x17770
  +345: call cmdline_parser_internal   yubihsm_pkcs11.c:885
    +224: call getopt_long@plt         ← global symbol collision
              SIGSEGV at libc.so.6+0xf457d (inside getopt_long,
              just before callq to match_prefix at +0xf4585)

Packages

• Crashing: https://dl.rockylinux.org/pub/sig/9/core/x86_64/core-infra/Packages/y/yubihsm-shell-2.6.0-1.el9.core.x86_64.rpm
  Build-id: 52bad84ecc864085f949b162c3e946b2e9578467
• Debuginfo: https://dl.rockylinux.org/pub/sig/9/core/x86_64/core-infra-debug/Packages/y/yubihsm-shell-debuginfo-2.6.0-1.el9.core.x86_64.rpm

Suggested fix

Replace the getopt_long-based INIT_ARGS parser in C_Initialize with a simple key=value string parser that uses only local/stack variables. The gengetopt-generated cmdline_parser_internal is the proximate problem; a 20-line strchr/strncmp parser for the small number of options that INIT_ARGS supports would have no global state and no PLT calls into libc's option machinery.

Full crash dump and some AI analysis

yubihsm_pkcs11_segfault.txt