segmentation violation getting RSA key on F40

[es-fabricemarie]

I've created a bug report for Google KMS PKCS11 library but it looks like it could be a libp11 issue: GoogleCloudPlatform/kms-integrations#28

I'm using the latest packages of Fedora 40. openssl-pkcs11-0.4.12-8.fc40.x86_64

The backtrace of the coredump is:

#0  0x00007fe7c81ca0d2 in pkcs11_getattr_alloc (ctx=ctx@entry=0x3f591e987bb78c92, session=4802441702199765720, 
    object=object@entry=7857815905065540909, type=type@entry=288, value=value@entry=0x7ffc7285e5e0, 
    size=size@entry=0x7ffc7285e5d8) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_attr.c:62
#1  0x00007fe7c81ca8b0 in pkcs11_getattr_bn (ctx=ctx@entry=0x3f591e987bb78c92, session=<optimized out>, 
    object=object@entry=7857815905065540909, type=type@entry=288, bn=bn@entry=0x7ffc7285e640)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_attr.c:92
#2  0x00007fe7c81d07df in pkcs11_get_rsa (key=0x70bda0)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:197
#3  0x00007fe7c81d0b50 in pkcs11_get_evp_key_rsa (key=0x70bda0)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:265
#4  0x00007fe7c81cea12 in pkcs11_get_key (key0=key0@entry=0x70bda0, object_class=<optimized out>)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_key.c:456
#5  0x00007fe7c81ceaaa in pkcs11_rsa (key=0x70bda0) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:34
#6  pkcs11_get_key_size (key=0x70bda0) at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:332
#7  pkcs11_private_encrypt (padding=1, key=0x70bda0, to=0x72ae50 "\232\326o", 
    from=0x741940 "010\r\006\t`\206H\001e\003\004\002\001\005", flen=51)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:91
#8  pkcs11_rsa_priv_enc_method (flen=51, from=0x741940 "010\r\006\t`\206H\001e\003\004\002\001\005", 
    to=0x72ae50 "\232\326o", rsa=<optimized out>, padding=1)
    at /usr/src/debug/openssl-pkcs11-0.4.12-8.fc40.x86_64/src/p11_rsa.c:384
#9  0x00007fe7c7dbfd86 in RSA_sign (type=<optimized out>, 
    m=m@entry=0x7ffc7285ebb0 "\372\b\334r\022\b\232\357\320̈́\232dW1,\267\304B軅\342\373\230\214,Z\201\266A\n", 
    m_len=m_len@entry=32, sigret=sigret@entry=0x72ae50 "\232\326o", siglen=siglen@entry=0x7ffc7285eb44, 
    rsa=rsa@entry=0x6f7820) at crypto/rsa/rsa_sign.c:307
#10 0x00007fe7c7dc2a91 in pkey_rsa_sign (ctx=0x6ff9c0, sig=0x72ae50 "\232\326o", siglen=0x7ffc7285ec50, 
    tbs=0x7ffc7285ebb0 "\372\b\334r\022\b\232\357\320̈́\232dW1,\267\304B軅\342\373\230\214,Z\201\266A\n", tbslen=32)
    at crypto/rsa/rsa_pmeth.c:178
#11 0x00007fe7c7d4f91b in EVP_DigestSignFinal (ctx=<optimized out>, sigret=0x72ae50 "\232\326o", siglen=0x7ffc7285ec50)
    at crypto/evp/m_sigver.c:677
#12 0x00007fe7c7da0e04 in PKCS7_SIGNER_INFO_sign (si=si@entry=0x70d780) at crypto/pkcs7/pk7_doit.c:934
#13 0x00007fe7c7da2025 in do_pkcs7_signed_attrib (mctx=<optimized out>, si=0x70d780) at crypto/pkcs7/pk7_doit.c:711
#14 PKCS7_dataFinal (p7=p7@entry=0x6f02f0, bio=bio@entry=0x5e87c0) at crypto/pkcs7/pk7_doit.c:833
#15 0x0000000000403103 in IDC_set (p7=p7@entry=0x6f02f0, si=si@entry=0x70d780, image=<optimized out>) at idc.c:216
#16 0x0000000000402947 in main (argc=<optimized out>, argv=<optimized out>) at sbsign.c:274

I'm happy to help with the debugging/testing.

Thanks.

[es-fabricemarie]

with some old fashion debugging printf I get this:

DEBUG DEBUG In pkcs11_get_session(): ctx is at 0x7405f0
DEBUG DEBUG In pkcs11_get_session(): ctx->method is at 0x7f38cf2fe220
DEBUG DEBUG In pkcs11_get_session(): ctx->method->C_GetSessionInfo function at 0x7f38cdcd799e
DEBUG DEBUG In pkcs11_get_session(): returning
DEBUG DEBUG: in pkcs11_get_evp_key_rsa(), before pkcs11_get_rsa(), key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_rsa, before pkcs11_get_session(), key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG In pkcs11_get_session(): ctx is at 0x7405f0
DEBUG DEBUG In pkcs11_get_session(): ctx->method is at 0x7f38cf2fe220
DEBUG DEBUG In pkcs11_get_session(): ctx->method->C_GetSessionInfo function at 0x7f38cdcd799e
DEBUG DEBUG In pkcs11_get_session(): after slot session head manipulation: ctx is at 0x7405f0
DEBUG DEBUG In pkcs11_get_session(): after slot session head manipulation: ctx->method is at 0x7f38cf2fe220
DEBUG DEBUG In pkcs11_get_session() after slot session head manipulation: ctx->method->C_GetSessionInfo function at 0x7f38cdcd799e
DEBUG DEBUG In pkcs11_get_session(): returning
DEBUG DEBUG: in pkcs11_get_rsa, after pkcs11_get_session(), key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_rsa, after pkcs11_getattr_bn-1, key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_rsa, we're in 'success:', key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_rsa, in 'success:', after pkcs11_put_session key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_rsa, in 'success:', after pkcs11_put_session key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_rsa RETURNING key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_evp_key_rsa(), after pkcs11_get_rsa(), key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x7f38cf2fe220
DEBUG DEBUG: in pkcs11_get_evp_key_rsa(), before pkcs11_get_rsa(), key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x2e3031342e322e31
DEBUG DEBUG: in pkcs11_get_rsa, before pkcs11_get_session(), key=0x87bbc0 slot=0x860390 ctx=0x7405f0 methods=0x2e3031342e322e31
DEBUG DEBUG In pkcs11_get_session(): ctx is at 0x7405f0
DEBUG DEBUG In pkcs11_get_session(): ctx->method is at 0x2e3031342e322e31

I believe the methods pointer should always stay the same for a specific ctx? Here it gets changed somehow.
As soon as the lib calls C_GetSessionInfo on the "modified" ctx->methods pointer it sigsevs.

[es-fabricemarie]

On Fedora 40, when I do something like this:

    echo "We are going to use Fedora 39 openssl lib just to sign the binary"
    mkdir /var/tmp/fedora-39-ssl-libs/
    gunzip fedora39_libcrypto.so.3.gz
    gunzip fedora39_libssl.so.3.gz
    mv fedora39_libcrypto.so.3 /var/tmp/fedora-39-ssl-libs/libcrypto.so.3
    mv fedora39_libssl.so.3 /var/tmp/fedora-39-ssl-libs/libssl.so.3
    export LD_LIBRARY_PATH=/var/tmp/fedora-39-ssl-libs/

Then everything works and there is no segmentation violation.

The difference of OpenSSL versions are as such:

Fedora 39:

• openssl-libs-3.1.1-4.fc39.x86_64

Fedora 40:

• openssl-libs-3.2.1-2.fc40.x86_64