openssl: Undefined behavior around calls to generic functions like OPENSSL_LH_HASHFUNC

We started seeing a complaint from LLVM17 sanitizer via some application nominally using OpenSSL:

crypto/lhash/lhash.c:299:12: runtime error: call to function err_string_data_hash through pointer to incorrect function type 'unsigned long (*)(const void *)'
[...]/crypto/err/err.c:184: note: err_string_data_hash defined here
    #0 0x7fa569e3a434 in getrn [...]/crypto/lhash/lhash.c:299:12
    #1 0x7fa569e39a46 in OPENSSL_LH_insert [...]/crypto/lhash/lhash.c:119:10
    #2 0x7fa569d866ee in err_load_strings [...]/crypto/err/err.c:280:15
[...]

Strangely there were not other reports, despite the fact that obviously this idiom is used for various LHASH and in STACK also.

Unsurprisingly this seems to be impacting many other C-based projects whose authors are not especially thrilled either:

https://github.com/systemd/systemd/issues/29972 https://github.com/python/cpython/issues/111178 https://github.com/php/php-src/commit/ca22505c8ca02950385f523e6056a9e129cf2289

There seems also to even be some conversation in the original LLVM thread on whether or not this is truly UB: https://reviews.llvm.org/D148827

About this issue

  • Original URL
  • State: open
  • Created 3 years ago
  • Comments: 35 (32 by maintainers)

Commits related to this issue

Most upvoted comments

For historical context, that “weird dereference” was to deal with a compiler bug that crashed for various versions of Borland C on windows unless you used the explicit pointer dereference approach.

Keep in mind that lhash predates SSLeay - that code had been to a lot of unusual places long before it went into SSLeay.

Revert callbacks to be generic, and do type convesion in callbacks (large change set, but smaller than (1), easier to implement, eliminates type safety of various callbacks)

Is this option to make all the callbacks be void*-typed? I do not think this is viable. These callbacks are exposed publicly, and callers have relied on the callbacks being typed for a very, very long time. It would be unreasonable to pull the rug out from under the whole OpenSSL ecosystem and say that they’re void*-typed now.

That is, the C requirement to match the type signature when you call the function means that these two APIs are incompatible:

void sk_FOO_pop_free1(STACK_OF(FOO) *sk, void (*free_func)(FOO *));
void sk_FOO_pop_free2(STACK_OF(FOO) *sk, void (*free_func)(void *));

If you told the caller that the API is sk_FOO_pop_free1, they will write the function with a FOO* parameter. If you then change it to sk_FOO_pop_free2, you have broken their code.

For function pointers which are not exposed publicly, you can change the type signature all you like. But changing any public ones is an API break.

@nhorman Regarding https://github.com/openssl/openssl/issues/22896#issuecomment-1837250395, you need to keep reading a few sentences further.

n1256, section 6.3.2.3, paragraph 8 says:

A pointer to a function of one type may be converted to a pointer to a function of another type and back again; the result shall compare equal to the original pointer. If a converted pointer is used to call a function whose type is not compatible with the pointed-to type, the behavior is undefined.

(“Compatible” is defined in 6.2.7.)

n1256, section 6.5.2.2, paragraph 9 says:

If the function is defined with a type that is not compatible with the type (of the expression) pointed to by the expression that denotes the called function, the behavior is undefined.

Both paragraphs also appear in the C89 draft you linked to. See also https://github.com/openssl/openssl/issues/22896#issuecomment-1834579201 which paraphrased that rule.

@thesamesam , ok, I think I’m starting to get it now. CFI, as a run time validation tool for, among other things, indirect pointer calls, might modify some data in the function pointer for validation purposes, and if the type conversion isn’t made properly, CFI breaks. Even if the conversion is effectively a no-op for the purposes of our run time operation in isloation, other tools that rely on those conversions happening may break, and thats whats ubsan is warning about here.

Ok, thank you for the education. It would seem to me then, that the most reasonable solution (though its ugly), would be to modify DEFINE_LHASH_OF_EX such that the the definition of lh_##type##_new accept function pointers whos arguments are all void *, and perform the explicit cast to the proper type in the respective _hash function implementation.

Note: This is also applicable to the compare function that gets passed to lh_##type##_new

So, I’ll start this by admitting that I didn’t/don’t fully understand the issue here, but I did some reading, and I think I have a slightly better handle on it now, though I’m still struggling to see why this is a problem. For the benefit of future readers, I wanted to get my thoughts down here.

The facts (and fact-adjacent info) llvm’s ub sanitizer is complaining about undefined behavior. Particularly, it is complaining that this code in our hash table library:

static OPENSSL_LH_NODE **getrn(OPENSSL_LHASH *lh,
                               const void *data, unsigned long *rhash)
{   
    OPENSSL_LH_NODE **ret, *n1; 
    unsigned long hash, nn;
    OPENSSL_LH_COMPFUNC cf;

    hash = (*(lh->hash)) (data);    <===== Here
    *rhash = hash;     
... 

Is making a call through a function pointer where the prototype of lh->hash which is:

 unsigned long (*OPENSSL_LH_HASHFUNC) (const void *);

Doesn’t match the signature of the function it’s pointing to, which in this case is:

static unsigned long err_string_data_hash(const ERR_STRING_DATA *a)

Additional info/Questions

C89 seems pretty vague on this issue. It indicates in A.6.5.7 that pointers can be cast to objects, and vice versa, and makes no mention of how arguments are handled when that is done. 3.3.16 notes that assignments (which is further clarified in c99) that:

In simple assignment ( = ), the value of the right operand is converted to the type of the assignment expression and replaces the value stored in the object designated by the left operand

If we assume that parameters to function calls are handled as assignments, then the above should be considered safe in C89, as the pointer to void is converted to a pointer to ERR_STRING_DATA, and everything is ok.

C99 has significantly more to say about this, in section 6.5.2.2:

7 If the expression that denotes the called function has a type that does include a prototype,
the arguments are implicitly converted, as if by assignment, to the types of the
corresponding parameters, taking the type of each parameter to be the unqualified version
of its declared type. The ellipsis notation in a function prototype declarator causes
argument type conversion to stop after the last declared parameter. The default argument
promotions are performed on trailing arguments.

Which I think matches what we’re doing here as lh->hash has a prototype defined in the OPENSSL_LHASH structure (the type is OPENSSL_HASH_COMPFUNC), and so the arguments are implicitly converted, which is what we want.

So far, so good, the section on assignments (6.5.16) says, relevant to this operation, that an assignment has to conform to at least one of the following constraints:

1. the left operand has qualified or unqualified arithmetic type and the right has
arithmetic type (**We don't conform here, but it's not applicable because we're working with pointer types)**

2. the left operand has a qualified or unqualified version of a structure or union type
compatible with the type of the right (**We don't conform, but this isn't applicable as we're dealing with pointer types**)

3. both operands are pointers to qualified or unqualified versions of compatible types,
and the type pointed to by the left has all the qualifiers of the type pointed to by the
right (**This is applicable and I think we conform**)

4. one operand is a pointer to an object or incomplete type and the other is a pointer to a
qualified or unqualified version of void, and the type pointed to by the left has all 
the qualifiers of the type pointed to by the right (**this is applicable, and I think we conform**)

5. the left operand is a pointer and the right is a null pointer constant (**Not applicable**)

Speaking to items (3) and (4), if we assume that The assignment operator constraints apply to the type conversions, as suggested by 6.5.2.2, taking the const ERR_STRING_DATA pointer in the called function to be the left operand of the assignment expression, and the passed void * data as the right operand in the expression, they are both pointers to qualified versions of compatible types (at least it seems to me that const void * and const ERR_STRING_DATA * should be compatible), and they both meet the conditions of having all the same qualifiers. As such it seems this code meets the constraints (3) and (4), and so the call behavior should not be considered undefined behavior.

Given the above, I’m struggling to understand how this isn’t a conformant operation. Can someone elaborate?

the paragraphs you refer to are not relevant as the ‘prototype’ in question refers to the prototype in the function pointer type, which is still mismatched with the real function type, and hence, the wrong conversions happen at call time

Or is llvm really triggering on the fact that the getrn function calls the hash function like this:

hash = (*(lh->hash)) (data);

Rather than this:

hash = lh->hash(data);

which @levitte and I noted in #22375 is just deeply wierd. Section 6.5.3.2 of C99 expressly notes that indirection of a function pointer results in the function designator, and so is fine, but I could see how ubsanitizer might trip up on such an operation, if its instrumentation recorded that as a different type.

it’s not. see:

/tmp$ echo 'static void f(unsigned) {} void (*foo)(unsigned) = f;' > a.c; echo 'int main(void) { extern void (*foo)(int); (*foo)(1); foo(1); }' > b.c; clang -O3 -fsanitize=function a.c b.c && ./a.out 
a.c:1:23: warning: omitting the parameter name in a function definition is a C2x extension [-Wc2x-extensions]
    1 | static void f(unsigned) {} void (*foo)(unsigned) = f;
      |                       ^
1 warning generated.
b.c:1:43: runtime error: call to function f through pointer to incorrect function type 'void (*)(int)'
a.c: note: f defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior b.c:1:43 in 
b.c:1:54: runtime error: call to function f through pointer to incorrect function type 'void (*)(int)'
a.c: note: f defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior b.c:1:54 in 
/tmp$

Note that you need >= Clang 17 for this.

At least your last question should be easy to answer if @jwalch could be so kind to change this line and rerun the test.

This isn’t about alignment and is a true positive. A function with type void (T*) cannot be called through a function pointer of type void (*)(void *). You can do the cast, but when you call it, it needs to be cast back to the original type.

I don’t think the proposed fix is (necessarily) right either. This is actually a pretty thorny consequence of how OpenSSL’s STACK_OF(T) and LHASH_OF(T) types do type erasure. The question is whether functions like lh_TYPE_new, sk_TYPE_pop_free, and sk_TYPE_set_cmp_func are expected to take void*- or T*-based function pointers. You have to pick one or the other.

For STACK_OF(T), both the docs and long-standing practice force us to pick the T* version. The T* version is also more type-safe. LHASH_OF(T) is interesting because the docs actually imply a void* pointer. But looking at the code, I think this may just be a doc bug. The helper macro does define type-specific function pointers.

Now, how do you get it cast to the right type if you pick the T* one and do type erasure? It’s pretty thorny but you basically need to pass in type-aware thunks. Here’s what I did for BoringSSL. It’s pretty gross, but fixes the UB.

As you go to do that, you’ll quickly discover that the standard library qsort and bsearch functions are actually practically impossible to use to implement OpenSSL’s API. (They’re missing a void* userdata or context parameter to pass in a closure.) bsearch_r and qsort_r fix this, but are a portability nightmare. We ultimately just reimplemented both in the library to avoid UB.