openssl: [OpenSSL 1.1.0] Segfault in EC_POINT_is_on_curve when curves mismatch

There seems to be a problem in OpenSSL 1.1.0 that produces a segmentation fault when EC_POINT_is_on_curve checks whether a point (from some given curve) is on a different curve. In most cases it correctly returns a 0 (meaning that the check is false since the point is trivially not in the curve), but for some pairs of curves (e.g., a point from curve_nid=715 is checked against curve_nid=714) it segfaults. So far it seems it affects version 1.1.0, but not 1.0.2 and 1.1.1.

Code that produces the segfault

// gcc -Wall segfault.c -o segfault -lcrypto
// gcc (Ubuntu 7.3.0-16ubuntu3) 7.3.0

#include <stdio.h>
#include <stdlib.h>
#include <openssl/ec.h>
#include <openssl/bn.h>

int main(int argc, char** argv)
{
    
    const EC_GROUP *group = EC_GROUP_new_by_curve_name(715);
    const EC_POINT *gen = EC_GROUP_get0_generator(group);

    const EC_GROUP *wrong_group = EC_GROUP_new_by_curve_name(714);
    
    BN_CTX *ctx = BN_CTX_new(); 
    int res = EC_POINT_is_on_curve(wrong_group, gen, ctx);
    printf("res: %d\n", res);
    BN_CTX_free(ctx);

    return 0;
}

Additional info

We have reproduced the problem in at least 5 different machines with different OS (all Unix based). The following information is particular to my machine.

david@david-NUC7i7BNHX:~/dev/pruebas$ openssl version -a
OpenSSL 1.1.0g  2 Nov 2017
built on: reproducible build, date unspecified
platform: debian-amd64
compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/lib/ssl\"" -DENGINESDIR="\"/usr/lib/x86_64-linux-gnu/engines-1.1\"" 
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"

david@david-NUC7i7BNHX:~/dev/pruebas$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04 LTS
Release:	18.04
Codename:	bionic
david@david-NUC7i7BNHX:~/dev/pruebas$ uname -a
Linux david-NUC7i7BNHX 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Backtrace

(gdb) run
Starting program: /home/david/dev/pruebas/segfault 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:141
141	../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:141
#1  0x00007ffff7a0ad6a in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#2  0x00007ffff7a0ae01 in BN_mod_mul_montgomery () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#3  0x00007ffff7a8efb8 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
#4  0x0000555555554885 in main ()
(gdb) 

Credit John Pacific (@tuxxy), Justin Myles Holmes (@jMyles), Michael Egorov (@michwill) and myself (David Nuñez, @cygnusv)