annoy: Initializing an AnnoyIndex crashes on AMD processors

I was trying to get a pip-installed Annoy v1.16.3 running on an m5a.large AWS Instance, but it crashed with Illegal instruction (core dumped) trying to execute

annoy.AnnoyIndex(512, "angular")

This doesn’t occur on Intel-based m5.large Instances.

Doing some digging (details below), it seems to be because Annoy requires the AVX-512 instruction set, which isn’t available on AMD processors.

m5a.large’s AMD EPYC 7000 series processor supports these instruction sets (from /proc/cpuinfo):

fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save

The Annoy library seems to use these instruction sets (using this utility):

    MODE64 (call)
    AVX512 (vmovdqa64)
    VLX (vmovdqa64)
    AVX2 (vextracti128)
    AVX (vmovups)
    NOVLX (vmovups)
    CMOV (cmovns)
CPU Generation: Unknown

About this issue

Original URL
State: open
Created 4 years ago
Comments: 34

Commits related to this issue

Update setup.py avoid illegal instruction per https://github.com/spotify/annoy/issues/472 — committed to Qiao126/annoy by Qiao126 5 months ago

Most upvoted comments

For anyone that has issues with this and builds annoy in docker, I used this line to override the ANNOY_COMPILER_ARGS environment variable per @erikbern’s suggestion. Thanks!

ENV ANNOY_COMPILER_ARGS -D_CRT_SECURE_NO_WARNINGS,-DANNOYLIB_MULTITHREADED_BUILD,-mtune=native

+12

mmaybeno on Aug 5, 2020

Any news on this? im having the same issue.

conradogarciaberrotaran on May 5, 2020

If anyone wants to give it a shot and try to fix this, I think the best approach is something like

Make the distance function calculation a template argument of AnnoyIndex (but not AnnoyIndexInterface)
At the point of instantiation of the index https://github.com/spotify/annoy/blob/master/src/annoymodule.cc#L153 – add (a) and #ifdef that checks if AVX is supported (b) checks if (__builtin_cpu_supports("avx2")) in runtime

Maybe there’s other solutions too!

erikbern on Aug 5, 2020

Hello may i know the exact line that you used to install annoy in docker? Not sure how to pass this argument into my dockerfile

It has been a few years, and there is probably a better way to do this now. But originally included these in my dockerfile

FROM python:3.7.5-slim-stretch

ENV ANNOY_COMPILER_ARGS -D_CRT_SECURE_NO_WARNINGS,-DANNOYLIB_MULTITHREADED_BUILD,-mtune=native

From there you can include anything else in your dockerfile including installing annoy.

mmaybeno on Mar 11, 2022

This may be the problem I have been seeing on our setup as well. Everything is built in docker images, so originally I wasn’t sure why I was getting failures. And since Annoy is integrated into our sagemaker pipeline, failures were some what opaque.

Plus, images built in our CI/CD pipeline were the only instances which would cause the error, but local docker builds would be fine. I’ll experiment with these flags to see if the issue is resolved. Side note, the only reason why I guess it was a problem with annoy was comparing the two docker images with https://github.com/GoogleContainerTools/container-diff.

mmaybeno on Aug 1, 2020