taichi: pip-installed Taichi crashes on Google colab kernels

FINALLY!!! I identified the problem! Colab kernels have a libtcmalloc library installed and env variable LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 set. Somehow it causes libstdc++ to use libunwind instead of libgcc_s for stack unwinding on exception. For some reason this causes abort during unwinding complex calls.

Running LD_PRELOAD= python t.py, where t.py is some taichi program works, even on GPU kernels. I’m looking for a way to make work inside colab cells as well.

@ppwwyyxx Thanks for pointing this out. I agree that closing stale issues using bots is not a good idea, and will prevent further misuse like this.

@znah After some searching, it turns out that we are now blocked at https://github.com/taichi-dev/taichi/issues/1059 - if we can remove all C++ exceptions (which I believe is necessary), then the system will not involve libunwind and we can run Taichi on colab. It may take some time for people (@sjwsl and @lin-hitonami) to fully remove throw IRModified etc. - if you’d like to help that would be awesome!

@strongoier I stand corrected, it appears the ‘minimal’ taichi code I was using was incorrect (though the lack of error messages makes things a bit hard to decipher). Apologies for pinging you all, seems to work well now 😃 Excited to try taichi out

FYI and off-topic: this opinion from pytorch author: https://twitter.com/soumithchintala/status/1451213207750721538 may lead the maintainers to reconsider whether it’s a good idea to “auto-close stale issues”. I personally agree with his opinion. What’s more valid (and also used in projects I maintained) is to auto-close invalid issues (e.g. those missing necessary information).

Sorry about that. The bitcode loading issue should be fixed in v0.5.6. The buildbots are currently working on compiling/releasing the new version.

I made a workaround, its pretty ugly, but it makes Taichi run in Colab notebook cells! https://colab.research.google.com/github/znah/notebooks/blob/master/taichi_colab.ipynb https://twitter.com/zzznah/status/1232321076014788608

I’d like to reopen this issue. The problem is still there, and I think supporting colab environment would greatly increase Taichi user adoption.

Interesting observation from the Colab team: Taichi works when using tcmalloc_minimal instead of tcmalloc. Relevant bits of documentation:

To use TCMalloc, just link TCMalloc into your application via the "-ltcmalloc" linker flag.

You can use TCMalloc in applications you didn't compile yourself, by using LD_PRELOAD:

   $ LD_PRELOAD="/usr/lib/libtcmalloc.so" 
LD_PRELOAD is tricky, and we don't necessarily recommend this mode of usage.

TCMalloc includes a heap checker and heap profiler as well.

If you'd rather link in a version of TCMalloc that does not include the heap profiler and checker (perhaps to reduce binary size for a static binary), you can link in libtcmalloc_minimal instead.

also this

NOTE: When compiling with programs with gcc, that you plan to link
with libtcmalloc, it's safest to pass in the flags

 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free

when compiling.  gcc makes some optimizations assuming it is using its
own, built-in malloc; that assumption obviously isn't true with
tcmalloc.  In practice, we haven't seen any problems with this, but
the expected risk is highest for users who register their own malloc
hooks with tcmalloc (using gperftools/malloc_hook.h).  The risk is
lowest for folks who use tcmalloc_minimal (or, of course, who pass in
the above flags :-) ).

I’m continuing the investigation.

The real way to rectify this issue is to fix a bug somewhere in either clang, or in (nongnu) libunwind, or in tcmalloc. I don’t feel like being capable to do this. I’ll discuss potential solutions with the Colab team.

It’s even trickier. I suspect some ABI incompatibility between clang and libunwind, that manifests itself only on unwinding complex virtual calls. So quite few programs are probably affected.