snowflake-connector-python: SNOW-856569: Segmentation Fault in cache.py pickle dump

@sfc-gh-mkeller Glad the repro repo was useful.

I’ve tested with that repo against two commits - the fix from @peterallenwebb’s PR on this branch and the (at the time of writing this) tip of your current PR on this one. Didn’t get any segfaults in 20 runs for either of them via for run in {1..20}; do make test; done > results.txt.

I also did a quick test of 10 runs of dbt compile --threads 30 in our dbt project where we originally saw this error repro and also didn’t get any segfaults there.

Wouldn’t call my testing alone conclusive, but on previous builds that almost certainly would’ve triggered at least one segfault, so I’d be optimistic about the fixes at least.

In PR #1635 I try to fix the underlying issue. I’d greatly appreciate it if you guys could help me test the code ❤️ I forked @verhey’s repo (thanks a ton for the repro!) to verify that this works with my changes, but I’d like to get some more organic testing to be done as well!

I was able to avoid the segfault by replacing these lines:

https://github.com/snowflakedb/snowflake-connector-python/blob/v3.0.4/src/snowflake/connector/cache.py#L527-L528

with

                    with open(tmp_file, "wb") as w_file:
                        w_file.write(pickle.dumps(self))

I tried this after noting that the faulting thread reliably had the same native stack trace:

#3  PyDict_GetItemWithError (op=0xffffbb175d70, key=key@entry=0xdddddddddddddddd) at ../Objects/dictobject.c:1441
#4  0x00000000005b4e88 in save (self=self@entry=0xffff606027d0, obj=0xffffa4126770, pers_save=pers_save@entry=0) at ../Modules/_pickle.c:4385
#5  0x00000000005b5dbc in batch_dict_exact (self=self@entry=0xffff606027d0, obj=obj@entry=0xffffa41409b0) at ../Modules/_pickle.c:3281
#6  0x00000000005b6188 in save_dict (self=self@entry=0xffff606027d0, obj=obj@entry=0xffffa41409b0) at ../Modules/_pickle.c:3335
#7  0x00000000005b4cf8 in save (self=self@entry=0xffff606027d0, obj=0xffffa41409b0, pers_save=pers_save@entry=0) at ../Modules/_pickle.c:4323
#8  0x00000000005b5dbc in batch_dict_exact (self=self@entry=0xffff606027d0, obj=obj@entry=0xffff605fe350) at ../Modules/_pickle.c:3281
#9  0x00000000005b6188 in save_dict (self=self@entry=0xffff606027d0, obj=obj@entry=0xffff605fe350) at ../Modules/_pickle.c:3335
#10 0x00000000005b4cf8 in save (self=self@entry=0xffff606027d0, obj=0xffff605fe350, pers_save=pers_save@entry=0) at ../Modules/_pickle.c:4323
#11 0x00000000005b4094 in save_reduce (self=self@entry=0xffff606027d0, args=args@entry=0xffff60653ad0, obj=obj@entry=0xffffa413d960) at ../Modules/_pickle.c:4229
#12 0x00000000005b4b94 in save (self=self@entry=0xffff606027d0, obj=obj@entry=0xffffa413d960, pers_save=pers_save@entry=0) at ../Modules/_pickle.c:4472
#13 0x00000000005b5570 in dump (self=self@entry=0xffff606027d0, obj=obj@entry=0xffffa413d960) at ../Modules/_pickle.c:4519
#14 0x00000000005b5a74 in _pickle_dump_impl (module=module@entry=0xffffa411fbf0, obj=obj@entry=0xffffa413d960, file=file@entry=0xffff60602d10,

That suggested a problem with the CPython implementation of pickle.dump(). Perhaps part of the issue is that the object being pickled is modified by another thread during serialization?

I verified the fix using the repro project (https://github.com/verhey/snowflake-thread-repro) created by @verhey. I was able to trigger the segfault every few runs with that method, and found that with the change above there was no segfault in over 20 runs.

@A132770 This python stack trace, particularly with the thread stopped at line 528 in cache.py, fits the pattern exactly.

In our case we found it temporarily sufficient to roll back to before the cache change was implemented (2.7.9). Looking at some of our various containers looks like some slightly newer versions that include the cache change (e.g. 2.7.12) also did not manifest the segfaults – so it’s not quite clear to me at what version this issue started, and why it seemed to be ‘activated’ on 6/28. Hope this helps someone!

I did re-run my testing with your latest code in #1635 , @sfc-gh-mkeller, and found that the segfaults were fixed.

I can’t share our entire dbt project, but here’s an example repo that I’ve been able to get the error to repro in.