alphafold: Multimer predictions degraded after updating from 2.2.3 to 2.2.4?
Hello, this is ranking_debug.json from a test multimer run that I had performed with version 2.2.3:
{ "iptm+ptm": { "model_1_multimer_v2_pred_0": 0.8584136076107545, "model_2_multimer_v2_pred_0": 0.8055674020453231, "model_3_multimer_v2_pred_0": 0.6845710872128711, "model_4_multimer_v2_pred_0": 0.8423123043127994, "model_5_multimer_v2_pred_0": 0.8676285262204777 }, "order": [ "model_5_multimer_v2_pred_0", "model_1_multimer_v2_pred_0", "model_4_multimer_v2_pred_0", "model_2_multimer_v2_pred_0", "model_3_multimer_v2_pred_0" ] }
and this is from an equivalent job, carried out using the same exact input, after updating to 2.2.4:
{ "iptm+ptm": { "model_1_multimer_v2_pred_0": 0.40774006137839586, "model_2_multimer_v2_pred_0": 0.44277279108467366, "model_3_multimer_v2_pred_0": 0.442618376771567, "model_4_multimer_v2_pred_0": 0.5631646773883024, "model_5_multimer_v2_pred_0": 0.48998768122405234 }, "order": [ "model_4_multimer_v2_pred_0", "model_5_multimer_v2_pred_0", "model_2_multimer_v2_pred_0", "model_3_multimer_v2_pred_0", "model_1_multimer_v2_pred_0" ] }
Is anyone else experiencing the same? I am not getting any obvious error upon running 2.2.4, so it’s not clear to me why there should be such a significant difference…
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 33
Hi @lucajovine. Thanks very much for sharing this. These look very consistent to me so I would agree. I will close this issue but please reopen if the issue arises again. Best – Hamish
It’s OK, I’ll take anything!!
I’m beginning to suspect the old jax library was more consistent across GPUs, while the new jax library is not. Unfortunately, I don’t have access to any high-end GPUs (like A100), so I can’t do any tests on my side.
I think one informative test to do would be: run on 2.2.3 with A100 and non-A100, then repeat the same experiment with 2.2.4 on A100 and non-A100.
Hi thanks very much for this. I will investigate and get back to you.