graphql-engine: deadlock with metadata apply and running subscriptions

This was seen by a user (Howard Wang#8417 Discord) in v1.3.2 when applying metadata:

2020-10-20 18:31:42 UTC:172.17.5.63(54744):t5br6hpuBHCetN33jJczdXRvTwfFXjBpANW5@hasura:[4585]: DETAIL: Process 4585 waits for AccessExclusiveLock on relation 17646 of database 16396; blocked by process 458.

Process 458 waits for AccessShareLock on relation 17824 of database 16396; blocked by process 4585.

Process 4585: DROP FUNCTION IF EXISTS hdb_views.“notify_hasura_channel_post_created_INSERT”() CASCADE

Process 458: SELECT “_subs”.“result_id” , “_fld_resp”.“root” AS “result” FROM UNNEST(($1)::uuid[], ($2)::json[]) AS…

This is causing hasura metadata apply command to fail and hence rollback of the desired upgrade.

About this issue

Original URL
State: closed
Created 4 years ago
Reactions: 1
Comments: 16 (8 by maintainers)

Most upvoted comments

@markerdmann First thanks for working with us on this issue. We have found why DROP FUNCTION was being called needlessly and it turned out to be a CLI issue: https://github.com/hasura/graphql-engine/issues/6115

We will fix this ASAP in v1.3.3

tirumaraiselvan on Nov 2, 2020

I initially assumed it was related to Subscriptions because of this Issue title, but I tested it locally and those long-running queries still appear when I cut off Subscriptions. I’ll do some further investigation tomorrow to see if I can find the exact GQL that’s connected to those long-running queries.

Subscription queries look like SELECT "_subs"."result_id" ... whereas SELECT coalesce(json_agg("root"), '[]) ... are regular queries. If you have the spreadsheet saved from before, then just directly copying that query and running against your prod DB would be safe.

Thanks for sending the migrations/metadata, we will continue investigating what might be causing DROP FUNCTION to be executed in the first place.

tirumaraiselvan on Nov 1, 2020

@tirumaraiselvan This is what blocked state looks like for us. Let me know if you need additional detail.

markerdmann on Oct 30, 2020