pg_net: Webhooks fail when multiple inserts are done
Bug report
Describe the bug
I have a webhook configured to watch a table for inserts. On an insert a POST request should be fired to an API hosted on Vercel. This works when one insert operation is carried out. If multiple inserts are carried out: .insert([{message: 1}, {message: 2}, {message: 3}]) then the webhooks arenβt fired at all. If instead we carry out multiple inserts by calling the Supabase API multiple times then the webhook is fired for only some of the inserts.
To Reproduce
Steps to reproduce the behavior, please provide code snippets or a repository:
-
Configure a webhook on a table to watch for insert operations and call an external API
-
Use the Supabase JS library to perform multiple inserts like:
let { data, error } = await supabase .from("example_table") .insert([{message: 1}, {message: 2}, {message: 3}, {message: 4}, {message: 5}, {message: 6}, etc ]) .select("message"); -
Check logs for external API to confirm it didnβt receive any requests
-
See error
Something to note is that I was inserting 32 things at once.
Expected behavior
For each insert, I would expect the external API to be called. It would also help if there was a log in Supabase that showed what happens when the webhook is run.
Screenshots
If applicable, add screenshots to help explain your problem.
System information
- OS: N/A
- Browser (if applies): N/A
- Version of supabase-js: 2.0.0-rc.8
- Version of Node.js: v16.17.0 on desktop, 18.x on Vercel (per their docs)
Additional context
Add any other context about the problem here.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Reactions: 16
- Comments: 57 (14 by maintainers)
Commits related to this issue
- test: add cloud test for reproducing timeouts This uses NixOps wrapper scripts to reproduce https://github.com/supabase/pg_net/issues/86. The steps are: ``` $ net-cloud-deploy $ net-cloud-reproduce... — committed to steve-chavez/pg_net by steve-chavez 2 years ago
- test: add cloud test for reproducing timeouts This uses NixOps wrapper scripts to reproduce https://github.com/supabase/pg_net/issues/86. The steps are: ``` $ net-cloud-deploy $ net-cloud-reproduce... — committed to steve-chavez/pg_net by steve-chavez 2 years ago
- test: add cloud test for reproducing timeouts This uses NixOps wrapper scripts to reproduce https://github.com/supabase/pg_net/issues/86. The steps are: ``` $ net-cloud-deploy $ net-cloud-reproduce... — committed to supabase/pg_net by steve-chavez 2 years ago
Weβre experiencing the same problem. When inserting records in sequence all the webhooks are received. But if we insert 30 rows in parallel none of the webhooks send and the
net/_http_responsetable is full ofTimeout was reachederrors.Any new updates on this?
Iβm running into this issue as well. I spun up a clean environment to reproduce, and did some testing. Supabase project
qutskxfebtlgwyzsonvd. The setup is:userstablequeuetablequeuefor each userqueueinsert to an HTTPS endpoint, an AWS API Gateway200response (typically in <80ms)sleep(5)I ran a handful of tests (spreadsheet here, screenshot below). It looks like a pretty linear relationship between the number of requests (in my case, the number of
usersfor which the cron job inserts items into thequeue) and the webhook timeout configuration. By my math, the webhook needs about 33.33ms per item to process. If only 20ms per item is available within the webhook timeout configuration, most or all of the events fail to be sent to the API Gateway.Unlike the note from @lohrmann above, I did see some partial successes.
I did some testing in this area and found dropped requests due to ephemeral port exhaustion through docker networking. Limited testing of
pg_neton bare metal did not reproduce dropped requests.https://blog.sequinstream.com/benchmarking-pg_net-part-1/
pg_net has been super inconsistent. also tends to hang up sporadically. tried to build an event-driven app but decide to give up & rely on cron job instead
Any new updates on this?
By increasing the default timeout (
2 seconds) to4 seconds, all the above requests succeed on Supabase cloud:Will increase the default to 5 seconds.
The fix for this issue is now available on Supabase cloud π.
Make sure to check your pg_net version:
If you donβt have that version, you can upgrade by going to https://supabase.com/dashboard/project/_/settings/infrastructure or you can also create a new project.
Solved in https://github.com/supabase/pg_net/pull/139. Note that even at 1M requests we got no unexpected timeouts.
Issue still happening
Fantastic write up!
@steve-chavez Limited testing. With .10 and a 5 second timeout I can do a bulk insert of 1000 rows and send 1000 realtime broadcast endpoint calls from a row trigger function and pg_net with no error. (tried twice). With 2 second timeout it fails in bunches of 100 or so working then not working. With a 1 second time a batch of 100 failed at 30 (1 try and was first try).
I did not test at 5 seconds before so donβt know if it is working better or not.
Can anyone confirm that this still happens on pg_net
0.10.0? To confirm the version do:Iβm no longer able to reproduce this previous error (https://github.com/supabase/pg_net/issues/86#issuecomment-1886040988) on Supabase cloud. We recently upgraded our libcurl version, which likely cleared the bug.
Yup, basically the webhook behaviour is flakey. I havenβt tried it recently. I wanted to use this to build an event driven architecture but canβt without it being reliable. Ended up using QStash as a replacement for the webhook. Would love Supabase to have some kind of full featured event queue like SQS/Eventbridge/Service Bus.
I want to test this out locally, but I cannot get that version in the CLI. I upgraded my preview environment last night, linked locally, and restarted so I do have the latest available docker images.
Isnβt the docker image supposed to bring the newer version?
Is there any ETA on this? Itβs blocking me and I would need to do some workaround. Thanks!
This bug has been open for a year, and itβs really frustrating that issues like this are not properly addressed by the Supabase team. They are adding other advanced features like branching, but the core remains unstable, and I donβt see a significant effort to fix that. I migrated a relatively small project from Firestore/Firebase to Supabase because I believed in their marketing, and now I really regret this decision. Currently, Iβm paying more for significantly worse quality:
Webhooks are unstable. In my previous architecture, I heavily relied on Firestore triggers to perform some operations in the background. Now, they can randomly fail without any option to retry, which is a HUGE problem when something needs to happen in the background of our app.
Realtime functionality is useless; it only works fine when you have an active tab in the browser. Then, you need to figure out the whole mechanism for refetching data and reconnecting after you go from inactive. I donβt even want to mention how painful this is on mobile (React Native). With Firebase, I didnβt even have to think about it; it just worked out of the box.
Supervisor / pg_bouncer - Random failures when connecting to the DB from cloud functions / AWS Lambda (Connection terminated unexpectedly).
Upgrading your db? Okay, our app will be unavailable for minutes or hours, and then you need to figure out why something doesnβt work.
Storage - You can upload from RN, but first, you need to convert the entire file into base64 and then use ArrayBuffer.
But yes, Supabase is open-sourced. I guess Iβll tell my clients - our app doesnβt work properly, but at least you can read the source code.