github-task-list-completed: Check seems to be stalling lately (GitHub API checks.create slowness)
I’ve noticed this a few times. Checking a box triggers a check and I end up in this limbo state where the check never completes
task-list-completed Expected — Waiting for status to be reported
Checking/unchecking/adding comments doesn’t seem to retrigger anything. I can share a repo if that would help.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 6
- Comments: 97 (58 by maintainers)
Commits related to this issue
- feat(logs): added github response statuses to logs to help debug issues such as #25 — committed to stilliard/github-task-list-completed by stilliard a year ago
- feat(logs): added X-GitHub-Request-Id: and context id, header requested by github to help with debugging ref #25 — committed to stilliard/github-task-list-completed by stilliard 8 months ago
Had some time tonight to play around with the new idea.
I’ve set up a new load balancer and a couple nodes for now (may add more as needed) dedicated to this app.
These are now in San Francisco, which may perform better than the uk location previously on DO. That said, just doing some searching around, it might be that githubs servers are in Virginia so moving the servers to the DO new york datacenter may actually be better but I can do that another night.
Monitoring it atm and it seems to be running fine however this time of day is often has less traffic for the service anyway, the real test will be tomorrow onwards.
Let me know if you notice any issues, also worst case I can switch it back to the old setup with a single url setting change in github just in case as well.
Been a couple weeks and not had any further issues. I think we’re good to close this one off again. If anyone runs into the same issue feel free to reply or open a new ticket any time. Thanks!
We have also been experiencing this issue. In the case of the screenshot below, this PR is approved and all tasks are checked. Removing a task or re-checking it does not seem to “kick” it.
Is there a workaround in place? Perhaps merging the main brach back into the PR will re-check the task status?
Hi @Walavouchey
No reply back from Github yet sorry.
Here’s the logs for your check:
all logs show it was good and we sent to github, but no reply from them yet.
Np, sorry it’s still causing issues.
Ah yes the checks stay in the
in_progressstate if there’s checklists waiting to be completed rather than saying it failed. It doesn’t actually stall or hold up our side as in progress, instead we just tell github it’s in progress and we cleanup and go on to the next request, each normally completing in under a second but we do end up waiting when the requests back to github take a while to complete. Thein_progressstate pattern rather than failing we borrowed from the WIP app so that any other checks that fail show as a full fail and you can tell the difference between actual fails vs missing tasks as they are still in progress.My hope is once we have the new logs github can see why sometimes they end up queuing their side.
I have a change I need to make my side to help with reporting this to github, they want me to add additional logs for the responses. As far as we can see it’s an issue getting through to github but often by the time their support replies it’s resolved so they need some ids from the response headers from their requests to help debug.
I’ll see what I can do to get those in later today.
I’ll give it a try if it happens again, thank you! We really appreciate using the functionality in our team 👍
Hi @seanweiss91 I’m seeing the same today too, checked the server and we’re receiving the request from github right away and sending it back to them in often in under a second or 2, but, Github seems to be having delays again as the request we send back to them to save the check response is often taking multiple minutes.
What can help is updating the PR quickly, like unchecking, waiting a few seconds and checking again, or updating the name by adding a full stop or similar. These little updates don’t speed anything up our side, but they trigger another request, which we again send to github and what I’ve seen is github will on some requests be slow, but others faster. Meaning sometimes, that 2nd request will complete way before the first one 😅 .
Hope this helps, I’ll try reporting to Github again soon if it keeps up though.
Hey @dave-yotta Sorry for not replying sooner! I’ve been away for about a week but back shortly. I’ll take a look into the debug tools they’ve sent early next week, thanks for chasing them about it.
Thanks for reporting it to github, let me know if they reply asking for anything about the checks code etc. it’s all open source on here if they want to take a look too.
Oh…! Ok, If it happen next time I will wait for enough time to make sure. Thank you ^^
Hi @namse
Hmm, looks ok atm for me but occasionally Github can still be slow. Maybe give it a little extra time for Github to realise we’ve sent it back to them lol
Here’s the log from our side for the last hour for that PR ID:
Sep 06 14:50:31 INFO (probot): PR #323: Request received Sep 06 14:50:31 INFO (probot): PR #323: Main comments api lookup complete Sep 06 14:50:31 INFO (probot): PR #323: Review comments api lookup complete Sep 06 14:50:32 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 14:50:32 INFO (probot): PR #323: Complete and sending back to github Sep 06 14:54:34 INFO (probot): PR #323: Request received Sep 06 14:54:34 INFO (probot): PR #323: Main comments api lookup complete Sep 06 14:54:34 INFO (probot): PR #323: Review comments api lookup complete Sep 06 14:54:35 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 14:54:35 INFO (probot): PR #323: Complete and sending back to github Sep 06 15:00:53 INFO (probot): PR #323: Request received Sep 06 15:00:53 INFO (probot): PR #323: Main comments api lookup complete Sep 06 15:00:53 INFO (probot): PR #323: Review comments api lookup complete Sep 06 15:00:54 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 15:00:54 INFO (probot): PR #323: Complete and sending back to github Sep 06 15:01:02 INFO (probot): PR #323: Request received Sep 06 15:01:02 INFO (probot): PR #323: Request received Sep 06 15:01:02 INFO (probot): PR #323: Main comments api lookup complete Sep 06 15:01:02 INFO (probot): PR #323: Main comments api lookup complete Sep 06 15:01:02 INFO (probot): PR #323: Review comments api lookup complete Sep 06 15:01:03 INFO (probot): PR #323: Review comments api lookup complete Sep 06 15:01:03 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 15:01:03 INFO (probot): PR #323: Complete and sending back to github Sep 06 15:01:03 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 15:01:03 INFO (probot): PR #323: Complete and sending back to github Sep 06 15:20:56 INFO (probot): PR #323: Request received Sep 06 15:20:57 INFO (probot): PR #323: Main comments api lookup complete Sep 06 15:20:57 INFO (probot): PR #323: Review comments api lookup complete Sep 06 15:20:58 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 15:20:58 INFO (probot): PR #323: Complete and sending back to github Sep 06 15:21:12 INFO (probot): PR #323: Request received Sep 06 15:21:13 INFO (probot): PR #323: Main comments api lookup complete Sep 06 15:21:13 INFO (probot): PR #323: Review comments api lookup complete Sep 06 15:21:13 INFO (probot): PR #323: Diff comments api lookup complete Sep 06 15:21:13 INFO (probot): PR #323: Complete and sending back to github
Hey, So Github has updated their status page and the PR issues appear resolved which is awesome!
To help I’ve made a couple tweaks to retry requests based on how the WIP check works. I’ve also upgraded the repo to the latest probot release so that any other upstream fixes to requests from github are applied as well to be safe.
Hopefully the combination works well, testing on a couple repo’s the updates run on the server side in less than a second still but on the github UI they are updating in a few seconds too.
If there’s any further issues with checks not completing then please check to see if github is having issues over on https://www.githubstatus.com/ and if nothing is reported please either comment here or start a new issue if needed. :octocat:
Thanks for sending those over. Yep external service but what’s weird is the server is fine, barely using a quarter of it’s resources though today.
Here’s the logs for those ID’s from the last 3 hours: (UK timezone)
Looks like both were sent to us, we checked the PR and it’s comments (using the 3 api endpoints for comments) and then the done call is when we sent back to github to say if it’s all ok or not.
No warnings or errors in the logs around these times either.
I think I need to raise another ticket with Github about this as for I’ve seen this a number of times now where the checks are hitting the server fine, checks running quick and we respond back to say it’s done fast too but then no update on the Github UI itself. I’m not sure if they just aren’t receiving the responses properly sometimes or they queue them and for some checks they are prioritised higher than ours maybe? Will contact them and find out what’s up.
Hi @vamcs
Might be worth trying though normally editing the name or body of the PR will help send it over to us again.
From my debugging, normally github send it to us quickly, but then we make several API requests back to github to check the comments, review comments & diff comments of the review, lately I’ve noticed these can at random times take a while for github to respond on.
My checks seem to be running ok for me atm but could be an intermittent issue.