lighthouse-check-action: PR commenting not working for me (Help?)

No problem @bhatvikrant - glad I could kinda help at least. I did some research and it seems like using a comment URL to be associated with commit is the only straightforward option (for now at least) when using the deployment_status event. The reason is that the deployment_status event doesn’t associate a PR. With that said, I’ll need to close this and hopefully the concluding thoughts below can help others.

Conclusion

When using the deployment_status event to trigger a workflow, Lighthouse Check Action PR comments will not work out of the box because under the hood we construct use github.context.payload.pull_request.url to post comments unless specified by the user. deployment_status event doesn’t provide pull request data.

Solution with Caveat

In order to enable comments, you would need to populate commentUrl, similar to our example, to comment on the associated commit. There is a caveat to this in that if you have an open PR, you will have a new comment with Lighthouse results for every commit, which can get noisy. This is different from using the pull_request event, which by default only posts one comment and updates it for each new commit after.

Possible Hacky Solutions for PR Comments

• Use GitHub API to get associated PRs to the commit as explained here. The issue with that is you could have multiple PRs and you would somehow need to choose one if so. This could work for small projects in a case that you only have one PR open at a time. You would then need to use this PR # to construct a URL per the PR review API and populate commentsUrl with it.
• I haven’t really solidified this idea, but apparently there is a way to share artifacts between workflows. Maybe there could be a way to somehow share a corresponding PR, but I’m not really sure how that could work… just a half-baked idea. Also, my source is Stack Overflow, so probably not the best idea.

No problem - and thanks for reporting this one.

Oh shoot, midnight hours got me hallucinating that I couldn’t even see this blunder. Thank you once again @adamhenson.

Hey @bhatvikrant - but if you’re using a different repo than before then why do you still have bhatvikrant/prep-station as part of the commentUrl. Maybe you should just change it to this:

commentUrl: https://api.github.com/repos/${{ github.repository }}/commits/${{ github.sha }}/comments

Yep, okay looks like it will just keep creating new comments. So, this method will work, but your PR will get flooded.

If you were to do this on a PR triggered action, then only one comment would show and it would keep editing it.

I don’t know if there is a better solution at this time, but I’ll take some more time on it and get back to you.

So, it created a new commit

Great, no prob - I just want to see if the 2nd gets edited or if a new comment is created 🤞

Okay, awesome - looks like it worked @bhatvikrant

Can you do 2 more things.

1. Remove that 1st comment with the scores, but keep the last?
2. Push up an empty commit to trigger the action again.

In theory there should only be one comment that is edited when additional commits are pushed. I want to see if this works as expected.

Interestingly, that does seem to work!

Alrighty, I ve added that.

Thanks @bhatvikrant. Regarding the docs, that’s something we’ll have to do because it will require in-depth knowledge of this project. It’s on the roadmap though… we’ll get there.

Okay, well that’s interesting. I think this is a clever way of populating the URLs and I’m not so sure how we can get this to work.

I wonder if you do the following - it will show up on PRs 🤔 . Want to give it a shot?

with:
  accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
-  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/pulls/2/reviews
+  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/commits/${{ github.sha }}/comments
  outputDirectory: /tmp/artifacts
  urls: '${{ github.event.deployment_status.target_url }}'

Hi @bhatvikrant. Sorry for the troubles and thanks for providing all those details. Our documentation isn’t that great, especially on this subject. There are plans to improve. The intention of this feature was to comment on PRs, however I feel this may need to be revisited to be more robust.

My theory is that because you are not triggering this on pull_request it isn’t working correctly (even though you have a PR open). Unless you explicitly specify commentUrl, then this GitHub Action will use context.payload.pull_request.url, which I’m guessing will be empty if not triggered by a pull_request. Your example is triggered by deployment_status.

To test my theory, could you make this change? Yes, we’re hard-coding the PR, but this will at least isolate the issue (hopefully), and then we can brainstorm on next steps.

with:
  accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
+  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/pulls/2/reviews
  outputDirectory: /tmp/artifacts
  urls: '${{ github.event.deployment_status.target_url }}'