coveralls-python: Github Actions raises a 422 error

My understanding of it is that if you are using the Github Actions natively provided GITHUB_TOKEN secret or ${{ github.token }}, you use the github service. Coveralls then works some internal magic, querying build and authentication info via GitHub APIs using that token. If you manually provide your coveralls token via a secret, you use github-actions (or really anything that’s not a special-cased service name).

I think I have found the fix for this issue, and have launched PR #385 with the fix.

For users finding this issue: Without using that PR, the same solution can be achieved in anyone’s GitHub Actions workflow as follows, as long as a version of coveralls is used that supports the COVERALLS_SERVICE_JOB_ID env var (such as coveralls>=3.3.0):

For sending the result of each parallel run:

    - name: Send coverage result to coveralls.io
      shell: bash -l {0}
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        COVERALLS_PARALLEL: true
        COVERALLS_SERVICE_JOB_ID: ${{ github.run_id }}
      run: |
        coveralls --service=github

For sending the finish request for parallel runs:

    - name: Send coverage finish to coveralls.io
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: |
        coveralls --service=github --finish

If this solution does not work for you, or if you find that you have to use the service name “github-actions”, please comment on this issue.

In case it can help others, I got stuck on 422 with the following config:

            - name: "Coveralls"
              env:
                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
              run: |
                  coveralls --service=github

Tried out other environment variables, setting the service name to both “github” and “github-actions”, to no avail.

Then I disabled my repository on https://coveralls.io/ (deleting all its data) and re-enabled it manually. After that operation, the config above started working again.

Had no problems with coveralls on Sep 28 and then subsequently, Nov 10, it stopped working with this 422 error. The first job report succeeds, but whenever a subsequent parallel job tries to post the coverage report, it fails with 422. Something seems to have changed between those dates.

In any case, the suggestion by @andy-maier fixed the issue.

# ... matrix job
       - name: Coveralls
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          COVERALLS_PARALLEL: "true"
+          COVERALLS_SERVICE_JOB_ID: ${{ github.run_id }}
         run: |
           pip install --upgrade coveralls
           coveralls --service=github
+  finish:
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - name: finish coveralls
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          pip install --upgrade coveralls
+          coveralls --service=github --finish

@elParaguayo Note that I removed and added one of our repos (https://github.com/zhmcclient/python-zhmcclient) from coveralls.io, and that did not fix the issue with this new occurrence of HTTP 422.

This has just started out of the blue for us.

I thought it might have been linked to an upgrade to coverage as the timings worked but downgrading that package didn’t fix the issue.

We had been using the --service=github flag. I’ve tried changing to github-actions and removing it completely. Neither worked.

I’d really prefer not to have to delete the data from coveralls. Is that really the only option here?

I understand! it’s been around for a long time and you guys have tried to work around it and a lack of documentation on it admirably. We’re good. Hopefully we can put it to bed soon.

Indeed, you understanding what the issue is, relaying it and vouching for it internally is the best possible outcome. So we’re good! (Sorry for being very direct and possibly harsh. I let my frustration with this year-old issue bubble out. Sorry!)

I’m ok with that but, personally, I’m just a support engineer so my goal is to help users get their builds sending coverage to coveralls and provide background on the Coveralls API to integrators.

I’m happy to relay the request as a feature change, which I’ll do now, and even argue for it.

(That said, I think we’re on the same page that without a unique identifier from Github Actions, Coveralls won’t work as designed for parallel jobs, but that this should be limited to incorrect links back to CI jobs on the builds page.)

Upon doing so, are we good for our purposes here?

Update: I have connected my request to your issue (https://github.com/coverallsapp/github-action/issues/59) @TimoRoth

In my original PR adding support for the actual ‘github’ service type, I also introduced use of the env variable called GITHUB_TOKEN: https://github.com/TheKevJames/coveralls-python/commit/f597109b62fadaf900af79d4f08a7debee5229e2

Before that, I don’t think it was used or supported at all. I’m not sure at which point it got its double role to also get the coveralls repo token passed in, but the original idea there was that GITHUB_TOKEN is always expected to have the actual GITHUB_TOKEN, and not something else. If you for some reason need/want to pass in the actual coveralls token, there’s a separate env var for that. So the change of the default service from ‘github’ to ‘github-actions’ is still confusing me.

I still think it’s odd to put anything but the GITHUB_TOKEN into the env var called GITHUB_TOKEN. Putting the coveralls provided token in there seems very confusing for both users and obviously the code. Was it ever documented to be like that?

@TimoRoth I tend to agree but afaik GITHUB_TOKEN was introduced by coveralls-python, and needed to be used in my example; but that original variable name makes sense since the creator expected to be passing ${{ secrets.GITHUB_TOKEN }}.

Not just odd, using a token like COVERALLS_REPO_TOKEN (regardless of what env var it goes into) doesn’t work in pull requests, as GitHub doesn’t expose secrets in pull requests (for obvious reasons). Recommending people to do that is irresponsible.

@liskin I wasn’t aware of that, though I see you mentioned it previously. Is that just a recommendation from Github? or will Github Actions simply not surface secrets in the context of a PR? My expectation would be that it wouldn’t surface those for forked PRs (that makes sense to me).

This also seems to go against my experience with this PR: https://github.com/afinetooth/flask-rollup/pull/1

Where, after changing all token references to COVERALLS_REPO_TOKEN, this Action run succeeded for the pull_request build: https://github.com/afinetooth/flask-rollup/runs/2452128424

Please note, I’m not saying Coveralls recommends using COVERALLS_REPO_TOKEN. In fact, we don’t. Our implementation of the Coveralls Github Action uses ${{ secrets.GITHUB_TOKEN }}. The issue is just that any incoming CI job with a service_name other than github will be treated like a job from a generic CI, which expects COVERALLS_REPO_TOKEN.

@TimoRoth @liskin @TheKevJames please stand by. I incorrectly reproduced the use case from @zgoda so I will need to re-try with his correction here to re-verify my assumptions.

@TimoRoth yes, unfortunately we received several bug reports from folks who had done things the other way around, which led to the original work in trying to guess which service name was correct. You might be onto something with the length check, I think that’s about the only potentially deterministic way of solving this I’ve seen so far.

Aside: I’ve just released @vchrombie 's fix as v3.0.1, so the Github retries should now work, at least!

Are there really repositories that supply a variable called GITHUB_TOKEN, and put the Coveralls Token in there? Pretty sure in my original PR, bringing up proper support for GitHub Actions, I just assumed that GITHUB_TOKEN would be said Actions-Provided secret, and if it’s not set, use the github-actions service instead of GitHub. Was there an issue with that approach?

If some people really do that, it should be possible to determine the kind of token from the length, even though that’s a bit hacky. GitHubs tokens all appear to be 40 characters long, while the Coveralls tokens seem to be 32~33 characters long.

@TheKevJames Do we have some examples of situations where github-actions service name is needed? Most of the commits linked to this issue are switching back to github, and that is what we’ll need to do in taskwiki as well. The tests there use the GITHUB_TOKEN provided by Actions, no COVERALLS_REPO_TOKEN, and there is COVERALLS_PARALLEL=true. With github-actions service name, we get 422.

I was involved in https://github.com/TheKevJames/coveralls-python/pull/227/ which made it possible to do test matrix builds with coveralls correctly, so I might be able to remember something, but my memory of it is somewhat foggy.

(I also think that service_job_id only needs to be specified with github-actions service name. With github, it’s worked just fine without it, as it should: GH Actions don’t provide any unique GITHUB_JOB_ID. It is probably harmless to pass null, though.)

Just in case the additional information helps: In our projects, the 422 error only shows up on Python 3.9 with latest package levels. Not on Python 3.9 with our minimum package levels, and not on older Python versions. See for example this Actions run: https://github.com/pywbem/pywbem/actions/runs/484671222