puppeteer: [Bug]: Get a 404 when trying to download Firefox during its version transition

hm, so the build.id is a date?

That’s right. Firefox buildIds are timestamps with the format YYYYMMDDhhmmss. We ensure the uniqueness of these dates by relying on the timestamp of the commit that the current nightly is based upon.

Does it mean that we would need to parse it to reconstruct the URL?

Yeah, I’m not aware of any place that exposes a string like YYYY/MM/YYYY-MM-DD-hh-mm-ss or the full URL of a nightly.

it does not seem to be good to expose such buildId to the users?

I’m sorry, I’m not sure to follow. Could you expand on this?

I guess versions like 115.0b8 are much better and they uniquely identify a binary in all cases except for the nightly channel?

That’s right. Mozilla has a stricter policy before builds other than nightlies reach https://archive.mozilla.org/pub/firefox/release/. Several teams are involved to ensure the quality of the build. This process also ensure versions like 115.0b8 are unique.

Is there a bug to follow for the automation process of the nightly releases?

I didn’t manage to find one. I remember we had conversation about this topic. I’ll file a bug.

Do you think it would be possible for Mozilla to provide a repo with version metadata following CfT formats […] I believe it would make it easier for everyone to download Firefox versions and will also mitigate the 404 error happening during the manual process.

I’m sorry, that will require some development. I don’t think Mozilla has the bandwidth to implement this, at the moment. Mozilla provides multiple ways to download either the latest build or a specific one. We also have https://buildhub.moz.tools/ which provides an ElasticSearch endpoint to look build up (https://buildhub2.readthedocs.io/en/latest/user.html#id1).

@JohanLorenzo cool, thanks for sharing! It does look like we can use it! Can we do the same for other Firefox channels (see https://github.com/puppeteer/puppeteer/issues/10420).

Awesome! 😃 Yes, a similar approach works on DevEdition, Beta, Release, and ESR. For all these cases, fetch https://product-details.mozilla.org/1.0/firefox_versions.json,

{
    "FIREFOX_DEVEDITION": "115.0b8",      // Use this field for beta too, these 2 are always in sync, these days
    "LATEST_FIREFOX_VERSION": "114.0.2",
    "FIREFOX_ESR": "102.12.0esr",
}

Then, you can build URLs like:

https://archive.mozilla.org/pub/devedition/releases/115.0b8/linux-x86_64/en-US/firefox-115.0b8.tar.bz2

(replace devedition by firefox for beta, release and ESR)

Also, how can we fetch the binary for a specific buildId? (i.e., the reverse lookup).

Yes! 😃 Once you know the buildId, you can build this kind of URLs: https://archive.mozilla.org/pub/firefox/nightly/YYYY/MM/YYYY-MM-DD-hh-mm-ss-mozilla-central/{path_to_binary}

E.g.: https://archive.mozilla.org/pub/firefox/nightly/2023/06/2023-06-20-09-44-33-mozilla-central/

As for the original issue, it looks like firefox_versions.json might be updated before all binaries are available for download. Do you have a solution for that?

The ideal solution would be for Mozilla to automate this manual process[1]. In the meantime, falling back to the previous major version of nightly should do the trick. By the way, only Nightly is impacted by this bug. DevEdition, Beta, Release, and ESR are all fully automated which means the data at https://product-details.mozilla.org/1.0/firefox_versions.json is accurate.

[1] https://docs.mozilla-releng.net/en/latest/procedures/release-duty/merge-duty/merge_duty.html#bump-nightly-version-and-release-dates-in-shipit