mkdocs-material: ⚠️ Social cards broken – update to 9.5.17

Wow, what just happened while I was sleeping? I’m currently on vacation, and albeit I try to stay on top of things, answer questions, fix bugs, and be responsive nonetheless, I will likely not be able to resolve this in the next few days, as we need to think carefully here before we decide on what to do, as it is a very, very fundamental change impacting DX big time. I’m confident that we can find other ways to cope with this while I’m gone, maybe even work towards a solution.

Context

Before we started using the downloads endpoint, we fetched the font files from Google Fonts’ CSS files, but that led to some problems where characters were not available, as those files are heavily optimized. I can’t really find the commit now, but a user suggested that we could switch to the downloads endpoint for a better experience, and it served us well for years.

However, I wasn’t aware that it was internal. Google switching this endpoint off is inconvenient for us, to put it mildly, but I think we can’t really complain if we have used an unofficial API until now. Thus, we need to look for other solutions.

Directions

We cannot instruct the social plugin to use the developer API, as it requires an API key. This means that every user of Material for MkDocs that wants to use the social plugin would need to register for a Google API key prior to using it. It would also mean this key would need to be passed along as a secret everywhere the project is built, which for Open Source projects would mean that contributors would not be able to build your documentation, since the API key must be kept secret and considered sensitive, or it would defeat its purpose. Thus, this is a dead end from my perspective.

Before we decide how we proceed, we should evaluate multiple possibilites. A first PR by @alexvoss is up that might help to remedy the situation, so you can help us collect feedback on this PR. We’re open to other suggestions, including entirely different font catalogs (i.e. switching away from Google Fonts), etc., but we can’t just bandaid this, since everybody using the social plugin is impacted.

Mitigation

The only save mitigation that does not rely on caching is to disable the social plugin. If you’re using Insiders, you could theoretically just download the font via Google Fonts web interface and unpack it in .cache/plugins/social/fonts. This does not work for the community edition, since the Insiders edition is a more elaborate rewrite. We can match folder locations if that helps, though.

I’ll try to find some time here and there during my vacation and be as helpful as possible, but please understand that right now, I might be a little slow to respond.

For the current non-insiders build, the social plugin expects files to be in the .cache/plugins/social/ folder, prefixed by {FontFamily}- (code). For example, placing Roboto in cache manually looks something like this:

$ mkdir -p .cache/plugin/social/
$ cp ~/Downloads/Roboto/*.ttf .cache/plugin/social/
$ tree .cache/plugin/social
.cache/plugin/social
├── Roboto-Black.ttf
├── Roboto-BlackItalic.ttf
├── Roboto-Bold.ttf
├── Roboto-BoldItalic.ttf
├── Roboto-Italic.ttf
├── Roboto-Light.ttf
├── Roboto-LightItalic.ttf
├── Roboto-Medium.ttf
├── Roboto-MediumItalic.ttf
├── Roboto-Regular.ttf
├── Roboto-Thin.ttf
└── Roboto-ThinItalic.ttf

1 directory, 12 files

Just a note that the PR I am working on will not require manual manipulations of the fonts beyond the downloading and putting the .zip in a directory somewhere that can be configured with a fonts_dir option.

Edit: PR has just gone in. Hope it is of some use and we’ll see a fix released soon. Might take a little because of timezones.

We don’t yet support MkDocs 1.6, because MkDocs has had a history of breaking changes in minor releases, which is why we’ve limited to <1.6 for now. We’ll try to upgrade to 1.6 quickly, but I currently need to finish other things for Material for MkDocs, so I’m happy to collaborate if somebody wants to go ahead and do the legwork ☺️

Edit: This is now tracked in #7076.

@janaka I never worked with poetry too deeply, so I’m not sure if there are any missing logs etc., but the failed CI says it installed version 9.5.13: https://github.com/docqai/docq/actions/runs/8590257742/job/23537555949#step:6:296

And in the other working CI where you disabled the social plugin it says it installed 9.5.17: https://github.com/docqai/docq/actions/runs/8590353932/job/23537780963#step:6:296

Quite bizarre, perhaps the issue lies in the 2 different version definitions? https://github.com/docqai/docq/blob/8c88f4c7019654167d6d1075414ee79fdf393cfe/pyproject.toml#L51 https://github.com/docqai/docq/blob/8c88f4c7019654167d6d1075414ee79fdf393cfe/pyproject.toml#L77

pip with requirements.txt is so much simpler 😸 perhaps you could clear the caches too 🤔

First, sorry for what Google bought you last week, I feel the pressure on you. mkdocs-material is so popular, so… you pay the celebrity price. For the moment the easiest thing for me to do is to disable the social plugin as it was never an essential bit.

Thanks for acknowledging! Yeah, I tried to fix this as fast as possible during my vacation, and yes, there’s a lot of pressure on me, but I can handle that. The funding we have makes that possible, as I can put icreasing weight on more shoulders. I also always try to provide mitigations as fast as possible, albeit it was “disable the plugin” this time.

In addition to the default Roboto fonts, I am also using the RedHat ones, but boths are “google fonts”. As I do have my own theme on top of material-theme for reusability across projects, I am really looking for a solution to embed the fonts inside the theme package, so we no longer need to download fonts from google in order to build. Still, there is not documentation about how to achieve this.

Yes, we’ll consider adding the ability to provide a custom font directory in the Insiders version of the social plugin after we sorted out the best way how to deal with Google Fonts. What we released with 9.5.17 is our first take on it, and we need to see if this fixes all problems or if we need to change things. After the social plugin can be considered stable again, we’ll look into providing more options. Note that the fonts will likely need to have specific names, so we keep code duplication at a minimum, but I guess this is a reasonable requirement for custom fonts that you control anyway.

The simplest way to achieve what you want is to whitelist the .cache/plugins/social/fonts directory in your .gitignore, and build the project locally which will download the fonts, and check the fonts into version control.

Another change that could prove useful, if the time allows, would be to catch zipfile.BadZipFile exception and throw a clear error message when it happens instead of one where we would need to guess which download might have had failed. Important to display the original URL and hopefully http code received too.

Yes, we’re going to improve error reporting when we settle on the final font loading logic. We’ll first collect some feedback on the current draft, so we can iterate more quickly. Then we’re going to improve resilience.

First, sorry for what Google bought you last week, I feel the pressure on you. mkdocs-material is so popular, so… you pay the celebrity price. For the moment the easiest thing for me to do is to disable the social plugin as it was never an essential bit.

In addition to the default Roboto fonts, I am also using the RedHat ones, but boths are “google fonts”. As I do have my own theme on top of material-theme for reusability across projects, I am really looking for a solution to embed the fonts inside the theme package, so we no longer need to download fonts from google in order to build. Still, there is not documentation about how to achieve this.

Another change that could prove useful, if the time allows, would be to catch zipfile.BadZipFile exception and throw a clear error message when it happens instead of one where we would need to guess which download might have had failed. Important to display the original URL and hopefully http code received too.

IMHO, custom fonts can be an extra feature, but should not be the replacement for what we currently have, because it means much more effort to get started. We need to keep it as simple as possible. Our philosophy says that customization should always be possible, but never mandatory.

For right now the only thing I can suggest is to turn off the social plugin. There should be a fix tomorrow, I think. IMHO downloading the fonts from Google Fonts was a convenience function that now no longer works. I will start looking at a PR to fix this issue with a local font option. That leaves open the possibility of automating it should that become feasible again but gives us a plan B.