microk8s: snap auto-refresh breaks cluster

Today a have experienced a crash of the PRODUCTION microk8s 3-nodes “HA” cluster. It just auto-updated to 1.21.5 ! As a programmer, admin, my mind even cannot comprehend what people deciding for the crucial services packaging have in mind to choose such a DNA broken tool as a snap??? Why at all UBUNTU uses it, when it hardly suitable even for desktop apps, and not suitable for services at all??? What is some medic.stuff would buy their adverting as “highly available” and people die because it auto-updates??? They must drop snap for anything aside the desktop apps, and better drop it at all and use proven by years .deb …

I am not sure why you mention only security and in quotes.

Your point was that security is paramount and absolute, that it should be the excuse that makes this problem okay, it’s not, it’s an excuse that only exasperates this problem and the whole of snap for servers in general.

Snaps are fine for user apps, those can deal with being restarted, crashing, shutting down, again and again. Server apps need more delicacy, planning, and oversight. Any admin/operator would not want the developer control over when, how, and why something will update, they want complete control over their systems, and the snaps auto-updating feature is a complete insult to that.

Any update that breaks the cluster is defeating its own purpose.

I’m glad you agree, then? I’d rather have a cluster which is outdated and vulnerable, and possibly get hacked, if it’s about my own oversight and my own fault (at least then i can tune it to my own schedule and my own system). With auto-update, and even the update window, that control is taken away from me, as now i have to scramble to make sure the eventual update will not fuck with my system, and then to do it manually, safe, and controlled to make sure it does not fuck over the data. (which it did for me, 1.2TB of scraping data, all corrupted because docker didnt want to close within 30 seconds, after which it got SIGKILLd)

As a sysadmin, I control a developer’s software, when, where, and how. The developer doesn’t control my system, unless I tell it to. And even then, only on my own conditions.

Snaps violated this principle, and that’s why I’m incredibly displeased with them.

@ktsakalozos the point of “security” is pretty moot if it breaks everything while updating it, it’s defeating its own purpose.

The Kubernetes project ships a few releases every month [1]. These releases include security, bug and regression fixes. Every production grade Kubernetes distribution should have a mechanism to release such fixes even before they are released from upstream. For MicroK8s this mechanism is the snaps. Snaps allow us to keep your Kubernetes infrastructure up to date not only with fresh Kubernetes binaries but also update/fix integrations with underlying system and the Kubernetes ecosystem.

If you do not want to take the risk of automated refreshes you have at least two options:

• use a snapstore proxy [3] to test revisions as they come and block those you do not want,
• set the snap refresh window in a convenient time for you [2]. This will not completely stop refreshes but will allow you to postpone them.

[1] https://github.com/kubernetes/kubernetes/releases [2] https://docs.ubuntu.com/snap-store-proxy/en/ [3] https://snapcraft.io/docs/keeping-snaps-up-to-date

For anyone reading this with the same issue, snapd currently doesnt allow any indefinite auto-update disabling, other than this suggestion through a forum thread with a bigger umbrella issue discussion about this; https://forum.snapcraft.io/t/disabling-automatic-refresh-for-snap-from-store/707/268

TL;DR: snap download foo ; snap install foo.snap --dangerous, replace foo with application in question.

Personally, the fact that snapd doesn’t allow more comprehensive and cooperative options like (web)hooks on new snap version, and have an external system handle manual refreshes one-by-one (for example by draining a node first before refreshing it, running some canary checks on it, then doing the same one-by-one for the rest, reverting entirely on first error (+ autoemail describing failed upgrade)), would be of great help and would even be still within the ethos of keeping snaps up-to-date, but the system to trust sysadmins to make it so needs to be there.

A note that is probably useful for people that have commented in the issue:

Starting with snapd 2.58, it is possible to indefinitely hold MicroK8s (and any other installed snap packages) from updating with the following command:

sudo snap refresh --hold microk8s

See also the “Control updates” section in the snapd documentation

Not stale

Yes I underestand that… so if is a political of canonical & snap dont change that (really i think that is a wrong idea dont let the user disable that)… why then waste the time in this issue??? snap team is not listen to users, because without use a workaround like discuss in this issue… or better, go to the microk8s kubernetes dristribution official doc and install it on a production cluster, you will mess all when autorefresh change something… that’s from my sysadmin point of view is faaaaaaaaaar from a production grade system.

@evildmp Do you really think you gave a satisfying professional response to my statement ? You have missed a chance to clarify:

‘As a director of engineering at canonical I’d like to assure you that you can expect a professional and complete documentation for our products. And this also includes controversial fixes or alternative or internal usage instructions for our products from our staff or our users even if we don’t recommend those to our customers. We encourage and enforce transparency. We are committed to leaving the choice to our users and customers to use and deploy our products in a way that best suits their needs even if we don’t agree with it or consider it harmful. Of course we will flag those with a big fat warning label’.

As long as you don’t say that the best answer to this issue still comes from your former employee [see: Bypassing store refresh]. As long as interns don’t have the courage or the companies allowance to give some really helpful background information I’m afraid its not cynical to say then hopefully more people are leaving the company to make up their minds finally. That’d be a really sad conclusion for the friends of Canonical, Ubuntu and Microk8s.

…but given that snap is what it is, and microk8s is what it is, I’ll try to bring this back to something constructive.

After some experimentation, it looks like the basic problem is that microk8s.stop brings down the k8s infrastructure (including the hostpath provisioner) before the containers have had a chance to react to the SIGTERM and finish gracefully. Would it be possible to change either microk8s.stop or the snap refresh hook so that the containers are allowed to terminate before the infrastructure disappears?

This is my test script, which I piped to a hostpath-provisioned folder. When bringing the container down via kubectl, it counts to 30 seconds. When doing microk8s.stop, I see nothing after “Waiting for SIGTERM”, even though I can see the process live for 30 seconds after that (in ps).

import os
import signal
import time

def handle_sigterm(*args):
    print('Got SIGTERM')
    i = 0
    while True:
        time.sleep(1)
        i += 1
        print(f'{i} seconds since SIGTERM')

if __name__ == '__main__':
    print(f'PID {os.getpid()}')
    signal.signal(signal.SIGTERM, handle_sigterm)
    print('Waiting for SIGTERM')
    while True:
        time.sleep(10)

@ktsakalozos I’m sure you do understand, that simultaneous update of the package on whatever number of the nodes, WILL lead to service stop, as ALL the nodes restarts the service simultaneously? For kubernetes, you (normally) do not restart the node, you drain it, making sure other nodes catch up for the pods, and than upgrade/maintain. So you just CANNOT distribute patches like you do. Yes, you just cannot. It may sound a litle strong, but may you imagine when lives depend on the stuff you do, or mass transport, or… whatever, where multiple persons affected? You DO name your microk8s SUPER reliable, So ANY, even minor upgrades must be done by the admin, and admin might choose than to implement auto-upgrade, because the admin will do it right, making node drain, etc.

I suspect people behind snap even scoff at us, as seems they are not allowing delaying auto-refresh more than for 60 days.

# snap set system refresh.hold=2050-01-01T15:04:05Z
# snap refresh --time
timer: 00:00~24:00/4
last: today at 10:22 UTC
hold: in 60 days, at 10:22 UTC
next: today at 17:25 UTC (but held)

You are given this option. You can block, schedule or postpone refreshes.

Block

False, point me where exactly in your comment where you said that, and keep in mind I don’t consider the snap store proxy to be a solution in this.

Great, thanks snap autorefresh, you have crashed my entire cluster with this: /snap/microk8s/2338/bin/dqlite: symbol lookup error: /snap/microk8s/2338/bin/dqlite: undefined symbol: sqlite3_system_errno

This isn’t stale, this is still a problem

@evildmp , It has nothing to do with opensource and etc. I do not criticize the code, but the wrong packaging way itself, which breaking WHOLE idea of having microk8s, and i’m using straight words for that. Once again I will try to explain what is wrong, it’s of course useless, but anyways. Looking at the meaning of the messages of other user of this thread: everyone tells the same, words different. Everyone says “It’s broken, it’s unacceptable, change it, do something”. If there is no other way to multiplatform distiribution of microk8s, just state in the docs, in front, very visible and understandable “DO NOT USE IT FOR PRODUCTION WHEN INSTALLED VIA SNAP BECAUSE YOUR SERVICE MAY BREAK RANDOMLY, UNEXPECTEDLY”. And this way, no one will have any issues. We will just look for other k8s from the start, and use microk8s (probably, not really) for dev only, and we will not experience the problems randomly and unexpectedly in production, because, according to the front page it’s rock solid reliable. Stating in the docs somewhere deep deep “install manually if you do not like snap”, changes nothing, just making users even more disappointed.

@vazir …

The approach of the mikrok8s team is not professional. The sole purpose of kubernetes is to provide a platform for running fault tolerant services, and it is completely ruined by packaging it into totally unsuitable, and DNA broken “snap” tool. It’s hard to say more. I personaly got rid of all microk8s on my servers and migrated to k3s. Next is to replace back ubuntu by plain Debian.

@evildmp if I were to guess, a large share of the frustration is not personal, but rather aimed at snap’s packaging in-of-itself, which is then a main root to cause this problem.

The solution offered is a hack, an explicit circumvention of the problem, which does not do much to offer a satisfying resolution, nor does it help lighten the burden that the problem caused, it only cripples the effectiveness of the platform, while a better solution is available from snap’s side, while they do not wish to give developers those tools, out of political and ideological reasons, that explicitly tear away control from users in a patriarchal fashion, in the sense that the developers would like to think they know their users’ systems better than the users would. (Which, imo, that is maybe true for normal application users, this becomes far less true for developers, and very much not true for system administrators, for which snaps all have the same attitude)

I don’t want to perpetuate the cycle here, at the very least know this; it wasn’t personal, the frustration is high, and this issue is just one part of the knot where the pressure became too high.

@a-hahn Hi. In respect of the documentation, no matter how many warnings or comments are added to things, sadly many people don’t read them. There is no hidden agenda, this is simply a matter of responsibility. If you buy a car and want to disconnect all the warning lights, that is also up to you, but you wouldn’t expect to find the instructions to do so printed in the owners manual. Please add your method as a comment on discourse if you like.

@ktsakalozos @evilnick well i understand that its ‘hard to recommend’ but I think it is well worth mentioning because its an obvious and simple solution for many users. Trying to hide that piece of information just hurts your / Canonicals reputation as a first class solution provider. This is even more true as snaps often stack upon each other (when you consider microk8s installed in lxc containers both installed via snaps) making troubleshooting much more difficult as it needs to be. Even more I can’t see anything positive about auto-updates as they are a backdoor into the installation and who knows which hidden features or paywalls might be introduced in the future. Misguiding people never pays in the long run. I think you should think twice about not accepting the docs update. I feel alarmed by now.

Interesting read from a former employee of canonical working in the snap advocacy team on howto disable auto-updates efficiently: https://popey.com/blog/2021/05/disabling-snap-autorefresh/ TLDR: snap download microk8s && snap install microk8s --dangerous prevents any autorefreshing

Yup. I guess there are really two issues at play here:

1. A reasonable way to disable auto-updates, which are harmful in some circumstances. Controversial, may not happen for reasons I don’t agree with but that’s how it is.
2. Fix corruption of stateful containers on microk8s.stop. Uncontroversial (probably), will make # 1 less harmful, also helps e.g. non-snap-triggered restarts and reboots.

Should # 2 get a separate issue?

Based on what I’ve read here and elsewhere, I think that snap-the-philosophy is fundamentally unsound for high availability or production server deployments. When it comes to snap-the-technology, I find it darkly amusing that it has found a way of restarting k8s that corrupts my postgresql db. Which it now does reliably 4 times a day.

@ktsakalozos Also, if you want to reproduce, 3 nodes HA 1.21/stable with rook-ceph installed, osds on each nodes, some pods that use volumes from these osds, refresh one of the nodes and that should do the trick.

@Dart-Alex i would recommend moving away from using snaps, if this is your experience, as i expect nobody from canonical will be giving this issue a genuine consideration.

@vazir I am interested to know how did the cluster crash and what did you have to do to recover it? Thank you.

I had to restart all 3 nodes to bring it up again.

you do however understand that MicroK8s cannot stop shipping patch releases and updates.

For the rest, you mentioned, you probably understand, that all you suggested are the hacks, and totally not suitable for a distribution? As nobody suggests “stopping patches distribution”, but do you understand that NO ONE in the clear mind auto-updates critical services, like databases, kubernets clusters, and similar stuff? When you deploy a highly resilient, highly available, highly redundant stuff, which as k8s, which on it’s own ATTEMPTS to be resistant to failures, trying to keep itself alive… AND YOU DO kind of “kill -9” it from the outside??? What is the whole reason of developing microk8s at all than if you totally ruin it’s redundancy and resiliency by simply mischoosing the distribution tool?

@ktsakalozos It’s one of the first things we tried but it did not help. Not sure about now, it’s a staging server and after hours now.

@vazir Absolutely, it’s now become a priority for us to migrate away from microk8s. It’s a bit irresponsible that the docs/github page does not have a big warning saying “microk8s should not be used in any kind of situation where multiple people depend on it” being it’s obviously a complete deal-breaker for any kind of serious usage. Going by this thread this has been an issue from day 1 and is never going to change with the unwillingness of snap to make autorefresh easy to disable. The only way any kind of auto-update system is reasonable for serious applications is if you’re Microsoft and have complete control over every line of update code you ship as well as gigantic resources to test every possible setup and scenario. And even then they still let you disable it on systems meant for serious use cases.

disable updates for a given package would satisfy most people who are complaining here

You are given this option. You can block, schedule or postpone refreshes. You are even given the option to try an “offline” manual deployment that never updates. Although this may suit your needs, it is very hard from the Kubernetes distribution perspective to call this a good practice.