cluster-api: machine-controller: block deletion with finalizer/annotation

User Story

As an operator I would like to have the ability to build custom components that integrate into the life-cycle of a machine. Specifically, I would like to have the ability to perform different actions between the time a machine is deleted in the api and the time the machine is deleted from the cloud.

One use-case is to ensure that a new master machine has been successfully created and joined the cluster before deleting a master machine. This might be useful in case there are disruptions during replacement and we need the disk of the existing master to perform some disaster recovery operation.

This method is mostly designed to let users/operators ‘hook’ into the the lifecycle of a machine as they see fit. This might also be helpful for developing integrations with new clouds or other types of infrastructure that need more fine-grained signaling.

Detailed Description

Support a new finalizer. Finalizer should have a known-name, and possibly support a list (or csv, etc) that different controllers can append/remove from; this would allow arbitrary number of controllers without having to limit integrations.

Ex:

machines-delete-hooks: "myfinalizer1,myfinalizer2"

Perhaps we should do this before drain? Perhaps we should have two separate finalizers, one before drain, one before delete? I think probably just having the finalizer work before drain is going to cover most of what someone would want, and we could always add a second one later.

Goals

• 1. Allow users/operators more control over a machine’s life cycle.

Non-Goals

• 1. API changes.

Anything else you would like to add:

During discussions of adding drain to the machine-controller, there was talk about letting an external component handle that. Users could implement their own server-side drain controller (or use a community one) and use the exclude-draining annotation to replace the machine-controller’s built-in drain functionality if they so chose. There are probably lots of different use cases and this would just be one tool we add for flexibility.

/kind feature