harvester: [BUG] Error message about invalid backup target doesn't go away

both the secret and setting should be reset on both Harvester and LH sides, additionally, I think we should keep the original backup data on the remote server for now, as an enhancement we can add an option to ask users whether to clean up all backups or keep it when updating the backup target setting.

@FrankYang0529 @guangbochen

Suppose the backup target setting changes as: initial-> S3 -> NFS…, if we only call h.updateBackupTargetSecret when S3, then it means in the NFS backup, the previous S3 secret is till remaining there.

It looks we need to update all information of backup target setting whenever changes happens (initial -> s3/nfs -> nfs/s3 -> init -> any…)

OnBackupTargetChange

	h.updateBackupTargetSecret (when S3)
		getBackupSecretData
		h.secretCache.Get
		 h.secrets.Create
		 h.secrets.Update
		h.updateLonghornBackupTargetSecretSetting
			h.longhornSettingCache.Get
			 h.longhornSettings.Create
			 h.longhornSettings.Update

// OnBackupTargetChange handles backupTarget setting object on change
func (h *TargetHandler) OnBackupTargetChange(key string, setting *harvesterv1.Setting) (*harvesterv1.Setting, error) {
..
	if target.Type == settings.S3BackupType {
		if err = h.updateBackupTargetSecret(target); err != nil {
			return nil, err
		}
	}
...
}

IMO I would prefer the following approaches, feel free to comment on it, thanks.

1. should be fine to continue the process and validate it via the webhook validations.
2. prefer to reject and ask the user to either remove the pending VM backups first or try reset later.
3. should be rejected as the current approach.
4. keep silent and do nothing.

@guangbochen @FrankYang0529

To reset the backup target to the default value, we need to consider a few specifications:

1. If some vm backup has been configured, should “reset backup target” give the user information to confirm the action?

2. If a vm backup is right happening, what to do when “reset backup target”? e.g., prompt the user to try later, or delete the vm backup first

3. If backup target is the default value(NULL), when the user creates VM backup, allow or deny?

4. If backup target is the default value(NULL), for those existing VM backups, how to process? a. be silent, loop until a valid backup target is configure then do the work b. report warning/error each time when it tries to do backup c. others…

Your idea?

When the backup target config is failed, the error will always be there, to notify the user. User can modify the config to a correct one, or clear/reset the error config. The latter does not work now.

We may either add a new button beside “show/hide backup target”, called “reset/clear backup target”; or in the “edit” pop up, enable “Use the default value” button (it is not usable now), to reset/clear current config.

“Use the default value” button looks to be a better option. UI is checking why it is disabled. @guangbochen @WuJun2016 @n313893254