terraform-provider-azurerm: Azure RM 2.0 extension approach incompatible with ServiceFabricNode extension requirements of being added at VMSS creation time.

@tombuildsstuff - I don’t think it’s an issue with the service fabric extension as such, Azure clearly allows extensions to require they are added at creation time so Terraform needs to account for that option. I doubt the service fabric team are going to change the behaviour, so the new scale set resource is going to remain incompatible.

Just a quick note, beta functionality for this shipped in v2.26.0. To make use of it there’s a new env var you can set to allow the use of the new extension property on both azurerm_linux_virtual_machine_scale_set and azurerm_windows_virtual_machine_scale_set. Set ARM_PROVIDER_VMSS_EXTENSIONS_BETA=true to use the new block.

I also included a basic example of the extension in use with Service Fabric extension in the examples folder also. If there’s feedback / issues with the beta functionality, please open a new issue (or add detail to any already open if someone else gets there first 😄 )

Thanks!

Thanks @rhys96 - I’m working through behaviours for create and update changes at the moment, if I can make use of your code I’ll add you to the co-authors list 👍

This also affects custom script extension with azurerm_linux_virtual_machine_scale_set. By not in lining extension with the VMSS object, the extension installs and is stuck in “Creating” state until you run a rolling upgrade on the VMSS. Unfortunately, you can’t easily set the upgrade state to “automatic” as that requires installation of a probe.

The only workaround is to use cloud-init, but cloud-init doesn’t block on failures of the script. Also, cloud-init doesn’t have the ability to take secrets like custom script’s “protectedSettings” field.

Additionally this results in a performance penalty, since applying extensions in two steps causes the whole deployment to take much longer as it has to wait until all nodes are up before applying the extension, and the slowness of the rolling upgrade.

We want to use azurerm_linux_virtual_machine_scale_set to get access to new features like ephemeral OS disks.

I’m taking a look into this at the moment. It’s a deceptively tricky thing to re-introduce, but we’ll do what we can.

I was about to start a PR to create this feature because I’m dependent on this feature for #8120, however @tombuildsstuff mentioned you were about start something. If it helps, I started to add the following in virtual_machine_scale_set.go:

func VirtualMachineScaleSetExtensionSchema() *schema.Schema {
	return &schema.Schema{
		Type:     schema.TypeList,
		Optional: true,
		Elem: &schema.Resource{
			Schema: map[string]*schema.Schema{

				"name": {
					Type:         schema.TypeString,
					Required:     true,
					ValidateFunc: validation.StringIsNotEmpty,
				},

				"publisher": {
					Type:         schema.TypeString,
					Required:     true,
					ValidateFunc: validation.StringIsNotEmpty,
				},

				"type": {
					Type:         schema.TypeString,
					Required:     true,
					ValidateFunc: validation.StringIsNotEmpty,
				},

				"type_handler_version": {
					Type:         schema.TypeString,
					Required:     true,
					ValidateFunc: validation.StringIsNotEmpty,
				},

				"auto_upgrade_minor_version": {
					Type:     schema.TypeBool,
					Optional: true,
					Default:  true,
				},

				"enable_automatic_upgrade": {
					Type:     schema.TypeBool,
					Optional: true,
					Default:  true,
				},

				"force_update_tag": {
					Type:     schema.TypeString,
					Optional: true,
				},

				"protected_settings": {
					Type:             schema.TypeString,
					Optional:         true,
					Sensitive:        true,
					ValidateFunc:     validation.StringIsJSON,
					DiffSuppressFunc: structure.SuppressJsonDiff,
				},

				"provision_after_extensions": {
					Type:     schema.TypeList,
					Optional: true,
					Elem: &schema.Schema{
						Type: schema.TypeString,
					},
				},

				"settings": {
					Type:             schema.TypeString,
					Optional:         true,
					ValidateFunc:     validation.StringIsJSON,
					DiffSuppressFunc: structure.SuppressJsonDiff,
				},
			},
		},
	}
}

func ExpandVirtualMachineScaleSetExtension(input []interface{}) (*[]compute.VirtualMachineScaleSetExtension, error) {
	extensions := make([]compute.VirtualMachineScaleSetExtension, 0)

	for _, v := range input {
		raw := v.(map[string]interface{})

		settings := map[string]interface{}{}
		if settingsString := (raw["settings"].(string)); settingsString != "" {
			s, err := structure.ExpandJsonFromString(settingsString)
			if err != nil {
				return nil, fmt.Errorf("unable to parse settings: %s", err)
			}
			settings = s
		}

		provisionAfterExtensionsRaw := raw["provision_after_extensions"].([]interface{})
		provisionAfterExtensions := utils.ExpandStringSlice(provisionAfterExtensionsRaw)

		protectedSettings := map[string]interface{}{}
		if protectedSettingsString := raw["protected_settings"].(string); protectedSettingsString != "" {
			ps, err := structure.ExpandJsonFromString(protectedSettingsString)
			if err != nil {
				return nil, fmt.Errorf("unable to parse protected_settings: %s", err)
			}
			protectedSettings = ps

		}

		extension := compute.VirtualMachineScaleSetExtension{
			Name: utils.String(raw["name"].(string)),
			VirtualMachineScaleSetExtensionProperties: &compute.VirtualMachineScaleSetExtensionProperties{
				Publisher:                utils.String(raw["publisher"].(string)),
				Type:                     utils.String(raw["type"].(string)),
				TypeHandlerVersion:       utils.String(raw["type_handler_version"].(string)),
				AutoUpgradeMinorVersion:  utils.Bool(raw["auto_upgrade_minor_version"].(bool)),
				EnableAutomaticUpgrade:   utils.Bool(raw["enable_automatic_upgrade"].(bool)),
				ProtectedSettings:        protectedSettings,
				ProvisionAfterExtensions: provisionAfterExtensions,
				Settings:                 settings,
			},
		}

		extensions = append(extensions, extension)
	}

	return &extensions, nil
}

func FlattenVirtualMachineScaleSetExtension(input *[]compute.VirtualMachineScaleSetExtension) []interface{} {
	if input == nil {
		return []interface{}{}
	}

	output := make([]interface{}, 0)

	for _, v := range *input {
		// To do
	}

	return output
}

This bug also affects custom script extension with azurerm_windows_virtual_machine_scale_set.

Sam issue here - no chance to get this working… I use Application Gateway and no LB so the health extension is the only option…

Hi,

Do we have an ETA on when this will be fixed? Currently getting around this by attaching the load balancer probe to scaleset, but I would prefer to de-couple them and set UpgradePolicy to Manual. Would be good to know if it’s going to be picked up in any upcoming Milestone.