azure-linux-extensions: (Custom Script Extension) Script failure causes following ARM template deployment to fail

When a script invoked by the Custom Script extension returns an error code, I’ve observed that the next ARM template deployment will also fail. The real problem is that the VM is the resource that fails. This poses a problem for custom scripts executed in ARM template deployments, because the custom script dependsOn the VM. So, it will never run again and have a chance to correct itself. (We use a timestamp in settings to ensure the script runs on each deployment).

Here’s the error output from New-AzureRmResourceGroupDeployment:

VERBOSE: 12:41:22 AM - Create template deployment 'azuredeploy-####-####'.
VERBOSE: 12:41:30 AM - Resource Microsoft.Network/virtualNetworks '#####' provisioning status is succeeded
VERBOSE: 12:41:33 AM - Resource Microsoft.Storage/storageAccounts '#####' provisioning status is succeeded
VERBOSE: 12:41:33 AM - Resource Microsoft.Network/publicIPAddresses '#####' provisioning status is succeeded
VERBOSE: 12:41:36 AM - Resource Microsoft.Compute/virtualMachines '#####' provisioning status is running
VERBOSE: 12:41:36 AM - Resource Microsoft.Network/networkInterfaces '#####' provisioning status is succeeded
New-AzureRmResourceGroupDeployment : 12:42:12 AM - Resource Microsoft.Compute/virtualMachines '#####' failed with message 'The resource 
operation completed with terminal provisioning state 'Failed'.'
At C:\#####\scripts\Deploy.ps1:65 char:11
+ $deploy = New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $T ...
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

New-AzureRmResourceGroupDeployment : 12:42:12 AM - VM has reported a failure when processing extension 'shexec'. Error message: "Script returned an 
error.
---stdout---
---errout---
(error output)
".
At C:\#####\scripts\Deploy.ps1:65 char:11
+ $deploy = New-AzureRmResourceGroupDeployment -Name ((Get-ChildItem $T ...
+           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [New-AzureRmResourceGroupDeployment], Exception
    + FullyQualifiedErrorId : Microsoft.Azure.Commands.Resources.NewAzureResourceGroupDeploymentCommand

We can work around this problem by invoking custom scripts outside of ARM templates, but it still means we need an extra deploy to fix the issue. Alternatively, I think that we can invoke “/bin/true” after a failed ARM template deployment to get the extension back into a non-error state. However, I can see this issue affecting others, and working around it in this manner is non-intuitive.