azure-sdk-for-python: VirtualMachineExtension metadata error handling response from create_or_update: KeyError "Use twice the key"

• Package Name: azure-mgmt-compute
• Package Version: 12.0.0
• Operating System: Debian 10
• Python Version: 3.7.3

Describe the bug

After calling VirtualMachineExtension.create_or_update, the response coming back from the server cannot go through a deserialize/serialize cycle, likely due to something with the metadata handling (there are two fields with the key “type” in the structure, but at different levels):

Calling create_or_update with:

('confab-test-virtual-machine-extension-rg',
 'confab-test-virtual-machine-extension-vm',
 'confab-test-virtual-machine-extension',
 {'auto_upgrade_minor_version': True,
  'location': 'eastus',
  'protected_settings': {'commandToExecute': 'powershell.exe -ExecutionPolicy '
                                             'Unrestricted -File '
                                             'appgatewayurl.ps1'},
  'publisher': 'Microsoft.Compute',
  'settings': {'fileUris': ['https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1']},
  'tags': {'_tuono_version': 'v0.1.2+unittest'},
  'type_handler_version': '1.10',
  'virtual_machine_extension_type': 'CustomScriptExtension'})

Note there is no “type” property; the server fills that in on the response:

(Pdb) pp result._result
<azure.mgmt.compute.v2019_12_01.models._models_py3.VirtualMachineExtension object at 0x7ff6952fdfd0>
(Pdb) pp result._result.as_dict()
{'auto_upgrade_minor_version': True,
 'id': '/subscriptions/.../resourceGroups/confab-test-virtual-machine-extension-rg/providers/Microsoft.Compute/virtualMachines/confab-test-virtual-machine-extension-vm/extensions/confab-test-virtual-machine-extension',
 'location': 'eastus',
 'name': 'confab-test-virtual-machine-extension',
 'provisioning_state': 'Succeeded',
 'publisher': 'Microsoft.Compute',
 'settings': {'fileUris': ['https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1']},
 'tags': {'_tuono_version': 'v0.1.2+unittest'},
 'type': 'Microsoft.Compute/virtualMachines/extensions',
 'type_handler_version': '1.10',
 'virtual_machine_extension_type': 'CustomScriptExtension'}

Now if you attempt to deserialize this result it will fail with the error:

(Pdb) from azure.mgmt.compute.models import VirtualMachineExtension
(Pdb) pp VirtualMachineExtension.from_dict(result._result.as_dict())

*** msrest.exceptions.DeserializationError: Unable to deserialize to object: type, KeyError: 'Use twice the key: "virtual_machine_extension_type"'

When I get the error indicating “Use twice the key”, which is a KeyError ,I added a breakpoint so I could investigate:

> /home/jking/confab/confab-cloud/.tox/debug/lib/python3.7/site-packages/msrest/serialization.py(1358)_deserialize()
-> raise KeyError('Use twice the key: "{}"'.format(attr))
(Pdb) l
1353                    for key_extractor in self.key_extractors:
1354                        found_value = key_extractor(attr, attr_desc, data)
1355                        if found_value is not None:
1356                            if raw_value is not None and raw_value != found_value:
1357                                breakpoint()
1358 ->                             raise KeyError('Use twice the key: "{}"'.format(attr))
1359                            raw_value = found_value
1360
1361                    value = self.deserialize_data(raw_value, attr_desc['type'])
1362                    d_attrs[attr] = value
1363            except (AttributeError, TypeError, KeyError) as err:

(Pdb) pp data
{'auto_upgrade_minor_version': True,
 'id': '/subscriptions/.../resourceGroups/confab-test-virtual-machine-extension-rg/providers/Microsoft.Compute/virtualMachines/confab-test-virtual-machine-extension-vm/extensions/confab-test-virtual-machine-extension',
 'location': 'eastus',
 'name': 'confab-test-virtual-machine-extension',
 'provisioning_state': 'Succeeded',
 'publisher': 'Microsoft.Compute',
 'settings': {'fileUris': ['https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1']},
 'tags': {'_tuono_version': 'v0.1.2+unittest'},
 'type': 'Microsoft.Compute/virtualMachines/extensions',
 'type_handler_version': '1.10',
 'virtual_machine_extension_type': 'CustomScriptExtension'}

(Pdb) pp attr
'virtual_machine_extension_type'
(Pdb) pp attr_desc
{'key': 'properties.type', 'type': 'str'}

(Pdb) pp found_value
'Microsoft.Compute/virtualMachines/extensions'
(Pdb) pp key_extractor
<function last_rest_key_case_insensitive_extractor at 0x7f9a2fab1048>

(Pdb) pp attributes
{'auto_upgrade_minor_version': {'key': 'properties.autoUpgradeMinorVersion',
                                'type': 'bool'},
 'force_update_tag': {'key': 'properties.forceUpdateTag', 'type': 'str'},
 'id': {'key': 'id', 'type': 'str'},
 'instance_view': {'key': 'properties.instanceView',
                   'type': 'VirtualMachineExtensionInstanceView'},
 'location': {'key': 'location', 'type': 'str'},
 'name': {'key': 'name', 'type': 'str'},
 'protected_settings': {'key': 'properties.protectedSettings',
                        'type': 'object'},
 'provisioning_state': {'key': 'properties.provisioningState', 'type': 'str'},
 'publisher': {'key': 'properties.publisher', 'type': 'str'},
 'settings': {'key': 'properties.settings', 'type': 'object'},
 'tags': {'key': 'tags', 'type': '{str}'},
 'type': {'key': 'type', 'type': 'str'},
 'type_handler_version': {'key': 'properties.typeHandlerVersion',
                          'type': 'str'},
 'virtual_machine_extension_type': {'key': 'properties.type', 'type': 'str'}}

It appears the metadata for “virtual_machine_extension_type” is referencing the “type” field instead of the “virtual_machine_extension_type” field. It extracts the wrong content, and then the code thinks that a key appears twice in the data.

incorrect:
(Pdb) pp key_extractor("virtual_machine_extension_type", {"key": "properties.type", "type": "str"}, data)
'Microsoft.Compute/virtualMachines/extensions'

correct:
(Pdb) pp key_extractor("virtual_machine_extension_type", {"key": "properties.virtual_machine_extension_type", "type": "str"}, data)
'CustomScriptExtension'

To Reproduce Run the following script:

import unittest

from azure.mgmt.compute.models import VirtualMachineExtension


class TestVirtualMachineExtensionSerialization(unittest.TestCase):
    def setUp(self):
        self.raw = {
            'auto_upgrade_minor_version': True,
            'id': (
                '/subscriptions/.../resourceGroups/confab-test-virtual-machine-extension-2-rg'
                '/providers/Microsoft.Compute/virtualMachines/confab-test-virtual-machine-extension-2-vm'
                '/extensions/confab-test-virtual-machine-extension-2'
            ),
            'location': 'eastus',
            'name': 'confab-test-virtual-machine-extension-2',
            'provisioning_state': 'Succeeded',
            'publisher': 'Microsoft.Compute',
            'settings': {
                'fileUris': [
                    (
                        'https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples'
                        '/master/application-gateway/iis/appgatewayurl.ps1'
                    )
                ]
            },
            'tags': {'_tuono_version': 'v0.1.2+unittest'},
            'type': 'Microsoft.Compute/virtualMachines/extensions',
            'type_handler_version': '1.10',
            'virtual_machine_extension_type': 'CustomScriptExtension'
        }

    def test_from_dict(self):
        assert VirtualMachineExtension.from_dict(self.raw).virtual_machine_extension_type == "CustomScriptExtension"


if __name__ == "__main__":
    unittest.main()

Result:

confab@bf0bc13796ae:/data$ python3 /tmp/bug.py
E
======================================================================
ERROR: test_from_dict (__main__.TestVirtualMachineExtensionSerialization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1357, in _deserialize
    raise KeyError('Use twice the key: "{}"'.format(attr))
KeyError: 'Use twice the key: "virtual_machine_extension_type"'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/bug.py", line 34, in test_from_dict
    assert VirtualMachineExtension.from_dict(self.raw).virtual_machine_extension_type == "CustomScriptExtension"
  File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 322, in from_dict
    return deserializer(cls.__name__, data, content_type=content_type)
  File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1294, in __call__
    return self._deserialize(target_obj, data)
  File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1364, in _deserialize
    raise_with_traceback(DeserializationError, msg, err)
  File "/usr/local/lib/python3.7/site-packages/msrest/exceptions.py", line 51, in raise_with_traceback
    raise error.with_traceback(exc_traceback)
  File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1357, in _deserialize
    raise KeyError('Use twice the key: "{}"'.format(attr))
msrest.exceptions.DeserializationError: Unable to deserialize to object: type, KeyError: 'Use twice the key: "virtual_machine_extension_type"'