azure-sdk-for-python: VirtualMachineExtension metadata error handling response from create_or_update: KeyError "Use twice the key"
- Package Name: azure-mgmt-compute
- Package Version: 12.0.0
- Operating System: Debian 10
- Python Version: 3.7.3
Describe the bug
After calling VirtualMachineExtension.create_or_update, the response coming back from the server cannot go through a deserialize/serialize cycle, likely due to something with the metadata handling (there are two fields with the key “type” in the structure, but at different levels):
Calling create_or_update with:
('confab-test-virtual-machine-extension-rg',
'confab-test-virtual-machine-extension-vm',
'confab-test-virtual-machine-extension',
{'auto_upgrade_minor_version': True,
'location': 'eastus',
'protected_settings': {'commandToExecute': 'powershell.exe -ExecutionPolicy '
'Unrestricted -File '
'appgatewayurl.ps1'},
'publisher': 'Microsoft.Compute',
'settings': {'fileUris': ['https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1']},
'tags': {'_tuono_version': 'v0.1.2+unittest'},
'type_handler_version': '1.10',
'virtual_machine_extension_type': 'CustomScriptExtension'})
Note there is no “type” property; the server fills that in on the response:
(Pdb) pp result._result
<azure.mgmt.compute.v2019_12_01.models._models_py3.VirtualMachineExtension object at 0x7ff6952fdfd0>
(Pdb) pp result._result.as_dict()
{'auto_upgrade_minor_version': True,
'id': '/subscriptions/.../resourceGroups/confab-test-virtual-machine-extension-rg/providers/Microsoft.Compute/virtualMachines/confab-test-virtual-machine-extension-vm/extensions/confab-test-virtual-machine-extension',
'location': 'eastus',
'name': 'confab-test-virtual-machine-extension',
'provisioning_state': 'Succeeded',
'publisher': 'Microsoft.Compute',
'settings': {'fileUris': ['https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1']},
'tags': {'_tuono_version': 'v0.1.2+unittest'},
'type': 'Microsoft.Compute/virtualMachines/extensions',
'type_handler_version': '1.10',
'virtual_machine_extension_type': 'CustomScriptExtension'}
Now if you attempt to deserialize this result it will fail with the error:
(Pdb) from azure.mgmt.compute.models import VirtualMachineExtension
(Pdb) pp VirtualMachineExtension.from_dict(result._result.as_dict())
*** msrest.exceptions.DeserializationError: Unable to deserialize to object: type, KeyError: 'Use twice the key: "virtual_machine_extension_type"'
When I get the error indicating “Use twice the key”, which is a KeyError ,I added a breakpoint so I could investigate:
> /home/jking/confab/confab-cloud/.tox/debug/lib/python3.7/site-packages/msrest/serialization.py(1358)_deserialize()
-> raise KeyError('Use twice the key: "{}"'.format(attr))
(Pdb) l
1353 for key_extractor in self.key_extractors:
1354 found_value = key_extractor(attr, attr_desc, data)
1355 if found_value is not None:
1356 if raw_value is not None and raw_value != found_value:
1357 breakpoint()
1358 -> raise KeyError('Use twice the key: "{}"'.format(attr))
1359 raw_value = found_value
1360
1361 value = self.deserialize_data(raw_value, attr_desc['type'])
1362 d_attrs[attr] = value
1363 except (AttributeError, TypeError, KeyError) as err:
(Pdb) pp data
{'auto_upgrade_minor_version': True,
'id': '/subscriptions/.../resourceGroups/confab-test-virtual-machine-extension-rg/providers/Microsoft.Compute/virtualMachines/confab-test-virtual-machine-extension-vm/extensions/confab-test-virtual-machine-extension',
'location': 'eastus',
'name': 'confab-test-virtual-machine-extension',
'provisioning_state': 'Succeeded',
'publisher': 'Microsoft.Compute',
'settings': {'fileUris': ['https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples/master/application-gateway/iis/appgatewayurl.ps1']},
'tags': {'_tuono_version': 'v0.1.2+unittest'},
'type': 'Microsoft.Compute/virtualMachines/extensions',
'type_handler_version': '1.10',
'virtual_machine_extension_type': 'CustomScriptExtension'}
(Pdb) pp attr
'virtual_machine_extension_type'
(Pdb) pp attr_desc
{'key': 'properties.type', 'type': 'str'}
(Pdb) pp found_value
'Microsoft.Compute/virtualMachines/extensions'
(Pdb) pp key_extractor
<function last_rest_key_case_insensitive_extractor at 0x7f9a2fab1048>
(Pdb) pp attributes
{'auto_upgrade_minor_version': {'key': 'properties.autoUpgradeMinorVersion',
'type': 'bool'},
'force_update_tag': {'key': 'properties.forceUpdateTag', 'type': 'str'},
'id': {'key': 'id', 'type': 'str'},
'instance_view': {'key': 'properties.instanceView',
'type': 'VirtualMachineExtensionInstanceView'},
'location': {'key': 'location', 'type': 'str'},
'name': {'key': 'name', 'type': 'str'},
'protected_settings': {'key': 'properties.protectedSettings',
'type': 'object'},
'provisioning_state': {'key': 'properties.provisioningState', 'type': 'str'},
'publisher': {'key': 'properties.publisher', 'type': 'str'},
'settings': {'key': 'properties.settings', 'type': 'object'},
'tags': {'key': 'tags', 'type': '{str}'},
'type': {'key': 'type', 'type': 'str'},
'type_handler_version': {'key': 'properties.typeHandlerVersion',
'type': 'str'},
'virtual_machine_extension_type': {'key': 'properties.type', 'type': 'str'}}
It appears the metadata for “virtual_machine_extension_type” is referencing the “type” field instead of the “virtual_machine_extension_type” field. It extracts the wrong content, and then the code thinks that a key appears twice in the data.
incorrect:
(Pdb) pp key_extractor("virtual_machine_extension_type", {"key": "properties.type", "type": "str"}, data)
'Microsoft.Compute/virtualMachines/extensions'
correct:
(Pdb) pp key_extractor("virtual_machine_extension_type", {"key": "properties.virtual_machine_extension_type", "type": "str"}, data)
'CustomScriptExtension'
To Reproduce Run the following script:
import unittest
from azure.mgmt.compute.models import VirtualMachineExtension
class TestVirtualMachineExtensionSerialization(unittest.TestCase):
def setUp(self):
self.raw = {
'auto_upgrade_minor_version': True,
'id': (
'/subscriptions/.../resourceGroups/confab-test-virtual-machine-extension-2-rg'
'/providers/Microsoft.Compute/virtualMachines/confab-test-virtual-machine-extension-2-vm'
'/extensions/confab-test-virtual-machine-extension-2'
),
'location': 'eastus',
'name': 'confab-test-virtual-machine-extension-2',
'provisioning_state': 'Succeeded',
'publisher': 'Microsoft.Compute',
'settings': {
'fileUris': [
(
'https://raw.githubusercontent.com/Azure/azure-docs-powershell-samples'
'/master/application-gateway/iis/appgatewayurl.ps1'
)
]
},
'tags': {'_tuono_version': 'v0.1.2+unittest'},
'type': 'Microsoft.Compute/virtualMachines/extensions',
'type_handler_version': '1.10',
'virtual_machine_extension_type': 'CustomScriptExtension'
}
def test_from_dict(self):
assert VirtualMachineExtension.from_dict(self.raw).virtual_machine_extension_type == "CustomScriptExtension"
if __name__ == "__main__":
unittest.main()
Result:
confab@bf0bc13796ae:/data$ python3 /tmp/bug.py
E
======================================================================
ERROR: test_from_dict (__main__.TestVirtualMachineExtensionSerialization)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1357, in _deserialize
raise KeyError('Use twice the key: "{}"'.format(attr))
KeyError: 'Use twice the key: "virtual_machine_extension_type"'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/bug.py", line 34, in test_from_dict
assert VirtualMachineExtension.from_dict(self.raw).virtual_machine_extension_type == "CustomScriptExtension"
File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 322, in from_dict
return deserializer(cls.__name__, data, content_type=content_type)
File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1294, in __call__
return self._deserialize(target_obj, data)
File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1364, in _deserialize
raise_with_traceback(DeserializationError, msg, err)
File "/usr/local/lib/python3.7/site-packages/msrest/exceptions.py", line 51, in raise_with_traceback
raise error.with_traceback(exc_traceback)
File "/usr/local/lib/python3.7/site-packages/msrest/serialization.py", line 1357, in _deserialize
raise KeyError('Use twice the key: "{}"'.format(attr))
msrest.exceptions.DeserializationError: Unable to deserialize to object: type, KeyError: 'Use twice the key: "virtual_machine_extension_type"'
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (9 by maintainers)
It fixed a test in the repository that was expecting an error, so I would agree with you.
I’d like to stress the urgency for a fix on this… Any virtual machine with an extension installed cannot be deserialized by azure-mgmt-compute 12.0 (the current version).