azure-sdk-for-go: ARM as a code does not work

Goal:

To be able to construct all Azure objects in code and Marsal/Unmarshal them in a transparent way while interacting with ARM API and constructing versioned payload for ARP API to consume

We would like to be able to construct versioned JSON from Objects and interact with ARM Templates instead of individual versioned APIS.
We would like to be able to generate ARM Templates from code without the need to manipulate JSON files before we produce the files.

Use-case: We are using SDK to create ARM templates as a code. We are using v24 SDK version for now due to internal dependencies. We managed our ARM templates as a code in the form of:

func (g *generator) vnet() *network.VirtualNetwork {
	return &network.VirtualNetwork{
		....
		Name:     to.StringPtr(vnetName),
		Type:     to.StringPtr("Microsoft.Network/virtualNetworks"),
		Location: to.StringPtr(g.cs.Location),
	}
}

And later on:

t := arm.Template{
		Schema:         "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
		ContentVersion: "1.0.0.0",
		Resources: []interface{}{
			g.vnet(),
			g.ipAPIServer(),
			.....
		},
	}

After this, we have object representation of ARM template.

Challenges:

SDK representation does not contain Version field[1] (and it is mandatory in JSON payload when sending to API server), so we have to do some hacking to inject them. We do this based on Type and map we maintain. But to do this we need to do[2]: a) marshal to bytes b) Marshal back to var azuretemplate map[string]interface{} to keep values c) Do manipulation and inject required fields (version) d) work with azuretemplate map[string]interface{} instead of more native object.
A newer version of SDK does not contain Type field too in the struct[3], so our model does not work as there is no way to map objects to the version anymore.

Suggestion:

Remove // Type - READ-ONLY; Resource type. types from the SDK, so enabling usage SDK object in more code native way.

Potentially introduce the ability to construct ARM code as a code with all subfields required by ARM to accept the payload.

[1] Due to the big file you might need to go RAW on github. https://github.com/Azure/azure-sdk-for-go/blob/v27.0.0/services/network/mgmt/2018-07-01/network/models.go#L19030-L19053

[2]

b, err := json.Marshal(template)
	if err != nil {
		return nil, err
	}

	var azuretemplate map[string]interface{}
	err = json.Unmarshal(b, &azuretemplate)
	if err != nil {
		return nil, err
	}

	err = arm.FixupAPIVersions(azuretemplate, versionMap)
	if err != nil {
		return nil, err
	}

[3] https://github.com/Azure/azure-sdk-for-go/blob/v28.0.0/services/network/mgmt/2018-07-01/network/models.go#L18798-L18814

/cc @amanohar @jim-minter @serbrech @ritazh

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 3
Comments: 24 (7 by maintainers)

Most upvoted comments

@jhendrixMSFT Thanks. I was looking for this.

I think generating separate types used solely for constructing a template is the likely solution; it would probably go in a sub-package to keep all template-specific code grouped together, e.g. github.com/Azure/azure-sdk-for-go/services/network/mgmt/2019-06-01/network/networkdeployment.

I think, having separate types will work for use cases when you construct template from scratch. But, will be tricky for use cases when you want to get some existing resource from the REST API and create a template based on it: there will be type mismatch and the SDK user will have to do a lot of manual data copying.

I’m currently looking into options ot solve this using type embeding. Hopefully will come back to you with something early next week.

m1kola on Sep 26, 2019

@tombuildsstuff I think type TemplateDeploymentObject struct would not work as proposed as each resource is versioned individually, and is treated as a separate object in the payload.

We already wrapping this in the work of:

t := arm.Template{
		Schema:         "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
		ContentVersion: "1.0.0.0",
		Resources: []interface{}{
			g.vnet(),
			g.loadBalancer(), 
		},
	}

| out of interest since you’re importing the SDK to build up the structs anyway - have you considered using the SDK’s Clients (and associated waiters) rather than generating an ARM Template?

When operating with big numbers of objects it would cause a separate set of problems (which would require much more maintenance to keep up). In an example, our code currently handles dependencies, arm.FixupDepends so we can just pre-append multiple objects into the Resources slice and don’t care about ordering and creation flow. ARM will do this for us. Updates of the objects become seamless too.

Overall it makes our code base testable. As not we have defined the output of our “Infrastructure” which we can version between releases, and we can use hash’es, mocking, etc to test our code behavior. If we use clients directly, we don’t have defined view of our Object without high-level API mocking for each endpoint. And lack of mock clients in the codebase we would need to mock each and every client * version to have extensive testing.

And finally - constructing code this way is so much nicer 😃

mjudeikis on Sep 17, 2019