terraform-provider-nomad: Terraform fails to notice when a Nomad job has changed

This issue was originally opened by @blalor as hashicorp/terraform#14038. It was migrated here as part of the provider split. The original body of the issue is below.

Terraform Version

Terraform v0.9.3

Affected Resource(s)

nomad_job

Terraform Configuration Files

provider "nomad" {
    address = "http://localhost:4646"
}

resource "nomad_job" "example" {
    jobspec = "${file("${path.module}/example.nomad")}"
}

Debug Output

Log for 2nd terraform apply: apply.log.txt

Console output:

$ TF_LOG=TRACE TF_LOG_PATH=apply.log.txt terraform apply
nomad_job.example: Refreshing state... (ID: example)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Expected Behavior

Terraform should have noticed that the job had changed in the cluster and updated it.

Actual Behavior

Nuttin’, honey.

Steps to Reproduce

nomad agent -dev
terraform apply
nomad status example shows one allocation for task group grp

change task group count:

 curl -sfS localhost:4646/v1/job/example | \
     jq '.TaskGroups[0].Count = 2 | {"Job": .}' | \
     curl -is -X PUT -d@- localhost:4646/v1/job/example

nomad status example shows two allocations for example.grp
terraform plan shows no change
terraform apply makes no change
nomad status example still shows two allocations

About this issue

Original URL
State: open
Created 7 years ago
Reactions: 11
Comments: 37 (18 by maintainers)

Commits related to this issue

Resolved. Notice when a Nomad job has changed. Resolved #1. Terraform fails to notice when a Nomad job has changed To have this, you need to maximize the use of Job Meta. (eg: Docker Image) This patc... — committed to liemle3893/terraform-provider-nomad by saboteurkid 4 years ago
the namespace trials #1 - everything is broken — committed to hashicorp/terraform-provider-nomad by gogococo 4 years ago
Add test for https://github.com/hashicorp/terraform-provider-nomad/issues/1 — committed to remilapeyre/terraform-provider-nomad by remilapeyre 4 years ago
job: read job submission source If available, use the job submission source to detect changes to `jobspec`. This can mitigate drift detection problems such as #1. — committed to hashicorp/terraform-provider-nomad by lgfa29 6 months ago
job: prevent partial update on error The default behaviour of the Terraform SDK is to copy the plan result into state which could result in partial state updates, where Terraform state is updated, bu... — committed to hashicorp/terraform-provider-nomad by lgfa29 6 months ago
job: prevent partial update on error The default behaviour of the Terraform SDK is to copy the plan result into state which could result in partial state updates, where Terraform state is updated, bu... — committed to hashicorp/terraform-provider-nomad by lgfa29 6 months ago
job: prevent partial update on error (#412) The default behaviour of the Terraform SDK is to copy the plan result into state which could result in partial state updates, where Terraform state is up... — committed to hashicorp/terraform-provider-nomad by lgfa29 6 months ago
job: read job submission source If available, use the job submission source to detect changes to `jobspec`. This can mitigate drift detection problems such as #1. — committed to hashicorp/terraform-provider-nomad by lgfa29 6 months ago
job: add support for job submissions (#405) If available, use the job submission source to detect changes to `jobspec`. This can mitigate drift detection problems such as #1. Read HCL2 variables ... — committed to hashicorp/terraform-provider-nomad by lgfa29 6 months ago

Most upvoted comments

Hi @ketzacoatl, I wholeheartedly agree that this is something that must be addressed. This shortcoming makes the Nomad provider just about useless for Day 2 operations.

As noted in #15 , there are a few different options for addressing this. The solution partially hinges on whether we want to try to find a general solution that doesn’t require modifying and re-releasing the provider every time Nomad releases a new version. On the other hand, with the Nomad product team taking ownership of this TF provider, we can potentially address such a workflow a little better than before. And it may be prudent to find a temporary solution to this issue while we work on a generic/unversioned provider.

The Nomad team is committed to addressing this; I will post an update here soon giving an idea as to the timetable, but my intention is to either resolve this issue as part of the upcoming 1.3.0 version of the provider (targeting Nomad 0.8.x) or the 1.4.0 version (targeting Nomad 0.9.x).

Thank you for your patience and persistence on this issue.

cgbaker on Dec 6, 2018

Update: this was not resolved in the 1.4.0 release of the provider, but we’re still tracking this.

cgbaker on Jun 5, 2019

Hi @eliburke 👋

No plans to get this fixed unfortunately. The work on #149 was a brave attempt to try and map all possible jobspec fields into a Terraform resource schema, but that was not a sustainable approach. As Nomad evolves it becomes almost impossible to manually keep up.

We think that a better approach would be to leverage the Nomad OpenAPI project that is able to auto-generate a Nomad job spec, and the Terraform Provider Framework which is a new and more flexible way to create providers.

But this will take a significant effort that is not in our roadmap yet.

lgfa29 on Jul 15, 2022

Nomad 0.8.x API support is available in the Nomad v1.3.0 that released today. I will look at finding a longer-term solution for this in upcoming versions; having said that, even if we continue version the Nomad providers, we pledge to be much more responsive in updating the Nomad provider going forward.

cgbaker on Dec 20, 2018

Thanks for the feedback @chuckyz. Creating a new resource seems to be the best approach we have right now, so we will move forward with that (but with a better name than nomad_job_new 😄).

lgfa29 on Aug 24, 2020

The underlying issue here also produces the following behavior:

run a job with the terraform provider
nomad stop that job manually
re-apply the terraform plan, the job is still stopped

ketzacoatl on Feb 22, 2018

Reading @jorgemarey comment again, I noticed this part:

We detect changes by having a shasum of the rendered job and seeing if it changes with the value stored for the previous resource version.

While not quite related to this issue, it made me realize that we can use the job submission data (if available) to detect changes to jobspec, which should help mitigate drift problems like this when the submission data is available.

lgfa29 on Dec 19, 2023

Is there any plan to release v1.5 soon? Is there more work to finish out this feature? #149 hasn’t been touched in almost 2 years.

eliburke on Jul 8, 2022

we were waiting for the update 2.0 plugin SDK to see if it helped deal with this. we’re actively looking at it now; we want this issue dealt with in the next few months as part of the Nomad 1.0 milestone.

cgbaker on Jul 14, 2020

@paddycarver, I’d love to help resolve this problem, as IMO it prevents serious use of Terraform to manage nomad, do you have any guidance or recommendations on how you would like to address the problem?

cc @katbyte / @radeksimko

ketzacoatl on Jun 28, 2018

Just made a comment on https://github.com/hashicorp/terraform-provider-nomad/pull/238#issuecomment-1833877526 about this.

jorgemarey on Nov 30, 2023

Thanks for the feedback @chuckyz. Creating a new resource seems to be the best approach we have right now, so we will move forward with that (but with a better name than nomad_job_new smile).

Would it make sense to incorporate it in #149?

Yup. #149 is being developed exactly to solve this issue 👍

lgfa29 on Oct 29, 2020

I am stoked to see this on the 1.5 milestone, rock on!

ketzacoatl on Sep 13, 2019

@cgbaker, With renewed development efforts here, what are your thoughts on this issue?

ketzacoatl on Dec 6, 2018

@mitchellh I’d be very grateful for your feedback/guidance here.

ketzacoatl on Jul 20, 2018

My observation is the same as @ketzacoatl.

It would be just more operator friendly if users could interact via Terraform, rather than nomad status

My versions are Terraform 0.11.5 Nomad (built from master) (0.8.0-dev)

shantanugadgil on Mar 31, 2018

Hi @ketzacoatl! I can’t say for sure whether this is a verified bug, nor can I explain the behaviour. I’ll try to look into this soon and come back with a bit more information, and a fix if necessary. Apologies for the delays on this.

paddycarver on Feb 22, 2018