terraform-provider-grafana: Cannot create (slack) grafana_contact_point - error 400 type should not be empty

Terraform Version

  • Terraform v1.1.7
  • Terraform Grafana Provider: 1.28.0
  • Grafana: Enterprise v9.1.2

Affected Resource(s)

Please list the resources as a list, for example:

  • grafana_contact_point

If this issue appears to affect multiple resources, it may be an issue with Terraform’s core, so please mention this.

Terraform Configuration Files

resource "grafana_contact_point" "testing" {
    name = "testing"
    slack {
        url = var.slack_webhook
        recipient = "testing-channel-abc"
    }
}

Debug Output

Contains too much information I cannot share.

Panic Output

grafana_contact_point.testing: Creating...
╷
│ Error: status: 400, body: {"message":"invalid object specification: type should not be an empty string","traceID":"00000000000000000000000000000000"}
│
│   with grafana_contact_point.testing,
│   on alert_contact_points.tf line 1, in resource "grafana_contact_point" "testing":
│    1: resource "grafana_contact_point" "testing" {
│
╵

Expected Behavior

Endpoint is created

Actual Behavior

Grafana throws an error 400: “invalid object specification: type should not be an empty”

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. Use my config as described above
  2. terraform apply

Important Factoids

Runs on K8s with Istio

Other stuff

  • Also tried it against Grafana 9.x version.
  • I’ve checked, but type is not a field I can add, it should be inferred from the list.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Comments: 19 (9 by maintainers)

Most upvoted comments

Core grafana issue on this: https://github.com/grafana/grafana/issues/54556

As far as I can tell, our docs are very vague as to which flavors of “database-compatible” cloud tools we actually support. This page simply says MySQL and nothing else… https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#database

In practice, we don’t actively test Grafana on every “MySQL-compatible” database product on every cloud provider… however I can confirm that both native MySQL and GCP Cloud SQL w/MySQL interface work correctly here, as we use both internally often. I’m sharing the issue above around in order to get a more definitive answer on this.

@alexweav following up for @wiardvanrij - the RDS engine version we are using is 5.7.mysql_aurora.2.07.2.

Thanks!

So, this caused the server to return a 500. You can see this first request in the log.

This then exposed a second bug deep in the golang-client project, underneath this provider. It has retry logic for failed requests. When a retry happens, it tries to read an io.Reader containing the request body twice!

The second read returns an empty slice of bytes, which is then marshalled into an empty Contact Point struct, which has… an empty string for Type! Hence, type should not be an empty string.

This bug has been around for about a year now, interesting that we only just now found it.