terraform-provider-google: Slack Notification Channel creation via terraform no longer works

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

I’m using v1.1.5

Affected Resource(s)

  • google_monitoring_notification_channel

Terraform Configuration Files

resource "google_monitoring_notification_channel" "alert_notification_channel" {
  enabled = true
  display_name = "test slack alerts"
  type = "slack"
  labels = {
    "channel_name": "#my-channel-name"
  }
  sensitive_labels {
    "auth_token": "xoxb-0000000000-1111111111111-Abbc1defg2HIjkL3MNoPQRs4"
  }

Debug Output

google_monitoring_notification_channel.alert_notification_channel["test slack alerts"]: Refreshing state... [id=projects/my-gcp-project/notificationChannels/1231231231231231231]

  # google_monitoring_notification_channel.alert_notification_channel["test slack alerts"] has changed
  ~ resource "google_monitoring_notification_channel" "alert_notification_channel" {
        id                  = "projects/my-gcp-project/notificationChannels/1231231231231231231"
        name                = "projects/my-gcp-project/notificationChannels/1231231231231231231"
      + user_labels         = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Panic Output

Expected Behavior

When we attach this notification channel to a GCP Monitoring Alert, and the Alert is triggered, an alert message should be sent to the provided slack channel.

Actual Behavior

When an incident arises and the alert happens, no Slack messages are sent.

However, if we create Notification Channels for Pubsub/Pagerduty/etc via Terraform, these all work as expected.

My guess is that this is related to GCP’s update for the creation of Slack notification channels… now, when you create a Slack notification channel in the GCP console, it pulls up a new screen/UI that asks for the slack channel you want to connect to, and auto-generates a connection to the slack channel. If I create a Slack notification channel in the GCP console using this method and attach it to an alarm, the slack messages are sent as-expected when an incident is raised. But, when I send alerts through the Slack notification channel created by terraform, no slack messages are sent when an incident is raised.

Steps to Reproduce

  1. terraform apply
  2. Attach notification channel to a GCP monitoring alert
  3. Trigger the alert to raise an incident, which should cause a message to be sent to the provided slack channel

Important Factoids

I created a Slack App via https://api.slack.com/apps and installed the app in my Slack workspace. I invited the Slack App bot into the channel we are trying to notify via the Notification Channel. I retrieved the auth_token from Settings -> Install App -> Bot User OAuth Token, and am passing that value to my terraform resource.

I’ve tried creating notification channels linked to various different slack apps and different Slack channels, but none of them will work when spun up with Terraform.

References

Similar to https://github.com/hashicorp/terraform-provider-google/issues/9564 . Seems like Slack notification channels simply don’t work when they are created via terraform.

  • b/275101438

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Reactions: 78
  • Comments: 21

Most upvoted comments

I think I found a workaround for this if you want to define it all in terraform:

resource "google_monitoring_notification_channel" "website_slack_alerts" {
  project      = "project123"
  display_name = "Slack Staging Website Alerts Channel"
  enabled      = "true"
  type         = "slack"
  labels = {
    "channel_name" = "#test-channel"
  }
  sensitive_labels {
    auth_token = data.google_secret_manager_secret_version.slack_bot_user_oauth_token.secret_data
  }

  depends_on = [ 
    google_secret_manager_secret.bot_secret
  ]
}

data "google_secret_manager_secret_version" "slack_bot_user_oauth_token" {
  secret  = "staging-website-slack-notifications-bot-token"
  project = "project123"

  depends_on = [ 
    google_secret_manager_secret.bot_secret
  ]
}

# This creates the secret, but not the data.  Add it manually.  
# The apply will fail till this is added because website_slack_alerts can't get the secret version!
# You could use secret_manager_secret_version if you wanted the secret in code too.
# This will be the token that starts with "xoxb"
resource "google_secret_manager_secret" "bot_secret" {
  project   = "project123"
  secret_id = "staging-website-slack-notifications-bot-token"

  replication {
    automatic = true
  }
}

On the slack side, I created the app, and added the channel.join and chat.write permissions. I also manually installed it in the channel I was targeting after installing it to the workspace. The token is the Bot User OAuth Token from the OAuth and Permissions section.

I’m using version 4.46.0 of the provider.

Hope this helps!

Running into the same problem. As described by @johan-torres-data, the team field seems to be missing. When inspecting requests sent by gcp’s to Slack’s API I see that it tries to send, next to the auth_token and channel_name; the team name. These are valid API labels, but for some reason Terraform doesn’t accept team during configuration.

Hi everyone, same issue here, I tried to create the channel using terraform but it didn’t work, so I created another one from the UI and that one works, the only difference that I noticed is one field called team that in my case is the Slack-workspace name, that label is not available within the “google_monitoring_notification_channel” resource. Each time that I want to set up a new channel Slack asks me to allow Monitoring to have some permissions, not pretty sure if could be related with that. In the next picture, you can see that the first has a team label, that was created by the UI the second one uses terraform. Notificationchannels

We actually got in touch with responsible team and this is what I heard back

The long term goal is to have full TF parity with console/cli. Lots of working being done at the moment. I have +1 the ticket and will be booking a follow up meeting with the PM to discuss progress

So I hope sometime in near future we would get this done.

Any update regarding this issue? As mentionned above the only solution in order to use the slack channel for me was to copy past in my alert resource the channel name that can be found with the following command:

gcloud beta monitoring channels list

Can concur here around the Team not being included. Current workaround is super ugly but essentially create the slack alert manually and copy down the notification channel ID which is something like:
"projects/<gcp_project_id>/notificationChannel/<NotificationchannelID>" Also frustrating that you can’t just pass the notification channel name. This “workaround” at the very least gives your created Alerts a working channel to reference in environments that will be stable for a while.

Yeah, the issue is resolved after it’s reported internally as well! We should be good at using the following:

                    labels={
                        "channel_name": <slack channel>,
                        "team": "<team name>",
                    },

Have migrated my workaround to the above in May this year which is working perfectly fine

Another possible (albeit terrifying workaround) is to use slacks feature to send emails to a channel:

  1. Rightclick the channel in slack
  2. View Channel Details
  3. Integrations
  4. Send Email to this channel
  5. Use the provided email with a email notification channel that can be configured through terraform

It’s not pretty but it works.