cortex: Alertmanager fails to read fallback config

Description

I’m trying to use fallback_config_file with Alertmanager service but it fails with:

 msg="GET /api/prom/configs/alertmanager (500) 78.226µs Response: \"Failed to initialize the Alertmanager\\n\"

Details

I’m running 1.4.0 binary release form GitHub and I have fallback_config_file configured to point to a file I configured the same way I used to for normal Alertmanager. When I check the logs I do not see either of these two errors: https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/alertmanager/multitenant.go#L186 https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/alertmanager/multitenant.go#L190 But when I query the API:

curl -sv http://localhost:9101/api/prom/configs/alertmanager -H 'X-Scope-OrgID: 0'

I get back:

Failed to initialize the Alertmanager

But the code that triggers this error doesn’t actually show what caused it because err is discarded: https://github.com/cortexproject/cortex/blob/23554ce028c090a4a3413ac0e35e5e1dc9fa929f/pkg/alertmanager/multitenant.go#L476-L485 So I have no clue what I’m doing wrong. The file is in place, it has read permissions for the service.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 35 (32 by maintainers)

Most upvoted comments

Just so you know, based on https://github.com/cortexproject/cortex/issues/3395#issuecomment-717144961 I’ve decided to abandon configuring Alertmanager via Cortex. Do with this issue what you will.

This part touches a bit on this subject, we provide several options to store per-tenant configuration (some for historical purposes), you can revise the flags for options - we recommend using any sort of object store for them (S3, GCS, etc).

Interesting. Let me try building your PR branch and see what actual error I get from it.