cortex: The -target=all flag does not include alertmanager

I’m running a 3 host cluster with 1.3.0-rc.1 and I’m trying to run Alertmanager serivce with the following config:

alertmanager:
  external_url: 'https://alerts.example.org/'
  cluster_bind_address: '10.1.33.0:9094'
  cluster_advertise_address: '10.1.33.0'
  data_dir: '/var/tmp/cortex/alertmnanager'
  retention: 120h
  storage:
    type: local
    local:
      path: '/var/tmp/cortex/alerts'

But I don’t see the service listening on port 9094 at all:

admin@master-03.do-ams3.metrics.hq:~ % sudo netstat -lpnt | grep cortex
tcp        0      0 10.1.33.0:9095          0.0.0.0:*               LISTEN      27780/cortex        
tcp6       0      0 :::9092                 :::*                    LISTEN      27780/cortex 

And I don’t see the service in the return value from /services:

 % curl -s http://10.1.33.0:9092/services | grep -A1 '<tr>' | grep td
					<td>memberlist-kv</td>
					<td>store</td>
					<td>server</td>
					<td>ring</td>
					<td>distributor</td>
					<td>ingester</td>
					<td>query-frontend</td>
					<td>querier</td>
					<td>ruler</td>
					<td>table-manager</td>

Both configured directories exist and are owned by service user. I’m not sure what more I’m supposed to do.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 24 (21 by maintainers)

Most upvoted comments

This should be front-and-center in the docs for Alertmanager.

Yes we should define “tenant”. #3023

This looks ok: https://en.wikipedia.org/wiki/Multitenancy “A tenant is a group of users who share a common access with specific privileges to the software instance.” In Cortex, those “users” need to be understood as both human and system accounts.

The way we use it in Cortex a single tenant could have multiple Prometheus.

That is a very good point. Can we reopen this issue, and rename it to document “all” target properly?

Though question, what’s the advantage of running alertmanager via Cortex as opposed to running Alertmanager directly?

I will leave answering that question to others, as my knowledge of AM is very low 😃 (Sorry!) (@gotjosh @jtlisi @annanay25?)