cortex: Can't statically (w/o SRV lookup) set memcached servers

Hello, sorry for all the issues I’m creating today 😄

I have the ingesters working but not being able to push to memcached (in aws) I tried pushing an item with another pod in the same network and namespace and I was able to do it:

root@ubuntu-649f64d646-482db:/# telnet test-cortex.xxxx.cfg.use1.cache.amazonaws.com 11211
Trying 10.80.17.123...
Connected to test-cortex.xxxx.cfg.use1.cache.amazonaws.com.
Escape character is '^]'.
get greeting
END
set greeting 1 0 11
Hello world
STORED
get greeting
VALUE greeting 1 11
Hello world
END

But ingester is complaining:

ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.529243532Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.535683451Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.539257288Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.541753931Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"

The relevant part in the config:

      -memcached.hostname=test-cortex.xxxxx.cfg.use1.cache.amazonaws.com:11211
      -memcached.timeout=100ms
      -memcached.service=memcached

I wonder if it’s related to https://github.com/cortexproject/cortex/issues/1501

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 15 (4 by maintainers)

Most upvoted comments

As I read in Loki’s Slack channel I solved it using “addresses” instead “host” with the address of my memcached. It worked.

PS: “addresses” option isn’t documented in the official documentation.

Source: https://grafana.slack.com/archives/CEPJRLQNL/p1616448036226400?thread_ts=1616253261.201300&cid=CEPJRLQNL

Sure @chancez. Feel free to move on. I just wanted to warn you, in case it was a large work (I didn’t check).