cortex: Can't statically (w/o SRV lookup) set memcached servers
Hello, sorry for all the issues I’m creating today 😄
I have the ingesters working but not being able to push to memcached (in aws) I tried pushing an item with another pod in the same network and namespace and I was able to do it:
root@ubuntu-649f64d646-482db:/# telnet test-cortex.xxxx.cfg.use1.cache.amazonaws.com 11211
Trying 10.80.17.123...
Connected to test-cortex.xxxx.cfg.use1.cache.amazonaws.com.
Escape character is '^]'.
get greeting
END
set greeting 1 0 11
Hello world
STORED
get greeting
VALUE greeting 1 11
Hello world
END
But ingester is complaining:
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.529243532Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.535683451Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.539257288Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
ingester-84c4b85c4d-sfdsm ingester level=error ts=2019-07-10T16:49:07.541753931Z caller=memcached.go:230 msg="failed to put to memcached" name= err="memcache: no servers configured or available"
The relevant part in the config:
-memcached.hostname=test-cortex.xxxxx.cfg.use1.cache.amazonaws.com:11211
-memcached.timeout=100ms
-memcached.service=memcached
I wonder if it’s related to https://github.com/cortexproject/cortex/issues/1501
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (4 by maintainers)
As I read in Loki’s Slack channel I solved it using “addresses” instead “host” with the address of my memcached. It worked.
PS: “addresses” option isn’t documented in the official documentation.
Source: https://grafana.slack.com/archives/CEPJRLQNL/p1616448036226400?thread_ts=1616253261.201300&cid=CEPJRLQNL
Sure @chancez. Feel free to move on. I just wanted to warn you, in case it was a large work (I didn’t check).