prometheus-operator: unable to change configuration

What did you do? I have a tectonic cluster. Prometheus-operator was installed as part of it. I want to increase resource requests and retention by editing Prometheus object (not sure if this is the best way) kubectl edit Prometheus k8s --namespace=tectonic-system After saving, Statefulset is getting recreated but after a few minutes, its getting killed. The new one, recreated after this has default settings.

What did you expect to see? Stateful set with updated configuration

What did you see instead? Under which circumstances? Always the same configuration

Environment Tectonic 1.6.7-tectonic.1 Kubernetes 1.6.7+tectonic.1 tectonic-monitoring 1.3.0

• Kubernetes version information:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.6", GitCommit:"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2", GitTreeState:"clean", BuildDate:"2017-06-16T18:34:20Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.7+coreos.0", GitCommit:"c8c505ee26ac3ab4d1dff506c46bc5538bc66733", GitTreeState:"clean", BuildDate:"2017-07-06T17:38:33Z", GoVersion:"go1.7.6", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

tectonic-installer

• Prometheus Operator Logs:

Nothing strange. Dozens of similar lines

ts=2017-07-17T06:18:54Z caller=operator.go:619 component=prometheusoperator msg="sync prometheus" key=tectonic-system/k8s
ts=2017-07-17T06:18:54Z caller=operator.go:972 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-07-17T06:18:54Z caller=operator.go:584 component=prometheusoperator msg="update handler" old=7003568 cur=7006925
ts=2017-07-17T06:18:54Z caller=operator.go:321 component=alertmanageroperator msg="update handler" old=7003568 cur=7006925
ts=2017-07-17T06:18:54Z caller=operator.go:619 component=prometheusoperator msg="sync prometheus" key=tectonic-system/k8s
ts=2017-07-17T06:18:54Z caller=operator.go:972 component=prometheusoperator msg="updating config skipped, no configuration change"
ts=2017-07-17T06:18:54Z caller=operator.go:584 component=prometheusoperator msg="update handler" old=7006925 cur=7006926
ts=2017-07-17T06:18:54Z caller=operator.go:321 component=alertmanageroperator msg="update handler" old=7006925 cur=7006926

In kubectl describe pod just before restoring default configuration.

  4s		4s		1	kubelet, <hostname>	spec.containers{prometheus-config-reloader}	Normal		Killing		Killing container with id docker://d5a160f8ee734b7423e088711f757f258a20b6120c4dc409a99f12541044e86a:Need to kill Pod
  2s		2s		1	kubelet, <hostname>	spec.containers{prometheus}			Normal		Killing		Killing container with id docker://0a08bab80d70dc12666940e3fa0a2d2fe51b28ec80ce7305181354eb84942e40:Need to kill Pod

Am I doing something wrong? Is there a correct way to change settings for the operator?