prometheus-operator: Problem with secret refresh v0.42.1

What happened?

We were running v0.39.0 with additionalScrapeConfigs: Runtime changes to the referenced secret were being updated by prometheus-operator and the config changes loaded into Prometheus. After upgrading to 0.42.1 when prom-op starts the config is correctly loaded from the additionalScrapeConfigs secret and Prometheus updated. However after the initial update the configuration in Prometheus doesn’t get updated when the configuration in the secret later changes.

when the secret is updated and resourceVersion changes then the prom-op logs show

level=debug ts=2020-09-28T13:08:47.038331903Z caller=operator.go:1583 component=prometheusoperator msg="updating Prometheus configuration secret"
level=debug ts=2020-09-28T13:08:47.042560064Z caller=operator.go:909 component=prometheusoperator msg="Secret updated"

But /etc/prometheus/config/prometheus.yaml.gz in the prometheus-config-reloader is not updated, nor is it updated after the resyncPeriod, however restarting the prometheus-operator binary does update the configuration, but only when it first initialises, after that it goes back to the behaviour described above.

Did you expect to see something different?

I expected changes to the additionalScrapeConfigs secret to continue to be loaded into Prometheus.

How to reproduce it (as minimally and precisely as possible):

Run v0.42.0 prom-op with additionalScrapeConfigs. Once everything is running, alter the configuration in the secret, the changes will not be loaded into Prometheus

Environment

Prometheus-operator v0.42.1, Prometheus v2.20.1, K8s v1.18.6

• Prometheus Operator version:

Prometheus-operator v0.42.1

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T21:52:18Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, etc.

• Manifests:

I can send full configs if relevant, but I think there is enough in the description.

• Prometheus Operator Logs:

see above

Anything else we need to know?:

Assuming the problem was something to do with this https://github.com/prometheus-operator/prometheus-operator/pull/3355 I tried setting --secret-field-selector='type=Opaque'