experimenter: Experiment end date can be incorrect if overridden in external config

Ah interesting, we could potentially use that to display “Experiment analysis was skipped” in the sidebar instead of “Experiment results not yet ready”. Could that be helpful at all?

If an analysis is skipped, then there won’t be any analysis result data. So GCS won’t have any JSON files for overall, weekly, daily, … results. This is the case for the following experiment which has skip=true: https://experimenter.services.mozilla.com/nimbus/set-default-as-first-screen-100-roll-out

https://github.com/mozilla/jetstream/pull/860 adds an external_config JSON object to the metadata jetstream exports for each experiment. external_config is null if no external config exists or if the external config does not override any of the parameters:

The experiment parameters that have not been changed and are the same as in Experimenter will be null. I also added skip which indicates that analysis is skipped for an experiment and url which links to the config in Github. Does this look reasonable?

So, if these properties are exposed via the analysis endpoint, then maybe we should override display of the local database values with the latest from analysis? Maybe even update them in the DB locally?

Yeah, that’s what I was thinking with my “I wonder if we should or if it’s plausible to pull in that metadata and change the experiment data from within Experimenter” comment. But… if the changes only impact the analysis data, updating the DB might be more trouble than it’s worth IMO, if we can simply display the overrides in a nice way at the top of the Results page, like with “Analysis date range: X - X” and probably also mention if there was a control branch/proposed enrollment override when one exists.

Probably should get @jaredlockhart’s opinion.

To me, this sounds like we “pass the ball” from Experimenter Console to Jetstream after experiment launch. (i.e. the source of truth has moved, no more edits allowed from Experimenter Console)

Yes, that sounds right 👍

So, if these properties are exposed via the analysis endpoint, then maybe we should override display of the local database values with the latest from analysis? Maybe even update them in the DB locally?

I think that would be ideal. Not sure how much work that would be on the Experimenter side. It might also be good enough to just show the parameters the analysis is based in some section on the Results page for now.

This is probably a function of me not yet being familiar with this part of Nimbus, but: Does this point to a general problem with source of truth for an experiment?

Should the end date of an experiment be overridden in external config? What’s the use case & rationale for that?

Not asserting that this shouldn’t happen, so much as trying to understand when & why it would happen. e.g. Is it expected that Experimenter Console should poll external configs to update its database and published DTOs? Do changes need to be pushed to Remote Settings based on external configs?

The config can override reference_branch, end_date, start_date, and proposed_enrollment. I think more properties could be added to that list in the future, but nothing is planned at the moment. Yeah, updating the experiment settings in Experimenter would probably be ideal.