client_golang: API client Series() call can't handle minTime/maxTime set by prometheus
This is a bit of a convoluted error so I’ll do my best to explain.
If an API call to /series
is made without a start/end time prometheus sets minTime/maxTime as the value (https://github.com/prometheus/prometheus/blob/master/web/api/v1/api.go#L462). If I then turn around and use this time with the API client (to presumably send the same request) I get an error:
{
"status": "error",
"errorType": "execution",
"error": "bad_data: cannot parse \"292277025-08-17T23:12:54.999-08:00\" to a valid timestamp"
}
This is because the times used for minTime/maxTime can’t be formatted to an RFC3339Nano timestamp. So for my use case in promxy I get a call from grafana (in this example) with no start/end time – so prometheus sets the time to these min/max times but then I cannot pass that time down to another prometheus to fetch the data, as its not a valid timestamp.
Go playground showing the behavior: https://play.golang.org/p/hM_DWyv0JMK
So I’m not sure what we want to do exactly to handle this. Locally I have copied the min/max times into the client package and are optionally not setting the start/end times if they match. I’m not a major fan of the copy/paste of vars, but unfortunately those times are also private in the prometheus api/v1 package (https://github.com/prometheus/prometheus/blob/master/web/api/v1/api.go#L441). Alternatively we could make it so the client doesn’t set the start/end if start/end are time.IsZero()
which would allow the client/caller to pick when they don’t send the timestamp (instead of us transparently doing it in the client).
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 17 (17 by maintainers)
Commits related to this issue
- Only set start/end if the time is not Zero Fixes #614 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> — committed to jacksontj/client_golang by jacksontj 5 years ago
- Add tests to ensure we can marshal and unmarshal our min/max times Related to https://github.com/prometheus/client_golang/issues/614 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> — committed to jacksontj/prometheus by jacksontj 5 years ago
- Add tests to ensure we can marshal and unmarshal our min/max times Related to https://github.com/prometheus/client_golang/issues/614 Signed-off-by: Thomas Jackson <jacksontj.89@gmail.com> — committed to jacksontj/prometheus by jacksontj 5 years ago
- Change all time formatting to UTC and off of time.RFC3339Nano Prometheus has issues parsing RFC3339Nano timestamps if the year has more than 4 digits, in addition it is the second-pass parse attempt.... — committed to jacksontj/client_golang by jacksontj 5 years ago
- Change all time formatting to UTC and off of time.RFC3339Nano Prometheus has issues parsing RFC3339Nano timestamps if the year has more than 4 digits, in addition it is the second-pass parse attempt.... — committed to jacksontj/client_golang by jacksontj 5 years ago
- Add tests to ensure we can marshal and unmarshal our min/max times (#5734) * Add tests to ensure we can marshal and unmarshal our min/max times Related to https://github.com/prometheus/client_gola... — committed to prometheus/prometheus by jacksontj 5 years ago
- Change all time formatting to UTC and off of time.RFC3339Nano Prometheus has issues parsing RFC3339Nano timestamps if the year has more than 4 digits, in addition it is the second-pass parse attempt.... — committed to jacksontj/client_golang by jacksontj 5 years ago
- Change all time formatting to UTC and off of time.RFC3339Nano Prometheus has issues parsing RFC3339Nano timestamps if the year has more than 4 digits, in addition it is the second-pass parse attempt.... — committed to jacksontj/client_golang by jacksontj 5 years ago
The fix we put in on the server side is a hack really because of the shortcomings of stdlib time parse. The prom API already supports 2 timestamp formats, so it makes sense to change this client to the one that “just works”
In addition to the server fix, IMO we should do https://github.com/prometheus/client_golang/pull/617 as a (1) simplification and (2) more complete workaround on the client side.