influxdb: Allow manual triggering of CQs
Feature Request
Up until 1.1, InfluxDB had an HTTP endpoint that allowed triggering CQs manually. This endpoint was added during development of the CQ service, for debugging purposes and early troubleshooting. It wasn’t intended to be part of the API and was eventually removed. However, manually triggering CQs is useful in some cases (e.g., running CQs on recently imported historic data) and we should add this feature in a way that’s consistent with other query operations, as part of the query language.
Proposal:
Add an EXECUTE CONTINUOUS QUERY statement to InfluxQL.
Syntax:
execute_continuous_query_stmt = "EXECUTE CONTINUOUS QUERY" query_name on_clause
[ where_clause ] .
With the semantic restriction that the where_clause may only contain a time range. Expressions filtering on tags, fields, etc. will return an error.
Examples:
Trigger a continuous query to run for the current time:
EXECUTE CONTINUOUS QUERY mycq ON mydb
Trigger a continuous query to run for the past 60 days:
EXECUTE CONTINUOUS QUERY mycq ON mydb WHERE time > now() - 60d
Trigger a continuous query to run for a 30 day block in the past:
EXECUTE CONTINUOUS QUERY mycq ON mydb WHERE time > now() - 60d and time < now - 30d
About this issue
- Original URL
- State: closed
- Created 7 years ago
- Reactions: 7
- Comments: 18 (7 by maintainers)
@dugwood, EVERY and FOR don’t do it for us because of the unpredictable delay between the measurement being recorded by a device and being uploaded to the database. As I noted in my earlier comments, the delay could be anything between a few minutes (typical) to several hours (not uncommon) and several weeks (not so common. but entirely possible). To write a CQ with EVERY and FOR to cover that time range is simply not reasonable. Hence my suggestion that the CQ be redefined as a mechanism to invoke a previously defined stored SELECT INTO query at some regular interval (EVERY and FOR), and that same stored query can be invoked by the application when new data is uploaded.
(Note that in our application we are not using Telegraf or any of the other Tick stack packages. We have our own HTTP API for receiving batches of measurement data which is then written to the InfluxDB database.)
This would be very useful to us also. Our data collection system, by design, does not guarantee to upload data in real time, so the CQ facility in its current form will miss some measurements. (Most are uploaded within a few minutes of being collected but it’s not unusual for there to be a delay in excess of an hour or more, possibly even a couple of weeks in extreme cases. The lateness of the data does not reduce its usefulness for analysis.)
The current workaround is to run a SELECT INTO query to handle historical data, but this can quickly become an application maintenance nightmare: the same SELECT INTO has to be written for the CQ and for any application logic or ad-hoc scripts that load measurements into InfluxDB.
Incidentally, I would prefer that the
EXECUTE CONTINUOUS QUERYfeature would allow selection on tags as well as by time range. (We can manage if it does not, but it’s going to be rather inefficient without tag selection.)