VictoriaMetrics: Importing data from InfluxDB with vmctl causes unnecessary data duplication on series that add tags
Describe the bug
Tl;dr: adding a new tag to a time series in InfluxDB (thus creating a new series) causes vmctl to duplicate all subsequently added data when later importing into Victoriametrics
I have, in InfluxDB, a time series called rank that tracks the rank of users in the context of some game. Executing show series on <db> in influxdb shows me the following series:
rank,username=blah
rank,alias=blah,username=blah
We’ll call the first one series “A”, and the second series “B”.
At some point in the past, only series A existed as I didn’t need the alias tag. But, a need for it arose so I added it, creating series B.
This is two series - HOWEVER - at a certain point in time, which we’ll call the cutoff point - I stopped adding data to series A and cut over to adding data only to series B.
However, when importing with vmctl, the result was that series B data was copied into series A, as well as series B being present as well. Meaning, the data in series B was completely duplicated by vmctl.
To elaborate: in my original InfluxDB data, these two series do not overlap in terms of timespan, at all. Series A ends at an exact moment, and series B begins. But what vmctl has done, is copy all of series B into series A resulting in Victoriametric’s copy of series A spanning the length of A+B.
I suspect this happens because vmctl does not understand how InfluxDB works. InfluxDB behaves more like a SQL database where SELECTing FROM rank WHERE username=xyz is going to show you applicable data, as opposed to the series grouping Victoriametrics/Prometheus use.
Vmctl should notice that time series returned from InfluxDB may have different tags than the series vmctl asked for, and should know to omit this data during import, instead of erroneously duplicating it.
To Reproduce
- Create 2 series as above in influxdb
- Attempt to migrate the data to Victorametrics
- See that data is duplicated
Version
vmctl version vmctl-20241002-113624-tags-v1.104.0-0-g0d4f4b8f7d
Logs
No response
Screenshots
No response
Used command-line flags
Vmctl import command:
docker run -it --rm victoriametrics/vmctl:v1.104.0 influx --influx-database mydatabase --influx-addr http://influx01:8086 --vm-addr http://vmetrics:8480 --verbose --vm-account-id 0 --vm-concurrency 8
Additional information
No response
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 1
- Comments: 15 (9 by maintainers)
Commits related to this issue
- vmctl: fixed import duplicate data when query result contains multiple series (#7330) ### Describe Your Changes Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301 When querying with... — committed to VictoriaMetrics/VictoriaMetrics by jiekun 2 years ago
- vmctl: fixed import duplicate data when query result contains multiple series (#7330) ### Describe Your Changes Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301 When querying ... — committed to VictoriaMetrics/VictoriaMetrics by jiekun 2 years ago
- vmctl: fixed import duplicate data when query result contains multiple series (#7330) ### Describe Your Changes Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301 When querying ... — committed to truepele/VictoriaMetrics by jiekun 2 years ago
- vmctl: fixed import duplicate data when query result contains multiple series (#7330) ### Describe Your Changes Fix https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7301 When querying ... — committed to truepele/VictoriaMetrics by jiekun 2 years ago
Hi @jiekun, thanks again!
I gave the updated vmctl a try, and it worked for the test dataset! However, for the real data set, it errored out with
2024/10/25 19:31:58 influx error: request failed for "some_measurement"."some_field": response doesn't contain field "time".I think that error is coming from here: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cd2222aa957274030770dac6a95ec5050c0a3751/app/vmctl/influx/influx.go#L225
Which looks to be something to do with chunking/batching? I guess influx is replying differently for larger responses?
Anyway, I put together a script to generate a larger test dataset. This creates 6 months worth of points with one point every 15 minutes, and a tag change in the middle: https://gist.github.com/dpedu/714ee98fbe054197a7cbe7f8db47990c. This uses the module
influxdb==5.3.1. I could dump a csv instead if you would prefer.Anyway, with that larger set I’m able to reproduce the error I hit on the original dataset:
I appreciate you looking into this!
I don’t want to share my actual data but I can put together some test data that demonstrates the issue.
This is a csv dump from influx:
From this query:
Which looks like this when queried from influx and when queried from VictoriaMetrics after being copied over with vmctl:
The VictoriaMetrics chart is duplicated with one of the series hidden client-side via Grafana’s UI so that the overlapping of the two series can be seen.
VM query:
testdupe_value{db="testdb"}Influx query:SELECT "value" FROM "testdupe" WHERE $timeFilter GROUP BY "foo"::taghttps://github.com/VictoriaMetrics/VictoriaMetrics/pull/7330 has been merged and will be available in the next release.
@jiekun no worries at all, I really appreciate your efforts here.
I agree, 4 series sounds right - double from before because VictoriaMetrics interprets each measurement as its own series.
Anyhow - I tested the new image with my original dataset once more and everything looks great! Thank you so much for your efforts here.
Hi @jiekun, looks good with the sample dataset! However, I did find one more edge case that seems new: I tried it against my original dataset too and it ended up importing no data at all.
I narrowed it down to this: it doesn’t seem to like measurements with multiple fields.
The previous test data looked like this:
But if I add a second “column”, so it looks like this:
Then, vmctl imports nothing, whereas multiple fields did work previously:
I made a slight change to the sample data script: on line 28, it just repeats the value twice now, which is enough to reproduce this: https://gist.github.com/dpedu/fbb3efe1ddb571e85a584fcd5a3d4441
Thanks again for looking!