VictoriaMetrics: Importing data from InfluxDB with vmctl causes unnecessary data duplication on series that add tags

Describe the bug

Tl;dr: adding a new tag to a time series in InfluxDB (thus creating a new series) causes vmctl to duplicate all subsequently added data when later importing into Victoriametrics

I have, in InfluxDB, a time series called rank that tracks the rank of users in the context of some game. Executing show series on <db> in influxdb shows me the following series:

rank,username=blah
rank,alias=blah,username=blah

We’ll call the first one series “A”, and the second series “B”.

At some point in the past, only series A existed as I didn’t need the alias tag. But, a need for it arose so I added it, creating series B.

This is two series - HOWEVER - at a certain point in time, which we’ll call the cutoff point - I stopped adding data to series A and cut over to adding data only to series B.

However, when importing with vmctl, the result was that series B data was copied into series A, as well as series B being present as well. Meaning, the data in series B was completely duplicated by vmctl.

To elaborate: in my original InfluxDB data, these two series do not overlap in terms of timespan, at all. Series A ends at an exact moment, and series B begins. But what vmctl has done, is copy all of series B into series A resulting in Victoriametric’s copy of series A spanning the length of A+B.

I suspect this happens because vmctl does not understand how InfluxDB works. InfluxDB behaves more like a SQL database where SELECTing FROM rank WHERE username=xyz is going to show you applicable data, as opposed to the series grouping Victoriametrics/Prometheus use.

Vmctl should notice that time series returned from InfluxDB may have different tags than the series vmctl asked for, and should know to omit this data during import, instead of erroneously duplicating it.

To Reproduce

  • Create 2 series as above in influxdb
  • Attempt to migrate the data to Victorametrics
  • See that data is duplicated

Version

vmctl version vmctl-20241002-113624-tags-v1.104.0-0-g0d4f4b8f7d

Logs

No response

Screenshots

No response

Used command-line flags

Vmctl import command:

docker run -it --rm victoriametrics/vmctl:v1.104.0 influx --influx-database mydatabase --influx-addr http://influx01:8086 --vm-addr http://vmetrics:8480 --verbose --vm-account-id 0 --vm-concurrency 8

Additional information

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 1
  • Comments: 15 (9 by maintainers)

Commits related to this issue

Most upvoted comments

Hi @jiekun, thanks again!

I gave the updated vmctl a try, and it worked for the test dataset! However, for the real data set, it errored out with 2024/10/25 19:31:58 influx error: request failed for "some_measurement"."some_field": response doesn't contain field "time".

I think that error is coming from here: https://github.com/VictoriaMetrics/VictoriaMetrics/blob/cd2222aa957274030770dac6a95ec5050c0a3751/app/vmctl/influx/influx.go#L225

Which looks to be something to do with chunking/batching? I guess influx is replying differently for larger responses?

Anyway, I put together a script to generate a larger test dataset. This creates 6 months worth of points with one point every 15 minutes, and a tag change in the middle: https://gist.github.com/dpedu/714ee98fbe054197a7cbe7f8db47990c. This uses the module influxdb==5.3.1. I could dump a csv instead if you would prefer.

Anyway, with that larger set I’m able to reproduce the error I hit on the original dataset:

# docker run -it --rm victoriametrics/vmctl:heads-fix-vmctl-influxdb-query-result-duplicate-0-g1cbb6e6a5-amd64 influx --influx-database testdb2 --influx-addr http://influx01:8086 --vm-addr http://vminsert:8480 --verbose --vm-account-id 0
InfluxDB import mode
2024/10/25 19:50:08 Exploring scheme for database "testdb2"
2024/10/25 19:50:08 fetching fields: command: "show field keys"; database: "testdb2"; retention: "autogen"
2024/10/25 19:50:08 found 1 fields
2024/10/25 19:50:08 fetching series: command: "show series"; database: "testdb2"; retention: "autogen"
2024/10/25 19:50:08 found 2 series
Found 2 timeseries to import. Continue? [Y/n] Y
VM worker 0:↖ ? p/s
VM worker 1:↖ ? p/s
Processing series: 0 / 2 [▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒] 0.00%
2024/10/25 19:50:09 influx error: request failed for "testdupe2"."value": response doesn't contain field "time"

I appreciate you looking into this!

I don’t want to share my actual data but I can put together some test data that demonstrates the issue.

This is a csv dump from influx:

name,time,foo,qux,value
testdupe,1729792944190676547,bar,,0
testdupe,1729792974395474289,bar,,1
testdupe,1729793004516631734,bar,baz,2
testdupe,1729793034637707302,bar,baz,3

From this query:

> SELECT * FROM testdupe;
name: testdupe
time                foo qux value
----                --- --- -----
1729792944190676547 bar     0
1729792974395474289 bar     1
1729793004516631734 bar baz 2
1729793034637707302 bar baz 3

Which looks like this when queried from influx and when queried from VictoriaMetrics after being copied over with vmctl:

image

The VictoriaMetrics chart is duplicated with one of the series hidden client-side via Grafana’s UI so that the overlapping of the two series can be seen.

VM query: testdupe_value{db="testdb"} Influx query: SELECT "value" FROM "testdupe" WHERE $timeFilter GROUP BY "foo"::tag

https://github.com/VictoriaMetrics/VictoriaMetrics/pull/7330 has been merged and will be available in the next release.

@jiekun no worries at all, I really appreciate your efforts here.

I agree, 4 series sounds right - double from before because VictoriaMetrics interprets each measurement as its own series.

Anyhow - I tested the new image with my original dataset once more and everything looks great! Thank you so much for your efforts here.

Hi @jiekun, looks good with the sample dataset! However, I did find one more edge case that seems new: I tried it against my original dataset too and it ended up importing no data at all.

I narrowed it down to this: it doesn’t seem to like measurements with multiple fields.

The previous test data looked like this:

> select * from testdupe2 limit 2;
name: testdupe2
time                foo qux value
----                --- --- -----
1698619629427265000 bar     0    
1698620529427265000 bar     1    

But if I add a second “column”, so it looks like this:

> select * from testdupe3 limit 2;
name: testdupe3
time                foo qux value1 value2
----                --- --- ------ ------
1698619629427265000 bar     0      0
1698620529427265000 bar     1      1

Then, vmctl imports nothing, whereas multiple fields did work previously:

2024/10/28 22:44:53 Import finished!
2024/10/28 22:44:53 VictoriaMetrics importer stats:
  idle duration: 0s;
  time spent while importing: 869.562283ms;
  total samples: 0;
  samples/s: 0.00;
  total bytes: 0 B;

I made a slight change to the sample data script: on line 28, it just repeats the value twice now, which is enough to reproduce this: https://gist.github.com/dpedu/fbb3efe1ddb571e85a584fcd5a3d4441

Thanks again for looking!