influxdb: cli write: Error: Failed to write data: bufio.Scanner: token too long
I’m seeing the error:
Error: Failed to write data: bufio.Scanner: token too long.
when i try to write a csv data that is ~164MB to influxdb. Based on https://stackoverflow.com/questions/21124327/how-to-read-a-text-file-line-by-line-in-go-when-some-lines-are-long-enough-to-ca , it looks like we might need to evaluate if we should switch to bufio.Reader? https://github.com/influxdata/influxdb/blob/f09ee881fb33ca06474ca499deac0c5cd1bd9f91/write/batcher.go#L67
I can send the file offline.
➜ influxdb git:(master) ✗ influx query -c tools 'from(bucket: "apps") |> range(start: -1m)' -r | influx write -c default --format csv -b apps
Error: Failed to write data: bufio.Scanner: token too long.
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 16 (15 by maintainers)
Just a note - the solution to this must not be to allocate the user’s batch size as a single heap allocation - that’s a pathway to crashing OOM.
Looks like the actual bug is here where
nis assigned to, but the rest of the code is assuming thatnis the same aslen(p).I’d be inclined to avoid using
nand just uselen(p)when appropriate - it’s no more efficient to assign it to a local variable.