vector: Strategy device_and_inode does not work properly

Hi

vector version 0.8.2

I have few logs files, for example:

$ ls
logfile_json.log
logfile_json.1
logfile_json.2
logfile_json.3

When logfile_json.log reaching 100MB it will be renamed to logfile_json.1 as well as other log files to next number. Also, my vector config:

    [sources.in_logs]
        type = "file"
        include = ["/var/log/myapp/logfile_json*"]
        exclude = ["*gz"]
        glob_minimum_cooldown = 1000
        ignore_older = 86400
        max_read_bytes = 4096

      [sources.in_logs.fingerprinting]
          strategy = "device_and_inode"

When vector starts - it detects all files and starts to read each of them and saves offsets properly. I checked inode number after renaming - new file gets new inode, renamed file saves his inode. Also, after renaming files vector detects changes and logs new_path and old_path properly.

However, vector does not read logs from those files, that were detected by device_and_inode strategy. debug log level did not clarify situation, because renaming detection shows even in info log level

After changing strategy to checksum it works well.

Ask me to provide something else, if you will need more information.

About this issue

  • Original URL
  • State: closed
  • Created 4 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

We’ve been busy adding first-class integration with k8s. There’s a new kubernetes_logs source, and we made sure to address the issue you’ve encountered here in that source. More specifically, we’re using a new implementation of the fingerprinter that checksums just the first line. We’re going to ship this in the next release (0.11).

@Hoverbear it’s linux - timberio/vector:0.8.2-debian