telegraf: [[input.kapacitor]] Invalid character 'x' in string escape code

E! [inputs.win_perf_counters] Error in plugin: error while getting value for counter \Processor(_Total)% Processor Time: A counter with a negative denominator value was detected.

Not sure on this one, could be a Telegraf bug or an issue with the perf counters. How often does it occur?

E! [outputs.influxdb] When writing to [http://…8086]: Post http://…:8086/write?db=SQL: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

This is a general timeout writing to the output, may not be something to worry about unless it happens frequently.

E! [inputs.win_perf_counters] Error in plugin: (pdhErr=258) The system cannot find message text for message number 0x%1 in the message file for %2.

I think this one is new to me, would be interesting if you could narrow down the reproduction steps for this.

E! [outputs.influxdb] When writing to [http://…:8086]: received error partial write: unable to parse 'sqlserver_requests

This one is a prime candidate for who is to blame for the corrupted data, but I know you said earlier that you disable the sqlserver plugin and still had the issue. There is some additional discussion aboutt this in #6976 and #7037.

I do recommend either disabling this query with exclude_query = [ 'SqlRequests'] or converting the statement_text tag to a field using the converter processor.

Here are some builds of #7696, based on 1.14.4. It includes utf-8 checking whenever metrics are created, if it finds an invalid byte sequence it will be replace with the replacement char and will log a message. There are still other places the corrupted data could be coming from, but this is the most likely place to catch it.

I’ll have to think about exactly how/if we can roll this out to an official version, since I know some people depend on being able to send binary data.

Can you start by running this on your gateway node and keeping an eye out log messages?

• telegraf-1.14.0~0d45f626-0.x86_64.rpm
• telegraf_1.14.0~0d45f626-0_amd64.deb
• telegraf-1.14.0~0d45f626_windows_amd64.zip

@danielnelson I have redirect pretty much all measurements to a new database (bucket) except the ones that come from sql plugin. Since, you said that people reported sqlserver input cause such bug. I’ll report back in next weeks. Hopefully, with no corrupt measurements in new bucket.

I ran jsonlint on the output to get a bit more info:

$ cat ~/kapa.txt | jsonlint -s -
<stdin>:5:78702: Error: JSON strings may not use the \x hex-escape
   |  At line 5, column 78702, offset 78854
   |  String started at line 5, column 78701, offset 78853
<stdin>:5:78706: Error: JSON strings may not use the \x hex-escape
   |  At line 5, column 78706, offset 78858
   |  String started at line 5, column 78701, offset 78853
<stdin>:5:78710: Error: JSON strings may not use the \x hex-escape
   |  At line 5, column 78710, offset 78862
   |  String started at line 5, column 78701, offset 78853
<stdin>:5:78714: Error: JSON strings may not use the \x hex-escape
   |  At line 5, column 78714, offset 78866
   |  String started at line 5, column 78701, offset 78853
<stdin>: has errors

Here is the section of the file containing the issue, formatted.

{
    "name": "ingress",
    "tags": {
        "cluster_id": "85ac55eb-351e-46c1-ade0-a784dd04625f",
        "database": "SERVERS",
        "host": "ALTAIR",
        "measurement": "\xb2\x8e\xee\x80disk",
        "retention_policy": "autogen",
        "server_id": "c64c5903-e50b-459c-a519-b30f9791d943",
        "task_master": "main"
    }
}

It appears that Kapacitor is not properly escaping non utf-8 characters in the JSON, and it might be that this value cannot be correctly escaped at all for JSON.

https://github.com/influxdata/kapacitor/blob/master/services/httpd/handler.go#L543-L554

I can move this issue over to Kapacitor, we won’t be able to fix it in Telegraf, but first let’s try to track down where is measurement is coming from. Does it appear in InfluxDB if you show measurements?

Can you attach the output from the link?:

curl http://localhost:9092/kapacitor/v1/debug/vars -o kapa.txt