influxdb: Timeouts after a few writes to InfluxDB

Hello,

I’m trying to write data ( a bunch of Windows perfmon counters) every 5 seconds to InfluxDB 0.9.4 with this Powershell code

$authheader = "Basic " + ([Convert]::ToBase64String([System.Text.encoding]::ASCII.GetBytes("InfluxAdmin:m3MOvzjdaidjaocrGGd2n")))
$uri = "http://$($InfluxDBServer):$($InfluxDBServerPort)/write?db=db_grafana01"
Invoke-RestMethod -Headers @{Authorization=$authheader} -Uri $uri -Method POST -Body $Metrics

This works fine when I do this manually one time. But when I make a loop and make it write these counters every 5 seconds, I get this error after the third time my PS script is trying to write data.

Error sending metrics to the Graphite Server. Please check your configuration file. 
PSMessageDetails      : 
Exception             : System.Net.WebException: The operation has timed out
                           at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
                           at System.Net.HttpWebRequest.GetRequestStream()
                           at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.SetRequestContent(WebRequest request, String content)
                           at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.FillRequestStream(WebRequest request)
                           at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
                           at System.Management.Automation.CommandProcessor.ProcessRecord()
TargetObject          : 
CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], WebException
FullyQualifiedErrorId : System.Net.WebException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at SendMetrics, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 143
                        at Send-BulkInfluxDBMetrics, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 229
                        at Start-StatsToInfluxDB, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 403
                        at <ScriptBlock>, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 424
                        at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

I tried augmenting the commit-timeout in influxdb.conf from 50ms to 500ms and later to 1000ms, but it still times out. Also tried to augment shard-writer-timeout from 5s to 10s and write-timeout from 5s to 10s, but this also didn’t help.

What else can I do to prevent me from getting timeout error? There is only one server sending data to InfluxDB and I’m only sending a few counters. The server is not under heavy load or anything.

This is an example of the data I’m sending each invoke-restmethod:

 hdd.e-drive.avg.diskreadqueuelength,host=server1,region=test value=0 
 cpu.usage,host=server1,region=test value=2.30835600969209 
 hdd.total.avg.diskwritequeuelength,host=server1,region=test value=0 
 nic.vmxnet3ethernetadapter.packetsreceivednon-unicast-sec,host=server1,region=test value=7.94371726813738 
 nic.vmxnet3ethernetadapter.packetssentunicast-sec,host=server1,region=test value=10.9226112436889 
 mem.pages-sec,host=server1,region=test value=0 
 mem.pagesinput-sec,host=server1,region=test value=0 
 hdd.c-drive.avg.diskreadqueuelength,host=server1,region=test value=0 
 nic.vmxnet3ethernetadapter.bytesreceived-sec,host=server1,region=test value=1040.626962126 
 hdd.total.avg.diskreadqueuelength,host=server1,region=test value=0 
 mem.availablembytes,host=server1,region=test value=1602 
 hdd.c-drive.avg.diskwritequeuelength,host=server1,region=test value=0 
 nic.vmxnet3ethernetadapter.packetssentnon-unicast-sec,host=server1,region=test value=0 
 nic.vmxnet3ethernetadapter.packetsreceivedunicast-sec,host=server1,region=test value=5.95778795110304 
 nic.vmxnet3ethernetadapter.bytessent-sec,host=server1,region=test value=9820.42047273484 
 system.processorqueuelength,host=server1,region=test value=0 
 hdd.e-drive.avg.diskwritequeuelength,host=server1,region=test value=0 
 system.threads,host=server1,region=test value=910 

Any help to get this working properly is welcome.

About this issue

  • Original URL
  • State: closed
  • Created 9 years ago
  • Comments: 15 (6 by maintainers)

Most upvoted comments

@willemdh - bit late for you probably but anyone else having this issue sounds similar to what i was having, firing off multiple calls in a single powershell session,

Check out the System.Net.ServicePointManager, has a default of 2 connections per endpoint, i pretty much do the following

$ServicePoint = [System.Net.ServicePointManager]::FindServicePoint(“http://influxendpoint/write…”) #you could do something like this or manually clear out as required. $ServicePoint.ConnectionLimit = 10 #start clean $ServicePoint.CloseConnectionGroup(“”)

Invoke Webrequests here…

#keep using as needed. $ServicePoint.CloseConnectionGroup(“”)