influxdb: Timeouts after a few writes to InfluxDB
Hello,
I’m trying to write data ( a bunch of Windows perfmon counters) every 5 seconds to InfluxDB 0.9.4 with this Powershell code
$authheader = "Basic " + ([Convert]::ToBase64String([System.Text.encoding]::ASCII.GetBytes("InfluxAdmin:m3MOvzjdaidjaocrGGd2n")))
$uri = "http://$($InfluxDBServer):$($InfluxDBServerPort)/write?db=db_grafana01"
Invoke-RestMethod -Headers @{Authorization=$authheader} -Uri $uri -Method POST -Body $Metrics
This works fine when I do this manually one time. But when I make a loop and make it write these counters every 5 seconds, I get this error after the third time my PS script is trying to write data.
Error sending metrics to the Graphite Server. Please check your configuration file.
PSMessageDetails :
Exception : System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.SetRequestContent(WebRequest request, String content)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.FillRequestStream(WebRequest request)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()
TargetObject :
CategoryInfo : NotSpecified: (:) [Invoke-RestMethod], WebException
FullyQualifiedErrorId : System.Net.WebException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
ErrorDetails :
InvocationInfo : System.Management.Automation.InvocationInfo
ScriptStackTrace : at SendMetrics, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 143
at Send-BulkInfluxDBMetrics, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 229
at Start-StatsToInfluxDB, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 403
at <ScriptBlock>, C:\Nagios\NAF\NAF_Sources\Tools\Grafana\naf_send_metric_to_influxdb.ps1: line 424
at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}
I tried augmenting the commit-timeout in influxdb.conf from 50ms to 500ms and later to 1000ms, but it still times out. Also tried to augment shard-writer-timeout from 5s to 10s and write-timeout from 5s to 10s, but this also didn’t help.
What else can I do to prevent me from getting timeout error? There is only one server sending data to InfluxDB and I’m only sending a few counters. The server is not under heavy load or anything.
This is an example of the data I’m sending each invoke-restmethod:
hdd.e-drive.avg.diskreadqueuelength,host=server1,region=test value=0
cpu.usage,host=server1,region=test value=2.30835600969209
hdd.total.avg.diskwritequeuelength,host=server1,region=test value=0
nic.vmxnet3ethernetadapter.packetsreceivednon-unicast-sec,host=server1,region=test value=7.94371726813738
nic.vmxnet3ethernetadapter.packetssentunicast-sec,host=server1,region=test value=10.9226112436889
mem.pages-sec,host=server1,region=test value=0
mem.pagesinput-sec,host=server1,region=test value=0
hdd.c-drive.avg.diskreadqueuelength,host=server1,region=test value=0
nic.vmxnet3ethernetadapter.bytesreceived-sec,host=server1,region=test value=1040.626962126
hdd.total.avg.diskreadqueuelength,host=server1,region=test value=0
mem.availablembytes,host=server1,region=test value=1602
hdd.c-drive.avg.diskwritequeuelength,host=server1,region=test value=0
nic.vmxnet3ethernetadapter.packetssentnon-unicast-sec,host=server1,region=test value=0
nic.vmxnet3ethernetadapter.packetsreceivedunicast-sec,host=server1,region=test value=5.95778795110304
nic.vmxnet3ethernetadapter.bytessent-sec,host=server1,region=test value=9820.42047273484
system.processorqueuelength,host=server1,region=test value=0
hdd.e-drive.avg.diskwritequeuelength,host=server1,region=test value=0
system.threads,host=server1,region=test value=910
Any help to get this working properly is welcome.
About this issue
- Original URL
- State: closed
- Created 9 years ago
- Comments: 15 (6 by maintainers)
@willemdh - bit late for you probably but anyone else having this issue sounds similar to what i was having, firing off multiple calls in a single powershell session,
Check out the System.Net.ServicePointManager, has a default of 2 connections per endpoint, i pretty much do the following
$ServicePoint = [System.Net.ServicePointManager]::FindServicePoint(“http://influxendpoint/write…”) #you could do something like this or manually clear out as required. $ServicePoint.ConnectionLimit = 10 #start clean $ServicePoint.CloseConnectionGroup(“”)
Invoke Webrequests here…
#keep using as needed. $ServicePoint.CloseConnectionGroup(“”)