go-carbon: [Q] Context Canceled Errors on Large Queries to Carbonserver

I have a few large queries that are causing me issues. I originally had some larger queries with a lot of wildcards that would cause me errors like this:

"error": "find failed, can't expand globs"

I increased the max globs in the config and that seemed to fix the problem for most of the larger queries, but there’s still one left that was causing me issues, so I increased the max globs to an absurd value. This time I get a different error though:

ERROR [access] find failed "format": "carbonapi_v3_pb", "runtime_seconds": 4.00071045, "reason": "Internal error while processing request", "error": "could not expand globs - context canceled", "http_code": 500

INFO [carbonserver] slow_expand_globs

My question is what do I need to change to get this query to run. Skimming through the carbonserver code, it looks like this error isn’t necessarily related to the value of max globs, but is more like an issue with timeouts. On our python based graphite this query takes about 8 or 9 seconds, and the timeout is set much higher than that in our config.

Our go-carbon setup is running on a 16 core box with EBS volumes for storage, and currently hovering around 8 million metrics. Here is our go-carbon config:

[common]
user = "root"
graph-prefix = "carbon.agents.{host}"
metric-endpoint = "local"
metric-interval = "1m0s"
max-cpu = 14

[whisper]
data-dir = "/opt/graphite/storage/whisper"
schemas-file = "/opt/go-carbon/storage-schemas.conf"
aggregation-file = "/opt/go-carbon/storage-aggregation.conf"
workers = 6
max-updates-per-second = 0
max-creates-per-second = 0
hard-max-creates-per-second = false
sparse-create = false
flock = true
enabled = true
hash-filenames = true
compressed = true
remove-empty-file = false

[cache]
max-size = 0
write-strategy = "noop"

[udp]
listen = ":2003"
enabled = true
buffer-size = 0

[tcp]
listen = ":2003"
enabled = true
buffer-size = 0

[pickle]
listen = ":2004"
max-message-size = 67108864
enabled = true
buffer-size = 0

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
read-timeout = "30s"

[grpc]
listen = "127.0.0.1:7003"
enabled = true

[tags]
enabled = false

[carbonserver]
listen = ":8080"
enabled = true
buckets = 10
metrics-as-counters = false
read-timeout = "60s"
write-timeout = "60s"
query-cache-enabled = false
query-cache-size-mb = 2000
find-cache-enabled = false
trigram-index = true
scan-frequency = "2m30s"
trie-index = false
cache-scan = false
max-globs = 100000000
fail-on-max-globs = true
max-metrics-globbed  = 100000000
max-metrics-rendered = 1000000
graphite-web-10-strict-mode = true
stats-percentiles = [99, 98, 95, 75, 50]

[pprof]
listen = "localhost:7007"
enabled = false