go-carbon: [BUG] newly created metrics not fetchable even with cache-scan turned on

Describe the bug We have migrated our test environments from [grafana -> graphite-web] -> [ graphite-web -> go-carbon] to [grafana-> carbonapi] -> [ carbonserver -> go-carbon] . Existing and updated metrics are fetchable in a timely manner. However, newly created metrics are not visible until persisted to disk.

We have an internal tool that creates a new random metric once a minute and time show long it takes for the metric to become fetchable. The delay in our low-usage test environment is a few seconds, while the delay on our high-usage test environment is as long as 12 minutes.

Go-carbon Configuration:

# go-carbon's   /etc/carbon/carbon.conf
[common]
user = "_graphite"
metric-endpoint="local"
metric-interval = "1m0s"
max-cpu = 6

[whisper]
data-dir = "/graphite/whisper"
schemas-file = "/etc/carbon/storage-schemas.conf"
aggregation-file= "/etc/carbon/storage-aggregation.conf"
workers = 4
max-updates-per-second = 1000
max-creates-per-second = 1000
sparse-create = true
flock = true
enabled = true
hash-filenames = false

[cache]
max-size = 5000000
write-strategy = "max"

[tcp]
listen = ":2003"
enabled = true
buffer-size = 5000000

[logging]
logger = ""
file = "/var/log/carbon/carbon.log"
level = "info"
encoding = "mixed"
encoding-time = "iso8601"
encoding-duration = "seconds"

[carbonlink]
listen  = "127.0.0.1:7002"
enabled = true
read-timeout = "30s"

[receiver.http]
protocol = "http"
listen = ":2006"

[carbonserver]
# Please NOTE: carbonserver is not intended to fully replace graphite-web
# It acts as a "REMOTE_STORAGE" for graphite-web or carbonzipper/carbonapi
listen = ":8080"
enabled = true
buckets = 10
metrics-as-counters = false
read-timeout = "90s"
write-timeout = "90s"
query-cache-enabled = true
query-cache-size-mb = 1024
find-cache-enabled = true
trigram-index = false
scan-frequency = "5m0s"
cache-scan = true
max-globs = 100
fail-on-max-globs = false
graphite-web-10-strict-mode = false
internal-stats-dir = ""
# Calculate /render request time percentiles for the bucket, '95' means calculate 95th Percentile. To disable this feature, leave the list blank
stats-percentiles = [99, 98, 95, 75, 50]

#carbonapi configuration
listen: "0.0.0.0:8080"
cache:
  type: "mem"
  size_mb: 2000
  defaultTimeoutSec: 60
upstreams:
    graphite09compat: false
    buckets: 10
    keepAliveInterval: "30s"
    maxIdleConnsPerHost: 100
    timeouts:
        find: "10s"
        render: "30s"
        connect: "30s"
    backendsv2:
        backends:
          -
            groupName: "go_carbon"
            protocol: "carbonapi_v3_pb"
            lbMethod: "broadcast"
            maxTries: 3
            maxBatchSize: 0
            keepAliveInterval: "10s"
            maxIdleConnsPerHost: 1000
            timeouts:
                find: "10s"
                render: "30s"
                connect: "30s"
            servers:
                - "http://10.10.6.109:8080"
                - "http://10.10.6.197:8080"
                - "http://10.10.2.77:8080"
                - "http://10.10.2.24:8080"