btrdb-server: BTrDB API service crash

This isn’t a fully baked issue yet since I’m having trouble reproducing the behavior, however, I thought I’d log it in case I can refine the information later.

I seem to have been able to take down our BTrDB API service while sending curl requests to our deployed copy of Mr Plotter. While I haven’t been able to completely crash it again I did notice the following.

Specifically, I was sending posts to the /csv endpoint while experimenting with different JSON values in the payload to see the effects on the data returned. I noticed that when sending a request for an alignment queryType of “windows” that I could get a non deterministic response from the DB. Sometimes I would get data, sometimes I would get an error message, and sometimes I’d get an error trying to connect.

Here’s the JSON payload with the auth token removed:

json={"StartTime":1494405229456,"EndTime":1494405345800,"UUIDS":["2e59878a-62f8-57ff-8834-16de04a45d62"],"Labels":["NavyYard/ pmu3003151/ L1MAG"],"QueryType":"windows","WindowText":"3","WindowUnit":"seconds","UnitofTime":"ms","PointWidth":44,"_token":""}

The primary thing that seemed to cause it to fail was changing the PointWidth value, but I’m not sure if I’m simply seeing patterns in randomness.

Here is the curl command I was sending (with auth token stripped again):

curl --request POST \
  --url https://viz.predictivegrid.com/csv \
  --header 'accept: text/csv' \
  --header 'accept-encoding: gzip, deflate, br' \
  --header 'accept-language: en-US,en;q=0.8' \
  --header 'cache-control: no-cache' \
  --header 'content-type: text/plain' \
  --header 'dnt: 1' \
  --header 'origin: https://viz.predictivegrid.com' \
  --header 'postman-token: 32b4c31a-fe7c-156c-2d40-cea40fa1b1e3' \
  --header 'referer: https://viz.predictivegrid.com/' \
  --header 'upgrade-insecure-requests: 1' \
  --header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' \
  --data 'json={"StartTime":1494405229456,"EndTime":1494405345800,"UUIDS":["2e59878a-62f8-57ff-8834-16de04a45d62"],"Labels":["NavyYard/ pmu3003151/ L1MAG"],"QueryType":"windows","WindowText":"3","WindowUnit":"seconds","UnitofTime":"ms","PointWidth":44,"_token":""}'

That either resulted in data being returned, or this error message:

Could not complete CSV query: rpc error: code = Internal desc = transport is closing

Or it flat out refused to connect.

As I said I haven’t been able to reproduce a full crash, but when I did all the API requests stopped responding (e.g. I noticed I could no longer query for streams) although Jerry (@pingthings) relayed that he didn’t see anything unusual in the logs.

I’ll update this if I figure out more.