influxdb: Data spontaneously dropped from shard, retention policy possibly fails

Bug report

System info: influxdb:1.3.3, running in docker persisted via host mount

Steps to reproduce:

1. Write data to default database (starting using influx 0.10)
2. Upgrade database and continue to write
3. Change shard duration from 1 week to 3 months
4. Change retention policy to 3650d
5. Continue writing

Expected behavior:

Writes continue and data is persisted for 10 years in line with the retention policy (and the expiration date of the shards).

Actual behavior:

• 50% of the series from a shard (378 default autogen 2017-01-16T00:00:00Z 2017-04-10T00:00:00Z 2027-04-08T00:00:00Z) are no longer queryable (there is a 3 month hole in all the data)
• Writes continue to succeed (for the other 50% of time series)
• The shard itself was not empty (300MB vs. 550MB before)
• In the client code there were no “drop shard” commands used, delete from is used with specific time series (but never for exactly the shard period)

Additional info:

The relevant part of the output of show shard groups:

220 default      autogen          2016-12-19T00:00:00Z 2016-12-26T00:00:00Z 2026-12-24T00:00:00Z
197 default      autogen          2016-12-26T00:00:00Z 2017-01-02T00:00:00Z 2026-12-31T00:00:00Z
196 default      autogen          2017-01-02T00:00:00Z 2017-01-09T00:00:00Z 2027-01-07T00:00:00Z
195 default      autogen          2017-01-09T00:00:00Z 2017-01-16T00:00:00Z 2027-01-14T00:00:00Z
293 default      autogen          2017-01-16T00:00:00Z 2017-01-23T00:00:00Z 2027-01-21T00:00:00Z
378 default      autogen          2017-01-16T00:00:00Z 2017-04-10T00:00:00Z 2027-04-08T00:00:00Z
432 default      autogen          2017-04-10T00:00:00Z 2017-07-03T00:00:00Z 2027-07-01T00:00:00Z
433 default      autogen          2017-07-03T00:00:00Z 2017-09-25T00:00:00Z 2027-09-23T00:00:00Z
563 default      autogen          2017-09-25T00:00:00Z 2017-12-18T00:00:00Z 2027-12-16T00:00:00Z
564 default      autogen          2017-12-18T00:00:00Z 2018-03-12T00:00:00Z 2028-03-09T00:00:00Z

Output of show retention policies on "default":

name     duration   shardGroupDuration replicaN default
----     --------   ------------------ -------- -------
autogen  87600h0m0s 2016h0m0s          1        true