thanos: Query errors after vertical compaction (penalty): Error executing query: sum and count timestamps not aligned

Thanos, Prometheus and Golang version used:

thanos, version 0.22.0 (branch: non-git, revision: non-git) build user: root@b63a1c94421f build date: 20210727-14:31:15 go version: go1.16.2 platform: linux/amd64

Object Storage Provider:

S3 storage

What happened:

After enabling vertical compaction and it being finished we see errors for longer range queries in Thanos query:

Example queries can be as simple as sum(up) over 4w.

Error executing query: sum and count timestamps not aligned

What you expected to happen:

Queries to work as they did before vertical compaction.

How to reproduce it (as minimally and precisely as possible):

1. Enable vertical compaction and let it finish:

--deduplication.func=penalty --deduplication.replica-label=prometheus_replica

1. Query long time range with sum(up)

Full logs to relevant components:

Anything else we need to know:

The issue triggers reliably with downsampled data. We keep 4w of raw data and older data only as 5m and 1h downsampled and whenever a downsampled block gets queried above error gets returned.