zipkin: Determine how to address the next Elasticsearch index pattern break
Many of you will recall that we had to redo everything last change to the index constraints: dropping of multi-type.
One thing we decided to do to accommodate this was to use a delimited as clearly we need to have separate indexing and retention for spans vs dependency links.
Unfortunately, we seem bit again… Looks like Elasticsearch will also drop support for the colon pattern soon.
[2018-10-24T11:32:00,033][WARN ][o.e.d.c.m.MetaDataCreateIndexService] index or alias name [zipkin:span-2018-10-24] containing ':' is deprecated and will not be supported in Elasticsearch 7.0+
First, we should strongly push back on Elasticsearch as this causes another big round of work for us amplified by the ecosystem. It isn’t quite cool to have to keep hitting mines like this. The knock on effects include double-reads, data migrations, all sorts of things, it isn’t fun especially for a volunteer group.
Anyway, assuming pushback fails we need to address this thing:
- figure out which character is ok now.
- make conditional code (again) to address this not only here, but also in the dependency linker which is completely different codebase
- communicate to custom ingesters about this and why and what to do about it.
About this issue
- Original URL
- State: closed
- Created 6 years ago
- Comments: 23 (21 by maintainers)
Commits related to this issue
- WIP: attempts support of Elasticsearch 7.x There are multiple issues to resolve, not just the colon banning. See https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html... — committed to openzipkin/zipkin by deleted user 5 years ago
- WIP: attempts support of Elasticsearch 7.x There are multiple issues to resolve, not just the colon banning. See https://www.elastic.co/guide/en/elasticsearch/reference/7.x/breaking-changes-7.0.html... — committed to openzipkin/zipkin by deleted user 5 years ago
- Supports Elasticsearch 7.x (#2398) Fixes #2219 — committed to abesto/zipkin by adriancole 5 years ago
I have a work testing now, which will use single-character wildcard on search
_and explicit characters on insert. So, for <7 index-type delimiter:and >=7-. It occurred to me that since we use the hyphen delimiter elsewhere, if this becomes banned by ES in the future it will break everything else. Instead of waiting for a special character to be cleared and not working with ES7 at all, better to assume they won’t break hyphen until that’s not the case.