spring-boot: ECS structure logging is not compatible with all collectors as it does not use the nested format
It would be nice with an option to get ecs logging as nested json like
{ "ecs": {"version": "8.4"}, "service": {"name": "myservice"}}
instead of what we get today
{ "ecs.version": "8.4", "service.name" : "myservice"}
The reason for asking is that collecting and processing of logs would be easier.
We are using Spring Boot 3.4.4.
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Reactions: 5
- Comments: 20 (11 by maintainers)
Commits related to this issue
- Fix "Used nested format for ECS structure logging" test on Windows See gh-45063 — committed to spring-projects/spring-boot by philwebb 2 years ago
- Used nested format for ECS structure logging Update `ElasticCommonSchemaStructuredLogFormatter` implementations so that nested JSON is used for entries that previous has a '.' in the name. This forma... — committed to deepharness/spring-boot by philwebb 2 years ago
Hei @nosan and thanks again for your feedbak.
Interesting, last year, we faced mapping issues with OpenSearch when using the flat format and the nested mapping definition from ECS. It appears that this has been resolved, and it now behaves like Elasticsearch.
I have tested it again, and while OpenSearch accepts both flat and nested JSON with a nested mapping definition, the saved JSON document is not converted to nested format.
This is unfortunate, as you can have both flat and nested documents within the same index when the same type of logs comes from different sources. When operating solely within Elasticsearch or OpenSearch, it works because they have implemented an abstraction that facilitates the conversion between formats, allowing for searches across different formats.
However, this approach fails when using data from these indexes for further external processing in another system, as those documents can contain both formats. Here is a simple check I ran with
jqto search a JSON file with 2 documents saved in the same opensearch index with the same mapping, one with flat format and the other with nested:The two formats are not compatible; you must search using different methods because they have different structures.
As you can see, these are two incompatible representations of the same type of object, resulting in a lack of unified results. You have to search in different ways because they are different data structures.
We could continue discussing the pros and cons of using flat versus nested JSON formats when defining a JSON structure. Personally, I prefer the nested format because it eliminates the need to interpret the document’s structure and how to use it.
One thing is certain: one should never mix flat and nested formats in the same document, as it can lead to confusion. Although both formats conform to the JSON RFC 8259 standard (see: https://www.rfc-editor.org/rfc/rfc8259.txt), the entire issue surrounding flat versus nested formats contributes to a lack of standardization and various problems. This is primarily because interpreting a JSON dataset with attributes that contain dots (.) and nested format is arbitrary and depends on the developer’s implementation.
Although it may be an unspoken convention that dots (.) signify the flat version of a nested structure, it’s crucial to recognize that this is not a formal standard. The consequences of this ambiguity contribute to the challenges we are discussing.
In our case, processing of logs in flat format with the Fluent Bit agent and Logstash is problematic, as they deliver JSON in nested format. If the flat parts are not converted to nested format before saving, the resulting data structure will become messy, which will lead to issues depending on the software used for further data processing.
regards and good weekend 😃
Thank you, @nosan, for investigating this issue further.
“…In summary, both flat and nested formats are compatible …”, as a source, but only when using Elasticsearch as the storage infrastructure because elasticsearch transforms flat format to the standard nested format before saving the log.
If you examine the JSON code of the document saved by Elasticsearch after sending the flat version, you’ll see it has been also converted into a nested json document to adhere to the standard.UPDATE:
If elasticsearch works as opensearch this statement is not true: “If you examine the JSON code of the document saved by Elasticsearch after sending the flat version, you’ll see it has been also converted into a nested json document to adhere to the standard”
OpenSearch accepts both flat and nested JSON with a nested mapping definition now, but the saved flat JSON document is not converted to nested format.
Check https://github.com/spring-projects/spring-boot/issues/45063#issuecomment-2797072716 for full details.
The ECS standard is now utilized by many products following the merger of OpenTelemetry and Elastic standards to create a unified open standard. Refer to the following links for more information:
It would greatly enhance the Spring project if it could deliver logs in the ECS nested format. This would enable standardized log storage without the need to convert from flat to nested format before saving the log, when used with products other than Elasticsearch, such as Opensearch.
The conversion from flat to nested format to be able to save the standardized log, can be quite tedious and resource-intensive when processing millions of logs if you don’t use elasticsearch.
regards
Thanks for letting us know. I have totally overlooked the sentence from the docs when implementing this format:
The reason we don’t currently write the error message in ECS format is because we assuming that writing any other ECS message might fail in the same way. That’s why we currently throw an exception and let the logging system report it directly to the console. I’ve opened #45217 to see if we can log the message without the context data if an exception occurs.
I’ve opened #45218 to allow users to opt-out of our MDC logging and also relocate the JSON (for example to move it under the labels field as suggested in https://github.com/spring-projects/spring-boot/issues/45063#issuecomment-2809416948).
Unfortunately our existing
logging.structured.json.rename.*property can’t be used with the ECS nested format.@rafaelma I think we should consider this a bug, although it’s too risky to fix in 3.4. I’ve pushed something to
mainso that we can hopefully fix this for our 3.5.0-RC1 release. If you get a chance, could you please try the latest SNAPSHOT (once https://github.com/spring-projects/spring-boot/actions/runs/14457193995 has finished)According to the ECS Reference:
I also checked the ECS Logging Java repository elastic/ecs-logging-java and found they have a similar issues:
However, the ECS Java logging library continues to use dot notation and relies on this processor afterward. It seems they chose to keep dot notation because they can’t ensure that all fields are correctly nested especially considering fields provided through
ILoggingEvent.getKeyValuePairs()orILoggingEvent.getMDCPropertyMap(), since these may contain dots as well, and for example if you set a custom field usingMDClike thisMDC.put("geo.continent_name", "North America")then according to the ECS Reference , theElasticCommonSchemaStructuredLogFormattershould ideally convert this into nested JSON, as shown below:Personally, if Spring Boot chooses to support the nested JSON format, I don’t see any advantage in maintaining support for the current format, as only the nested JSON format fully complies with the ECS specification.
I set up OpenSearch using Docker Compose and submitted several JSON documents to it. Both formats are supported by OpenSearch.
Screenshot
I hope I haven’t overlooked anything, and I apologize once again for my earlier incorrect comment: https://github.com/spring-projects/spring-boot/issues/45063#issuecomment-2793987496
Hello, thanks for looking at this 😃
ECS stores data in a nested format, our example will be saved as
{ "ecs": {"version": "8.4"} }, and the mapping definition of the ECS standard needed by e.g. elasticsearch or opensearch defines also the nested format. It’s important to note that the translation from{ "ecs.version": "8.4" }to{ "ecs": {"version": "8.4"} }before saving the data according to the ECS standard is done by the Elastic Agent using the decode_json_fields processor in the elastic agent ref: https://www.elastic.co/guide/en/fleet/current/decode-json-fields.html.For reference, you can check the mapping representation of the ECS standard for our example here: https://github.com/elastic/ecs/blob/main/generated/elasticsearch/composable/component/ecs.json The ECS documentation https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html refers to attributes like
ecs.version, but this is merely a simplified representation of a JSON nested attribute in the documentation for presentation purposes.If you use another agent than the elastic-agent (e.g.fluent-bit) to process logs and spring delivers them as e.g.
{ "ecs.version": "8.4" }, you have to transform this to the nested version{ "ecs": {"version": "8.4"} }before you can save it according to the standard.Spring doesn’t deliver the logs according to the ECS standard format and it will only work without problems if you collect these logs with the elastic-agent that will convert them to the right ECS format before sending them to storage.
regards, Rafael Martinez Guerrero
Thanks @nosan. Indeed @kajh we’re not going to offer a format that’s not backed by ECS