spring-boot: ECS structure logging is not compatible with all collectors as it does not use the nested format

It would be nice with an option to get ecs logging as nested json like

{ "ecs":  {"version": "8.4"}, "service": {"name": "myservice"}}

instead of what we get today

{ "ecs.version": "8.4", "service.name" : "myservice"}

The reason for asking is that collecting and processing of logs would be easier.

We are using Spring Boot 3.4.4.

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 5
  • Comments: 20 (11 by maintainers)

Commits related to this issue

Most upvoted comments

Hei @nosan and thanks again for your feedbak.

Interesting, last year, we faced mapping issues with OpenSearch when using the flat format and the nested mapping definition from ECS. It appears that this has been resolved, and it now behaves like Elasticsearch.

I have tested it again, and while OpenSearch accepts both flat and nested JSON with a nested mapping definition, the saved JSON document is not converted to nested format.

This is unfortunate, as you can have both flat and nested documents within the same index when the same type of logs comes from different sources. When operating solely within Elasticsearch or OpenSearch, it works because they have implemented an abstraction that facilitates the conversion between formats, allowing for searches across different formats.

However, this approach fails when using data from these indexes for further external processing in another system, as those documents can contain both formats. Here is a simple check I ran with jq to search a JSON file with 2 documents saved in the same opensearch index with the same mapping, one with flat format and the other with nested:

$ cat test3.json

{
    "docs":[
	{
	    "_index": "test_ecs",
	    "_id": "1",
	    "_version": 1,
	    "_score": null,
	    "_source": {
		"@timestamp": "2025-04-10T18:03:42.230354Z",
		"log.level": "ERROR",
		"process.pid": 30859,
		"process.thread.name": "main",
		"service.name": "gh-45063",
		"log.logger": "task.gh45063.Gh45063Application",
		"message": "MyError",
		"error.type": "java.lang.IllegalStateException",
		"error.message": "Error Test",
		"error.stack_trace": "java.lang.IllegalStateException: Error Test\n\tat task.gh45063.Gh45063Application.main(Gh45063Application.java:14)\n",
		"ecs.version": "8.11"
	    },
	    "fields": {
		"@timestamp": [
		    "2025-04-10T18:03:42.230Z"
		]
	    },
	    "sort": [
		1744308222230
	    ]
	},
	{
	    "_index": "test_ecs",
	    "_id": "2",
	    "_version": 1,
	    "_score": null,
	    "_source": {
		"@timestamp": "2025-04-10T18:04:14.257693Z",
		"log": {
		    "logger": "task.gh45063.Gh45063Application",
		    "level": "ERROR"
		},
		"process": {
		    "pid": 30877,
		    "thread": {
			"name": "main"
		    }
		},
		"service": {
		    "name": "gh-45063"
		},
		"message": "MyError",
		"error": {
		    "type": "java.lang.IllegalStateException",
		    "message": "Error Test",
		    "stack_trace": "java.lang.IllegalStateException: Error Test\n\tat task.gh45063.Gh45063Application.main(Gh45063Application.java:14)\n"
		},
		"ecs": {
		    "version": "8.11"
		}
	    },
	    "fields": {
		"@timestamp": [
		    "2025-04-10T18:04:14.257Z"
		]
	    },
	    "sort": [
		1744308254257
	    ]
	}
    ]
}

The two formats are not compatible; you must search using different methods because they have different structures.

$ cat test3.json |jq '.docs[]|select(._source.log.level == "ERROR") |._source.process.pid'
30877
$ cat test3.json |jq '.docs[]|select(._source."log.level" == "ERROR") |._source."process.pid"'
30859

As you can see, these are two incompatible representations of the same type of object, resulting in a lack of unified results. You have to search in different ways because they are different data structures.

We could continue discussing the pros and cons of using flat versus nested JSON formats when defining a JSON structure. Personally, I prefer the nested format because it eliminates the need to interpret the document’s structure and how to use it.

One thing is certain: one should never mix flat and nested formats in the same document, as it can lead to confusion. Although both formats conform to the JSON RFC 8259 standard (see: https://www.rfc-editor.org/rfc/rfc8259.txt), the entire issue surrounding flat versus nested formats contributes to a lack of standardization and various problems. This is primarily because interpreting a JSON dataset with attributes that contain dots (.) and nested format is arbitrary and depends on the developer’s implementation.

Although it may be an unspoken convention that dots (.) signify the flat version of a nested structure, it’s crucial to recognize that this is not a formal standard. The consequences of this ambiguity contribute to the challenges we are discussing.

In our case, processing of logs in flat format with the Fluent Bit agent and Logstash is problematic, as they deliver JSON in nested format. If the flat parts are not converted to nested format before saving, the resulting data structure will become messy, which will lead to issues depending on the software used for further data processing.

regards and good weekend 😃

Thank you, @nosan, for investigating this issue further.

“…In summary, both flat and nested formats are compatible …”, as a source, but only when using Elasticsearch as the storage infrastructure because elasticsearch transforms flat format to the standard nested format before saving the log. If you examine the JSON code of the document saved by Elasticsearch after sending the flat version, you’ll see it has been also converted into a nested json document to adhere to the standard.

UPDATE:


If elasticsearch works as opensearch this statement is not true: “If you examine the JSON code of the document saved by Elasticsearch after sending the flat version, you’ll see it has been also converted into a nested json document to adhere to the standard”

OpenSearch accepts both flat and nested JSON with a nested mapping definition now, but the saved flat JSON document is not converted to nested format.

Check https://github.com/spring-projects/spring-boot/issues/45063#issuecomment-2797072716 for full details.


The ECS standard is now utilized by many products following the merger of OpenTelemetry and Elastic standards to create a unified open standard. Refer to the following links for more information:

It would greatly enhance the Spring project if it could deliver logs in the ECS nested format. This would enable standardized log storage without the need to convert from flat to nested format before saving the log, when used with products other than Elasticsearch, such as Opensearch.

The conversion from flat to nested format to be able to save the standardized log, can be quite tedious and resource-intensive when processing millions of logs if you don’t use elasticsearch.

regards

Thanks for letting us know. I have totally overlooked the sentence from the docs when implementing this format:

The document structure should be nested JSON objects. If you use Beats or Logstash, the nesting of JSON objects is done for you automatically. If you’re ingesting to Elasticsearch using the API, your fields must be nested objects, not strings containing dots.

What might be a bug is that the The name ‘error’ has already been written error message is not written in ECS format. For us, I think that means it will not be imported into Opensearch, so we will be able to know what has gone wrong

The reason we don’t currently write the error message in ECS format is because we assuming that writing any other ECS message might fail in the same way. That’s why we currently throw an exception and let the logging system report it directly to the console. I’ve opened #45217 to see if we can log the message without the context data if an exception occurs.

I was not aware that Spring automatically added error.type so I think the fix for this is to remove error.type line above from our code.

I’ve opened #45218 to allow users to opt-out of our MDC logging and also relocate the JSON (for example to move it under the labels field as suggested in https://github.com/spring-projects/spring-boot/issues/45063#issuecomment-2809416948).

Unfortunately our existing logging.structured.json.rename.* property can’t be used with the ECS nested format.

@rafaelma I think we should consider this a bug, although it’s too risky to fix in 3.4. I’ve pushed something to main so that we can hopefully fix this for our 3.5.0-RC1 release. If you get a chance, could you please try the latest SNAPSHOT (once https://github.com/spring-projects/spring-boot/actions/runs/14457193995 has finished)

According to the ECS Reference:

The document structure should be nested JSON objects. If you use Beats or Logstash, the nesting of JSON objects is done for you automatically. If you’re ingesting to Elasticsearch using the API, your fields must be nested objects, not strings containing dots.

I also checked the ECS Logging Java repository elastic/ecs-logging-java and found they have a similar issues:

However, the ECS Java logging library continues to use dot notation and relies on this processor afterward. It seems they chose to keep dot notation because they can’t ensure that all fields are correctly nested especially considering fields provided through ILoggingEvent.getKeyValuePairs() or ILoggingEvent.getMDCPropertyMap(), since these may contain dots as well, and for example if you set a custom field using MDC like this MDC.put("geo.continent_name", "North America") then according to the ECS Reference , the ElasticCommonSchemaStructuredLogFormatter should ideally convert this into nested JSON, as shown below:

{
  "geo": {
    "continent_name": "North America"
  }
}

Personally, if Spring Boot chooses to support the nested JSON format, I don’t see any advantage in maintaining support for the current format, as only the nested JSON format fully complies with the ECS specification.

when used with products other than Elasticsearch, such as Opensearch.

I set up OpenSearch using Docker Compose and submitted several JSON documents to it. Both formats are supported by OpenSearch.

ScreenshotImage

I hope I haven’t overlooked anything, and I apologize once again for my earlier incorrect comment: https://github.com/spring-projects/spring-boot/issues/45063#issuecomment-2793987496

Hello, thanks for looking at this 😃

ECS stores data in a nested format, our example will be saved as { "ecs": {"version": "8.4"} }, and the mapping definition of the ECS standard needed by e.g. elasticsearch or opensearch defines also the nested format. It’s important to note that the translation from { "ecs.version": "8.4" } to { "ecs": {"version": "8.4"} } before saving the data according to the ECS standard is done by the Elastic Agent using the decode_json_fields processor in the elastic agent ref: https://www.elastic.co/guide/en/fleet/current/decode-json-fields.html.

For reference, you can check the mapping representation of the ECS standard for our example here: https://github.com/elastic/ecs/blob/main/generated/elasticsearch/composable/component/ecs.json The ECS documentation https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html refers to attributes like ecs.version, but this is merely a simplified representation of a JSON nested attribute in the documentation for presentation purposes.

If you use another agent than the elastic-agent (e.g.fluent-bit) to process logs and spring delivers them as e.g. { "ecs.version": "8.4" }, you have to transform this to the nested version { "ecs": {"version": "8.4"} } before you can save it according to the standard.

Spring doesn’t deliver the logs according to the ECS standard format and it will only work without problems if you collect these logs with the elastic-agent that will convert them to the right ECS format before sending them to storage.

regards, Rafael Martinez Guerrero

Thanks @nosan. Indeed @kajh we’re not going to offer a format that’s not backed by ECS