nats-server: NATS Server corrupts the meta.inf file when it gets killed/closed/exited during startup stream loading time.

Defect

Make sure that these boxes are checked before submitting your issue – thank you!

• Included nats-server -DV output
• Included a [Minimal, Complete, and Verifiable example] (https://stackoverflow.com/help/mcve) Logs are included. nats-server.exe -DV [2684] 2023/05/26 01:08:32.952160 [INF] Starting nats-server [2684] 2023/05/26 01:08:32.953198 [INF]   Version:  2.9.16 [2684] 2023/05/26 01:08:32.953720 [INF]   Git:      [f84ca24] [2684] 2023/05/26 01:08:32.954231 [DBG]   Go build: go1.19.8 [2684] 2023/05/26 01:08:32.954257 [INF]   Name:     NAHUTXJA2HWFEV45HQ534LEZ54ZKESIX5NZ6VJKMYKY5XZ5H767XTS2G [2684] 2023/05/26 01:08:32.954257 [INF]   ID:       NAHUTXJA2HWFEV45HQ534LEZ54ZKESIX5NZ6VJKMYKY5XZ5H767XTS2G [2684] 2023/05/26 01:08:32.954257 [DBG] Created system account: “$SYS” [2684] 2023/05/26 01:08:32.955842 [INF] Listening for client connections on 0.0.0.0:4222 [2684] 2023/05/26 01:08:32.955842 [DBG] Get non local IPs for “0.0.0.0” [2684] 2023/05/26 01:08:32.960573 [DBG]   ip=1.7.19.7 [2684] 2023/05/26 01:08:32.962145 [INF] Server is ready [2684] 2023/05/26 01:08:32.963720 [DBG] maxprocs: Leaving GOMAXPROCS=4: CPU quota u

Versions of nats-server and affected client libraries used:

Version:  2.9.16

OS/Container environment:

Windows Server 2019 Version 1809 (OS Build 17763.4010)

Steps or code to reproduce the issue:

1. Start the Nats-server, Enable Jetstream, Create a stream, Add many messages (possibly of bug size) this is to increase the time when nats-server starts up as per step No.5

2. Now Stop the nats-server process.

3. Have a script to kill/stop the nats-server service upon demand.

4. Start the nats-server service. (having the logs enabled)

5. When the server starts to "Starting restore for stream ‘$G > streamname’ ", Kill the nats-server.exe.

6. Start the nats-server process newly again.

7. We see the error " Error unmarshalling stream metafile “C:\\DataStore\EventData\jetstream\$G\streams\STREAM_EI\meta.inf”: invalid character ‘h’ looking for beginning of value". This invalid character can change every time.

8. The stream could not be recovered at all.

9. This kill can also occur other times before the server is saying it is ready.

10. Also the same issue happens when System Time changes when server is doing step No. 5

Expected result:

When we start the service, the stream should be recoverable.

Actual result:

When we start the service and follow the steps, the stream meta file is updated / corrupted. leaving the stream un usable.

Actual Logs

LogToSahre.txt Some path and stream names are redacted, please do not mind those.