opensim-core: Locale settings are inconsistent between `TimeSeriesTable`/`Storage` and applications

I am on a fresh Ubuntu 22.04 install and installed the latest OpenSim (main branch build).

When trying to load this test data test.mot through the OpenSim GUI onto a model:

inDegrees=yes
name=ik_example_imu_data_orientations
DataType=double
version=3
OpenSimVersion=4.5.1-2024-09-27-cbf9c74c0
endheader
time	pelvis_tilt
3	0.0
3.005	90.0

… I am receiving this error:

[warning] Storage: FileAdpater failed to read data file.
Timestamp at row 0 with value 3,000000 is greater-than/equal to timestamp at row 1 with value 3,000000
	Thrown at TimeSeriesTable.h:533 in validateRow().

It seems to only accept the data if the time starts from 0. Also this does not work for me either:

...
time pelvis_tilt
0	0.0
0.1	90.0

Only if the timestamps are full seconds it seems to pass. Everything worked on my old system with OpenSim 4.5 install (code below should be identical to 4.5.1). Not sure how I can even debug this as I get the same results using the official examples from the OpenSim Modules --> Code --> Python --> OpenSenseExample

Whats the problem here?

Regarding code:

https://github.com/opensim-org/opensim-core/blob/c9336a887f6a1649b2cabbe3ae92d62d696e2955/OpenSim/Common/Storage.cpp#L205-L230

https://github.com/opensim-org/opensim-core/blob/cbf9c74c065194dd4ac2d350bbd580a09a575848/OpenSim/Common/TimeSeriesTable.h#L518-L537

About this issue

  • Original URL
  • State: open
  • Created 2 years ago
  • Comments: 19 (11 by maintainers)

Commits related to this issue

Most upvoted comments

@nickbianco @ksomml I have created a minimum reproduction of the issue here. The example sets the locale locally in the code (to Finnish but any locale which uses commas for numerics would work) so you do not have to modify your system locale at all to replicate the issue.

Now that I can reliably reproduce the issue I’m going to starting working on the fixes described above.

iirc, I wrote the comment because we had a problem in prod where dutch students in TU Delft were ending up with incompatible data files when they were shared with the english speakers. I probably only tested it in a very specific situation, and it was probably borked in that specific way. It wouldn’t surprise me if the same bug also happens for reading via different APIs. It stems from an inconsistent use of raw file writers vs. formatted file writers (e.g. FILE write() vs. std::ostream).

It seems like actually fixing the issue would first involve removing usages of FILE* (this is in a c++ class so it does not make sense to use C file IO) and subsequent usages of IO::OpenFile. It is mainly only used here and here. It seems like both of those can be replaced with IO::OpenInputFile and IO::OpenOutputFile. Further usages of FILE * , for example here would need to be written to use std::fstream.

Once that is complete and there is only one place where the file IO happens then the stream can be imbued with the correct locale in the OpenInputFile and OpenOutputFile methods and things would work as expected.

iirc, I wrote the comment because we had a problem in prod where dutch students in TU Delft were ending up with incompatible data files when they were shared with the english speakers. I probably only tested it in a very specific situation, and it was probably borked in that specific way. It wouldn’t surprise me if the same bug also happens for reading via different APIs. It stems from an inconsistent use of raw file writers vs. formatted file writers (e.g. FILE write() vs. std::ostream).

@alexbeattie42 , I’m actually feeling quite seen ^_^ - I think you’re probably one of the first people to read OSC’s source 😄

It looks like @adamkewley has used this solution in OpenSimCreator if you also want to check that for reference.

<del>Alright, i’ll try it out the fix in IO.cpp first and check if these changes work.</del> If so I will open a new PR for a clean commit history. Maybe that way some maintainers will have a deeper look into this as a change in IO.cpp could affect many things.

/edit: Actually I will try it out in Storage.cpp due to the reason mentioned.

I agree that in research the commas are not used for numerics generally but the system locale is important for things like date and calendar display settings within the system and on the desktop.

I think I’ve figured out where the problem could be solved and found this article and this post very helpful.

Here is where the file stream is opened and on line 281 it is opened. So I believe you could do something like this to change the locale to en_US for that specific ifstream without impacting the system configuration.