opensim-core: Locale settings are inconsistent between `TimeSeriesTable`/`Storage` and applications
I am on a fresh Ubuntu 22.04 install and installed the latest OpenSim (main branch build).
When trying to load this test data test.mot through the OpenSim GUI onto a model:
inDegrees=yes
name=ik_example_imu_data_orientations
DataType=double
version=3
OpenSimVersion=4.5.1-2024-09-27-cbf9c74c0
endheader
time pelvis_tilt
3 0.0
3.005 90.0
… I am receiving this error:
[warning] Storage: FileAdpater failed to read data file.
Timestamp at row 0 with value 3,000000 is greater-than/equal to timestamp at row 1 with value 3,000000
Thrown at TimeSeriesTable.h:533 in validateRow().
It seems to only accept the data if the time starts from 0.
Also this does not work for me either:
...
time pelvis_tilt
0 0.0
0.1 90.0
Only if the timestamps are full seconds it seems to pass. Everything worked on my old system with OpenSim 4.5 install (code below should be identical to 4.5.1). Not sure how I can even debug this as I get the same results using the official examples from the OpenSim Modules --> Code --> Python --> OpenSenseExample
Whats the problem here?
Regarding code:
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 19 (11 by maintainers)
Commits related to this issue
- Add locale requirements Fixes issue https://github.com/opensim-org/opensim-core/issues/3924 — committed to ksomml/opensim-core by ksomml 2 years ago
- Add locale requirements — committed to opensim-org/opensim-gui by ksomml 2 years ago
@nickbianco @ksomml I have created a minimum reproduction of the issue here. The example sets the locale locally in the code (to Finnish but any locale which uses commas for numerics would work) so you do not have to modify your system locale at all to replicate the issue.
Now that I can reliably reproduce the issue I’m going to starting working on the fixes described above.
It seems like actually fixing the issue would first involve removing usages of
FILE*(this is in a c++ class so it does not make sense to use C file IO) and subsequent usages ofIO::OpenFile. It is mainly only used here and here. It seems like both of those can be replaced withIO::OpenInputFileandIO::OpenOutputFile. Further usages ofFILE *, for example here would need to be written to usestd::fstream.Once that is complete and there is only one place where the file IO happens then the stream can be imbued with the correct locale in the
OpenInputFileandOpenOutputFilemethods and things would work as expected.iirc, I wrote the comment because we had a problem in prod where dutch students in TU Delft were ending up with incompatible data files when they were shared with the english speakers. I probably only tested it in a very specific situation, and it was probably borked in that specific way. It wouldn’t surprise me if the same bug also happens for reading via different APIs. It stems from an inconsistent use of raw file writers vs. formatted file writers (e.g.
FILE write()vs.std::ostream).@alexbeattie42 , I’m actually feeling quite seen ^_^ - I think you’re probably one of the first people to read OSC’s source 😄
It looks like @adamkewley has used this solution in OpenSimCreator if you also want to check that for reference.
<del>Alright, i’ll try it out the fix in
IO.cppfirst and check if these changes work.</del> If so I will open a new PR for a clean commit history. Maybe that way some maintainers will have a deeper look into this as a change inIO.cppcould affect many things./edit: Actually I will try it out in
Storage.cppdue to the reason mentioned.I agree that in research the commas are not used for numerics generally but the system locale is important for things like date and calendar display settings within the system and on the desktop.
I think I’ve figured out where the problem could be solved and found this article and this post very helpful.
Here is where the file stream is opened and on line 281 it is opened. So I believe you could do something like this to change the locale to
en_USfor that specificifstreamwithout impacting the system configuration.