osrm-backend: windows osrm-extract of North America failing with I/O Error occurred
Newcomer to this project, and was able to get the build generated with the latest files, after running into a couple of other issues For anyone reading this in the future, here were the errors that may help you troubleshoot during Windows build:
- The build failed when there was a space in the filepath during cmake – removing spaces in the path where the source was stored fixed that.
- When I first tried build/release/osrm-extract.exe, there were error messages about missing DLLs – copying dependencies tbb.dll and libexpat.dll from osrm-deps/lib/ into build/release/ remedied this issue.
Now on to the thing I don’t know how to fix : ) My environment is a W10 Pro 64-bit installation with 32GB of RAM, 2TB SSD, and Ryzen 7 3700X CPU
I attempted to extract the North America pbf, but am hitting an I/O issue. (log at the bottom) I can successfully extract a smaller .pbf (Washington state), but probably don’t have enough RAM for the whole of NA.
Whenever it begins the below step, I reliably run into the error:
Generating edge-expanded graph representation
I/O error occurred: ..\..\..\map_pbfs\north-america-latest.osrm : /extractor/barriers (possible cause: "Invalid argument") (at include\storage/tar.hpp:52)
I’ve tried to find a solution in the issues here, but no luck. I’ve tried 10+ variations, and no matter what I try it always errors at that point. Remedies I’ve tried:
- I’ve tried to compile with stxxl, but saw no difference. – I checked the default stxxl location and also created a .stxxl.txt file, but didn’t see the temp file get created.
- Attempted various number of threads by using the -t argument (as low as -t 2).
- Set Windows virtual memory allocation to 200GB, which allocated a large pagefile.sys (as I understand, this is similar to allocating a Linux swap file)
Is there a way I can get more detail about this error? I read in one of the issue threads that this particular step in the extract was Memory heavy, and several others have hit errors here before as well. If it is a RAM limitation, I would expect to see the virtual memory to be used, and then the process would just take a few orders of magnitude longer.
I’d prefer to run extract locally (without having to rent a machine), so let me know if there is any additional information I can provide to assist in debugging this issue. Any and all suggestions and help are appreciated.
command and output:
osrm-extract.exe …....\map_pbfs\north-america-latest.osm.pbf -p …..\profiles\car.lua [info] Parsed 0 location-dependent features with 0 GeoJSON polygons←[0m [info] Using script …..\profiles\car.lua←[0m [info] Input file: north-america-latest.osm.pbf←[0m [info] Profile: car.lua←[0m [info] Threads: 16←[0m [info] Parsing in progress…←[0m [info] input file generated by osmium/1.8.0←[0m [info] timestamp: 2020-07-30T20:59:02Z←[0m [info] Using profile api version 4←[0m [info] Found 3 turn restriction tags:←[0m [info] motorcar←[0m [info] motor_vehicle←[0m [info] vehicle←[0m [info] Parse relations …←[0m [info] Parse ways and nodes …←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Using profile api version 4←[0m [info] Parsing finished after 439.976 seconds←[0m [info] Raw input contains 1256813774 nodes, 105408556 ways, and 91104 relations, 349251 restrictions←[0m [info] Sorting used nodes … ok, after 4.44185s←[0m [info] Erasing duplicate nodes … ok, after 0.50473s←[0m [info] Sorting all nodes … ok, after 27.1483s←[0m [info] Building node id map … ok, after 55.3839s←[0m [info] Confirming/Writing used nodes … ok, after 19.9171s←[0m [info] Writing barrier nodes … ok, after 0s←[0m [info] Writing traffic light nodes … ok, after 0s←[0m [info] Processed 257548049 nodes←[0m [info] Sorting edges by start … ok, after 62.3636s←[0m [info] Setting start coords … ok, after 98.7579s←[0m [info] Sorting edges by target … ok, after 32.4669s←[0m [info] Computing edge weights … ok, after 124.666s←[0m [info] Sorting edges by renumbered start … ok, after 47.7229s←[0m [info] Writing used edges … ok, after 9.78324s – Processed 271992387 edges←[0m [info] Writing way meta-data … ok, after 0.603115s – Metadata contains << 30512033 entries.←[0m [info] Sorting used ways … ok, after 0.307405s←[0m [info] Collecting start/end information on 29 maneuver overrides…ok, after 0.375136s←[0m [info] Collecting start/end information on 29 maneuver overrides…ok, after 0.034667s←[0m [info] Collecting start/end information on 349251 restrictions…ok, after 2.53855s←[0m [info] Collecting start/end information on 349251 restrictions…ok, after 2.91023s←[0m [info] writing street name index … ok, after 0.481568s←[0m [info] extraction finished after 936.353s←[0m [info] Generating edge-expanded graph representation←[0m [info] RAM: peak bytes used: <not implemented on Windows>←[0m ←[31m[error] I/O error occurred: …....\map_pbfs\north-america-latest.osrm : /extractor/barriers (possible cause: “Invalid argument”) (at include\storage/tar.hpp:52)←[0m
About this issue
- Original URL
- State: closed
- Created 4 years ago
- Comments: 23 (10 by maintainers)
As it turns out, getting docker running on top of a Windows AWS instance is not an easy task, and I ultimately failed to get it running. Nested virtualization is not enabled in EC2, and every time I enabled hyper-v on a metal machine (including using amazon’s AMI), RDP stopped working and I could no longer access the instance.
What I did discover, however, is a workaround to get docker working with (more) swap memory on Windows, which allowed me to use osrm-extract on the whole of North America overnight with just 32GB of RAM and a decent SSD. As @danpat mentions, there is likely still a problem with error handling on windows native binaries. This workaround is more of a workaround for the Docker limitation than it is a change to osrm itself.
What I did:
I did not need to provide -m or --memory-swap parameters to the docker command.
Command and results
Additional Notes:
Edit, added screenshots of the Docker settings screens:
