organicmaps: Unable to run map_generator on osm.pbf
Describe the issue I am trying to run maps_generator to Saudi Arabia
I am running this command:
python -m maps_generator --countries="Saudi Arabia_North, Saudi Arabia_South" --skip="Coastline"
Getting this error:
python -m maps_generator --countries="Saudi Arabia_North, Saudi Arabia_South" --skip="Coastline"
Loading configuration from /mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/var/etc/map_generator.ini
[2023-09-02 15:40:01,923] INFO env Start setup ...
[2023-09-02 15:40:01,924] INFO env Check for the osmctools binaries...
[2023-09-02 15:40:02,055] ERROR env Uncaught exception: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/__main__.py", line 250, in <module>
main()
File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/__main__.py", line 232, in main
env = Env(
File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/generator/env.py", line 390, in __init__
for k, v in self.setup_osm_tools().items():
File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/generator/env.py", line 555, in setup_osm_tools
if all([is_executable(t) for t in tmp_paths]):
File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/generator/env.py", line 555, in <listcomp>
if all([is_executable(t) for t in tmp_paths]):
File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/utils/file.py", line 43, in is_executable
return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
File "/usr/lib/python3.10/genericpath.py", line 30, in isfile
st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
osm.pbf file from:
http://download.geofabrik.de/asia/gcc-states-latest.osm.pbf
this is my map_generator.ini:
[Developer]
# Path to the `organicmaps` source code repository:
OMIM_PATH: /mnt/c/users/k/Documents/organicmaps
# A path with the generator_tool binary:
BUILD_PATH: ${Developer:OMIM_PATH}/../omim-build-release
[Main]
# A special small planet file will be downloaded if DEBUG is set to 1.
DEBUG: 0
# A main working directory. There is a subdirectory created for each generator run
# which contains the planet and other downloads, temporary build files, logs and completed MWMs.
MAIN_OUT_PATH: ${Developer:OMIM_PATH}/../maps_build
# Path for storing caches for nodes, ways, relations.
# If it's not set then caches are stored inside the directory of the current build.
# CACHE_PATH: ${Main:MAIN_OUT_PATH}/cache
[Generator tool]
# Path to the data/ folder in the repository:
USER_RESOURCE_PATH: ${Developer:OMIM_PATH}/data
# Features stage only parallelism level. Set to 0 for auto detection.
THREADS_COUNT_FEATURES_STAGE: 0
[Osm tools]
# Path to osmctools binaries:
OSM_TOOLS_PATH: ${Developer:OMIM_PATH}/../osmctools
# If the binaries are not found neither in the configured path nor system-wide,
# then the tools are built from the sources:
OSM_TOOLS_SRC_PATH: ${Developer:OMIM_PATH}/tools/osmctools
[Logging]
# maps_generator's general (python output only) log file path and name.
# More detailed logs that include output of the `generator_tool` binary
# are located in the `logs/` subdir of a particular build directory,
# e.g. `maps_build/2023_06_04__20_05_07/logs/`.
LOG_FILE_PATH: ${Main:MAIN_OUT_PATH}/generation.log
[External]
# Planet file location. It should be a dump of OSM data in osm.pbf format.
# By default its an entire planet from "planet.openstreetmap.org".
# Or set it to a particular country/region extract from e.g. [Geofabrik](http://download.geofabrik.de/index.html).
# Note that an entire planet generation takes 40+ hours on a 256GB RAM server (and 1TB+ disk space).
# Stick to smaller extracts unless you have a machine this large.
# Here and further, its possible to specify either an URL (to be downloaded automatically)
# or a local file path like file:///path/to/file.
# A sample URL to download a latest OSM dump for North Macedonia:
PLANET_URL: file://mnt/c/users/k/Downloads/gcc-states-latest.osm.pbf
# Location of the md5 checksum of the planet file:
PLANET_MD5_URL: file://mnt/c/users/k/Downloads/gcc-states-latest.osm.pbf.md5
# A base url to the latest_coasts.geom and latest_coasts.rawgeom files.
# For example, if PLANET_COASTS_URL = https://somesite.com/download/
# then the https://somesite.com/download/latest_coasts.geom url will be used to download latest_coasts.geom and
# the https://somesite.com/download/latest_coasts.rawgeom url will be used to download latest_coasts.rawgeom.
# Comment to skip getting the coastlines files.
# PLANET_COASTS_URL:
# Should be 'true' for an entire planet build to make a special routing section in World.mwm
# for alerting about absent regions without which the route can't be built.
NEED_BUILD_WORLD_ROADS: false
# Subway file location, see docs/SUBWAY_GENERATION.md if you want to generate your own file.
# Comment to disable subway layer generation.
#SUBWAY_URL: https://cdn.organicmaps.app/subway.json
# Location of the EXPERIMENTAL GTFS-extracted public transport transit files:
# TRANSIT_URL:
# Urls for production maps generation.
# UGC_URL:
# HOTELS_URL:
# PROMO_CATALOG_CITIES:
# POPULARITY_URL:
# FOOD_URL:
# FOOD_TRANSLATIONS_URL:
# SRTM_PATH:
# ISOLINES_PATH:
# UK_POSTCODES_URL:
# US_POSTCODES_URL:
[Stages]
# Set to 1 to update the entire OSM planet file (as taken from "planet.openstreetmap.org")
# via an osmupdate tool before the generation. Not for use with partial planet extracts.
NEED_PLANET_UPDATE: 0
# If you want to calculate diffs you need to specify where the old maps are.
DATA_ARCHIVE_DIR: ${Generator tool:USER_RESOURCE_PATH}
# How many versions in the archive to use for diff calculation:
DIFF_VERSION_DEPTH: 2
[Common]
# Default parallelism level for the most of jobs. Set to 0 for auto detection.
THREADS_COUNT: 0
[Stats]
# Path to rules for calculating statistics by type:
STATS_TYPES_CONFIG: ${Developer:OMIM_PATH}/tools/python/maps_generator/var/etc/stats_types_config.txt
And I’ve run these commands before:
./tools/unix/build_omim.sh -r desktop
./tools/unix/build_omim.sh -r generator_tool
./tools/unix/build_omim.sh -r world_roads_builder_tool
System information:
- Operating system and its version: [Windows 11 + WSL=Ubuntu 22.04.]
About this issue
- Original URL
- State: closed
- Created 10 months ago
- Comments: 26 (13 by maintainers)
Commits related to this issue
- [generator] fix building of osmctools resolves #5908 Signed-off-by: Harry Bond <endim8@pm.me> — committed to RedAuburn/organicmaps by deleted user 10 months ago
- [generator] fix building of osmctools resolves #5908 Signed-off-by: Harry Bond <endim8@pm.me> — committed to RedAuburn/organicmaps by deleted user 10 months ago
- [generator] fix building of osmctools resolves #5908 Signed-off-by: Harry Bond <endim8@pm.me> — committed to RedAuburn/organicmaps by deleted user 10 months ago
@vng Right, my device has 64 GB of RAM. I made changes to the ‘settings.py’ file, and It was done successfully. For WSL, it was successful after:
mv /mnt/c/Users/k/Documents/organicmaps /home/k/organicmaps@biodranik @vng @pastk @RedAuburn Thank you very much for your efforts and time in solving this problem, and thank you for this wonderful application–node_storage=mem (means memory) will give bad_alloc if you don’t have enough RAM. Our production server has 128Gb RAM + 132Gb Swap.
Looks like this check is not valid anymore. Should set at least 128.
@abdullahO2 You can patch and set
NODE_STORAGE = "map"in settings.py Honestly saying, better to move it again into map_generator.ini