organicmaps: Unable to run map_generator on osm.pbf

Describe the issue I am trying to run maps_generator to Saudi Arabia

I am running this command:

python -m maps_generator --countries="Saudi Arabia_North, Saudi Arabia_South" --skip="Coastline"

Getting this error:

python -m maps_generator --countries="Saudi Arabia_North, Saudi Arabia_South" --skip="Coastline"
Loading configuration from /mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/var/etc/map_generator.ini
[2023-09-02 15:40:01,923] INFO env Start setup ...
[2023-09-02 15:40:01,924] INFO env Check for the osmctools binaries...
[2023-09-02 15:40:02,055] ERROR env Uncaught exception: stat: path should be string, bytes, os.PathLike or integer, not NoneType
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/__main__.py", line 250, in <module>
    main()
  File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/__main__.py", line 232, in main
    env = Env(
  File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/generator/env.py", line 390, in __init__
    for k, v in self.setup_osm_tools().items():
  File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/generator/env.py", line 555, in setup_osm_tools
    if all([is_executable(t) for t in tmp_paths]):
  File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/generator/env.py", line 555, in <listcomp>
    if all([is_executable(t) for t in tmp_paths]):
  File "/mnt/c/users/k/Documents/organicmaps/tools/python/maps_generator/utils/file.py", line 43, in is_executable
    return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
  File "/usr/lib/python3.10/genericpath.py", line 30, in isfile
    st = os.stat(path)
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

osm.pbf file from: http://download.geofabrik.de/asia/gcc-states-latest.osm.pbf

this is my map_generator.ini:

[Developer]
# Path to the `organicmaps` source code repository:
OMIM_PATH: /mnt/c/users/k/Documents/organicmaps
# A path with the generator_tool binary:
BUILD_PATH: ${Developer:OMIM_PATH}/../omim-build-release


[Main]
# A special small planet file will be downloaded if DEBUG is set to 1.
DEBUG: 0
# A main working directory. There is a subdirectory created for each generator run
# which contains the planet and other downloads, temporary build files, logs and completed MWMs.
MAIN_OUT_PATH: ${Developer:OMIM_PATH}/../maps_build
# Path for storing caches for nodes, ways, relations.
# If it's not set then caches are stored inside the directory of the current build.
# CACHE_PATH: ${Main:MAIN_OUT_PATH}/cache


[Generator tool]
# Path to the data/ folder in the repository:
USER_RESOURCE_PATH: ${Developer:OMIM_PATH}/data
# Features stage only parallelism level. Set to 0 for auto detection.
THREADS_COUNT_FEATURES_STAGE: 0


[Osm tools]
# Path to osmctools binaries:
OSM_TOOLS_PATH: ${Developer:OMIM_PATH}/../osmctools
# If the binaries are not found neither in the configured path nor system-wide,
# then the tools are built from the sources:
OSM_TOOLS_SRC_PATH: ${Developer:OMIM_PATH}/tools/osmctools


[Logging]
# maps_generator's general (python output only) log file path and name.
# More detailed logs that include output of the `generator_tool` binary
# are located in the `logs/` subdir of a particular build directory,
# e.g. `maps_build/2023_06_04__20_05_07/logs/`.
LOG_FILE_PATH: ${Main:MAIN_OUT_PATH}/generation.log


[External]
# Planet file location. It should be a dump of OSM data in osm.pbf format.
# By default its an entire planet from "planet.openstreetmap.org".
# Or set it to a particular country/region extract from e.g. [Geofabrik](http://download.geofabrik.de/index.html).
# Note that an entire planet generation takes 40+ hours on a 256GB RAM server (and 1TB+ disk space).
# Stick to smaller extracts unless you have a machine this large.
# Here and further, its possible to specify either an URL (to be downloaded automatically)
# or a local file path like file:///path/to/file.
# A sample URL to download a latest OSM dump for North Macedonia:
PLANET_URL: file://mnt/c/users/k/Downloads/gcc-states-latest.osm.pbf
# Location of the md5 checksum of the planet file:
PLANET_MD5_URL: file://mnt/c/users/k/Downloads/gcc-states-latest.osm.pbf.md5
# A base url to the latest_coasts.geom and latest_coasts.rawgeom files.
# For example, if PLANET_COASTS_URL = https://somesite.com/download/
# then the https://somesite.com/download/latest_coasts.geom url will be used to download latest_coasts.geom and
# the https://somesite.com/download/latest_coasts.rawgeom url will be used to download latest_coasts.rawgeom.
# Comment to skip getting the coastlines files.
# PLANET_COASTS_URL:
# Should be 'true' for an entire planet build to make a special routing section in World.mwm
# for alerting about absent regions without which the route can't be built.
NEED_BUILD_WORLD_ROADS: false
# Subway file location, see docs/SUBWAY_GENERATION.md if you want to generate your own file.
# Comment to disable subway layer generation.
#SUBWAY_URL: https://cdn.organicmaps.app/subway.json

# Location of the EXPERIMENTAL GTFS-extracted public transport transit files:
# TRANSIT_URL:

# Urls for production maps generation.
# UGC_URL:
# HOTELS_URL:
# PROMO_CATALOG_CITIES:
# POPULARITY_URL:
# FOOD_URL:
# FOOD_TRANSLATIONS_URL:
# SRTM_PATH:
# ISOLINES_PATH:
# UK_POSTCODES_URL:
# US_POSTCODES_URL:


[Stages]
# Set to 1 to update the entire OSM planet file (as taken from "planet.openstreetmap.org")
# via an osmupdate tool before the generation. Not for use with partial planet extracts.
NEED_PLANET_UPDATE: 0
# If you want to calculate diffs you need to specify where the old maps are.
DATA_ARCHIVE_DIR: ${Generator tool:USER_RESOURCE_PATH}
# How many versions in the archive to use for diff calculation:
DIFF_VERSION_DEPTH: 2


[Common]
# Default parallelism level for the most of jobs. Set to 0 for auto detection.
THREADS_COUNT: 0


[Stats]
# Path to rules for calculating statistics by type:
STATS_TYPES_CONFIG: ${Developer:OMIM_PATH}/tools/python/maps_generator/var/etc/stats_types_config.txt

And I’ve run these commands before:

./tools/unix/build_omim.sh -r desktop
./tools/unix/build_omim.sh -r generator_tool
./tools/unix/build_omim.sh -r world_roads_builder_tool

System information:

  • Operating system and its version: [Windows 11 + WSL=Ubuntu 22.04.]

About this issue

  • Original URL
  • State: closed
  • Created 10 months ago
  • Comments: 26 (13 by maintainers)

Commits related to this issue

Most upvoted comments

@vng Right, my device has 64 GB of RAM. I made changes to the ‘settings.py’ file, and It was done successfully. For WSL, it was successful after: mv /mnt/c/Users/k/Documents/organicmaps /home/k/organicmaps @biodranik @vng @pastk @RedAuburn Thank you very much for your efforts and time in solving this problem, and thank you for this wonderful application

./generator_tool
  --threads_count=1
  --intermediate_data_path=/home/k/maps_build/2023_09_07__23_43_46/intermediate_data
  --cache_path=/home/k/maps_build/2023_09_07__23_43_46/intermediate_data
  --osm_file_type=o5m
  --osm_file_name=/home/k/maps_build/2023_09_07__23_43_46/planet.o5m
  --node_storage=mem
  --user_resource_path=/home/k/organicmaps/data
  --preprocess=true 

–node_storage=mem (means memory) will give bad_alloc if you don’t have enough RAM. Our production server has 128Gb RAM + 132Gb Swap.

Looks like this check is not valid anymore. Should set at least 128.

NODE_STORAGE = "mem" if total_virtual_memory() / 10 ** 9 >= 64 else "map"

@abdullahO2 You can patch and set NODE_STORAGE = "map" in settings.py Honestly saying, better to move it again into map_generator.ini