x

osm2pgsql Import | Server Specifications


  1. osm2pgsql Import | Server Specifications · Yonggan (Gast) · 04.11.2021 12:40 · [flux]

    Hello together,

    i am currently working on an App with OSM Integration.
    I am currently using the official OpenStreetMap Tile Server and the official Nomatim Server for searching locations. I have tried to setup my own servers with this two docker containers:

    https://github.com/mediagis/nominatim-docker
    https://github.com/Overv/openstreetmap-tile-server

    I have tried the nomatim Import and it took me more than a month on my specs and didn´t even finished.
    So i wanted to ask for reference what server specs i have roughly have to use to import both in less then a month.

    I am trying to import the Geofabrik Europe Map.

    Would be nice if somebody could assist me with this 🙂


    • Re: osm2pgsql Import | Server Specifications · Jochen Topf (Gast) · 04.11.2021 16:31 · [flux]

      The documentation contains some information on this, see https://nominatim.org/release-docs/late … /#hardware and https://osm2pgsql.org/doc/manual.html#s … quirements . For Europe you should have 64 GB RAM, maybe it will work with 32 GB RAM, something like 600 GB SSD (don't try with a spinning hard disk, it is far too slow), at least 4 CPUs, more is better. Those numbers are for each of Nominatim and osm2pgsql, not together! I don't now anything about those docker containers, so maybe they do that already, but make sure to tune your PostgreSQL setup (instructions on that is in the documentation, too). Tuning or not tuning your database setup can make the difference between this running in a few hours, or taking days.


    • Re: osm2pgsql Import | Server Specifications · Nop (Gast) · 04.11.2021 20:55 · [flux]

      I can't say for Nominatim, but the numbers above for osm2pgsql seem to be excessive. My server imports almost all of Europe, the import is finished in less than 2 hours.

      It is a Hexacore with 64GB available, but during import the peak memory usage is below 48GB - including all other things running on the machine. The resulting database for mapnik rendering on the SSD is less than 40GB, so any SSD above 128GB should be fine.

      I don't know what your docker containers are doing, of course. My database configuration is fine-tuned for performance and osm2pgsql is using a nodelist file to reduce memory consumption, if you use it without a nodelist and force it to cache all nodes in main memory, the consumption is much higher, then you might need > 64GB.

      As for SSDs, in my experience two smaller SSDs mounted separately work better than one large SSD or RAID. Put the europe.pbf and the nodelist on one SSD and the database on another, then both SSDs can run with their full read and write performance during import.


    • Re: osm2pgsql Import | Server Specifications · Yonggan (Gast) · 06.11.2021 11:32 · [flux]

      Thank you that helped me a lot 🙂