Fixes #18

This PR provides the ability for a user to specify an arbitrary vector schema via a configuration file. Configuration is via a YML file which specifies the contents of the configured layers. Documentation for the schema definition format is provided in planetiler-custommap/README.md.

The schema language allows for the following features:

• Ability to define layers and features within layers based on tag filters
• Ability to specify attributes as a function of tags
• Ability to specify feature zoom levels as a function of tagging or feature area
• Ability to specify data sources in a configuration file

A collection of unit tests are provided, as well as a sample schema intended to support OWG vector tile exploration under openstreetmap/operations#565

A typical invocation looks something like:

(java/docker launch) generate-custom --schema=schema_file.yml --download --area=rhode-island

This PR does not include handling for line merging, which needs a deeper investigation to deal with attributes that have minimum zooms greater than the overall feature minimum zoom, which would allow certain linestring features to merge only at certain lower zooms.

Context

this addresses point 2 of https://github.com/onthegomap/planetiler/issues/167#issuecomment-1115483329

Overview

the compact DB mode splits the tiles table into tiles_shallow and tiles_data

• tiles_shallow contains the coordinates plus a reference on the data ID
• tiles_data contains the data ID plus the actual tile data

this allows to de-duplicate content since multiple tiles can reference the same data

in this mode, tiles is realized as a view that joins the two tables tiles_shallow and tiles_data

Impact

• it can significantly de-crease the mbtiles file size iff there's many duplicates (ocean tiles) - in case of Australia: 3.1GB -> 1.7GB
• it has little to no impact on writing mbtiles (if no dupes it's slower, and the more dupes the faster but not significantly)
• reading from the mbtiles is 10-15% slower because of the indirection over the view

some more insights can be found here: PR-impact.txt

Concept

The main idea to realize this is to create a hash over the tile data, and build a map between that hash and a data ID.

Failed Approaches

I was a bit surprised that the impact on writing the mbtiles is little to none. The only explanation I have is that we have to do more inserts than before (tiles_shallow, tiles_data). I tried to offload this "dual insert" from code to the DB. So in planetiler I produced just one insert but the insert was into a view - which then either just inserted into tiles_shallow or both tiles_shallow and tiles_data. This, however, increased the overall mbtiles generation time by almost 20%.

Next

I'd like to address point 1 in https://github.com/onthegomap/planetiler/issues/167#issuecomment-1115483329 either in this or a follow-up PR

This is a rough idea right now (don't merge yet) to explore alternate TileCoord encodings. it deserves some better tests for edge cases as right now it's blindly ported from C++ to java

The current TileCoord uses a fixed 14 bits for X and Y and the remaining bits for Z. This instead defines a mapping from a 32-bit unsigned encoded integer to X,Y,Z like this:

Tile 0 is (0,0) on z0 Tile 1-4 are positions on the z1 Hilbert curve; each side of the curve is length 2, so tile 1 is (0,0), tile 2 is (0,1), tile 3 is (1,1) and tile 4 is (1,0) Tile 5-20 are positions on the z2 Hilbert curve etc...

Drawbacks:

• decoding/encoding might more expensive
• does not correspond to index insertion order in MBTiles, but there may be other ways to mitigate this

Potential benefits of doing it this way:

• support z15 without widening the TileCoord encoding to 64 bits
• If features are ordered this way and memory-mapped on disk, reads should have better locality for 2D queries vs. striped Z/X/Y ordering
• for some other tools I'm developing this would aid interoperability, such as a bitmap format for sparse ocean representation or a future ordering requirement for PMTiles

This will allow rendering with —maxzoom=6 while having zoom 6 rendered the same way as if —maxzoom=14.

maxzoomForRendering will be used for simplifly algorithms. The default is MAX_ZOOM(14) so by default whatever maxzoom is used you will get the same render as if maxzoom=14

This PR adds Planetiler.addShapefileDirectorySource, which can be used to add all Shapefiles matching a glob pattern in a directory to a single named source. Other than grouping related shapefiles together, the main benefit here is that the files can then be processed concurrently.

Was mentioned in Slack a couple weeks back: https://osmus.slack.com/archives/C031V9E9RMG/p1668717797875339

The main feedback on https://news.ycombinator.com/item?id=28986762 was that "flatmap" is ambiguous and not unique. Before publishing to maven central, package managers, etc should decide whether or not to keep "flatmap" or use something different.

Pros:

• easier to find through search.
• It's still early so there are too many inbound links yet

Cons:

• there are still some links, forks, and stars so need to be careful to preserve those

Any suggestions?

Describe the bug The names of POI/buildings are missing using the default basemap profile.

Expected behavior The output mbtiles should include the name tags of buildings and POIs but they are dropped.

To Reproduce

1. Download Malaysia, Singapore, and Brunei from geofabrik
2. Run java -Xmx1g -jar planetiler.jar --download --osm_path="malaysia-singapore-brunei-old.osm.pbf"
3. Inspect location lat: 1.29788, lon: 103.77677

Long description I am generating vector tiles for Singapore and have been using tilemaker prior to planetiler. Using Way 427077072 as an example, the building name in OSM is 'Ridge View Tower' and I would expect the output mbtiles to contain the same information.

In the planetiler mbtiles, however, this name field is missing. Inspecting the planetiler mbtiles using QGIS reveals that planetiler only saves housenumbers and does not save POI/building names.

Here are some screenshots describing the problem:

OSM groundtruth: Planetiler output: Tilemaker output:

Is there a way I can edit the planetiler profile to match the detailed output of tilemaker/OSM ?

Environment (please complete the following information):

• OS: Windows 10
• Java version and distribution: [e.g. Eclipse Temurin 17.0.2]

The osm pbf data comes from daylightmap. Problems occur when executing to natural earth. Here are the logs：

0:00:00 DEB - argument: config=null (path to config file)
0:00:00 DEB - argument: area=planet (name of the extract to download if osm_url/osm_path not specified (i.e. 'monaco' 'rhode island' 'australia' or 'planet'))
0:00:00 INF - Using in-memory stats
0:00:00 INF [overall] - 
0:00:00 INF [overall] - Starting...
0:00:00 DEB - argument: madvise=true (default value for whether to use linux madvise(random) to improve memory-mapped read performance for temporary storage)
0:00:00 DEB - argument: storage=ram (default storage type for temporary data, one of [ram, mmap, direct])
0:00:00 DEB - argument: threads=36 (num threads)
0:00:00 DEB - argument: write_threads=1 (number of threads to use when writing temp features)
0:00:00 DEB - argument: process_threads=35 (number of threads to use when processing input features)
0:00:00 DEB - argument: bounds=Env[-180.0 : 180.0, -85.0511287798066 : 85.0511287798066] (bounds)
0:00:00 DEB - argument: feature_read_threads=2 (number of threads to use when reading features at tile write time)
0:00:00 DEB - argument: loginterval=10 seconds (time between logs)
0:00:00 DEB - argument: minzoom=0 (minimum zoom level)
0:00:00 DEB - argument: maxzoom=14 (maximum zoom level (limit 14))
0:00:00 DEB - argument: skip_mbtiles_index_creation=false (skip adding index to mbtiles file)
0:00:00 DEB - argument: optimize_db=false (optimize mbtiles after writing)
0:00:00 DEB - argument: emit_tiles_in_order=true (emit tiles in index order)
0:00:00 DEB - argument: force=false (overwriting output file and ignore disk/RAM warnings)
0:00:00 DEB - argument: gzip_temp=false (gzip temporary feature storage (uses more CPU, but less disk space))
0:00:00 DEB - argument: mmap_temp=true (use memory-mapped IO for temp feature files)
0:00:00 DEB - argument: sort_max_readers=6 (maximum number of concurrent read threads to use when sorting chunks)
0:00:00 DEB - argument: sort_max_writers=6 (maximum number of concurrent write threads to use when sorting chunks)
0:00:00 DEB - argument: nodemap_type=array (type of node location map, one of [noop, sortedtable, sparsearray, array])
0:00:00 DEB - argument: nodemap_storage=ram (storage for node location map, one of [ram, mmap, direct])
0:00:00 DEB - argument: nodemap_madvise=true (use linux madvise(random) for node locations)
0:00:00 DEB - argument: multipolygon_geometry_storage=ram (storage for multipolygon geometries, one of [ram, mmap, direct])
0:00:00 DEB - argument: multipolygon_geometry_madvise=true (use linux madvise(random) for temporary multipolygon geometry storage)
0:00:00 DEB - argument: http_user_agent=Planetiler downloader (https://github.com/onthegomap/planetiler) (User-Agent header to set when downloading files over HTTP)
0:00:00 DEB - argument: http_timeout=30 seconds (Timeout to use when downloading files over HTTP)
0:00:00 DEB - argument: http_retries=1 (Retries to use when downloading files over HTTP)
0:00:00 DEB - argument: download_chunk_size_mb=1000 (Size of file chunks to download in parallel in megabytes)
0:00:00 DEB - argument: download_threads=10 (Number of parallel threads to use when downloading each file)
0:00:00 DEB - argument: min_feature_size_at_max_zoom=0.0625 (Default value for the minimum size in tile pixels of features to emit at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: min_feature_size=1.0 (Default value for the minimum size in tile pixels of features to emit below the maximum zoom level)
0:00:00 DEB - argument: simplify_tolerance_at_max_zoom=0.0625 (Default value for the tile pixel tolerance to use when simplifying features at the maximum zoom level to allow for overzooming)
0:00:00 DEB - argument: simplify_tolerance=0.1 (Default value for the tile pixel tolerance to use when simplifying features below the maximum zoom level)
0:00:00 DEB - argument: osm_lazy_reads=false (Read OSM blocks from disk in worker threads)
0:00:00 DEB - argument: compact_db=true (Reduce the DB size by separating and deduping the tile data)
0:00:00 DEB - argument: skip_filled_tiles=false (Skip writing tiles containing only polygon fills to the output)
0:00:00 DEB - argument: tmpdir=data/tmp (temp directory)
0:00:00 DEB - argument: only_download=false (download source data then exit)
0:00:00 DEB - argument: download=true (download sources)
0:00:00 DEB - argument: temp_nodes=data/tmp/node.db (temp node db location)
0:00:00 DEB - argument: temp_multipolygons=data/tmp/multipolygon.db (temp multipolygon db location)
0:00:00 DEB - argument: temp_features=data/tmp/feature.db (temp feature db location)
0:00:00 DEB - argument: osm_parse_node_bounds=false (parse bounds from OSM nodes instead of header)
0:00:00 DEB - argument: only_fetch_wikidata=false (fetch wikidata translations then quit)
0:00:00 DEB - argument: fetch_wikidata=false (fetch wikidata translations then continue)
0:00:00 DEB - argument: use_wikidata=true (use wikidata translations)
0:00:00 DEB - argument: wikidata_cache=data/sources/wikidata_names.json (wikidata cache file)
0:00:00 DEB - argument: lake_centerlines_path=data/sources/lake_centerline.shp.zip (lake_centerlines shapefile path)
0:00:00 DEB - argument: free_lake_centerlines_after_read=false (delete lake_centerlines input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: lake_centerlines_url=https://github.com/acalcutt/osm-lakelines/releases/download/latest/lake_centerline.shp.zip (lake_centerlines shapefile url)
0:00:00 DEB - argument: water_polygons_path=data/sources/water-polygons-split-3857.zip (water_polygons shapefile path)
0:00:00 DEB - argument: free_water_polygons_after_read=false (delete water_polygons input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: water_polygons_url=https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip (water_polygons shapefile url)
0:00:00 DEB - argument: natural_earth_path=data/sources/natural_earth_vector.sqlite.zip (natural_earth sqlite db path)
0:00:00 DEB - argument: free_natural_earth_after_read=false (delete natural_earth input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: natural_earth_url=https://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip (natural_earth sqlite db url)
0:00:00 DEB - argument: osm_path=data/sources/planet.osm.pbf (osm OSM input file path)
0:00:00 DEB - argument: free_osm_after_read=false (delete osm input file after reading to make space for output (reduces peak disk usage))
0:00:00 DEB - argument: osm_url=aws:latest (osm OSM input file url)
0:00:00 DEB - argument: mbtiles=output20221110.mbtiles (mbtiles output file)
0:00:00 DEB - argument: transliterate=true (attempt to transliterate latin names)
0:00:00 DEB - argument: languages=am,ar,az,be,bg,br,bs,ca,co,cs,cy,da,de,el,en,eo,es,et,eu,fi,fr,fy,ga,gd,he,hi,hr,hu,hy,id,is,it,ja,ja_kana,ja_rm,ja-Latn,ja-Hira,ka,kk,kn,ko,ko-Latn,ku,la,lb,lt,lv,mk,mt,ml,nl,no,oc,pl,pt,rm,ro,ru,sk,sl,sq,sr,sr-Latn,sv,ta,te,th,tr,uk,zh (languages to use)
0:00:00 DEB - argument: only_layers= (Include only certain layers)
0:00:00 DEB - argument: exclude_layers= (Exclude certain layers)
0:00:00 DEB - argument: boundary_country_names=true (boundary layer: add left/right codes of neighboring countries)
0:00:00 DEB - argument: transportation_z13_paths=false (transportation(_name) layer: show all paths on z13)
0:00:00 DEB - argument: building_merge_z13=true (building layer: merge nearby buildings at z13)
0:00:00 DEB - argument: transportation_name_brunnel=false (transportation_name layer: set to false to omit brunnel and help merge long highways)
0:00:00 DEB - argument: transportation_name_size_for_shield=false (transportation_name layer: allow road names on shorter segments (ie. they will have a shield))
0:00:00 DEB - argument: transportation_name_limit_merge=false (transportation_name layer: limit merge so we don't combine different relations to help merge long highways)
0:00:00 DEB - argument: transportation_name_minor_refs=false (transportation_name layer: include name and refs from minor road networks if not present on a way)
0:00:00 DEB - argument: mbtiles_name=OpenMapTiles ('name' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_description=A tileset showcasing all layers in OpenMapTiles. https://openmaptiles.org ('description' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_attribution=<a href="https://www.openmaptiles.org/" target="_blank">&copy; OpenMapTiles</a> <a href="https://www.openstreetmap.org/copyright" target="_blank">&copy; OpenStreetMap contributors</a> ('attribution' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_version=3.13.1 ('version' attribute for mbtiles metadata)
0:00:00 DEB - argument: mbtiles_type=baselayer ('type' attribute for mbtiles metadata)
0:00:00 DEB - argument: help=false (show arguments then exit)
0:00:00 INF - Building BasemapProfile profile into output20221110.mbtiles in these phases:
0:00:00 INF -   download: Download sources [natural_earth]
0:00:00 INF -   lake_centerlines: Process features in data/sources/lake_centerline.shp.zip
0:00:00 INF -   water_polygons: Process features in data/sources/water-polygons-split-3857.zip
0:00:00 INF -   natural_earth: Process features in data/sources/natural_earth_vector.sqlite.zip
0:00:00 INF -   osm_pass1: Pre-process OpenStreetMap input (store node locations then relation members)
0:00:00 INF -   osm_pass2: Process OpenStreetMap nodes, ways, then relations
0:00:00 INF -   sort: Sort rendered features by tile ID
0:00:00 INF -   mbtiles: Encode each tile and write to output20221110.mbtiles
0:00:00 INF [download] - 
0:00:00 INF [download] - Starting...
0:00:02 INF [download] - Downloading https://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip to data/sources/natural_earth_vector.sqlite.zip
0:00:12 INF [download] -  natural_earth: [  80M  18%   8M/s ]  cpus: 0.4 gc:  0% heap: 35M/118G direct: 41k postGC: 46M
0:00:22 INF [download] -  natural_earth: [ 189M  44%  10M/s ]  cpus: 0.2 gc:  0% heap: 45M/118G direct: 41k postGC: 46M
0:00:32 INF [download] -  natural_earth: [ 291M  67%  10M/s ]  cpus: 0.2 gc:  0% heap: 24M/118G direct: 41k postGC: 52M
0:00:42 INF [download] -  natural_earth: [ 373M  86% 8.2M/s ]  cpus: 0.1 gc:  0% heap: 32M/118G direct: 41k postGC: 52M
0:00:47 INF [download] - Finished downloading https://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip to data/sources/natural_earth_vector.sqlite.zip
0:00:47 INF [download] -  natural_earth: [ 434M 100%  10M/s ]  cpus: 0.1 gc:  0% heap: 38M/118G direct: 25k postGC: 52M
0:00:47 INF [download] - Finished in 48s cpu:10s avg:0.2
0:00:47 INF [download] -   download-natural_earth_chunk-downloader  1x(14% 7s sys:3s done:2s)
0:00:47 INF - no wikidata translations found, run with --fetch-wikidata to download
0:00:47 DEB - ✓ 242G storage on / (/dev/sda2) requested for read phase disk, 1.5T available
0:00:47 DEB -  - 242G used for temporary feature storage
0:00:47 DEB - ✓ 364G storage on / (/dev/sda2) requested for write phase disk, 1.5T available
0:00:47 DEB -  - 242G used for temporary feature storage
0:00:47 DEB -  - 121G used for mbtiles output
0:00:47 INF - ⚠️️ 115G JVM heap requested for read phase, 118G available (increase JVM -Xmx setting)
0:00:47 INF -    - 82G used for array node location cache (switch to sparsearray to reduce size)
0:00:47 INF -    - 11G used for multipolygon way geometries
0:00:47 INF -    - 21G used for temporary profile storage
0:00:47 DEB - ✓ 0 temporary files and 16G of free memory for OS to cache them
0:00:47 INF - Using merge sort feature map, chunk size=2000mb max workers=36
0:00:48 INF - dataFileCache open start
0:00:48 INF [lake_centerlines] - 
0:00:48 INF [lake_centerlines] - Starting...
0:00:50 INF [lake_centerlines] -  read: [  59k 100%  33k/s ] write: [    0    0/s ] 0    
    cpus: 4.8 gc:  4% heap: 406M/118G direct: 254k postGC: 182M
    read( -%) ->    (0/1k) -> process( -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%) ->   (0/86k) -> write( -%)
0:00:50 INF [lake_centerlines] - Finished in 2s cpu:9s avg:4.8
0:00:50 INF [lake_centerlines] -   read     1x(79% 1s)
0:00:50 INF [lake_centerlines] -   process 35x(5% 0.1s wait:2s)
0:00:50 INF [lake_centerlines] -   write    1x(0% 0s wait:2s)
0:00:50 INF [water_polygons] - 
0:00:50 INF [water_polygons] - Starting...
0:01:00 INF [water_polygons] -  read: [ 2.1k  15%  211/s ] write: [  11M 1.1M/s ] 2G   
    cpus: 34.6 gc:  8% heap: 2.9G/118G direct: 66M postGC: 2.7G
    read(86%) ->  (747/1k) -> process(84% 80% 82% 85% 86% 85% 81% 83% 82% 85% 85% 84% 85% 85% 83% 86% 82% 87% 82% 85% 82% 82% 81% 82% 83% 83% 83% 85% 83% 86% 84% 79% 83% 83% 82%) -> (15k/86k) -> write(24%)
0:01:10 INF [water_polygons] -  read: [ 3.9k  28%  183/s ] write: [  20M 899k/s ] 3G   
    cpus: 34.8 gc:  7% heap: 18G/118G direct: 66M postGC: 2.9G
    read(73%) ->  (988/1k) -> process(91% 92% 92% 93% 93% 91% 92% 92% 92% 93% 92% 91% 92% 92% 90% 91% 91% 92% 93% 93% 92% 90% 92% 91% 92% 91% 92% 92% 91% 92% 93% 91% 92% 92% 92%) -> (17k/86k) -> write(22%)
0:01:20 INF [water_polygons] -  read: [   6k  42%  209/s ] write: [  34M 1.3M/s ] 3G   
    cpus: 34.9 gc:  6% heap: 18G/118G direct: 66M postGC: 2.1G
    read(41%) ->   (1k/1k) -> process(93% 93% 94% 94% 93% 94% 94% 94% 93% 93% 94% 94% 93% 93% 94% 93% 92% 92% 94% 92% 94% 91% 92% 93% 93% 94% 94% 93% 93% 94% 93% 94% 94% 94% 94%) -> (33k/86k) -> write(31%)
0:01:30 INF [water_polygons] -  read: [ 9.9k  69%  386/s ] write: [ 118M 8.3M/s ] 6.8G 
    cpus: 16.3 gc:  0% heap: 22G/118G direct: 66M postGC: 1.2G
    read( 2%) ->   (1k/1k) -> process(42% 41% 42% 41% 41% 42% 42% 41% 42% 41% 41% 43% 41% 42% 53% 47% 43% 41% 41% 46% 41% 43% 43% 43% 42% 40% 41% 56% 45% 41% 42% 43% 41% 41% 47%) -> (82k/86k) -> write(98%)
0:01:40 INF [water_polygons] -  read: [  13k  96%  392/s ] write: [ 208M 8.9M/s ] 11G  
    cpus: 13.5 gc:  0% heap: 14G/118G direct: 66M postGC: 1.2G
    read( -%) ->  (591/1k) -> process(36% 37% 35% 36% 36% 36% 36% 36% 36% 36% 35% 36% 36% 36% 36% 35% 36% 35% 36% 36% 36% 36% 35% 35% 36% 35% 35% 36% 36% 35% 36% 36% 36% 36% 35%) -> (78k/86k) -> write(100%)
0:01:42 INF [water_polygons] -  read: [  14k 100%  369/s ] write: [ 222M 8.7M/s ] 10G  
    cpus: 13.4 gc:  0% heap: 2.5G/118G direct: 66M postGC: 1.2G
    read( -%) ->    (0/1k) -> process( -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%) ->   (0/86k) -> write( -%)
0:01:42 INF [water_polygons] - Finished in 52s cpu:22m50s gc:2s avg:26.4
0:01:42 INF [water_polygons] -   read     1x(39% 20s sys:1s wait:26s done:3s)
0:01:42 INF [water_polygons] -   process 35x(68% 35s wait:13s)
0:01:42 INF [water_polygons] -   write    1x(56% 29s sys:7s wait:22s)
0:01:42 INF [natural_earth] - unzipping /home/cq/planetiler/data/sources/natural_earth_vector.sqlite.zip to data/tmp/natearth.sqlite
0:01:46 INF [natural_earth] - 
0:01:46 INF [natural_earth] - Starting...
0:01:56 INF [natural_earth] -  read: [ 334k  96%  33k/s ] write: [   4k  399/s ] 12G  
    cpus: 3.5 gc:  0% heap: 8.1G/118G direct: 66M postGC: 1.2G
    read(70%) ->    (-/1k) -> process( 6%  6%  5%  6%  5%  6%  6%  6% 13%  5%  6%  6%  6%  5%  5%  5%  6%  6%  6%  5%  6%  6%  6%  6%  6%  6%  5%  6%  6%  6%  5%  5%  5%  6%  5%) -> (25k/86k) -> write( 0%)
0:01:56 INF [natural_earth] -  read: [ 349k 100%  25k/s ] write: [  33k  49k/s ] 10G  
    cpus: 3.3 gc:  0% heap: 8.6G/118G direct: 66M postGC: 1.2G
    read( -%) ->    (0/1k) -> process( -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%  -%) ->   (0/86k) -> write( -%)
0:01:56 INF [natural_earth] - Finished in 11s cpu:37s avg:3.4
0:01:56 INF [natural_earth] -   read     1x(67% 7s wait:3s)
0:01:56 INF [natural_earth] -   process 35x(6% 0.6s wait:10s)
0:01:56 INF [natural_earth] -   write    1x(0% 0s wait:11s)
0:01:57 INF [osm_pass1] - 
0:01:57 INF [osm_pass1] - Starting...
Worker osm_pass1_read-1 died
java.io.UncheckedIOException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: type, datasize
        at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:165)
        at com.onthegomap.planetiler.worker.WorkerPipeline$Empty.lambda$fromGenerator$0(WorkerPipeline.java:170)
        at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41)
        at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: type, datasize
        at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
        at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:68)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
        at org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader.parseFrom(Fileformat.java:1471)
        at com.onthegomap.planetiler.reader.osm.OsmInputFile.readBlobHeader(OsmInputFile.java:96)
        at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:153)
        ... 7 more
Exception in thread "main" com.onthegomap.planetiler.util.Exceptions$FatalPlanetilerException: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: type, datasize
        at com.onthegomap.planetiler.util.Exceptions.throwFatalException(Exceptions.java:29)
        at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:389)
        at com.onthegomap.planetiler.stats.ProgressLoggers.awaitAndLog(ProgressLoggers.java:374)
        at com.onthegomap.planetiler.reader.osm.OsmReader.pass1(OsmReader.java:221)
        at com.onthegomap.planetiler.Planetiler.lambda$addOsmSource$0(Planetiler.java:192)
        at com.onthegomap.planetiler.Planetiler.lambda$ifSourceUsed$6(Planetiler.java:652)
        at com.onthegomap.planetiler.Planetiler.run(Planetiler.java:535)
        at com.onthegomap.planetiler.basemap.BasemapMain.run(BasemapMain.java:51)
        at com.onthegomap.planetiler.basemap.BasemapMain.main(BasemapMain.java:14)
        at com.onthegomap.planetiler.Main.main(Main.java:50)
Caused by: java.util.concurrent.ExecutionException: java.io.UncheckedIOException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: type, datasize
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
        at com.onthegomap.planetiler.stats.ProgressLoggers.await(ProgressLoggers.java:383)
        ... 8 more
Caused by: java.io.UncheckedIOException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: type, datasize
        at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:165)
        at com.onthegomap.planetiler.worker.WorkerPipeline$Empty.lambda$fromGenerator$0(WorkerPipeline.java:170)
        at com.onthegomap.planetiler.worker.Worker.lambda$new$0(Worker.java:41)
        at com.onthegomap.planetiler.worker.Worker.lambda$new$1(Worker.java:68)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: type, datasize
        at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
        at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:68)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:191)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:203)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:208)
        at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:48)
        at org.openstreetmap.osmosis.osmbinary.Fileformat$BlobHeader.parseFrom(Fileformat.java:1471)
        at com.onthegomap.planetiler.reader.osm.OsmInputFile.readBlobHeader(OsmInputFile.java:96)
        at com.onthegomap.planetiler.reader.osm.OsmInputFile$EagerReader.forEachBlock(OsmInputFile.java:153)
        ... 7 more

Describe the bug

In Germany Planetiler is generating only few refs for major roads like motorways or trunks. The reason seems to be that Planetiler calculates the rank of a relation based on if it's part of a set of preconfigured networks (like US:). If these are not set, Planetiler uses other tags for hiking networks etc to figure out the rank. See the code here.

Here are two examples from Germany where this approach does not work properly which leaves major roads without road signs. Motorway: https://www.openstreetmap.org/relation/2166961 Trunk: https://www.openstreetmap.org/relation/1506786

Expected behavior As these are important roads, it would be good to have road signs for these roads.

Screenshots

Here is a comparison on z11 between OMT (has road signs) and Planetiler (no road signs):

Potential fix

The transportation_name_minor could be only used for minor roads. This would invalidate the fix for #145.

So how about we apply a low rank for relations with network=e-road and when getting the relations of a road, we order them by rank? So right now, the default rank is 3, so maybe e-roads could be rank 4?

Currently, if a shapefile has attributes with letters like ä, ö, ü, etc, the resulting tiles show some strange letters. This pull requests fixes this issue by hard-coding the shapefile mode to UTF-8.



Adds newlines to the OpenMapTiles table file. My motivation for this is to simplify customization of the basemap profile. For example, with the hot-air balloon I often land in places where it matters if the highway=track has tracktype=grade1 or tracktype=grade5. The tracktype tag is not part of OMT, so I added it manually to the basemap profile. It is super fun to customize the tiles like this. After this pull request, adding new tags will only insert a new line and result in a simpler git diff. Merging upstream changes will also be easier.

This PR pulls ZIP file handling out of ShapefileReader so that it can be reused for different readers. The changes here can be applied to other reader types as we add them.

Added:

• support for reading multiple .shp files from a single .zip
• support glob matching within a .zip file

There's a breaking change in naming: addShapefileDirectorySource becomes addShapefileGlobSource. I figured addShapefileDirectorySource was new enough that the more accurate name was worth the churn, as the pattern can now be applied both to ZIP archives and directories.

Another change in behavior: previously, if you had a .zip with two .shp files, only one would be read. Now (unless a glob pattern filters one out), both will be read.

My workflow is usually to code in visual studio code without much java support. Then I go to the terminal, compile and run the application.

Maybe this is a bit strange if you are used to working with intelij or another good ide, but this is how I usually code...

So for me it is super helpful if I can run the configurable schema standalone, like I can run the things in the examples folder standalone...

Let me know if this is the right approach.

Exposes the Feature::setMinPixelSize function in the configurable schema as min_pixel_size.

This follows the implementation of min_zoom.

Related to https://github.com/onthegomap/planetiler/issues/349

Unblocks ZeLonewolf/openstreetmap-americana#618 CC @zstadler

Internationalized name:xx names are missing from roads in the transportation_name layer.

The following object show internationalized names in omt-tools:

Same location in my planetiler server (link)

How can I run the shortbread.yml configurable schema on the full planet?

When I do something like --area planet, I get an error that geofabrik does now know this area.

Is hardcoding the planet download url like this the best solution?

   osm:
     type: osm
-    url: '${ args.osm_url }'
+    url: https://planet.openstreetmap.org/pbf/planet-latest.osm.pbf