igorbrejc.net

photo credit: Omar Eduardo

My previous post about PBF reading successes was written way too prematurely. It turned out my PBF reading code had some serious bugs which made reading look much faster than it actually was (one of the reasons was that I neglected to read OSM node keys/values when written in PBF dense node format).

I’ve subsequently written some extensive tests, comparing OSM database contents from XML and PBF file of the same area (thanks Geofabrik) on an object by object basis, so I’m now 95% sure the PBF code works OK. Performance-wise the (final?) results are much less glamorous than it looked initially: PBF reading is “only” 2.5 times faster than reading OSM.bz2 files, while in memory consumption terms, they are pretty much the same. I curious what other OSM software like osmosis has to say about these results.

I had hoped I could speed the PBF reading by spreading the work on several processor cores. What I did is to use Microsoft’s Parallel Extensions library to separate the fetching of PBF file blocks from the actual parsing of them into two (or more) cores. This resulted in only about 10% increase of the overall speed (tested on my two-core machine, so on more cores the result could be better).

It actually proved pretty hard to do a decent job of separating work in some balanced fashion. Since the file reading is sequential, this can only be done by one thread/core, so you want to put as little other work to that core as possible. As soon as file block bytes are fetched from the file, they are delegated to another core to parse it (in terms of protocol buffers) and then extract OSM objects from it. The problem is that you don’t want to enqueue too many file blocks at the same time, since this takes up valuable memory (which is already filled with extracted OSM objects). So I ended up using a blocking queue, which means the main thread (which reads the file) will wait until at least one core is available before filling the queue with another file block.

I’ve also tried micro-management strategy – using multiple cores to extract individual OSM objects, but this only really works for ways and relations. Current PBF extracts use dense nodes format, which is delta-encoded and thus forces you to read things sequentially on a single thread of execution. I guess this is the price of having a format that wants to satisfy two different (and inherently conflicting) goals: less space and less CPU.

I’m fairly new to Parallel Extensions and there are probably better ways of handling this, but I’ll leave it for the future.

Anyway, a new Maperitive release is out, grab it from the usual place.

UPDATE: the post below was based on premature assumptions that my new PBF code is actually working. It turns out it had a number of serious bugs which made reading look faster than it actually is. Here’s a followup post.

For the last couple of days I’ve been working on a PBF file reader for Maperitive. PBF file is a binary file for storing OSM geo data using Google’s protocol buffers.

It’s been a steep learning curve, since I had to learn three things at the same time: protocol buffers, using protobuf-net library for .NET and understanding the PBF format. I’m mostly satisfied with the protobuf-net library, although the lack of any new development activity worries me a little bit.

I’ve finished most of the PBF reading stuff this evening and I was eager to test the new code against the old XML reader. I’ve used Geofabrik’s Denmark data, here are some rough results:

• PBF file loads 7.6 times quicker than the .OSM.bz2 file. This is a really good result, mostly thanks to the way the PBF format has been designed.
• Loading of PBF data uses a quarter less memory than the XML file. I’m talking about the memory used in the process of loading, not for storing the loaded OSM data – the data is internally stored in the exactly same way both for PBF and XML reading. This result surprised me a bit, I guess the extra memory consumed by the XML reader is due to the XML parser itself and/or the fact that a lot more strings are generated when reading XML OSM tags. PBF uses string tables and thus saves a lot of space by reusing common strings.

The first Maperitive release of 2011 is out! Download link, as always: http://maperitive.net/download/

There are many new goodies inside, including:

• Commands for FTP uploading and zipping files.
• Pipelining generated files from commands like generate-tile and export-bitmap to the above commands.
• generate-tiles command now has the ability to detect whether tile contents have changed since the last run (using tile fingerprinting). This way only the actually modified tiles can be uploaded to an FTP server, saving you a lot of time and bandwidth.
• Scripts now have the ability to reference external files using relative paths.
• Icons can now be placed on lines and rotated in the same way shapes can.
• I added two new keyboard shortcuts, one for focusing on the map (Ctrl+M) and the other focusing on the command prompt (Ctrl+Enter).
• Various SVG export bug fixes and improvements. SVG paths are now generated in a more optimal fashion, reducing the size of the generated SVG file even further.
• Various other bug fixes.

Scripting Web Maps Generation

In the last couple of weeks I’ve been working on my own hiking Web map. This is sort of me dogfooding of Maperitive. In the process I’ve fixed a number of bugs and added the above mentioned commands for automating the process of creating and maintaining Web maps. To see what I’m talking about, here’s a sample script which generates a web map using my (soon to be published) hiking rendering rules and uploads it to an FTP server:

use-script-dir
clear-map
use-ruleset location=hiking.txt
load-source Stajerska.osm.bz2
load-source Stajerska.ibf
load-image file=Stajerska-hillshading.png background=false
set-bounds 15.14,46.39,15.92,46.
generate-tiles minzoom=11 maxzoom=15 use-fprint=true
ftp-upload host=myftp.com user=me pwd=secret remote-dir=hikingmap/tiles

That’s it! In the current version the script must be run inside the Maperitive GUI, but I plan to add a pure headless console for these kinds of tasks.

What’s Next

I have a long list of features waiting to be implemented, and the list doesn’t seem to get any shorter with time. But the main focus will be on even better automatic scripting support and improved map rendering quality.