igorbrejc.net

This weekend I was playing around with Kosmos by trying to load large maps (by “large” I mean country-size large). I have never really done this before, since I was more concentrated on smaller areas when I developed the software.

The initial results weren’t really encouraging – Kosmos managed to load the OSM extract of Austria (25 MB of .osm.bz2 file), but the GUI was burning around 1 GB or RAM and it took some time to load. Then I got into investigating how this could be improved.

I managed to do a few things which greatly improved the performance:

• OSM tags were stored as C# generic Dictionaries (for each OSM element). This is a problem, since typically there are no more than few tags for an element. It turns out the ListDictionary class is better suited for small dictionaries, since it uses linked lists instead of hash tables.
• A lot of memory was being taken by “created_by” tags for each OSM node. Since this tag isn’t really much used when rendering maps, I decided to ignore it when loading OSM files (OSMXAPI ignores it too, by the way). This can save a lot of space, since most nodes do not have any other tags – which means the tags dictionary doesn’t even have to be created for such a node.
• Immediately after loading the map, the garbage collector is now called to dispose of any temporary objects. I know, GC isn’t supposed to be called manually by applications, but I guess in these extreme circumstances it can be forgiven – especially since it frees up several MB of RAM immediately.
• The last thing to do was to write a custom OSM XML loader. Until now I used XmlSerializer classes autogenerated from OSM XSD. This has several disadvantages: you have to load the whole OSM file into memory in order to (de)serialize it and also produces a large quantity of intermediate objects which then have to be translated into “proper” OSM objects. But by writing a custom loader which reads XML sequentially (using XmlReader) both of these obstacles can be overcome. Still, writing XML parsing using XmlReader isn’t easy (in fact is real pain in the you know where).

Anyway, after these modifications Kosmos loads Austria data more quickly and with much less memory footprint (around 400 MB). It’s still a slow experience on my machine (E6400, 4 GB RAM), but at least it’s improvement.

I tried to load progressively larger country extracts – the largest one that did load so far without throwing the OutOfMemoryException was France (54 MB). The screenshot above shows Czech Republic (43 MB).

NOTE: by doing this I wasn’t really aiming to achieve some kind of “planet coverage” with Kosmos – it’s virtually impossible for Kosmos GUI to draw the whole planet interactively. But I would like for non-interactive Kosmos Console at least to be able to generate tiles for larger areas (UK or Germany, for example, or even the whole of Europe) in some reasonable time – but there’s a lot of things to do before this is possible. I guess the first steps have been made…

I’ll be upgrading my blog to the latest WordPress version this weekend, so there will probably be some glitches (the download site should work though). Sorry for any inconvenience.

UPDATE: the migration was easier than I expected – hopefully no hidden problems will appear later. If you encounter anything strange, please send me a note. Thanks!

Yes, Kosmos has reached a new version (download the latest package from here). It contains some cool new features, but I have spent most of the work on improving the user experience. But let’s start with new features:

Mobile GMaps support

I’ve already blogged about this. In short, you can now use Kosmos-generated maps on with a Java mobile application called Moble GMaps (MGMaps). The idea is to generate map tiles from your Kosmos project, copy them to your phone and tell MGMaps to use them. Since the map is stored on the phone, you don’t need to use the mobile network or pay anything to view it. I’ll soon write instructions on how to set up the whole thing on the Kosmos OSM Wiki pages. UPDATE: the instructions are here.

Error list window

When loading projects, parsing rendering rules etc. Kosmos GUI will now display all the errors and warnings in a special “error list” window. This is combined with much improved handling of various errors in rendering rules – instead of causing Kosmos to crash, you should now be able to see all the detected errors in this window. In case some a parameter has an invalid value, the rendering rules engine will now use a default value – typical example is entering an invalid color value.

Handling of rendering rules errors has been a longstanding issue with many users – hopefully this has been improved now, but there will probably be some bugs, so please report if you encounter any.

Changes in the Kosmos console

Kosmos console has a new command called “zappy” which allows you to download OSM data from the OSMXAPI server. This has been already available in Kosmos GUI, but I wanted to extend it to command line support since it can be used for automating certain tasks. A simple example: you can now periodically download new OSM data for a certain area and generate fresh tiles from this data – so your tile server will offer up-to-date maps.

By the way, the command line options for generating and serving tiles have been changed. Instead of having to manually edit parameters in the Kosmos project file, you can now specify everything you need from the command line. Also, these parameters will be stored in a special file together with generated tiles, so when serving tiles, you only need to specify the location of the tiles cache. I’ll write more about this on the OSM Wiki later. UPDATE: the instructions have been updated.

Exporting georeferenced bitmaps with world files

All bitmaps generated in Kosmos (except the tiles) are now equipped with a world file, which should allow users to use these bitmaps in other GIS tools. However, this is only a partial solution – you still cannot import bitmaps into Kosmos and also GeoTIFFs are not supported. I will implement importing of bitmaps with world files in later versions, but only if they use Transverse Mercator projection. Supporting other types of projections (by using image rectification) is not planned in the near future since it’s beyond the scope of this application (read: I just don’t have enough time to do it).

NOTE/WARNING: you will not be able to open bitmaps from projects created with previous versions of Kosmos, since the new version does not use the georef information embedded in the project file – it now uses world files for this. Unfortunately backward compatibility is one of the things I chose to sacrifice due to lack of time – mostly because I think all the bitmaps generated in Kosmos can be regenerated without much problems.

Other new stuff in brief

• Generating relief contours in GUI is now customizable – you can specify the elevation interval. I also wanted to add customization of how the contours are rendered, but this will have to wait for the next release.
• Kosmos can now read GPX v1.0 files
• Entering of direct OSM wiki URLs for rendering rules (like http://wiki.openstreetmap.org/index.php/Kosmos_General_Purpose_Rules) is now allowed.
• Data files that cannot be loaded in the GUI project are now greyed-out.
• The same way can now appear in multiple multipolygons without causing Kosmos to crash.
• Various bug fixes.

That’s it – the next task is to update the Kosmos documentation in the OSM Wiki. After that, I’ll try to get back on Kosmos 3D stuff which I neglected for the past few months.

I hope you enjoy the new version and please report any bugs you discover (there’s bound to be a few of them ).

The work on the new version of Kosmos is in full speed. I planned to release it in August, but I had too little spare time for it – most of my weekends were spent hiking and mapping and was also pretty busy on my “real” work.

Anyway, I updated the “Kosmos Future” page on OSM wiki pages with the latest status. What I am really excited about is the new feature – showing Kosmos-generated maps on mobile phones using MGMaps, MGMaps is a free Java ME application for showing Web maps from various sources (OpenStreetMap, Yahoo, MSN and more). The default behavior is to download map tiles on demand, but the user can also download them manually and store them in the phone’s memory.

This “stored maps” feature of MGMaps gave me the idea of providing a way to show Kosmos maps with MGMaps. I added support for generating MGMaps-style map tiles in Kosmos console – you simply generate the tiles for a specified Kosmos project and then copy them to your phone. The results can be seen here (NOTE: the “Osmarender” text on the bottom of the screen is shown because I had to mark my tiles with one of the map types supported by MGMaps, and I chose Osmarender).

Apart from the fact that using this method you do not overload OSM servers by mass-downloading tiles, it also provides a way to have maps customized to your needs on your phone to help you with the navigation (MGMaps can connect to a GPS receiver). I’m already thinking about using this solution when hiking instead of Garmin + mkgmaps combination. Note however that MGMaps work for Java ME-enabled phones only.