Archive for the 'Cartography' Category

Published by breki on 12 Mar 2011

Maperitive: Plans For The Second Year

Fantasia
Creative Commons License photo credit: spitfirelas

Today Maperitive celebrates its first birthday! It has been a productive year and Maperitive has learnt a lot of new stuff.

This post is dedicated to my wishes and plans for features I want Maperitive to learn in its next year. Some of these features I planned even before I started working on the software, while others are more fresh and indicate a shift in focus. My initial goal was to concentrate more on the GUI and less on the features, but as time passed the GUI became less and less important. This is mostly due to my decision to implement scripting support, which made a lot of things easier to implement through command-line than by writing GUI code.

This doesn’t mean the GUI will be totally forgotten. It does however mean I will prioritize other things over it. My general goal is for Maperitive to become a tool for making high-quality maps with emphasis on scripting and automation.

So the following subsections describe some major things I want to see in Maperitive in the near future.

Better OSM Relations Support

This is something I’ve already started working on. The idea is to provide a better way to work with OSM relations. I’m still in the brainstorming phase on this one, but I can give a some examples of things I want users to be able to do:

  • If an OSM way is shared by two or more relations, I want the way to be treated as a single entity. Think way which represents both a country and a municipal boundary. I want only the country boundary to be shown in this case.
  • Aggregate OSM tags for two or more relations sharing the same way. Think 5 bicycle route names drawn in a single box for such a way.

Extensibility

I’ve started playing with IronPython as a way to allow users to specify custom code for various parts of the rendering pipeline. Some examples pop up in my mind:

  • Custom painting of the map (after the map elements have been rendered). This would allow drawing user-defined grids, labels, logos, legends etc.
  • Individual map element customization, like setting the color of the line based on value of some OSM tag. Or setting the width of the line based on how many bicycle routes cross a way.

I already did some testing with IronPython and it looks very promising.

Improved Scripting

IronPython could also be used to make the existing Maperitive scripting much more powerful. The current script language (if you can call it like that) is just too limiting – no variables, no loops, no branches etc.

I also plan to add a pure console application for headless scripting, so you won’t need to look at the GUI when running scripts.

Automatic Label Placement

This is the Holy Grail of cartography – having map labels arrange themselves in a nice looking way. I plan to devote a lot of time on this one.

Label Abbreviations

Closely related to label placement, I’ve actually implemented some parts of this feature. But before releasing it to the public, there is a lot more to be done.

Other Map Projections

It’s time to cut the ropes Mercator has tied us with. Mercator projection is nice for Web maps, but there is a lot of other interesting map projections that can be used to show OpenStreetMap data. Implementing this won’t be as simple as it sounds – Mercator projection has some pretty nice properties which make rendering on screen much easier than some other projections. And this is important for real-time renderer like Maperitive. But I guess it can be done.

Data Mining

Planet.osm is getting bigger and bigger and increasingly difficult to consume. I want to create a toolset for extracting data from large OSM files. Yes, I know there’s osmosis, but I want to have a good integration of data mining with Maperitive. And this is another point of entry for Python: writing custom filter code. Still in brainstorming phase.

Using Databases

I’ve played with spatialite and PostGIS back in 2009 and I managed to do some integration with Kosmos. I expect this to be one of the harder things to implement. I want to avoid hacky approach Mapnik uses for storing OSM data in a database – my wish is to keep the data structure in the original form.

Conclusion

That’s it, really. I don’t like writing long posts. I guess you’ll have to be patient and some of the things mentioned above will start to trickle in. Have a good day!

Published by breki on 25 Feb 2011

Maperitive Build 1138

A new build is here! The major new thing is support for SRTM1 and custom digital elevation models (DEMS). Well, when I say “custom”, I actually mean anything that’s compatible with SRTM *.hgt files. You can read more about this in Maperitive’s online book.

I wanted to do a little test of the new functionality so I decided to choose a small area of Alps as my testing ground. I chose Alps because there’s a good DEM source for Alps, which I wrote about some time ago: Viewfinder’s DEM so I can compare it with standard SRTM3 data.

The results are stunning. Here’s a sample hiking map of Lake Brienz using standard SRTM3 DEM (notice the “white” spots which are missing elevation data):

Brienzersee hiking map using SRTM3 DEM

And now for something completely different, a man with three buttocks:

Brienzersee hiking map using Viewfinder's Alps SRTM1 DEM

I especially like the slopes shading north of the lake.

Published by breki on 05 Jan 2011

Maperitive Build 1094

Maperitive Hiking Map Sample

The first Maperitive release of 2011 is out! Download link, as always: http://maperitive.net/download/

There are many new goodies inside, including:

  • Commands for FTP uploading and zipping files.
  • Pipelining generated files from commands like generate-tile and export-bitmap to the above commands.
  • generate-tiles command now has the ability to detect whether tile contents have changed since the last run (using tile fingerprinting). This way only the actually modified tiles can be uploaded to an FTP server, saving you a lot of time and bandwidth.
  • Scripts now have the ability to reference external files using relative paths.
  • Icons can now be placed on lines and rotated in the same way shapes can.
  • I added two new keyboard shortcuts, one for focusing on the map (Ctrl+M) and the other focusing on the command prompt (Ctrl+Enter).
  • Various SVG export bug fixes and improvements. SVG paths are now generated in a more optimal fashion, reducing the size of the generated SVG file even further.
  • Various other bug fixes.

Scripting Web Maps Generation

In the last couple of weeks I’ve been working on my own hiking Web map. This is sort of me dogfooding of Maperitive. In the process I’ve fixed a number of bugs and added the above mentioned commands for automating the process of creating and maintaining Web maps. To see what I’m talking about, here’s a sample script which generates a web map using my (soon to be published) hiking rendering rules and uploads it to an FTP server:

use-script-dir
clear-map
use-ruleset location=hiking.txt
load-source Stajerska.osm.bz2
load-source Stajerska.ibf
load-image file=Stajerska-hillshading.png background=false
set-bounds 15.14,46.39,15.92,46.
generate-tiles minzoom=11 maxzoom=15 use-fprint=true
ftp-upload host=myftp.com user=me pwd=secret remote-dir=hikingmap/tiles

That’s it! In the current version the script must be run inside the Maperitive GUI, but I plan to add a pure headless console for these kinds of tasks.

What’s Next

I have a long list of features waiting to be implemented, and the list doesn’t seem to get any shorter with time. But the main focus will be on even better automatic scripting support and improved map rendering quality.

Published by breki on 19 Nov 2010

Maperitive vs. Adobe Illustrator

Maperitive -> SVG -> Adobe Illustrator

It’s been a hard fight, but I’ve finally worked out most (all?) of Adobe Illustrator’s quirks and bugs in SVG importing and ways to go around them. I can now officially say that Adobe’s support for SVG is lousy (so much for their professed commitment to open standards). I even managed to export SVGs from Illustrator which then could not be imported back into it (“Can’t open the illustration”).

Anyway, SVGs now look pretty nice in Illustrator, but there is a price to pay: they need to generated in a different way than for Inkscape, so there is a new setting available in the export-svg command. They certainly look better than SVGs produced by the Export tab on the OSM.org’s map site and they are structured in a more usable way (better layering and reuse of shapes, use of actual text lettering instead of graphic paths etc).

Here’s a sample SVG map of the Dublin’s center, so you can take a look (warning 1: do not try to open the file in a browser, it is a compressed SVG (SVGZ) file which only Illustrator and Inkscape know how to handle, warning 2: although the map file is not very large, it may take a while for Illustrator to open and show it).

Expect this feature to be available in the next Maperitive release (within days).

Published by breki on 20 Oct 2010

OpenStreetMap: What’s Wrong With The Picture

Justin O’Beirne wrote a couple of blog posts about his “outside” view on OpenStreetMap Web maps (Mapnik layer, basically). When I say “outside” view, I mean that he talks about how and what data is presented on the map from the point of a visitor to the OSM site, and not from the point of someone who is an active OSM mapper and knows the root causes of these problems.

And this is just the point: a well intentioned criticism from someone outside of the community should be received as such. I think the OSM community is more and more becoming self-centered and disregards some basic issues about why this project exists in the first place. Why should Justin (or anybody else) care what “Mapnik” or “Osmarender” means? Why should he care about the tagging mess which resulted from the anarchical way the project is (not) being led? Not everyone wants to become a mapper – most of people just want to find something on the map.

Reading through various OSM mailing lists and forums, one gets the feeling there is very little concern about how the data that is being collected by hardworking individuals will be useful in a practical way. I see two main problems here:

  • Inconsistency of how things are tagged. And the project’s inability to set some strict quality guidelines for tagging. The “everybody can tag the way she likes” slogan starts to wear off once you want to use such data for something more than just displaying it on the OSM web map.
  • High barrier to entry if you want to access the data. Sorry, but not everyone has the technical means and knowledge to import 12 GB of zipped planet.osm XML file into a database and then run queries just so he can access the latest data for his local area. OSMXAPI is great, but it’s unstable and has a limit of how much data can be retrieved. Country extracts help, but the problem is that they are country-oriented. What if I do not want my data to be cut along the border? Some time ago I suggested providing grid extracts instead of country ones – the user would choose which grid cells to download and then merge the data himself.

Anyway, enough ranting… Going back to coding.

Published by breki on 22 May 2010

Maperitive’s New Text Formatting Features

Maperitive Text Formatting

I’m in the middle of implementing new features for formatting text in Maperitive. The above map shows just some of them:

  • Word wrapping: a new text-max-width property specifies the maximum width of the text. Everything else will be moved to a new line.
  • Alignment: text-align-horizontal and text-align-vertical are the new properties for this.
  • Offsetting: you will be able to move the text horizontally (text-offset-horizontal) or vertically (text-offset-vertical) from the original point. This enables showing icons for peaks and their names pushed below to prevent overlapping.
  • Some other text-related stuff I haven’t yet implemented, but let’s leave this as a surprise.

The new version will probably be released sooner than planned because I decided to leave the SVG exporting work for another release.

Published by breki on 22 Apr 2010

Kerning Or How To Outfox GDI+

500px-Metal_type_kerning.svg[1] (image taken from Wikipedia)

I’m working on a text-on-path algorithm for Maperitive to be able to render things like street names. GDI+ does not have a built-in function for drawing texts on a path, so I’m stuck with writing my own.

Although there are a lot of implementations of such an algorithm out there, they all miss one important thing: kerning. Kerning is basically telling a character to move a bit closer to its neighbor if there is enough space for them to huddle together:

Kerning[1] (image taken from Wikipedia)

Since the path can consist of several line segments (and even curves), you need to render each character (glyph to be more precise) separately to be able to rotate it using the angle of the line segment it will be drawn on. The problem is determining how far from the previous character to place the next character. GDI+ offers a method called MeasureString, but calling it on a single isolated character returns a character width value without any kerning taken into account, so characters like V and A can appear too spread apart. There is another method, MeasureCharacterRanges, which can measure each individual character width for a given text, but it has a practical limit of 32 characters and is pretty processor- and resource-intensive. I’ve used it for the text-on-path algorithm for Kosmos but the results weren’t very satisfactory.

From my (basic) understanding of how kerning usually works on computer typography, a font which has kerning information stores kerning offsets for character pairs. So, for example, V and A will have a different (usually stronger) kerning than say, character pair V and T. The problem with GDI+ is that you don’t really have access to this information.

One option would be to use old GDI’s font and text functions to fetch the information about the kerning pairs, but these need to be P/invoked and thus will probably not work on Mono, which makes them unusable for Maperitive – I’m really trying to make Maperitive run on multiple platforms.

So after spending a day in brainstorming, I got an idea on how to handle kerning myself. It’s really simple: I will calculate the kerning myself (using the mentioned MeasureString method). The formula is simple:

<character kerned width> = <character non-kerned width> – (<character pair non-kerned width> – <character pair kerned width>)

I can get all of the three values on the right of the equation using the MeasureString: first I’ll measure each character’s individual (non-kerned) width and then I’ll measure the width when they are drawn together. The difference between these is the actual kerning, which is then used to calculate the kerned width of the first character.

The added twist to this tale is that I’ll keep this information in a cache, to keep the expensive calls to the MeasureString method to the minimum. This cache could even be persisted to the disk. Of course, each font family and font style dictates its own kerning, so the cache needs to be extended with this information.

One potential problem is how to handle kerning information for different font heights. Since Maperitive supports continuous zooming levels, the number of different font heights used is (almost) limitless. I’ve decided to cut some corners here: the kerning will be calculated (and stored) for some nominal font height (say 100 em) and then interpolated for any needed height. We’ll see how that goes… next time.

Published by breki on 28 Nov 2009

Kosmos: Preview Of SVG Support

Some time ago I wrote about experimenting with Cairo drawing engine to render maps. The basic idea was to use Cairo to render SVG map files which can then be manually post-processed using some vector graphics editor like Inkscape.

Unfortunately I worked on lot of other stuff after that, so the experiment was put on hold. But a few weeks ago I started working on a new rendering rules engine for Kosmos and I wanted to write some unit tests to try it out – and the experimental map drawing code seemed a good fit.

The rules engine is far from complete, but I can already reveal a few details of its capabilities. First, you specify a set of features, for example a forest, a motorway, a town. In accordance to some common GIS concepts, there are three types of features: point, line and area feature.

Then you specify how each of these features should be rendered. This consists of a directed graph of commands which supports inheritance of rendering properties (something in the direction of MapCSS, but not as liberal).

I won’t bore you with the details (at least until a proper parser for these rendering rules is available), but I think this new system will be much more powerful than the existing “flat” Wiki table-based rules one. Also, I’ve been working on improving support for relations and multipolygons.

Anyway: back to Cairo: I’ve started implementing more detailed drawing logic and I had quite a few problems figuring out how to render text using Cairo. It turns out the API for drawing text in Cairo is pretty poor (it even crashes when trying to render non-ASCII chars), so the authors of Cairo recommend using Pango, a library for high-quality text rendering. Luckily, it has Mono wrappers and it wasn’t too difficult to make it write out texts on the bitmap.

I’ve written few unit tests for some simple OSM map rendering and here are the results (the left map is rendered using GDI+, the right one using Cairo):

map2gdimap2cairo

I’ve used some simple rendering rules:

  • texts come from OSM tags equipped with “name” tag
  • lines come from any non-closed OSM ways
  • orange-filled areas come from any areas (closed OSM ways).

As a special treat, I’ve also created a SVG file from the same map, you can download it from here: http://downloads.igorbrejc.net/osm/misc/map1.svg

Published by breki on 03 Nov 2009

DSL For Rendering Rules

Doodle
Creative Commons License photo credit: The Pack

Yesterday I started thinking about a new rendering rules system for Kosmos. The existing system (using Wiki tables) was OK for a while, but it’s starting to show its old age: there is no rule inheritance, the rules are cumbersome to edit (because of wikitext), the selectors are not very powerful, rules are not reusable etc.

I started by investigating how similar software is doing these things. Mapnik has a more elaborate XML-based ruling system. It’s powerful, but it seems too verbose for what I had in mind – I want the rules to be simple to write and even simpler to read. And I want to stay away from XML: XML is easy to parse but not that friendly to humans. And writing expressions using XML character entities (example: “<Filter>[CARTO] &gt;= 2 and [CARTO] &lt; 5</Filter>”) is really unfriendly and error-prone.

The next thing to look at was MapCSS, a CSS-like language for map stylesheets and is a brainchild of Richard Fairhurst, the author of Potlatch, the excellent web-based OSM editor. I like the idea, but I’m worried how it could be implemented in the context of a database-driven map engine. The databases can be huge and you want to minimize (and optimize) the queries run on them, but MapCSS seems to allow too many combinations to make it usable for generating maps in (almost) real-time, and that’s what Kosmos is about. When working efficiently with the database, the rendering engine wants to collect all the related objects in one go, and then decide how to render them. So, for example, if it wants all highways to be rendered red, it’ll fetch all the highways from the database (with a single query) and then go and paint them all red. MapCSS doesn’t really allow this: you need to process each OSM object on its own and go through the list/tree of MapCSS rules in order to know whether (and how) to render it. Maybe I’m wrong and there is a better way to approach this, I’ll have to discuss this further with Richard and see what he thinks.

So now I’m thinking about defining my own domain-specific language (DSL) which would be used to specify rendering rules. I don’t have any specifics yet, but there are some things worth thinking about:

  • The DSL should not be XML-based: angular brackets are not for humans.
  • Separating definition of features from the definition of styles: features tell you what something represents. A forest, for example, is a feature that corresponds to an area tagged with natural=wood or landuse=forest. How it is rendered depends on the type of the map: on topo maps you would render it in green color. On driving maps maybe you wouldn’t render it at all. So the idea is to have a features definitions that could be stored in one place (one wiki page) and then reused all across the styling rules. Right now this is not possible in Kosmos.
  • Expressions: the selector expressions should be more powerful than just simple logical operators. The same goes for expressions for style attributes.

Anyway, this all means I have to implement some kind of a parser for the DSL. So I started looking into possible C# libraries/tools for writing parsers: ANTLR, Coco/R, GOLD, Irony.NET, … There is also a good article called Grammars and parsing with C# 2.0. Time to do some refreshing of knowledge lost way back from my university days…

Published by breki on 23 Oct 2009

Kosmos And Spatial Databases, Part 1

Kosmos Label Placement Sample

Last couple of months I’ve been working on database support for Kosmos. This work started as a request to produce bitmaps of the whole UK in the British national grid reference system from the OSM data.

The current version of Kosmos loads all the OSM data into memory. This obviously proves to be a problem for larger areas. Some time ago I worked on optimizing Kosmos memory usage and it proved to be quite successful, but there is still a physical limit of how large an area can be rendered.

The latest UK OSM data takes about 200 MB of zipped XML (2.5 GB unzipped). This is way too much for the existing in-memory system, so I had to find another way to render UK without reading the XML file directly.

After a bit of investigation, the obvious candidate for OSM data storage proved to be PostgreSQL / PostGIS. By “obvious” I mean that there were already some tools which could import OSM data into the PostgreSQL database, namely Osmosis. I was totally new to PostgreSQL, but the actual installation of the database engine was pretty easy.

The next step was importing of OSM data using Osmosis. Although this can be achieved with a few simple steps, there were quite a few “tricks” I had to learn the hard way before the performance of the database was satisfactory enough to produce map renderings in a realistic time:

  • The PostgreSQL database engine uses non-optimal default settings, so you need to do a bit of investigation to set certain things up. One tip: if you have several disks, make sure your data is stored on a disk separate from the system.
  • Osmosis OSM DB import command has a few settings on its own which can have a dramatic effect on the DB query performance. First of all, you need to use the extended OSM DB schema which contains bbox column for ways, otherwise the spatial queries would always have to contain multiple table joins which will terribly slow things down.
  • Kosmos rendering rule engine was designed for instant-access in-memory data source. The way the data was fetched wasn’t really compatible with slow DB sources, so I had to do a lot of refactoring on the engine before the performance was good enough for any serious work.

After a few weeks of work, I finally managed to get some decent results. The rendering code generated UK for approximate zoom levels 7, 8 and 9 in about two hours on my machine, with the level of detail similar to Mapnik’s layer of the OSM main map. This may seem a lot of time, but considering there were about 11 million nodes and 1.5 million ways to process, I’m quite pleased. And there are further improvements still possible in the rendering engine which could reduce this time at least by half.

British National Grid

One of the biggest worries I had was how the rendering engine will behave when rendering using a map projection and reference system different from the “standard” Mercator and WGS-84. This actually proved to be the easiest thing to solve: the rendering engine internally uses the National grid coordinates and the spatial DB queries transform those transparently to WGS-84 (and back). I also needed to implement a new map projection, but this was just a matter of writing a few lines of math code.

Other Improvements

Since the map was supposed to be “professional” looking, there were a few things that needed to be implemented or improved on the existing rendering engine. These features will probably be included in the next generation of Kosmos, which will be released some time next year.

Text Placement

One of the first (and hardest) things needed for a good looking map is to make sure the text labels do not overlap each other. This is called (automatic) label placement. Good algorithms for label placement are quite complex and I didn’t really have time to implement a full-blown algorithm, so I chose to do a simple point-selection: the algorithm removes labels for smaller places (towns) until there are no more overlaps (see the sample map at the beginning of the post). I’ll write more about this feature some other time.

Better Relations Support

Some of the features needed to be rendered are now defined in OSM using relations. One example of this is national parks. So a new algorithm for consolidating relation’s ways into a single polygon was implemented.

Better Sea Filling Support

The coastline processing algorithm is now more resilient: it will ignore poorly connected coastlines and will still render all of those which are properly defined. This improvement is already in use in the new GroundTruth version released a week ago.

What Next?

During the work on PostgreSQL support I learned about SpatiaLite, a spatial extension to sqlite. As you may know, sqlite is a popular self-contained database engine which stores the data into a single file. So I decided to do a bit of playing around with it and see if it can also be used in Kosmos. I was a bit skeptical whether SpatiaLite will be fast enough for map rendering, but I can already say that it managed to achieve some very good results. I’ll write about SpatiaLite in the next blog post.