igorbrejc.net

Although I haven’t been posting much on Kosmos, this doesn’t mean that nothing is happening. In fact, a lot of new stuff is in development. so much so that I’m having trouble concentrating on finishing stuff .

There are main fronts I’ve been battling since I’ve decided to do a big Kosmos rework: the GUI and the rendering engine. Although I started the rework by concentrating on the GUI code, for various reasons the effort moved to rewriting the rendering engine, especially the rules system.

The problem with this is that now there is a lot of “todo” code on all fronts and it left me wondering what the realistic date for releasing Kosmos v3 would be. Even when the rendering engine has been completed, there would still be a huge amount of work needed to finish the GUI framework.

During a hiking trip a few days ago I started thinking how this could be speeded up. I got the idea that instead of waiting for full-fledged GUI to be finished it would be better to provide a simplified user interface as an intermediate / temporary solution (going back to the old v2 GUI code was out of the question).

So my idea was to provide a set of commands which could be executed by the user through a command window. I know, I know, this is a bit geeky and not too friendly, but I see several good points with this approach:

• The time for releasing the first (“alpha”) version of Kosmos v3 would be shortened by at least two to three months.
• Users will be able to test the two main improvements: the new rendering rules system and the improved graphics engine.
• The idea for the commands is to make them automatically available through menus, so it wouldn’t be a so user-unfriendly after all.
• My plan is to release Kosmos more often than before (at least once a week), so the users would experience continuous improvements to it. Later I would try to implement some kind of auto-updater.

Now let’s see what are the new features I’ve been working on.

Rules Engine

I’ve been writing about this before. The new engine has now been more or less implemented, but I still have to write the parser for it. Also, I want to be able to support the old Wiki-tables rules format (obviously), so there’s some work there before the release. All I can say right now is that the new system will be much more powerful regarding selecting elements to be drawn and much easier to control, without unnecessary repetition when writing rules.

Map Rendering

The idea was to rewrite the graphics code to be able to support different outputs (like SVG) and I think I’ve made a good start. There’s still a lot of work (now there are two separate code bases for GDI+ and for Cairo). Also, the rendering will probably not be as quick as in Kosmos v2, but this is because I plan to work on graphics optimization later.

Scripting

This is a result of the decision to provide a command-line interface inside the Kosmos GUI. The natural next step to this is writing a sequence of such commands. This could then become a powerful automation tool and could also replace the existing Kosmos.Console application. I’m still thinking how best to provide such a scripting functionality. One idea is to have an interpreter of some common scripting language (like Ruby, Python or Lua), but I don’t want to go over the top with this. And I don’t want to open a new front I wouldn’t be able to fight

Spatial Databases

I introduced this topic a few months ago. It is very exciting to see Kosmos working with PostGIS and SpatiaLite, although this feature will not be released very soon – I still think Kosmos is currently most useful when working with smaller map areas which can be loaded from an XML file. Once the general scenarios have been covered, I’ll try to take some time to finish the DB support.

Misc

There’s a lot of other stuff I’d like to see in Kosmos, like better GPX support. The rendering rules will support specifying how to draw GPX features, too. Also, there will be functions like calculating the area size (and using this to filter things when rendering). I would also like to have routing. And editing GPS data (and even an OSM editor). But I guess these things will come later, after v3 has been released and stabilized. Too many ideas and too little time, that’s the problem….

Preview

For those of you who endured to the end of this post, I’m attaching a screenshot of the prototype of what I call “Kosmos Commander”:

Some time ago I wrote about experimenting with Cairo drawing engine to render maps. The basic idea was to use Cairo to render SVG map files which can then be manually post-processed using some vector graphics editor like Inkscape.

Unfortunately I worked on lot of other stuff after that, so the experiment was put on hold. But a few weeks ago I started working on a new rendering rules engine for Kosmos and I wanted to write some unit tests to try it out – and the experimental map drawing code seemed a good fit.

The rules engine is far from complete, but I can already reveal a few details of its capabilities. First, you specify a set of features, for example a forest, a motorway, a town. In accordance to some common GIS concepts, there are three types of features: point, line and area feature.

Then you specify how each of these features should be rendered. This consists of a directed graph of commands which supports inheritance of rendering properties (something in the direction of MapCSS, but not as liberal).

I won’t bore you with the details (at least until a proper parser for these rendering rules is available), but I think this new system will be much more powerful than the existing “flat” Wiki table-based rules one. Also, I’ve been working on improving support for relations and multipolygons.

Anyway: back to Cairo: I’ve started implementing more detailed drawing logic and I had quite a few problems figuring out how to render text using Cairo. It turns out the API for drawing text in Cairo is pretty poor (it even crashes when trying to render non-ASCII chars), so the authors of Cairo recommend using Pango, a library for high-quality text rendering. Luckily, it has Mono wrappers and it wasn’t too difficult to make it write out texts on the bitmap.

I’ve written few unit tests for some simple OSM map rendering and here are the results (the left map is rendered using GDI+, the right one using Cairo):

I’ve used some simple rendering rules:

• texts come from OSM tags equipped with “name” tag
• lines come from any non-closed OSM ways
• orange-filled areas come from any areas (closed OSM ways).

As a special treat, I’ve also created a SVG file from the same map, you can download it from here: http://downloads.igorbrejc.net/osm/misc/map1.svg

photo credit: The Pack

Yesterday I started thinking about a new rendering rules system for Kosmos. The existing system (using Wiki tables) was OK for a while, but it’s starting to show its old age: there is no rule inheritance, the rules are cumbersome to edit (because of wikitext), the selectors are not very powerful, rules are not reusable etc.

I started by investigating how similar software is doing these things. Mapnik has a more elaborate XML-based ruling system. It’s powerful, but it seems too verbose for what I had in mind – I want the rules to be simple to write and even simpler to read. And I want to stay away from XML: XML is easy to parse but not that friendly to humans. And writing expressions using XML character entities (example: “<Filter>[CARTO] &gt;= 2 and [CARTO] &lt; 5</Filter>”) is really unfriendly and error-prone.

The next thing to look at was MapCSS, a CSS-like language for map stylesheets and is a brainchild of Richard Fairhurst, the author of Potlatch, the excellent web-based OSM editor. I like the idea, but I’m worried how it could be implemented in the context of a database-driven map engine. The databases can be huge and you want to minimize (and optimize) the queries run on them, but MapCSS seems to allow too many combinations to make it usable for generating maps in (almost) real-time, and that’s what Kosmos is about. When working efficiently with the database, the rendering engine wants to collect all the related objects in one go, and then decide how to render them. So, for example, if it wants all highways to be rendered red, it’ll fetch all the highways from the database (with a single query) and then go and paint them all red. MapCSS doesn’t really allow this: you need to process each OSM object on its own and go through the list/tree of MapCSS rules in order to know whether (and how) to render it. Maybe I’m wrong and there is a better way to approach this, I’ll have to discuss this further with Richard and see what he thinks.

So now I’m thinking about defining my own domain-specific language (DSL) which would be used to specify rendering rules. I don’t have any specifics yet, but there are some things worth thinking about:

• The DSL should not be XML-based: angular brackets are not for humans.
• Separating definition of features from the definition of styles: features tell you what something represents. A forest, for example, is a feature that corresponds to an area tagged with natural=wood or landuse=forest. How it is rendered depends on the type of the map: on topo maps you would render it in green color. On driving maps maybe you wouldn’t render it at all. So the idea is to have a features definitions that could be stored in one place (one wiki page) and then reused all across the styling rules. Right now this is not possible in Kosmos.
• Expressions: the selector expressions should be more powerful than just simple logical operators. The same goes for expressions for style attributes.

Anyway, this all means I have to implement some kind of a parser for the DSL. So I started looking into possible C# libraries/tools for writing parsers: ANTLR, Coco/R, GOLD, Irony.NET, … There is also a good article called Grammars and parsing with C# 2.0. Time to do some refreshing of knowledge lost way back from my university days…

Last couple of months I’ve been working on database support for Kosmos. This work started as a request to produce bitmaps of the whole UK in the British national grid reference system from the OSM data.

The current version of Kosmos loads all the OSM data into memory. This obviously proves to be a problem for larger areas. Some time ago I worked on optimizing Kosmos memory usage and it proved to be quite successful, but there is still a physical limit of how large an area can be rendered.

The latest UK OSM data takes about 200 MB of zipped XML (2.5 GB unzipped). This is way too much for the existing in-memory system, so I had to find another way to render UK without reading the XML file directly.

After a bit of investigation, the obvious candidate for OSM data storage proved to be PostgreSQL / PostGIS. By “obvious” I mean that there were already some tools which could import OSM data into the PostgreSQL database, namely Osmosis. I was totally new to PostgreSQL, but the actual installation of the database engine was pretty easy.

The next step was importing of OSM data using Osmosis. Although this can be achieved with a few simple steps, there were quite a few “tricks” I had to learn the hard way before the performance of the database was satisfactory enough to produce map renderings in a realistic time:

• The PostgreSQL database engine uses non-optimal default settings, so you need to do a bit of investigation to set certain things up. One tip: if you have several disks, make sure your data is stored on a disk separate from the system.
• Osmosis OSM DB import command has a few settings on its own which can have a dramatic effect on the DB query performance. First of all, you need to use the extended OSM DB schema which contains bbox column for ways, otherwise the spatial queries would always have to contain multiple table joins which will terribly slow things down.
• Kosmos rendering rule engine was designed for instant-access in-memory data source. The way the data was fetched wasn’t really compatible with slow DB sources, so I had to do a lot of refactoring on the engine before the performance was good enough for any serious work.

After a few weeks of work, I finally managed to get some decent results. The rendering code generated UK for approximate zoom levels 7, 8 and 9 in about two hours on my machine, with the level of detail similar to Mapnik’s layer of the OSM main map. This may seem a lot of time, but considering there were about 11 million nodes and 1.5 million ways to process, I’m quite pleased. And there are further improvements still possible in the rendering engine which could reduce this time at least by half.

British National Grid

One of the biggest worries I had was how the rendering engine will behave when rendering using a map projection and reference system different from the “standard” Mercator and WGS-84. This actually proved to be the easiest thing to solve: the rendering engine internally uses the National grid coordinates and the spatial DB queries transform those transparently to WGS-84 (and back). I also needed to implement a new map projection, but this was just a matter of writing a few lines of math code.

Other Improvements

Since the map was supposed to be “professional” looking, there were a few things that needed to be implemented or improved on the existing rendering engine. These features will probably be included in the next generation of Kosmos, which will be released some time next year.

Text Placement

One of the first (and hardest) things needed for a good looking map is to make sure the text labels do not overlap each other. This is called (automatic) label placement. Good algorithms for label placement are quite complex and I didn’t really have time to implement a full-blown algorithm, so I chose to do a simple point-selection: the algorithm removes labels for smaller places (towns) until there are no more overlaps (see the sample map at the beginning of the post). I’ll write more about this feature some other time.

Better Relations Support

Some of the features needed to be rendered are now defined in OSM using relations. One example of this is national parks. So a new algorithm for consolidating relation’s ways into a single polygon was implemented.

Better Sea Filling Support

The coastline processing algorithm is now more resilient: it will ignore poorly connected coastlines and will still render all of those which are properly defined. This improvement is already in use in the new GroundTruth version released a week ago.

What Next?

During the work on PostgreSQL support I learned about SpatiaLite, a spatial extension to sqlite. As you may know, sqlite is a popular self-contained database engine which stores the data into a single file. So I decided to do a bit of playing around with it and see if it can also be used in Kosmos. I was a bit skeptical whether SpatiaLite will be fast enough for map rendering, but I can already say that it managed to achieve some very good results. I’ll write about SpatiaLite in the next blog post.

The new version of GroundTruth has just been released (download link). As I mentioned yesterday, the main work was on filling sea polygons based on the natural=coastline OSM tag. One additional feature is the ability to set the background color of the map (even if there are no coasts in your area). All of this is configured using the Options section of the rendering rules:

Sometimes it is not possible for the GT’s coastline processing algorithm to close up all of the coastlines and the resulting map can get ‘flooded’. In such a case I suggest you should use the new ‘–nosea’ option, which turns off sea polygon filling. Unfortunately a lot of the OSM planet extracts are lousy and they don’t include enough of the coastline to be able to connect it to the extract boundary, so I guess ‘-nosea’ option will have to be used quite frequently.

Some Background Information

This version of GroundTruth contains a lot of new (and improved) code which I was working on for the past several months. The work is primarily targeted towards a new generation of Kosmos tools, but since GT shares common geometry and OSM libraries with Kosmos, it can benefit from these libraries too.

photo credit: Joe Penniston

I’m on vacation today since I planned to do a hiking trip, but the weather turned out to be worse than was forecast, so I decided to stay home. And since I haven’t written any posts for some time, now is a good opportunity to fill you in about what has been happening with Kosmos in the last few months.

The main news is that I’m working hard on a new version of Kosmos. I’ve refactored a lot of the code, cleaned up some stuff I mentioned in the previous post. The “spring cleaning” of code turned out to extend to the summer and (soon) the fall, hopefully it will not extend to the winter (although the winter is the best time to do programming – nothing to do outside, long nights etc… )

Looking from the outside the new version will probably not be much different – the only really noticeable difference will be the fact that Kosmos no longer uses DockPanel Suite for tabbed document interface (TDI). The main reason for this is that it is tricky to control the TDI layout programmatically, which makes persisting the user’s choice of layout so that it is restored when Kosmos is restarted difficult. Also, the library is no longer being maintained and I don’t have the time to do it myself.

Instead, I implemented my own TDI framework, which is much simpler, has fewer features and doesn’t look so nice, but it is easier to control. I’m thinking about extending it to support multiple monitors, but since I still have only one monitor at home, this will be difficult to test .

The main difference with the new version is what the whole “spring cleaning” effort was all about – to streamline the internal design and make it much easier to add new features to Kosmos. My long-term plan is for Kosmos to be a platform for all kinds of applications – an OSM editor, a hiking planner, there are a lot of ideas. We’ll see if my time allows for this. I will also try to extract a GPL map rendering library which could then be used by anyone interested.

Of course, I will also try to fix some bugs that were reported by users.

When I finish the streamlining, the next step will be to migrate Kosmos to the new rendering engine I mentioned I was working on half a year ago. This is where things will start to get more interesting. I’m also thinking about introducing a new rendering rules system (maybe something in the direction of MapCSS). So, lot of ideas, lot of plans…

Anyway, this is all for now – I just wanted to let you know that Kosmos is not dead . Next time I will probably have some screenshots to show.

NASA has moved the SRTM tiles again (they are now stored under v2.1 directory, but I don’t know if there’s any difference in the content compared to the v2 version). So GroundTruth’s and Kosmos’ contours functionality stopped working.

Unfortunately I’m in the middle of a major refactoring of the whole of my OSM code in preparation for Kosmos v3 and support for other sources of DEM, so it’s kind of difficult to produce running versions of my software: the builds are broken, I’ve temporarily stopped my build server etc.

But I’ve decided to take a few hours and prepare a new build for GroundTruth at least (Kosmos will have to wait, too much changes in the code). Since I didn’t have much time to test it, it is “officially” marked as an experimental build, and I’ve put it under http://downloads.igorbrejc.net/osm/groundtruth/experimental/ directory (just use the ZIP with the latest date). I hope it works.

BTW: if NASA decides to move the tiles again, there’s a new setting in the configuration file (GroundTruth.exe.config):

<add key="SrtmServerUrl" value="http://dds.cr.usgs.gov/srtm/version2_1/"/>

You’ll be able to change the URL yourself this time

For the past couple of weeks I’ve been busy on extending my digital elevation model (DEM) library with some new features, including the ability to use DEM sources other than the “standard” SRTM3. I’ve managed to produce a working version of the code which supports Viewfinder’s DEM1 data for Alps and I was dying to test it on Julian Alps, especially the area around Triglav, the highest mountain in Slovenia. The SRTM3 data for Julian Alps is very poor, with large chunks of height data missing. Here’s a sample contours map generated in Kosmos using that low-quality SRTM data:

As you can see above, the actual Triglav mountain does not have any contours. Compare that with the contours generated with Viewfinder’s data:

Not only all the contours are there (and seem to correspond with reality), but also the resolution is three times better than that of SRTM3. I’m pretty impressed with Viewfinder’s data, I have to admit.

This is only an intermediate experiment, I have to do some more work on the library before I integrate it into GroundTruth and Kosmos. But the results so far are very promising.

BTW: I contacted Jonathan de Ferranti, the author of Viewfinder’s site, and asked him about terms of use of these DEM tiles. Basically he gave a wholehearted approval of using it for open-source projects as long as the original authorship is respected. He does not claim copyright on these tiles since they originated from various sources, be he still likes to be informed about any large-scale use of them.

photo credit: State Library of New South Wales collection

I’ve succumbed to the Twitter craze (well, not really, just experimenting . You can follow me on http://twitter.com/breki74

Topics? Development, OpenStreetMap, Kosmos and other projects I’m involved in, hiking, photography, who knows what else…

I finally managed to integrate my new digital elevation model (DEM) processing code into Kosmos. It was a lot more work than I anticipated, mostly because the existing Kosmos’ source code is getting large, difficult to maintain and even more difficult to introduce new functionality. For me it is the best example of why SOLID principles are not just a developers’ buzzword, but that they do have an impact on maintainability of the code as the code base gets larger and larger. Although Kosmos’ code is highly modular, uses all the paradigms of object-oriented programming, even uses inversion of control (IoC) containers quite a lot, this is not enough: certain portions of the code are pretty bloated with multiple responsibilities, which is a first no-no when we talk about SOLID principles: keeping the design clean by separating responsibilities into separate services/components.

When changing the code which serves more than one responsibility, it is difficult to separate these responsibilities and difficult to test the impacts. Unit tests in fact become mini-integration tests and you typically need more setup code in order for a test to be able to run. Which produces unit tests that are fragile and, like the “main” code, difficult to maintain.

I mentioned the usage of IoC containers (Castle Windsor actually). The problem is that you need some mileage to get things right: a common mistake is to drag the container around in your code and use it explicitly to create objects. This defeats the purpose of the IoC container: to wire things together without your intervention. I agree with Joshua Flanagan when he says that the “container” word was unfortunate choice: it should be called a “composer”. Kosmos was one of the first projects where I used an IoC container and it shows: there are a lot of things which I would have done differently now.

All the more reason for me to continue on my effort of rewriting the critical parts from scratch. But that’s a story for some other blog post.

Anyway, I hope the new IBF thing works. The bad news is that I needed to change/hack some code, so I cannot exclude a possibility of a bug or two. The other bad news is that the old “.dat” contours format is no longer supported, so you’ll need to recreate contours for your projects.

The good news, on the other hand, is that we switched to the NASA’s new SRTM server, since the old one has gone to retirement. And even more: the new IBF code is much better that the old Srtm2Osm one, the contours are now placed properly on the map, the contour generator now knows how to correctly ignore data voids and the contour files are much smaller.

I also made changes to GroundTruth to cut the produced IBF tiles exactly around the specified map boundary.

Oh yes, the download links:

• Kosmos: http://downloads.igorbrejc.net/osm/kosmos/ 
• GroundTruth: http://downloads.igorbrejc.net/osm/groundtruth/

Enjoy! And if you find the stuff I’m making useful, please consider a small donation. It will be used for books, software and equipment, all for the benefit of producing even better stuff in the future:

Thanks!