Archive for the 'development' Category

Published by breki on 05 Mar 2010

Disposing Of The Evidence

Garbage?
Creative Commons License photo credit: wok

In the previous post we discussed various relationships between singletons and transients in an inversion of control container. I introduced WindsorContainerInspector, a little utility class for analyzing the container and showed a sample unit test which looks for singletons that rely on transient objects.

Now I’ll show a few other checks that I think are good to perform on the container. They involve disposability of objects.

Disposable Transients

Let’s say we have a transient service IWebClient:

public interface IWebClient : IDisposable
{
void DownloadMeSomething(string somethingUrl);
}

As you can see, the service implements IDisposable. Now let’s say we have a class DownloadData which depends on this service:

public class DownloadData
{
public DownloadData (IWebClient webClient) {...}

...
}

Note the constructor dependency on IWebClient. What’s wrong with this picture?

What happens when the DownloadData is no longer used? The webClient is not disposed. We could hold up some valuable system resource by not disposing of such an object. And since we know IWebClient is defined as transient in our IoC container, we can be pretty sure that DownloadData is the only one using the received IWebClient instance. So we are responsible for this instance!

In other words, our DownloadData should also be disposable. We can say that, in general, components that depend on disposable transients should also be disposable:

[Test, MultipleAsserts]
public void TransientsThatRelyOnDisposableTransientsShouldAlsoBeDisposable ()
{
    Type disposableType = typeof(IDisposable);
 
    IList<KeyValuePair<ComponentModel, ComponentModel>> dependencies = inspector.FindDependencies (
        (a, b) =>
        {
            return a.LifestyleType == LifestyleType.Transient && false == disposableType.IsAssignableFrom (a.Implementation)
                && b.LifestyleType == LifestyleType.Transient && disposableType.IsAssignableFrom(b.Service);
        });
 
    foreach (KeyValuePair<ComponentModel, ComponentModel> dependency in dependencies)
    {
        Assert.Fail (
            "Transient component {0} relies on the disposable transient component {1}, but it itself is not disposable",
            dependency.Key.Implementation.FullName,
            dependency.Value.Implementation.FullName);
    }
}

Some might say that this kind of check can be done by static analysis like FxCop. FxCop will detect this only if DownloadData stores the IWebClient in a class field. But we could have used IWebClient parameter just within the constructor and then forget about it:

public class DownloadData
{
public DownloadData (IWebClient webClient)
{
webClient.CheckConnection();
}
}

In such cases no warning will be issued by FxCop.

Disposable Implementations Of Non-Disposable Services

What if we had another service, called IMailChecker:

public interface IMailChecker
{
bool AnyNewMail();
}

and we implemented a GMailChecker:

public class GMailChecker : IMailChecker, IDisposable
{
public bool AnyNewMail() {...}
public void Dispose() {...}
}

Notice GMailChecker implements IDisposable in addition to IMailChecker interface. But IMailChecker itself is not defined as disposable. How will the user of IMailChecker service know whether the object it holds should be disposed or not?

One way would be to use reflection to determine this. But this is very cumbersome and easy to forget when coding. My way of thinking is that if there’s reasonable possibility the implementations of the service will require disposability, you should declare the service interface as disposable. We can check for this condition with the following unit test:

[Test, MultipleAsserts]
public void TransientsThatRelyOnDisposableTransientsShouldAlsoBeDisposable ()
{
    Type disposableType = typeof(IDisposable);
 
    IList<KeyValuePair<ComponentModel, ComponentModel>> dependencies = inspector.FindDependencies (
        (a, b) =>
        {
            return a.LifestyleType == LifestyleType.Transient && false == disposableType.IsAssignableFrom (a.Implementation)
                && b.LifestyleType == LifestyleType.Transient && disposableType.IsAssignableFrom(b.Service);
        });
 
    foreach (KeyValuePair<ComponentModel, ComponentModel> dependency in dependencies)
    {
        Assert.Fail (
            "Transient component {0} relies on the disposable transient component {1}, but it itself is not disposable",
            dependency.Key.Implementation.FullName,
            dependency.Value.Implementation.FullName);
    }
}

Below is the updated the implementation of WindsorContainerInspector. Next time we will be discussing issues (= mostly problems) around disposing stuff received from the Windsor container.

public class WindsorContainerInspector
{
    public WindsorContainerInspector(IWindsorContainer container)
    {
        this.container = container;
    }
 
    public IList<KeyValuePair<ComponentModel, ComponentModel>> FindDependencies(
        Func<ComponentModel, ComponentModel, bool> dependencyPredicate)
    {
        List<KeyValuePair<ComponentModel, ComponentModel>> dependencies = new List<KeyValuePair<ComponentModel, ComponentModel>>();
 
        foreach (GraphNode node in container.Kernel.GraphNodes)
        {
            ComponentModel dependingNode = (ComponentModel) node;
 
            foreach (GraphNode depender in node.Dependents)
            {
                ComponentModel dependerNode = (ComponentModel)depender;
 
                if (dependencyPredicate(dependingNode, dependerNode))
                    dependencies.Add(new KeyValuePair<ComponentModel, ComponentModel>(dependingNode, dependerNode));
            }
        }
 
        return dependencies;
    }
 
    public IList<ComponentModel> FindComponents (Func<ComponentModel, bool> predicate)
    {
        List<ComponentModel> components = new List<ComponentModel>();
 
        foreach (GraphNode node in container.Kernel.GraphNodes)
        {
            ComponentModel component = (ComponentModel) node;
            if (predicate(component))
                components.Add(component);
        }
 
        return components;
    }
 
    private readonly IWindsorContainer container;
}

Published by breki on 01 Mar 2010

The Delicate Dance Of Transience

H2O
Creative Commons License photo credit: Flowery Luza*

I’m in the final stages of polishing Maperitive before the first release. Well, not really polishing, I have a bunch of tasks yet to do, but I think I can see the light at the end of the tunnel.

As #I stated a while back#, one of the reasons I decided to do a major rewrite of the Kosmos code was to introduce inversion of control (IoC) into the architecture. Maperitive (ex-Kosmos) now heavily relies on Windsor Castle do to the wiring. I can say that investing in IoC (and sticking to SOLIDs) now starts to pay dividends: I can add new features with more ease and they tend to affect a relatively minor part of “old” code.

The problem with using IoC containers like Windsor Castle is when the components registered exceed a certain number that you can handle in your brain (Maperitive currently has about 130 components in IoC container). Then the whole thing gets a life of its own: even if you write extensive unit tests for individual parts, it gets more and more difficult to predict how the “organism” will behave as a whole.

When used in a Windows desktop application like Maperitive, the problem gets compounded by the intricate dependencies between different lifestyles of components. This is what I want to discuss in this post.

I’ll restrict the discussion to singletons and transients, since I only use these two types in Maperitive. I’ll give examples of components in terms of a mapping application (which Maperitive is), but I think it won’t be a problem to translate this to your own areas. I’ll start with the simple case of…

Singletons Depending On Singletons

This is the easy one: singletons are components which typically share the same lifespan as the application. A Map is a good example of a singleton: the user expects the map to be available at all times (after all, what would Google Earth look like without the map?). User typically uses a mouse to move around the map. And we luckily have only one mouse on our machines, so this is an example of another singleton (I’m a bit oversimplifying the stuff here, bear with me). So the Map subscribes to the Mouse events and this can be seen as singleton to singleton dependency.

These kind of dependencies are easy to handle, since the number of created objects is typically small and they all share the same lifespan. Also, singletons are (typically) created automatically by the IoC container at the start of the application so you don’t really need to explicitly create and destroy them yourself.

Transients Depending On Singletons

This is the second common relationship in a desktop application. You have a short-term task to do (usually as a response to user’s actions) and you create a transient object to do this task. One example would be a WebClient which downloads the map data from the server: once the download is done, you can kill the client (hmmm that sounds tempting).

Transients are typically created using factories. Windsor Castle offers a cool feature called TypedFactoryFacility which makes creating factories a lot easier (you don’t need to write the actual creation code, just specify your factory interface according to facility’s conventions).

A WebClient needs to know whether it has to provide credentials for the Web proxy. We can store this information in a Configuration object, which in our case is a singleton.

A transient component states its dependency on a singleton through constructor or property injection. Again, this scenario should be easy to handle, since the transient object has a shorter lifespan than the singleton it depends upon.

Transients Depending On Transients

This is where things can get messy. There are two possible scenarios when a transient object needs another transient object:

  1. They both share the same lifespan. Let’s say our WebClient wants to check if internet connection is up before sending a request to the download server. It can use a Ping service to ping google.com (it’s immoral, I know). Since both WebClient and Ping will be used for a short time, you can inject the Ping dependency in WebClient’s constructor. When WebClient dies, Ping should also die.
  2. The lifespans overlap, but are not the same.  Example: out WebClient has detected a communication problem with the download server and sends a notice to the user. The notice is displayed as a modeless dialog, which can stay on the screen until the user clicks on the Close button. So Notice is a transient object which does not die together with its creator, WebClient: once WebClient notifies the user, it is no longer needed. Notice, however, will live on until the user chooses to close it.

In the second case, having the Notice dependency in WebClient’s constructor or property is not an ideal option, for two reasons:

  1. It is not guaranteed the notice will even be used: if the download went OK, there is no need to notify the user. If your Notice component is expensive to create, this could be an issue.
  2. By using a constructor injection for a transient component, you are effectively claiming the ownership of this component. This conflicts with the fact that the component will live on even after your main component dies.

How do you solve this problem? My suggestion is to use factories – factories are usually singleton objects, so you end up with “transient depending on singleton” case. Even better: Windsor Castle’s TypedFactoryFacility also offers a mechanism to release components created with such factory.

The last (and most problematic) case is…

Singletons Depending On Transients

The problem with this is that your supposedly transient object will live the whole duration of the application lifetime. While in some cases this could be a valid scenario, in lot of cases it’s merely an oversight of the developer (especially when confronted with a huge number of components in a system).

Windsor Castle Inspector

The solution I used in Maperitive is to write a unit test which checks for singleton-to-transient relationships in my container and fails if it finds any:

[Test]
public void SingletonsShouldNotRelyOnTransients()
{
    IWindsorContainer container = Program.CreateWindsorContainer(false);
 
    WindsorContainerInspector inspector = new WindsorContainerInspector(
        container);
    IList<KeyValuePair<ComponentModel, ComponentModel>> dependencies 
         = inspector.FindDependencies(
        (a, b) =>
            {
                return (a.LifestyleType == LifestyleType.Singleton 
                    || a.LifestyleType == LifestyleType.Undefined)
                    && b.LifestyleType == LifestyleType.Transient;
            });
 
    Assert.AreEqual(0, dependencies.Count);
}

Of course, the condition for the test failure could be a bit modified to exclude any “special cases".

The code uses the WindsorContainerInspector, a little utility class I wrote to assist in inspecting the container:

public class WindsorContainerInspector
{
    public WindsorContainerInspector(IWindsorContainer container)
    {
        this.container = container;
    }
 
    public IList<KeyValuePair<ComponentModel, ComponentModel>> FindDependencies(
        Func<ComponentModel, ComponentModel, bool> dependencyPredicate)
    {
        List<KeyValuePair<ComponentModel, ComponentModel>> dependencies 
            = new List<KeyValuePair<ComponentModel, ComponentModel>>();
 
        foreach (GraphNode node in container.Kernel.GraphNodes)
        {
            ComponentModel dependingNode = (ComponentModel) node;
 
            foreach (GraphNode depender in node.Dependents)
            {
                ComponentModel dependerNode = (ComponentModel)depender;
 
                if (dependencyPredicate(dependingNode, dependerNode))
                    dependencies.Add(new KeyValuePair<ComponentModel, ComponentModel>(
                        dependingNode, 
                        dependerNode));
            }
        }
 
        return dependencies;
    }
 
    private readonly IWindsorContainer container;
}

Right now WindsorContainerInspector offers only one method: FindDependencies, which looks for dependencies based on the supplied predicate. I already have some additional ideas on what other possible problems to detect in the container, but I’lll write about this next time.

Published by breki on 20 Feb 2010

Paper Prototyping

A screenshot from my yesterday’s Maperitive paper prototyping session:

Paper prototyping

Tools of the trade:

Paper prototyping - tools

Published by breki on 15 Feb 2010

Am I Doing Amateur Software?

Ouchy Creative Commons License photo credit: 1Happysnapper(photography)

Just read Amateur Software on The Endeavour blog. I liked the following line:

Volunteers do what they want to do by definition. The problem is that the reverse is also true: volunteers do not do what they do not want to do. And for software developers, writing documentation usually falls in the “do not want to do” column. So does making software easy to install. So does testing in multiple environments.

I had to reread the “do not” part several times before I got it into my head :) .

I guess I fall into this “amateur” category. To some extent. I don’t like writing documentation. But I somehow forced myself to write at least something about the software I’m developing. That’s why I like the concept of an open Wiki – the users can participate in this painful activity, too.

Published by breki on 05 Feb 2010

Poor Man’s Task Tracking Tool

31.01.
    - indicate the command prompt has focus
    - added detection of changed OSM files
    - implemented get-info command
    - started working on the error reporting func.
02.02.
    - implemented MailService
    - implemented AppDiagnostics
    - added error reporting form, but it still needs to be beautified
    - support for polygons/polylines consolidated from relations
04.02.
    - solved the problem: why is footway drawn ABOVE residential road?
        - it's because the footway feature is defined AFTER the residential road
    - fixed: bug with form disposing        
    - if a way is used in a relation, it should be ignored elsewhere
        -written tests for this        
05.02.        
    - TODO: the rules and features should be specified in the reverse order
        - from the most important to the least important
    - TODO: how to solve layering problem?
 
 
 
- TODO: beautify error reporting dialog
- TODO: find an icon for Maperitive
- TODO: implement multipolygon polyline rendering
    - TODO: write tests for multipolygon rendering

This is an excerpt from Maperitive’s Todo.txt file. I use it as a log of the stuff I did for the current day and also as a “database” of the tasks I still have do. On various projects during the years I used Trac, Bugzilla, on Kosmos I used ToDoList for a while, but nothing beats the simplicity of a text file.

The history log is a recent “invention”: since Maperitive is a pet-project, the time allocated to it is very unevenly distributed – sometimes I do a lot of work in a single day (usually on weekends without find weather or hangovers), but then I have to leave it untouched for several days. So often it happens that I forget what I was working on – and this history log helps me to quickly refresh the memory. Also, it’s a good psychological tool: I take a look and the log entries and see that some work has actually been done and I’m (slowly) progressing towards the first release.

The added benefit is that I can copy&paste these log entries into SVN commit comments. And of course, Todo.txt is kept under the source control like the rest of the code.

Published by breki on 28 Nov 2009

Kosmos: Preview Of SVG Support

Some time ago I wrote about experimenting with Cairo drawing engine to render maps. The basic idea was to use Cairo to render SVG map files which can then be manually post-processed using some vector graphics editor like Inkscape.

Unfortunately I worked on lot of other stuff after that, so the experiment was put on hold. But a few weeks ago I started working on a new rendering rules engine for Kosmos and I wanted to write some unit tests to try it out – and the experimental map drawing code seemed a good fit.

The rules engine is far from complete, but I can already reveal a few details of its capabilities. First, you specify a set of features, for example a forest, a motorway, a town. In accordance to some common GIS concepts, there are three types of features: point, line and area feature.

Then you specify how each of these features should be rendered. This consists of a directed graph of commands which supports inheritance of rendering properties (something in the direction of MapCSS, but not as liberal).

I won’t bore you with the details (at least until a proper parser for these rendering rules is available), but I think this new system will be much more powerful than the existing “flat” Wiki table-based rules one. Also, I’ve been working on improving support for relations and multipolygons.

Anyway: back to Cairo: I’ve started implementing more detailed drawing logic and I had quite a few problems figuring out how to render text using Cairo. It turns out the API for drawing text in Cairo is pretty poor (it even crashes when trying to render non-ASCII chars), so the authors of Cairo recommend using Pango, a library for high-quality text rendering. Luckily, it has Mono wrappers and it wasn’t too difficult to make it write out texts on the bitmap.

I’ve written few unit tests for some simple OSM map rendering and here are the results (the left map is rendered using GDI+, the right one using Cairo):

map2gdimap2cairo

I’ve used some simple rendering rules:

  • texts come from OSM tags equipped with “name” tag
  • lines come from any non-closed OSM ways
  • orange-filled areas come from any areas (closed OSM ways).

As a special treat, I’ve also created a SVG file from the same map, you can download it from here: http://downloads.igorbrejc.net/osm/misc/map1.svg

Published by breki on 12 Nov 2009

Changing The Build Server

Recently I’ve upgraded my home development machine to Windows 7. It is useful to do these cleanings from time to time, since it forces you to take a step back from your current configuration and think whether something new might be better.

So yesterday while I was preparing a new release of GroundTruth, I was missing my CruiseControl.NET server and started to reinstall it. After hitting a few snags I got bored with the whole idea: I constantly keep resolving the same issues with CC.NET installations (since I use it both at home and at work). Remembering I once played with Hudson and liked it, I decided to give it a serious try.

Setting Up Hudson

The installation was really easy – just download the .war file and run it using java. I did have to move the home directory from the default user’s profile directory to my data disk in order to make the whole installation more portable.

This is what I like about Hudson: no hassle with XML configuration files, you can configure it with a user-friendly Web GUI. Also, you don’t need IIS – the Hudson has its own integrated web server. And it even provides a button for installing Hudson as a Windows service!

The only real drawback with Hudson is that it’s a Java application and if you want to extend it with your own plug-ins, you need to write Java code. Which is fine, but not as accessible for .NET developers. But lately I’ve started using the build server just for building, labeling and packaging and I don’t really need any special plug-ins for that. Anyway, Hudson already has a lot of plug-ins, some of them even for .NET, so I don’t think I will need to write my own any time soon.

Good Bye Cruising

I’ve had quarrels with CC.NET before. Now I think of CC.NET as a nice introduction into CI world, but after a while you need something else. My view is that the .NET world needs a new open source CI project, which would build upon experience of CC.NET, both positive and negative. This is what I would like to see:

  • No hassle: just copy a single executable and run it. No IIS setup, no nothing. Web server comes with the package. Windows service installation with a single mouse click.
  • Portability: server configuration has to be separated from the server executables. If a new version of the build server arrives, the upgrade be as simple as overwriting a single executable file. No messing around with Web.configs, dashboard.configs etc.
  • Web-based installation & upgrade: I like how WordPress is doing things: you can upgrade your WordPress installation using the Web dashboard. It would also be nice to be able to install plug-ins just by pointing to its home URL.
  • Simple GUI: simple as in “Google search simple”. 95% of the time you only need a build radiator and nothing else. Everything else should be accessible, but not on the first page. And think simple permalinks. BTW: in my view, even Hudson’s GUI could be improved in this aspect.
  • Interactive GUI: I want to see a live build log, without manual refreshing. More Ajax please.

Published by breki on 03 Nov 2009

DSL For Rendering Rules

Doodle
Creative Commons License photo credit: The Pack

Yesterday I started thinking about a new rendering rules system for Kosmos. The existing system (using Wiki tables) was OK for a while, but it’s starting to show its old age: there is no rule inheritance, the rules are cumbersome to edit (because of wikitext), the selectors are not very powerful, rules are not reusable etc.

I started by investigating how similar software is doing these things. Mapnik has a more elaborate XML-based ruling system. It’s powerful, but it seems too verbose for what I had in mind – I want the rules to be simple to write and even simpler to read. And I want to stay away from XML: XML is easy to parse but not that friendly to humans. And writing expressions using XML character entities (example: “<Filter>[CARTO] &gt;= 2 and [CARTO] &lt; 5</Filter>”) is really unfriendly and error-prone.

The next thing to look at was MapCSS, a CSS-like language for map stylesheets and is a brainchild of Richard Fairhurst, the author of Potlatch, the excellent web-based OSM editor. I like the idea, but I’m worried how it could be implemented in the context of a database-driven map engine. The databases can be huge and you want to minimize (and optimize) the queries run on them, but MapCSS seems to allow too many combinations to make it usable for generating maps in (almost) real-time, and that’s what Kosmos is about. When working efficiently with the database, the rendering engine wants to collect all the related objects in one go, and then decide how to render them. So, for example, if it wants all highways to be rendered red, it’ll fetch all the highways from the database (with a single query) and then go and paint them all red. MapCSS doesn’t really allow this: you need to process each OSM object on its own and go through the list/tree of MapCSS rules in order to know whether (and how) to render it. Maybe I’m wrong and there is a better way to approach this, I’ll have to discuss this further with Richard and see what he thinks.

So now I’m thinking about defining my own domain-specific language (DSL) which would be used to specify rendering rules. I don’t have any specifics yet, but there are some things worth thinking about:

  • The DSL should not be XML-based: angular brackets are not for humans.
  • Separating definition of features from the definition of styles: features tell you what something represents. A forest, for example, is a feature that corresponds to an area tagged with natural=wood or landuse=forest. How it is rendered depends on the type of the map: on topo maps you would render it in green color. On driving maps maybe you wouldn’t render it at all. So the idea is to have a features definitions that could be stored in one place (one wiki page) and then reused all across the styling rules. Right now this is not possible in Kosmos.
  • Expressions: the selector expressions should be more powerful than just simple logical operators. The same goes for expressions for style attributes.

Anyway, this all means I have to implement some kind of a parser for the DSL. So I started looking into possible C# libraries/tools for writing parsers: ANTLR, Coco/R, GOLD, Irony.NET, … There is also a good article called Grammars and parsing with C# 2.0. Time to do some refreshing of knowledge lost way back from my university days…

Published by breki on 27 Oct 2009

C#: Service Caching Proxies With The A Little Help From Some Functional Code

On a current project we’re working on in our company, we are developing a web application which accesses the back-end through some web services. Nothing special really, except that certain web services provide pretty static information like lookup tables, which don’t change very often, so it’s not really necessary to refetch them all the time.

Since we are heavily relying on the dependency injection and all the back-end services are exposed through C# interfaces, it wasn’t really hard to develop a new implementation of such an interface which calls the back-end service and then caches the results in the HTTP cache to be used by any subsequent calls.

Here’s a simple example of a service interface:

public interface IBackendService
{
    List<string> ListSomeStrings(string parameter1, int parameter2);
}

I won’t bore you with an implementation of this, since it’s really a dumb one. What’s interesting is how the cached implementation looks like:

public class BackendServiceCachingProxy : CachingProxyBase<IBackendService>, IBackendService
{
    public BackendServiceCachingProxy(
        IBackendService wrappedService, 
        Cache cache,
        TimeSpan slidingCacheExpiration)
        : base(wrappedService, cache, slidingCacheExpiration)
    {
    }

    public List<string> ListSomeStrings(string parameter1, int parameter2)
    {
        return CallServiceMethod<string>(
            () => ConstructCacheKey("SomeStrings", parameter1, parameter2),
            s => s.ListSomeStrings(parameter1, parameter2));
    }
}

Our caching wrapper inherits from a base class CachingProxyBase. Before showing you the code for CachingProxyBase, let me just quickly explain what the BackendServiceCachingProxy code does.

We’re using HTTP cache, so we have to supply it to the class in the constructor. We also specify how long the cache should be valid (slidingCacheExpiration parameter).

Each service method in the caching wrapper now uses CallServiceMethod method to implement the cached service method calls. You specify two parameters:

  1. A function delegate for constructing the cache key. This key should be unique for each unique combination of method’s input parameters. The CachingProxyBase offers a helper method ConstructCacheKey to help you with this task.
  2. A function delegate for calling the method on the service. This delegate will be used for calling the actual service implementation which will contact the back-end.

 

CachingProxyBase

And now the implementation of the CachingProxyBase:

public abstract class CachingProxyBase<TService>
{
    public void ExpireAllCachedData()
    {
        foreach (System.Collections.DictionaryEntry entry in cache)
            cache.Remove(entry.Key.ToString());
    }

    protected CachingProxyBase(
        TService wrappedService,
        Cache cache,
        TimeSpan slidingCacheExpiration)
    {
        this.wrappedService = wrappedService;
        this.cache = cache;
        this.slidingCacheExpiration = slidingCacheExpiration;
    }

    protected Cache Cache
    {
        get { return cache; }
    }

    protected TimeSpan SlidingCacheExpiration
    {
        get { return slidingCacheExpiration; }
    }

    protected TService WrappedService
    {
        get { return wrappedService; }
    }

    protected TValue CallServiceMethod<TValue>(
        Func<string> constructCacheKeyFunc,
        Func<TService, TValue> serviceMethodFunc)
        where TValue : class
    {
        string cacheKey = constructCacheKeyFunc();

        TValue cachedValue = GetCachedValue<TValue>(cacheKey);

        if (cachedValue == null)
        {
            cachedValue = serviceMethodFunc(WrappedService);
            CacheValue(cacheKey, cachedValue);
        }

        return cachedValue;
    }

    protected string ConstructCacheKey(
        string function, 
        string environmentId, 
        params object[] args)
    {
        StringBuilder cacheKeyBuilder = new StringBuilder();
        cacheKeyBuilder.AppendFormat(
            CultureInfo.InvariantCulture,
            "Proxy-{0}-{1}",
            function,
            environmentId);

        foreach (object arg in args)
            cacheKeyBuilder.AppendFormat(CultureInfo.InvariantCulture, "-{0}", arg);

        return cacheKeyBuilder.ToString();
    }

    private void CacheValue(string cacheKey, object value)
    {
        Cache.Insert(
            cacheKey,
            value,
            null,
            Cache.NoAbsoluteExpiration,
            SlidingCacheExpiration);
    }

    [SuppressMessage("Microsoft.Design", "CA1004:GenericMethodsShouldProvideTypeParameter")]
    private T GetCachedValue<T>(string cacheKey) 
    {
        return (T)cache.Get(cacheKey);
    }

    private readonly TService wrappedService;
    private readonly Cache cache;
    private readonly TimeSpan slidingCacheExpiration;
}

TService is a generic parameter representing the service interface the proxy is enhancing with the caching. I think the code is pretty self-explanatory. The ExpireAllCachedData method is exposed to be used in unit tests to make sure the cache is empty before doing any work.

Using It With Windsor Castle

I’ve made a helper method for configuring in Windsor Castle a specific back-end service with or without caching:

[SuppressMessage("Microsoft.Design", "CA1004:GenericMethodsShouldProvideTypeParameter")]
protected void RegisterServiceWithOptionalCache<TService, TImplementation, TCachedImplementation>(
    IWindsorContainer container,
    TimeSpan slidingCacheExpiration)
    where TImplementation : TService
    where TCachedImplementation : TService
{
    string serviceName = typeof(TService).Name;

    if (IsCachingUsed)
    {
        container.Register(
            Component.For<TService>().ImplementedBy<TCachedImplementation>()
                .ServiceOverrides(ServiceOverride.ForKey("wrappedService").Eq(serviceName))
                .Parameters(Parameter.ForKey("slidingCacheExpiration").Eq(slidingCacheExpiration.ToString())));

        if (log.IsDebugEnabled)
            log.DebugFormat("Registered {0}", typeof(TCachedImplementation).Name);
    }

    container.Register(
        Component.For<TService>().ImplementedBy<TImplementation>()
            .Named(serviceName));            
}

 

Conclusion

There are probably more elegant ways of doing this (probably using Windsor’s interceptors and reflection), but I think this code does its job well if you have a small number of service methods to cover.

Published by breki on 23 Oct 2009

Kosmos And Spatial Databases, Part 1

Kosmos Label Placement Sample

Last couple of months I’ve been working on database support for Kosmos. This work started as a request to produce bitmaps of the whole UK in the British national grid reference system from the OSM data.

The current version of Kosmos loads all the OSM data into memory. This obviously proves to be a problem for larger areas. Some time ago I worked on optimizing Kosmos memory usage and it proved to be quite successful, but there is still a physical limit of how large an area can be rendered.

The latest UK OSM data takes about 200 MB of zipped XML (2.5 GB unzipped). This is way too much for the existing in-memory system, so I had to find another way to render UK without reading the XML file directly.

After a bit of investigation, the obvious candidate for OSM data storage proved to be PostgreSQL / PostGIS. By “obvious” I mean that there were already some tools which could import OSM data into the PostgreSQL database, namely Osmosis. I was totally new to PostgreSQL, but the actual installation of the database engine was pretty easy.

The next step was importing of OSM data using Osmosis. Although this can be achieved with a few simple steps, there were quite a few “tricks” I had to learn the hard way before the performance of the database was satisfactory enough to produce map renderings in a realistic time:

  • The PostgreSQL database engine uses non-optimal default settings, so you need to do a bit of investigation to set certain things up. One tip: if you have several disks, make sure your data is stored on a disk separate from the system.
  • Osmosis OSM DB import command has a few settings on its own which can have a dramatic effect on the DB query performance. First of all, you need to use the extended OSM DB schema which contains bbox column for ways, otherwise the spatial queries would always have to contain multiple table joins which will terribly slow things down.
  • Kosmos rendering rule engine was designed for instant-access in-memory data source. The way the data was fetched wasn’t really compatible with slow DB sources, so I had to do a lot of refactoring on the engine before the performance was good enough for any serious work.

After a few weeks of work, I finally managed to get some decent results. The rendering code generated UK for approximate zoom levels 7, 8 and 9 in about two hours on my machine, with the level of detail similar to Mapnik’s layer of the OSM main map. This may seem a lot of time, but considering there were about 11 million nodes and 1.5 million ways to process, I’m quite pleased. And there are further improvements still possible in the rendering engine which could reduce this time at least by half.

British National Grid

One of the biggest worries I had was how the rendering engine will behave when rendering using a map projection and reference system different from the “standard” Mercator and WGS-84. This actually proved to be the easiest thing to solve: the rendering engine internally uses the National grid coordinates and the spatial DB queries transform those transparently to WGS-84 (and back). I also needed to implement a new map projection, but this was just a matter of writing a few lines of math code.

Other Improvements

Since the map was supposed to be “professional” looking, there were a few things that needed to be implemented or improved on the existing rendering engine. These features will probably be included in the next generation of Kosmos, which will be released some time next year.

Text Placement

One of the first (and hardest) things needed for a good looking map is to make sure the text labels do not overlap each other. This is called (automatic) label placement. Good algorithms for label placement are quite complex and I didn’t really have time to implement a full-blown algorithm, so I chose to do a simple point-selection: the algorithm removes labels for smaller places (towns) until there are no more overlaps (see the sample map at the beginning of the post). I’ll write more about this feature some other time.

Better Relations Support

Some of the features needed to be rendered are now defined in OSM using relations. One example of this is national parks. So a new algorithm for consolidating relation’s ways into a single polygon was implemented.

Better Sea Filling Support

The coastline processing algorithm is now more resilient: it will ignore poorly connected coastlines and will still render all of those which are properly defined. This improvement is already in use in the new GroundTruth version released a week ago.

What Next?

During the work on PostgreSQL support I learned about SpatiaLite, a spatial extension to sqlite. As you may know, sqlite is a popular self-contained database engine which stores the data into a single file. So I decided to do a bit of playing around with it and see if it can also be used in Kosmos. I was a bit skeptical whether SpatiaLite will be fast enough for map rendering, but I can already say that it managed to achieve some very good results. I’ll write about SpatiaLite in the next blog post.

Next »