photo credit: Omar Eduardo

My previous post about PBF reading successes was written way too prematurely. It turned out my PBF reading code had some serious bugs which made reading look much faster than it actually was (one of the reasons was that I neglected to read OSM node keys/values when written in PBF dense node format).

I’ve subsequently written some extensive tests, comparing OSM database contents from XML and PBF file of the same area (thanks Geofabrik) on an object by object basis, so I’m now 95% sure the PBF code works OK. Performance-wise the (final?) results are much less glamorous than it looked initially: PBF reading is “only” 2.5 times faster than reading OSM.bz2 files, while in memory consumption terms, they are pretty much the same. I curious what other OSM software like osmosis has to say about these results.

I had hoped I could speed the PBF reading by spreading the work on several processor cores. What I did is to use Microsoft’s Parallel Extensions library to separate the fetching of PBF file blocks from the actual parsing of them into two (or more) cores. This resulted in only about 10% increase of the overall speed (tested on my two-core machine, so on more cores the result could be better).

It actually proved pretty hard to do a decent job of separating work in some balanced fashion. Since the file reading is sequential, this can only be done by one thread/core, so you want to put as little other work to that core as possible. As soon as file block bytes are fetched from the file, they are delegated to another core to parse it (in terms of protocol buffers) and then extract OSM objects from it. The problem is that you don’t want to enqueue too many file blocks at the same time, since this takes up valuable memory (which is already filled with extracted OSM objects). So I ended up using a blocking queue, which means the main thread (which reads the file) will wait until at least one core is available before filling the queue with another file block.

I’ve also tried micro-management strategy – using multiple cores to extract individual OSM objects, but this only really works for ways and relations. Current PBF extracts use dense nodes format, which is delta-encoded and thus forces you to read things sequentially on a single thread of execution. I guess this is the price of having a format that wants to satisfy two different (and inherently conflicting) goals: less space and less CPU.

I’m fairly new to Parallel Extensions and there are probably better ways of handling this, but I’ll leave it for the future.

Anyway, a new Maperitive release is out, grab it from the usual place.

Fortunately Gallio provides a class called TestContext which contains the current information about the running test and which you can use to determine if the latest test is successful or not. This can then be used to run your custom handling code during the test teardown:

        [TearDown]
        protected virtual void Teardown()
        {
            if (TestContext.CurrentContext.Outcome.Status == TestStatus.Failed)
            {
                using (TestLog.BeginSection("Failed web page HTML"))
                    TestLog.Write(WebDriver.PageSource);
            }
        }

In the above snippet, we record the current Web page’s HTML code into Gallio’s log (the TestLog class). To avoid spamming the log, we do this for failed tests only.

Gallio provides a powerful framework which I think is very much underused, mostly because the documentation is not very detailed (to say the least).

Castle.MicroKernel.Resolvers.DependencyResolverException: Could not resolve non-optional dependency for ‘Karta.DataSources.OsmFileMapDataSource’ (Karta.DataSources.OsmFileMapDataSource). Parameter ‘fileName’ type ‘System.String’

I tried to reproduce this behavior using a simple unit test, but I couldn’t, so I’m posting the actual code. This is where the exception occurs:

return windsorContainer.Resolve<OsmFileMapDataSource>();

And this is how OsmFileMapDataSource constructors look like:

        public OsmFileMapDataSource(
            string fileName,
            IFileSystem fileSystem,
            IMapDataLayerFactory layerFactory)
        {
            ...
        }

        public OsmFileMapDataSource(IMapDataLayerFactory layerFactory)
        {
           ...
        }

Needless to say, both IFileSystem and IMapDataLayerFactory are registered in the container (IMapDataLayerFactory is registered as a typed factory, by the way). OsmFileMapDataSource is also registered as an implementation of itself. And I’m using version 2.1.0.6655 of the library.

What’s strange about this is that if I move the second constructor in front of the first one, the component is resolved without problems. I’m not sure if this is intended behavior, but I doubt the order of constructors should be a determining factor on how components are resolved.

But as I said, I couldn’t reproduce this behavior using a simplified test code, so I guess I should start debugging it instead.

For the last week or so I’ve been busting my fingers with one of the harder things to implement in a desktop GUI: application responsiveness when executing longer-running tasks. By longer-running I mean something that takes more than a couple of seconds.

Simple desktop applications tend to run everything synchronously: when the user presses a button, the action gets run. After the action finishes, the control is given back to the user. Simple, but totally crappy. The problem is that the action gets run on the same thread that services the GUI, so until the action finishes, the application will not be able even to refresh itself or respond in any meaningful way to user clicks or key presses. And since there is no refresh, you cannot show any progress indicators to the user. I know I wouldn’t want to wait half an hour for something to finish without some reassurance that the application is still alive and not waiting for the electricity to run out.

Maperitive already had a lot of responsiveness code implemented, but it was still an “under the construction” design. The additional complication was the fact I want Maperitive to have a good script runner and scripts can take a very long time (downloading OSM data, generating tiles etc.). After a lot of trials and errors I finally managed to implement the whole thing in a consistent package. And believe me when I say it was not easy.

So what is new:

• When running scripts (or longer tasks), Maperitive draws an indicator on the map (see the screenshot above) and launches a “sort of modal” mode – most of GUI controls are disabled so the script doesn’t get confused by some inadvertent user action. However, the application is still responsive: you can view the progress of the script in the command log.
• Aborting scripts: as the indicator says, you can press the escape key to abort the script. If you prefer torturing your mouse instead of your keyboard, there’s an “Abort task” button in the bottom right corner which does the same thing.

Not all of running tasks have been switched to the new system. Loading OSM files is one example of a task that will still block the GUI, but I will gradually improve these things.

You can download the latest release at http://maperitive.net/download/

Enjoy!

Example URL: http://somewhere.else.com:9999/service/packages?page=2#fragment

The table below contains values for various properties of Uri for that example URL:

A design problem I was trying to solve yesterday: I have a component which depends on two implementations of the same interface:

public class HelpCommand (    
   ICommandDescriptor oneLinerDescriptor,     
   ICommandDescriptor detailedDescriptor)     
{     
   ...     
}     

HelpCommand prints out human-readable descriptions of various commands in Maperitive. If the user specifies the exact command, HelpCommand prints out a detailed description. On the other hand, if the command parameter is not specified, HelpCommand lists all of the commands and prints a one-liner description for each of them. So there are two implementations of the ICommandDescriptor interface: OneLinerDescriptor and DetailedDescriptor classes.

The problem I was having with this design is that all of my *Command components are registered in one go, using the convention like

container.Register (AllTypes.Of<ICommand>().FromAssembly (typeof(ICommand).Assembly)    
   .Configure (c => c.Named (c.Implementation.Name).LifeStyle.Transient));

so I’m letting Windsor do all the wiring magic without me being able to provide any custom registration logic for individual commands like HelpCommand. So how do I tell Windsor which component to use for oneLinerDescriptor and which for detailedDescriptor?

Bad Solution

Of course, I could simply specify the concrete implementations in HelpCommand’s constructor:

public class HelpCommand (    
   OneLinerDescriptor oneLinerDescriptor,     
   DetailedDescriptor detailedDescriptor)     
{     
   ...     
}     

But this couples the three classes together and goes against the “rely on abstractions, not implementations” mantra. It also makes unit testing of HelpCommand more difficult, since I have to drag the two descriptor classes around (and they could have their own dependencies to satisfy, which further bloats the unit testing code).

Factories?

I could use a factory to get the implementations I need, but that would be wasteful: a simple constructor dependency is much slicker and it doesn’t introduce an additional component (factory) into the picture. Also, factories are more suited for components which are created later, on-demand. In my case I want the descriptors to be available right at the beginning of HelpCommand’s lifecycle.

Resolving By Name

So I got an idea: why not use the name of the constructor parameter as an indicator of what kind of implementation I need?

This requires writing a custom subresolver:

    public class SubDependencyResolverByName : ISubDependencyResolver
    {
        public SubDependencyResolverByName(IKernel kernel, string componentPrefix)
        {
            this.kernel = kernel;
            this.componentPrefix = componentPrefix;
        }

        public bool CanResolve(
            CreationContext context, 
            ISubDependencyResolver contextHandlerResolver, 
            ComponentModel model, 
            DependencyModel dependency)
        {
            if (dependency.IsOptional)
                return false;

            return FindCandidateComponent(dependency.DependencyKey, dependency.TargetType) != null;
        }

        public object Resolve(
            CreationContext context, 
            ISubDependencyResolver contextHandlerResolver, 
            ComponentModel model, 
            DependencyModel dependency)
        {
            ComponentModel dependencyResolvingComponent = FindCandidateComponent (dependency.DependencyKey, dependency.TargetType);
            return kernel.Resolve(dependencyResolvingComponent.Name, dependency.TargetType);
        }

        protected ComponentModel FindCandidateComponent (string dependencyKey, Type targetType)
        {
            foreach (GraphNode node in kernel.GraphNodes)
            {
                ComponentModel visitedComponent = (ComponentModel) node;
                if (visitedComponent.Service != targetType)
                    continue;

                if (componentPrefix == null)
                {
                    if (0 == string.Compare(visitedComponent.Name, dependencyKey, StringComparison.InvariantCultureIgnoreCase))
                        return visitedComponent;
                }
                else
                {
                    if (false == visitedComponent.Name.StartsWith(componentPrefix, StringComparison.InvariantCultureIgnoreCase))
                        continue;

                    string nameWithoutPrefix = visitedComponent.Name.Substring(componentPrefix.Length);
                    if (0 == string.Compare (nameWithoutPrefix, dependencyKey, StringComparison.InvariantCultureIgnoreCase))
                        return visitedComponent;                    
                }
            }

            return null;
        }

        private readonly IKernel kernel;
        private readonly string componentPrefix;
    }

…and attaching it to the container:

container.Kernel.Resolver.AddSubResolver (
   new SubDependencyResolverByName (container.Kernel, "auto_"));

Results

Initially I had problems with this new resolver: it tried to be too smart and it messed with the existing (non-problematic) component wiring. So I decided to limit its scope only on dependency components which have a container ID starting with a certain prefix (in my case “auto_”). So if I want OneLinerDescriptor and DetailedDescriptor to be auto-detected by the resolver, I need to register them as “auto_OneLinerDescriptor” and “auto_DetailedDescriptor” in the Windsor container (casing is ignored). The resolver is currently limited to constructor dependencies only, but I see no reason it shouldn’t work for property injections too.

Is name-matching a good approach to auto-wiring components? I don’t know, time will tell. I certainly feel there’s something missing in how Windsor resolves dependencies by the default: type-matching is fine for simple scenarios, but as soon as you have more than one implementation of the same service interface, you need to specify service overrides or rely on "black magic” to wire things for you.

Example request URL: http://localhost:65107/Services.svc/services?page=30

The table below contains values for various path-related properties of HttpRequest for that example request URL:

string applicationFullPath = Assembly.GetExecutingAssembly().Location;
string applicationDirectory = Path.GetDirectoryName(
    Path.GetFullPath(applicationFullPath));

log.InfoFormat(
    "Setting the current directory to '{0}'...", 
    applicationDirectory);
Environment.CurrentDirectory = applicationDirectory;

This helps avoiding problems with .NET framework implicitly accessing various file resources stored together with your binaries. I had a situation where I wanted to validate an XML file using XSD schema files stored in my application directory. The code worked OK in the integration tests but failed when run inside a Windows service. It turns out .NET Framework’s XML validation code wanted to import certain dependent XSD schema files from the current directory, which is, in the case of Windows services, C:\Windows\System32 by default. Of course, the needed XSD files weren’t there, but the validation code doesn’t really tell you this, it just reports an unhelpful validation error.

I’m working on a text-on-path algorithm for Maperitive to be able to render things like street names. GDI+ does not have a built-in function for drawing texts on a path, so I’m stuck with writing my own.

Although there are a lot of implementations of such an algorithm out there, they all miss one important thing: kerning. Kerning is basically telling a character to move a bit closer to its neighbor if there is enough space for them to huddle together:

(image taken from Wikipedia)

Since the path can consist of several line segments (and even curves), you need to render each character (glyph to be more precise) separately to be able to rotate it using the angle of the line segment it will be drawn on. The problem is determining how far from the previous character to place the next character. GDI+ offers a method called MeasureString, but calling it on a single isolated character returns a character width value without any kerning taken into account, so characters like V and A can appear too spread apart. There is another method, MeasureCharacterRanges, which can measure each individual character width for a given text, but it has a practical limit of 32 characters and is pretty processor- and resource-intensive. I’ve used it for the text-on-path algorithm for Kosmos but the results weren’t very satisfactory.

From my (basic) understanding of how kerning usually works on computer typography, a font which has kerning information stores kerning offsets for character pairs. So, for example, V and A will have a different (usually stronger) kerning than say, character pair V and T. The problem with GDI+ is that you don’t really have access to this information.

One option would be to use old GDI’s font and text functions to fetch the information about the kerning pairs, but these need to be P/invoked and thus will probably not work on Mono, which makes them unusable for Maperitive – I’m really trying to make Maperitive run on multiple platforms.

So after spending a day in brainstorming, I got an idea on how to handle kerning myself. It’s really simple: I will calculate the kerning myself (using the mentioned MeasureString method). The formula is simple:

<character kerned width> = <character non-kerned width> – (<character pair non-kerned width> – <character pair kerned width>)

I can get all of the three values on the right of the equation using the MeasureString: first I’ll measure each character’s individual (non-kerned) width and then I’ll measure the width when they are drawn together. The difference between these is the actual kerning, which is then used to calculate the kerned width of the first character.

The added twist to this tale is that I’ll keep this information in a cache, to keep the expensive calls to the MeasureString method to the minimum. This cache could even be persisted to the disk. Of course, each font family and font style dictates its own kerning, so the cache needs to be extended with this information.

One potential problem is how to handle kerning information for different font heights. Since Maperitive supports continuous zooming levels, the number of different font heights used is (almost) limitless. I’ve decided to cut some corners here: the kerning will be calculated (and stored) for some nominal font height (say 100 em) and then interpolated for any needed height. We’ll see how that goes… next time.

photo credit: wok

In the previous post we discussed various relationships between singletons and transients in an inversion of control container. I introduced WindsorContainerInspector, a little utility class for analyzing the container and showed a sample unit test which looks for singletons that rely on transient objects.

Now I’ll show a few other checks that I think are good to perform on the container. They involve disposability of objects.

Disposable Transients

Let’s say we have a transient service IWebClient:

public interface IWebClient : IDisposable
{
    void DownloadMeSomething(string somethingUrl);
}

As you can see, the service implements IDisposable. Now let’s say we have a class DownloadData which depends on this service:

public class DownloadData
{
    public DownloadData (IWebClient webClient) {...}

    ...
}

Note the constructor dependency on IWebClient. What’s wrong with this picture?

What happens when the DownloadData is no longer used? The webClient is not disposed. We could hold up some valuable system resource by not disposing of such an object. And since we know IWebClient is defined as transient in our IoC container, we can be pretty sure that DownloadData is the only one using the received IWebClient instance. So we are responsible for this instance!

In other words, our DownloadData should also be disposable. We can say that, in general, components that depend on disposable transients should also be disposable:

[Test, MultipleAsserts]

public void TransientsThatRelyOnDisposableTransientsShouldAlsoBeDisposable ()

{

    Type disposableType = typeof(IDisposable);

    IList<KeyValuePair<ComponentModel, ComponentModel>> dependencies = inspector.FindDependencies (

        (a, b) =>

        {

            return a.LifestyleType == LifestyleType.Transient && false == disposableType.IsAssignableFrom (a.Implementation)

                && b.LifestyleType == LifestyleType.Transient && disposableType.IsAssignableFrom(b.Service);

        });

    foreach (KeyValuePair<ComponentModel, ComponentModel> dependency in dependencies)

    {

        Assert.Fail (

            "Transient component {0} relies on the disposable transient component {1}, but it itself is not disposable",

            dependency.Key.Implementation.FullName,

            dependency.Value.Implementation.FullName);

    }

}

Some might say that this kind of check can be done by static analysis like FxCop. FxCop will detect this only if DownloadData stores the IWebClient in a class field. But we could have used IWebClient parameter just within the constructor and then forget about it:

public class DownloadData
{    
    public DownloadData (IWebClient webClient) 
    {
        webClient.CheckConnection();
    }    
}

In such cases no warning will be issued by FxCop.

Disposable Implementations Of Non-Disposable Services

What if we had another service, called IMailChecker:

public interface IMailChecker
{
    bool AnyNewMail();
}

and we implemented a GMailChecker:

public class GMailChecker : IMailChecker, IDisposable
{    
    public bool AnyNewMail() {...}
    public void Dispose() {...}
}

Notice GMailChecker implements IDisposable in addition to IMailChecker interface. But IMailChecker itself is not defined as disposable. How will the user of IMailChecker service know whether the object it holds should be disposed or not?

One way would be to use reflection to determine this. But this is very cumbersome and easy to forget when coding. My way of thinking is that if there’s reasonable possibility the implementations of the service will require disposability, you should declare the service interface as disposable. We can check for this condition with the following unit test:

[Test, MultipleAsserts]

public void TransientsThatRelyOnDisposableTransientsShouldAlsoBeDisposable ()

{

    Type disposableType = typeof(IDisposable);

    IList<KeyValuePair<ComponentModel, ComponentModel>> dependencies = inspector.FindDependencies (

        (a, b) =>

        {

            return a.LifestyleType == LifestyleType.Transient && false == disposableType.IsAssignableFrom (a.Implementation)

                && b.LifestyleType == LifestyleType.Transient && disposableType.IsAssignableFrom(b.Service);

        });

    foreach (KeyValuePair<ComponentModel, ComponentModel> dependency in dependencies)

    {

        Assert.Fail (

            "Transient component {0} relies on the disposable transient component {1}, but it itself is not disposable",

            dependency.Key.Implementation.FullName,

            dependency.Value.Implementation.FullName);

    }

}

Below is the updated the implementation of WindsorContainerInspector. Next time we will be discussing issues (= mostly problems) around disposing stuff received from the Windsor container.

public class WindsorContainerInspector

{

    public WindsorContainerInspector(IWindsorContainer container)

    {

        this.container = container;

    }

    public IList<KeyValuePair<ComponentModel, ComponentModel>> FindDependencies(

        Func<ComponentModel, ComponentModel, bool> dependencyPredicate)

    {

        List<KeyValuePair<ComponentModel, ComponentModel>> dependencies = new List<KeyValuePair<ComponentModel, ComponentModel>>();

        foreach (GraphNode node in container.Kernel.GraphNodes)

        {

            ComponentModel dependingNode = (ComponentModel) node;

            foreach (GraphNode depender in node.Dependents)

            {

                ComponentModel dependerNode = (ComponentModel)depender;

                if (dependencyPredicate(dependingNode, dependerNode))

                    dependencies.Add(new KeyValuePair<ComponentModel, ComponentModel>(dependingNode, dependerNode));

            }

        }

        return dependencies;

    }

    public IList<ComponentModel> FindComponents (Func<ComponentModel, bool> predicate)

    {

        List<ComponentModel> components = new List<ComponentModel>();

        foreach (GraphNode node in container.Kernel.GraphNodes)

        {

            ComponentModel component = (ComponentModel) node;

            if (predicate(component))

                components.Add(component);

        }

        return components;

    }

    private readonly IWindsorContainer container;

}