The challenge, to summarise, is to devise an application which can search through very large log files for lines which match the time (the name of the challenge will give away the purpose for those familiar with the grep tool). The challenge focuses specifically on HAProxy logs and although they’re not hard to generate yourself they do provide a sample perl generator. The main behaviour of the application is that a user should be able to provide either a time or time range and optionally a log path (with a default) and have the lines within the scope of the log file returned:

$ tgrep 8:42:04 

[log lines with that precise timestamp] 

$ tgrep 10:01 

[log lines with timestamps between 10:01:00 and 10:01:59] 

$ tgrep 23:59-0:03 

[log lines between 23:59:00 and 0:03:59]

For the purpose of this challenge, the log files can be assumed to have been rotated every 24 hours and you will therefore only need to worry about the time of a line and not a date changeover.

A successful application in the strict interpretation of this challenge would also need to work on Linux and would therefore likely need to be compiled with Freepascal.

What I like about this challenge is that there are many levels of solution which you could dive into. Clearly, as the challenge states, you are unlikely to achieve any real speed when searching through log files of many gigabytes if you perform a linear search. The size of the files involved will also cause problems any solution which isn’t careful about memory consumption.

For my first iteration I initially opted for a straightforward interpolative search, which turned out to be much more buggy than I’d originally thought however luckily, I had written my tests first even though I had to resort to a form of integration testing instead of using mock objects (Nick Hodges would not be happy!). It’s a shame that none of the current Mocking libraries appear to be compatible with Delphi 2009 so I thought I’d leave the mocks until a later iteration. My initial iteration works and although I haven’t performance tested it fully yet, I fully expect it to pale in comparison with some of the excellent tgrep entries made available via github. Of particular note is EvansBee’s C version, which in the few tests that I ran was very quick.

For my second iteration, I expect that it can be optimised in real-world conditions by taking account of the peaks and troughs in traffic that I would imagine reddit get during certain intervals of the day.

If anyone feels like they might like to give the tgrep challenge a go with some absurdly clever way of tackling this problem in delphi/fpc then feel free to comment/get in contact. It might be fun to put them all up on github so that we can all compare and contrast. If someone was really clever then they might even be able to hookup some kind of clever super project which builds a bunch of git submodules and tests them all.

Links

• tgrep challenge – the original job ad, explaining the details.
• A perl log sample generator – It is perl, but it works.
• github tgrep projects – Some of these contain some interesting ideas, if you’re stuck for an approach.

Holy cow, all those flashy words in a single description, that must mean this is some enterprisey buzzwordy new fangled thing, right? Nope. This is an extension of all those ideas that we know that we “should” and “could” have always been doing, and in fact some of you may already be doing it.

Continuous Delivery, in a nutshell, is about delivering your software in small, very frequent incremental updates through a huge quantity and quality of automation in build, testing and deployment. Traditionally, the release process for many teams culminates in a lockdown period for release followed by many hours or days of manual release process, followed by triage and observation. This is a broken and problematic approach and we all know it.

Instead, continuous delivery is the way forwards for delivering software and in future blog posts I will aiming to cover more of the implementation process, potential problems and cultural impact.

My interest was piqued when I attended a presentation by one of the authors of Continuous Delivery, which as the name would suggest is the currently definitive book on the subject. David and Jez developed the principles for Continuous Delivery at Thoughtworks, where Jez currently also works as the Product manager for Go (yes, neither this Go, nor this Go).

Continuous Delivery is about automating everything from build to deployment into production. This also means that methodologies such as Continuous Integration and Agile testing could be viewed as a specific subset of Continuous Delivery. I also recently attended ‘Evolving Continuous Delivery‘ with the London Continuous Integration Meetup where Chris Read gave us the quote of the night:

Until your code is in production making money or doing what it is meant to do, you have simply wasted your time

The Value proposition for Continuous Delivery is very simple: getting business value into production as quickly as possible. By repeatedly deploying to production in a controlled manner we also:

• Validate the business decisions which we’ve made more quickly by taking small steps towards our goal and gaining feedback before you’ve invested the full cost.
• Allow us to deliver our business value at lower risk because we make smaller changes at each step. Smaller changes are less risky than large packages of change.
• Encourages the culture that nothing is truly “done” until it is delivered to your users and more importantly proven before delivery.

Every build artefact should be considered a potentially releasable artefact, your comprehensive suite of tests are there to prove that it is not suitable. With each test passed, the build should be promoted through the pipeline to the next test. In order to be confident that your release is not risky, you need to be able to test everything in a release. Obviously, this means testing code changes, but also deployment processes and configuration. Being certain of your configuration also means testing your system configuration as well as your application configuration. We’ve all seen how “minor” OS patches can cause massive knock on effects to the performance or stability of an application unknowingly, it’s vital that you should be able to monitor and roll back. When you begin to consider what conditions you may want to roll back or draw attention to your release, you may need add specific monitoring and instrumentation to your application to monitor if you notice:

• Increased Latency
• Increased Load
• Lost Dependencies after a configuration change
• Poorer Business Performance
• Failed Basic Functionality of the application

Finally, deploying continuously may require cultural and process changes to the way you develop your products. In particular, your SCM and branching strategy may need to reflect the need to release partially implemented features as disabled (also known as feature toggles).  A presentation worth watching on this topic is Chuck Rossi on ‘How facebook releases software‘. Your application architecture may also need to be updated to allow continuous deployment in a safe manner.

Continuous Delivery isn’t just for Web Applications (although it, clearly has massive benefits to them) but as the Google Chrome team have demonstrated, can provide great benefits for Desktop applications (even Delphi ones!) too although requires different tools and approaches. Imagine being able to push a series of very efficient diffs to all the users of your application every time it successfully passes a full range of exhaustive tests.

Many companies currently use some form of Continuous Deployment to manage their operations, such as Flickr, Etsy, Netflix and more who I’ve forgotten. Continuous Delivery is a topic that impacts all areas of operations, development, marketing, and possibly any regulatory concerns. I intend on diving into detailed posts of various areas of the overall topic.

Further Reading

• Continuous Delivery by Jez Humble and Dave Farley – The Book
• Jez Humble at DevOps Days 2010 – A presentation by Jez on Continuous Delivery
• Push the Button – A recent talk by Sam Newman, a principle consultant at Thoughtworks on Continuous Delivery.
• The Value Proposition – An article about the value proposition on Continuous Delivery
• Continuous Delivery using Build Pipelines and Ant – A great end to end article on the Build Tools side by James Betteley, who I met at the London CI meetup.

I would like to thank my good friend Kingsley Davies for encouraging me to pickup this vein of blogging.

I had hosted my Delphi headers for the yajl json library on Github at Delphi-yajl, which were sadly in a broken state as I was unable to solve some of my conversion problems myself. I recently received a pull request on Github for my project from the ever active Delphi’er on github Jason T Penny. He very kindly forked my Delphi-yajl repo and submitted a set of changes which fixed up my botched conversion job and pushed it back. As a result, I’m pleased to announce that the Delphi yajl code is in working order and I now plan on adding a higher level wrapper which allows slightly more Delphi-like usage but if you need lightning fast SAX-style json parsing then feel free to try it out. I can imagine that it would be perfect for situations where you need to parse a stream of json, such as if you were streaming the Twitter Streaming API.

He also has some pretty cool other repos on his Github profile, including his GitForDelphi Project. GitForDelphi is a conversion of the libgit2 headers. libgit2 is portable, pure C implementation of the core git methods which allow you to effectively implement git functionality without having to call git command line (much cooler). The Delphi bindings are in a working state already and in order to return the favour, I do plan on helping him out by contributing some tests. I would encourage all Delphi developers to have a hunt around on Github as there are some interesting little gems to be found and it’s a very cool community to explore.

Further Reading

• An introduction to Git -The Github introduction to Git.
• A Bootcamp guide to Gtihub – The best guide to getting involved to github.
• libgit2 – The amazing native linkable implementation of git.
• Delphi-yajl – Delphi header translations for yajl
• yajl – The yajl (Yet Another Json Library) home.

The F# compiler and libraries were also recently released as Open Source under the Apache 2.0 license which should make for some exciting developments in the future, not to mention better support for Mono.

I also found F# a good way to understand the slightly functional constructs that appeared in Delphi Prism 2011. I’ve only just started beginning to learn F# but I thought I would highlight some of the particularly interesting differences that I’ve found compared to Delphi.

Identifiers and let bindings

Identifiers are the way which you assign names to your values so that you can reference them later, using the let keyword. To a Delphi programmer this may sound like the concept of variables, however there is one important difference in F#: Variables are immutable and therefore normally can’t be changed and once a value has been assigned to it. Consider the following:

open System

let x = 1
printfn "%d" x

let x = 2
printfn "%d" x

This will produce a compiler error where we attempt to reassign a new value to the identifier x. F# shares the type inference approach of C# 3.0 onwards so you do not need to explicitly declare the types of your variables as you do in Delphi (while Delphi Prism shares the C# approach). It should also be noted that as in Delphi, you can also assign identifiers to methods. For example, in Delphi you might do something like the following:

var
  i: integer;
  myAdder: TFunc<integer, integer, integer>;
begin
  myAdder :=
    function(a, b: integer): integer
    begin
      result := a + b;
    end;
  i := myAdder(1, 2);
end;

The equivalent F# code might look like this where F# Functions are values in their own right anyway:

// Could also be written in longform as:
// let myAdder = fun a b -> a + b

let myAdder a b = a + b
let i = myAdder 1 2

If you’re coming from Delphi, it is also worth noting that identifier names are case-sensitive.

Scope

F# defines scope, by default, using whitespace and indentation creates new scope, and the end of that particular scope is signaled by the end of the indented block. The following example demonstrates this, by reusing the identifier ‘z’ as a midway calculation half way through a block.

let z = 1

let y =
    let z = 3 * 2
    z / 2

printfn "%d" y

This code also demonstrates that like the Ruby language, you do no need to explicitly state a return value for a method. This can be quite freeing in many ways.
It should also be noted that you can redefine identifiers with the let keyword whilst in the scope of a function.

Pattern Matching

Another favoured building block for functional programming is pattern matching. A very simple example of this might be compared to series of case or if statements in Delphi. For example, if we look at an implementation of the FizzBuzz problem in Delphi:

	// if ((i mod 15 = 0)) then
        if ((i mod 3 = 0) and (i mod 5 = 0)) then
        begin
          WriteLn('FizzBuzz');
        end
        else if (i mod 3 = 0) then
        begin
          WriteLn('Fizz');
        end
        else if (i mod 5 = 0) then
        begin
          WriteLn('Buzz');
        end
        else
        begin
          WriteLn(i);
        end;
      end;

And then using pattern matching in F#:

// FizzBuzz Solution - @jamiei
open System

let FizzBuzz() =
    let nums = [1..100]
    let printbuzz x =
        match x with
        // | x when x%15=0 -> printfn("FizzBuzz")
        | x when x%3=0 && x%5=0 -> printfn("FizzBuzz")
        | x when x%3=0 -> printfn("Fizz")
        | x when x%5=0 -> printfn("Buzz")
        | _ -> printfn "%d" x;
    nums |> List.iter(fun x -> printbuzz x)
FizzBuzz()
let _ = Console.ReadLine()

Pattern matching must either be complete (i.e. must match all possible values of an identifier) or incomplete and include a wildcard (“_” in the above example) or the compiler will raise an exception at runtime. The compiler will also emit a warning for rules which will never be matched.

let rand = new Random()
let randNum = rand.Next(0, 100);
let coinToss =
    match randNum % 2 = 0 with
    | true -> "heads"
    //| false -> "tails"
printfn "%A" coinToss

Pattern matching may appear to be a simple tool but it’s an extremely concise way of controlling the flow of an application, particularly to get rid of what would otherwise turn into a spaghetti ball of if statements. Pattern matching can also be active, which means that they allow you to execute a function to determine the match.

Units of measurement

If you were listening carefully to the Podcast of Delphi.org when marc hoffman was talking to Jim McKeeth about the future of Delphi Prism, you’ll remember that he mentioned that this might be coming to Delphi Prism. This represents a pretty nifty way of marking the units of a value which makes you less likely to accidentally make a mistake when combining them with less effort than in Delphi where you would have to declare and create a class for each. The following code in F#, will throw an exception:

[<Measure>]type centimetre
[<Measure>]type inch
let distanceToMyThumb = 1<inch>
let distanceToMyOtherThumb = 2.5<centimetre>
let newVol = distanceToMyThumb + distanceToMyOtherThumb

Summary

F# is a very expressive language and the more I read and use it, the more I find that the expressions and constructs that it allows give me new ways to think about certain problems. It is also great that it is built upon the .NET Framework and can interact fully with the standard .NET libraries. My F# test modules can also interact with and be interacted by any Delphi Prism applications that I need it to. If your curiosity is piqued by functional programming and you are already familiar with the .NET framework then F# is a great way to try your hands.

Further Information

You can find more information on F# here:

• F# MSDN Home
• F# Community Samples
• An Introduction to Functional Programming for .NET Developers

MEF is a new library in .NET Framework 4 and Silverlight 4 that addresses the problem of easily extending and componentising applications by providing a nifty framework. Adding an easy ability to extend your application, whether it be for the purposes of componentising your application or allowing others to extend your application has always been available to Delphi users in the guise of frameworks like RemObjects Hydra, the TMS Plugin Framework or building your own homebrew solution but MEF is very nice framework.

MEF is incredibly powerful and I’ll only really be scratching the surface and introducing it here.

Imagine an application which runs a sequential series of processes on an imaginary input file, not exactly rocket science but this is only a basic introduction after all.

First I begin by defining an Interface in a separate assembly which all my plugins will adhere to:

  IFileAction = public interface
    method ProcessFile(path: String);
    property Name: String read;
    property Version: String read;
  end;

We then write our first plugin, which we’ll conveniently call the MyHelloPlugin. First create a new assembly and reference the base plugin assembly which contains our IFileAction interface and the System.ComponentModel.Composition assembly. We also need to ensure that we reference these in our uses clause of the unit containing our Hello Plugin class. The My Hello Plugin class looks like this:

  [Export(typeof(IFileAction))]
  MyHelloPlugin = public class(IFileAction)
  const
     MYNAME = 'The My Hello Plugin';
     MYVERSION = '1.0.1';
  public
    method ProcessFile(path: String);
    property Name: String read MYNAME;
    property Version: String read MYVERSION;
  end;

This should be pretty self explanatory but we have two properties for our name and version and a ProcessFile method which will be called by our host app, providing it with a path for the file that we’re processing. You’ll notice the Export attribute which denotes what we’re exporting. We’re specifying the type of the export but in many cases you may not even need to do this. MEF has a very powerful ExportMetaData system for providing information about what you’re exporting which is also worth reading up on.

Next, of course, we need to actually consume the plugins in our main application. To manage my collection of plugins, I’ll create a small, basic class to hold and manage them:

  PluginList = class
  public
    [ImportMany(typeof(IFileAction))]
    property FileActions: IList<IFileAction> read write;
    constructor;
  end;

Here I’m using the ImportMany attribute to indicate that I’m expecting to import many IFileAction instances and then providing a list with which to populate (which, as always, you must remember to instantiate before you attempt to fill it!).

Next I’ll load up my Hosting catalog and compose the parts of my application.

  var aggregateCat := new AggregateCatalog();
  // var catalog := new AssemblyCatalog(System.Reflection.Assembly.GetExecutingAssembly());
  // var catalog := new AssemblyCatalog('plugins\MyHelloPlugin.dll');
  var dirCatalog := new DirectoryCatalog('plugins');

  // Add our Catalogs here,
  aggregateCat.Catalogs.Add(dirCatalog);

  var container := new CompositionContainer(aggregateCat);
  // Create our plugin hosting object
  var pluginList := new PluginList();
  // Compose the parts.
  container.ComposeParts(pluginList);

You’ll note that I create an AggregateCatalog which isn’t strictly necessary in this case because I’m only adding one ComposablePartCatalog to it but in other uses this would also allow me to separately load single Assemblies or a collection of DirectoryCatalog’s. I then load the catalog into a composition container before creating my aforementioned PluginList class and finally pulling all the parts together. It really is that simple. You can test what has been loaded using our now populated PluginList.FileActions collection, like this:

  for each plugin: IFileAction in pluginList.FileActions do
  begin
    Console.WriteLine('Loaded ' + plugin.Name + ', version ' + plugin.Version);
  end;

You can now create a new and entirely different plugin assembly (or many, many more) and providing it conforms to our IFileAction interface and is either in our target directory or loaded manually, it will now appear in our list as if by magic.

Once you’ve begun using MEF, you’ll find out how powerful it can be and how easy it makes building modular applications.

If anyone is interested in me uploading the complete project set for this sample then please let me know in the comments below and I can upload it to github. If anyone else has any excellent samples of MEF usage that they can publish we could even start a sample repository.

Further Reading

• The MEF Architecture Overview
• The MEF Programming Guide
• The MSDN System.ComponentModel.Composition Guide

I still believe that participating in these communities and making visible “noise” does credit to the delphi community. Some of you may remember that I extolled the virtues of this point just over a year ago in another post, The Delphi Community: From the outside.

This is all particularly relevant in a month when, the largest community gathering, delphi.non-technical has been filled to the brim with little which might give a potential newcomer to the community any kind of welcome feeling what-so-ever. Also this month a set of stats generated from the GitHub API showed that Delphi was the 39th largest language on there, another online property that I advocated our further involvement with. Without saying that we should be much higher, I would say that I would like to see more Open Source Delphi projects on there. We could also do with more of a vocal presence in many other areas too but I guess this requires organic community growth (allbeit slightly prompted).

The recently unveiled previews of RAD Studio XE may not be the cross-platform upgrade that some people hoped it would be, it does represent the next product version of the Delphi we know and love. If what the Embarcadero staff are saying in the newsgroups of the quality this release is true then it will be a worthy addition to the family (if not as groundbreaking as we’d originally hoped). Indeed I liked the look of the new integrated tools for the Delphi IDE revealed today and will await the next previews but even if that isn’t your cup of tea it still leaves plenty to shout about.

What I’m advocating is that we retain a little perspective and focus on what is in the Delphi XE release and not what isn’t and use this as a conversation starting point to start making more noise about Delphi in “public places” with a view to reinvigorating the community. Which reminds me: what other communities/online properties could the Delphi community benefit from making better use of?

In other news: I’ve also got what I consider to be an exciting Delphi community based side-project which you may all potentially get to see very soon…

Visual Studio 2010 Support

I’ll start with the most obvious change: Delphi Prism now supports Visual Studio 2010 which was only released just over a month ago. The Visual Studio 2010 IDE looks great, feels faster than 2008 and most importantly, in my experience, is very stable.

Visual Studio 2010 with Delphi Prism

The shell is based on WPF and has had considerable work put into allowing it to be flexible and extensible which I understand has allowed a whole host of new plugins to flourish. It’s an extremely minor point but I’ve also found it much better in allowing themes for the code editor, springing forth sites like http://studiostyles.info/. My favourite scheme  is Son of Obsidian which works marvellously for C# code and works reasonably well for Delphi Prism code too. It’s great that we can take advantage of community extras like this, although it would also be great if we could tailor some of these themes specifically for Delphi Prism.

As marc mentioned back on episode #39 of the Podcast at Delphi there are a lot of free minor improvements that come with moving to Visual Studio 2010 (not financially free but that were inherent with the migration).

Other cool stuff includes support for Silverlight 4 and the “Paste C# as Oxygene” feature which allows you to translate code snippets on the fly. This release also includes MonoDevelop which includes support for Mac and Windows.

For, If and Case Expressions

For, if and case I hear you say? Haven’t we had those for as long as we can remember? Not like this we haven’t. Delphi Prism now supports them as expressions. This allows you to do operations like this:

  var sourceList: Array of Integer := [1 ,2 ,4 ,8 ,16 ,32 ,64 ,128];
  var sqList := for each num in sourceList yield num * num;

Now this sample could just as easily be done with a LINQ operation or a separate loop but this is another option available at your disposal and introduces a slightly Functional Programming feel. Delphi Prism now also supports If expressions and my favourite expression addition: case.

These expressions, as with so many language features hold the potential for misuse but they can also make things a little more concise:

method NotifierStr(newMails: integer): string;
begin
  result := 'You have ' + newMails.ToString() + ' new ' +
    case newMails of
      0: 'emails';
      1: 'email';
    else
      'emails';
    end;
end;

Please forgive the horrible string concatenation but I wanted to demonstrate it in this way because I thought it might help make it a little clearer.

Dynamic

Delphi Prism 2011 includes support for the (in some cases) contraversial .NET 4 dynamic type. To quote Scott Hanselman on dynamics: A dynamic is a statically typed dynamic type. Dynamics leave a lot more till run-time and allow a much cleaner interop with the Dynamic Scripting languages built upon the CLR.

For example, the ExpandoObject enables you to add and delete members at runtime, something might be useful when serialising and deserialising JSON or XML where you do not know the spec ahead of time:

dynamic tweetresult := new ExpandoObject();
tweetresult.Id = 1243535345;
tweetresult.Username = 'delphifeeds';

The Delphi Prism 2011 implementation has a required dependency upon a separate assembly, RemObjects.Oxygene.Dynamic.dll, which is freely re-distributable.

For a much better idea of what dynamics are and can be used for, I would refer you to Scott Hanselman’s whirlwind tour of dynamics and the MSDN reference on using dynamics.

Tail Calls

Tail Calls are another feature that could be seen as similar to a Functional Programming Language construct. They are essentially an exit statement followed by a call to a method. They have to be explicitly enabled and disabled for a method using the {$TAILCALL ON} flag. This allows you to code recursive functions, particularly useful for maths based methods. This concept will probably take a bit of getting used to and are something I’d like to cover in subsequent posts.

Tuples

A Tuple is an ordered list of Elements. They can contain 1 to 7 different references and types with an 8th reference being used to reference another Tuple if you need to hold more than 7. They are similar to an array and a class/record but unlike an array can contain more than one type and require considerably less declaration than a class/record. A Tuple in Delphi Prism might be used as follows:

  var telephoneList := new Dictionary<Tuple<string, integer>, String>();
  telephoneList.Add(Tuple.Create('Josh Moore', 1283843), '+0014158343131');
  Console.WriteLine(telephoneList[Tuple.Create('Josh Moore', 1283843)]);

They are particularly useful for returning more than one value from a method where you would otherwise need to explicitly declare a class or record to hold the data otherwise. Tuples are a good way to solve a specific problem and will be more familiar to those who have used them in other languages such as the Tuple support in Python.

Parallel Loops (not actually a new feature)

Parallel loops have been a part of the Delphi Prism (and formerly Oxygene) language for a long, long time. However what I hadn’t appreciated was that the Parallel Task Library upon which it relied was officially bundled into the .NET 4.0 Framework libraries. As such I felt it necessary to re-raise it and show just how nifty it is here even though it isn’t really new to Delphi Prism 2011.

Just to recap, with the parallel keyword you can parallelise a loop.

As a fully fledged .NET 4 language we can also use the AsParallel() method in order to return a ParallelEnumerable source capable of being queried with the .NET Parallel LINQ library.

The rough and ready example below demonstrate these two techniques:

class method ConsoleApp.Main(args: array of string);
begin
  var anagramletters := 'mite';
  var filepath := 'mydictionary.txt';

  var dictList := new List<string>();
  dictList.AddRange(System.IO.File.ReadAllLines(filepath));
  Console.WriteLine('Added ' + dictList.Count + ' lines to the file');

  var sortedList := new Dictionary<string, string>();

  for parallel i: Integer := 0 to dictList.Count - 1 do
  begin
    sortedList[dictList.Item[i]] := dictList[i].Sort();
  end;

  var matchingwords := sortedList.AsParallel().Where(w -> ((w.Value.Length = anagramletters.Length) and (w.Value = anagramletters.Sort()))).Select(w -> w.Key);

  Console.WriteLine('Found: ' + matchingwords.Count().ToString() + ' results');
  matchingwords.ToList().ForEach(x -> Console.WriteLine(x));

  Console.ReadLine;

end;

The code above reads a dictionary file in and then builds an index using a parallel loop before finding the words that are anagrams of the provided string using a Parallel LINQ query.

Extension Methods

I consider it a slightly more minor point as I’ve already briefly talked about the new method of defining Extension Methods, but this is now baked into Delphi Prism 2011 and is a pretty useful tool to have. I used the new Extension Methods syntax in the Parallel Example above to add an implementation specific Sort() method to my strings like so:

interface

extension method String.Sort : String;

implementation

extension method String.Sort : String;
begin
  var c := self.ToCharArray();
  Array.Sort(c);
  result := new string(c);
end;

Summary

Overall Delphi Prism 2011 is a pretty significant release and contains lots of new features that represent a lot of interesting new potential both on the language side and the IDE side, congratulations to the RemObjects team.

Further Reading

• What’s new in Delphi Prism 2011 – The Official Embarcadero page.
• New Features – The Delphi Prism Documentation Wiki page listing newly added features.
• Delphi Prism 2011 Release Notes – Is what it says on the tin.
• Using Type dynamic in .NET 4.0 – MSDN reference on the dynamic type.
• What is tail-recusion – StackOverflow answer on Tail Recursion which might help you to better understand Tail Calls in Delphi Prism.

I have been toying with the idea of trying to convert the yajl parser bindings to Delphi in order to build a wrapper on top of the original C dll. yajl (Yet Another JSON Library) is a small fast SAX style JSON parser written and open sourced in C over at lloyd’s yajl GitHub page. I wanted to be able to use this library in particular because it’s very fast, small, portable and able parse from a stream. Plus, I had never really done any conversion of this nature on C header files before so I thought it might be a small, easy project to start with… I was a perhaps a little naive on this front!

I’ll leave you with a disclaimer early on: the code that I’ve been working on and have shared is not currently in a working state. This is the first part of my journey and I do intend on posting further as I learn from the major mistakes that I make in the process.

Approach

The 3 header files that I need to translate can be found in the src/api directory:

• yajl_common.h
• yajl_gen.h
• yajl_parse.h

I’m not sure what a *typical* process for conversion might look like but I started out by trying to use Dr Bob’s HeadConv tool as a way of getting me started and doing some of the legwork. However, the callback structure used by the parsing callbacks confused the tool and it never produced anything of use. After this setback I tried to install Rudy Velthuis‘s Conversion Helper Tool but I wasn’t able to load this into the Delphi 2009 IDE.

Having failed to get any of the conversion tools to do any of the work for me, I accepted my task and began to convert the headers from scratch. There were two resources that I found invaluable: the first is the Type Conversion table half way down the page on Dr Bob’s HeadConv page.  The second was the comprehensive Pitfalls of Conversion guide from Rudy.

As I began converting the headers I realised that although I thought I was translating the headers without a problem, the devil is in the detail and as my first major conversion project, I didn’t have the specfic knowledge necessary to correct my specific mistakes. As a result, please correct me if I have misunderstood any bits which I have picked up in the process.

Const parameters

One issue that I did notice is that there were quite a few parameters marked const in the header files, such as this declaration in yajl_parse:

   YAJL_API unsigned char * yajl_get_error(yajl_handle hand, int verbose,
                                            const unsigned char * jsonText,
                                            unsigned int jsonTextLength);

Rudy’s guide states that where possible Delphi programmers should ignore the const declaration as it’s easy to get wrong depending on whether the argument is passed by value or by reference, complicated by the issue that it means something subtly different in C. I translated this heading and simply ignored the const keyword:

    Tyajl_get_error = function(handle: yajl_handle;
                               verbose: integer;
                               jsonText: PChar;
                               jsonTextLegnth: Cardinal): PChar; stdcall;

Multiple Indirection Parameters

There was one declaration that I had to do a bit of searching for: it was in the yajl_gen file, in the declaration of yajl_gen_get_buf function:

YAJL_API yajl_gen_status yajl_gen_get_buf(yajl_gen hand,
                                              const unsigned char ** buf,
                                             unsigned int * len);

I had never seen the double asterisk before and found it surprisingly difficult to reliably Google for it. Luckily, this comprehensive page on C pointers was pretty useful in explaining that it is in fact a technique called multiple indirection which essentially creates a pointer to a pointer, in this case to a char.

    Tyajl_gen_get_buf = function (handle: Tyajl_gen;
                                 out buf: PChar;
                                 out len: Cardinal): yajl_gen_status; stdcall;

I wasn’t sure if this was indeed the correct translation but if you know otherwise then please let me know in the comments below.

Record Alignment and Enum size

When reading up on the perils of record alignment and enum sizes I found this blog post from Vlad loan Topan on Record Alignment and Enum sizes. Both problems are potentially applicable to the yajl header files. In his post, Vlad mentions that in order to line up the enum sizes you will need to use the {Z4} compiler directive to change the size from the Delphi default of 1 to the C compatible 4 as so:

type
  {$Z4}
  yajl_status = (
                  // no error was encountered
                  yajl_status_ok,
                  // a client callback returned zero, stopping the parse
                  yajl_status_client_canceled,
  ... snip ...

Reading through the Record alignment section of Rudy’s article I understood that I needed to search through the yajl project source to look for an Alignment setting. However, I couldn’t find a reference to the pragma directive anywhere in  the  source.

Next Steps

Well, it doesn’t actually work yet but I have put the Delphi-yajl project on GitHub where you can view my efforts so far, fork it and hopefully provide feedback or code contributions to show me where I’m going wrong. I have also setup a Delphi-yajl page here to serve as a placeholder for when it actually works!

Futher Reading

• yajl – The yajl project home page.
• Dr Bob’s Header Conversion Tool – The useful tool from the ever excellent Dr Bob.
• The Pitfalls of Conversion – Rudy Velthuis’s long but excellent article on commonly encountered problems when converting
• Conversion Helper Package – The Conversion package helper from Rudy but I’m uncertain as to how compatible it is any more.

So what is the Rx Framework?

It’s a framework for asynchronous programming from Microsoft Research (including Erik Meijer & Wes Dyer amongst others) which could potentially be para-phrased as LINQ to Events. The basic problem is that Asynchronous programming is hard, the exacerbating issue for this problem is that it is entirely necessary for modern development. It is pretty common for Desktop applications to need to chain a series of events together with the correct order and sometimes correct time but doing so at present can land you in a spaghetti-like sea of event callbacks. Enter the Microsoft Rx Framework.

What do I need?

The Framework requires a minimum .NET Framework version of .NET 3.5 SP1 or .NET 4 but downloads for both versions are available on the Microsoft Research site. I couldn’t work out why but Delphi Prism appears to have a problem with the Libraries use of Extension Methods but this issue appears to be fixed in the latest Beta of Delphi Prism. I now suspect it might have started working as a result of the updated support for Extension methods which marc hoffman recently posted about.

How does it work?

In Brief: The Rx Framework introduces a of a pair of interfaces, the IObserver & IObservable, which allow you to represent push-based observable collections, along with a library of extension methods (the bit that wasn’t previously working in Delphi Prims) that implement the Standard LINQ Query Operators.

And…

I always find that the easiest way to understand it is to view some examples. First you’ll need to make sure reference the System.CoreEx and System.Reactive assemblies.

As with always, we’ll start with an entirely useless example just to demonstrate a basic example which uses an Observable Timespan interval to print out a number which doubles every 2 seconds:

class method ConsoleApp.Main(args: array of string);
begin
  var oneNumberPerSecond := Observable.Interval(TimeSpan.FromSeconds(2));

  var numbersTimesTwo := from n in oneNumberPerSecond
                        select n * 2;

  Console.WriteLine('Numbers * 2:');

  numbersTimesTwo.Subscribe(num -> begin Console.WriteLine(num) end);

  Console.ReadKey();
end;

Clearly, this is a contrived example but I think this hints at a few of the more powerful features and versatility of the Rx Framework. The first thing to note is that this is an Asynchronous process, imagine the same code using the standard .NET Timer class. This is a demonstration of the power of the Observable Interfaces and how it can allow standard interfaces to concepts that would otherwise complicate our code such time windows to be easily expressed. The second lines to notice is the highlighted LINQ To Events code on lines 5 and 6 which show how easily you can filter specific events from any events. This is a truly powerful and important concept because it is fairly common to generate a stream of events, only some of which we want to react to and the Linq extensions allow us to express this in a succinct way.

This is only a basic console application but it’s not hard to imagine how this could make the complex ordering and timing of GUI events in many applications much easier. As another example I have translated the C# ISubject ping pong example which mimics the classic Ping Pong Actor example that the Scala folks get so excited about.

First of all the Ping ISubject and Pong ISubject classes:

namespace PingPongExample;

interface

uses
  System.Collections.Generic,
  System.Linq,
  System.Text;

type
  Pong = public class (ISubject<Ping, Pong>)
  private
  protected
  public
    method OnError(exception: System.Exception);
    method OnCompleted;
    method Dispose;
    method Subscribe(observer: System.IObserver<PingPongExample.Pong>): System.IDisposable;
    method OnNext(value: PingPongExample.Ping);
  end;

implementation

method Pong.Dispose;
begin
  OnCompleted();
end;

method Pong.OnCompleted;
begin
  Console.WriteLine('Pong finished Ponging');
end;

method Pong.OnError(exception: System.Exception);
begin
  Console.WriteLine('Exception');
end;

method Pong.OnNext(value: PingPongExample.Ping);
begin
  Console.WriteLine('Pong received Ping..');
end;

method Pong.Subscribe(observer: System.IObserver<PingPongExample.Pong>): System.IDisposable;
begin
  result := Observable.Interval(TimeSpan.FromSeconds(2.5))
                .Where(n -> n < 10)
                .Select(n -> Self)
                .Subscribe(observer);
end;

end.

namespace PingPongExample;

interface

uses
  System.Collections.Generic,
  System.Linq,
  System.Text;

type
  Ping = public class (ISubject<Pong, Ping>)
  private
  protected
  public
    method OnNext(value: Pong);
    method OnError(error: Exception);
    method OnFinished;
    method Subscribe(observer: System.IObserver<PingPongExample.Ping>): System.IDisposable;
    method OnCompleted;
    method Dispose;
  end;

implementation

method Ping.OnNext(value: Pong);
begin
  Console.WriteLine('Ping received Pong..');
end;

method Ping.OnError(error: Exception);
begin
  Console.WriteLine('Exception');
end;

method Ping.OnFinished;
begin
  Console.WriteLine('Completed Ping Ponging');
end;

method Ping.Dispose;
begin
  OnFinished();
end;

method Ping.OnCompleted;
begin

end;

method Ping.Subscribe(observer: System.IObserver<PingPongExample.Ping>): System.IDisposable;
begin
  result := Observable.Interval(TimeSpan.FromSeconds(2))
                .Where(n -> n < 10)
                .Select(n -> Self)
                .Subscribe(observer);
end;

end.

And then our setup code:

class method ConsoleApp.Main(args: array of string);
begin
  var ping := new Ping();
  var pong := new Pong();

  Console.WriteLine('Any key to stop..');

  var pingSubscription := ping.Subscribe(pong);
  var pongSubscription := pong.Subscribe(ping);

  Console.ReadKey();

  pongSubscription.Dispose();
  pingSubscription.Dispose();

  Console.WriteLine('Ping Pong Completed..');
end;

You can embrace IObservable in your own business objects quite easily and begin pushing events throughout your application asynchronously:

type
  Invoice = class

  private
    paidDate: DateTime;
    paidSubject: Subject<Invoice>;
  public
    constructor;
    method MarkPaid(paidDate: DateTime);
    property Paid: IObservable<Invoice> read paidSubject.Hide();
  end;

  ConsoleApp = class
  public
    class method Main(args: array of string);
  end;

implementation

class method ConsoleApp.Main(args: array of string);
begin
  var myInvoice := new Invoice();
  myInvoice.Paid.Subscribe(n -> Console.WriteLine('Paid'));
  myInvoice.MarkPaid(DateTime.Now);
  Console.ReadLine();
end;

constructor Invoice;
begin
  Self.paidSubject := new Subject<Invoice>;
end;

method Invoice.MarkPaid(paidDate: DateTime);
begin
  Self.paidDate := paidDate;
  Self.paidSubject.OnNext(Self);
end;

It’s true that the examples above can be replicated quite easily without using the Rx Framework but within a short time of experimenting with the Rx Framework, I would hope that you should be able to see that it allows asynchronous event handling patterns to be expressed in a standard and more expressive manner.

I realise that this post is a little short on more practical examples but in my next post I’ll expand on some slightly more practical examples and some examples which demonstrate the Frameworks Power in handling GUI events in WPF Applications. It is also worth noting that the Rx Team recently released the RxJs framework which brings the Reactive Programming environment to Javascript.

Further Reading

For further information I would recommend:

• The Rx Wiki
• Wes Dyer’s introduction to the Rx Framework
• Wes Dyer and Erik Meijer from MS Research explaining the Rx Framework
• 101 Rx Samples (not quite 101 yet!)

Does the framework appeal to you? As always, if you have any thoughts or questions about using the Rx Framework with Delphi Prism then let me know in the comments below.

Last time, I had created an attribute which could only be attached to methods returning strings and used a home-made caching class, the Aspect looked like this:

namespace AOPCacheLibrary;

interface

uses
  RemObjects.Oxygene.Cirrus.*,
  RemObjects.Oxygene.Aspects;

type
   [AttributeUsage(AttributeTargets.Method)]
  MemCacheAttribute = public class(System.Attribute, IMethodImplementationDecorator)
  private
  public

    method HandleImplementation(Services: IServices; aMethod: IMethodDefinition);
  end;

implementation

method MemCacheAttribute.HandleImplementation(Services: IServices; aMethod: IMethodDefinition);
begin
  // Get a Reference to our Result Value
  var newResult := new ResultValue();

  // Get our MemCache Object
  var cacheType := Services.FindType('AOPCacheLibrary.MemCache');
  // Setup our static methods.
  var getDataProc := new ProcValue(new TypeValue(cacheType), 'getData', [new NamedLocalValue('cacheKey')]);

  // Assign the result of our cache request to a local variable.
  var getDataAssignment := new AssignmentStatement(new NamedLocalValue('strData'), getDataProc);

  // CACHE DATA USED: To assign our Cache'd data to the Method Result
  var useCacheData := new AssignmentStatement(newResult, new NamedLocalValue('strData'));
  // CACHE DATA NOT USED: To assign the result of our OriginalBody to the Method Result.
  var useOriginalMethodResult := new AssignmentStatement(new NamedLocalValue('resultOriginal'), newResult);

  aMethod.SetBody(Services,
    method begin
      var strData: String;
      // Get our Cache Key Name
      var cacheKey := MemCache.getKeyFromMethod(Aspects.MethodName, Aspects.GetParameters);

      try
        // See if we have anything in the cache for this key.
        unquote(getDataAssignment);
        // Replace this
        unquote(useCacheData);
      except
        on E: Exception do
        begin
          var resultOriginal: String;
          // No cache data found, execute the original method body.
          Aspects.OriginalBody;

          // Assign the result of the Original Method to a new variable.
          unquote(useOriginalMethodResult);
          // Send the result of the Original Method to the Cache for next time.
          MemCache.setData(cacheKey, resultOriginal);
        end;
      end;
    end);
end;

end.

Modifying my original code into working with the Microsoft Enterprise Caching black wasn’t particularly hard. If you haven’t worked with the Microsoft P&P Enterprise Library Caching Block before I highly recommend the Quickstart guide. The Enterprise Caching Block allows you to configure your cache using configuration file and change the backing store to suit own your needs. I would certainly recommend tweaking the expiration time and frequency of cache cleaning times before you begin.

Whilst rewriting the code of my Aspect I found it absolutely invaluable to have the Cirrus documentation wiki open and a piece of paper next to my keyboard with the approximate psuedo-code that I was intending to build with Cirrus written on it. I did actually find that despite the fact that the code I was building was not complicated, I did require a reminder of the statements and structure of it.

One additional problem that I did encounter very quickly on my modification was that I had a method which would generate a string key from the method name being called and its parameters which looked like this:

class method CacheKeyFactory.GetKeyFromMethod(aName: String; Args: Array of object): String;
begin
  var argConcat: String;

  argConcat := aName;
  for argTemp in Args do
  begin
    argConcat := argConcat + argTemp.ToString();
  end;

  Result := argConcat;
end;

Which I had intended to inject into the target class using the AutoInjectIntoTargetAttribute which as the name suggests creates the method tagged within the class you are targetting with your attribute. The problem I encountered was that I used this Attribute on a method within an IMethodImplementationDecorator attribute which meant that it worked perfectly when I applied my attribute to just one method within a class but applying my attribute to more than one meant that the AutoInjected method got injected more than once – causing a conflict (Is this a bug? I was told not but it seems odd to me).

[Update: 22/11/2009] I have reported this issue as QC: 79705. [/Update]

Therefore as a workaround, I was forced to move this key generating method into a static class elsewhere in the assembly but maybe someone can work out a way around this limitation.

I also encountered an annoying trait of the Cirrus Framework which is that some errors produce a fairly unhelpful message and lead you to a line in the Project’s build file which whilst being logical can be infuriating when you’re presented with this:

A rather confusing exception and causal code identification.

After a tiny bit of tweaking, My next iteration of the Caching Attribute looks like this:

type
  [AttributeUsage(AttributeTargets.Method)]
  MethodCacheAttribute = public class(System.Attribute, IMethodImplementationDecorator)
  private
  public
    method HandleImplementation(Services: IServices; aMethod: IMethodDefinition);
  end;

implementation

method MethodCacheAttribute.HandleImplementation(Services: IServices; aMethod: IMethodDefinition);
begin
  // Get a Reference to our Result Value
  var methodResult := new ResultValue();

  // Get a reference to our Static Key Factory.
  var cacheKeyGen := Services.FindType('CirrusCachingAspect.CacheKeyFactory');
  // Generate our cache key from the method name and it's parameters
  var getCacheKey := new AssignmentStatement(new NamedLocalValue('cacheKey'), new ProcValue(new TypeValue(cacheKeyGen), 'GetKeyFromMethod', [aMethod.Name, aMethod.GetParameterArrayValue()]));

  // Get our CacheManager Type and our CacheFactory Type
  var cacheType := Services.FindType('Microsoft.Practices.EnterpriseLibrary.Caching.CacheManager');
  var cacheFactoryType := Services.FindType('Microsoft.Practices.EnterpriseLibrary.Caching.CacheFactory');

  // Get a Reference to our Cache Instance.
  var getCacheInstance := new AssignmentStatement(new NamedLocalValue('methodCache'), new ProcValue(new TypeValue(cacheFactoryType), 'GetCacheManager'));

  // Get data from our cache.
  var getDataFromCache := new ProcValue(new NamedLocalValue('methodCache'), 'GetData', [new NamedLocalValue('cacheKey')]);

  // Cast our result from the Cache to the Result Type.
  var valueCastToResultType := new UnaryValue(new NamedLocalValue('cacheData'), UnaryOperator.Cast, aMethod.Result);

  // Assign the result of our cache request to a local variable.
  var getDataAssignment := new AssignmentStatement(new NamedLocalValue('cacheData'), getDataFromCache);

  // Assign our Cache'd data to the Method Result
  var useCacheData := new AssignmentStatement(methodResult, valueCastToResultType);

  // Add our data to the Cache for next time
  var addDataToCache := new StandAloneStatement(new ProcValue(new NamedLocalValue('methodCache'), 'Add', [new NamedLocalValue('cacheKey'), methodResult]));

  // Key not found in the cache, insert the original body and add a call to the Cache.Add() method.
  var notFoundInCache := new BeginStatement();
  notFoundInCache.Add(new PlaceHolderStatement());
  notFoundInCache.Add(addDataToCache);

  // If found in the cache.
  var ifCacheResult := new IfStatement(new BinaryValue(new NamedLocalValue('cacheData'), new NilValue(), BinaryOperator.Equal), notFoundInCache, useCacheData);

  aMethod.SetBody(Services,
    method begin
      var cacheData: Object;
      var methodCache: CacheManager;
      var cacheKey: String;

      // Get out cache instance
      unquote(getCacheInstance);

      // Generate a key for this method call.
      unquote(getCacheKey);

      // Try to retrieve the data from the cache.
      unquote(getDataAssignment);

      // If data from cache = nil then
      //    call original method
      //    add to cache
      // else
      //    return the data from the cache
      unquote(ifCacheResult);

    end);
end;

Which may look quite complex at first but when you look at the actual statements and values being passed around, you can quickly get a feel for how the code works. We can then easily apply this attribute and cache the result of any method in a class automatically like this:

type
  Flickr = public class
  private
    _username: string;
  public
    method Flickr(username: String);
    [Aspect:MethodCache]
    method GetPhotos: XmlDocument;
    [Aspect:MethodCache]
    method GetUsername: String;
    [Aspect:MethodCache]
    method GetAuthLevel(user: String): Integer;
  end;

Where GetPhotos, GetUsername and GetAuthLevel will all have their results cached automatically by our Aspect. You can see the resulting AOP’d code disassembled in the reflector below:

The RedGate .NET Reflector showing our Cirrus doctored caching code.

There is still one slight limitation in our method of key generation which is that it will currently only generate a unique key for a method with parameters if the Types of the methods parameters provide a proper implementation of ToString() that identifies them. Before you mention it, I had considered using the GetHashCode() method for this purpose but the default implementation of the GetHashCode method does not guarantee unique return values for different objects which makes it no more useful than relying on the user to implement the ToString() method.

I have made my implementation available on GitHub as the CirrusCachingAttribute where you can feel free to download and play with the source but please note that you will need to download and install the Enterprise Library from Microsoft first. I have also added a stub to the Code section on my site for the Cirrus Caching Attribute.

There is also a Standard Library of Aspects over on code.remobjects.com which provides a good set of Aspects from which you can learn more about how Cirrus works.