12

Introduction

I have been toying with the idea of trying to convert the yajl parser bindings to Delphi in order to build a wrapper on top of the original C dll. yajl (Yet Another JSON Library) is a small fast SAX style JSON parser written and open sourced in C over at lloyd’s yajl GitHub page. I wanted to be able to use this library in particular because it’s very fast, small, portable and able parse from a stream. Plus, I had never really done any conversion of this nature on C header files before so I thought it might be a small, easy project to start with… I was a perhaps a little naive on this front!

I’ll leave you with a disclaimer early on: the code that I’ve been working on and have shared is not currently in a working state. This is the first part of my journey and I do intend on posting further as I learn from the major mistakes that I make in the process.

Approach

The 3 header files that I need to translate can be found in the src/api directory:

I’m not sure what a *typical* process for conversion might look like but I started out by trying to use Dr Bob’s HeadConv tool as a way of getting me started and doing some of the legwork. However, the callback structure used by the parsing callbacks confused the tool and it never produced anything of use. After this setback I tried to install Rudy Velthuis‘s Conversion Helper Tool but I wasn’t able to load this into the Delphi 2009 IDE.

Having failed to get any of the conversion tools to do any of the work for me, I accepted my task and began to convert the headers from scratch. There were two resources that I found invaluable: the first is the Type Conversion table half way down the page on Dr Bob’s HeadConv page.  The second was the comprehensive Pitfalls of Conversion guide from Rudy.

As I began converting the headers I realised that although I thought I was translating the headers without a problem, the devil is in the detail and as my first major conversion project, I didn’t have the specfic knowledge necessary to correct my specific mistakes. As a result, please correct me if I have misunderstood any bits which I have picked up in the process.

Const parameters

One issue that I did notice is that there were quite a few parameters marked const in the header files, such as this declaration in yajl_parse:

   YAJL_API unsigned char * yajl_get_error(yajl_handle hand, int verbose,
                                            const unsigned char * jsonText,
                                            unsigned int jsonTextLength);

Rudy’s guide states that where possible Delphi programmers should ignore the const declaration as it’s easy to get wrong depending on whether the argument is passed by value or by reference, complicated by the issue that it means something subtly different in C. I translated this heading and simply ignored the const keyword:

    Tyajl_get_error = function(handle: yajl_handle;
                               verbose: integer;
                               jsonText: PChar;
                               jsonTextLegnth: Cardinal): PChar; stdcall;
Multiple Indirection Parameters

There was one declaration that I had to do a bit of searching for: it was in the yajl_gen file, in the declaration of yajl_gen_get_buf function:

YAJL_API yajl_gen_status yajl_gen_get_buf(yajl_gen hand,
                                              const unsigned char ** buf,
                                             unsigned int * len);

I had never seen the double asterisk before and found it surprisingly difficult to reliably Google for it. Luckily, this comprehensive page on C pointers was pretty useful in explaining that it is in fact a technique called multiple indirection which essentially creates a pointer to a pointer, in this case to a char.

    Tyajl_gen_get_buf = function (handle: Tyajl_gen;
                                 out buf: PChar;
                                 out len: Cardinal): yajl_gen_status; stdcall;

I wasn’t sure if this was indeed the correct translation but if you know otherwise then please let me know in the comments below.

Record Alignment and Enum size

When reading up on the perils of record alignment and enum sizes I found this blog post from Vlad loan Topan on Record Alignment and Enum sizes. Both problems are potentially applicable to the yajl header files. In his post, Vlad mentions that in order to line up the enum sizes you will need to use the {Z4} compiler directive to change the size from the Delphi default of 1 to the C compatible 4 as so:

type
  {$Z4}
  yajl_status = (
                  // no error was encountered
                  yajl_status_ok,
                  // a client callback returned zero, stopping the parse
                  yajl_status_client_canceled,
  ... snip ...

Reading through the Record alignment section of Rudy’s article I understood that I needed to search through the yajl project source to look for an Alignment setting. However, I couldn’t find a reference to the pragma directive anywhere in  the  source.

Next Steps

Well, it doesn’t actually work yet but I have put the Delphi-yajl project on GitHub where you can view my efforts so far, fork it and hopefully provide feedback or code contributions to show me where I’m going wrong. I have also setup a Delphi-yajl page here to serve as a placeholder for when it actually works!

Futher Reading

Tags: , , ,

12 Comments

  1. Istvan on the 1st April 2010 remarked #

    the len param in the Tyajl_gen_get_buf function should be made “var” or better yet “out” because it will receive the size of the new buffer in “buf” param, which imho should also be set to PChar

  2. Pawel Glowacki on the 1st April 2010 remarked #

    Have you tried types from new Delphi 2010 “dbxjson.pas” unit for parsing JSON?

  3. jamiei on the 1st April 2010 remarked #

    @Istvan: You’re right, It was a careless mistake to omit the out keywords. I can’t remember why for the buf parameter, I went to an array of byte there instead of straight to a PChar, which is what I did in other places.

    @Pawel: I have not tried it yet as I’m currently using Delphi 2009. I would also be interested in seeing a performance comparison of yajl vs the parser in dbxjson. Maybe the subject of another post..

  4. Christian Wimmer on the 1st April 2010 remarked #

    PChar is really a PWideChar in >= D2009.
    But char* in C is (often) a PAnsiChar in Delphi.

    Check your call conventions. It doesn’t seem to be stdcall but cdecl!

    Also check the real enum size in a compiler (sizeof(yourenum)) to be sure that it is really an 4 byte enum.
    Record alignment is a separate subject and has nothing to do with the enum size.
    But I read some parts and didn’t find a problem with it (yet).

  5. Xepol on the 1st April 2010 remarked #

    Be careful of double indirection char **buf can often mean that you are passing the address of a pointer, and that pointer is to char data.

    Var/Out to PChar are not aways appropriate – sometimes NULL is a perfectly valid value for those fields, and you can not accomplish this with var/out. In this case the null would indicate that you do not need that pointer populated on return.

    and ya, C Headers are hard to convert. I found that none of the header converters currently work at all.

    I found that most of the real pain comes from the pre-processor nonsense.

  6. jamiei on the 1st April 2010 remarked #

    @Christian: Thank-you, I had forgotten that there might be some issues with the unicode changes. I will check the call convention and enum size shortly.

    How would you recommend I work out what I need to check on the C side in order to assess any record alignment issues?

    @Xepol: Interesting point re Var/Out, I will check the yajl documentation to see if any could be expected to return null.

    As an aside, why do you think the header converters have not been kept up to date? Lack of demand or lack of time?

  7. Rudy Velthuis on the 4th April 2010 remarked #

    I’m sorry you couldn’t install the conversion helper package. It should install in Delphi 2009, but I may have to change some of the directories used in the project. I’ll investigate this and post an update.

    Thanks for mentioning my article, too. :-)

  8. Rudy Velthuis on the 4th April 2010 remarked #

    FWIW, WRT the convertpack.zip file: I now created different projects for versions 2006-2010. They have settings that should make them compile and load in each setup, as they are not compiled to the Delphi directory anymore. I also added an updated, recent convertpack.ini which is the result of many conversions I have done myself.

  9. jamiei on the 28th April 2010 remarked #

    Thanks Rudy!

  10. William Meyer on the 23rd July 2010 remarked #

    My own experience with HeadConv was very disappointing. There was another tool I found at the time (Delphi 2 days) which did a better job. In fairness to Dr. Bob, the presence of any non-trivial macros makes the process extremely difficult.

    I was able to reach perhaps the 80% level with the tool, then was on my own, and had some help from folks on the newsgroups, especially Wayne Niddery and Rudy Velthuis.

  11. Menjanahary on the 22nd February 2012 remarked #

    Have considered to embbed obj files instead of using yajl.dll?

Leave a Comment