I’ve worked on a number of projects with Cucumber now, and I think it is a terrific tool. However, I’ve been seeing it used in ways that it was not originally intended for, the consequence being acceptance-test suites that grow increasingly slow, brittle and difficult to maintain.

These problems occur when developers start using Cucumber to drive overly-detailed tests. The problem of excessive UI testing with Cucumber has been blogged about in the past by Aslak Hellesøy – the creator of Cucumber – but I thought I’d cover some of my own experiences and opinions in this post.

The Thing About Developers

We developers are a funny bunch – we’ll take a new tool and flog it to death, often pushing it way past what it was originally intended for. In some ways, this is only human, but the problem is that these failed experiments stay with us in the form of difficult-to-maintain code.

To make matters worse, when we discover the tool can’t do absolutely everything we hoped, there’ll be a backlash and we’ll declare the tool to be rubbish. Those of us lucky enough to have the chance will move onto the next big thing, sometimes going right back to where we were in the first place, and failing to acknowledge the more modest (but not insignificant) advancement that the tool gave us in the first place. Those unlucky souls left behind will have to clean up the post-party mess, cursing those who moved on and the wretched tools and techniques that they used.

The technology and process landscape is littered with such roadkill. And I believe Cucumber is in danger of suffering the same fate.

What Should Happen…

So how is Cucumber supposed to be used? Well, it’s worth remembering that Cucumber is primarily intended to be a collaboration tool, and only secondarily an automated testing tool. This is a point emphasised in the recently-published Cucumber Book by Hellesøy and Matt Wynne.

However, in my experience, this collaboration can be difficult to achieve. Business people mightn’t want to write acceptance tests in that format, or might not even be able to; the format might be too far from what they’re used to. They need developers to help them, but developers aren’t particularly well known for their collaboration skills. Which brings me to…

What Actually Happens

So what usually happens is that developers dive in and start to write the features and scenarios themselves. At first this goes well, but it rapidly spirals out of control. For developers are, by necessity, detail-focussed, and their acceptance tests rapidly begin to reflect this fact. Being obsessed with DRY principles, they begin to re-use steps.  To achieve this, step arguments are introduced. Data tables next. Examples get out of control. Lots of state begins to be stored between step definitions. Steps need to be executed in a particular order.

And before you know it, you’ve got a real mess on your hands. New team members find it harder and harder to understand the step implementations. When they want to add another small feature, they try and contrive extra data that will wend and weave its way through the system and exercise their code change. And if it all gets too hard, they’ll start copying-and-pasting scenarios and tweaking them slightly to test the particular path they want to cover. The number of scenarios explodes and the test suite slows down.

It can get to a point where Cucumber starts to replace the role of unit-testing. This is unfortunate because whilst Cucumber is great, it’s a poor choice for a unit testing tool. There are two reasons this is the case. Firstly: acceptance tests, functional tests, integration tests; call them what you will, but one property they all share is that they test the system as a whole rather than particular units. Hence the increasingly convoluted test-data gymnastics that we have to contrive in order to test new changes. Secondly,  compared to writing your tests in code, all Cucumber is adding is an additional layer of indirection via a pseudo-natural-language syntax. If non-technical people aren’t reading those steps, all you’re adding is additional hoops to jump through for no benefit.

The Golden Rule

I have a basic criteria for whether Cucumber is going to add value to your project. In short, if your scenarios aren’t understandable to business people, there’s no point in using Cucumber.

If developers are going to write the acceptance tests, they need to exercise extreme diligence. They need to ensure that each scenario should be almost immediately comprehensible to a business person. A scenario should not be full of so many gory details and twists and turns that a business person glazes over when reading it.

Of course, this is easier said than done. It’d require developers to regularly go and get a business person to take a look at our new feature files and scenario definitions. Yet, for many developers, this is an inexplicably hard thing to do.

A strategy that I’ve seen some developers use to stop their Cucumber tests getting out of control is to avoid tables, examples, and even arguments in steps unless absolutely necessary. Better to have comprehensible steps with some repetition than highly-refactored steps, but unreadable scenarios.

If you’re a developer who doesn’t think you’re going to be able to keep your feature files comprehensible, that’s OK – just don’t use Cucumber. Whilst in some ways this is a shame (because I think that it’s healthy and beneficial for developers to write a couple of high-level scenarios for new functionality that they are going to implement), in other ways it’s a relief, because if you just can’t help yourself when it comes to getting down to the nitty-gritty, I’d much rather you wrote your tests in code than that you wrote them in Cucumber.

A Third Way

So what if you still really want to use Cucumber, despite the fact that you can’t trust yourself to write scenarios that are comprehensible to the business, and the business is incapable of writing scenarios at all?

I have found testers to be great people to drive the development of feature files (or at least have a red-hot go at kicking them off). They can interact with the business to get a high-level picture, fill out the details, then get developers to write the step definition code (which developers just love to do because they just love to code). Testers are far enough away from the code to avoid unit-testing, yet close enough to check the important details.

The only trap testers should avoid falling into is this: when the developers have got all of the automated acceptance tests passing, testers should most definitely not use those tests as a script for their own manual testing (a strange phenomenon I have seen on a couple of occasions). Instead, they should use their new-found time and freedom to engage in exploratory testing.

Summing Up

I believe that Cucumber is only appropriate if non-technical people are going to be reading (and preferably writing) the steps.  Whilst in theory Cucumber is a great tool for behaviour-driven development at any level, I’ve seen too many developers struggling to write comprehensible scenarios and step-definition code that is understandable to other developers, let alone non-technical people. The result is often a Cucumber test suite that is slow, brittle and difficult to maintain.

I’d rather that developers wrote their test in plain-old code, unless a business person is going to look a them. I also believe that, rather than trying to put a natural-language wrapper around their tests, developer effort would be better expended just trying to make test code more understandable.

High-level acceptance tests written in Cucumber are great, but let’s not go overboard. Let’s not let Cucumber become another technology roadkill, run down by over-zealous developers. And let’s not leave ourselves a test-suite legacy that we’ll all regret.

Before we begin, let me give you a quick background. I’m a web developer who likes to dabble a bit in UI design. The majority of the projects I work on are non public facing, so they are often without the aid of a graphic designer. This usually leads to the production of a fully functioning web app that looks like a mixed bag of CSS tutorials. Sounding familar?

Enter Twitter Bootstrap.

Bootstrap is an open-source, cross-browser collection of CSS and HTML conventions. It is essentially a front-end toolkit for rapidly developing stylish web applications. It covers everything from forms, typography, tables and buttons to layouts and dynamic components like tabs, menus and pagination. It is built on with the power of LESS and has support for many jQuery plugins (modals, dropdowns, alerts etc).

One of the strong points of Bootstrap is how easy it is to integrate into a project and use in conjunction with other technologies. To give you an example, my most recent project used Bootstrap in conjunction with display tag. Display tag is a JSP tag library for displaying tabled data, with built in column sorting and pagination. The integration between Boostrap and display tag was seamless. I only needed to set the CSS classes on the display table to use Boostrap CSS classes:

<display:table class="table table-striped table-condensed" name="results">
...
</display:table>

Bootstrap even took care of the table row stripping (although display tag has support for this too). Here is a picture of the result: Another thing I like about Bootstrap is that it makes use of custom HTML 5 data attributes to activate/link the majority of it’s dynamic content. For example, have a look at the code required to pop up a modal:

<a class="btn" data-toggle="modal" href="#theModal">Launch Modal</a>

<div class="modal" id="theModal">
  <div class="modal-header">
    <a class="close" data-dismiss="modal">×</a>
    <h3>Super Cool Modal</h3>
  </div>
  <div class="modal-body">
    <p>Some body text</p>
  </div>
  <div class="modal-footer">
    <a href="#" class="btn btn-primary">Save</a>
    <a href="#" class="btn">Close</a>
  </div>
</div>

Note the data-toggle=”modal” attribute on the button, which means clicking the ‘Launch Modal’ button will make ‘theModal’ div appear as a modal. There is absolutely no JavaScript required! I like this because it keeps the code clean and I don’t need a high working knowledge of JavaScript to make basic UI components work. This also helps minimise code on the ‘view’ side of things. If you’re not already sold, here is another code example for creating drop down menus:

<ul class="nav">
  <li class="dropdown active" id="menu1">
    <a class="dropdown-toggle" data-toggle="dropdown" href="#menu1">
      Listings
      <b class="caret"></b>
    </a>
    <ul class="dropdown-menu">
      <li><a href="/list/1.html">Update</a></li>
      <li><a href="/list/2.html">Create</a></li>
      <li><a href="/list/3.html">Delete</a></li>
    </ul>
  </li>

  <li class="dropdown" id="menu2">
    <a class="dropdown-toggle" data-toggle="dropdown" href="#menu2">
      Other
      <b class="caret"></b>
    </a>
    <ul class="dropdown-menu">
      <li><a href="/other/1.html">Reviews</a></li>
      <li><a href="/other/2.html">Users</a></li>
    </ul>
  </li>
</ul>

As you can see, no JavaScript! The drop down menu’s are identified with data-toggle=”dropdown”, and Bootstrap takes care of the rest!

Although I’m extremely pleased with Boostrap, there are a few concerns/issues that I have. The most obvious is that if you use Boostrap on all your projects, guess what all your projects are going to look like? I think the Twitter look and feel is very nice, but the novelty may wear off after a while. I know you can get around this by overlaying your own CSS customisations to the Boostrap core, but we are starting to head back into a mixed bag of CSS tutorials problem. Maybe to a lesser degree though, since we would have a solid CSS base (using LESS) to work off. Another issue I’ve found with Bootstrap, is that it doesn’t have native support for a date picker. This was unfortunate because our search UI required the user to enter a start and end date. Thankfully with a bit of googling I was able to find this little widget, which did the job fine. I’m pretty sure something like this will makes it’s way into Bootstrap soon.

I should also point out that Bootstrap isn’t the first or only one of it’s kind, there are quiet a few floating around the web. I’ve also looked into jQuery UI, Zurb Foundation, YUI and HTML Kickstart. Zurb Foundation and HTML Kickstart are very similar in offerings to Boostrap, but in my opinion Twitter Bootstrap has that little bit of extra visual design polish to it’s UI. YUI and jQuery UI are heavier frameworks with a bit more emphasis on JavaScript than Boostrap. I always felt like I needed to do a lot more work on the CSS side of things when using jQuery UI or YUI. To their credit though, I think they have better API’s and documentation than Bootstrap. Though this is possibly because they need it? I found Bootstrap to have a very small learning curve compared to jQuery UI or YUI.

As stated on the official website, Bootstrap is ‘built for and by nerds’. It allows me as a developer to create a gorgeous UIs in less time, which means a happy client and in turn a happy boss. Bootstrap makes me look good. So if you’re a front-end developer and have yet to use Bootstrap, I’d recommend giving it a shot.

Weird Al's not the only one who knows how to go large with Jackson

A couple of months ago I was asked to build a processor that would take a JSON file; perform a few elementary checks and transformations and upload the resulting records into a Couch DB. I hadn’t done any JSON, or Couch and, naturally, the whole thing had to be done yesterday – but at first glance it didn’t look like much of a challenge.

However, looking a little deeper, my processor was going to be part of a replication process across two databases housed in separate enterprises. The replication was going to be based on a daily snapshot – each JSON file would be a copy of the entire database – and a few back-of-the-envelope calculations suggested that the files may become rather large (40+GB) over time.

There was also a fairly tight processing window within which the upload had to be complete. Repressed memories of chasing down and remediating an old DOM based XML project plagued with OutOfMemoryExceptions and very slow processing times warned that some thought was required.

Obviously I wasn’t going to be able to be able to read the whole file into RAM to transform it and fortunately, I didn’t need to – the file was a collection of a couple of million simple objects with no cross references. So to minimize the amount of memory required by the processor, I simply needed to parse the large file into a stream of objects and transform them either individually or in small collections.

Jackson

Key “JSON Parsing” into your favourite search engine and you find about thirty different technologies competing for your attention. Also about a hundred different opinions as to which was “the best”. After a brief review, Jackson recommended itself though the tutorial which indicated that it included a Streaming API, as well as a DOM like Tree Model and JAXB style object bindings.

Given the memory and time concerns above, my first concept was to use the Streaming API to identify object boundaries within the input file to break it down into small chunks and to then use a JAXB style object binding to do the simple transformations required (mostly field name changes). The object binding piece was beautifully trivial:

public class Business {
    @JsonView({ BriefView.class, FullView.class })
    private String businessId;

    @JsonView({ BriefView.class, FullView.class })
    private String name;

    @JsonView({ FullView.class })
    private String address;

    ...
}

Using the JsonView concept, I was able to use the same value class to render different JSON requests for various external services supporting the checking and transformations required.

The Streaming API piece wasn’t going so smoothly. Although it was straight-forward to identify the object boundaries in the JSON token stream, there didn’t seem to be a natural (i.e. easy) way of accessing the underlying character stream in order to create the smaller work unit for the object bindings to use.  A re-think was necessary.

The object bindings were giving me an easy way of creating a collection of value objects in memory from a JSON stream – and conversely of creating a JSON stream from a collection of value objects. If I could get the Jackson object binding to create a stream of value objects, then I could dispense with the streaming API altogether. Some trawling through the Jackson API soon showed this wasn’t only possible, but easy:

JsonFactory f = new MappingJsonFactory();
JsonParser jp = f.createJsonParser(reader);

do {
    Business business = jp.readValueAs(Business.class);

    if (business == null) {
        handler.complete(context);
        break;
    }

    numberOfBusinessesReceived++;
    handler.handleBusiness(business, context);

} while (true);
jp.close();

The object handler here applies some cheap validation rules and aggregates a bunch of objects into a file for later (in memory) processing and upload to Couch. The context object carries some extra information – like the name of the file being processsed.

The magic provided by Jackson is the readValueAs method, which reads enough of the stream to create an object. Pushing a 10GB of two million records file through the processor running on an ordinary desktop with 256M of heap space yielded a processing time of 6 minutes – well within the required processing time window.\

Conclusion

The creators of Jackson have learnt the lessons from XML parsing well. Jackson offers a range of parsing approaches that allows developers to choose the right approach to the task at hand. In short, Jackson eats massive JSON files – without raising a sweat.

Yow! Brisbane 2011

I recently had the pleasure of attending the Brisbane Yow! Conference.  This was a great conference with talks to interest developers across many languages platforms and experience.

This blog is a summary of the more interesting talks that I attended and my thoughts on them.

Keynote – Top 10 JVM Erroneous Zones – Cameron Purdy

The opening talk of day one was a very technical discussion from an Oracle VP, taking us through what he thought were areas that can/could/should be improved in the Java Virtual Machine.  It was a reasonably low level talk for a Keynote but this was a developer’s conference so not entirely unexpected.

Purdy spoke about low-level extensibility in terms of abstracting on primitive types and the limitations of building word length into compilers.  He also touched on precision, lack of continuations support and made comparisons with other languages such as C++ and Scala to support his points.

These points were then used to relate back to development work being done at Oracle on the JVM.  The case was made that since Oracle took over JVM development and maintenance, many improvements and efficiencies in the language have been made. As Java remains a flagship product for Oracle they will continue to do so going forward.

What I took from the talk was that someone driving development is being constructively critical of the language, which is undergoing a lot of criticism in  light of the popularity of dynamic languages such as Groovy.

Lean and Agile  – Continuous Design – Mary Poppendieck

This talk from the Lean Luminary was not quite what I expected. It focused more on a wider view of product design from conception to market place whereas I thought it would be covering incremental design, how best to manage that within an agile environment and how best to integrate that with a continuous delivery approach.

Although the talk was a little broader than I expected, Mary continually reaffirmed the 3M philosophy of make a little, sell a little and learn a little.  This was then discussed in terms of product cycles and how important design conception is and how it can then be refined through iterations of testing and using the design to the point that it can be taken to market.

A lot of comparisons were made between Google, Apple and Dropbox in terms of defining and refining products before they go to market and what comes after go live in terms of design.

An interesting point was her discussion on enabling toggling of features in production so that unfinished features do not disrupt the continuous delivery cycle. She discussed doing small releases via load balancing to test out new features on a smaller audience in case they fail and drive people away.

Overall this was a great talk to be listening to if you were taking a new product to market or worked within a start up type environment.   Of course it was quite a privilege to hear Mary Poppendieck present and she is an excellent presenter.

 DevOps – We’re all Developers Now – Julian Simpson

This talk eventually got underway after a few technical hitches with the speaker’s laptop not co-operating with the projector.  You definitely don’t want technical difficulties in front of a room full of developers.

Once the talk began, it was basically a quick overview of DevOps and then mainly a comparison of Puppet and Chef.  Both Puppet and Chef are automation tools for Sys Admins.   The talk was very interesting showing how a sys admin would manage the installation of an application using the Nginx web server and Jetty.  Examples of how this is configured in Puppet and Chef where then shown illustrating how the configuration is then version controlled to provide change control management and one version of truth.

The power of this was demonstrated very well as Julian discussed how the configuration defined where specific components were to be installed across many virtual servers, allowing the administrator to roll out changes very quickly to a scalable number of machines.  Very powerful and quite dangerous – all the more reason for source control and proper change control!

DevOps is a fast growing and very interesting development in the industry and is leading to more responsibility for the developer as the gap between development and production is closing….

I thought this was a very good talk and I am looking closely at the various DevOps movements within our clients.

Transitioning To The Cloud – Challenges Transitioning To The Cloud – Rod Johnson

A very interesting talk from Rod Johnson, the founder of Spring.  This talk was Rod discussing what he sees as the issues and challenges facing the drive of software into the cloud.

He defined the differences between Infrastructure as a service (IaaS), Platforms as a service (PaaS) and Software as a service (SaaS) and then spoke of the trends he thinks will happen with these.  These were namely that there will be more growth in PaaS and likened it to the second coming of the main frame.  He talked about a standardized stack such as a Ruby or a Java stack, which might see developer’s choice narrow and will also see the tools as part of the platform.

It was very interesting hearing someone of Rod’s experience and foresight discussing the cloud and what he saw it meaning for developers in the future.

The Interactive Web – The Live Web – Dan Ingalls.

Dan Ingalls is one of the founders of Smalltalk so it was quite surreal to have him present a demonstration on a JavaScript web IDE for building HTML 5 applications.

The Living Kernel is a component architecture for HTML and runs in the browser.  It lets developers build and compose with existing components in a canvas like fashion and then tie them together with events, animations and scripting to quickly pull together visual applications.

While the examples shown in the talk were all relatively lightweight there was some scope of the power of this show when he quickly generated a graph based on Unix output to show CPU utilization of a server.  This would then be saved as a HTML report.  His entire presentation was also built using the tool, which showed its versatility.

What was missing in this talk was the business benefits of these tools.  He mentioned towards the end that they generate some reporting for come clients using this but it was only mentioned in passing.

Better Testing for Less Work – QuickCheck – John Hughes

This very entertaining talk was based around the speakers experience using QuickCheck.

QuickCheck is a testing tool, which based on a set of rules will then generate a specified number of tests with random data.  If an error is found, then the tool will focus on reproducing that error as many times as possible to determine the simplest cause of the error from the random data it generated.

John Hughes explained the tool at a high level and briefly showed how to specify a property rule but the basis of his talk was his experiences using QuickCheck in industry.  He discussed the use of the tool in car industry software and how he identified an issue in a production database.

John Hughes was such an entertaining presenter the entire audience was hanging on every word and we all erupted in applause at the end of the talk.  His passion for his work was evident and this translated into a surprisingly interesting talk on a low level-testing tool.

Keynote – Escape from the Ivory Tower – Simon Peyton-Jones

You could be forgiven for calling Simon Peyton-Jones  “Monty” and John Hughes’ “Python”.

Both went to university together at Cambridge and have the same passion, humour and comic delivery which makes for an excellent presentation. This was a wonderful follow on from the John Hughes talk.

This talk covered the history of Haskell, a functional programming language which has largely been academic until recently.

With the rise of parallel programming Haskell has attracted far more interest and suddenly Simon as a co-creator of Haskell is very much in demand as a speaker.

The speech centered around the history of Haskell and the differences between Haskell and other functional programming languages.  For non-functional programmers Simon recommended some exposure to Haskell to help them think about programming in a different way.

Again the speaker was very passionate about his topic and his presentation was so well put together that the audience was very much engaged.

JavaScript Everywhere

The next few talks that I attended painted a better picture for me on the rise and rise of JavaScript.

Web As A Platform – Dart, a new programming language for structured web programming  – Kasper Lund

This talk introduced Dart, a new programming language (almost) released by Google, still very much in its formative years.

Through coding up examples, the presenter highlighted a number of the language features mainly comparing them to JavaScript.   I took it that Dart was positioning itself to rival JavaScript. Dart can either run on it’s own VM or can be compiled down to JavaScript… which leads me into the next talk.

Web As A Platform – The Web Browser is a Transitional Technology – Allen Wirfs-Brock

This talk was based on the premise that we are moving into a new era of computing. We went from corporate computing to personal computing and now into what Mozilla’s Allen Wirfs-Brock labeled the Ambient Computing era.

Ambient is defined as “completely enveloping” so I take it he was trying to capture the fact the now people have multiple devices which they want and use all the time, therefore we are enveloped in the technology.  Bit of a stretch for me I’m afraid… I don’t think that phrase is going to stick.

In any case, he made good points that the new gadgets entering the market do not typically use a browser any more to interact with the web.  However a lot of them still use the same technology stack just without the browser chrome.  This web platform can sit directly on new hardware losing the PC and the browser.  To relate back to the title of the speech, the browser was a PC based, transitional technology to get us to this point.

The biggest point I believe Allen made was that JavaScript was directly built into this technology stack and was the only language present.  By this fact alone it foretells a bright future for JavaScript as it would be very difficult for an emerging technology (such as Dart) to proliferate to the point that all vendors across the Internet and gadget market will replace JavaScript in this platform.

I guess that’s why the Dart guys have ensured Dart can compile down to JavaScript….

Web As A Platform – NodeJS and the JavaScript-Everywhere Strategy – Matthew Eernisse

Yammer’s Matthew Eernisse talked about JavaScript on the server side and the rise of NodeJS.  In the talk Eernisse wrote a web server from scratch to show how quickly you can get off and running with NodeJS.  He discussed some of the tools available to help with NodeJS development and highlighted some of the trouble shooting lessons he had learned.  With so many callbacks and asynchronous code it can be difficult finding root causes of issues.  Eernisse showed how he changed his debug to help address those issues.

The majority of the Yammer upload apps are NodeJS applications and Shine Technologies has also done some major project work with a Melbourne client on various NodeJS apps.

This talk complemented the previous ones on highlighting the rise and rise of JavaScript to me.

The only other talk I should mention was the Keynote by Damien Conway on extending Perl to implement quantum physics. This was genius in PowerPoint but while I enjoyed it and followed it to a degree, I could not hope to summarise it due to its complexity.  Presentations like Conway’s go a long way to inspiring people in the field to take their profession far beyond what anyone thought was possible.  Sensational stuff!

Summary

There were some other interesting talks, which rounded out the conference plus a good chance to meet other developers and people in the industry as well as the speakers themselves who were very approachable.

This was a very well run conference and I will be lining up next year again for it.

Alistair Sutton is a Senior Consultant for Shine Technologies and has recently moved to Brisbane to open a Brisbane branch for Shine.

Synchronising advanced data structures with a backend sucks. Especially when the sync is both ways; user updates to server, server updates to user. Couchbase for Mobile attempts to solve this issue by bringing CouchDb and it’s spiffy replication to Android.

What? Why?

Why would you ever want CouchDb on your phone, you ask. Well, along with giving you some fancy pants replication, which I’ll rant about soon, some of the benefits over Android’s built in SQLite support are:

• Unstructured data storage in JSON documents
• Map/reduce for querying
• Versioning of documents
• And more..

In a bit more detail ..

JSON documents with whatever you want in them

The data in CouchDb is document oriented, in that it stores and returns your content in JSON documents. These documents can have whatever fields and whatever JSON acceptable values you wish without any table changes. This gives you the power to support dynamically added fields. If you wanted to create a form that accepts additional user defined fields, CouchDb or something like it is what you want.

A CouchDb document, requested with a GET to /database/boo, is read and written something like

{
  "_id": "booboo",
  "_rev": "gibberish",
  "favouriteFoods": [ "Bangers", "Mash" ],
  "bodyWeight": 300
}

While a horribly non standard representation of a SQL table structure, that allows dynamically added fields, would be something like

Form#1()
FormValue#1(
  formId: 1,
  fieldName: "favouriteFoods",
  fieldValue: "Bangers",
  fieldType: "String"
)
FormValue#2(
  formId: 1,
  fieldName: "favouriteFoods",
  fieldValue: "Mash",
  fieldType: "String"
)
FormValue#3(
  formId: 1,
  fieldName: "bodyWeight",
  fieldValue: "300",
  fieldType: "Number"
)

Retrieved with queries like

SELECT * FROM Form
SELECT * FROM FormValue WHERE formId = 1

As you can see the CouchDb approach is much nicer, and in SQLite’s case the structure of Form and FormValue need to be defined when the table is created. Meaning updates to the structure of these tables involve fancy scripts or complicated SQLiteOpenHelper.onUpgrade functions.

Map/reduce functions

CouchDb has a different method of querying the data compared to normal SQL databases, depending on your usage this could be a deal maker or a deal breaker.

In CouchDB you create views of your data, written in Javascript, emitting a key and a value. You then look up the data you want using the key you specified. You can also reduce the values for summing or other fun things. For example:

"byName": {
  "map": function (doc) {
    if (doc.name) {
      emit(doc.name, 1);
    }
  },
  "reduce": function (key, values, rereduce) {
    return sum(values);
  }
}

When this has been indexed you will have an index like:

"Jim", 1
"Jim", 1
"Jack", 1

Calling it with reduce will give you:

"Jim", 2
"Jack", 1

Versioning of documents

CouchDb versions documents. Every time you update a document the previous revision will still exist and be accessible (until compaction which I’ll ignore), which may be useful for basic undo or rollback functionality.

Everything CouchDb gives you

And all of the other things CouchDb provides, like hugs, kittens and obese indexes (don’t quote me).

Replication

The main benefit you’ll see with using CouchDb for your application (presumably) is data synchronisation via replication.

CouchDb supports really easy start and stop replication, with it’s own conflict resolution. You could have multiple databases that haven’t seen each other in months, tell them about each other and each of them will eventually catch up with the other as far as documents go. This is awesome for “when I want” syncing, and offline editing of data that syncs when a connection is established again. Some use cases:

Taking notes

This would be great for a note taking application that supports mobile as well as web edits. The user could add notes on both sides and sync to make both sides consistent when they want / when a wireless connection is available.

Real time when available

You could have replication going any time the application is in the foreground. This would allow for applications that work in real time but don’t explode when the network is lost.

And I’m sure other more exciting things my boring brain chose to shield me from.

I’m sold. How do I make the magic happen?

First up, make an Android 2.x project in your favourite Java IDE (Eclipse), which you have obviously already set up for Android development.

Next you need to get the latest version of CouchDb for Android (still in Developer preview), here. This website also covers the instructions for getting your project CouchDb friendly, basically it’s “run an Ant script and hope it’s not doing anything too horrible”. Without going too into it, it’s configuring your project for CouchDb, Erlang (which CouchDb uses behind the scenes) and some funky native libraries.

You can also add Ektorp, a Java API for CouchDb with Android support. This wraps the CouchDb calls so you don’t have to generate your own HTTP requests. That’s really up to you, but the code examples below use this, and Couchbase recommends it as far as Android CouchDb libraries go.

I’m listening..

If you’re anything like me, you’ve probably skipped to this section or closed the page when you didn’t instantly see code, but here goes.

Starting CouchDb

First we’ll set up a CouchbaseDelegate in our main Activity. You use this to capture startup errors and kick off initialisation steps / replication when the database has started.

ICouchbaseDelegate couchDelegate = new ICouchbaseDelegate() {
  @Override
  public void exit(String error) {
    // Show the error to the user
  }

  @Override
  public void couchbaseStarted(String host, int port) {
    // CouchDb has started, set up your connector
    // in our case initialise Ektorp
    startEktorp(host, port);
  }
};

Now we’ll actually request for CouchDb to be started, referencing the delegate we created above. In my case I’m calling this when the activity starts.

private void startCouch() {
  CouchbaseMobile couch = new CouchbaseMobile(
    getBaseContext(), couchDelegate
  );
  couchConnection = couch.startCouchbase();
}

CouchbaseMobile will do its magic and eventually call your delegate.

Connecting to the database

Now that CouchDb has started, we’ll connect using the host and port the delegate gave us.

clientBuilder = new AndroidHttpClient.Builder();
httpClient = (AndroidHttpClient)clientBuilder
  .host(host)
  .port(port)
  .build();
dbInstance = new StdCouchDbInstance(httpClient);
// The last parameter true will create the database
// if it doesn't exist.
dbConnector = dbInstance.createConnector(DATABASE_NAME, true);

The dbConnector is now ready for normal database use. Let the CRUD begin.

Creating the initial design documents

CouchDb is pretty useless without a view over your data (unless you know ids or want to just work with all documents). So after CouchDb has started you should create (or update) the design documents specifying your views.

designDocId = "_design/slyncy";
byNameView = "byName";
byNameMap = "function(doc) {if (doc.name) emit(doc.name, doc);}";
DesignDocument dDoc = new DesignDocument(designDocId);
dDoc.addView(byNameView, new DesignDocument.View(
  byNameMap
));
dbConnector.create(dDoc);

Putting a list adapter on top of the view

Assuming you’re displaying the data in a list, you’d want to wrap a view in a custom CouchbaseViewListAdapter. There’s a horrible implementation in the example code showing basic view mapping.

ViewQuery query = new ViewQuery()
  .designDocId(designDocId)
  .viewName(byNameViewName)
  .descending(true);

// The third parameter specifies that you should listen to
// the _changes feed from CouchDb enabling this means the
// list will automatically update for you when data changes
adapter = new CouchbaseViewListAdapterImpl(
  dbConnector, query, true
);

The final parameter (_changes: true) is pretty powerful. It works like Android content provider data change notifications, allowing the list to update when the data changes. Delete a document and it will be removed from the list.

Something Ektorp is missing with their base CouchbaseViewListAdapter class is support for include_docs when querying the view. This means your document will have to be emitted as an indexed value, which could lead to obese indexes if your data set is big enough.

Starting replication

Starting replication is pretty simple. You build a ReplicationCommand, specifying the source, target and continuous (once until complete (false), or continually run until shut down (true)).

pushReplicationCommand = new ReplicationCommand.Builder()
  .source(DATABASE_NAME)
  .target(syncUrl)
  .continuous(true)
  .build();

EktorpAsyncTask pushReplication = new EktorpAsyncTask() {
  @Override
  protected void doInBackground() {
    dbInstance.replicate(pushReplicationCommand);
  }
};
pushReplication.execute();

Switch the source and target around for pull replication. Depending on how your system is you would probably want both (user updates -> server, server updates -> user).

CRUD

After the connection is set up, CRUD is simple. You just call create/update on a document (where document is a Map in my case), or delete with a given id and revision.

dbConnector.create(document);
dbConnector.get(Map.class, id);
dbConnector.update(document);
dbConnector.delete(id, revision);

Summing up

So, as you can see, connecting to CouchDb on Android is fairly simple, if a little verbose, but that’s the Java way.

You can find a working (you know, sometimes) example application here, based on AndroidGrocerySync.

Concerns

Data Security

Sure, replication sounds dandy, but how do I stop the user replicating changes of documents they don’t own?

CouchDb has a method for this called Validation Functions. Basically you can define a Javascript function that will look at an update and allow you to reject the change. The last parameter of these functions is a user context. So using this you can work out who’s writing and if their write is valid. For example:

function (newDoc, savedDoc, userCtx) {
  if (savedDoc) {
    // This is an update, make sure the user
    // isn't watching the world burn.
    if (newDoc.userName !== userCtx.name) {
      throw({ unauthorised: 'You shall not pass ' + userCtx.name });
    }
  }
}

Internal phone security

A completely uninformed concern (I’m sure Couchbase has thought of it) is security from other applications running on the phone. What if you have private data you don’t want Mr Naughty’s application to push to his cloud of deceit?

Completely based on a Stack Overflow thread here, it seems that connecting to Couch and access the database itself is secure enough, but the data is written to external storage, which isn’t secure at all.

Conclusion

I’ll sum it up in pros and cons form.

Pros:

• CouchDb on Android makes replication easy. It’s basically allout of the box, just point and say go
• The JSON strucure of CouchDb documents gives your application a lot of flexibility and a nice way to store data.
• The actual code behind it isn’t too painful or that far away from ‘typical Android’ code.

Cons:

• Starting up CouchDb is slow, from what I’ve heard mostly due to loading an Erlang interpeter. You’ll see the lag between starting the application and initialising CouchDb if you run the example application
• Map/Reduce while providing some friendliness also removes the possibility of on the fly SQL style queries
• Packaging CouchDb with your application adds about 10mb of bloat to your installation.

All up I’ll give it 3.33 out of 5 overly intelligent mice, ready to take over the world.

Links / More interesting reads:

• Example Github project: https://github.com/dbrain/slyncy
• A nice setup guide here

Note: Images randomly pilfered from Google Image Search.

Getting started with the development of location-based services on Android is relatively easy thanks to the well documented location API. However, getting more serious shows there is still much uncharted territory. One such area concerns the accuracy of real-life location data, which I have recently taken a close look at. In short, location data can be far more accurate than Google’s conservative estimates – at least with the phone that I used.

The Android developer documentation gives estimates on expected accuracy of the location data. This estimate varies according to the location provider used. Android distinguishes two location provider types with the following accuracy:

• Fine-grained location provider like the GPS provider with an accuracy of less than 100 m.
• Coarse-grained location provider like the network location provider with an accuracy of 100 – 500 m

One look at the Google Maps application makes it clear that the estimates for the accuracy are very conservative: in many cases the user location is shown on Google Maps with an accuracy of a few meters, much better than the estimates stated above.

My interest is in the individual accuracy of real-world data rather than an average accuracy estimate. Each time a location provider sends a location update, the accuracy of this individual location is given as well. In this article I look at this accuracy as reported by the location provider, and  compare the location accuracy of the two main location providers to actual locations.

Location Providers

Location providers are the sources of location data in Android. Even though the Android API specifies location providers in a very generic way, in practice there are two main location providers: The GPS location provider and the network location provider. Because they use different ways to obtain location data they differ in accuracy, minimum notification interval, time to first fix (TTFF), and battery consumption.

The way you receive location information in Android is to register a Location Listener with the Location Manager. At registration the location provider, notification interval and notification distance have to be specified:

locationManager.requestLocationUpdates(provider, notificationInterval, notificationDistance, locationListener);

The Location Manager  will call the onLocationChanged() method of the listener if the distance since the last location update for the registered location provider is greater then the notificationDistance, or the time since last location update is greater than the notificationInterval.

GPS Location Provider

The most obvious location provider is the GPS. GPS provides data with high accuracy: Less than 100 m is the official number, but real accuracy is often around 2 m – 20 m. Once it is connected to satellites it can provide location updates in intervals as short as a few seconds.

Sounds like an ideal location provider, but the GPS also has its draw-backs. For one, the initial connection to the satellites – Time to First Fix (TTFF) – can be quite slow. GPS traditionally needs to connect to at least three satellites for a 2D location, and initial transmission of so-called Ephemeris data takes a minimum of 30 seconds for a satellite. Once the Ephemeris data has been transmitted and connection to the satellite is made, location updates are very fast.

To speed up the TTFF, manufacturers have come up with a new way to deliver the initial Ephemeris data to the GPS: They use an Internet connection to deliver the data much faster than the transmission from the satellite. Android uses this so-called Assisted GPS (A-GPS) to speed up TTFF to 5-15 seconds, but only if you have an Internet connection.

Another draw-back of the GPS provider is that the battery consumption is quite high, and continued use of the the GPS will drain the battery. Even though GPS location updates can be received every few seconds, the Android developers guide recommends update intervals for the GPS of greater than 10 minutes. This limits the type of applications that can benefit from using the GPS.

Last but not least: Since GPS uses a radio signal, solid objects – especially those made from metal – will obstruct the signal. In practice, this means the GPS will not work in most buildings (GPS is not affected by clouds, rain, or snow, though).

Accuracy

Each location update from a location provider comes with the accuracy of the data in meters. This accuracy is given for each location update. To visualise the location data and the accuracy for each location I wrote an Android application that records the location data in a database. Drawing the recorded data on a Google Map helps visualise the data. The following map shows the data recorded for the GPS provider during a walk through Melbourne.

GPS provider data compared to actual location

The black line is the actual path I took during the walk. The white dots are the locations recorded by the GPS provider, while the red circles around the dots indicate the accuracy per location as reported by the GPS provider.

As expected the accuracy of the GPS provider is quite good: Most of the time the accuracy is within 30 m, some times as good as 2 m. That is much better than the conservative estimate of less than 100 m stated in the documentation.

An interesting observation is that sometimes the actual location is not within the circle of accuracy given by the GPS provider. The location marked with 1 on the map is about 25 m off the actual location, with an accuracy radius of about 5 m. Even with the radius of accuracy the result is still 20 m off the actual location.

Another observation that shows the draw-back of the GPS: if we are indoors or have otherwise bad reception there may not be a GPS signal, which means the location is completely lost. Marker 2 shows a section of the walk without any GPS signal. First I walked through a building from the Flight Centre to the cafe “The Journal”. Even after I passed though this building and walked down down the narrow Flinders Lane the GPS reception didn’t improve, so there are no readings at all during this time.

Network Location Provider

The network location provider uses both WiFi hotspots and cell towers known to the Android device to approximate a location of a user. When the location provider is polled, the IDs of the WiFi hotspots and cell towers in the area are sent via Internet to the Google Location Server, a database with location information on WiFi hotspots and cell towers. Google Location Server returns an approximate location of the user.

The data comes from a combination of cell and WiFi hotspot information, and can only be obtained when the device can access the Internet to query the Google Location Server. Accuracy of the data is specified somewhere between 100 m – 1000 m depending on the information available. WiFi hotspots allow an accuracy of 100 m – 500 m, while cell towers only allow for an accuracy greater 500 m. This means the network provider will be very inaccurate in areas without WiFi hotspots. Conversely, the more hotspots and cell towers are in the area the greater the accuracy.

The network provider can give first location results within seconds. This is much faster than the GPS location provider, which even using A-GPS may take up to a minute to give back first results.

Accuracy

The following map visualises data recorded for the network provider when walking through Melbourne.

Network provider data compared to actual location

The black line again is the actual path taken, while the white dots are the locations recorded by the network provider. The turquoise circles indicate the accuracy reported by the network provider. The walk took about 30 minutes, during that time 16 points where within a reasonable accuracy (I discarded those points with an accuracy of greater than 100 m as too inaccurate, just to make the map clearer).

The map shows that at least in an urban area the network provider gives enough data with an accuracy of less than 100 m. Most of them were actually in the area of 50 m. This is not as good as the GPS, but still quite impressive. The network provider also has number of advantages over the GPS: It works both indoors and outdoors, starts up quickly, and doesn’t drain the battery.

WiFi Hotspot Location Data

During recording of the data I stumbled across an interesting anomaly: The network location provider put me in a totally different location than where I actually was. Curiously this happened every time I connected to my WiFi hotspot at home.

Wrong WiFi Hotspot Location

The location marked with 1 is near my home, while the location marked with 2 is the place where I used to live. Once I arrived home the network provider placed me in location 2 instead of near location 1. Apparently Google thought I was in my old home once I logged into my home network, because it had previously registered the location of my WiFi hotspot in location 2.

This made me think about how Google actually obtains the location of a WiFi hotspot. It used to be possible to register the location of your WiFi hotspot with Google by hand. This option apparently does not exist anymore. As this discussion shows, the data for the network provider actually comes from data Android GPS providers sent to the Google Location Server at some other time:

“[...] before the GPS acquires a “fix”, Android will send the Wifi’s MAC/SSID and Cell Tower base stations CID/LAC around you to Google’s server and Google’s server will send back your location information; after the GPS acquires a “fix”, your device will send the Wifi MAC/SSID, Cell Tower CID/LAC, and your GPS location to Google’s server and Google’s server will collect this information to refine their own Wifi database.”

The discussion also shows an Email reply from a Google employee stating that updating location information on WiFi hotspots is done automatically when a GPS provider sends new location data, but “may require a significant amount of data from you and other users before changes are made to Google’s location database”.  That explains why my WiFi hotspot is still registered at my old home in the Google Location Server.

Conclusion

Comparing the location data received from the location providers with the actual location data shows that both location providers are much better in accuracy than the conservative estimates of the documentation.

For the hightest accuracy the GPS provider needs to be used, but is not suitable for all situations. The limitation of GPS to outdoor use is quite constraining, as is the battery consumption and TTFF, especially when used without A-GPS.

The results of network provider for me came as a surprise: It is quite accurate while doing without the draw-backs of GPS like power-consumption,  TTFF, and restriction to outdoor use. It does lead to large inaccuracies in areas without WiFi hotspots.

Clearly a combination of both provides will yield the best results. How they are mixed depends on the application. A foreground application with emphasis on accuracy would use the GPS provider as the primary provider, with the network provider as a backup, while a background application with emphasis on low power consumption would use the network provider as its primary location provider with the GPS as a backup. Varying the update intervals allows fine-tuning the providers to a specific application.

Code

If you want to have a look at the location data on your Android device I have uploaded the code to GitHub. It consists of an Android application that records the data as well as an HTML page that displays the data on Google Maps.

Of the many excellent sessions at this week’s Swipe Conference, the one titled “Build Better Cocoa Apps Using Game Mechanics” by Paris Buttfield-Addison was unique in its topic. In it, Paris outlined how gamification is currently viewed as something that can be quickly slapped onto an existing product in order to increase its appeal, whereas in reality, successful gamification requires carefully thought-out integration, for when done correctly it should form a core part of your app’s user experience.

In case you’re new to the concept, gamification is the application of game-like mechanics to non-game products in an effort to increase engagement with the user, motivating them to spend more time or money on your product.

The current trend in gamification is rather worrying, with seemingly everyone adding it to the list of things their product needs – as though it were a requirement for being successful. The reality of the situation is that these ‘gamification grafts’ do nothing to enhance the user experience at all, providing mainly virtual badges for completing rather menial tasks – for example, pushing a button. This lack of challenge contributes to a failure to keep a user engaged, for they quickly become bored and move on.

A better approach to gamification would be to find a way to enhance the core functionality through the use of game-like mechanics. An example of successful gamification is Stack Overflow. The core functionality is to provide people with a place to both ask and answer questions. However, game play elements come into it by rewarding users with a reputation system; the more you participate on the site, the more your reputation increases. Your reputation is then an indication of how helpful you are in the community. With higher reputation comes privileges that extend to helping with moderation of the site. This reputation system sets itself up in a unique way so that it encourages people to actively take part in the community and rewards them in the process.

Perhaps the most important point that was made during Paris’ session was that not all products are suited to gamification. An app that displays the weather has no use for leader boards or an elaborate point system – they bring nothing meaningful to the user experience. However something like Foursquare, whose core functionality is to provide you with a means of sharing your location with others, benefits greatly from the addition of game play elements, effectively turning an otherwise dull concept into an engaging experience that rewards and encourages exploration of your surroundings.

In conclusion, the concept of gamification is definitely here to stay, but as developers and designers, we need to be vigilant and not give in to the urge to simply slap on a half-baked implementation. Instead we need to take the extra time to evaluate if it will actually enhance the existing user experience.

We’ve all heard developers say it: “I’m a terrible drawer” or “I’ve got no design skills”. Perhaps we’re even guilty of saying it ourselves – I know I am. But after attending this year’s Swipe Conference I now subscribe to the opinion that this is no longer acceptable. We are all responsible for the design of the app we are building; developer, designer, tester, or producer: every member of the team is accountable for helping shape the app’s design and interactions.

For far too long this has represented how teams have collaborated.

Why must we all be accountable?

We – as developers – are iPhone experts. We know what can and can’t be done technically. We have toyed with and tried building (with both successes and failures) many ideas before. But before that, we are users; in fact, not just users but experienced super users. Sure we know how to code, but this is really just a small part of creating an app with an amazing user experience.

But I can’t draw

During the Swipe Conference we had a 30-minute session that addressed this exact problem. But the fact is, that it’s not a problem at all. When talking about working together to help build an intuitive interactive app, I’m not talking about creating the entire visual representation of your app. I’m talking about presenting ideas, and thinking through them as a team.

This 30-minute session taught us such a simple lesson, how to use a pen, how to draw a basic shape, a square, a circle, and how to colour them in. This is all you need to do, in order to present your ideas to the team. Sketch some simple shapes that represent the objects in your app, shade the background to bring the elements to the front, and colour them in. It’s simpler than a picture you would have drawn in kindergarten. “I can’t draw” is not an excuse!

So I’ve drawn my picture, now what?

At the end of this session, we were all encouraged to hand in the sketches of the iPhone interfaces that we had created. The organisers then took these sketches and stuck them up on the wall. And guess what, with just a black pen, a grey texta, and an orange texta (the swipe conference colour), we had hundreds of ideas that both conveyed an idea and were immediately understandable.

Just as we were able to do at Swipe Conference, everyone in an iPhone team can and must spend time doing the same thing. Pin your ideas on a wall, tear them down, change them, evolve them, give feedback, and work together to evolve from an infinite number of ideas from each team member into one highly refined and fantastic experience.

Brilliant apps with intuitive interactions

This is how we need to think about working

By having a team where everyone can provide input, we help to create an experience that can only be beneficial. I know from some of the projects I’ve worked on, that it can be increasingly frustrating when you’re involved building the inner workings of the app, but cannot influence its overall direction. With such a great device to work on, and the potential it provides to user and developers alike, it would be a gross injustice to continue working in a counter productive way.

Let the designers turn your ideas into beautiful pixel-perfect designs, and let the developers transform these designs into interactive working interfaces – but also work together with everyone to allow your app to become a masterful interactive experience.

Yesterday was the last day of Swipe Conference so I thought I would take this time to reiterate one of the points I took from the first presentation by Josh Clark.  Josh covered quite a few topics and if you haven’t already you should check out his book Tapworthy: Designing Great iPhone Apps.

In short:

• Gestures can be brilliant … if the context they are used in feels natural
• If you’re using gestures make sure your users will find them
• If you don’t think your users will figure out your gestures easily, don’t overload your users with lots of help hints all at once; instead, let them “unlock” them over time, like a reward for using your app
• Downside though is that there is no consensus on what a 3 finger swipe gesture might do. Every iOS app that uses this gestures decides it for themselves.

It was obvious from the way Josh presents that he has so much passion for touch / gesture based devices. What I personally took away from his presentation was that gestures can be awesome shortcuts within your app. In a lot of cases gestures are a natural progression with how we interact with real world items.

So let’s drill in a little more to some of these points.

Gestures are brilliant… if used in the correct context

How cool are gestures?! As a developer I have the freedom to implement any gesture I want, to perform any actions I desire. But it’s not really that simple – you need to consider if the gesture you’re using makes sense for the functionally you’re tying it to. For example, do you believe your users will associate a cool 4 finger pinch gesture in your app with bringing up a feedback email form?

Bad example, I know – because of course you would say: thats just stupid; why would you do that?! But these are exactly the kind of questions we need to ask ourselves – not only as developers, but as UX designers.

How will users learn your gestures?

If you want to add a gesture to your app because it’s a cool shortcut, how are you going to let your users know those gestures exist within your app?

Take a look at Apple for example. Their iBook app allows a user to turn a page by swiping from right to left; and yet in Contacts app, performing this action will most likely prompt you to delete one of your contacts. This is just confusing to a user.

I highly recommend that you watch Cathy Shive‘s presentation (when the Swipe Conference presentations are released – I will update this post when it happens). Cathy highlighted a lot of great reasons why it’s so important to get your metaphors right. If it looks like a book and doesn’t function at all like a book, then why are you making your app look like a book?

If the gesture you’re adding into your app is not something a user will instinctively know to try out, then you need to give them some hints to be able to find this shortcut.

There are a number ways you can do this:

• Possibly draw attention to the part of the app that has this gesture – maybe it could pulse for the first time to signify it is interactive
• Perform an animation that hints to the type of gesture that may be used to replay something
• If your app has quite a few shortcuts maybe only provide one hint each time the user starts up your app

Don’t forget to give your users a chance to explore your app. Users love to explore. They may stumble upon your gestures themselves.

Don’t overload your new users with “helpful” hint messages

I really liked the example Josh highlighted, which was if you were to be given an in-depth explanation before you were to read a book, you would instantly feel like using a book was quite complicated. For example: you turn the page from left to right, you read from left to right (depending on culture) and when you reach the end of a line you jump down to the next line and continue; to bookmark you can fold over a page and return to it in the future; when done, simply close the book (there’s a funny explanatory diagram along similar lines that you can find here).

The same goes for an iOS app. Feel free to explore gestures; make use of them as shortcuts within your app, but if you need to explain to a user how to use your app, then don’t just throw up a big old welcome message that tells a user every gesture they can perform – I guarantee you they will glaze over it. If it looks and feels complicated, then you’re just putting off your users.

In short, ease your users into finding these hidden gems within your app.

I hope swipeconf runs again next year!

Swipe Conference was great! I had a great time and all the presenters were excellent. It was cool to see so many iOS developers / UX designers out there who are really trying to build the best iOS apps they can.

Shine was recently involved with helping a client bring an outsourced mobile web site in-house. The site was essentially a guide for browsing business and event information for cinemas, restaurants and bars in your area. Bringing this web site in-house was mainly an exercise in re-writing the outsourced instance as a component to the client’s own internally developed NodeJS/Express app server (currently used to serve their main – non-mobile – web site).

The NetBiscuits platform was used to handle all the device specific porting which allowed the development effort to focus almost entirely on business logic and backend data processing. The combined learning curve of NetBiscuit’s own markup language with Shine’s Jazz template system was simple enough to make the presentation layer development a breeze. This blog will talk about the process involved in developing for NetBiscuits and some of the problems and solutions encountered on the way.

So what is NetBiscuits?
NetBiscuits is a hosted platform technology that enables web apps to reach mobile devices. It’s essentially a knowledge base that maps mobile devices and their browsers to a presentation profile, translating your mobile site into the format expected by whatever mobile client that is making the request.

It’s worth noting that the concept of porting web content to mobile devices and their browsers is big. Seriously. The impact of standardized mobile web development is perhaps lost on those of us who’ve never worked in it before. Where common web development concerns itself with compliance across 4 or 5 main browsers, mobile web development must cater to hundreds. Even in the days when Nokia ruled the world the support and maintenance of mobile web apps for an ever-growing device list was not just for the final phase of a project but an ongoing full time job.

Maximizing your audience still means supporting the lowest denominator, but these days you can expect your low-end browsers to run javascript. This old school WAP and J2ME developer acknowledges that times have moved on but pays due credit to the people behind NetBiscuits and their mobile platform.

Working with NetBiscuits
You develop your mobile web app in mostly the same way as any other web app with the difference that instead of dynamically generating HTML you’re generating BiscuitML. Your production URL will point to your NetBiscuits account (hosted on their cloud) which in turn points to your app server.

For as long as your app server talks to the NetBiscuits platform in BML, NetBiscuits will talk fluently to all devices that hit your mobile site. This means your development efforts can focus purely on the business logic without the concern of tailoring content to multiple screen sizes and browser types.

So what can you do with BML? Pretty much everything you can do with HTML. For most of the devices hitting your mobile site NetBiscuits will convert your BML to HTML. Writing markup in BML is done with Biscuit tags that comprise of familiar page elements such as; list, table, form, image etc. More specialised tags are available to handle ad-banners, analytics, maps and media. A script tag exists to provide various levels of javascript support ranging from handling browser events completely within the markup to simply relying on an external js file.

Challenges with Session Handling
Environmental constraints meant that client side session management was the only option available. This in turn meant that support for cookies and javascript was a requirement for any browser hitting the site. Cookies would store the user’s state code (see below) and the preferred list of cinema venues for returning movie times.

<TEXT>
    <richtext>
        Select your state:[br]
    </richtext>
    ...
    <richtext>
        <event type="onclick">
            <!-- Custom set cookie function -->
            <action eval="setStateCodeCookie('VIC');" />
            <action eval="return true;" />
        </event>
        [url="path/to/main/page/?{URLParams}"]VIC[/url]
    </richtext>
    ...
</TEXT>

Maintaining sessions within cookies is certainly made easier in the Tomcat/JSP model where the HTTPSession interface provides easy access to user data. In such a scenario, session data are largely managed by the server and a user is identified by a cookie-stored session id. In our instance we ended up pulling session data directly from cookies after discovering some limitations with BML.

Amidst the wealth of documentation for BML and its rich internet features there were examples of how to get/set/delete cookies from script tags. This led us to the false belief that a cookie could be interrogated and manipulated fully from within the markup. Wrong. BML’s getCookie function exists only to test the presence of a cookie, not to read/extract its data. Cookie handling was therefore relegated to an external js file which also meant taking control away from the markup for certain browser events. Specifically, the onclick event had to be hijacked from certain anchor tags to implement redirects for re-written URLs.

<TEXT>
    <richtext>
        Find Movies:[br]
    </richtext>
    ...
    <richtext>
        <event type="onclick">
            <!-- Override default behaviour and use
                 an externally defined custom handler
                 to perform redirect -->
            <action eval="getPreferredCinemasForMovieSearch('{URLParams}');" />
            <action eval="return false;" />
        </event>
        [url="#"]Browse by Cinemas[/url]
    </richtext>
    ...
</TEXT>

Hijacking the Onclick Event
Retrieving a user’s cinema preferences (above) is one example of URL re-writing trickery. Retrieving their state code is another. A state code would populate all URLs presented to the user. New users would have their state code persisted in URL parameters for the duration of their session. Returning users would need their session initialised by extracting their state code from the cookie.

Since cookie data wasn’t available at BML generation time, extraction would have to happen at URL click time. So this meant triggering a javascript function on the browser using the anchor tag’s onclick event.

Performing cookie data extraction at ‘click time’ effectively turns the handler function into an event-swallowing black hole, since it must also perform the page redirect leaving no control to pass back to the browser and it’s anchor tags. A consequence is that any URL parameters normally processed within the markup must now be passed to the javascript function for re-writing to the URL.

Ideals, Alternatives and Trade Offs
Relying on the BML to handle all things cookie/session related would have been the most ideal solution. URLs could have been re-written at BML generation time using cookie data without any javascript intervention. With the benefit of hindsight, the next best alternative would have been to utilize the onload event instead of hijacking the onclick event. This is because the onload event allows for the markup to retain control of its anchors – since re-writing happens well before URL click time.

This works fine for initialising a returning user’s state code since ALL URLs are modified with this data in the same way. However in situations where we needed to be more selective (extracting a user’s cinema preferences for only certain URLs), additional effort is required to identify/isolate the subset of URLs to operate on. Not all URLs need specific preference data appended to them. So the trade off is down to the assembly and redirect of URLs for a pre-determined anchor using the onclick method versus isolating a target set of anchors and appending data to an existing URL using the onload method.

<page>
    <script>
        <event type=”onload”>
            <!-- Append cookie data to a
                 target set of anchor tags -->
            <action eval="appendCookieDataToTargetURLs('{cookieName}',
                                                       '{targetId}',
                                                       '{URLParams}');" />
        </event>
    </script>
...
</page>

Conclusion
On the one hand it makes sense that BML can only provide superficial support for cookies given that its markup is never processed directly by the browser. On the other hand, if the NetBiscuits platform is communicating with client browsers and processing the BML on our behalf, could it not set its own session cookie and expose it through a set of functions within the markup? Where is NetBiscuit’s HTTPSession?! Perhaps the logistics of supporting potentially thousands of such sessions makes this concept infeasible.

To summarise, there may be further options available for enabling a mobile website that warrant a comparison but our experience with NetBiscuits was a happy and successful one. Cookie handling aside, the presentation layer development for our mobile app was super smooth. We also made use of a couple of the specialised tags for analytics and maps with relative ease.

The absence of full cookie support just means a little extra work. Perhaps we’ll see this built out in a future implementation of BML? In the meantime, its just old-school javascript know-how to enable cookie based sessions for your web apps. Of course, all this needs to be put in perspective with the main advantage provided by NetBiscuits: standardised mobile web development! Easy to overlook this significance when granted in such an accessible and ready-to-use format.