Shine Technologies Blog

One of my pet hates in software development is repetitive tasks: a complicated deployment process, tricky configuration of an application, repetitive editing motions that are just a little too messy for a find/replace. All of the above have, in the past, killed my concentration, dulled my senses and otherwise numbed my brain. I quickly realized that learning the ins and outs of a few powerful tools can take away a lot of the drudgery that sometimes crops up in my day-to-day work. Further, combining these tools makes it possible to automate just about everything that I need to do on a regular (or even not-so-regular) basis.

What’s important about everything that follows isn’t necessarily the process/tools itself, but what you get out of learning them. This is just what works for me, what has accumulated over a few years.

0. Pick a Good Editor (or IDE)

This first point is a half-hearted gesture because I don’t actually expect anyone to listen to me.

Use vim.

In all seriousness, if your editor doesn’t support basic macros and/or keyboard recording you’re going to encounter situations where you get slowed down by dull, repetitive editing. Even in an IDE that supports refactoring, macros will save you time and brain-dead copy/pastes. Automating repetitive tasks in editing source code is one of the simplest ways you can help yourself.

1. Learn Your Shell (or Learn Your IDE)

I think the Pragmatic Programmer said it well before I even knew what a “shell” was: the command-line is powerful.

Becoming very familiar with a powerful shell is one of the best things you can do for yourself as a developer. Yes, the syntax can be funky and the semantics confusing but over time you come to love it. For me, it’s the Unix shell: in *nix, the shell becomes the glue between all the other tools you use to get work done — including custom scripts and programs that take care of the heavy lifting for you. Windows users might be interested in PowerShell, which is a similar offering from Microsoft — you have the power of the .NET framework at your fingertips.

I understand, too, that folks accustomed to IDEs tend to get a little irked by shell programming. I still strongly recommend giving it a go, but scripting or extending your IDE to automate certain tasks you perform on a regular basis is a great start. If you find you lose a lot of time uploading and modifying configuration files to various servers, you may want to write an Eclipse plugin to make that process as simple as the click of a button.

Whatever path you choose — learning the more exotic aspects of a shell or the plugin API of your favourite IDE — learn it well. The only way to learn it well is to use it all the time.

2. Learn a Dynamically Typed Programming Language

If your shell scripts start to look like a train wreck when you need to perform complex operations, you may want to consider falling back to a scripting language: Perl, Python and/or Ruby are all great choices — bonus points if you mix the three to meet your needs. Languages like these will give you a more structured environment in which to express what you’re trying to achieve (at the cost of it being slightly more difficult to invoke external processes than, say, Bash — a standard Unix shell).

I think it’s important to note that I mention “dynamically typed” languages here because they generally don’t have a separate compilation step: bytecode compilation occurs as part of the execution of a Perl/Python/Ruby program. Thus, a small change doesn’t require an ant/make call to pull your code together again. Remember: you’re trying to automate a process, not create more process. Build management is overhead.

By the same token, if your program needs low-level access to hardware or raw speed for processing large quantities of data — don’t be afraid to fall back to a language like C or Java if you’re sure of your reasons for doing so. Needless to say, situations such as this are rare but not impossible.

3. Appreciation of the Supporting Cast

ssh, scp, rsync, svn/cvs/git, diff, patch, wget, sed, grep, find … the list of handy *nix command-line tools goes on and on. These are smaller tools that — with a good shell — can easily be combined to automate more complicated process. The output of these programs can also be piped into a script of your own doing to perform more complicated operations; then, the output of that program can be piped into another and so on.

This is the big thing I miss when working in Windows: even with PowerShell in play to alleviate the pain that is DOS, it has terrible tool support out of the box. Having the .NET framework on hand is great, but even that’s no substitute for the *nix command-line tools mentioned above. Rant aside, there are ways around most of the shortcomings in Windows and it’s possible to emulate some of the more useful command-line *nix tools. It just means you may need to put in more effort than a *nix counterpart at times.

For folks interested in automating with their IDE, your “supporting cast” might mean third party Java/.NET libraries which your automation plugin(s)/scripts can use to perform more complicated operations.

4. Invest Time in Learning Your Tools

Really, it all comes down to learning the tools from steps 1-3 backwards. Windows, Linux or OSX — you should be an expert in how your tools work and ideally use them on a daily basis. Really getting to know your tools might mean stepping out of your comfort zone a little to learn the nuts and bolts of how your shell ticks, or taking some time out of your day to learn certain esoteric API calls, or the semantics of some new construct in your scripting language of choice.

The goal of such practice is to surround yourself with an easily extensible suite of tools where every action feels like it’s an obvious choice. These tools will help to take away some of the unavoidable “noise” on the fringes of any software development project, and let you get on with getting your job done.

5. Write Lots of Scripts to Automate Common Tasks

This is at the core of automation: combine the above tools to get things done. Copy files to a server using scp, have your script extract and install it via ssh, manipulate configuration files using sed, grep and perl before finally starting the application and storing the process ID in a file with a quick shell command. All without you typing much more than “deploy”.

Automated tasks should be easy to kick off, require little-to-no input from you and — ideally — be fast, so you don’t have to wait long to determine if anything went haywire.

Automate at the First Sign of Resistance

Once you become accustomed to your tools — and again, by no means am I implying that those described here are the best or the only — you’ll step back when you start feel some resistance performing a task throughout your day:

Hmm. I don’t want to have to go through that 12-step deployment process all over again.

It becomes second nature to whip up a script that not only does the complicated deployment but verifies that the application is up and running as expected, checking a process exists and scanning the logs for hints indicating that everything’s fine. Over time you should find your automated tasks are easier to write and “feel” much more reliable than any manual process.

And, hopefully, you’ll be happier for it.

Share This

We have a new release of the GWT/Spring Shine Reference project out here (ver 0.3), which is an important improvement on the previous releases.

• Now includes proper Data Layer : using Spring 2.5 annotations, Hibernate 3.2 and integrated HSQLDB runtime memory database.
• Upgraded GWT gui to include save/delete/select functionality linking into the exposed Spring services
• Maven now builds deployable war files for both the presentation (gwt) and service (spring) layers that can be dropped into a standard Tomcat install.

All of this should “just work” straight out of the box - tested on linux (fedora), mac and windows systems. Follow the “Getting Started” instructions on the project page.

Share This

This was just too painful! Unfortunately I would have to recommend to people to stay away from Ganymede if you use Subversion for version control until they sort out connectors, update sites etc. After installing on 4 machines (3 macs, 1 linux) i’ve finally got 3 out of 4 working - but one of them (one of the identical macs!) just wont fix and the others were such hit and miss affairs. In theory, all you need to do is add in the update site as detailed on the Polarian site

Update Sites - Direct your Eclipse update manager to both the following update sites…
* Subversive plug-in update site :
http://download.eclipse.org/technology/subversive/0.7/update-site/

* Subversive SVN Connectors update site
http://www.polarion.org/projects/subversive/download/eclipse/2.0/update-site/

I realise that its to do with license issues etc, but Eclipse really has to “just work” out of the box with Subversion soon. The update manager really needs work too. Whoever put the “Close” button where they did while hiding off the “Install” button needs to go on a design course - so fed up clicking “close” just when i’ve chosen “install”….!

Share This

The Problem

As I work, I like to be able to do very regular checkins to our Subversion server (ensuring the tests pass beforehand, of course). This gives me an easy fallback position if I make some change that causes test breakages. Sometimes it’s just easier to rollback and start again than to try and figure out how I broke it. This is especially the case when refactoring.

We use Crucible for our code reviews, connected to our Subversion server. The problem is that when I want somebody to review a new feature or bug fix, the reviews might need to span a bunch of these fine-grained changesets. Crucible allows you to have multiple changesets in a review, but I don’t want people to have to review every little changeset - I want them to be able to just look at the difference between the start point and the end point.

A Solution

To get around this, I’ve been using Git to track my local changes, a custom script to bundle my changes into a single changeset, then git-svn to commit them to my Subversion server when I am ready.

I’m not going to give you a tutorial on Git here, but here are the basic steps involved:

1. Get Git and git-svn.
2. Clone your remote SVN repository into a local Git repository. If your SVN repository has a lot of history, this may take a while - but this is a one-off event. Go get a coffee whilst you wait.
3. Once it’s done, start coding. At this point it might be worth reading some of the introductory Git documentation to get an understanding of how it works. Commit your changes to your local Git repository as per normal Git usage.
4. Periodically rebase to the remote SVN repository using
   git svn rebase

   . This will ensure you’ve got the latest changes in the SVN repository.

5. Repeat steps 3 and 4 until you’re happy with your changes. Create branches if you want, but don’t merge between them (we’ll discuss why later).
6. When you’re ready to check-in to SVN, first use a custom bash script that was written by my colleague Tom Lee. Why do this? Well, to commit from Git to Subversion, you’d normally just use:
   git svn dcommit

   However, the problem with this is that’ll it commit a new changeset to SVN for each Git commit you’ve done. We don’t want this. Instead, we want to bundle everything up into a single commit first.

   The way to do is to create a temporary Git branch, and merge all of your changes into it. This will squash them into a single changeset, which you can then commit to SVN in one hit. Whilst you can do this manually if you want, Tom’s script does this for you automatically.

   Put the script in the same directory as your other Git executables and make it executable too. You can then run it as follows:

   git-prepare-svn-commit -m ‘Added some new feature’

   It’ll bundle everything up into a single changeset with the provided commit message. Note that this script has only been tested with Ubuntu - your mileage may vary.

7. Now you can check it in:
   git svn dcommit

   and it’ll just commit a single bundled changeset to your Subversion repository.

Additional Benefits of using Git

Git has a number of features that are very appealing. One that stood out for me is the ability to very quickly and easily create branches locally and then instantly switch between them. Often I would have a number of branches going at the same time, each one for different code enhancements or defect fixes.

In theory I could probably do all of this using Subversion branches, but Git offered one additional advantage - I don’t require a network connection to use it. I regularly work with it offline, only having to find a network connection when I’m ready to upload my changes to Subversion. I also have the added bonus that I don’t end up with a bazillion branches hanging around on the Subversion server.

It’s worth noting that I’ve never merged between local Git branches that originated from the Subversion server - indeed, the git-svn documentation recommends against it (see the ‘Caveats’ section of the git-svn Manual Page). However, this wasn’t ever a problem for me; by committing finished work to Subversion, switching to a Git branch containing unfinished work and then rebasing against the Subversion repository, code could easily be transmitted between branches.

Drawbacks of using Git
I found Git to have a steep learning curve. In particular, it’s two-stage approach to commits took a little getting used to. This problem was exacerbated by the fact that I was using git-svn as well. The best example of this was the extreme bewilderment I experienced when I first encountered merge conflicts during a rebase against the Subversion repository.

The best piece of advice I can give you if you encounter such a conflict during a rebase is to do what Git tells you. Git will provide a number of ‘what you can do next’ instructions when it encounters a conflict that it needs you to resolve. Read these instructions carefully and follow them. I tried to short-cut them and ended up hopelessly tangled up.

Finally, there’s not much Git tool support out there at the moment, so you’re pretty much gonna be doing it all from the command-line.

Credits
Thanks to Tom Lee for breaking a trail to this solution - I knew what I wanted and thought Git might provide a solution, but Tom was the one who actually figured it out and put it all together. He also pulled me out of a few holes that I dug myself into during the learning process.

Share This

Following on from Tom’s post on Handling AJAX Errors using Prototype & Rails, I thought I’d take a look at how to ensure that all Ajax calls are subject to decent error handling - or any other sort of handling you want to do.

Towards the end of his post, Tom showed how he overrode form_remote_tag to include his error handling hooks. The only catch is that form_remote_tag isn’t the only way to trigger an Ajax call - there’s also form_remote_for, link_to_remote and a bunch other methods in ActionView::Helpers::PrototypeHelper.

You could override all of them if you wanted to, but it might just be easier to override the granddaddy of them all: remote_function. As far as I can tell, all of the PrototypeHelper methods use it, and you might even have to use it directly yourself for particularly sticky problems (that’s why it’s part of the public API). It’d look something like this in your helper class:

def remote_function_with_user_feedback(options)
    # .. Do extra stuff...
    remote_function_without_user_feedback(options)
end
alias_method_chain :remote_function, :user_feedback

Of course, that does beg the question: what exactly is the ‘extra stuff’?

Well, you could setup a callback that notifies the user if a server error occurs:

options[:failure] = (options[:failure] || '') +
        'alert("An error has occured on the server. Please try again later."); '

Or you could even try out the more sophisticated network-outage detection code in Tom’s original blog entry.

Alternately, have you ever clicked a button in an Ajax app and been left wondering why nothing seems to be happening? Wouldn’t it be good if something started spinning to tell you that ’stuff’s happening’? Well here’s how you could do it:

options[:before] = (options[:before] || '') +
        "$('spinner').src = '#{image_path "#active-spinner.gif"}';"
    options[:complete] = (options[:complete] || '') +
        "$('spinner').src = '#{image_path "#inactive-spinner.gif"}';"

Note that in each case, it’s important not to overwrite any existing Javascript callbacks that may have been provided to remote_function - instead we just concatenate our own Javascript code to them. Thanks to Tom for pointing this out.

It’s also worth remembering that you’re extending a pretty important Rails method here so make sure you get it right…and by that I mean make sure you’ve got some tests

Share This

I recently did a presentation to the Melbourne Ruby Users Group entitled ‘Flex and Rails’. It demonstrates how we’ve used Flex to build a rich web interface for a real-world Rails application. It includes a demo and code snippets. I’ve made a streaming screencast of the presentation - you’ll find it here (requires QuickTime). Check it out and let me know what you think.

Share This

Having accomplished one of my main short term goals in life, I thought I would reflect on the recent events that earned me my first certification outside of Uni.

You might be thinking, whats the big deal about answering a couple of Java questions, nabbing the cert and moving on with life. Besides, we all sat for the tricky test tailored by Shine that got us this job. The only way to answer your question is to try the questions in the diagnostic exam by Whizlabs or head over to the Java Ranch website and try their mock exams.

A couple of us sitting for the exam tried out the questions on Whizlabs and had our butt kicked. Things we thought we knew about Java. The hardest part about achieving our goal was the preparation. Our initial plan was to pass the exam by December last year, however, other commitments at the client site threw our schedule off course. By the time we got back to studying which was around February, our knowledge of the first few chapters were stale. To make it worse, I had to regain my motivation to put in additional hours into the preparation which is seperate from Shine’s training time.

February was rather short and as March approached I was entering panic mode. By this time, I had covered about five chapters which left me with another five more to go. The next few weeks were spent flying through the remaining chapters and trying out the self-tests that came with at the end of each chapter. I tried out the Whizlabs exam to get an estimate of where I am at and I was rather disappointed with my score of 30%. The good thing is, I get to study the questions I got wrong and work on my weak points. I tried the questions on the CD that came attached with the SCJP book I was studying and I found the questions much tougher and trickier. The whole objective is to thoroughly test your knowledge.

I decided to stop doing the questions on the CD as the results I was getting were rather demoralizing. I spent Easter finishing up the last chapter and doing some practice exams with Whizlabs. One of the things that I look out for while doing an exam, is the clock. For example, if you had 72 questions to answer within 3 hours, you would have about 2.5 minutes to answer every one of them. I adopted an approach of answering the “easy” questions that I am confident with first and “Marking” those that I could not answer or I was unsure off. If you still have about an hour left after your first run, you can relax a little and review those questions that you have not answered. Try your best to give an answer and since its multiple-choice, analyze each answer and deduce which one feels right. You should still have abit of time left after answering your questions, so go back to the ones that you have marked. From personal experience, I found that I was able to tackle the questions that I was unsure about after I had answered the rest of them. Some tough questions really do not mix well with stress and you will be amazed at your ability to tackle them after the initial hurdle.

I was really nervous at the actual exam but the valuable experience gained with Whizlabs gave me that confidence to see it through. The questions are actually easier than the Whizlab’s ones but they are designed to keep you engrossed! So whatever you do, watch the clock, mark tough questions and review!

P.S. I have never passed a practice exam on Whizlabs before. Nearly did, but not counted as a pass. I scored 70% on the actual exam and its the first exam I ever passed. If you are coping well with the Whizlabs exams, you know you are doing well.

Share This

Ruby on Rails comes with a lot of nice helper methods for generating the JavaScript driving the AJAX calls to your controllers. Handling the responses from the HTTP server becomes a snap too, with Rails providing a few simple callbacks to handle the response from the server:

But what if the server doesn’t get a chance to respond at all? What if the user’s browser has been unexpectedly disconnected from the Internet? What if your server - god forbid - has crashed?

Most AJAX applications in the wild will simply sit there forever waiting for a response. Certainly, that’s what happened to us when we first tested such a scenario. This is unacceptable: if the browser is unable to communicate with the server, we need to let the user know somehow.

Internally, Rails uses a cross-platform Javascript library called Prototype to do the heavy lifting when making AJAX calls. When you call a helper method like link_to_remote, Rails generates code to instantiate a Prototype Ajax.Request object (or a Ajax.Updater object in some cases) to make the remote call when a certain event occurs. In the case of link_to_remote, this will be when a user clicks on the generated link.

Many of the helper methods accept a common set of options which provide callbacks for handling success or failure using the following callbacks:

• :success will be executed if the server responds to the AJAX request with any sort of 200 HTTP OK code.
• :failure will be executed if the server responds to the AJAX request with anything *but* a 200 code.
• :complete will be executed after the AJAX request has finished, irrespective of the result.

These callbacks can be mapped to corresponding callbacks in Prototype on a one-to-one basis. For the above callbacks, the corresponding Prototype callbacks are called onSuccess, onFailure and onComplete, respectively. Unfortunately, only the “complete” callback will be triggered if some fatal error occurs when communicating the server - “failure” will only be triggered if we get a response back from the remote host indicating a server-side. We have no way of determining what will happen if the browser is unable to communicate with the server at all.

So how do we detect server outage? Well it just turns out that if your browser is unable to communicate with the server, an exception will be thrown. We can actually use this as a basis for detecting server outages and other classes of errors that might occur browser-side. For this purpose, Prototype provides onException which is triggered by your browser if any sort of exception is thrown while the AJAX call is in progress. If we can tap into this callback, we can ensure our AJAX methods will dutifully report any problems that might occur.

Unfortunately Rails doesn’t yet provide access to this callback out-of-the-box, so there is no “:exception” callback available to the Rails helpers. We don’t want to clutter our templates with ugly JavaScript: we’d prefer to keep with the existing Rails conventions if we can. So what can we do?

PrototypeHelper#build_callbacks is used by remote_function to generate the Javascript responsible for making an AJAX call in helper methods like link_to_remote, form_remote_tag and others. It turns out that this little method exposes virtually all the callbacks available to Prototype *but* onException.

This is the hook we’re looking for: here we override and adapt the original code in PrototypeHelper#build_callbacks to add the following method to our ApplicationHelper:

include ActionView::Helpers::PrototypeHelper

  # ...

  def build_callbacks(options)
    callbacks = {}
    options.each do |callback, code|
      name = 'on' + callback.to_s.capitalize
      if CALLBACKS.include?(callback)
        callbacks[name] = "function(request){#{code}}"
      elsif callback == :exception
        callbacks[name] = "function(request,exception){#{code}}"
      end
    end
    callbacks
  end

This makes the :exception callback available to Rails using the familiar notation seen for the other types of callback. :exception is somewhat different in that it provides the exception object thrown internally available to your callbacks as the “exception” variable. Outside of that, it’s just like any other callback.

With this new callback available to us, we can then override the AJAX helper methods to include our error handling code in ApplicationHelper. As an example, here’s how we’d do it for form_remote_tag:

alias old_form_remote_tag form_remote_tag

  def form_remote_tag(options={}, &block)
    # In the event of a network failure (or some other error), we want to display an error message
    options[:exception] = (options[:exception] || '') +
      'alert("Anerror occured while communicating with the server. Please try again later.");' +
      # Bubble the javascript exception up to prevent the caller from continuing execution (failure
      # to do this may result in multiple error messages being displayed!).
      'throw exception;'
    old_form_remote_tag(options, &block)
  end
end

With this code in place, any attempts to submit a form via AJAX will include our special error detection code, so we can rest at ease knowing our users will be notified if something goes horribly wrong at the network or language leve.

When using this technique to check, you have to be quite careful in your JavaScript code. Where any language- or browser-level exception that was thrown during the AJAX request was once ignored, it will be caught by the :exception callback. This means syntax errors in the response Javascript, bad content types (e.g. a content type of “text/javascript” with a body of HTML) or other errors will trigger the :exception callback. Unfortunately there doesn’t seem to be an obvious way to get more information about the nature of the error. For our purposes, however, this seemed to be enough if we were careful with the AJAX responses.

A further improvement to this approach might use a timeout mechanism to provide a hard limit for AJAX requests: if the server doesn’t respond with data within X seconds, abort the request and display an error. This would remove the reliance on the browser reporting errors via an exception, resulting in somewhat more robust code - the feedback may not be as immediate as if one were using exceptions, however.

Don’t leave your users out in the dark: keep them informed if things go wrong with an AJAX request.

Share This

A while ago Ben wrote about how Ruby on Rails development sucks on Windows. It seems that the move by the Rails Core team from subversion to Git has inflamed this issue further.

My favourite comment is:

If you’re freaking out, calm down. Rails and the developers behind it have snubbed Windows far, far worse in the past :).

Share This

Once in a while new technologies mature and coalesce together to provide a new and viable platform that radically change the traditional development landscape. In recent years Ruby on Rails has been the poster child of all that is new and groovy; the “must learn” skill for developers, leaving behind those “legacy” Java systems built over the last ten years.

It looks like GWT (Google Web Toolkit) is challenging those ideas. We’ve written an article explaining the steps involved putting together a multi-tiered system using GWT and Spring together (with both layers built by Maven). This also comes with an Open Source reference project showing in detail how to get up and running very quickly. Full details here.

What makes me particularly happy about all this is , is that it looks like there is still life in the old Java dog yet

Share This