yob.id.au

New Zealand as Australia's Data Centre?

Sun, 08 Jan 2012 00:00:00 +1100

Like many rubyists I’ve often used Slicehost to host my virtual servers.

As a networking nerd who cares about latency (see It’s the Latency Stupid ) I’ve always wanted to host my client sites in Australia, however data prices here made it prohibitively expensive. Slicehost in the States was often the next best option.

Some time ago they were purchased by Rackspace and they’re finally preparing to migrate everyone to the Rackspace cloud platform.

I was happy enough with the migration, however it prompted me to do a quick scan of other options and I found something surprising. I can run a VPS at Rimuhosting’s Auckland data centre for about the same price as their US datacentre and it’s only a 50ms RTT from Melbourne.

It’s not unusual to find data costs with Australian VPS providers (including Rimu) hovering around AUD$2/Gb. At rimuhosting’s Auckland datacentre it’s NZD$0.20/Gb (currently about AUD$0.15).

What’s going on here? Are data prices in New Zealand cheaper than Australia? Is Rimuhosting oversubscribing their transit links? Are they buying budget transit? Are other New Zealand companies cheaper than their Australian competitors or is this an edge case?

Whatever the reason, I moved a clients VPS from the US to Auckland two weeks ago and couldn’t be happier with the result.

Preflight In Ruby

Fri, 29 Jul 2011 00:00:00 +1000

The PDF spec is a beast. Amongst a long list of features it allows transparencies, embedding of flash, video and javascript, content encryption and multiple colour spaces.

Such features have their place, but sometimes you just need to say no. Standards like PDF/A and PDF/X are subsets of the full PDF spec that target particular use cases like archiving or printing.

To check a given PDF conforms to a required standard you run a preflight check. There are expensive programs that can preflight, but they’re ….. expensive, and generally hard to script.

I recently released the preflight rubygem for performing preflight checks from ruby. It’s not as full featured as the commercial options, but it’s free and may solve 80% of your problem.

Start by installing the gem

    gem install preflight

To check a file passes a PDF/X-1a preflight it’s just three lines.

    require 'preflight'

    preflight = Preflight::Profiles::PDFX1A.new

    puts preflight.check("somefile.pdf").inspect

If you need more control over the profile, you can build your own profile.

    require "preflight"

    class MyPreflight
      include Preflight::Profile

      rule Preflight::Rules::MaxVersion, 1.4
      rule Preflight::Rules::DocumentId
      rule Preflight::Rules::OnlyEmbeddedFonts
      rule Preflight::Rules::MinPpi, 300
    end

    preflight = MyPreflight.new

    puts preflight.check("somefile.pdf").inspect

For a list of available rules, check out the classes in the Preflight::Rules namespace - each class has documentation describing how to use it in a profile.

If the provided rules don’t cover everything you need, you can also write your own.

For more details and examples, check out project on github.

Ethical Superation Followup

Sun, 24 Jul 2011 00:00:00 +1000

Some time ago I got angry with my faux-ethical superannuation and did some research on alternatives. Then I put off making a decision for 18 months. I excel at delaying hard decisions, and comparing Super funds is definitely hard.

This week I finally decided I needed to do something about it. I’m pessimistic about the short term prospects for global markets anyway, so I’ve decided to simultaneously lower my risk profile and move my superannuation to Australian Ethical.

Super company documentation makes comparing returns almost impossible, but I’m confident I’ll get returns I’m happy with whilst avoiding companies making money in unethical and unsustainable ways.

Db2fog

Sat, 23 Jul 2011 00:00:00 +1000

Several years ago Xavier Shay published his db2s3 gem for backing up the database from your rails project to Amazon S3.

Xavier’s original code depended on the aws-s3 gem. aws-s3 works well enough, however it appears to be unmaintained and has a few rough edges. In particular, it doesn’t cleanly handle S3 regions outside US-east. It also uses class variables for configuration, making it difficult to authenticate with alternate accounts at other points in my application.

The solution to these issues was to switch the dependency from aws-s3 to fog, a newer gem that is actively maintained. As a bonus, it also brings us support for additional storage providers like Rackspace cloudfiles and Ninefold.

After checking with Xavier I’ve forked his code and released a new gem - db2fog.

To use it, start by adding it to the Gemfile in your rails app:

    gem 'db2fog'

Then configure it by putting this in config/initializers/db2fog.rb:

    DB2Fog.config = {
      :aws_access_key_id     => 'yourkey',
      :aws_secret_access_key => 'yoursecretkey',
      :directory             => 'bucket-name',
      :provider              => 'AWS'
    }

Now use the rake tasks to perform backups and restores. Run them manually first, then add them to cron for automatic backups. I recommend running a few backup every few hours and the clean task daily.

    rake db2fog:backup:full  RAILS_ENV=production
    rake db2fog:backup:clean RAILS_ENV=production

That’s it! A dead simple way to get offsite backups that will probably cost you only cents per month. For details and examples for configuring sotrage providers like Rackspace check the project README on github.

Scan Like A Pro

Tue, 28 Sep 2010 00:00:00 +1000

Several years ago I read an article on scanning personal paperwork and reducing physical clutter. I liked the concept (except the part about using perforce!) and have since refined my own workflow to achieve something similar.

These days I scan and OCR nearly all paperwork I deal with (bills, contracts, letters, statements, etc), store them in a shallow directory structure and commit them to a git repository.

Step One: Scanner

For scanning, I use a document scanner at a clients office. I’m there a couple of days a week, so every now and again I take a pile of paper in and feed it through. They have a Xerox DocuCentre-II C2200 - it can scan 75 double sided pages at a time and saves PDFs to a windows share. Depending on the document, I scan in colour or greyscale at 600 dpi - the saved files are larger, but I deal with that later.

Step Two: OCR

The resulting PDFs have a single scanned image per page - I really wanted to OCR them if possible and make the content searchable. I explored various open source options that could run on my Linux systems, but none worked as well as I hoped. They generally had accuracy issues or would only output the recognised characters to a text file.

A few years ago I was working at a University, so I splashed out on an academic copy of Adobe Acrobat Pro 9.0 for Windows. Dual booting is a pain, but the OCR support is second to none and makes the hassle worthwhile.

Version 9.0 or later of Acrobat Pro has an OCR feature called ClearScan. After recognising the characters on each page, it converts them to a custom vector font and replaces the scanned image of that character with the vector character. This makes the text indexable by standard PDF search tools (like spotlight, etc) and has a massive impact on the file size. A 24 page text-only document I scanned last week went from 11Mb to 700Kb.

To use ClearScan open the scanned PDF Acrobat, select the Document menu, then choose “OCR Text Recognition” and “Recognise text using OCR”. Check the settings in the window that pops up and use the edit button to ensure “PDF output style” is set to Clearscan and “Downsample” is set to Lowest (600 dpi).

Step Three: Git

As a final step I store the files in a simple directory structure and save them to a git repository. A version control system might seem a bit odd, as the files generally don’t change much (if at all) but git makes it easy to store a backup of all the files on my server.

I group the documents by type and year. Anything more complex is unnecessary as I usually find what I need by searching. Here’s a sample of the directory structure:

    banking/2010/20100630 - statement.pdf
    banking/2010/20100720 - letter.pdf
    banking/2010/20100731 - statement.pdf
    insurance/2009/20090317 - contents renewal.pdf
    insurance/2010/20100315 - contents renewal.pdf
    receipts/2010/20100915 - laptop.pdf

Step Four: Profit

It has proven to be incredibly useful to have all my documents scanned and searchable. Not only is there less stuff floating around my house, but looking up records has become a trivial task.

I regularly look for files to send to my accountant and to check when I can expect a new bill to arrive. Recently I also had to look up receipts as part of an insurance claim.

Now that I have a standard workflow setup, the whole process is relatively little work and the results are convenient.

Extending Formtastic

Thu, 19 Aug 2010 00:00:00 +1000

Formtastic is a great little gem for cleaning up form code in your Rails view. It provides a neat DSL for building forms and outputs semantic markup with plenty of hooks for you to style with.

The project readme has extensive samples, but here’s what a simple form might look like:

    <% semantic_form_for @meeting do |form| %>
      <% form.inputs do %>
        <%= form.input :title %>
        <%= form.input :week %>
      <% end %>
      <% form.buttons do %>
        <%= form.commit_button %>
      <% end %>
    <% end %>

Formtastic is smart enough to detect your attribute types and renders an appropriate form element. If you don’t like what it chooses, overriding the element type is as easy as an extra option to the input method.

One limitation is that it only supports the standard form elements available in the rails API. What if you want something different - like some fancy image based thing? or a non-standard date picker? or some new-fandangled HTML5 input type?

The readme has a tantalising hint on custom inputs, but doesn’t go into much detail

    If you want to add your own input types to encapsulate your own logic or
    interface patterns, you can do so by subclassing SemanticFormBuilder and
    configuring Formtastic to use your custom builder class.

Imagine your model has a date attribute that always stores a Monday. You want users to pick a date by selecting from a list of options formatted with commercial weeks (week 22 2010, week 23 2010, etc).

Start by creating lib/my_custom_builder.rb. Create a class that extends the default formtastic builder and add your own input type. The default builder has a heap of helper methods you can use, so have that open at the same time.

    class MyCustomBuilder < Formtastic::SemanticFormBuilder
      def commercial_week_input(method, options = {})
        options = set_include_blank(options)
        html_options = options.delete(:input_html) || {}
        input_name = generate_association_input_name(method)
        selected_value = @object.send(input_name)
        selected_value = selected_value.strftime("%Y-%m-%d") if selected_value.respond_to?(:strftime)
        select_options = strip_formtastic_options(options).merge(:selected => selected_value)

        collection = mondays.map { |mon| [mon.strftime("Week %W, %Y"), mon.strftime("%Y-%m-%d")]}

        self.label(method, options_for_label(options)) <<
          self.select(input_name, collection, select_options, html_options)
      end

      private

      # return an array of dates for the next year of mondays
      #
      def mondays
        this_monday = Date.today.beginning_of_week.to_date
        (0..51).map { |i| this_monday + (i*7) }
      end
    end

Then edit your formtastic settings in config/initializers/formtastic.rb and tell it to use your new builder instead of the default

    Formtastic::SemanticFormHelper.builder = MyCustomBuilder

Restart your app, and then use the new input type in a form

    <% semantic_form_for @meeting do |form| %>
      <% form.inputs do %>
        <%= form.input :title %>
        <%= form.input :week, :as => :commercial_week %>
      <% end %>
      <% form.buttons do %>
        <%= form.commit_button %>
      <% end %>
    <% end %>

Flash 10 On 64bit Debian

Sat, 22 May 2010 00:00:00 +1000

Debian recently removed flash from their AMD64 to security issues and Adobe dropping official support.

I dislike flash as much as the next self respecting web developer, but some site (like google analytics) are pretty useless without it.

If you need a temporary solution to getting it up and running, manually download and install the old flash9 package from backports

    wget http://backports.mithril-linux.org/pool/contrib/f/flashplugin-nonfree/flashplugin-nonfree_1.4~bpo40+1_amd64.deb
    sudo dpkg -i flashplugin-nonfree_1.4~bpo40+1_amd64.deb

For further reading, check out the backports page for flashplugin-nonfree.

It’s ye olde, but it works.

New Mysql Ruby Bindings

Thu, 20 May 2010 00:00:00 +1000

There’s been some recent discussions about the way ruby 1.9 handles encodings and the exceptions people are running into, particularly when running rails.

A lot of the time, the exceptions are caused by database drivers passing back multi-byte strings incorrectly tagged as ASCII-8BIT instead of UTF-8.

The work around was ususally to do something like the monkey patch described here. I’ve used that particular patch in production for a few months and it works fine, but monkey patches make me squirm.

This week I was finally able to replace the patch with an alternative mysql gem that retrieves all strings from the database as utf-8 and marks them correctly.

Brian Mario has released mysql2. The basic API is different to the standard mysql gem, but he’s included an ActiveRecord adapter that acts as a drop-in replacement for the mysql one.

To give mysql2 a go in rails install the gem:

    gem install mysql2

Then update your database.yml:

    development:
      adapter: mysql2
      database: myapp_dev
      username: foo
      password: bar
      host: localhost

Make sure you get mysql2 version 0.1.6 or higher, in lower versions the ActiveRecord adaptor behaved slightly differently from the default Mysql one.

Brian is incredibly responsive to bug reports, so if you have any issues make sure you report them on the github project.

Ethical Superannuation

Sat, 16 Jan 2010 00:00:00 +1100

For all of my (admittedly short) professional life, my superannuation has been in Australian Super’s Sustainable Balanced fund.

I was more or less happy with this. Conventional wisdom says that non-profit industry super funds tend to perform better than commercial funds due to lower fees, and the sustainable option meant my money wasn’t being invested in too many companies I disagreed with.

However, I recently received a letter from Australian Super informing me of an upcoming change in their sustainable investment policies. The key paragraph was:

In the past our investment manager excluded companies that it believed to
receive a significant proportion of their revenue (at least 5%) from the
manufacture of sale of alcohol or tobacco, the operation of gaming
facilities or the manufacture of gambling equipment, uranium extraction or
the manufacture of weapons or armaments. This automatic exclusion will no
longer take place.

Hah. Imagine that. Presumably they were sick of missing investment returns from companies like BHP that are involved in uranium.

Whatever their reasoning, chances are my money is now being invested in places I’d prefer it not to be, so I’ve been researching my options.

The obvious alternative is Australian Ethical Super. Their investment policies gel strongly with my preferences and my non-super investments with them have been performing well.

Being a commercial fund, a comparison of fees was in order. The results were surprising and a little disappointing.

Australian Super’s Fees:

$78/year administration costs
0.77% of account balance
0% of contributions

Australian Ethical Fees:

$41/year administration costs
2.15% of account balance, calculated weekly
4.1% of contributions

For the 08-09 financial year, Australian Ethical fees for my super balance would’ve been a little more than triple what I actually paid Australian Super. Bugger.

All this is ignoring historical returns for each fund, a topic that would demand an entirely separate blog post. The short of it is that over the previous 5 years, Australian Ethical seems to have performed worse on comparable funds.

So do I move? Are there other ethical super options out there? What premium am I willing to cop in return for having my money invested in ethical companies?

Intro To The Pdf File Structure

Thu, 12 Nov 2009 00:00:00 +1100

For a couple of years I’ve been involved in various PDF related Ruby projects - most recently in prawn, a pure ruby PDF generation library.

Prawn is a long way from implementing all parts of the spec and there has been no shortage of people who desire support for various features. Many of these users are the kind who will submit a patch to an open source project when they find a missing feature, yet this has only happened to prawn in a small number of cases.

The PDF spec is epic (version 1.7 is over 1300 pages), yet the basic structure is less complicated than it first appears. Don’t be put off! We’d love more contributors to prawn, so this post is intended to be a basic introduction to reading a PDF file - the first step in understanding how prawn does the reverse process. I’ve created a simple PDF to use as an example.

The best way to think of the file is a set of objects (strings, hashes, symbols, integers, arrays, etc) in an object tree that descends from a single “root” hash.

To start navigating the example file by eye, open it in a text editor and browse to the last line. Just above the end of the file token (%%EOF) is a number that indicates a byte offset in the file.

That byte offset points to the “xref” token on line 46. Everything between here and the “trailer” token several lines below is called the Cross Reference Table (xref table to its friends). “0 6” indicates that this file has 5 objects in it numbered 1-5 (plus a number 0 null object). The following lines indicate object 1 is at byte offset 15, object 2 at offset 71, etc.

After the byte offsets you’ll see the following lines:

    << /Info 1 0 R
    /Size 6
    /Root 3 0 R
    >>

This is called the trailer and it is a serialised hashmap. In Ruby syntax, this would look something like:

    {
      :Info => "1 0 R",
      :Size => 6,
      :Root => "3 0 R"
    }

This is the top of our object tree. “1 0 R” is the syntax for pointing to object #1. Using our xref table we can see object 1 is at byte 17 and it looks like this:

    << /Creator (Prawn)
    /Producer (Prawn)
    >>

It’s another hash that looks like this in Ruby:

    { :Creator => "Prawn", :Producer => "Prawn" }

Merge this with our trailer hash and our object tree now looks like:

    {
      :Info => { :Creator => "Prawn", :Producer => "Prawn" },
      :Size => 6,
      :Root => "3 0 R"
    }

Repeat this process for every time you see “n 0 R” (called an indirect object) and you will have a complete PDF object graph.

Prawn’s job is to do the reverse - build an object graph that conforms to the spec and serialise it to a file in the correct format. Prawn has all the tools for serialising standard ruby objects into their PDF representation, so the trick is to find the correct place to add your new information (an extra font, an advanced colour definition, whatever) and insert it as a ruby object. Usually this means something like adding an entry to an array or hash.

Once that’s done, prawn will handle the serialisation when a user calls render.

For homework, download a copy of the PDF spec and read chapter 7 (particularly sections 7.3, 7.5, 7.7, 7.8 and 7.9). Also check out lib/prawn/pdf_object.rb to see how prawn converts each Ruby object into PDF equivalents.