A Real XML API and Rails

I recently implemented an XML API that I intended to be used outside of a web browser. Much of the words others have written on the topic are ways to get a Javascript framework to use the authentication_token magic. Some others show the GET side and mention the DELETE but omit the PUT and POST methods.

Here are things I've learned:

• To ensure XML data is returned, use the correct HTTP header: Accept: application/xml
• Rails assumes that multipart forms and url-encoded forms are from browsers. You can't use them in a default Rails setup if you want to avoid the authentication_token check.
• To POST or PUT, use a header of Content-Type: application/xml and include XML data.

It is rather unfortunate that Rails assumes that the encoding ties into a browser. It should be possible to use any encoding so long as the XML data types in the headers are correct. This is probably a bug, but it might be that if you receive XML through an API, you should send XML too.

Example

curl --user user:password -H 'Accept: application/xml' \
  -H 'Content-Type: application/xml' \
  -X POST -d '<?xml version="1.0" encoding="UTF-8"?>
<item>
  <name>foo</name>
  <description>bar</description>
  <price>100</price>
</item>' http://localhost:3000/items/create

Rails 2.0 and cool error handling

Rails is a great framework, but it has gained something of a bad reputation in terms of error reporting. Everyone has seen them -- those ugly 500-status "we're sorry, but something has gone wrong" messages.

In a finished application, this just doesn't cut it. It used to be necessary to trap the error using something like this:

def rescue_action_in_public(exception)
  case exception
  when ::ActiveRecord::RecordNotFound
    render :file => "#{Rails.public_path}/404.html", :status => 404
  else
    render :file => "#{Rails.public_path}/500.html", :status => 500
  end
end

The problem is that, as error causes begin to pile up, this method gets messy. Plus, you might want to render something different in one controller than in another, and would either need to duplicate all the logic, put controller-specific cases in the ApplicationController, or call ApplicationController::rescue_action_in_public directly.

Enter Rails 2.x error handling. Drop this in your controller or ApplicationController.

rescue_from(ActiveRecord::RecordNotFound) do |e|
  render :file => "#{Rails.public_path}/404_record_not_found.html", :status => 404
end

By building up a rich set of rescue_from handlers in the ApplicationController, and only overriding the ones you wish to make per-controller, you have fine-grained control. And they work exactly the same in development and testing as they do in production.

Ruby Regular Expression Gotchas

I love Ruby. I love Ruby on Rails. Rarely have I found a language or a framework that just works.

However, you still have to know the finer details sometimes.

I recently made a model for a DNS zone. The name in the model is the "front part" of a fully qualified domain name. For instance, if zone.name = "foo" then I would write the name into my name server's configuration files as "foo.example.com."

Knowing that people were evil, I saw that if a user put a string in like "example.com. NS hackerz-will-someday-rule-the-earth.ru.\nfoo" I would happily write out two strings, one being rather bad.

Knowing how easy this sort of data validation is in Rails, I made my model look like:

class Zone < ActiveRecord::Base
  validates_presence_of :name
  validates_uniqueness_of :name
  validates_format_of :name,
    :with => /^[a-zA-Z0-9\-\_\.]+$/,
    :message => "contains invalid characters."
end

Happy, I ran a few tests using my browser and found that I could not insert names with spaces, colons, tabs, etc. Then, several days later, I decided it was time to write tests for this.

require 'test_helper'
class ZoneTest < ActiveSupport::TestCase
  def test_name_with_newline_fails
    z = Zone.new(:name => "test\nzone")
    assert !z.valid?
    assert z.errors.on(:name)
  end

  def test_name_with_space_fails
    z = Zone.new(:name => "test zone")
    assert !z.valid?
    assert z.errors.on(:name)
  end
end

Imagine my surprise when test_name_with_space_fails() passed, and the one I was most worried about, test_name_with_newline_fails(), did not!

Not all regular expressions are alike

The problem is in what I thought ^ and $ actually matched. I thought these meant "match the beginning and ending of the string." However, it turns out it means "match the beginning and ending of each line contained in the string," where lines are divided by newlines. Ooops.

Changing ^ into \A and $ into \Z fixed this problem. Now I'm auditing all the code in this application to see if there are other problems like this.

This is just one thing to add to an ever-growing security checklist for my Rails work. It's also a very typical security hole: programmer error.

Mongrel, Apache, and Rails

When I first started running Rails applications on my web server, I chose to use FastCGI. Specifically, the mod_fcgid module, which had some features I wanted. It also has the unfortunate by-product of corrupting Apache's memory. Bad news.

I've since removed FastCGI entirely and moved to a proxy to mongrel_cluster setup. And I've started deploying with Capistrano.

Capistrano

I have a certain amount of concern with moving to a deployment system I knew very little about. Just like a new backup system, I feel like I'm handing the keys to my data over to something not written by me. And, while it is fairly simple to set up, Capistrano is somewhat complicated internally.

I already push out my operating system upgrades in an automated way. I compile NetBSD on one machine here at home, and push the binaries out to all my machines. This means about 7 machines rsync from the build box with one command. This can be scary, but I've been doing it for 5 years now, and it just works. How can a web site be scary compared to kernels and system binaries?

The answer is, it's not. If something breaks it is fairly easy to manually reconfigure if I need to. So, I've relaxed a bit. My concerns are still there, and I'm keeping a careful watch on how Capistrano runs each time I deploy. I have yet to do a *real* deployment after all! So far, I've not done a single migration, and have not had to roll back. And I'm pushing to a single machine, which runs the database as well as the site.

I suspect that, as I become comfortable with this new method to update my web sites, I'll start thinking of it as rsync++. It really is that simple.

mongrel_cluster

Mongrel is a vary amazing little widget. Sure, it's slower than Apache, but that's ok. Mongrel is still far, far faster than restarting Rails for each web hit, and far more reliable than mod_fcgid.

In my configuration, I run each site on ports 10000, 10010, 10020, etc. with up to 3 servers per. This means application #1 is on 10000 through 10002, with room to grow should I need to run more. If I find myself running more than 10 servers for a site it needs a new machine anyway, or more machines. And if that happens, I hope I'll have a budget.

Apache load balancing

This is a new feature in Apache 2.1, and apparently is very reliable with Apache 2.2. This is currently my favorite way to run a web site.

My configuration, which happens to be for this site:

<proxy balancer://blog>
  BalancerMember http://localhost:10010
  BalancerMember http://localhost:10011
  BalancerMember http://localhost:10012
</proxy>
<VirtualHost blog.flame.org:80>
  DocumentRoot /www/blog/flame-blog/current/public
  <directory "/www/blog/flame-blog/current/public">
    Options FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
  </directory>

  ProxyRequests off
  <proxy *>
    order deny,allow
    allow from all
  </proxy>

  RewriteEngine on

  # Check for maintenance file. Let apache load it if it exists
  RewriteCond %{DOCUMENT_ROOT}/system/maintenance.html -f
  RewriteRule . /system/maintenance.html [L]

  # Rewrite index to check for static
  RewriteRule ^/$ balancer://blog%{REQUEST_URI} [L,P,QSA]

  # Let apache serve static files (send everything via mod_proxy that
  # is *no* static file (!-f)
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
  RewriteRule .* balancer://blog%{REQUEST_URI} [L,P,QSA]
</VirtualHost>

It is important, at least on my host, to use localhost in the balancer destinations. This is due to mongrel suddenly running on IPv6 loopback (::1) rather than the usual IPv4 loopback (127.0.0.1). I don't know why this happened, but the localhost trick makes Apache try both addresses, and whichever works it will use.

This configuration makes Apache serve static content, and sends all other requests off to one of the Mongrel processes.

RailsMode (not ENV['RAILS_ENV'])

I've been spreading things like this all around my code, where I wanted to do something differently in production vs. development mode. Previously, I'd write something like this:

if ENV['RAILS_ENV'] == 'production'
  # perform magic, in production mode ...
end

While this is pretty simple, it just didn't feel very DRY. So, I decided to use this as a reason to learn about modules and mixins.

I have a generic plug-in in vendor/plugins that I put small bits of code like this. You might as well, but if you don't, you can drop this in a file in lib or in a helper.

module RailsMode
  def railsmode(*list)
    list.map! do |item|
      item.to_s
    end

    if block_given?
      if list.include?(ENV['RAILS_ENV'])
        yield
      end
    else
      return list.include?(ENV['RAILS_ENV'])
    end
  end
end

I also put this line in my environment.rb file:

include RailsMode

This mixes the module into the current class. Doing this in environment.rb makes it available everywhere in rails. Putting that line in a specific file would also work, such as a controller, or a helper.

With this, I can now write:

if railsmode(:production)
  # perform magic, in production mode ...
end

I can also check for multiple modes at once:

if railsmode(:production, :development)
  # perform magic, in production or development ...
end

And of course, who needs an if when I can pass in a block:

railsmode(:production) do
  # perform magic, in production mode ...
end