OnRails.org

Ever forget what all the Rails defined Date/DateTime/Time#strftime formats are? Or forget what ones you added to the project yourself?

Ala rake routes comes rake date_formats:

Date
====
            db:'%Y-%m-%d'   2008-08-20
  long_ordinal:'&proc'      August 20th, 2008
          long:'%B %e, %Y'  August 20, 2008
        rfc822:'%e %b %Y'   20 Aug 2008
        number:'%Y%m%d'     20080820
         short:'%e %b'      20 Aug

DateTime
========
            db:'%Y-%m-%d'   2008-08-20 16:56:21
  long_ordinal:'&proc'      August 20th, 2008 16:56
          long:'%B %e, %Y'  August 20, 2008 16:56
        rfc822:'%e %b %Y'   Wed, 20 Aug 2008 16:56:21 -0600
        number:'%Y%m%d'     20080820165621
         short:'%e %b'      20 Aug 16:56

Time
====
            db:'%Y-%m-%d %H:%M:%S'         2008-08-20 16:56:21
  long_ordinal:'&proc'                     August 20th, 2008 16:56
          long:'%B %d, %Y %H:%M'           August 20, 2008 16:56
        rfc822:'%a, %d %b %Y %H:%M:%S %z'  Wed, 20 Aug 2008 16:56:21 -0600
         short:'%d %b %H:%M'               20 Aug 16:56
        number:'%Y%m%d%H%M%S'              20080820165621
          time:'%H:%M'                     16:56

I’ve recently upgraded to OS X 10.5 (Leopard), and all-in-all, I’m pleased with the experience. My biggest issue has been the default stacks behavior—the icon changes to the last thing added to the stack, making visual identification unnecessarily cumbersome. I worked around this annoyance (as outlined here) by changing the sort to name rather than date added, and adding a dummy folder named “_1” that will sort to the top. For extra bonus points, I customized the icon of the dummy folder. For some yet unknown reason, the most recently downloaded item still peeks through from time to time, but it’s much better than before.

Maybe it’s my Windows history showing through, but I went with the “clean-sweep” erase and install method. For a non-developer, I’d probably recommend the upgrade (and in fact I used that method for my Father-in-law’s MacBook), but I had lots of custom bits scattered about my machine, and didn’t want to be chasing any incompatibility gremlins.

So now, to get my development environment set up on the new machine… Leopard includes a fairly complete Rails stack out of the box, with a non-broken Ruby, readline support, and most of the commonly used gems. Read more here.

MySQL was not included, but the latest installer (mysql-5.0.45-osx10.4-i686.dmg) for 10.4 from dev.mysql.com downloads worked (mostly) fine. The Server and the StartupItem install and operate correctly. The PrefPane installs, but does not appear to actually do … anything. I’ll have to work on that, but I can live without it for now. After a bit of manual hacking on my database dump file from Tiger (where I was running a 5.1.x beta of MySQL), all my databases are back in place.

One last piece that I needed for my Rails apps—RMagick. I know it’s possible to install RMagick and its dependencies, um, “autoRMagickally” via a package management system like MacPorts or Fink, but I prefer not to. For some background on why not, you can read this article at hivelogic. The last time I was rebuilding my laptop and desktop near the same time, I put together a shell script to automate the process of installing RMagick. I got it back out and dusted off the cobwebs, and voila! RMagick on Leopard. (note: replace wget with “curl -O”, if you don’t have wget installed on your machine) Here’s the code:

#!/bin/sh
wget http://download.savannah.gnu.org/releases/freetype/freetype-2.3.5.tar.gz
tar xzvf freetype-2.3.5.tar.gz
cd freetype-2.3.5
./configure --prefix=/usr/local
make
sudo make install
cd ..

wget http://superb-west.dl.sourceforge.net/sourceforge/libpng/libpng-1.2.22.tar.bz2
tar jxvf libpng-1.2.22.tar.bz2
cd libpng-1.2.22
./configure --prefix=/usr/local
make
sudo make install
cd ..

wget ftp://ftp.uu.net/graphics/jpeg/jpegsrc.v6b.tar.gz
tar xzvf jpegsrc.v6b.tar.gz
cd jpeg-6b
ln -s `which glibtool` ./libtool
export MACOSX_DEPLOYMENT_TARGET=10.5
./configure --enable-shared --prefix=/usr/local
make
sudo make install
cd ..

wget ftp://ftp.remotesensing.org/libtiff/tiff-3.8.2.tar.gz
tar xzvf tiff-3.8.2.tar.gz
cd tiff-3.8.2
./configure --prefix=/usr/local
make
sudo make install
cd ..

wget http://jaist.dl.sourceforge.net/sourceforge/wvware/libwmf-0.2.8.4.tar.gz
tar xzvf libwmf-0.2.8.4.tar.gz
cd libwmf-0.2.8.4
make clean
./configure
make
sudo make install
cd ..

wget http://www.littlecms.com/lcms-1.17.tar.gz
tar xzvf lcms-1.17.tar.gz
cd lcms-1.17
make clean
./configure
make
sudo make install
cd ..

wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/gs860/ghostscript-8.60.tar.gz
tar zxvf ghostscript-8.60.tar.gz
cd ghostscript-8.60/
./configure  --prefix=/usr/local
make
sudo make install
cd ..

wget ftp://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/current/ghostscript-fonts-std-8.11.tar.gz
tar zxvf ghostscript-fonts-std-8.11.tar.gz
sudo mv fonts /usr/local/share/ghostscript

wget http://imagemagick.site2nd.org/imagemagick/ImageMagick-6.3.5-9.tar.gz
tar xzvf ImageMagick-6.3.5-9.tar.gz
cd ImageMagick-6.3.5
export CPPFLAGS=-I/usr/local/include
export LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local --disable-static --with-modules --without-perl --without-magick-plus-plus --with-quantum-depth=8 --with-gs-font-dir=/usr/local/share/ghostscript/fonts
make
sudo make install
cd ..

sudo gem install RMagick

Rails makes things easy on developers—maybe too easy. It’s not uncommon to reference an association while iterating over a collection of objects, resulting in a performance-devouring N+1 queries being executed. Of course, this is easily fixed with eager loading of the association, but you get my point. Rails can be a big gun, and it’s easy to blow off your foot.

Once your application is moved into production, it becomes important to keep an eye on performance over time in order to get a feel for trends and plan capacity intelligently. Numerous Rails performance measuring tools exist, but I find that it helps to correlate performance of your application with simultaneous lower-level performance metrics of your system (CPU and Memory usage, Load Average, etc). Besides that, hey! Pretty Graphs! Enter munin, an open-source, extensible monitoring tool with a number of out-of-the-box plugins that are useful to SysAdmin type folks. There’s a pretty decent (though brief) writeup on howtoforge that explains a bit more about how to go about getting your own munin. Go ahead and check it out—I’ll chill here until you’re back.

That wasn’t too bad, huh? Okay, so at this point you should have a working munin instance. It will take a few minutes of data collection before your graphs look like anything, but going forward, you’ll have historical graphs of several key metrics for managing your server(s). It’s outside the scope of this article, but you can also set up munin to monitor multiple servers on your network, and/or alert you when critical threshold levels are passed.

Well, knowing CPU usage is great, but it would also be nice to have some idea what the average user experience is like on your site. Does your Rails app perform differently at different times of day? How long does it typically take for your app to render a page? Is the majority of the time spent in the database, or rendering? Is the response time about the same between production deployment versions? I wanted answers to these questions too, so I came up with a small ruby program that watches your Rails log in realtime, and when munin asks, provides summary information about the performance of your application. Below, I’ll go over the code section by section; for those who want it now, scroll to the bottom of the article to find the download link.

The basis of the solution is a Ruby array: we stuff values into it, then compute the average of all the values and clear the array each time munin pings us. Every time a value is added, we also check it against the maximum already seen, so we can report the maximum response time in addition to the average. We keep three of these objects around, one each for DB, Rendering, and Total response times. There is also a mode that lets you look at the current value without consuming it—useful for peeking inside without affecting the data that munin will ultimately see. Here’s the class that implements this functionality:

class Accumulator
  def initialize
    @values = Array.new()
    @max = 0
  end

  def add(value)
    @values << value
    @max = value if value > @max
  end

  def average(read_only=false)
    return_value = if @values.length == 0
      nil
    else
      @values.inject(0) {|sum,value| sum + value } / @values.length
    end
    @values = Array.new() unless read_only

    return_value
  end

  def max(read_only=false)
    return_value = @max
    @max = 0 unless read_only
    return_value
  end
end

In the next section of the code, we build our accumulators, and begin tailing the logfile to extract performance numbers. This requires the file-tail gem, available from rubyforge. Note that in my setup, this file resides in a subdirectory under lib in RAILS_ROOT. If you choose to place this file elsewhere, you’ll have to adjust the path to the logfile accordingly. Another thing to note: in our environment, the load balancer continually pings a “heartbeat” action on each node to make sure it is still responsive. As we will be hitting this action repeatedly, it is engineered to be as lightweight as possible. Therefore, any numbers from it are pretty meaningless to our overall statistics, so we don’t want to include them. To keep these numbers from skewing our results, we define an IGNORE_PATTERNS regexp (earlier in the code). If the request matches a pattern we want to ignore, its statistics are not collected.

LOGFILE = File.join(File.dirname(__FILE__), '..', '..', 'log', "#{RAILS_ENV}.log")
$response_data = { :total     => Accumulator.new(),
                   :rendering => Accumulator.new(),
                   :db        => Accumulator.new() }

Thread.abort_on_exception = true
logtail = Thread.new do
  File::Tail::Logfile.tail(LOGFILE) do |line|
    if line =~ /^Completed in /
      parts = line.split(/\s+\|\s+/)
      resp = parts.pop
      requested_url = resp[/http:\/\/[^\]]*/]
      next if requested_url =~ IGNORE_PATTERNS

      parts.each do |part|
        part.gsub!(/Completed in/, "total")
        type, time, pct = part.split(/\s+/)
        type = type.gsub(/:/,'').downcase.to_sym

        $response_data[type].add(time.to_f)
      end
    end
  end
end

So now we have a thread busy gathering our data—how can we expose the data to a munin plugin? There are multiple ways to do this, but I chose to use a small HTTP server listening to requests from the local machine only. We could build such a thing in Rails, but we really don’t need about 90% of the features Rails has to offer. Since we’re running Rails as a Mongrel cluster, we already have a perfect tool at our disposal for writing small HTTP request handlers in Ruby: Mongrel. Here are a couple of pages about how to get started writing Mongrel handlers—it’s pretty straightforward. Here’s our handler:

class ResponseTimeHandler < Mongrel::HttpHandler
  def initialize(method)
    @method = method
  end

  def process(request, response)
    response.start(200) do |head,out|
      debug = Mongrel::HttpRequest.query_parse(request.params["QUERY_STRING"]).has_key? "debug"
      head["Content-Type"] = "text/plain"
      output = $response_data.map do |k,v|
        value = v.send(@method, debug)
        formatted = value.nil? ? 'U' : sprintf('%.5f', value)

        "#{k}.value #{formatted}"
      end.join("\n")
      output << "\n"
      out.write output
    end
  end
end

h = Mongrel::HttpServer.new("127.0.0.1", PORT)
h.register("/avg_response_time", ResponseTimeHandler.new(:average))
h.register("/max_response_time", ResponseTimeHandler.new(:max))
h.run.join

The handler is generic so that it can call an arbitrary method on our collection of data arrays, so we can set up one URI for the average, and one for the maximum. With this in place, we can write a simple munin plugin script that uses Net::HTTP to query our Mongrel to get at the performance data.

Basically, a munin plugin has two usage scenarios. When called with the single argument “config”, it should output information about itself in a format that Munin understands. This includes how to label the chart, scaling information, how many series will be included on the chart, etc. When called with no arguments, the plugin should output the current values of each series. For more information on writing your own munin plugin, start with the HowToWritePlugins munin wiki page. And now, our plugin script:

#!/usr/bin/env ruby

# munin plugin to render rails response time graphs
# link to /etc/munin/plugins/avg_response_time and /etc/munin/plugins/max_response_time

require 'open-uri'
PORT = ENV['PORT'] || "8888"

def config
  title = File.basename($0).split('_').map{|s| s.capitalize }.join(' ')
  config=<<__END_CONFIG__
graph_title #{title}
graph_vlabel response time
graph_category rails
total.label total
rendering.label rendering
db.label db
__END_CONFIG__
  puts config
end

def get_data(read_only=false)
  qs = read_only ? '?debug' : ''
  puts open("http://127.0.0.1:#{PORT}/#{File.basename($0)}#{qs}").read
end

case ARGV.first
when 'config'
  config
when 'debug'
  get_data(true)
else
  get_data
end

The script will examine the name with which you linked it in to the munin plugins directory to determine which URI to query. I have also added a debug mode that will show you the current values, so you’re not consuming any data that munin needs to see for an accurate graph. The final piece is a small Daemons wrapper script to control the main log-tailing process, and you should be set. Make sure to restart munin-node so it will notice the new plugins, and after a while, you’ll see something like this:

It is worth noting that the numbers from the Rails log might not be 100% accurate, and this won’t replace the results that you can get from seriously profiling your application. Also, the information you are getting is a bit generic—all actions are lumped together, so there is not a lot of information about the cause of the performance problem. But, for insight into your production application performance, this setup should at least give you some indications about how well your baby is playing in the interwebs.

Download rails_log_monitor.rb

Download rails_response_time

I’ve updated my downloader from earlier to include all sorts of fancy options. It no longer requires wget, it just uses open-uri. It can give the files a fancy name. It can be told where to download the files to. It will skip files that won’t download for some reason. It will even butter your toast if you can find the correct command line switch.

It’s about 3 times bigger than the previous one. But maybe you can learn a little more about optparse, hpricot, file handling, and error handling along the way.

Here it is:

#!/usr/bin/env ruby

require 'optparse'

OPTIONS = { :Verbose => false,
            :Force => false,
            :DownloadDir => '.',
            :DescriptiveFilenames => true
          }
OptionParser.new do |opts|
  opts.banner = "Usage: #{$0} [options]"

  opts.on("-v", "--[no-]verbose", "Run verbosely, default #{OPTIONS[:Verbose]}") do |verbose|
    OPTIONS[:Verbose] = verbose
  end
  opts.on("-f", "--[no-]force", "Force downloads, default #{OPTIONS[:Force]}") do |force|
    OPTIONS[:Force] = force
  end
  opts.on("-d", "--[no-]descriptive", "Use long descriptive filenames, default #{OPTIONS[:DescriptiveFilenames]}") do |long|
    OPTIONS[:DescriptiveFilenames] = long
  end
  opts.on("-p", "--path PATH", "Path to download to, default #{OPTIONS[:DownloadDir]}") do |path|
    OPTIONS[:DownloadDir] = path
  end
  opts.on_tail("-h", "--help", "Print help message") do |help|
    puts opts
    exit
  end
end.parse!

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'fileutils'

BASE_URL = 'http://www.web2expo.com'

def log(str)
  puts str if OPTIONS[:Verbose]
end

def download(href, filename)
  url = "#{BASE_URL}#{URI.escape(href)}"
  download_file = File.join(OPTIONS[:DownloadDir], filename)
  if OPTIONS[:Force] || !File.exists?(download_file)
    log "downloading #{File.basename(href)}..."
    begin
      File.open(download_file, 'w') { |f| f.write(open(url).read)}
      log "\tsaved as #{download_file}"
    rescue Object => e
      FileUtils.rm(download_file)
      $stderr.puts "ERROR downloading #{url}: #{e.message}"
    end
  else
    log "skipping #{File.basename(href)}... already downloaded as #{download_file}"
  end
end

FileUtils.mkdir_p(OPTIONS[:DownloadDir])
h = Hpricot(open("#{BASE_URL}/pub/w/51/presentations.html"))
h.search('div.presentation').each do |presentation_node|
  href = presentation_node.at('a[@href^="/presentations/rails2007/"]')[:href]
  if OPTIONS[:DescriptiveFilenames]
    name = presentation_node.at('b a').inner_text.strip
    text = presentation_node.inner_text
    speaker = text[/Speaker\(s\):\s+(.*)\s*$/, 1]
    date = Date.parse(text[/Presentation Date:\s+(.*)\s*$/, 1])
    filename = [speaker, date, name, File.basename(href)].compact.map { |s| s.to_s.strip.gsub(/[^\w\.]/, '_').squeeze('_') }.join('-')
  else
    File.basename(href)
  end
  download(href, filename)
end

Updated: Now more bloated!

#!/usr/bin/env ruby

require 'rubygems'
require 'hpricot'
require 'open-uri'

base = 'http://www.web2expo.com'
h = Hpricot(open("#{base}/pub/w/51/presentations.html"))

h.search('div .presentation > a[@href^="/presentations/rails2007/"]').each do |a|
  url = "#{base}#{a[:href]}"
  if File.exists?(File.basename(url))
    puts "skipping #{url}... already downloaded"
  else
    puts "downloading #{url}..."
    `wget --quiet #{url}`
  end
end

So you’re building some righteous automation for your killer web 2.0 app, placing scripts in RAILS_ROOT/script that you can call from cron for nightly maintenance, etc. To bootstrap your rails environment, you decide to use the shebang feature of script/runner, available since changeset 5189. When you start to edit the script in TextMate (you are using TextMate, aren’t you?) there is no syntax highlighting to be found! It’s all plain text with no colors, and none of your ever-so-helpful keyboard macros work! Frightful. Well, take a deep breath, because together, we’re going to get the filetype detection magic working for you.

Before we get started, it’s helpful to know how filetype detection works. TextMate does a couple of different types of filetype detection—the first is based off of the extension, so if you named your script with a .rb extension, you are probably wondering what in the world I’m rambling about. Dude. It just works.

However, if you followed the rails convention for scripts, and did not use an extension with your filename, keep reading. The second type of detection works by scanning the so called “shebang” line at the top of the script which tells the shell (and in this case TextMate) which interpreter to use to evaluate your script—this is how we will tell TextMate that script/runner really means ruby.

First of all, you’ll need to fire up the Bundle Editor and select “Languages” from the drop-down filter. Expand the “Rails” node, and then select the “Ruby on Rails” language. On the right side, you should see the definition being used by TextMate to detect the Ruby on Rails scope. If you have not modified your bundle, you’ll probably see that it is using a fileTypes to look for .rxml files. This is where we want to insert the following line:

firstLineMatch = '^#!.*(script/runner)';

Here’s a screenshot of what it should look like when you are done:

Now go back to your script and enjoy all the colorized, scope-aware editing goodness that is TextMate!

This is an update of a previous post.

I just watched Jim Weirich's talk on Rake at the Rails Edge Studio and decided to update the command completion script to use Rake itself for caching the tasks. Err the Blog did an update on my script last month to include caching for a nice speedup, so I took that and made sure the cache is regenerated when the Rakefile or any rake tasks found in your rails app have been updated. I added this with just some straight ruby code, but Jim's talk made me realize that this type of thing is exactly is what rake is for. So, here is the updated version:

  #!/usr/bin/env ruby

  # Complete rake tasks script for bash
  # Save it somewhere and then add
  # complete -C path/to/script -o default rake
  # to your ~/.bashrc
  # Nicholas Seckar <nseckar@gmail.com>
  # Saimon Moore <saimon@webtypes.com>
  # http://www.webtypes.com/2006/03/31/rake-completion-script-that-handles-namespaces
  # http://errtheblog.com/post/33

  exit 0 unless File.file?(File.join(Dir.pwd, 'Rakefile'))

  require 'rubygems'
  require 'rake'
  DOTCACHE = File.join(File.expand_path('~'), ".rake_task_cache" , Dir.pwd.hash.to_s)
  RAKE_FILES = FileList[ __FILE__, 'Rakefile', 'lib/tasks/**/*.rake', 'vendor/plugins/*/tasks/**/*.rake']
  file DOTCACHE => RAKE_FILES do
    tasks = `rake --silent --tasks`.split("\n").map { |line| line.split[1] }
    require 'fileutils'
    dirname = File.dirname(DOTCACHE)
    FileUtils.mkdir_p(dirname) unless File.exists?(dirname)
    File.open(DOTCACHE, 'w') { |f| f.puts tasks }
  end

  Rake::Task[DOTCACHE].invoke
  tasks = File.read(DOTCACHE)

  exit 0 unless /\brake\b/ =~ ENV["COMP_LINE"]

  after_match = $'
  task_match = (after_match.empty? || after_match =~ /\s$/) ? nil : after_match.split.last
  tasks = tasks.select { |t| /^#{Regexp.escape task_match}/ =~ t } if task_match

  # handle namespaces
  if task_match =~ /^([-\w:]+:)/
    upto_last_colon = $1
    after_match = $'
    tasks = tasks.map { |t| (t =~ /^#{Regexp.escape upto_last_colon}([-\w:]+)$/) ? "#{$1}" : t }
  end

  puts tasks
  exit 0

Update: I fixed the bug that janfri pointed out. The bug caused the first task to be missed. I also changed it so it won't abort if rake isn't the first command on the command line. This will allow stringing multiple commands together. For instance:

rake db:migrate VERSION=0 && rake db:migrate

Enjoy

I got some basic rake command line completion working today using Jon Baer’s comment. Very simple, very easy:

complete -W '`rake—silent—tasks | cut -d ” ” -f 2`' -o default rake

However this didn’t work for the namespaced tasks in a rails app like rake test:units. Searching a little further I found a reference to some code Nicholas Seckar wrote on project.ioni.st. This used ruby to find the possible tasks for command completion. This looked promising, but it still didn’t work for namespaced tasks. A little more googling led me to what looked like the perfect link: Rake-completion script that handles namespaces. Alas, it only handled one level of namespacing. It worked nicely for rake test:units, but rake tmp:ses<TAB> would complete to rake tmp:clear instead of rake tmp:sessions:clear. Also, rake test:units <TAB> would complete to rake test:units units instead of giving me all the tasks again, just in case you want to run multiple tasks form the command line.

So, now what? Stand on the shoulders of others, naturally. Here is what I’m using now that handles multiple namespaces and multiple tasks per command line:

#!/usr/bin/env ruby

# Complete rake tasks script for bash
# Save it somewhere and then add
# complete -C path/to/script -o default rake
# to your ~/.bashrc
# Nicholas Seckar <nseckar@gmail.com>
# Saimon Moore <saimon@webtypes.com>
# http://www.webtypes.com/2006/03/31/rake-completion-script-that-handles-namespaces

exit 0 unless File.file?(File.join(Dir.pwd, 'Rakefile'))
exit 0 unless /^rake\b/ =~ ENV["COMP_LINE"]

after_match = $'
task_match = (after_match.empty? || after_match =~ /\s$/) ? nil : after_match.split.last
tasks = `rake --silent --tasks`.split("\n")[1..-1].collect {|line| line.split[1]}
tasks = tasks.select {|t| /^#{Regexp.escape task_match}/ =~ t} if task_match

# handle namespaces
if task_match =~ /^([-\w:]+:)/
  upto_last_colon = $1
  after_match = $'
  tasks = tasks.collect { |t| (t =~ /^#{Regexp.escape upto_last_colon}([-\w:]+)$/) ? "#{$1}" : t }
end

puts tasks
exit 0

Enjoy

Rails has a nice plugin system allowing to add common code to a project. A plugin should really be independent from any other plugins. But we also use plugins to share code among different projects we are working on and our code depends on existing plugins. The Rails development team want’s to keep the plugin system simple and didn’t provide an explicit way to handle these dependencies, which I believe is a good decision. There is a solution. Simply name the plugins in order off the dependencies you have. Let’s assume you want to add “my_very_own_plugin” plugin that depends on the enumation_mixin, then simply organize the /vendor/plugins folder as follows, et Voila!:.

/myrailsapp
    /vendor
        /plugins
            /01_acts_as_taggable
            /01_enumations_mixin 
            /01_acts_as_versioned
            /02_my_very_own_plugin

If we look at the Rails::Initializer we can see why this works. Note, this is only an extract of the full class that Rails provides to bootstrap your rails applicaton. The sort on line 4 here after allows this trick.

1
2
3
4
5
6
7
8

module Rails
  class Initializer
    def load_plugins
      find_plugins(configuration.plugin_paths).sort.each { |path| load_plugin path }
      $LOAD_PATH.uniq!
    end
  end
end