It took me a while to figure this out, so here are my notes:

1. Install Scratchbox as described here. This greatly simplifies cross-compiling things. It also provides you a way to run the compiled ruby interpreter on a bigger host machine using QEMU. This is essential because I rely on rubygems, and the Chumby is too small to execute the gem install commands. Besides, at least for the sqlite3-ruby and mongrel gems, I need to compile native code.
2. Install dependencies. I found the following libraries necessary:
   • OpenSSL 0.9.8n – avoid the 1.0.0 version, it causes problems with the Ruby interpreter.
   • Zlib 1.5.3 – same as installed on the Chumby.
   • Readline 6.1
   • A recent version of sqlite, since I need it for my gems later on.

   Compile the libraries and install them into /mnt/storage (use –prefix where necessary).

3. Adjust the environment to use the new libraries:
   BASEDIR=/mnt/storage/usr
export PATH=$PATH:$BASEDIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$BASEDIR/lib
4. Install Ruby and rubygems. I was not able to compile Ruby 1.8.7, but Ruby 1.8.6-p399 works nice. To configure everything, I use the following commands:
   export ac_cv_func_getpgrp_void=yes
export ac_cv_func_setpgrp_void=yes
./configure --target=arm-linux --host=arm-linux --disable-pthread --enable-wide-getaddrinfo --enable-zlib --with-zlib-dir=/mnt/storage/usr --enable-readline --with-readline-dir=/mnt/storage/usr --prefix=/mnt/storage/usr --with-openssl-dir=/mnt/storage/usr && make && make install

5. Install rubygems:
   
/mnt/storage/usr/bin/ruby setup.rb --prefix=/mnt/storage/usr
mkdir -p /mnt/storage/usr/lib/ruby/gems/1.8

6. Afterwards, the necessary gems can be installed using the gem command. It might be necessary to provide additonal path information to the gems that compile native extensions, for example
   gem install sqlite3-ruby -- --with-sqlite3-dir=/mnt/storage/usr

The result is an image with Ruby installed, and all necessary gems for my project. The tarball can be downloaded here. To install, scp it to the Chumby and log in:

scp *tarball* root@*chumby-ip*:/mnt/storage
ssh -l root *chumby-ip*
cd /mnt/storage
tar xvzf *tarball*


Now you have a directory /mnt/storage/usr with all tools permanently installed on your Chumby. In order to use it, please adjust your PATH and LD_LIBRARY_PATH appropriately:

export BASEDIR=/mnt/storage/usr
export PATH=$PATH:$BASEDIR/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$BASEDIR/lib


/me mumbles more automation needed. But not today.