BlogFish

I use Mongrel and mongrel_cluster for all of my Rails apps so it was the obvious choice for our Rails based Knowledge Management system at Peritor.

Our app ran fine in WEBrick on the local development boxes but we had strange errors on the production Mongrels:

NoMethodError: undefined method `find’ for Cluster:Module

Locally it was also running fine with Mongrel. It took some time to find out that the mongrel_cluster gem_plugin that we use on the production boxes installs a global module named Cluster and that this module overshadowed our Rails model named Cluster. I already wrote the author about it and hopefully mongrel_cluster will use a separate namespace in the next release. Until then we manage the Mongrels ourselves. Maybe this will spare somebody a long debugging session…

For a new project of mine we needed to operate Rails with HTTPS. Our setup is the same as I described in an earlier article about Mongrel and Apache 2.2 mod_proxy_balancer, so we have Apache 2.2 in front of a cluster of Mongrels.

After the initial plain HTTP setup was working fine we went on to configure HTTPS. The obvious way is to configure an Apache SSL virtual host, that proxies all requests to the Mongrel cluster (for more on how to setup the Mongrel cluster look here).

<VirtualHost _default_:443>
ServerName www.example.com:443
ServerAdmin webmaster@example.com
TransferLog /var/log/www/www.example.com/apache_ssl_transfer_log
ErrorLog /var/log/www/www.example.com/apache_ssl_error_log
CustomLog /var/log/www/www.example.com/apache_ssl_access_log combined

ProxyPass / balancer://mongrelcluster/
ProxyPassReverse / balancer://mongrelcluster/

SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/ssl/example.crt
SSLCertificateKeyFile /etc/ssl/private.key

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>
BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog /var/log/httpd-ssl_request.log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" 
</VirtualHost>

This setup works fine until you initiate an internal redirect in your rails code like this:

redirect_to :action => 'list'

As Rails does not know that is behind an HTTPS proxy it creates a redirection to a HTTP resource. This breaks your security and e.g. results in IE complaining about unsafe file transmission on POSTs. James Duncan Davidson has a nice solution for this annoyance.

The solution is to tell Rails that it is operated in HTTPS mode without breaking the development environment. This can be done by setting an environment variable with Apache in the request and checking for this variable in a before filter. If this variable is set, redirect to HTTPS resources. Otherwise use plain old HTTP.

In order to set an environment variable in Apache, include the following line in the SSL virtual host definition:

RequestHeader set X_ORIGINAL_PROTOCOL 'https'

Now create a before_filter in the ApplicationController that checks for this variable:

before_filter :set_ssl
...

def set_ssl
  if request.env.has_key? 'HTTP_X_ORIGINAL_PROTOCOL'
    if request.env['HTTP_X_ORIGINAL_PROTOCOL'] == "https" 
      request.env["HTTPS"] = "on" 
     end
  end
end

request.env[“HTTPS”] = “on” tells Rails to consider the request as an HTTPS request and therefore generate redirects that obey this.

One thing to watch out for is that the variable gets a “HTTP_” prefix set by Apache. So we set the variable “X_ORIGINAL_PROTOCOL” but check for “HTTP_X_ORIGINAL_PROTOCOL”.

Knowing this can save you some hours of debugging…

UPDATE:
After poking around in the ActionController sources there seems to be a much better and easier way. Just set this variable (in httpd.conf) and delete the before_filter:

RequestHeader set X_FORWARDED_PROTO 'https'

def ssl?
      @env['HTTPS'] == 'on' || @env['HTTP_X_FORWARDED_PROTO'] == 'https'
end

Unitl this week we used Lighttpd and FastCGI for MeinProf.de. The setup was nearly the same as described in the must read series scaling rails (1, 2, 3, 4) from poocs.net.

We used this setup from day 1 but always had some small issues with Lighttpd. Lighttpd was crashing every couple of days. Nothing dramatic, we had a script that monitored Lighttpd and restarted it if necessary. During the last weeks Lighttpd started to crash once a day and lately even once an hour. This was unacceptable and as we knew that we were going to get some serious press coverage in Germany we looked for alternatives.

43people and Basecamp use Apache 1.3 and FastCGI so this seemed like a good alternative. Just switch the webserver and we would be done. Unfortunately Apache 1.3 cannot loadbalance the FastCGI request and there is very little documentation on Apache 1.3 and remote FastCGI processes. Apache 2.0 is no better and has problems with mod_fastcgi. We needed remote FastCGI listeners as our hardware is quite old and we have many slow machines as opposed to a few fast ones that could use local FastCGI to handle the load.

Enter Mongrel.

Mongrel is a fast HTTP library and server for Ruby that is intended for hosting Ruby web applications of any kind using plain HTTP rather than FastCGI or SCGI. It is framework agnostic and already supports Ruby On Rails, Og+Nitro, and Camping frameworks.

With Mongrel your application server becomes a webserver that speaks HTTP so you “only” need to loadbalance and proxy normal HTTP request to it. Mongrel was stable during our tests so we looked for the HTTP proxy solution. Apache had always mod_proxy and could therefore proxy HTTP requests but we needed to loadbalancer these. The are extra packages for this kind of stuff like Balance but we wanted something more integrated and didn’t want to introduce more components.

Enter Apache 2.2 and mod_proxy_balancer.

Apache 2.2 introduced a new proxy module, mod_proxy_balancer. This module does exactly this, it balances proxy requests. You can define a cluster of proxies and use this cluster in your mod_proxy statement instead of just one proxy server.

With this setup we use Apache 2.2 to handle all incoming requests. Apache 2.2 uses mod_proxy to redirect the incoming HTTP requests to the mod_proxy_balancer cluster. The cluster consists of several Mongrel processes on each application server (and now also internal web server) and distributes the requests.

mod_proxy_balancer is more configurable that Lighttpd’s mod_fastcgi. For example you can specify load factors or routes for each cluster member. See the documentation for details.

Our httpd.conf looks like this:

First you define the cluster and tell it of which members it is composed of.

<Proxy balancer://myclustername>
  # cluster member 1
  BalancerMember http://192.168.0.1:3000 
  BalancerMember http://192.168.0.1:3001

  # cluster member 2, the fastest machine so double the load
  BalancerMember http://192.168.0.11:3000 loadfactor=2
  BalancerMember http://192.168.0.11:3001 loadfactor=2

  # cluster member 3
  BalancerMember http://192.168.0.12:3000
  BalancerMember http://192.168.0.12:3001

  # cluster member 4
  BalancerMember http://192.168.0.13:3000
  BalancerMember http://192.168.0.13:3001
</Proxy>

Then you proxy the location or virtual host to the cluster:

<VirtualHost *:80>
  ServerAdmin info@meinprof.de
  ServerName www.meinprof.de
  ServerAlias meinprof.de
  ProxyPass / balancer://meinprofcluster/
  ProxyPassReverse / balancer://meinprofcluster/
  ErrorLog /var/log/www/www.meinprof.de/apache_error_log
  CustomLog /var/log/www/www.meinprof.de/apache_access_log combined
</VirtualHost>

The slash at the end of the ProxyPass directive is very important.

Mongrel itself is startet on the cluster nodes like this:

# mongrel_rails start -d -e production -p 3000
# mongrel_rails start -d -e production -p 3001

Another nice feature of mod_proxy_balancer is the balancer-manager. It is a web interface to the configuration of the mod_proxy_balancer cluster through which you can query or edit your cluster nodes without the need to restart Apache.

In order to use balancer-manager include this in your configuration:

<Location /balancer-manager>
  SetHandler balancer-manager
</Location>

Of course you should protect this location through Apache’s require valid-user or Allow from directives.

So far this solution has proven much more stable (at least on FreeBSD) and was able to handle our peak traffic of 350.000 page request per day. In practice we use up to 8 Mongrel processes on each cluster node and it seems that Apache is the bottleneck and not our application servers as before. The next step for us is to introduce another web server that handles the incoming HTTP requests and has it’s own Mongrel cluster.