Amazon EC2 HighMemory Instances

Posted by Jonathan

Very very nice: 34.2 GB RAM and 68.4 GB RAM instances on EC2

# free -m
             total       used       free     shared    buffers     cached
Mem:         70007       2205      67801          0         28        595
-/+ buffers/cache:       1581      68425
Swap:            0          0          0

And

# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5550  @ 2.67GHz
stepping	: 5
cpu MHz		: 2666.760
cache size	: 8192 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu tsc msr pae mce cx8 apic mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr dca popcnt lahf_lm
bogomips	: 5336.34
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

[..]

We just included support for those machines in Scalarium, our EC2 cluster management plattform.

Webistrano/Capistrano problem with git

Posted by Jonathan

Recently I helped a friend debug a problem when deploying with Webistrano/Capistrano.

He was using a git repository and used SSH keys for authentication. Every time he tried to deploy he got this error:

 executing locally: "git ls-remote ssh://repo.example.com/git/myproject.git HEAD"
*** Could not save revision: Unable to resolve revision for 'HEAD' on repository 'ssh://repo.example.com/git/myproject.git'.

When running this command manually as the Webistrano user, everything worked fine.

We checked the usual suspects: the SSH key, the permissions on the SSH dirs/files, user, firewall & co. Everything seemed correct and worked when we ran the command by hand.

After a bit of tinkering I had the Eureka moment: the git command was not in $PATH when running under Passenger!

Git was installed and worked when we logged in as the Webistrano user. But when Passenger runs Webistrano it doesn't load all your shell config files. So if git is not in a standard location like /usr/bin or /bin Capistrano (which by this time will be called from Webistrano to do the actual deployment) will not find it.

I our case git was installed in /usr/local/bin and thus not in the default path. We ended up symlinking it to /usr/bin and everything worked like a charm.

I just committed a fix to Capistrano to make debugging such errors in the future easier. Capistrano will now check every local command it executes and see if it is in path. So with the latest version on github the error message would have looked like this:

 executing locally: "git ls-remote ssh://repo.example.com/git/myproject.git HEAD"
*** executable 'git' not present or not in $PATH on the local system!
*** Could not save revision: Unable to resolve revision for 'HEAD' on repository 'ssh://repo.example.com/git/myproject.git'.

So if you are running any shell commands under Passenger remember that it doesn't use a full login-shell.

Scotland on Rails 2009 Slides

Posted by Jonathan

I know, it is a bit late, but here are my slides from Scotland on Rails:


The slides are also available as a PDF download: Advanced Deployment

Scotland on Rails was again a great conference. A very interesting crowd in a very nice city. I'm looking forward to next year!

Webistrano 1.4 released

Posted by Jonathan

I just released Webistrano 1.4. Webistrano is a tool for managing Capistrano deployments and offers a rich web UI. It lets you manage projects with their stages and keep track who deployed which version to which servers.

Webistrano 1.4 brings many new features that make deployment easier. The most prominent are:

  • Recipe versioning - recipes are now versioned so that you can keep track of changes
  • Project cloning - you can now create a template project and clone it over and over again
  • Array parameters - support for arrays as values for configuration parameters
  • CAS-auth support - Single Sign-On support by delegating authentication to a CAS server. See the documentation
  • Enhaced UI - nicer overviews of deployments and many small fixes
  • Cancel deployments - a running deployment can now be canceled by Webistrano. The running Capistrano instance will be killed so use this feature with care
  • Track deployed revisions - Webistrano will track which revision was deployed. This way you always know which version is running where
  • Updated packages - Rails 2.1 and Capistrano 2.5.0

Apart from that some smaller enhancements and fixes went into the 1.4 release. See the CHANGELOG for a complete list.

Further, there is now a Webistrano mailing list at GoogleGroups.

Go get Webistrano from the project homepage as a download or checkout the source:

Download: webistrano-1.4.zip (3.4 MB)

# Development version:

svn co http://labs.peritor.com/svn/webistrano/trunk

# Stable version:

svn co http://labs.peritor.com/svn/webistrano/branches/1.4

RailsConf Europe 2008

Posted by Jonathan

Day two of RailsConf Europe 2008 is over and so are my two sessions.

On tutorial day Mathias and I did a 4h workshop on deploying and monitoring Rails applications. The tutorial went really well, apart from the AirportExpress base station not coping with 100 laptops connecting to it. In the practical part we had a FreeBSD server with 40 virtual machines running and helped the audience deploy an example application with git or svn and Mongrel or mod_rails.

On day two I held my Security on Rails session where I go over the various attacks and countermeasures against Rails applications. This session was also well received and I hope I could educate people a bit about WebAppSecurity.

The slides are available as PDF here: Security on Rails (PDF) Deploying and Monitoring Rails (PDF)

Further, you can find both presentations at slideshare.

Security On Rails
View SlideShare presentation or Upload your own. (tags: ruby rubyonrails)


The slides are available as PDF here: Security on Rails (PDF) Deploying and Monitoring Rails (PDF)

If you attended one of my sessions, I encourage you to rate them at the conference site.

So far my expectations have been met and I've could catch up with a lot of people. I'm looking forward to day three of RailsConf Europe!

Webistrano 1.3 released

Posted by Jonathan

I'm proud to announce Webistrano 1.3!

Webistrano is a Web UI for managing Capistrano deployments. It lets you manage projects and their stages like test, production, and staging with different settings. Those stages can then be deployed with Capistrano through Webistrano.

The 1.3 adds several new shiny features to Webistrano that make deployment easier:

  • Better Git support through Capistrano 2.2
  • Support for Phusion Passenger / mod_rails
  • Ability to temporary disable hosts for a deployment
  • A command-line interface with script/deploy
  • A simple permission system

The complete changelog is available through the Webistrano project site.

One very often demanded feature is the ability to temporary disable a host for a deployment. This is helpful when you want to deploy a stage without changing the stage configuration even if one or more hosts are down:

Another scenario is when you want to execute a task only on a limited set of servers.

The script/deploy command is a nice little gem, especially useful if you want to script Webistrano:

$ ruby script/deploy 
Usage: deploy [options] project stage
    -h, --help                       This message
    -e, --environment=ENV            RAILS_ENV for Webistrano (default: production)
    -u, --username=NAME              Webistrano username to use (default: admin)
    -t, --task=NAME                  Capistrano task to invoke (default: deploy)
    -d, --description=TEXT           Deployment comment for Webistrano records

Further, Webistrano now offers built-in tasks for managing mod_rails deployments. It will override the default deploy tasks and ask for the necessary configuration entries so that using mod_rails becomes even easier.

Upgrading from previous releases is very easy, see the Upgrading wiki page.

Webistrano 1.3 can be downloaded here. Webistrano is BSD-licensed and the project site is open for everybody. Please see the project page for more documentation and screenshots. There are even some screencasts.

Web 2.0 Expo Berlin

Posted by Jonathan

I'm just back from today's Web 2.0 Expo sessions and I'm not sure I will attend tomorrow. Many have written about this before, but the creative, social atmosphere is missing due to the conference labyrinth halls. Boy, I'm happy I haven't spend > 1.000 Euros on this. No real food, a lot of product presentations, not enough room for socializing and to many suits for my taste.

Still, I had some nice conversations and met some interesting people.

I did again a session on scaling with Amazon EC2 and S3, the slides can be found here.

This time a also talked a bit about how we use S3 and EC2 to drive our Webmail Portal product, PeritorMail at Peritor.

SlideShare | View

Also nice the AWS announcement of S3 being available in EU data centers. Now I'm only waiting for EC2 in the EU...

Webistrano 1.2 released

Posted by Jonathan

Webistrano 1.2 is out. You can get it at the Peritor Labs project page.

Version 1.2 brings the following changes and enhancements:

  • Fixes a bug in the execution of tasks with sudo
  • Highlight recipe syntax with the syntax gem
  • Include Capistrano 2.1.0 and git support
  • Add experimental support for ssh_keys and ssh_ports as normal configuration parameter (currently only one SSH key is supported)

Although Capistrano 2.1.0 is included, Webistrano will by default allocate a pty for each SSH command as the new Capistrano default of now doing so seems problematic on some hosts.

Further there where some minor fixes and UI enhancements. All available configuration parameter are now documented in the wiki.

Upgrading is done with a `RAILS_ENV=production rake db:migrate`, more information on how to upgrade can be found here.

With version 1.2 out of the door I will focus on bringing authorization to Webistrano so that deployment permissions can be tight to individual users and groups.

Get version 1.2 either from the download page or directly here: webistrano-1.2.zip

EC2 gets new instance types

Posted by Jonathan

Wow

Amazon EC2 gets two new types of instances, large and extra large EC2 instances. Basically a large instance that has 4 times the capacity (CPU, RAM, HDD) of the old, now default small instance type while the extra large instance type has 8 times the capacity.

Small Instance (default)

1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage (150 GB plus 10 GB root partition)
32-bit platform
I/O Performance: Moderate 
Price: $0.10 per instance hour

Large Instance

7.5 GB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage (2 x 420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High 
Price: $0.40 per instance hour

Extra Large Instance

15 GB memory
8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
1,690 GB instance storage (4 x 420 GB plus 10 GB root partition)
64-bit platform
I/O Performance: High 
Price: $0.80 per instance hour

The idea is that you specify the instance type in the RunInstances API call. All old tools that do not specify this parameter start a default instance type.

Very nice to see this so fast after the recent S3 SLAs.

If they would now allow to run EC2 instances in Europe there are no excuses left not to run nearly all applications on EC2.

Capistrano and Webistrano configuration parameter overview

Posted by Jonathan

Capistrano (and thereby Webistrano) is very flexible and there are a lot of variables that can change its default behaviour.

Unfortunately there is no one public list of all parameter to tweak. As the question about all of those comes up every now and then I decided to create a probably incomplete list of all configuration parameter for Webistrano and Capistrano.

The list can be found on the Webistrano project page.

If you have any additions or corrections, please tell me or use the project page directly to submit corrections.

A preview of Webistrano 1.1

Posted by Jonathan

It's been two weeks since the public release of Webitrano . Webistrano was downloded nearly 500 times and I got a lot of feedback. People seem to like the concept and having a tool to manage Capistrano deployments through a Web UI.

A lot of people wanted to tie recipes not to projects but to stages, so Webistrano 1.1 will let you do that. Further 1.1 will interpolate strings in Capistrano variables correctly so that you can refer to other variables inside configuration entries.

Also more recent versions of needle/net-ssh will be included so that Webistrano will not blow up if you have newer versions installed.

Apart from this new functionality/bug-fixing the UI got some major enhancements and now looks very promising:



I hope to get 1.1 out of the door shortly after RailsConfEurope here in Berlin. If you are there and have any feedback, I would love to hear it.

Webistrano - A Web UI for managing Capistrano deployments

Posted by Jonathan

I'm happy to announce the release of Webistrano - a Web UI for managing Capistrano deployments.

Webistrano is an internal application that I developed at Peritor for the easy handling of Capistrano deployments. I often had situations where our designer updated some images and wanted to update/deploy our Rails project. Capistrano offers a nice command line interface for doing this but this is not the right tool for a designer, especially if you have a complicated multi-stage environment with production, staging and test settings.

Further we wanted to be able to keep track of who deployed what when to which servers.

This is where Webistrano enters the stage.

Webistrano

Webistrano is a Ruby on Rails application that manages projects with their different stages (like production or testing) and leverages Capistrano to handle the deployment part. This way it is very easy to handle multi-stage and multi-client situations and keep an eye on all deployments. It further includes a simple email alerting system so that you get pinged if somebody deploys to the production servers.

We are using Webistrano for quite some time now and it has proven stable for our needs. I am releasing it under the BSD license. The Subversion repository is located at Rubyforge.

On the project page you can find two screencasts that show you how to get Webistrano running and explain some advanced concepts.

Version 1.0 can be downloaded here:
Make sure to watch the screencasts:  

I presented Webistrano last month at the Berlin Ruby User Group (then under the name 'Webcap') and the feedback was quite positive. I'd love to hear your feedback and suggestions.

UPDATE:
There was a problem with the Mephisto comment system that resulted in all posted comment being posted to /dev/null. The problem is fixed now, please re-submit your comments.

Debugging fun

Posted by Jonathan

I use Mongrel and mongrel_cluster for all of my Rails apps so it was the obvious choice for our Rails based Knowledge Management system at Peritor.

Our app ran fine in WEBrick on the local development boxes but we had strange errors on the production Mongrels:

NoMethodError: undefined method `find’ for Cluster:Module

Locally it was also running fine with Mongrel. It took some time to find out that the mongrel_cluster gem_plugin that we use on the production boxes installs a global module named Cluster and that this module overshadowed our Rails model named Cluster. I already wrote the author about it and hopefully mongrel_cluster will use a separate namespace in the next release. Until then we manage the Mongrels ourselves. Maybe this will spare somebody a long debugging session…

Mongrel and Rails behind Apache 2.2 and SSL

Posted by Jonathan

For a new project of mine we needed to operate Rails with HTTPS. Our setup is the same as I described in an earlier article about Mongrel and Apache 2.2 mod_proxy_balancer, so we have Apache 2.2 in front of a cluster of Mongrels.

After the initial plain HTTP setup was working fine we went on to configure HTTPS. The obvious way is to configure an Apache SSL virtual host, that proxies all requests to the Mongrel cluster (for more on how to setup the Mongrel cluster look here).

<VirtualHost _default_:443>
ServerName www.example.com:443
ServerAdmin webmaster@example.com
TransferLog /var/log/www/www.example.com/apache_ssl_transfer_log
ErrorLog /var/log/www/www.example.com/apache_ssl_error_log
CustomLog /var/log/www/www.example.com/apache_ssl_access_log combined

ProxyPass / balancer://mongrelcluster/
ProxyPassReverse / balancer://mongrelcluster/

SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile /etc/ssl/example.crt
SSLCertificateKeyFile /etc/ssl/private.key

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>
BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0
CustomLog /var/log/httpd-ssl_request.log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" 
</VirtualHost>

This setup works fine until you initiate an internal redirect in your rails code like this:

redirect_to :action => 'list'

As Rails does not know that is behind an HTTPS proxy it creates a redirection to a HTTP resource. This breaks your security and e.g. results in IE complaining about unsafe file transmission on POSTs. James Duncan Davidson has a nice solution for this annoyance.

The solution is to tell Rails that it is operated in HTTPS mode without breaking the development environment. This can be done by setting an environment variable with Apache in the request and checking for this variable in a before filter. If this variable is set, redirect to HTTPS resources. Otherwise use plain old HTTP.

In order to set an environment variable in Apache, include the following line in the SSL virtual host definition:

RequestHeader set X_ORIGINAL_PROTOCOL 'https'

Now create a before_filter in the ApplicationController that checks for this variable:

before_filter :set_ssl
...

def set_ssl
  if request.env.has_key? 'HTTP_X_ORIGINAL_PROTOCOL'
    if request.env['HTTP_X_ORIGINAL_PROTOCOL'] == "https" 
      request.env["HTTPS"] = "on" 
     end
  end
end

request.env[“HTTPS”] = “on” tells Rails to consider the request as an HTTPS request and therefore generate redirects that obey this.

One thing to watch out for is that the variable gets a “HTTP_” prefix set by Apache. So we set the variable “X_ORIGINAL_PROTOCOL” but check for “HTTP_X_ORIGINAL_PROTOCOL”.

Knowing this can save you some hours of debugging…


UPDATE:
After poking around in the ActionController sources there seems to be a much better and easier way. Just set this variable (in httpd.conf) and delete the before_filter:

RequestHeader set X_FORWARDED_PROTO 'https'
Rails will figure out the rest itself. The magic comes from these lines in request.rb:
def ssl?
      @env['HTTPS'] == 'on' || @env['HTTP_X_FORWARDED_PROTO'] == 'https'
end