BlogFish

Yesterday I gave a presentation about CouchDB and SimplyStored, our convenience Ruby library, at the Ruby User Group Berlin.

There is a recording of the presentation at ustream.tv.

Mathias and I wrote SimplyStored in order to easily interact with Ruby objects serialized in CouchDB. We use CouchDB as the main data store for Scalarium and so far it has been great. But it is a bit cumbersome to write all those map and reduce functions yourself.

SimplyStored generates the JavaScript map&reduce functions for handling associations or dynamic finders for you.

SimplyStored offers:

• Models
• Associations
• Callbacks
• Validations
• Dynamic finder
• S3 attachments
• Paranoid delete

    class User
      include SimplyStored::Couch

      property :login
      property :age
      property :accepted_terms_of_service, :type => :boolean
      property :last_login, :type => Time
    end

    user = User.new(:login => 'Bert', 
                    :age => 12, 
                    :accepted_terms_of_service => true, 
                    :last_login = Time.now)
    user.save

    User.find_by_age(12).login
    # => 'Bert'

    User.all
    # => [user]

    class Post
      include SimplyStored::Couch

      property :title
      property :body

      belongs_to :user
    end

    class User
      has_many :posts
    end

    post = Post.create(:title => 'My first post', 
                       :body => 'SimplyStored is so nice!', 
                       :user => user)

    user.posts
    # => [post]

    Post.find_all_by_title_and_user_id('My first post', user.id).first.body
    # => 'SimplyStored is so nice!'

    post.destroy

    user.posts(:force_reload => true)
    # => []
  

The code is on github and OpenSource: SimplyStored example code

Another thing I talked about is RockingChair. RockingChair is an in-memory CouchDB implementation that understands all of SimplyStored's functionality. We use it to speed up our tests and be able to run them in parallel.

Although Berlin looks a bit like Antarctica right now, I'm really looking forward to travel to Helsinki, Finland for Frozen Rails on May 7th.

Forzen Rails is organized by the guys from the HHLinuxCliub and Kisko Labs. I've met them in Amsterdam back in October and we had a great time. I'm sure Frozen Rails will be too. They have a great line-up organized with speakers like Yehuda Katz, Chris Wanstrath, Joseph Wilk, Jarkko Laine, Mike Dirolf, or Nizar Jouini. The ticket prices are reasonable, starting with 99 Euro for the early bird tickets. They even have a discount scheme that will give you a free ticket if you get 5 friends to sign-up. Regional conferences like this are always a lot of fun as you meet a lot of local developers and the organizers are very passionate. I hope to meet you at Frozen Rails 2010 in Helsinki.

The last past days in Amsterdam for Ruby En Rails 2009 were really great.

I arrived on Thursday and had the chance to discuss a possible security vulnerability in Rails I discovered a while back with Yehuda and Mislav during lunch.

Afterwards we went to the conference dinner and met many very nice people and had a long discussion about voting systems, European vs. American culture, gun laws, and political systems.

Friday was the first conference day and started for us with being 40 minutes late as we headed up to the wrong metro station. Yehuda was supposed to give a keynote as the first session... luckily the organizers swapped the sessions. So unfortunately we missed the first session but were in time for Yehuda to give his keynote.

Yehuda talked about the Rails/Merb merge and dissected what was achieved. It was a very good, in-depth presentation about the new features of Rails 3 and what left to do. I really liked the new router DSL, actions being Rack-apps, and the death of Rails Metal (the new, slicker possibility to build simpler/smaller controllers takes care of this.).

After the coffee break I gave my Rails Security presentation about common attacks against web applications and how you can protect against them in Rails. Usually this is quite heavy stuff and people tend to sit quiet and listen as this is new to most. In Amsterdam I had some very good questions and discussions on session fixation, JavaScript high-jacking, and app reconnaissance.

Then we listened to Julio Javier Cicchelli talk about Rubyists.EU, an effort to make the different European Rails communities easier to find and build a common European community.

We went to lunch with James which gave us some time to catch up. Took us a while to find a decent place around the conference hall but we managed to find a restaurant where we tried to estimate the global financial burden due to Internet Explorer and what will happen to Microsoft if they were charged for the extra work needed done.

After lunch Eloy educated us about the current state of MacRuby. I'm really looking forward to spending some time hacking Mac apps in Ruby.

The next session gave an overview about monitoring, performance, and the different tools like request-log-analyzer or Nagios.

The conference was closed by Jeremy giving a keynote about Rails, Ruby, and the current state of things in the community (and a sneak-peak at ActiveRelation and what it will be able to do). It was very well received and a good closing session.

Afterwards a big group headed for dinner. We ended up in a small, very local restaurant. It included very nice food, good discussions with new friends from Finnland, and a waiter/owner who runs the restaurant homepage with Rails and asked Yehuda for help :-)

The next day was titled Geek Day and included many lightning talks (that ended up being nearly full sessions and had very good content). Parallel to the great sessions about MongoDB, DataMapper, or experience reports there was a Rails Rumble featuring five teams going on.

The most notable session of the second day was Justin Halsall, dressed for Halloween, talking about BlockHelpers and view DSLs. It was a hilarious show.

The Rumble was a great idea. In contrast to the usual Rumble the teams got a specific challenge. They should build something that improves the situation with Rails dependencies and out-of-date gems laying around in vendor/gems. The winning team would get two tickets to RailsConf 2010.

One team extended builder to list outdated gems and got their changes even merged back to builder by Yehuda on the same day. Another team extended Webistrano to accept projects dependencies and display them on the stage page. Some teams build a command line tool to extract local dependencies like your gems or even the MySQL version and push those definitions to a central place. The winning team had the the most advanced idea regarding update notification and gem-sets for applications. I could really see something like this being integrated into gemcutter. Congratulations Ludo and Michel!

Ruby En Rails day two was celebrated with a big dinner and drinks. After a very nice evening we headed back to the hotel as everybody had an early flight out. Some were still discussing Rails, some were lucky to be able to enjoy Amsterdam longer.

I really enjoyed the conference and Amsterdam. Thank you Chris and Tim Obdam for organizing the whole event!

Recently I helped a friend debug a problem when deploying with Webistrano/Capistrano.

He was using a git repository and used SSH keys for authentication. Every time he tried to deploy he got this error:

 executing locally: "git ls-remote ssh://repo.example.com/git/myproject.git HEAD"
*** Could not save revision: Unable to resolve revision for 'HEAD' on repository 'ssh://repo.example.com/git/myproject.git'.

When running this command manually as the Webistrano user, everything worked fine.

We checked the usual suspects: the SSH key, the permissions on the SSH dirs/files, user, firewall & co. Everything seemed correct and worked when we ran the command by hand.

After a bit of tinkering I had the Eureka moment: the git command was not in $PATH when running under Passenger!

Git was installed and worked when we logged in as the Webistrano user. But when Passenger runs Webistrano it doesn't load all your shell config files. So if git is not in a standard location like /usr/bin or /bin Capistrano (which by this time will be called from Webistrano to do the actual deployment) will not find it.

I our case git was installed in /usr/local/bin and thus not in the default path. We ended up symlinking it to /usr/bin and everything worked like a charm.

I just committed a fix to Capistrano to make debugging such errors in the future easier. Capistrano will now check every local command it executes and see if it is in path. So with the latest version on github the error message would have looked like this:

 executing locally: "git ls-remote ssh://repo.example.com/git/myproject.git HEAD"
*** executable 'git' not present or not in $PATH on the local system!
*** Could not save revision: Unable to resolve revision for 'HEAD' on repository 'ssh://repo.example.com/git/myproject.git'.

So if you are running any shell commands under Passenger remember that it doesn't use a full login-shell.

Recently I've been made aware of people inside US Government organizations using my Ruby on Rails Security presentation as an excuse to limit Ruby on Rails adoption and projects inside those organizations.

They mandate that applications in Rails should be redone in Java because of the issues I covered.

It is not clear to me how anybody who saw/read this presentation would come to the conclusion that Rails is insecure. Every application is vulnerable. Some more and some less. Yes, this means that also Java applications are attackable.

It is my honest and strong belief that Ruby on Rails applications are not less secure than any other web application. In contrast, the Ruby on Rails framework provides several advanced security mechanisms that make it very easy to write secure applications. Further, Ruby on Rails enables very sane security options by default. Some of those defaults include auto-escaping, prevent Cross-Site Request Forgery, or protected from SQL-injection. I would even go as far as to state that the typical Rails application is more secure than the typical web application for those reasons.

The security of an application stands and falls with the knowledge and abilities of the people implementing and running it. Rails makes it very easy to write secure applications.

The conclusion I came to in my presentation still holds:

Ruby is by no means a "web app security silver bullet" but adding security is easy and not a pain like in many other frameworks

I hope this will change the opinion of some people and remove my presentation as their argument.

While debugging an unrelated problem in Nanite I've stumbled over this feature in Ruby:

You can rescue the call to Kernel.exit.

Calling exit will just raise a SystemExit exception that can catched as any other.

#!/usr/bin/env ruby

begin
  exit
rescue SystemExit => e
  puts "somebody wanted to #{e}"
end

puts "after exiting"

This is clearly documented in the RDoc but still I was surprised.

We are in the process of testing the new version of MeinProf.de and crossed a very interesting bug.

We are using fragment caching to speed up the serving of the university homepages. So far this is working great and helps a lot with coping with extra load.

The university homepages are served by unis/show.html.erb:

  ...
  <%= render :partial => 'shared/user_info' %>
  ...
  <% cache "uni_page_homepage_#{@uni.id}", :expires_in => 1.hour do %>
    <%= render :partial => 'unis/header' %>
    <%= render :partial => 'tabs_header' %>
    <div class="uni_desc">
      <div class="tab-content">
       ...
       <!-- a lot of uni content here-->
  <% end %>

The partial unis/_header.html.erb looks like this:

  ...
<% content_for("breadcrumb") do %>
  <%= uni_breadcrumb(uni, department) %>
  <div class="breakcrumb">
    ...
  </div>
<% end %>

<div class="subcolumns">
  <h1><%=h uni.name %></h1>
  <div class="c08l badge">
  <p>
    <strong>Homepage:</strong>....
   ....
  </p>

So far so good. When we deployed to our staging host we noticed that the university header was not always right. On the first request it was correct and on every subsequent request the header was missing the breadcrumb. Errors like this sounds like caching issues and it took us a while to figure it out.

The problem is the content_for block inside the partial. Once you think about it, it makes a lot of sense. During the first request the show-template gets evaluated. It calls the partial and the partial executes the content_for block. This block access the template binding/variables to inject content. The partial finishes rendering and stores the rendered content in memcached. Everything fine.

On the next request Rails loads up the action and renders the view. During the view rendering the fragment will be loaded from memcached. No bug here.

So why is the breadcrumb not showing up? Because the content_for block will not be executed. It is a side effect of the partial-rendering and the partial will not be rendered once it is cached. Only the resulting HTML-fragment will be loaded from the cache but the code-block is not executed. This means that there is not call to inject extra content to the main view.

So before you wrap some view code in fragment caching remember: do not fragment cache content_for-blocks. They will not be called. Always check cached views for content_for-blocks and move them outside the cache-call.

I know, it is a bit late, but here are my slides from Scotland on Rails:

The slides are also available as a PDF download: Advanced Deployment

Scotland on Rails was again a great conference. A very interesting crowd in a very nice city. I'm looking forward to next year!

As a consultant for Peritor I often review other peoples code. What I often encounter and always address is custom logic in Rake tasks.

Rake is perfect for automating tasks and making calls into your application from the command line. People often use it for doing maintenance, executing periodic jobs, or starting and stopping services. Nothing wrong with that, this is what Rake in Rails apps is for.

What I resent is having all this logic in the Rakefile, either inline in the task body or as a method in the Rakefile.

  # Rakefile
  
  task :ping_stale_users do
    stale_users = User.find(:all, 
                            :conditions => ["last_login < ? ", Time.now - 3.months])
    ...
    ...
    stale_users.each do |user|
      if user.check_something
        user.mail_invite
      end
    end
  end

This is just wrong for the following reasons:

• Testing is hard: methods and Rake tasks are hard to test. The environment is hard to create and you may have to shell-out to call rake.
• Reusing is hard: you cannot easily reuse this functionality from the console or an admin controller.
• POLA: Rakefile are not were people look for domain logic

The solution is simple and yet it seems underused. Just put your domain logic into a model or library and call it from the Rake task. This way it is much easier to test, reuse, and read.

  # Rakefile
  
  task :ping_stale_users do
    User.ping_stale_users
  end
  
  # User.rb
  def self.ping_stale_users
    stale_users = User.find(:all, 
                            :conditions => ["last_login < ? ", Time.now - 3.months])
    ...
    ...
    stale_users.each do |user|
      if user.check_something
        user.mail_invite
      end
    end
  end

Of course the method in the model can be cleaned up a bit more but this is not the point. The point is that we removed domain logic from the Rakefile and made it shorter and easier to understand. The logic is now somewhere where we can test and reuse it.

I just released Webistrano 1.4. Webistrano is a tool for managing Capistrano deployments and offers a rich web UI. It lets you manage projects with their stages and keep track who deployed which version to which servers.

Webistrano 1.4 brings many new features that make deployment easier. The most prominent are:

• Recipe versioning - recipes are now versioned so that you can keep track of changes
• Project cloning - you can now create a template project and clone it over and over again
• Array parameters - support for arrays as values for configuration parameters
• CAS-auth support - Single Sign-On support by delegating authentication to a CAS server. See the documentation
• Enhaced UI - nicer overviews of deployments and many small fixes
• Cancel deployments - a running deployment can now be canceled by Webistrano. The running Capistrano instance will be killed so use this feature with care
• Track deployed revisions - Webistrano will track which revision was deployed. This way you always know which version is running where
• Updated packages - Rails 2.1 and Capistrano 2.5.0

Apart from that some smaller enhancements and fixes went into the 1.4 release. See the CHANGELOG for a complete list.

Further, there is now a Webistrano mailing list at GoogleGroups.

Go get Webistrano from the project homepage as a download or checkout the source:

Download: webistrano-1.4.zip (3.4 MB)

# Development version:

svn co http://labs.peritor.com/svn/webistrano/trunk

# Stable version:

svn co http://labs.peritor.com/svn/webistrano/branches/1.4

RailsRumble 2008 is over - after 48 hours of hacking I'm proud to announce our contestant:

GoodBad.me - Twitter Till Judgement Day!

GoodBad.me is a twitter mashup that Thomas Metschke and I build over the last two days. Once you follow our bot on twitter, we subscribe to your tweets. We collect all your good and bad deeds by searching for #goodme or #badme (or the shorter #gd #bd) in your tweets.

We keep track of your deeds and generate stats about you. Your deed balance decides your character level. Are you just a boy scout or do you strive for beeing friends with Goldfinger?

All those questions can be answered at http://goodbad.me.

The idea is that you keep tweeting as always and by adding the #goodme tag you let the world know that you did something good.

In later iterations we want to allow others to tag your deeds/tweets and maybe allow you to gain points by donating to charity.

The site is online for a couple of hours and we already have several active users. If you want to keep track of your day's work, go join!

If you like the idea and our implementation, please vote for us!

Day two of RailsConf Europe 2008 is over and so are my two sessions.

On tutorial day Mathias and I did a 4h workshop on deploying and monitoring Rails applications. The tutorial went really well, apart from the AirportExpress base station not coping with 100 laptops connecting to it. In the practical part we had a FreeBSD server with 40 virtual machines running and helped the audience deploy an example application with git or svn and Mongrel or mod_rails.

On day two I held my Security on Rails session where I go over the various attacks and countermeasures against Rails applications. This session was also well received and I hope I could educate people a bit about WebAppSecurity.

The slides are available as PDF here: Security on Rails (PDF) Deploying and Monitoring Rails (PDF)

Further, you can find both presentations at slideshare.

The slides are available as PDF here: Security on Rails (PDF) Deploying and Monitoring Rails (PDF)

If you attended one of my sessions, I encourage you to rate them at the conference site.

So far my expectations have been met and I've could catch up with a lot of people. I'm looking forward to day three of RailsConf Europe!

Lately I've seen several people struggling with an error message like this:

`gem_original_require': no such file to load -- sqlite3/database

This is for the SQLite3 gem but I've seen it also with Capistrano.

The problem is that some files have incorrect permissions in the authors repository and those incorrect permission are replayed by the gem packaging.

The next version of SQLite and Capistrano will solve those problems, the short term fix is just:

$ sudo chmod -R a+r /usr/lib/ruby/gems/1.8/gems

If you are following my twitter stream you have probably read me rambling about a security hole in Rails < 2.1.

With Rails 2.1 out for a while I though I should describe the problem a bit as there are many Rails applications stuck in 1.2.3 or 2.0.

The story begins with me giving a talk about Ruby on Rails security at Dynamic Languages World Europe in Karlsruhe. While talking about SQL injection, Tobias Schlottke, a fellow German Rails developer, mentioned that he once saw ActiveRecord allowing random SQL in the :offset option. Everybody in the room agreed that shouldn't be possible or wanted.

Later that day I sat down with Steven Bristol and we tried to verify this. Playing with MySQL we only got ActiveRecord::StatementInvalid exceptions. So I installed PostgreSQL (damn you MacPorts!) and tried the same with PostgreSQL and SQLite. Both times I got straight SQL Injection. Looking through the code I found out that the :offset parameter was also prone to SQL injection:

# vulnerable controller code
User.find(:all, :limit => params[:limit])

User.find(:all, :limit => 10, :offset => params[:offset])

# with params[:offset] or params[:limit] set to '; DROP TABLE users;'
# you got a big problem ...

MySQL is also affected but thanks to its default setting of disallowing multiple SQL statements per API-call you cannot insert a new SQL clause. You can just manipulate the one executing by setting other parameters (like an offset).

I've seen a lot of code taking the :offset parameter straight from the HTTP params and all people I talked with thought of it being a big problem. Especially since the documentation speaks about :offset and :limit being two integers (so you, or at least I, would expect auto-quoting like done for User.find(params[:id])).

I wrote a patch that casts both :limit and :offset to integers and sent it to core@ hoping for it to be included before Rails 2.1. After a bit of discussion if this is a bug or a feature (as apparently :limit can also be '1, 5'), Steven got hold of DHH at RailsConf and of Aaron Bedra who apparently discovered this problem a couple of months ago. Steven wrote a new patch with Aaron that also checks for the '1, 5' syntax of :limit and David committed it for Rails 2.1.

Unfortunately the committed patch didn't include fixing the MySQL adapter as it overrides the method that adds the limit and offset options. This isn't as bad as it sounds as MySQL will not allow multiple SQL statements by default. David later committed a patch of mine fixing MySQL in edge (and the coming 2.1.1).

So if the new features alone will not bring you to Rails 2.1, here you have another reason to upgrade.

For you guys stuck in 1.2 or 2.0 land, either always cast your :limit and :offset options, or use those patches:

Rails 1.2.3 patch

Rails 2.0.2 patch

P.S: You guys using will_paginate are save as it casts the parameters by default.

Rails 2 introduced ActiveSupport::TestCase and friends, RoR's enhancement of Test::Unit.

Those extra classes made testing Rails controllers easier and removed the need for cluttered setup methods. Today DHH committed a new feature to ActiveSupport::TestCase (by Jay Fields) that allows Rails tests to match up with RSpec's and Shoulda's nicer declaration style test naming: test/do declaration style testing.

In plain Test::Unit each test would be a method named 'test_' followed by the name the test:

def test_email_format_is_validated
  ...
end

def test_invalid_credit_card_number_throws_exception
  ...
end

This works ok but is a bit clumsy and gets ugly with long method names. In edge you can now write the test like this

test 'email format is validated' do
  ...
end

test 'invalid credit card number throws exception' do
  ...
end

What happens in the background is that ActiveSupport::TestCase will just generate the test_email_format_is_validated method for you. What is missing is a nice integration with the test runner.

This brings Rails developers that envy RSpec's and Shoulda's declarative style to the same level. RSpec&co can still do more tricks but most developers I know really just lust for the it 'should do as I want it to' do ... end syntax and don't really care about the a.should == b.