BlogFish

Yesterday I gave a presentation about CouchDB and SimplyStored, our convenience Ruby library, at the Ruby User Group Berlin.

There is a recording of the presentation at ustream.tv.

Mathias and I wrote SimplyStored in order to easily interact with Ruby objects serialized in CouchDB. We use CouchDB as the main data store for Scalarium and so far it has been great. But it is a bit cumbersome to write all those map and reduce functions yourself.

SimplyStored generates the JavaScript map&reduce functions for handling associations or dynamic finders for you.

SimplyStored offers:

• Models
• Associations
• Callbacks
• Validations
• Dynamic finder
• S3 attachments
• Paranoid delete

    class User
      include SimplyStored::Couch

      property :login
      property :age
      property :accepted_terms_of_service, :type => :boolean
      property :last_login, :type => Time
    end

    user = User.new(:login => 'Bert', 
                    :age => 12, 
                    :accepted_terms_of_service => true, 
                    :last_login = Time.now)
    user.save

    User.find_by_age(12).login
    # => 'Bert'

    User.all
    # => [user]

    class Post
      include SimplyStored::Couch

      property :title
      property :body

      belongs_to :user
    end

    class User
      has_many :posts
    end

    post = Post.create(:title => 'My first post', 
                       :body => 'SimplyStored is so nice!', 
                       :user => user)

    user.posts
    # => [post]

    Post.find_all_by_title_and_user_id('My first post', user.id).first.body
    # => 'SimplyStored is so nice!'

    post.destroy

    user.posts(:force_reload => true)
    # => []
  

The code is on github and OpenSource: SimplyStored example code

Another thing I talked about is RockingChair. RockingChair is an in-memory CouchDB implementation that understands all of SimplyStored's functionality. We use it to speed up our tests and be able to run them in parallel.

Although Berlin looks a bit like Antarctica right now, I'm really looking forward to travel to Helsinki, Finland for Frozen Rails on May 7th.

Forzen Rails is organized by the guys from the HHLinuxCliub and Kisko Labs. I've met them in Amsterdam back in October and we had a great time. I'm sure Frozen Rails will be too. They have a great line-up organized with speakers like Yehuda Katz, Chris Wanstrath, Joseph Wilk, Jarkko Laine, Mike Dirolf, or Nizar Jouini. The ticket prices are reasonable, starting with 99 Euro for the early bird tickets. They even have a discount scheme that will give you a free ticket if you get 5 friends to sign-up. Regional conferences like this are always a lot of fun as you meet a lot of local developers and the organizers are very passionate. I hope to meet you at Frozen Rails 2010 in Helsinki.

The last past days in Amsterdam for Ruby En Rails 2009 were really great.

I arrived on Thursday and had the chance to discuss a possible security vulnerability in Rails I discovered a while back with Yehuda and Mislav during lunch.

Afterwards we went to the conference dinner and met many very nice people and had a long discussion about voting systems, European vs. American culture, gun laws, and political systems.

Friday was the first conference day and started for us with being 40 minutes late as we headed up to the wrong metro station. Yehuda was supposed to give a keynote as the first session... luckily the organizers swapped the sessions. So unfortunately we missed the first session but were in time for Yehuda to give his keynote.

Yehuda talked about the Rails/Merb merge and dissected what was achieved. It was a very good, in-depth presentation about the new features of Rails 3 and what left to do. I really liked the new router DSL, actions being Rack-apps, and the death of Rails Metal (the new, slicker possibility to build simpler/smaller controllers takes care of this.).

After the coffee break I gave my Rails Security presentation about common attacks against web applications and how you can protect against them in Rails. Usually this is quite heavy stuff and people tend to sit quiet and listen as this is new to most. In Amsterdam I had some very good questions and discussions on session fixation, JavaScript high-jacking, and app reconnaissance.

Then we listened to Julio Javier Cicchelli talk about Rubyists.EU, an effort to make the different European Rails communities easier to find and build a common European community.

We went to lunch with James which gave us some time to catch up. Took us a while to find a decent place around the conference hall but we managed to find a restaurant where we tried to estimate the global financial burden due to Internet Explorer and what will happen to Microsoft if they were charged for the extra work needed done.

After lunch Eloy educated us about the current state of MacRuby. I'm really looking forward to spending some time hacking Mac apps in Ruby.

The next session gave an overview about monitoring, performance, and the different tools like request-log-analyzer or Nagios.

The conference was closed by Jeremy giving a keynote about Rails, Ruby, and the current state of things in the community (and a sneak-peak at ActiveRelation and what it will be able to do). It was very well received and a good closing session.

Afterwards a big group headed for dinner. We ended up in a small, very local restaurant. It included very nice food, good discussions with new friends from Finnland, and a waiter/owner who runs the restaurant homepage with Rails and asked Yehuda for help :-)

The next day was titled Geek Day and included many lightning talks (that ended up being nearly full sessions and had very good content). Parallel to the great sessions about MongoDB, DataMapper, or experience reports there was a Rails Rumble featuring five teams going on.

The most notable session of the second day was Justin Halsall, dressed for Halloween, talking about BlockHelpers and view DSLs. It was a hilarious show.

The Rumble was a great idea. In contrast to the usual Rumble the teams got a specific challenge. They should build something that improves the situation with Rails dependencies and out-of-date gems laying around in vendor/gems. The winning team would get two tickets to RailsConf 2010.

One team extended builder to list outdated gems and got their changes even merged back to builder by Yehuda on the same day. Another team extended Webistrano to accept projects dependencies and display them on the stage page. Some teams build a command line tool to extract local dependencies like your gems or even the MySQL version and push those definitions to a central place. The winning team had the the most advanced idea regarding update notification and gem-sets for applications. I could really see something like this being integrated into gemcutter. Congratulations Ludo and Michel!

Ruby En Rails day two was celebrated with a big dinner and drinks. After a very nice evening we headed back to the hotel as everybody had an early flight out. Some were still discussing Rails, some were lucky to be able to enjoy Amsterdam longer.

I really enjoyed the conference and Amsterdam. Thank you Chris and Tim Obdam for organizing the whole event!

Recently I helped a friend debug a problem when deploying with Webistrano/Capistrano.

He was using a git repository and used SSH keys for authentication. Every time he tried to deploy he got this error:

 executing locally: "git ls-remote ssh://repo.example.com/git/myproject.git HEAD"
*** Could not save revision: Unable to resolve revision for 'HEAD' on repository 'ssh://repo.example.com/git/myproject.git'.

When running this command manually as the Webistrano user, everything worked fine.

We checked the usual suspects: the SSH key, the permissions on the SSH dirs/files, user, firewall & co. Everything seemed correct and worked when we ran the command by hand.

After a bit of tinkering I had the Eureka moment: the git command was not in $PATH when running under Passenger!

Git was installed and worked when we logged in as the Webistrano user. But when Passenger runs Webistrano it doesn't load all your shell config files. So if git is not in a standard location like /usr/bin or /bin Capistrano (which by this time will be called from Webistrano to do the actual deployment) will not find it.

I our case git was installed in /usr/local/bin and thus not in the default path. We ended up symlinking it to /usr/bin and everything worked like a charm.

I just committed a fix to Capistrano to make debugging such errors in the future easier. Capistrano will now check every local command it executes and see if it is in path. So with the latest version on github the error message would have looked like this:

 executing locally: "git ls-remote ssh://repo.example.com/git/myproject.git HEAD"
*** executable 'git' not present or not in $PATH on the local system!
*** Could not save revision: Unable to resolve revision for 'HEAD' on repository 'ssh://repo.example.com/git/myproject.git'.

So if you are running any shell commands under Passenger remember that it doesn't use a full login-shell.

Recently I've been made aware of people inside US Government organizations using my Ruby on Rails Security presentation as an excuse to limit Ruby on Rails adoption and projects inside those organizations.

They mandate that applications in Rails should be redone in Java because of the issues I covered.

It is not clear to me how anybody who saw/read this presentation would come to the conclusion that Rails is insecure. Every application is vulnerable. Some more and some less. Yes, this means that also Java applications are attackable.

It is my honest and strong belief that Ruby on Rails applications are not less secure than any other web application. In contrast, the Ruby on Rails framework provides several advanced security mechanisms that make it very easy to write secure applications. Further, Ruby on Rails enables very sane security options by default. Some of those defaults include auto-escaping, prevent Cross-Site Request Forgery, or protected from SQL-injection. I would even go as far as to state that the typical Rails application is more secure than the typical web application for those reasons.

The security of an application stands and falls with the knowledge and abilities of the people implementing and running it. Rails makes it very easy to write secure applications.

The conclusion I came to in my presentation still holds:

Ruby is by no means a "web app security silver bullet" but adding security is easy and not a pain like in many other frameworks

I hope this will change the opinion of some people and remove my presentation as their argument.

I know, it is a bit late, but here are my slides from Scotland on Rails:

The slides are also available as a PDF download: Advanced Deployment

Scotland on Rails was again a great conference. A very interesting crowd in a very nice city. I'm looking forward to next year!

As a consultant for Peritor I often review other peoples code. What I often encounter and always address is custom logic in Rake tasks.

Rake is perfect for automating tasks and making calls into your application from the command line. People often use it for doing maintenance, executing periodic jobs, or starting and stopping services. Nothing wrong with that, this is what Rake in Rails apps is for.

What I resent is having all this logic in the Rakefile, either inline in the task body or as a method in the Rakefile.

  # Rakefile
  
  task :ping_stale_users do
    stale_users = User.find(:all, 
                            :conditions => ["last_login < ? ", Time.now - 3.months])
    ...
    ...
    stale_users.each do |user|
      if user.check_something
        user.mail_invite
      end
    end
  end

This is just wrong for the following reasons:

• Testing is hard: methods and Rake tasks are hard to test. The environment is hard to create and you may have to shell-out to call rake.
• Reusing is hard: you cannot easily reuse this functionality from the console or an admin controller.
• POLA: Rakefile are not were people look for domain logic

The solution is simple and yet it seems underused. Just put your domain logic into a model or library and call it from the Rake task. This way it is much easier to test, reuse, and read.

  # Rakefile
  
  task :ping_stale_users do
    User.ping_stale_users
  end
  
  # User.rb
  def self.ping_stale_users
    stale_users = User.find(:all, 
                            :conditions => ["last_login < ? ", Time.now - 3.months])
    ...
    ...
    stale_users.each do |user|
      if user.check_something
        user.mail_invite
      end
    end
  end

Of course the method in the model can be cleaned up a bit more but this is not the point. The point is that we removed domain logic from the Rakefile and made it shorter and easier to understand. The logic is now somewhere where we can test and reuse it.

I just released Webistrano 1.4. Webistrano is a tool for managing Capistrano deployments and offers a rich web UI. It lets you manage projects with their stages and keep track who deployed which version to which servers.

Webistrano 1.4 brings many new features that make deployment easier. The most prominent are:

• Recipe versioning - recipes are now versioned so that you can keep track of changes
• Project cloning - you can now create a template project and clone it over and over again
• Array parameters - support for arrays as values for configuration parameters
• CAS-auth support - Single Sign-On support by delegating authentication to a CAS server. See the documentation
• Enhaced UI - nicer overviews of deployments and many small fixes
• Cancel deployments - a running deployment can now be canceled by Webistrano. The running Capistrano instance will be killed so use this feature with care
• Track deployed revisions - Webistrano will track which revision was deployed. This way you always know which version is running where
• Updated packages - Rails 2.1 and Capistrano 2.5.0

Apart from that some smaller enhancements and fixes went into the 1.4 release. See the CHANGELOG for a complete list.

Further, there is now a Webistrano mailing list at GoogleGroups.

Go get Webistrano from the project homepage as a download or checkout the source:

Download: webistrano-1.4.zip (3.4 MB)

# Development version:

svn co http://labs.peritor.com/svn/webistrano/trunk

# Stable version:

svn co http://labs.peritor.com/svn/webistrano/branches/1.4

RailsRumble 2008 is over - after 48 hours of hacking I'm proud to announce our contestant:

GoodBad.me - Twitter Till Judgement Day!

GoodBad.me is a twitter mashup that Thomas Metschke and I build over the last two days. Once you follow our bot on twitter, we subscribe to your tweets. We collect all your good and bad deeds by searching for #goodme or #badme (or the shorter #gd #bd) in your tweets.

We keep track of your deeds and generate stats about you. Your deed balance decides your character level. Are you just a boy scout or do you strive for beeing friends with Goldfinger?

All those questions can be answered at http://goodbad.me.

The idea is that you keep tweeting as always and by adding the #goodme tag you let the world know that you did something good.

In later iterations we want to allow others to tag your deeds/tweets and maybe allow you to gain points by donating to charity.

The site is online for a couple of hours and we already have several active users. If you want to keep track of your day's work, go join!

If you like the idea and our implementation, please vote for us!

If you are following my twitter stream you have probably read me rambling about a security hole in Rails < 2.1.

With Rails 2.1 out for a while I though I should describe the problem a bit as there are many Rails applications stuck in 1.2.3 or 2.0.

The story begins with me giving a talk about Ruby on Rails security at Dynamic Languages World Europe in Karlsruhe. While talking about SQL injection, Tobias Schlottke, a fellow German Rails developer, mentioned that he once saw ActiveRecord allowing random SQL in the :offset option. Everybody in the room agreed that shouldn't be possible or wanted.

Later that day I sat down with Steven Bristol and we tried to verify this. Playing with MySQL we only got ActiveRecord::StatementInvalid exceptions. So I installed PostgreSQL (damn you MacPorts!) and tried the same with PostgreSQL and SQLite. Both times I got straight SQL Injection. Looking through the code I found out that the :offset parameter was also prone to SQL injection:

# vulnerable controller code
User.find(:all, :limit => params[:limit])

User.find(:all, :limit => 10, :offset => params[:offset])

# with params[:offset] or params[:limit] set to '; DROP TABLE users;'
# you got a big problem ...

MySQL is also affected but thanks to its default setting of disallowing multiple SQL statements per API-call you cannot insert a new SQL clause. You can just manipulate the one executing by setting other parameters (like an offset).

I've seen a lot of code taking the :offset parameter straight from the HTTP params and all people I talked with thought of it being a big problem. Especially since the documentation speaks about :offset and :limit being two integers (so you, or at least I, would expect auto-quoting like done for User.find(params[:id])).

I wrote a patch that casts both :limit and :offset to integers and sent it to core@ hoping for it to be included before Rails 2.1. After a bit of discussion if this is a bug or a feature (as apparently :limit can also be '1, 5'), Steven got hold of DHH at RailsConf and of Aaron Bedra who apparently discovered this problem a couple of months ago. Steven wrote a new patch with Aaron that also checks for the '1, 5' syntax of :limit and David committed it for Rails 2.1.

Unfortunately the committed patch didn't include fixing the MySQL adapter as it overrides the method that adds the limit and offset options. This isn't as bad as it sounds as MySQL will not allow multiple SQL statements by default. David later committed a patch of mine fixing MySQL in edge (and the coming 2.1.1).

So if the new features alone will not bring you to Rails 2.1, here you have another reason to upgrade.

For you guys stuck in 1.2 or 2.0 land, either always cast your :limit and :offset options, or use those patches:

Rails 1.2.3 patch

Rails 2.0.2 patch

P.S: You guys using will_paginate are save as it casts the parameters by default.

Rails 2 introduced ActiveSupport::TestCase and friends, RoR's enhancement of Test::Unit.

Those extra classes made testing Rails controllers easier and removed the need for cluttered setup methods. Today DHH committed a new feature to ActiveSupport::TestCase (by Jay Fields) that allows Rails tests to match up with RSpec's and Shoulda's nicer declaration style test naming: test/do declaration style testing.

In plain Test::Unit each test would be a method named 'test_' followed by the name the test:

def test_email_format_is_validated
  ...
end

def test_invalid_credit_card_number_throws_exception
  ...
end

This works ok but is a bit clumsy and gets ugly with long method names. In edge you can now write the test like this

test 'email format is validated' do
  ...
end

test 'invalid credit card number throws exception' do
  ...
end

What happens in the background is that ActiveSupport::TestCase will just generate the test_email_format_is_validated method for you. What is missing is a nice integration with the test runner.

This brings Rails developers that envy RSpec's and Shoulda's declarative style to the same level. RSpec&co can still do more tricks but most developers I know really just lust for the it 'should do as I want it to' do ... end syntax and don't really care about the a.should == b.

The slides and a video of my Ruby on Rails Security session are now online. The 24C3 was a lot of fun, unfortunately I couldn't spend all 4 days there.

My talk covered most of the common web application vulnerabilities like Cross Site Scripting and Cross Site Request Forgery, SQL and Code injection, and deployment security and how they apply to Rails. Further Ruby on Rails specific issues like Rails plugin security, JavaScript/Ajax security, and Rails configuration were be examined and best practice solutions were introduced.

| View | Upload your own

The is also a Google video version: Ruby on Rails Security.

Get the slides (PDF - 1.6 MB) or the video (mkv - 95 MB). Other formats are available from the official mirrors or the torrent site.

Webistrano 1.2 is out. You can get it at the Peritor Labs project page.

Version 1.2 brings the following changes and enhancements:

• Fixes a bug in the execution of tasks with sudo
• Highlight recipe syntax with the syntax gem
• Include Capistrano 2.1.0 and git support
• Add experimental support for ssh_keys and ssh_ports as normal configuration parameter (currently only one SSH key is supported)

Although Capistrano 2.1.0 is included, Webistrano will by default allocate a pty for each SSH command as the new Capistrano default of now doing so seems problematic on some hosts.

Further there where some minor fixes and UI enhancements. All available configuration parameter are now documented in the wiki.

Upgrading is done with a `RAILS_ENV=production rake db:migrate`, more information on how to upgrade can be found here.

With version 1.2 out of the door I will focus on bringing authorization to Webistrano so that deployment permissions can be tight to individual users and groups.

Get version 1.2 either from the download page or directly here: webistrano-1.2.zip

The reason why wrote up my rant about not getting too exited about REST is that I had some fun time paying for my blind usage of REST.

While developing Webistrano I thought that this would be a nice project for playing with all the RESTful stuff that Rails currently offers. So I started to map my resources and thereby only allowing certain HTTP verbs to certain URLs.

This all works nice until reality in the form of IE hunts you down.

Until recently all my Ajax calls used HTTP POST for getting updates from the server. I used POST for so long that I didn't remember why. In Webistrano I use Ajax to periodically get status updates on a running deployment. As getting status updates translates perfectly to HTTP GET on the resource I used this code for it:

# controller
def show
  @deployment = @stage.deployments.find(params[:id])

  respond_to do |format|
    format.html # show.rhtml
    format.xml  { render :xml => @deployment.to_xml }
    format.js { render :partial => 'status' }
  end
end

# view _status.rhtml
<% unless @deployment.completed? %>
  <script type="text/javascript">    
    function update_status(){
      new Ajax.Updater('status_info','<%=h project_stage_deployment_path(current_project, current_stage, @deployment) %>',{
        method: 'get',
        evalScripts: true
      });
    }
    
    setTimeout(update_status, 3000);
  </script>
<% end %>    

This worked nicely in Safari and Firefox but Internet Explorer would update the status-div with the whole page. So you got the page-in-a-page effect. I've spend several hours trying to debug from where IE was getting this strange output and why there were no requests to the server. And then I found the answer and remembered why in the past I always used HTTP POST for my Ajax calls.

IE was caching the GET Ajax call.

In order to prevent IE from caching Ajax calls your need to either supply different parameters on each request or switch to POST. Switching to POST is not so easy as Rails will not allow POST requests to the .../deployments/1 resource. So unique parameters on each request it is:

# view _status.rhtml
<% unless @deployment.completed? %>
  <script type="text/javascript">    
    function update_status(){
      new Ajax.Updater('status_info','<%=h project_stage_deployment_path(current_project, current_stage, @deployment) %>',{
        method: 'get',
        evalScripts: true,
        parameters: {
          random_differentiator: Math.floor(Math.random()*50000) // work around IE caching bug
        }
      });
    }
    
    setTimeout(update_status, 3000);
  </script>
<% end %>

The alternative would be to define a custom action on the deployment resource that would answer to a HTTP POST but this destroys the whole "one resource URL, different representations" REST thing.

So long for RESTful Web Applications with IE.

REST is a nice theoretical concept and Rails is pushing hard in this direction. We all heard about RESTful Rails and map.resources but at least in my surrounding its adoption is still lacking.

At the last Berlin Ruby User Group meeting somebody asked if one of us had a running RESTful application and only one out of 40 responded. Then we got into the whole 'what is REST, when is your application RESTful, and is the edit-view part of REST...' discussion.

Yes, it would be wonderful if the world would be full of REST Web Services instead of SOAP Web Services but in the end we are talking about Web Services. If your application does not expose a service, there is no need for hardcore REST. That's why nobody raised their hand, nobody had a service to expose. Not yet, you could argue. And by doing it the RESTful way they would prepare for the day that their application is so successful that they start to think about exposing some resources. In in this case they would be prepared. Just add some responds_to magic and boom, you got your Web Service.

And I answer YAGNI. Do it when you need it, too much preparation is evil.

At the moment some people (especially in the Rails community) see REST as the new Golden Hammer. Just hammer with it once or twice on your application and boom out comes a Web Service for virtually no effort at all. And this is what scares me. Don't get me wrong. I like the idea of REST and definitely prefer it to the WS-* way of doing things. But making your application a Web Service that will serve anything, from JSON to XHTML and XML, just because you can will not solve your problems. It will create new ones because you couple different views and interaction schemes.

Seeing DHH at the RailsConf Europe keynote hacking custom MIME types in order to render a different view for the iPhone didn't make me jump up and down in a jolly manner. It made me thinking if the Rails community pushed too hard to the RESTful side. The iPhone required it's own view because although running a stock Browser it has a different interaction scheme. A different interaction scheme should force you to have a different logic in the controller. The responds_to blocks start to grow in complexity and size.

This is the point where you should apply the idea of de-coupling, the idea of services. Let the iPhone have it's own Web Application. Let than this new application access the common resources that the Desktop Browser application and the iPhone application share. This access can be done through just loading the same models through svn:externals or through a REST Web Service.

The REST concept should de-couple different clients (controller and views) from each other and from and back-end code (models).

Sometimes the responds_to stuff makes sense. If you mostly expose data, then offering JSON and XML besides XHTML makes sense if by doing this you save your clients the transformation or expose the data to more clients that you could not reach before. But it you have a Web Application with a lot of interaction then coupling the Desktop Browser version with the iPhone version, the mobile version, the JSON version, and the XML version does not make any sense to me.

For me REST is about exposing interconnected resources through a unified interface. And yes you should let the client choose the representation. But different clients will handle and interact differently with the resources. So please think before reaching for your Golden Hammer.