Creating FreeBSD ports for ruby gems

Posted by Jonathan

Creating FreeBSD ports for ruby gems involves creating a pkg-plist that lists all files and directories for a given port. This kind of sucks for gems that have a lot of rdoc documentation (like ActiveSupport). There are a couple of automatic solutions like pkg_trackinst but none is perfect.

While updating the ports for Rails 1.2.1 I had enough and created a small Ruby script to help with the generation of pkg-plists. Updating a Rubygem port now looks like this:

# cd /usr/ports/devel/rubygem-activesupport
# vim Makefile
(update PORTVERSION)
# make makesum

At first update the port version and create the new checksum. Now install the new version (without updating the pkg-plist, so will will get a lot of errors during deinstall).

# make install

Now run the script to generate the new pkg-plist out of the installed gem:

# /tmp/gen_plist.rb activesupport-1.4.0 > pkg-plist

The script assembles a pkg-plist of of all files and directories under /gems and /doc. Any other files like the rails executable for the Rails gem must be added by hand as they will not be installed under /usr/local/lib/ruby/gems/1.8/.

What’s left to do now is to cleanup manually the installation and test the pkg-plist through creating a package and deinstalling the port.

The small script looks like this:

#! /usr/bin/env ruby  

package_name = ARGV[0]

raise ArgumentError if (package_name.nil? || package_name == '')

prefix = "/usr/local" 
gem_lib_dir = "#{prefix}/lib/ruby/gems/1.8/gems/#{package_name}" 
gem_doc_dir = "#{prefix}/lib/ruby/gems/1.8/doc/#{package_name}" 

pkg_plist = ["%%GEM_CACHE%%", "%%GEM_SPEC%%"]

# add lib
lib_files = `find #{gem_lib_dir} -type f`.split("\n")
lib_dirs = `find #{gem_lib_dir} -type d`.split("\n").reverse
lib_dirs = lib_dirs.collect{|d| "@dirrm #{d}"}

# add doc
doc_files = `find #{gem_doc_dir} -type f`.split("\n")
doc_dirs = `find #{gem_doc_dir} -type d`.split("\n").reverse
doc_dirs = doc_dirs.collect{|d| "@dirrm #{d}"}

# assemble
pkg_plist += lib_files + doc_files + lib_dirs + doc_dirs

# change PLIST_SUBs
pkg_plist.each do |line|
  line.gsub!(gem_doc_dir, "%%GEM_DOC_DIR%%")
  line.gsub!(gem_lib_dir, "%%GEM_LIB_DIR%%")
end

# print out pkg-plist
pkg_plist.each do |line|
  puts line
end

Upgrading ports and preserve make options

Posted by Jonathan

FreeBSD uses make options while building ports in order to build a certain extension or (de)activate certain features.

MySQL 4.1 for example can be build with the following options:

WITH_CHARSET=charset
WITH_XCHARSET=list
WITH_COLLATION=collate
WITH_OPENSSL=yes
WITH_LINUXTHREADS=yes
WITH_PROC_SCOPE_PTH=yes
BUILD_OPTIMIZED=yes
BUILD_STATIC=yes
WITHOUT_INNODB=yes
WITH_NDB=yes

There are two ways of utilizing these options. If the port uses OPTIONS in the Makefile, you can configure the options with

# cd /usr/ports/databases/mysql41-server
# make config

You will get an ncurses screen from which you can choose your options. Your configuration will be saved in /var/db/ports/PORTNAME/options and each time you build this port you get the same options. Reconfiguration is done through calling make config in the port directory again.

But many ports do not use the OPTIONS framework in their Makefiles. MySQL for example does not.

If you want to use some of the build options you have to do it like this:

# cd /usr/ports/databases/mysql41-server
# make -DBUILD_OPTIMIZED install clean

The problem is that you build options are not saved. So when you have to update MySQL, you can’t just use portupgrade -a because your options will be lost. You have to upgrade MySQL yourself (make && make deinstall && make install clean) and remember to use all your options again.

The author of portupgrade though of this problem and introduced /usr/local/etc/pkgtools.conf. With pkgtools.conf you can specify arguments that portupgrade uses while upgrading a port.

MAKE_ARGS = {
# a) Separate them with the space
‘databases/mysql41-’ => ‘WITH_LINUXTHREADS=1 BUILD_STATIC=1’,

# b) Specify them using an array
‘databases/mysql41-
’ => [
‘WITH_LINUXTHREADS=1’,
‘BUILD_STATIC =1’,
],
}

So you just edit all your ports with the correct build options here and you’re done right? No.

Only portupgrade will use this file so if you happen to build any of the ports yourself, your options are not used. And worse, if a port gets updated THROUGH portupgrade as a dependency of another port, the options will NOT BE USED by portupgrade.

So the only solution left is to use /etc/make.conf. In /etc/make.conf you can specify global make options.

CPUTYPE?= pentium3
CFLAGS= -O2 -mmmx -msse -pipe
COPTFLAGS= -O2 -pipe
WITHOUT_X11=yes
WITHOUT_GUI=yes

The problem with this approach is that these options will be set in every make invocation. So all my ports are build without a GUI and without X11 if this options exists in their Makefiles.

So the correct solution is to conditionally set the make variables in /etc/make.conf because I want BUILD_STATIC only set if the mysql port is compiled. You can do this with checking the CURDIR make variable that points to the current directory:

.if ${.CURDIR:M/databases/mysql}
BUILD_STATIC=yes
BUILD_OPTIMIZED=yes
WITH_LINUXTHREADS=yes
.endif

Now every time make is called in databases/mysql41-server (or better in databases/mysql* so that if I compile the client or decide to move to MySQL 5, the options are still set) make will have my options set. Even if you use portupgrade everything works as portupgrade just calls make.

Not pretty but the only real solution if you want to preserve make options through upgrades automatically.

Typo 2.6.0 on FreeBSD

Posted by Jonathan

A quick howto install typo 2.6.0 on a blank FreeBSD installation. It is quite easy with Rails 1.0 and typo 2.6.0 in the ports.

Install

# cd /usr/ports/www/typo
# make install clean

Choose your database

# cd /usr/ports/databases/ruby-{mysql | sqlite | postgres }
# make install clean

Configure

# cd /usr/local/www/typo-2.6.0
# vim config/database.yml

Now start your database and create the database with the included schema in db/schema.{mysql | sqlite | psql }.sql

Run

# ruby script/server

Rails 1.0 now in the FreeBSD ports tree 0

Posted by Jonathan

I updated all Rails related ports and they are now commited:

So have fun with Rails 1.0 on FreeBSD:

# cvsup /etc/ports-supfile
# cd /usr/ports && make fetchindex
# cd www/rubygem-rails

# make install clean

or

# portupgrade rubygem-\*

If you have Rails already installed through the ports tree

Rails 1.0 released! 2

Posted by Jonathan

Rails 1.0 was released!

The 1.0 release has no new features over 0.14.x (the unofficial betas for 1.0) and is available through rubygems.

I will update the FreeBSD ports of rails asap as I already got some mails about the new release. With the new ports updating should look like:

# cvsup /etc/ports-supfile
# make fetchindex
# cd /usr/ports/www/rubygems-rails
# make deinstall clean && make install clean

Updating your applications is described here.

The new pkg_add

Posted by Jonathan

One of the new features of pkg_add(1) in OpenBSD 3.8 is the ability to update all installed packages. 3.8’s pkg_add will not do it automatically but rather list all packages for you to do it yourself (through pkg_add -r).

The pkg_add in current though will update all packages:

# dmesg | head -1
OpenBSD 3.8-current (GENERIC) #159: Tue Sep 27 22:21:33 MDT 2005
# export PKG_PATH=ftp://ftp.openbsd.org/pub/OpenBSD/snapshots/packages/i386/
# pkg_add -u
Candidates for updating arping-2.05p0 -> arping-2.05p0
No need to update ddclient-3.6.3
Candidates for updating dnsmasq-2.22 -> dnsmasq-2.22
Candidates for updating gdiff-2.8.1p0 -> gdiff-2.8.1p0
Candidates for updating gettext-0.10.40p3 -> gettext-0.10.40p3
Candidates for updating libiconv-1.9.2p1 -> libiconv-1.9.2p1
Candidates for updating libnet-1.0.2a -> libnet-1.0.2ap1
No need to update lzo-1.08p0
No need to update mergemaster-1.46p0
Candidates for updating openvpn-2.0 -> openvpn-2.0.2
Candidates for updating rsync-2.6.5 -> rsync-2.6.6
No need to update screen-4.0.2-static
Candidates for updating vim-6.3.61p0-no_x11 -> vim-6.3.85p0-no_x11
Candidates for updating wget-1.8.2 -> wget-1.10
Running the equivalent of pkg_add -r arping-2.05p0 dnsmasq-2.22 gdiff-2.8.1p0 gettext-0.10.40p3 libiconv-1.9.2p1 libnet-1.0.2ap1 openvpn-2.0.2 rsync-2.6.6 vim-6.3.85p0-no_x11 wget-1.10
libnet-1.0.2ap1 (extracting): complete
libnet-1.0.2a (deleting): complete
libnet-1.0.2ap1 (installing): complete
arping-2.05p0 (extracting): complete
arping-2.05p0 (deleting): complete
arping-2.05p0 (installing): complete
dnsmasq-2.22 (extracting): complete
dnsmasq-2.22 (deleting): complete
dnsmasq-2.22 (installing): complete
libiconv-1.9.2p1 (extracting): complete
libiconv-1.9.2p1 (deleting): complete
libiconv-1.9.2p1 (installing): complete
gettext-0.10.40p3 (extracting): complete
gettext-0.10.40p3 (deleting): complete
gettext-0.10.40p3 (installing): complete
gdiff-2.8.1p0 (extracting): complete
gdiff-2.8.1p0 (deleting): complete
gdiff-2.8.1p0 (installing): complete
openvpn-2.0.2 (extracting): complete
openvpn-2.0 (deleting): complete
openvpn-2.0.2 (installing): complete
rsync-2.6.6 (extracting): complete
rsync-2.6.5 (deleting): complete
rsync-2.6.6 (installing): complete
vim-6.3.85p0-no_x11 (extracting): complete
vim-6.3.61p0-no_x11 (deleting): complete
vim-6.3.85p0-no_x11 (installing): complete
wget-1.10 (extracting): complete
wget-1.8.2 (deleting): complete
wget-1.10 (installing): complete
Clean shared items: complete

Nice and clean. Grab a snapshot from ftp://ftp.openbsd.org/pub/OpenBSD/snapshots or use the upcoming 3.8.

Rubygems in the ports tree

Posted by Jonathan

It seems like my efforts to integrate Rubygems into the FreeBSD ports tree were not only for my own needs. Ports for the RGL and Stream rubygems were submitted by Ulrich Spoerlein and are now part of the ports tree.

RGL is a framework for graph datastructures and algorithms implemented in Ruby while Stream is an interface for external iterators.

I hope that this will be continue to be useful for FreeBSD users. My work on the OpenBSD integration is at a halt on the moment because I wait for the Ruby 1.8.2 integration. Due to the upcoming 3.8 release commits on the ports are bug-fixing only so this will be delayed until 3.8 will be released in November.

Typo 2.5 released

Posted by Jonathan

A new version of Typo, the Ruby On Rails based blog engine, was released. The new features include:

  • Static Caching, your typo blog will now serve thousands of pages per second with ease. All the active elements of the frontend were moved into JavaScript
  • First official release with the new Azure theme
  • Multiple theme support, please create your own!
  • Newly designed admin interface Textile, Markdown and Smartypants support, all bundled
  • Enhanced spam protection
  • Quick post functionality
  • Support for multiple users posting to the same blog
  • Powerful sidebar management with drag and drop goodness
  • Monthly archives
  • Multiple syndication support built in for 43things, del.icio.us, Flickr, upcoming.org, tadalist, technorati, 43places and much more
  • Ajax’ed out the wazoo
  • Edit/nuke comments while reading them in the frontend ( as admin )
  • Converters for most major blog software including import from RSS feeds.

See also Justin Palmer’s post for details.

I will update my blog and the FreeBSD port to 2.5 over the next days.

Rubygems now integrated in the FreeBSD ports tree

Posted by Jonathan

My patches have been commited. New rubygems can now be easily integrated into the FreeBSD ports tree.

The common Makefile in devel/ruby-gems handels the download and installation for the .gem file.

That means that the user is now able to manage gems through the ports tree and does not have to also use the gem command.
For example in order to install rails do:

# cd /usr/ports/www/rubygem-rails
# make install clean

And all the rubygems on which rails depends will be installed.

These rubygems-ports are available yet:

databases/rubygem-activerecord
mail/rubygem-actionmailer
devel/rubygem-rake
devel/rubygem-activesupport
www/rubygem-actionpack
www/rubygem-rails
www/rubygem-actionwebservice
www/rubygem-redcloth
www/typo

If you want to use the rubygem-* ports make sure to deinstall the manually installed gems before (not all of course, only those that would conflict with the ports version).

I also work on patches for OpenBSD and hope to get them integrated into OpenBSD 3.8.

Status of Rubygems in BSD ports trees 0

Posted by Jonathan

It’s been a while since I reported on my effort to integrate rubygems into the FreeBSD and OpenBSD ports tree.

My FreeBSD patches a waiting in the PR database for someone to commit them but I need to update them to 0.13.1.

Today I sent patches to ports@openbsd.org that integrate rubygems in the OpenBSD ports tree along with ports for Rake, Rails, Actionmailer, Actionpack, Actionwebservice, Activerecord, and Activesupport. Further I created a port for ruby-sqlite and sent a patch that updates Ruby to 1.8.2.

I hope that this time the patches will be reviewed and commited.

You can get my patches here:

gem-ports.tgz (ruby-gems and all the gem-* ports as several directories)
ruby_mega.patch (ruby-gems and all the gem-* ports as a big patch)
ruby-sqlite.tgz
ruby182.patch

New patch for RubyGems in the FreeBSD ports tree

Posted by Jonathan

I just created a new patch that contains all my rubygems-ports. I made some small changes to my last version after I got some feedback. The category used by the patch is now rubygem (gem was used before).

Again, feedback is appreciated.

I hope that this is the last version before I fill out a FreeBSD problem report in order to officially include it in the ports tree.

Get the patch here.

BTW all my patches were commited, so now ruby-fcgi, ruby-mysql and ruby-sqlite are now up-to-date on FreeBSD.

First version of RubyGems in the FreeBSD ports tree

Posted by Jonathan

After a weekend working on the infrastructure for RubyGems in the FreeBSD ports tree, I finally have a version that is worth releasing.

It works by wrapping the gem install calls in make install/pkg_add. I created the following ports:

  • devel/gem-rake
  • devel/gem-activesupport
  • databases/gem-activerecord
  • www/gem-actionpack
  • www/gem-actionwebservice
  • www/gem-rails
  • www/gem-redcloth
  • www/typo

The various gem ports include my Makefile.common (placed in devel/ruby-gems). This Makefile sets common install instructions and variables.
A new gem-port has now not much to tell in it’s Makefile.

From www/gem-actionpack/Makefile:

PORTNAME= actionpack
PORTVERSION= 1.8.1
CATEGORIES= www gem

MAINTAINER= jw@innerewut.net
COMMENT= Action Controller and Action View of Rails MVC Framework

MASTER_SITES= http://rubyforge.org/frs/download.php/4149/

BUILD_DEPENDS= ${LOCALBASE}/${SPEC_DIR}/activesupport-1.0.4.gemspec:${PORTSDIR}/devel/gem-activesupport
RUN_DEPENDS= ${BUILD_DEPENDS}

.include <bsd.port.pre.mk>
.include ”${PORTSDIR}/devel/ruby-gems/Makefile.common”
.include <bsd.port.post.mk>

You further have to create a pkg-plist file that lists all files installed by this gem-port. At first I thought about getting these from the gem specification as it lists all installed files. But the gemspec does not include the rdoc files and creating the pkg-plist file with pkg_trackinst/pkg_genplist is very easy.

For now the gem-rails and typo port have no webserver or database listed as a dependency. They come with WEBrick and installing MySQL or SQLite is not complicated. I plan to include them in the future.

You can get my work here. Read the README for instructions.

Please test this and comment!

Progress on my BSD ports

Posted by Jonathan

The last days I’ve been working on some ports for OpenBSD and FreeBSD.

I created an OpenBSD port for dnsmasq-2.22. dnsmasq is a caching DNS-forwarder and DHCP server designed for small networks. It is very easy to setup and configure. I use it for my home network as caching DNS server on FreeBSD and I wanted to be able to use it on OpenBSD too.

I already sent it to ports@openbsd.org and got some feedback. But actual testing feedback is welcome. You can get the port here. Just unpack it into /usr/ports/net, give it a try and let me know if you have any problems.

Then I spent some time working on the rails/ruby-gems stuff on FreeBSD. Again I wrote an email to ports@freebsd.org, asking if somebody will join my effort. But again no reaction. I guess I have to do it by myself. I also tried to join the rubygems-developers mailing list, but I got no response from the webinterface after registering. Maybe I have till Monday or Tuesday until an actual human will review the administration email.

Further I looked how PHP Pear compatibility is realized in the FreeBSD ports and thought about how do copy it for ruby-gems. I tried to figure out, how exactly a gem is installed. It seems like that the gem is a tar-archive with two GNUzip-files in it, data.gz and metadata.gz. data.gz (a tar archive) hold the actual files and libraries while metadata.gz is a compressed file, metadata, that hold the, suprise, metadata information like dependencies, author, and file list.

As far as I know, if a gem GEMNAME is installed, the content of data.gz is copied to

/usr/local/lib/ruby/gems/1.8/gems/GEMNAME

and a gem specification file (GEMNAME.gemspec) is installed in

/usr/local/lib/ruby/gems/1.8/specifications/

My problem is that the .gemspec file in /specifications/ is different from metadata (but the content seems nearly the same) and I have to know how the former is created from the latter.

If this hurdle is taken, I hope that gem support on FreeBSD is not far away. I then have to expand the ruby-gems port with a Makefile that holds all common information like the path to the gem directory and how to copy the files. Then each new gem-port would only have to name its URL and dependencies. Then common Makefile would download and extract the gem, get the names of the included files from the metadata file and copy the files to the above described directories.

I fear that it will be complicated than this. But it is a start.

Apart from this effort I finally filled out problem reports for my updates to ruby-mysql, ruby-sqlite, and ruby-fcgi as knu@freebsd.org (their maintainer and maintainer of Ruby on FreeBSD) did not react since 3 weeks. ruby-sqlite was committed this morning and I hope that the other updates will be committed soon.

I did not filled problem reports for my various rails ports as these are not gems but “normal” ruby libraries. They install into

/usr/local/lib/ruby/site_ruby/1.8/

and so they are not recognized by ruby-gems as needed for rails. But they could be of value for a non-rails developer who wants to use this libraries without ruby-gems so I’m not sure if they should be dumped in favour of the ruby-gems ports that are to come.

Ruby on Rails and the FreeBSD ports

Posted by Jonathan

Yesterday I tried to combine Ruby on Rails with the FreeBSD ports tree. RoR relies heavily on ruby-gems but I prefer installing programs with the ports tree.

The ideal solution would be an integration of the two like done for PHP-pear and Perl CPAN in the ports tree but this is heavy lifting.
I started with creating non-gems ports of the several packages within rails.

I created:

devel/ruby-actionsupport
www/ruby-actionpack
www/ruby-actionwebservice
databases/ruby-activerecord
mail/ruby-actionmailer

When I went on to www/rails I found out that the rails.tgz from the rubyforge website includes all packages in the vendor directory :-(

That means that I should have created only the www/rails port or (cleaner) I should extract the packages from the vendor directory and let the rails port depend on my ports. The latter would benefit if somebody wants to use the various packages alone.

I’m not sure what to do as my goal is to combine gems and the ports tree.

Apart from this work I created some update-diffs for ruby ports that are needed by rails. I sent them to their maintainer who seems to be someone really involved in the ruby-stuff on FreeBSD. Maybe I should talk to him about the ruby-gems stuff…

Anyway, you can find my work here:

rails-ports.tar.gz
ruby-fcgi.patch
ruby-mysql.patch
ruby-sqlite.patch