There is a nice ActiveRecord Rails plugin called rails-settings that does the following:
Settings is a plugin that makes managing a table of global key, value pairs easy. Think of
it like a global Hash stored in you database, that uses simple ActiveRecord like methods
for manipulation. Keep track of any global setting that you dont want to hard code into
your rails app. You can store any kind of object. Strings, numbers, arrays, or any object.
We have used this plugin in the past for global key / value pairs and it works great. However, for our Floxee app, we are using MongoDB / MongoRecord so this plugin would not work.
Lucky for us, MongoDB is already a document store so I was able to recreate similar code in about 15 lines of code inside of a “method_missing” method.
With this, we are now able to do things like:
>> AppSetting.jim = "is cool"
=> "is cool"
>> AppSetting.arr = ["jim", "was", "here"]
=> ["jim", "was", "here"]
>> AppSetting.arr.first
=> "jim"
>> AppSetting.hsh = {"hash_1"=>1, "hash_2"=>"demo"}
=> {"hash_1"=>1, "hash_2"=>"demo"}
>> AppSetting.hsh["hash_2"]
=> "demo"
•
I had the opportunity to share MongoDB with the Dallas.rb group this evening. Thanks @mully and @jnunemaker for contributing slides!
•
At RailsConf recently we met a lot of folks who gave us kudos on this blog. Many were surprised to learn that Locomotivation is brought to you by the Squeejee team.
So, we’ve decided to integrate our code blog a bit more with our web site. Sorry if the move caused your RSS reader to panic. Hopefully, once the dust settles we’ll have more regular content from our crazy ride on Rails.
•
Wynn and I started dabbling with CouchDB this week. As with any technology, step one when dabbling is installing the software.
For Mac OS X Leopard Intel users, there is a very nice CouchDB packaged zip file that can be downloaded from the Couch Projects Google Code site. The huge positive of going this route is that it is drop dead simple to install. Simply download the .zip file, extract the code, and double-click the “CouchDBX” file. You then simply click the “Play” icon and you are off and running with CouchDB.
The negative going this route is that, as the time of this blog post, this package runs CouchDB version 0.8.1 which is about 7 months old.
In order to run the new 0.9 version of CouchDB, you will need to pull the latest source code from the SVN trunk, install some prereqs, and compile from source. Here are the exact steps I took to get up and running.
NOTE: For details on each of these steps, you can look at the README file that is downloaded with the couchdb source code.
From the command line:
Browse to the following URL in a browser to view your CouchDB Futon admin tool:
http://127.0.0.1:5984/_utils/
•
I did some research on what was new in the Continuous Integration testing field in regards to Ruby this week. We have used CruiseControl.rb in the past which worked fine, but I remember the setup being a bigger pain than I thought it had to be. I was hoping for something simpler.
With this in mind, I decided to give Integrity a shot. Their title of “The Easy and Fun Automated Continuous Integration Server” was exactly what I was looking for. After installing, configuring, and playing around with it for the past day or so, I have not been disappointed.
With that being said, I did have to jump around to a few different sources to get everything running correctly on a server with Phusion Passenger, so I thought I would document the steps I had to take here.
Installation:
This will create an Integrity application directory with a couple files on your server, mainly config.yml, config.ru, and a “public” directory we will point our Apache virtual host to.
Edit config.yml to suit your needs. This file is very will commented and your edits should be self-explanatory. I kept the default Sqlite database configuration to keep things simple. However, due to this, I had to install Sqlite on our Ubuntu server via:
Setup a DNS entry to point to this server and create an Apache virtual host as you would any other Rails or Sinatra web application:
Once this is all complete, simply restart apache to access your new Integrity instance.
Project Creation:
Create a Test Database:
(You can skip this test if you can use Sqlite databases for testing. Our code inevitably has MySql specific SQL syntax in it which requires us to use MySql for our tests.)
Create a MySql user that will be used for Integrity tests:
Note that “IDENTIFIED BY PASSWORD” is optional. If it is not included, your user can log in without a password.
Project Creation:
Assuming the above was successful, you should now be able to browse to the Integrity URL you created. From here, click the “Add new project” link.
Customizing Your Build Script
The build script can be a tad bit tricky. It is our practice not to keep database.yml in Git for password security purposes. Because of this, Integrity will not know the test database configuration settings needed to run the tests. To get around this issue, we created a database.yml.test file with the db credentials Integrity would need to run a test similar to this:
Using MySql:
Using Sqlite:
In my Rails project, I then created the below custom Rake task in “lib/tasks/my_cool_application.rake”:
Once this is done and the files are added/committed/pushed to git, go back to your edit project page in Integrity and change your “Build Script” to be “rake test:move_test_db_config features” for Cucumber tests, for example.
Manual Build
After all of the above is complete, you should be able to click the “manual build” link for your new project to pull your code and run your tests!
Automated Tests Via Git Service Hooks
Integrity tests are automated with Git Service Hooks. Adding a service hook is drop-dead simple with GitHub. To do so:
- Browse to your project in GitHub
- Click on the “Admin” tab
- Click on “Service Hooks” in the header
- In the “Post-Receive URLs” box, add a URL similar to this:
- http://username:password@integrity.example.com/my_cool_application/push
- where “username” and “password” are the HTTP authentication credentials setup in the integrity/config.yml file
Once this is setup, your CI Integrity test will be run everytime you code is pushed to the appropriate branch in GitHub.
Notifiers
Integrity also provides a notification system to inform you when tests are run.
Current notifiers include email, Jabber, Campfire, IRC, Twitter, and Basecamp. These are installed independently of Integrity via Gems. Once a gem is installed on your CI server, just log back into your project within your Integrity web site and setup your notifier credentials. You are also free to write your own custom notifiers for applications that are not supported yet.
Resources:
Thanks to the following sites for assisting me with my original setup and this blog post:
•
Having fled the cubicle jungle of Corporate America, we have a strong distaste for status reports. That’s not to say we don’t believe in letting others on the team know what we’re doing. We just prefer to workstream our activities and let our work announce itself to the rest of the team.
Present.ly - our workstreaming tool
I hate to put Present.ly in a box by calling it “Twitter for Business” (because it is so much more than that), but that’s the easiest way to describe it.
Familiar Twitter feel
If you’re comfortable using Twitter, you’ll feel at home with Present.ly because it supports @replies, direct messages, and hashtags.
Questions and urgent messages
Present.ly adds some smarts to the display of messages or “tweets.” Messages that end in a question mark are displayed as questions to the group: 
Need to mark a message as urgent? Simply add three exclamation marks: 
Private and secure
Unlike Twitter, only your users see your messages.
Groups!
I really hope Twitter builds functionality for groups one day and when they do, I hope they rip off Present.ly’s implementation. Admins control who belongs to what groups. Perhaps the coolest feature of Present.ly groups is the broadcast. Broadcasts are like direct messages to a group except they are displayed as if sent from the group itself, which gets its own avatar. The following broadcast message
b system Elvis has left the building
would display as if it came from the System itself: 
How we use it
Much of our Present.ly use is still manual. We tweet each other questions and comments just like we do on Twitter. In fact, Present.ly let’s you cross-post to twitter by using the t! command:
t! This message would go to Twitter, too!
But, increasingly, we’ve automated much of our workstreaming by sending broadcast messages at certain points in our development cycle.
Source control checkins
One of the most important things we do as a development team is add code to source control. We use “GitHub”:http://github.com for our code repositories. GitHub has a nifty feature called Service Hooks, which allows you to send a message to a number of third party services for code commits to your repo. Setting up Present.ly really easy:
Once set up, GitHub will send a new broadcast message from the group you specify with each new code push. The message includes the commit message and a link to the commit back on GitHub:
Deploy notices
Before Present.ly we had been sending our deployment notices to Campfire using Tinder. While this works well, we really preferred to have our workstream in one spot and leav eour Campfires for discussion and chat. Since Present.ly implements the Twitter API, wiring up the deploy notices from Capistrano was a piece of cake.
Now everybody knows when the build is done:
How are you using it?
We’d love to hear how you’re using Present.ly in your workflow? Any cool uses we’re missing?
•
I’ve been building web applications for a dozen years now and I’m convinced there has never been a better time to be a web developer! I can’t imagine doing my job if it weren’t for the impact of some great people, sites, and software. This post isn’t a Ten Biggest style article. I won’t be thanking Tim Berners-Lee, although we all should.
Rather, this list is simply the extraordinary people and things that have made my job a joy over the past year.
People I am thankful for
Creator of jQuery, JavaScript ninja, John casts a large shadow on the web development landscape. Looking at John’s code has taught me more about JavaScript than every book I’ve ever bought on the subject.
Half of the err the blog duo, a third of the GitHub posse, Chris is the most energetic Rubyist I’ve seen. Just watching the amount of software his company puts out will either inspire you or send you into a shame spiral.
Makers of cool, free, and downright funny software, these guys write more less code than anybody around. I’m thankful that guys like this are around to keep us from taking ourselves too seriously.
I’ve never seen anyone single-handedly will a project into existence like JP. While there is no doubt that Redmine has a large and growing community behind it, I’m sure everyone would agree that Redmine exists because of the sheer will of Jean-Phillipe.
Sites I am thankful for
No other technology has impacted me more this year than Git, and I don’t think it would have happened without GitHub. We first signed up for GitHub because we wanted a turnkey central repository for our projects. But GitHub is so much more. GitHub brings social networking to source control. This means you can see what other folks are working on and discover new projects and tools all the time. It also means it’s never been as easy to contribute to open source. Want to improve an open source project? With GitHub, you just fork it and send a pull request. If the original owner goes off the grid, no worries, you’ve got your own repo and keep on chugging.
I’ll have to admit at first I didn’t get Twitter. I mean, take a look at the public timeline. Who cares when random folk are bathing? I’ve since learned that there is no better tool for finding those other folks that are just as weird as you. You know, those long lost members of your tribe(s). Folks who are interested in those things you bug everyone else about.
Soon after I climbed on the Twitter train, I longed for something as easy as Twitter to workstream all those source control checkin notices, build statuses, and “what I am doing” updates. The folks at Intridea have hit a homerun with Present.ly. The groups and hashtag functionality is awesome. Try it for your team!
Software I am thankful for
We’ve used a lot of issue trackers. Some great ones. Some workhorses. But we’ve never used anything as extensible as Redmine. Plugins, themes, and custom fields, oh my!
I want to thank Tim Pease for building such a cool little tool. I’m using Webby to write this post!
Blueprint has been a part of my standard toolbox since it came out. Blueprint brings sanity to my layout and typography while make it so easy developers can do CSS!
This year I’ll be looking to Compass to semantify my Blueprint.
I wasn’t in the market for a new JavaScript framework when I discovered jQuery, but I quickly fell in love with this one. jQuery lets me do more powerful things with less code than any other framework I’ve used. With comprehensive CSS3 (and bonus) selectors, chaining, and plugin architecture it’s a joy to use.
How about you?
What are the people, sites, and projects you’re thankful for this year?
•
I had the opportunity to give a talk at Lone Star Ruby Conf 2008. There are a lot of hardcore Rubyists here writing great web applications in Rails, Merb, and Waves. It was fun to knock around ideas with so many folks about how Javascript fits into the mix.
Until the slides are posted on the official LSRC site you can grab them from SlideShare.
•
We’ve been huge fans of named_scope since its introduction in Rails 2.1. If you are new to the named_scope concept, please read Ryan Daigle’s intro to named_scope and watch Ryan Bates’ named_scope RailsCast. Ryan Daigle also had a good follow up article explaining his utility scope gem.
With a recent project, we really started utilizing extensions with named_scope which is very powerful and cleaned up our code considerably.
For example, let’s say you have an issue tracking system that has an Entry model that belongs_to both a User model and a Project model. Using named_scope, we can easily access Entries for a date range filtered to both the Project and User level using named_scope lambdas:
To find the Entries for the last week for the first user, we would simply do:
u = User.first
u.entries.by_date(1.week.ago, Time.now)
To find the Entries for the last week for the first project, we would simply do:
p = Project.first
p.entries.by_date(1.week.ago, Time.now)
What happens if we want to see the sum of hours worked for a date range? This is where extensions come in very handy. Let’s add a “total_duration” extension to our named_scope:
Now to get the duration for the past week for the first user we simply do:
u = User.first
u.entries.by_date(1.week.ago, Time.now).total_duration
with similar logic at the project level.
To group all of the entries by project, we would do:
If you need the lambda arguments inside your named_scope extension, you will need to use “self.proxy_options”. For example, let’s say you need the from and to dates in order to calculate the average duration per weekday. That code would look something like this:
In the above example, “weekday_count” is a custom method we created to calculate the number of weekdays between 2 given dates.
The opportunities to simplify code using named_scope is great. As I was writing this post, one of our team members, Jason Derrett, created a named_scope to wrap a Xapian full text search into a Lesson model:
In this example, using Xapian to search for a keyword becomes as simple as:
Lesson.find_with_xapian "fishing"
How cool is that?!?
Are there any other cool named_scope tricks out there that any of you are using?
•
In my previous blog post, I wrote about the several full-text search engines that are available to us as Rails programmers and how we ultimately decided to go with Xapian.
Background
I am by no means a Xapian expert. I first heard about it less than a week ago. However, we have been very impressed with its combination of ease of installation and robust features.
Xapian is written in C++ and has been around in its current form since 2001. Its code base is derived from a previous search tool that went closed source.
- Written in C++
-
Feature rich including relevance feedback, boolean search operators, stemming, wildcard searching, and similar results
Its user list includes del.icio.us and GMane both of which are indexing tens of millions of records, so scalability should not be an issue. Here is a quote about scalability from the site:
People often want to know how Xapian will scale. The short answer is “very well” - a previous version of the software
powered BrightStation’s Webtop search engine, which offered a search over around 500 million web pages (around 1.5 terabytes of database files).
Searches took less than a second.
The largest recent installation we’re aware of is probably gmane, which currently indexes over 50 million mail messages.
On the other hand, indexing time could take a while if you have a huge set of documents. According to this article, it took about 5 hours to index the entire Wikipedia database using Xapian for offline Wikipedia access and searching.
Acts_As_Xapian is the Rails plugin for the Xapian. It was written in May 2008 by Francis Irving after he experienced some frustrations with Solr. Acts_As_Xapian has a google group at http://groups.google.com/group/acts_as_xapian.
Xapian Installation:
For Ubuntu Users:
sudo apt-get install libxapian15 libxapian-ruby1.8
For Mac / Other: (taken from the xapian.org install docs)
(version = 1.0.6 at the time of this writing)
wget http://oligarchy.co.uk/xapian/1.0.6/xapian-core-1.0.6.tar.gz
wget http://oligarchy.co.uk/xapian/1.0.6/xapian-bindings-1.0.6.tar.gz
Alternately if you haven’t installed wget on the mac:
curl -O http://oligarchy.co.uk/xapian/1.0.6/xapian-core-1.0.6.tar.gz
curl -O http://oligarchy.co.uk/xapian/1.0.6/xapian-bindings-1.0.6.tar.gz
tar zxvf xapian-core-versionnumber.tar.gz
tar zxvf xapian-bindings-versionnumber.tar.gz
cd xapian-core-versionnumber
./configure --prefix=/opt
make
sudo make install
If you don’t have root access to install Xapian, you can specify a prefix in your home directory, for example:
./configure—prefix=/home/jenny/xapian-install
cd xapian-bindings-<version>
./configure XAPIAN_CONFIG=/opt/bin/xapian-config
make
sudo make install
Acts_As_Xapian Installation:
Install the plugin:
script/plugin install git://github.com/frabcus/acts_as_xapian.git
Generate and execute the acts_as_xapian migration:
script/generate acts_as_xapian
rake db:migrate
Basic Model Code
To include fields to your index, you will need to add an “acts_as_xapian” call to the model you want to index.
class Lesson < ActiveRecord::Base
acts_as_xapian :texts => [:name, :description]
end
In this case, only the data found in the “name” and “description” attributes of our Lesson model will be searched.
A Quick Test Of Your Index
At this point, you can quickly test that your Xapian install, plugin, and model code are all working together via the following rake commands:
To build the index:
rake xapian:rebuild_index models="Lesson" RAILS_ENV=development
(You can add more models with a space delimeter. “Lesson User Tag”)
To update index:
rake xapian:update_index RAILS_ENV=development
To test index:
rake xapian:query models="Lesson" query="golf" RAILS_ENV=development
The Basic Search
Adding the this line of code
@search = ActsAsXapian::Search.new([Lesson], @lesson_search_params)
will return a ActsAsXapian::Search object with the following (From the acts_as_xapian README):
- description – a techy one, to check how the query has been parsed
- matches_estimated – a guesstimate at the total number of hits
- spelling_correction – the corrected query string if there is a correction, otherwise nil
- words_to_highlight – list of words for you to highlight, perhaps with TextHelper::highlight
- results – an array of hashes each containing:
- :model – your Rails model, this is what you most want!
- :weight – relevancy measure
- :percent – the weight as a %, 0 meaning the item did not match the query at all
- :collapse_count – number of results with the same prefix, if you specified collapse_by_prefix
So, if we want to return an ActiveRecord model object we could do something like this:
@lessons = @search.results.collect {|r| r[:model]}
To get the recommended spelling for a misspelled word(s) in a search, we would do:
@corrections = @search.spelling_correction
Similar Results
If you have a requirement to show similar or “You may also like” results based on a search result set or a single result, this can be done with a single line of code:
@similar_lessons = ActsAsXapian::Similar.new([Lesson], @lessons).results.collect {|r| r[:model]}
where @lessons in this case would be the list of lessons returned in the original search.
Updating The Index
Unfortunately, one drawback to Acts_As_Xapian is that the indexers are not updated automatically when data in an indexed model is added, modified, or deleted. When model data is changed, Xapian will put a record in the table ‘acts_as_xapian_jobs’ in your database to notify the update task to update the index. A cron job or something of the like will be needed to call the following rake task periodically to keep the index up-to-date:
rake xapian:update_index
Francis, the acts_as_xapian creator, commented on this on my original Mulling Over Our Ruby On Rails Full Text Search Options blog post.
with Xapian you can update and search simultaneously, and updates are immediate. However, only one thread can update a Xapian database at the same time. Since I wanted offline indexing anyway (as my index operation is risky, complex and slow, involving parsing Word documents, PDFs etc.), I didn’t try to find a solution that causes a second thread in the web application to, say, wait for the database lock. So acts_as_xapian currently only supports offline indexing.
Conclusion
This is a very basic tutorial meant to highlight the steps needed to get up and running with simple Xapian searches quickly. There are quite a few topics that I did not touch on such as:
- Sorting and grouping with the :values option in the model
- Advanced searching with the :terms option in the model
- Adding extended attributes from other models into a searched index. (Such as adding lesson comments to the Lesson index)
- Filtering searched data (in our case, we had to filter our results so only “active” lessons were returned)
- Highlighting keywords in the search results
•
