Hopefully, you’ve already figured out that we’ve moved all our blogging to the main Squeejee blog.
We decided to make this change in an effort to unify our efforts around the Squeejee brand, rather than splitting the tech stuff out into something else.
So head on over to http://squeejee.com/blog for all the latest goodness!
As you all probably well know by now, we’re big fans of MongoDB here at Squeejee. We’ve written about it, spoken about it, and ported mysql based apps over to it.
We’ve recently launched our first application built from the ground up on MongoDB. TweetSaver also happens to be the first production app that we know of utilizing John Nunemaker’s MongoMapper ORM.
Mongo’s schema-less nature made it easy to store all sorts of data from a number of APIs. Grabbing rich, deep data structures was easy because we didn’t have to relationally model the relationships between documents and subdocuments. We could simply ‘stash the hash.’
One of the coolest features of MongoDB is the ‘upsert.’ In ActiveRecord we do this with find_or_create_by_xxx. This performs two calls — a select and insert — for new records. MongoDB allows you to peform fire-and-forget saves by upserting:
myColl.update( { name: "Joe" }, { name: "Joe", age: 20 }, { upsert: true } );
where the name ‘Joe’ is your unique field.
One of the toughest things about going NO-SQL is having to think about the best way to model data for your application. Although MongoDB shines in storing deep, nested document structures, it also fully supports more relational database design. This choice means that you need to think through the tradeoffs up front. It’s a good idea to go through the scenarios in which you’ll consume pieces of data and determine how to store it.
Since MongoMapper is not ActiveRecord, all of those Rails gems and plugins that have ActiveRecord dependencies will not work. Your choices here are:
(a) Go with a MySQL hybrid approach to store a subset of data in MySQL. For example, we store our Users table in MySQL to take advantage of the number of Authentication plugins out there. (RestfulAuth / AuthLogic / TwitterAuth/ etc). However, even this is pretty simple to implement with MongoDB. Here is a Gist from John Nunemaker to implement authentication with his MongoMapper ORM: http://gist.github.com/147427
(b) Port over a solution to work with MongoDB. The nice part about this route is that there are no migrations to futz with. Just define your needed fields directly in your app, plugin, or gem.
It’s easy to play around with Mongo, see its speed and think that the laws of computing have been suspended. Alas, collections with hundreds of thousands of records still require indexing.
So, before you start your next project take a minute to see if MongoDB might be a good fit… it’s been a great addition to the Squeejee toolbox.
Thanks to Ron Evans for contributing ThumbFight support for our TwitterLand gem.
ThumbFight measures how people liked or disliked search terms based on the last one hundred tweets for the term. The results can be shown in a head-to-head matchup like this one for “Ruby vs. Python”.
To use the ThumbFight API, just upgrade to TwitterLand version 0.3.0: sudo gem install twitterland
My favorite of all the new Google Reader features released this week. Now catching up on older posts is not quite all-or-nothing.
Working on several API wrappers lately, one of my favorite gems is Mash from Michael Bleigh. Much like OpenStruct, Mash is basically a Hash with dot notation.
But unlike OpenStruct, you can pass a Hash to Mash and it will recursively do a deep dive and Mashify it. Now you get pretty accessor methods for all those parsed hashes from your JSON API calls:
# Return tweet referencing a URL
results = Twitterland::BackTweets.search('http://squeejee.com', 'OU812')
results.tweets.size
=> 25
results.tweets.first.from_user
=> "euromarianne"
results.items_per_page
=> 25
results.total_results
=> 3301
Mash includes a couple of helpful methods for traversing those deep hashes when you’re not certain if a key has been set:
mash = Mash.new
mash.first_name?
=> false
mash.first_name = 'Wynn'
mash.first_name?
=> true
The ! method provides multi-level assignment, kind of like mkdir -p for hashes:
mash = Mash.new
mash.these_droids!.will_do_nicely = true
sudo gem install mash
— posted by Wynn Netherland // @pengwynn
Today we received an email from Google AdSense letting us know they had disabled our AdSense account. The only explanation we got was that we were a “Significant risk to their AdWords advertisers”. Check out the full email.
Hello,
While going through our records recently, we found that your AdSense
account has posed a significant risk to our AdWords advertisers. Since
keeping your account in our publisher network may financially damage our
advertisers in the future, we’ve decided to disable your account.
Please understand that we consider this a necessary step to protect the
interests of both our advertisers and our other AdSense publishers. We
realize the inconvenience this may cause you, and we thank you in advance
for your understanding and cooperation.
If you have any questions about your account or the actions we’ve taken,
please do not reply to this email. You can find more information by
visiting
https://www.google.com/adsense/support/bin/answer.py?answer=57153.
Sincerely,
The Google AdSense Team
Google’s email is cryptic to say the least. It would have been nice to at least know the specific violations we commited that put their advertisers at risk. Not only did they cancel our account but they also confiscated our AdSense earnings. I just don’t understand the draconian nature of their actions. If we did something that violated the TOS then tell us. If we knew what we did wrong I promise we would make it right. I did file an appeal and will keep you updated on our communication with Google.
Here at Squeejee we do a lot of data mashups, and more and more of those are mashups of Twitter and 3rd-party services related to Twitter.
There are a bunch of Twitter-related sites out there that have great APIs to pull in their data. Since we’ve been using so many of them repeatedly in our projects we finally decided to pull them into a single gem. TwitterLand will give all you other Twitter app developers out there easy access to these great APIs in your ruby projects.
So far we’ve included:
sudo gem install twitterland
# Get follow cost for specified user
Twitterland::FollowCost.show('bradleyjoyce')
=> <Mash at_reply_index=24.0 average_tweets_per_day=6.87254901960784 average_tweets_per_day_recently=19.6396220282001 golden_index=3.0 milliscobles_all_time=324.02 milliscobles_recently=925.96 political_index=1.0 profile_image_url="http://s3.amazonaws.com/twitter_production/profile_images/179927752/bradley_normal.png" statuses_count=3505 twitter_created_at="2008/03/14 18:26:52 -0700" username="bradleyjoyce">
# request your api key at [http://twitter.grader.com/accessrequestform](http://twitter.grader.com/accessrequestform)
# get twitter grade for user
api_key = "OU812"
Twitterland::TwitterGrader.grade('bradleyjoyce', api_key)
=> 98.4183
api_key = 'OU812'
mt = Twitterland::Mrtweet.new(api_key,'bradleyjoyce')
# Check whether the given user is a MrTweet user.
mt.is_user
=> true
# Returns MrTweet statistics of the given user
mt.profile
=> <Mash conversation=0.225 conversation_percentile=53 frequency=8.98621 frequency_percentile=90 links=0.53 links_percentile=87 recommendations=1>
mt.profile.links
=> 0.53
# Returns the latest recommendations the given user received on MrTweet
mt.recommendations
=> [<Mash date=Thu Aug 06 01:02:54 -0500 2009 name="billtrammel" text="he is an entrepreneur, and one of the developers of TweetCongress.org, award-winning site promoting government transparency. Plus, he's a good friend.">]
mt.recommendations.first.name
=> "billtrammel"
# Returns the twitter_id's of 3 users that for the given user pays the most attention to
mt.most_attenion_towards
=> [15049040, 17993906, 22286046]
# Creates a recommendation from the given user, to another user (aka "friend_name")
reason = "Wynn is an awesome entrepreneur, rubyist, designer and friend! Follow him for his useful and entertaining tweets!"
friend_name = "pengwynn"
mt.recommend(reason,friend_name)
=> true
username = 'bradleyjoyce'
password = 'mypassword'
t = Twitterland::Twinfluence.new(username,password)
=> #<Twitterland::Twinfluence:0x317ecf8 @username="bradleyjoyce", @password="mypassword">
# gets twinfluence data for user
t.user('bradleyjoyce')
t.user('bradleyjoyce')
=> <Mash user=<Mash adrider=<Mash script=[<Mash type="text/javascript">, <Mash src="http://pagead2.googlesyndication.com/pagead/show_ads.js" type="text/javascript">]> centralization="24.224443556856" centralization_grade="0.0 Average - Resilient" description="Entrepreneur and Web Developer -- Floxee.com, TweetCongress.org, Sherflock.com, Squeejee.com" followers_count="1315" friends_count="1278" id="14688076" location=nil name="Bradley Joyce" screen_name="bradleyjoyce" second_order="7936664" second_order_grade="Rank: <b>#10,185</b> (93%)" social_capital="6035.4859315589" social_capital_grade="+1.4 High" statuses_count="3511" unix_timestamp="1249683954" url="http://bradleyjoyce.me" velocity="15535.453112211" velocity_grade="+0.1 Fast Average">>
tc = Twitterland::TwitterCounter.show('bradleyjoyce')
tc.rank
=> 37194
# available methods
tomorrow_2w
followers_2w_ago
followers_yesterday
followers_current
friends_current
next_month
growth_since_2w
started_followers
rank
user_id
growth_since
follow_days
tomorrow
next_month_2w
average_growth
average_growth_2w
# get grade for user
result = Twitterland::TweetBlocker.user('bradleyjoyce')
=> <Mash grade="a" score=100 url="http://twitter.com/bradleyjoyce" username="Bradley Joyce">
# report a user as spammer
Twitterland::TweetBlocker.report_spam('spamtest')
# check your rate limit status
Twitterland::TweetBlocker.rate_limit
=> <Mash hourly_limit=100 remaining_hits=100 reset_time="2009-08-11 23:12:41 UTC" reset_time_in_seconds=2746>
# Return tweet referencing a URL
results = Twitterland::BackTweets.search('http://squeejee.com', 'OU812')
results.tweets.size
=> 25
results.tweets.first.from_user
=> "euromarianne"
results.items_per_page
=> 25
results.total_results
=> 3301
http://github.com/squeejee/twitterland/
http://rdoc.info/projects/squeejee/twitterland
Copyright (c) 2009 Squeejee. See LICENSE for details.
The Ruby BestBuy Remix API gem. Remix is an API that gives you access to BestBuy.com’s product catalog data and more. Remixr is a Ruby wrapper for the Remix API.
sudo gem install remixr
Remixr.api_key = 'OU812' # get yours from http://remix.bestbuy.com/apps/register
client = Remixr::Client.new
# find stores within 50 miles of ZIP 76227
stores = client.stores({:area => ['76227', 50]}).fetch.stores
stores.first
=> {"city"=>"Denton", "longName"=>"Best Buy - Denton", "name"=>"Denton", "region"=>"TX", "address"=>"1800 S Loop 288, Ste 102 Bldg 1 ", "country"=>"US", "lng"=>-97.10067, "postalCode"=>"76205", "phone"=>"940-384-9581", "hours"=>"Mon: 10-9; Tue: 10-9; Wed: 10-9; Thurs: 10-9; Fri: 10-10; Sat: 10-10; Sun: 11-8", "storeId"=>827, "fullPostalCode"=>"76205", "lat"=>33.192524, "distance"=>9.79}
# fetch first page of products on sale below 20 bucks
products = client.products({:salePrice => {'$lt' => 20.00}}).fetch.products
# fetch only SKU and salePrice
products = client.products({:salePrice => {'$lt' => 20.00}}).fetch.products
You can also chain stores and products to return stores and nested product info or vice versa
# find stores within 50 miles of ZIP 76227 and products over three G's
stores = client.stores({:area => ['76227', 50]}).products({:salePrice => {'$gt' => 3000}}).fetch.stores
stores.first.products.first.shortDescription
#=> "ENERGY STAR Qualified 4 HDMI inputs; gray Touch of Color bezel; 16:9 aspect ratio"
All calls terminate in a call to fetch which takes the following options
:page - positive integer for page number
:show - comma delimited string or array of field names to show
:sort - hash or string of sort info {'fieldname' => 'asc|desc'}
We took a page out of MongoDB’s playbook and mapped conditional operators to text equivalents to avoid having these be keys to hashes:
{:salePrice => {'$gte' => 300.00}}
=> salePrice > 300.00
$gte - greater than or equal to : field > value
$lte - less than or equal to : field > value
$gt - greater than : field > value
$lt - less than : field > value
$ne - not equal to : field != value
More in the examples folder:
http://github.com/squeejee/remixr/tree/master/examples
http://rdoc.info/projects/squeejee/remixr
Copyright (c) 2009 Squeejee. See LICENSE for details.
As we’ve previously mentioned, at Squeejee, we’re big fans of MongoDB. We’re excited that some very smart people are starting to take notice of Mongo, too.
In this first post of what we hope will be a helpful introductory series, we’ll cover the use case of using a document store like MongoDB.
It’s rare that we are presented with a new project from a prospective client these days that doesn’t pull data from some third-party API. Twitter seems to be the most popular choice of late, perhaps because it adds value both as a content provider and a transportation medium. One of the challenges of integrating with Twitter in any large-scale way is creating your own local cache of Twitter data so you can ration those precious API calls and avoid their API rate limiting. When we built our own mashup, Tweet Congress we wanted to integrate with a number of third-party Twitter services that each pulled data from Twitter, analyzed it, and passed that value on via their own APIs. The common thread was that each of these services exposed data as JSON (JavaScript Object Notation). Taking these deep, rich data structures and breaking them out into relational database tables seemed silly (not to mention tedious). With the relational approach, if a service exposed a new value that we wanted to consume without fighting our framework, we’d have to extend the schema on our end to capture it.
MongoDB is a high-performance, open source, schema-free document-oriented database. This means we stash those deep data structures we get back from those nifty web services more easily. MongoDB uses BSON to store your documents (or objects). This BSON smells less like a buffalo and more like JSON, which you already know. The main difference is that BSON can serialize binary data (images, and even Word documents, oh the irony!) and dates. This schema-less storage means your app can store all that new data TwitterCounter returns you tomorrow without needing to change your database.
Certainly CouchDB (and many other tools that predate it), does document storage and does it well. We are still big CouchDB fans, however we found that querying CouchDB dynamically was a lot like getting a question answered at the DMV. CouchDB shines when you know how you’re going to query the data up-front, in static views of schema-less data.
MongoDB provides all that schema-less goodness without giving up any of the rich dynamic querying you get with a relational database. More on this in our next post.
Let’s talk specifically about Twitter for a moment. The Twitter API has methods to retrieve the social graph for a particular user, returning an array of integer user IDs for the user’s friends or followers. With a relational database, you’d normally insert a row for each of those values. So a user with a thousand friends would get a thousand database rows, each with something like user_id and friend_id (and id if you’re using some Rails magic). With MongoDB, we can just stash that array with the object itself — and query it: :conditions => {:friend_ids => current_user.id}
Another benefit of all this schema-less goodness is your application can define fields at run-time. On our Floxee platform, we support different custom fields on a per-site basis. Due to the magic of MongoDB, we can have custom fields for party, state, and district for all the congressfolk in Tweet Congress and fields like skills and soda_preference on our own Squeejee twitter directory. Just upload your CSV with some custom columns and we’ll create those fields on-the-fly (after sanitizing them of course).
In our upcoming posts we’ll cover Ruby bindings, performance, deployment, and more. Stay tuned!
One of the things I love about TextMate is that it’s not an IDE, and it feels lightning fast. That said, sometimes I forget to use some of the powerful features in TM. Here’s one I decided to start using:
Cmd-F2 to toggle a bookmark
F2 to navigate to the next bookmark
Shift-F2 to navigate to the previous bookmark
How many times have you scrolled back and forth between two (or more) chunks of code using the mouse wheel, Cmd-F, Cmd-G, page up, page down, or the dreaded scrollbar drag which makes your “mouse mileage” go up?
Just use F2 and you can quickly flip between your bookmarks. Note: You can also use the mouse to toggle the “*” that appears on your bookmarked lines of code.
Pro Tip: On a Mac laptop, consider changing this setting: System Preferences -> Keyboard & Mouse -> Use all F1, F2, etc as standard function keys to “checked”. That way you won’t have to hit the “fn” key to use the magical F2!
Carpal tunnel syndrome be damned!!
