Quick thoughts on Twitter’s use of Scala over Ruby

A little over a week ago an article appeared in The Register called Twitter jilts Ruby for Scala that was high on sensationalism and pretty short on substance. Basically all you could get out of the article is that Twitter is moving their middle-tier from Ruby to Scala because their Ruby implementation wasn’t performant.

The article caused quite the kurfuffle in the Ruby community, but over the last week much more has come to light regarding the switch.  Last night, Dave Thomas responded to the commotion with the, well, pragmatic response.  The Ruby community spent way too much time villifying Twitter without a whole lot of information.  Since then the Twitter folks have been very open with their rational for the switch.  Check out the comments to this post at Unlimited Novelty for some great insight from the Twitter developers.

I still wonder though whether or not Ruby could work for Twitter’s middle tier.

The first thing that bothers me is that it doesn’t appear that they tried to run their middle tier on JRuby at all.  One of the reasons for the switch to Scala mentioned in the Register article is that it runs on the JVM with it’s much more advanced garbage collection, threading, etc.  So why would you skip over JRuby when that’s exactly what it does?  If you take a look at The Great Ruby Shootout (December 2008) you’ll see that JRuby is over 3 times as fast as the Ruby 1.8.6 VM.  JRuby has gotten faster since then.  Also, the Ruby 1.9.1 VM is faster still.  Now, it might have been a little harder to get Twitter’s message queue working in 1.9.1, but I doubt it would have taken as long as implementing it in a whole new language with all the baggage of developing for Java.

The second thing that bothers me comes to light in Obie Fernandez’s post My Reasoned Response About Scala at Twitter.  The fact that Twitter had to basically develop a type system for Ruby is a big code smell to me.  I think it points to something fundamentally wrong with the way they built their system.  I can’t point to any clear-cut example, but it seems a little fishy that they would need to do that.

Finally, the third thing that bothers me, and really this piggy-backs on the second, is that the component that was originally migrated from Ruby to Scala was originally written in a day in a half.  I’m sure it was modified after that, but that seems to be a very short time for what turned out to be a critical piece of the Twitter infrastructure.  I may be wrong as I haven’t written a message queue, but the fact that there are many long lived open source message queue projects suggest to me that it isn’t so simple of a task.

I’d really love to know more of the details from Twitter’s side of the house.  They allude to the possibility of releasing a whitepaper in the comments to the Unlimited Novelty post.  That would be great for everyone involved.  After all, Twitter is probably the largest and highest volume Ruby/Rails and Scala app on the planet.  They have to have some very interesting insights into a lot of things.  It would be fascinating reading.

However, the root of the matter for me is that we, as developers, engineers, technologists, etc. are naturally curious and we go out of our way to grow and learn.  A member of the Twitter team re-wrote their messaging queue in his spare time, and you know what?  It worked better than what they had!  So they adopted it, and it’s presumably been a good experience for them and I imagine they’ve all grown as a result of the process.

Could Ruby have been made to work for Twitter?  Maybe, maybe not.  But they got curious, went out and tried something, and found a better (at least for them) solution.  How can we fault that?