So, you want to hire a Ruby on Rails developer.
You and everyone else in the Valley! Rails has been amongst the "it" technologies for web application development for some time now, and although it's got some tough competition coming from the likes of Node.js and Meteor, there's no denying its iterative power and robust community.
That being said, it's not the best framework for every web project out there. So the first question you should ask is...
Do I really need a Rails developer?
The best way to answer this question is to determine what you want your application to do. Do you need to push real time updates to the client, a la Trello? Do you just need to build an API for a mobile app or some front-end framework such as Backbone, Knockout, Angular, or Ember? Or, do you just need a blog or e-commerce site with little innovative application logic?
If the answer to any of these questions is yes, I would reconsider your choice of hiring a Rails developer for the job.
In the first case, you would be better off using Node.js or a similar socket server technology. These are built with data streaming and live client side updating in mind. While Rails can make this happen, the framework is not designed around this use case, and you have to be very careful which gems (Ruby's version of packages) you use.
In the final case, you don't need a web application, you need a Wordpress site. You'll get a much better deal if you just pay a PHP or Wordpress developer to bang out the minimal functionality and move on.
Now that you're sure you need a Rails developer, here are some litmus tests to make sure you're getting the right person for the job.
One of the most important marks of a great rails engineer is their ability to wield the Ruby language effectively. It's got a lot of power and magic under the hood, and this can be a good or bad thing. One line of Ruby can be equivalent to 100 lines of Java or C++.
What's the easiest way to determine Ruby chops as a semi or non-technical person? When I got hired for my first Rails job, they encouraged me to go through the Ruby Koans. The Koans walk you through the entirety of Ruby in a succinct way. Most Ruby developers I know have gone through the Koans already; all you have to do is ask them to push up their Koans repository to their Github and send you a screenshot of the final success image. This is the easiest way to see if they've at least explored the Ruby language thoroughly, and if they can do this, they're in good shape.
With dynamic languages such as Ruby and Python, tests are the best way to know that as you keep building your application, all the moving parts stay in working order.
Whether they drive their development by tests (a methodology affectionately dubbed TDD), or they write them once they've got a piece of functionality where they want it to be, having a solid grasp on what it means to develop a thorough test suite is key. Ask them about their testing methodology. If they don't have a clear answer, this is a huge red flag.
Gems are the way Ruby and Rails developers share packages of functionality that others might find useful. Gems are one of the most powerful parts of the Rails ecosystem, and any Rails developer worth their salary has a bundle of gems they keep in their back pocket to simplify their work.
An easy litmus test here is to ask them how they would go about developing user authentication in an app. If their answer doesn't include something about a gem, specifically Devise or Omniauth, I would ask them their rationale behind building their own authentication. I can't think of a Rails programmer in my network that builds their own authentication system, and I think there's a good reason for that.
An Open Source Legacy
What would be great is if the dev you're hiring has actually contributed to gems or built their own. If they have, you know you've got someone special. Hire them quickly because they have plenty of other exciting opportunities and won't be free for much longer.
One of the more difficult parts of being a Rails engineer is actually getting your application deployed and serving many users at a time. The deployment process has come a long way since the early days of the framework, but it's still tough to do.
If the engineer has deployed a Rails application to a server that they own, then they've got the chops they need.You don't need to worry about these folks.
If they've only deployed to Heroku that might be fine, too. Are you okay with being locked into that ecosystem? If so, then you can move on.
Do you need some more intense infrastructure? Are you required, for whatever reason, to use AWS, or Rackspace? If so, make sure your engineer understands this. It's okay that they don't have previous experience working with these platforms, but ask them to deploy a "Hello World" app to whichever platform you need to ensure they have the stuff to make it happen.
Deploying early and often is key in developing web apps. Make sure your developer can do so on the platform of your choice, and you're golden.
General Web App Chops
I imagine that many of you don't, though. In this case, it's important that your prospective hire knows these enough to get the job done. Ask them about their experience developing front ends for web apps. Ask them to show some of their favorite interfaces they've built in the past. Ask them if they enjoy this side of the work. If you need them to build your front end and they resent having to touch HTML, you're probably going to have a bad time.
It's all about the portfolio, baby. All this stuff aside, if the person you're hiring has an incredible portfolio of applications they've banged out in the past, that's the best litmus test of all, especially if they've built something with some overlap in functionality with what you need.
Ask them about the projects of which they are most proud. You don't need to see all their work, just their best work. Make sure that what you're asking for is similarly interesting to them and not beyond what they've shown themselves to be capable of in the past, and you're in for a treat. If you can provide a Rails engineer, or any engineer, really, with an awesome problem to solve, they will likely blow your mind with the quality of their work and pace at which they work.
If I was on-boarding a Rails engineer, this would be the steps that, as a Rails engineer myself, I would take to ensure I was getting the best bang for my buck.