Nick Gauthier

Rails Systems Tests with Headless Chrome on Windows Bash (WSL)

2017-09-08T00:00:00+00:00

I’ve been using the Windows Subsystem for Linux for a few months now, but one thing I hadn’t figured out was how to run selenium tests from it. WSL doesn’t support GUI applications, and Chrome is especially difficult to get working. So even thought I want to use chromedriver with the new headless option, Chrome still doesn’t work properly in WSL out of the box.

Thanks to a tweet from Jessie Frazelle in which she mentioned that you can run Windows binaries from WSL, I had the idea to give this another shot: have my Rails tests in WSL call out to Windows chromedriver.exe to drive a headless browser using Windows Chrome. Turns out it works! Here’s how to do it.

Start with a blog of course

So we need some kind of Rails app. I’m going to show you how to set it up so that if you just want to tinker you can try it out quickly.

$ rails new blog
$ cd blog
$ rails generate system_test hello
$ rails db:migrate
$ rails test:system

It should show no tests to run. Let’s write one to try it out. Edit test/system/hellos_test.rb:

require "application_system_test_case"

class HellosTest < ApplicationSystemTestCase
  test "visiting the index" do
    visit '/'
    assert page.has_content?('oh hai')
  end
end

If you run this, it won’t work, because Selenium can’t find the chromedriver:

$ rails test:system
Run options: --seed 9949

# Running:

E

Error:
HellosTest#test_visiting_the_index:
Selenium::WebDriver::Error::WebDriverError:  Unable to find chromedriver. Please download the server from http://chromedriver.storage.googleapis.com/index.html and place it somewhere on your PATH. More info at https://github.com/SeleniumHQ/selenium/wiki/Chr
omeDriver.

    test/system/hellos_test.rb:5:in `block in <class:HellosTest>'

Error:
HellosTest#test_visiting_the_index:
Selenium::WebDriver::Error::WebDriverError:  Unable to find chromedriver. Please download the server from http://chromedriver.storage.googleapis.com/index.html and place it somewhere on your PATH. More info at https://github.com/SeleniumHQ/selenium/wiki/Chr
omeDriver.




bin/rails test test/system/hellos_test.rb:4



Finished in 0.749730s, 1.3338 runs/s, 0.0000 assertions/s.
1 runs, 0 assertions, 0 failures, 1 errors, 0 skips

Install chromedriver

Go download chromedriver from Google. Put it somewhere you plan on adding to your path. Like Documents\bin.

Now, add that directory to your path:

Hit the Windows key and type environment
Open the “Edit the system environment variables” app
Click “Environment Variables”.
Click “Path”
Click “Edit”
Click “New”
Click “Browse”
Browse to your directory you’re going to use (I used Documents\bin)
Click OK, OK, OK

Now relaunch your terminal to pick up the new environment variables. You can echo $PATH to ensure it’s on there. You should be able to run:

$ chromedriver.exe -v
ChromeDriver 2.32.498550 (9dec58e66c31bcc53a9ce3c7226f0c1c5810906a)

Finally, we need to do one of two things:

Rename chromedriver.exe to chromedriver
Make a chromedriver bash wrapper inside WSL

This is because Selenium looks for a chromedriver binary, not chromedriver.exe. Renaming is the easiest, that’s what I did.

Run it and go headless

If you run your tests now, it should pop open a Windows Chrome window and fail the test (because we didn’t actually write code to pass it).

$ rails test:system
/home/nick/.gems/gems/railties-5.1.3/lib/rails/app_loader.rb:40: warning: Insecure world writable dir /mnt/c/Users/Nick/Documents/GitHub/blog in PATH, mode 040777
Run options: --seed 5589

# Running:

Puma starting in single mode...
* Version 3.10.0 (ruby 2.4.1-p111), codename: Russell's Teapot
* Min threads: 0, max threads: 1
* Environment: test
* Listening on tcp://0.0.0.0:50511
Use Ctrl-C to stop
2017-09-08 09:44:45 -0400: Rack app error handling request { GET / }
#<ActionController::RoutingError: No route matches [GET] "/">
/home/nick/.gems/gems/actionpack-5.1.3/lib/action_dispatch/middleware/debug_exceptions.rb:63:in `call'
/home/nick/.gems/gems/actionpack-5.1.3/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/home/nick/.gems/gems/railties-5.1.3/lib/rails/rack/logger.rb:36:in `call_app'
/home/nick/.gems/gems/railties-5.1.3/lib/rails/rack/logger.rb:24:in `block in call'
/home/nick/.gems/gems/activesupport-5.1.3/lib/active_support/tagged_logging.rb:69:in `block in tagged'
/home/nick/.gems/gems/activesupport-5.1.3/lib/active_support/tagged_logging.rb:26:in `tagged'
/home/nick/.gems/gems/activesupport-5.1.3/lib/active_support/tagged_logging.rb:69:in `tagged'
/home/nick/.gems/gems/railties-5.1.3/lib/rails/rack/logger.rb:24:in `call'
/home/nick/.gems/gems/actionpack-5.1.3/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/home/nick/.gems/gems/actionpack-5.1.3/lib/action_dispatch/middleware/request_id.rb:25:in `call'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
/home/nick/.gems/gems/activesupport-5.1.3/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
/home/nick/.gems/gems/actionpack-5.1.3/lib/action_dispatch/middleware/executor.rb:12:in `call'
/home/nick/.gems/gems/actionpack-5.1.3/lib/action_dispatch/middleware/static.rb:125:in `call'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
/home/nick/.gems/gems/railties-5.1.3/lib/rails/engine.rb:522:in `call'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'
/home/nick/.gems/gems/rack-2.0.3/lib/rack/builder.rb:153:in `call'
/home/nick/.gems/gems/capybara-2.15.1/lib/capybara/server.rb:44:in `call'
/home/nick/.gems/gems/puma-3.10.0/lib/puma/configuration.rb:225:in `call'
/home/nick/.gems/gems/puma-3.10.0/lib/puma/server.rb:605:in `handle_request'
/home/nick/.gems/gems/puma-3.10.0/lib/puma/server.rb:437:in `process_client'
/home/nick/.gems/gems/puma-3.10.0/lib/puma/server.rb:301:in `block in run'
/home/nick/.gems/gems/puma-3.10.0/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
[Screenshot]: tmp/screenshots/failures_test_visiting_the_index.png

E

Error:
HellosTest#test_visiting_the_index:
ActionController::RoutingError: No route matches [GET] "/"



bin/rails test test/system/hellos_test.rb:4



Finished in 6.261807s, 0.1597 runs/s, 0.0000 assertions/s.
1 runs, 0 assertions, 0 failures, 1 errors, 0 skips
/home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `waitpid2': Invalid argument (Errno::EINVAL)
        from /home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:138:in `process_exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:123:in `stop_process'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `ensure in stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/chrome/driver.rb:69:in `quit'
        from /home/nick/.gems/gems/capybara-2.15.1/lib/capybara/selenium/driver.rb:276:in `quit'
        from /home/nick/.gems/gems/capybara-2.15.1/lib/capybara/selenium/driver.rb:32:in `block in browser'
/home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `waitpid2': Invalid argument (Errno::EINVAL)
        from /home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:138:in `process_exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:123:in `stop_process'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `ensure in stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:69:in `block in start'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/platform.rb:138:in `block in exit_hook'

Also you’ll notice there’s an error at the end with waitpid2. I don’t know how to fix that. But it does run our test so I’m happy enough for now.

Let’s pass the test. Edit config/routes.rb:

Rails.application.routes.draw do
  root to: 'application#index'
end

Edit app/views/application/index.html.erb:

oh hai

And run. It should pass now.

Following this post from thoughtbot edit test/application_system_test_case.rb:

require "test_helper"

class ApplicationSystemTestCase < ActionDispatch::SystemTestCase
  driven_by :headless_chrome, screen_size: [1400, 1400]
end


Capybara.register_driver :headless_chrome do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    chromeOptions: { args: %w(headless disable-gpu) }
  )

  Capybara::Selenium::Driver.new app,
    browser: :chrome,
    desired_capabilities: capabilities
end

And run it, and it’s headless! Super cool.

$ rails test:system
/home/nick/.gems/gems/railties-5.1.3/lib/rails/app_loader.rb:40: warning: Insecure world writable dir /mnt/c/Users/Nick/Documents/GitHub/blog in PATH, mode 040777
Run options: --seed 28393

# Running:

Puma starting in single mode...
* Version 3.10.0 (ruby 2.4.1-p111), codename: Russell's Teapot
* Min threads: 0, max threads: 1
* Environment: test
* Listening on tcp://0.0.0.0:50621
Use Ctrl-C to stop
.

Finished in 4.973954s, 0.2010 runs/s, 0.2010 assertions/s.
1 runs, 1 assertions, 0 failures, 0 errors, 0 skips
/home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `waitpid2': Invalid argument (Errno::EINVAL)
        from /home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:138:in `process_exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:123:in `stop_process'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `ensure in stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/chrome/driver.rb:69:in `quit'
        from /home/nick/.gems/gems/capybara-2.15.1/lib/capybara/selenium/driver.rb:276:in `quit'
        from /home/nick/.gems/gems/capybara-2.15.1/lib/capybara/selenium/driver.rb:32:in `block in browser'
/home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `waitpid2': Invalid argument (Errno::EINVAL)
        from /home/nick/.gems/gems/childprocess-0.7.1/lib/childprocess/unix/process.rb:32:in `exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:138:in `process_exited?'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:123:in `stop_process'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `ensure in stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:83:in `stop'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/service.rb:69:in `block in start'
        from /home/nick/.gems/gems/selenium-webdriver-3.5.2/lib/selenium/webdriver/common/platform.rb:138:in `block in exit_hook'

Elm on Rails with Webpack

2017-08-30T00:00:00+00:00

Continuing from part 1: Preact on Rails with Webpack, today we’re going to look at how to set up Elm in front of Rails using Rails 5.1’s new webpack scaffolding.

Elm is a very interesting language to me for a couple of reasons. First, I’ve been doing Go for a few years now, and I’ve found its simplicity (one way to do things), static type system, and auto formatting to be very relieving. As a programmer, I have not had to think as hard about the details of solving a problem, and instead I can just go right to solving it. Elm seems to share some of these ideas too.

Elm has a formatter, like Go, so code always looks the same (and you don’t have to worry about typing it perfectly). It has a static type system that makes writing and refactoring code easier because it surfaces errors faster (at compile time) letting you skip right to behavioral errors in your code (through tests of course!) Elm also promises zero runtime errors, which sounds incredible!

In my opinion, the tradeoff Elm asks in exchange for these features is that it’s an additional language to put on your stack, which comes with it’s own overhead of learning the language and maintaining the tooling as part of your build process. Additionally, Elm takes additional effort to integrate with existing JavaScript through its ports system. The ports system makes sense to me, as creating a safe barrier into the “error-free” world of Elm. But it means we’re going to have some overhead.

Luckily, while Elm has its own binary tooling, they’ve made it distributable through npm and buildable through webpack, meaning that you can pretend like it’s another JavaScript framework (I mean, I guess it kind of is!) On top of that, it is a first-class citizen in the new Rails 5.1 webpack scaffolding, meaning Rails can set up Elm for us.

Let’s get started.

Prerequisites

Similar to Part 1, you’ll need this before we continue:

Ruby >2.2.2 (I’m using 2.4.0)
Rails >5.1 (I’m using 5.1.4.rc1)
NodeJS >6 (I’m using 6.11.2)
Yarn >0.20.1 (I’m using 0.27.5)

Creating a new Rails App with Webpack and Elm

Slightly different from part 1, we can use Rails’s Elm webpack scaffold to give us a jumpstart:

rails new hello \
  --skip-action-cable \
  --skip-sprockets \
  --skip-coffee \
  --skip-javascript \
  --skip-turbolinks \
  --webpack=elm

Note the last line, where instead of just --webpack to setup the webpack pipeline, we can say --webpack=elm and Rails will add Elm as a dependency and set it up with webpack to detect and package Elm files.

Using Webpacker @ master

Heads up! There’s currently a bug with automatic compilation so we’re going to use Webpack @ master. Once a release > 2.0.0 of rails/webpacker is released, you should be able to use that instead of this step.

Edit our Gemfile, and modify the webpacker line to use master from GitHub:

gem 'webpacker', github: 'rails/webpacker'

Then update it:

bundle update webpacker

As of the writing of this post, I’m on master@691389f.

Finally, reinstall webpacker:

rails webpacker:install

(you can answer a to allow all updates)

Hello Elm

Since Webpacker sets Elm up for us automatically, all we have to do is edit app/views/layouts/application.html.erb and remove the stylesheet tag and replace it with a webpack tag:

<!DOCTYPE html>
<html>
  <head>
    <title>Hello</title>
    <%= csrf_meta_tags %>

    <%= javascript_pack_tag 'hello_elm' %>
  </head>

  <body>
    <%= yield %>
  </body>
</html>

The Elm Webpacker install automatically creates a hello_elm pack for us to start with as a “hello world” example. In order to see this example though, we need a page in our app. We’re going to be lazy and just add a view to the application controller:

In config/routes.rb:

Rails.application.routes.draw do
  root to: 'application#index'
end

$ mkdir app/views/application
$ touch app/views/application/index.html.erb

Now run rails server open up your app, and it should say “Hello Elm!”. Pretty easy, huh!

Trimming the Fat

Webpack is great because it sets up everything we need to get started, but when using Elm it brings in a lot more than we need. First, we can start by removing old asset pipeline paths:

rm -rf app/assets lib/assets

Next, let’s open up package.json:

{
  "name": "hello",
  "private": true,
  "dependencies": {
    "autoprefixer": "^7.1.3",
    "babel-core": "^6.26.0",
    "babel-loader": "7.x",
    "babel-plugin-syntax-dynamic-import": "^6.18.0",
    "babel-plugin-transform-class-properties": "^6.24.1",
    "babel-plugin-transform-object-rest-spread": "^6.26.0",
    "babel-polyfill": "^6.26.0",
    "babel-preset-env": "^1.6.0",
    "coffee-loader": "^0.8.0",
    "coffee-script": "^1.12.7",
    "compression-webpack-plugin": "^1.0.0",
    "css-loader": "^0.28.5",
    "elm": "^0.18.0",
    "elm-webpack-loader": "^4.3.1",
    "extract-text-webpack-plugin": "^3.0.0",
    "file-loader": "^0.11.2",
    "glob": "^7.1.2",
    "js-yaml": "^3.9.1",
    "node-sass": "^4.5.3",
    "path-complete-extname": "^0.1.0",
    "postcss-cssnext": "^3.0.2",
    "postcss-loader": "^2.0.6",
    "postcss-smart-import": "^0.7.5",
    "precss": "^2.0.0",
    "rails-erb-loader": "^5.2.1",
    "resolve-url-loader": "^2.1.0",
    "sass-loader": "^6.0.6",
    "style-loader": "^0.18.2",
    "webpack": "^3.5.5",
    "webpack-manifest-plugin": "^1.3.1",
    "webpack-merge": "^4.1.0"
  },
  "devDependencies": {
    "elm-hot-loader": "^0.5.4",
    "webpack-dev-server": "^2.7.1"
  }
}

Because we’re not going to use any JavaScript except for some vanilla Elm bindings, we can remove a few packages:

yarn remove \
  coffee-loader \
  coffee-script \
  rails-erb-loader

And we can also remove Webpack’s loaders:

rm \
  config/webpack/loaders/coffee.js \
  config/webpack/loaders/erb.js

And we can also remove the following extensions from config/webpacker.yml:

.coffee
.erb
.jsx
.ts
.vue

Wrapping Up

Similar to part 1, I wanted to look at Elm’s page weight. I was sure it would be higher than Preact because we’re adding an entire language, but I was pleasantly surprised. To see production weight, compile the pack in production mode:

RAILS_ENV=production ./bin/webpack

Then run the server again:

rails server

Now our hello-elm JavaScript file weighs just 20KB. This was definitely less than I expected.

I was also surprised at how easy setting up Elm was. In fact, I was disappointed because I wanted to be challenged to actually write hello world in Elm on my own. So, I guess that means I’ll have to write a follow up post!

Preact on Rails with Webpack

2017-08-29T00:00:00+00:00

I needed to brush up my front-end skills, since I’ve been in Gopherland for a couple years now, so I decided to write a few tutorials about using Rails with modern front-end frameworks. I’m learning this as I go, so please feel free to comment or message me on Twitter if you have some tips about the setup.

In this article, we’re going to walk through how to set up Rails 5.1 with Webpack to use Preact as a front-end framework. Rails 5.1 is the first version of Rails to support Webpack as part of the application scaffolding process, which makes it easy (well … easier) to get started with.

Preact is an alternate implementation of React using native dom features and with a focus on being small and light and lower in dependencies. I like the idea of Preact because it seems more approachable because of its smaller source code. I know I can just dig through it if I encounter something unexpected. I mean, look at its source code (seriously, open that link, that’s Preact without ES6 and JSX).

Prerequisites

In order to run through this tutorial, you’ll need:

Ruby >2.2.2 (I’m using 2.4.0)
Rails >5.1 (I’m using 5.1.4.rc1)
NodeJS >6 (I’m using 6.11.2)
Yarn >0.20.1 (I’m using 0.27.5)

Creating a new Rails App with Webpack

The first thing we need to do is make a new Rails app with Webpack. We’re also going to remove all the things we won’t need, since we’ll be using Preact. Here’s how we set up our application:

rails new hello \
  --skip-action-cable \
  --skip-sprockets \
  --skip-coffee \
  --skip-javascript \
  --skip-turbolinks \
  --webpack

We’re going to skip action cable, sprockets, coffeescript, rails javascript helpers, and turbolinks. And we’re going to turn on Webpack.

Using Webpacker @ master

Edit our Gemfile, and modify the webpacker line to use master from GitHub:

gem 'webpacker', github: 'rails/webpacker'

Then update it:

bundle update webpacker

As of the writing of this post, I’m on master@691389f.

Finally, reinstall webpacker:

rails webpacker:install

(you can answer a to allow all updates)

Trimming the Fat

Webpacker’s install is great because it brings in everything you might need. But, we can trim down a bunch of its dependencies since we will just be using Preact. First off, lets remove some vestigal asset pipeline folders:

rm -rf app/assets/ lib/assets/

Also, edit app/views/layouts/application.html.erb and remove the stylesheet tag and replace it with a webpack tag:

<!DOCTYPE html>
<html>
  <head>
    <title>Hello</title>
    <%= csrf_meta_tags %>

    <%= javascript_pack_tag 'application' %>
  </head>

  <body>
    <%= yield %>
  </body>
</html>

Next, let’s check out our package.json:

{
  "name": "hello",
  "private": true,
  "dependencies": {
    "autoprefixer": "^7.1.3",
    "babel-core": "^6.26.0",
    "babel-loader": "7.x",
    "babel-plugin-syntax-dynamic-import": "^6.18.0",
    "babel-plugin-transform-class-properties": "^6.24.1",
    "babel-plugin-transform-object-rest-spread": "^6.26.0",
    "babel-polyfill": "^6.26.0",
    "babel-preset-env": "^1.6.0",
    "coffee-loader": "^0.8.0",
    "coffee-script": "^1.12.7",
    "compression-webpack-plugin": "^1.0.0",
    "css-loader": "^0.28.5",
    "extract-text-webpack-plugin": "^3.0.0",
    "file-loader": "^0.11.2",
    "glob": "^7.1.2",
    "js-yaml": "^3.9.1",
    "node-sass": "^4.5.3",
    "path-complete-extname": "^0.1.0",
    "postcss-cssnext": "^3.0.2",
    "postcss-loader": "^2.0.6",
    "postcss-smart-import": "^0.7.5",
    "precss": "^2.0.0",
    "rails-erb-loader": "^5.2.1",
    "resolve-url-loader": "^2.1.0",
    "sass-loader": "^6.0.6",
    "style-loader": "^0.18.2",
    "webpack": "^3.5.5",
    "webpack-manifest-plugin": "^1.3.1",
    "webpack-merge": "^4.1.0"
  },
  "devDependencies": {
    "webpack-dev-server": "^2.7.1"
  }
}

Wow, that’s a lot of stuff! We can remove the coffeescript packages since we’ll be using ES6:

yarn remove coffee-loader coffee-script

We can also remove the rails-erb-loader since we won’t be using any erb templates in our javascript:

yarn remove rails-erb-loader

Now we can remove those loaders from webpack:

rm config/webpack/loaders/coffee.js config/webpack/loaders/erb.js

And in config/webpacker.yml we can remove the following extensions:

.coffee
.erb
.ts
.vue

Just to make sure you didn’t break anything, you can run:

./bin/webpack

to bundle up your assets. Mine looks like this:

Hash: 77d3b57fb7806bb9009c
Version: webpack 3.5.5
Time: 262ms
                              Asset      Size  Chunks             Chunk Names
application-8f8bdd9f4ff51391ec46.js   4.47 kB       0  [emitted]  application
                      manifest.json  68 bytes          [emitted]
   [0] ./app/javascript/packs/application.js 515 bytes {0} [built]

Installing Preact

Installing Preact is as easy as:

$ yarn add preact
yarn add v0.27.5
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
success Saved lockfile.
success Saved 1 new dependency.
└─ preact@8.2.5
Done in 3.96s.

Of course, that doesn’t do anything to our app, because we’re not using Preact anywhere yet. In order to try it out, we’re going to need a page to work with in our app. I’m going to be lazy and just add a view to the application controller.

In config/routes.rb:

Rails.application.routes.draw do
  root to: 'application#index'
end

$ mkdir app/views/application
$ touch app/views/application/index.html.erb

Now we can run rails server and it should just be a blank screen, but in the console, we should see:

Hello World from Webpacker

Yay! Just for a sanity check, edit app/javascript/packs/application.js and edit the message and refresh your browser and you should see a new message (because Rails should recompile your assets for you with webpacker).

Additionally, we’re going to use JSX, so we need to install the babel JSX transformer:

yarn add babel-plugin-transform-react-jsx

And in order to map this transformer to Preact’s h method, add this line inside plugins in your .babelrc:

["transform-react-jsx", { "pragma":"h" }]

Preact Hello World

Now it’s time to actually use Preact (finally, whew). Edit app/javascript/packs/application.js and add this line:

import('hello.js')

That’s going to import app/javascript/hello.js (which we are about to write) into your app.

Here’s what we put into app/javascript/hello.js (which is straight from Preact’s tutorial):

import { h, render, Component } from 'preact';

render((
    <div id="foo">
        <span>Hello, world!</span>
        <button onClick={ e => alert("hi!") }>Click Me</button>
    </div>
), document.body);

Now, we should be able to refresh our site, and see the Preact demo. Click the button and it says “Hi!”. We did it.

Wrapping up

One of the things I was interested in is page weight. When running our app in dev mode, I saw two JS files, one is application.js, which contains a bunch of webpack code for dynamically loading all our JS files separately. This is great in development because we only have to rebuild what we change. So that’s why there’s a second “chunk” file also being loaded.

Doing a production asset build is as easy as:

RAILS_ENV=production ./bin/webpack

Then run our dev server again:

rails server

As long as you don’t touch any JS files our production built JS files are used. From here I was able to see that our application.js was just shy of 1kB, and our chunk with hello.js and preact in it are 3.7KB. Great! So we’re under 5KB, which is awesome.

I’m not sure if it’s possible to have just one application.js file (no chunks) so if you know how, let me know!

Linux Workstation PC Build

2016-11-01T00:00:00+00:00

I got great response on Twitter from people who were interested in building a Linux workstation. In this post, we’ll build a ridiculous PC for under $1,000, plus show a couple add-ons you can make to fit your use case.

Prices were noted at the time of the writing of the post, they may have changed!

Base Spec

This is the base version of the PC we’re going to build. You can only buy the parts here and you’ll have an amazing linux machine.

CPU $329

Intel Core i7 6700K 4.00 GHz Unlocked Quad Core Skylake Desktop Processor

This processor is a year old, and Kaby Lake is just coming out soon. However, Kaby Lake is not going to be a big boost over Skylake, it’s just a minor iteration. The Kaby lake version of this chip will be at 4.2 GHz.

But, this is unlocked! That’s the K at the end of the SKU. There are reports of folks overclocking it to 4.8 GHz (with voltage changes). I wouldn’t be surprised if you can push it to 4.5 GHz without changing the voltage. My trusty 3770k is still humming along at 4.0 GHz overclock, and it’s four and a half years old!

Motherboard $149

ASUS Z170-A

I love ASUS motherboards, I’ve been building on them for fifteen years. This is the cream of the crop. It’s got m2 for insane solid state speeds, ddr4 ram, and it can leverage the intel chip’s graphics to drive up to three displays over displayport. It’s got USB3 and gigabit LAN, of course. It also has a USB-C port, neat!

Speaking of displays, this combined with the intel chip above can drive a 4K display over displayport. So you don’t need additional graphics.

Also, it has 5-way optimization, which is an auto-tuning tool. You boot into the bios, tell it to do it’s thing, and it will auto-overclock the whole system!

RAM $85

Corsair Vengeance LPX 16GB (2x8GB) DDR4 DRAM 2133MHz

A pair of 8GB sticks brings us to 16GB. You can get two orders for a total of 32GB if you’d like. I don’t need more than 8GB on linux, but I have 16GB right now because I play games. I think 32GB is excessive, and you can always purchase it later. Or treat yourself!

Power Supply $70

Seasonic M12II 620

I love Seasonic supplies. They are very quiet. This one is also modular, so that keeps the clutter down inside the system. 620W is more than enough, even if you purchase the add-ons below (more graphics).

Case $75

NZXT S340

I love the look of this case. You can pick a different color if you’d like, they have a bunch of great options. NZXT’s build quality is great, and this case has two included 120mm fans, which will be great for cooling and quiet too.

SSD $185

Samsung 950 PRO 256GB M.2

The new M.2 SSDs are crazy. This one packs 2,200MB/s Read and 900MB/s write. Wow. It’s 256GB but the link I posted has a 512GB option that is 2,500MB/s and 1,500MB/s if you love speed.

NOTE: you can only put one of these in the system. Motherboard has only one M.2 slot. You can buy an M.2 PCIe card and RAID them, or buy a PCIe SSD, or buy a SATA SSD if you need more.

Base Spec Summary

OK, that’s it for the base!

Total before taxes: $893

If you’re used to Apple, this price may shock you. Welcome to the real world ;)

Add-ons

Here are two more things you may want to add to your system.

Liquid Cooler $105

NZXT Technologies Kraken X41

I love the new all-in-one liquid coolers. You don’t have to mess with pipes or leaks or anything. They’re entirely self-contained. This NZXT cooler fits on the front of the case (inside it) we picked. That way, air is taken in the front of the case, passes over the radiator, and is exausted via the rear and top fans.

This cooler goes on just like a normal air cooler, but it also attaches at a fan mount point. When you install the chip, just toss the intel fan cooler and use this one’s water block instead.

Coolers like these are more efficient and much much quieter too, even under load.

Graphics Card $230

Sapphire NITRO+ Radeon™ RX 480 4GB

The new RX 480 line from AMD is an amazing value. You can drop any card into this system and it will probably be fine, and if I offered every graphics option this post would be huge.

I use the RX 480 in my home desktop on linux and the compatibility is excellent.

I also play games on it at 2560x1600. It can play Overwatch completely maxxed out at 60fps.

Also, the motherboard we’re using can fit 3 graphics cards on it (but usually there are diminishing returns after 2). So you can buy a second one in a year or two at a nice discount and double up your graphics power.

Totally Maxxed Out

Totally Maxxed Out Total: $1444

(This includes a second pair of RAM, liquid cooler, graphics, and 512GB SSD).

Bonus Section: Peripherals

I really like the Dell P2715Q 27” 4K monitor, a Das Keyboard, a SteelSeries Sensei, and a Logitech C922x webcam.

On linux, a quick xset m 1 1 will kill your mouse acceleration, making you very accurate.

If you want a 144hz gaming monitor with freesync, the Pixio PX277 looks like a great deal. Apparently the stand is flimsy so you may want to buy a separate VESA stand.

Staying Motivated While Working Solo

2016-04-25T00:00:00+00:00

I recently started MeetSpace, a video conferencing app for distributed teams. It’s not the first time I’ve started a product company, nor is it the first time I’ve worked alone. But, it’s the first product I’ve worked on alone. In the past I had either partners or clients I’ve worked with on any project, but this is the first one where I’m the solely responsible person (insert irresponsible coworkers joke here :P ). In this post I’ll outline the ways that I keep myself motivated and working each day.

Accountable Person

One of the very first things I did was get someone to hold me accountable for my work. I’m very fortunate to have Brian Sierakowski of TeamPassword meeting with me weekly to hold me accountable. We also swap notes with what he’s doing over at TeamPassword.

Each week I tell Brian what I got done this week and what I expect to get done next week, and we compare it to what I said I’d get done last week. We also talk at a high level about strategy or sometimes just bat ideas back and forth. Eventually we just end up talking about video games.

It really motivates me to think about our weekly meeting (it’s on Friday morning) during the week. I know I’ll have to tell Brian if I miss a deadline on a task, especially if it’s something that’s not particularly difficult but maybe a chore to do. Having it on Friday morning also makes sure I don’t leave it until the last minute, because I want to have everything done by Thursday, but I can finish it Friday if I have to.

I would highly recommend getting an accountable person to help you out if you’re working solo. You should have a regular meeting with a specific agenda that covers what you did and what you will do. Even if you’re a freelancer and all you track is billable hours, you can simply text a family member and say “I billed 35/30 target hours this week”.

Working Contiguously in Time and Space

It’s very important for my productivity to work contiguously, both in time and space. I need to go to a separate location that is solely a work location, and I need to be there for one long chunk of time. Context switching in and out of work mode doesn’t work at all for me. I have to go to work, work until I am done, and then go home and not work. I have a very binary work brain.

I worked from home for a while when I first started freelancing, but after a few months I got a work space. My friends at Figure53 were kind enough to lease me a desk at their office. Later on, I ended up working out of offices I leased or coworking spaces (I’m currently at Workbar Arlington).

When I worked from home, I was billing hourly to clients, and that kept me very productive and honest with my time. I would work in 30 minute pomodoros modified to be 30 minutes working 5 minutes resting, and I would bill that as 30 minutes. That way, I couldn’t stop working inside a 30 minute chunk or I wouldn’t be able to bill it.

But, once my time was my own, this didn’t work. It was too easy to make stupid coffee videos when I was at home instead of working. So now, I go to an office.

Routine

Having a regular routine and schedule is very helpful because it keeps my expectations set very consistently on how each day is going to go. Every weekday looks like this:

Wake up (6:30am)
Make breakfast (usually coffee and scrambled eggs)
Bike to work (drive if weather sucks)
Work 8am-12pm and drink 32oz water
Lunch 12pm-12:30pm
Work 12:30pm-4:30pm and drink 32oz water, second coffee if I want
Bike/drive home
Exercise
Hang out (video games, movies, reading, cooking) and drink 32oz water
Sleep

I have two 4 hour work blocks with a lunch break in between. There’s not much to do at the office that isn’t working. Because my schedule is really consistent, I get very used to following the same schedule every day. This way there’s less of a nagging “hey let’s go to work late today / come home early” voice because those are the times I always work.

Task-driven

I mentioned that I discuss my tasks with an accountable person, and in a previous post on minimum viable project management I discuss how I manage the tasks on a project. What this really breaks down to is a todo-list (I use Wunderlist) with all of my tasks on it.

Anything that I do goes into the list, and generally those items go onto the list at least the week before I work on them. Each week (after the Friday accountability meeting) I scrub my list. This involves going through every task and seeing if their due dates are all still reasonable.

All tasks have a due date (almost always a Friday). This groups tasks into weeks. I can look at the tasks for a week and if it’s too much I bump some tasks to the next week, and cascade forwards until the roadmap looks good.

I delete any tasks I don’t think are important any more. Sometimes I add something like “ooh, it would be neat if…” If on Friday I don’t think that task is really important, I delete it.

At this point, I have one to two months of roadmap. Anything farther in the future is so poorly defined I just keep it as a thought. By the time I get low on tasks, my favorite thoughts for the product will be at the forefront.

When I am working, I look at Wunderlist’s automatic “Week” list, which shows everything due within 7 days. This is my list for the week, and I work top to bottom, checking things off as I go.

I also make use of recurring todos for marketing. That involves writing inbound content, publishing, republishing, tweeting, emailing, and also customer development type tasks. That way these come back around on a schedule.

One really nice aspect of this system is that I can actually get to a done state. When I finished everything for the week my list is blank. I am actually done. It’s wonderful. I usually will take a Friday afternoon off if I’m done. Life is short.

Varied Schedule

I have a routine, tasks, and recurring tasks. This comes together to form a very regular day and week. But each day is a little different. I like to have different kinds of work scheduled throughout the week so that day-by-day I’m doing different things.

Mondays are for marketing, creating content, following up on campaigns, and customer development in the morning. This is really motivating work because it reminds me of why I’m working on MeetSpace: all the people! Seeing a good piece of feedback or good reaction to a post is very rewarding. Then seeing a bad piece of feedback or bug report is really motivating to do a good job. I can’t sit on my butt if there’s a bug out there!

Monday afternoons I work on bugs and then technical debt. This puts me on a “clean slate”. Tuesday and Wednesday I work on the “big feature” of the week. Thursday is another marketing and content morning. Thursday afternoon is wrapping up features, bugs, and technical debt.

Friday is my accountability meeting, any remaining tasks, task planning for the next week, a product update for users, and any leftover bugs or debt.

Not Naturally Motivated

I’m writing this post to share that there’s really a lot that goes into getting myself to do work each day, week, and month. It’s not something that comes naturally to me. If I am at home I’ll play video games. At work I’ll surf random internet sites or social networks or talk to coworkers. If I don’t have a combination of a time and place to work, tasks due, someone who’s going to ask about them, people who are interested in them, and a regular but varied schedule, I don’t get much done at all.

So if you don’t feel naturally inclined to get lots done every day, don’t worry, you’re not weird. You’re probably pretty normal. I needed a lot of structure before I could really get lots done on MeetSpace.

Minimum Viable Project Management

2016-03-25T00:00:00+00:00

I started my career in software at a consultancy, and over the years I’ve spent more than half my time as a consultant or freelancer. In all this time across many projects, I’ve experimented with many ways of doing software project management. From processes like waterfall, agile, and kanban to tools like pivotal tracker, trello, github issues, and basecamp.

All these processes and tools have the same goal: turn thoughts into structured definition of work.

As the tools have evolved, they have focused on improving that goal through increased specification. By capturing more information on what the work is, how it will be done, and who will do it, they have furthered their goal of creating order from chaos.

My career is built upon my ability to turn creative thoughts into reality. My emotional attachment to each of my thoughts is strong and personal. I derive a lot of satisfaction in filling up a project management tool with all of my ideas.

What starts as a distillation quickly turns into hoarding. It seemed like all the ideas were at the forefront of consciousness, but after a week or two, I don’t recognize some of my tasks any more. Did I even put this in the icebox? How do we have 80 stories in here already?

It’s not long before the icebox becomes unmanageable and we declare bankruptcy. We call our work so far “done” and we make a new project, or we switch tools. And what a rush to start fresh! Now I can really just write down what is important, and none of that other crap. Then a week passes, and then another, and here we are again, mr icebox, my how you’ve grown.

Somehow I lost track of the fact that when I declare bankruptcy and start fresh, all the good ideas are still there. My best ideas are never forgotten. Like a commercial jingle they jump back into my head when I’m at the grocery store, or in the shower, or trying to fall asleep.

Over time, the best ideas grow and evolve through a mental kneading and rising and kneading and rising, until their wispy beginnings gain a firm but supple quality. I don’t notice or miss the other thoughts that came and went, and if I have to, I can go looking and find them again.

Simplify, simplify.

Every piece of information, every fleeting thought and precious idea that is written down and stored becomes a strap tying down what we’re creating, limiting its shape and size within the scope of its bindings. As more and more tasks and stories and requirements are added, the straps pile on and tighten until what’s left isn’t so much our hopes and dreams, but the only remaining shape that can fit within the bounds we’ve created.

I’m ditching my specification-oriented project management style for something more free-form and organic.

I will constrain my ability to constrain.

An entire use case, or feature, or flaw is one short sentence. Everything else is in my head, or in my words with my advisors, customers, and friends. Or on pages in my notebook I know I’ll never flip back to.

What about all the productive discussions with all the people involved? Those sessions are not for “information extraction”, instead, they are explorative and extemporaneous expositions. I may not write everything down but I am certainly shaped by it.

There is no icebox, there is no phase 2, there is no story landfill. Burn your trash, throw your ideas away. The good ones will never leave you.

Everything I am going to do is scheduled and ordered or it is not recorded. When it’s time to get started, I revisit the discussion. A lot may have changed since we last explored the idea. Now we can visit it in the present and move forwards without the constraints of the past limiting our thoughts.

There are 36 one-line tasks between now and the full paid release of MeetSpace, and that leaves me free to focus, work, and think.

Interview Notes

2016-03-21T00:00:00+00:00

I was asked recently to write up how I give interviews, and I realized that it could be very helpful to publish these publicly. As you will understand while reading, knowledge of my interview and its rubrick doesn’t give a potential candidate an edge. In fact, I think it would lead to a more accurate interview. Enjoy, and share your thoughts!

General Format

Part 1: Chat

30 minutes talking. We would talk about previous experience, current interests, what it’s like at the company they are interviewing for. No formal questions like “what vegetable are you?” only casual things. The goal of this phase is to establish our relationship and communication style so that we can speak easily when we get to the technical part. Candidates could be disqualified during this phase, though.

Examples of negative experiences and qualities:

Superiority complex / elitism / asshole: “well, actually”, patronizing me, talking poorly of past experiences or coworkers, bragging and gloating without specific examples (i.e. “I’m pretty much the best developer I’ve ever known” is bad but “I’ve done some really incredible things with postgresql on past projects” is good).
Inability to explain: if I ask about a past project or a language or anything, and the candidate can’t tell me about it at all, or explains very tersely and stops.
Lack of interest: it’s odd but sometimes you get someone who just doesn’t seem to even care about the interview or working for the company or anything.

Examples of positive experiences and qualities:

If I ask about something pretty simple, like “What did you work on at X corp” and they explode into enthusiasm and talk about it on their own with only minimal “ah”, “ok”, “cool!” from me for a while. This person really loves what they do.
Mutual feeling of at-ease when we speak. We’re having an honest conversation, there’s no positioning, choosing words carefully. We are just talking about stuff and establish a connection quickly. It can take until the end of the 30 minutes for this, but that’s ok. As an interviewer, you should set the example of speaking honestly and openly.
Opinions and willingness to share them. Doesn’t really matter if we agree or not, but I want to hear that the candidate has their own opinions on relevant subjects and is excited to discuss them.

Part 2: Code

One hour of coding. I’ve done two projects during this part since 2009. At first, it was always “build a rails blog” since I was only interviewing for Rails (although I had someone do it in Django because that’s what they knew, and they got the job!). Later on, it became “work with some json” because it was more of a general position but web background was part of it.

Here is how the JSON project would go:

I give them some JSON data. I would use some Codeship API data. It had a root object, and one of the values was an array with some computable properties.
I would ask them to give me one of the top level key’s values. At this point the person says “um, how do you want me to code that?” and I will say “you could do a script, or a binary, or a library, it all works for me!”. At this point I also mention that “there are no rules” for this interview.
“OK, before we do the next thing, anything you want to clean up?”
I ask for the number of items in the array value of the object (just count)
“OK, anything you want to clean up or refactor here?”
I ask to compute something on the array values (percentage where some bool is true, for example)
refactor again
I ask to compute something else, but more complicated (often this one is avg(timestamp-timestamp) to get average duration of an event).
refactor again

That usually takes the hour. If not, I will have a couple of harder and harder questions. The point is not how far we get, but how we do it.

At this point, you’re probably thinking “wow that is really easy, how is this effective?”. On the one hand, I am evaluating their ability to do the assignment. I have passed candidates who only were able to do the first task. But usually it’s because the language they were comfortable with had poor json support so we spent the whole time figuring that out. But that’s OK! That’s totally something programmers do all the time, and your ability to do that well is just as important as slamming out easy code.

Here is what I am really looking for when we do this part of the interview:

How is working with this person? I am acting as a “novice pair” so I don’t really know how to do a lot of stuff, but I’m still contributing a little bit. Do I enjoy coding with this person? Would I ask them to pair with me if I was stuck?
What is their attention to detail? How often did they miss little things or leave loose ends lying around? Are they rushing sloppily to get through as much as possible in the interview, or are they really trying to get it right? (Spoiler alert: this is how they will attack projects when they’re on the job too!)
Where is their balance point between productivity and perfection? More specifically, how much do they refactor when prompted? I have had people who just say “It’s great the way it is I wouldn’t change it” when there are obvious duplications. I’ve also had people who went full rabbit-hole on something really unimportant. I try to figure out their balance point. Generally, I will take people who are perfectionists, and in the middle, but not those who don’t see the possibility of improvement. (Spoiler alert: they are often “expert beginners” and they will not improve themselves while on the job. Why should they? They’re already perfect and they write perfect code the first time).
Do they ask for help? I really try to act like a pair, and hopefully from the chat section we are at ease. When they don’t receive enough clarification when I tell them the next part of the project, do they ask for clarification, or do they guess what I want? When they get stuck, will they ask me for help? It’s extremely important that the candidate ask for help during the exercise. Especially if they are more senior. I want to work on a team of people who get the most out of each other to do their best work. I have even had candidates ask me directly for the solution to one of my questions. This is actually a huge point in their favor! I will then politely say that while I do know the answer unfortunately this is an interview and they will need to solve it themselves, but I will usually give them a tip.
Comfort with their tools. I always have the interviewee work in their most comfortable and familiar environment. The language of their choice, the editor of their choice. Always on a computer they brought with them. They should be quick at the keyboard, quick on their editor, quick on the browser searching for help, quick to evaluate which stack overflow post is the most relevant, etc. If you were hiring a carpenter and they struggled with hitting a nail, they don’t hit a lot of nails. Many interviews try to throw a fastball and see if the interviewee can hit it. I set up a T-ball and see how far they can hit.

Boiling it down, these are the qualities I am evaluating:

Communication ability: honesty, frankness, and ease.
Attention to detail, not sloppy, no “messy room syndrome”
Always looking for improvement. Never satisfied. Understanding that perfection is both unnattainable but worth attempting.
Open to assistance from anyone, not afraid to ask for help, open to the thoughts and opinions of others.
Programming is a natural skill for them. They are quick and comfortable and experienced with what they claim as their experience level.

Conclusion

You may have noticed that the actual act of programming is a minority in the qualities I am assessing. That is because I truly believe it is a minority in the qualities of a great team member. You will be spending the majority of your week interacting with the people on your team. Those interactions have to be smooth and productive for you to be able to even get to the programming part of your job. The easier it is to talk to your teammates, debate, come to conclusions, get help, and solve problems, the more time you will spend implementing your solutions. It will also significantly affect your happiness and burnout.

Scraping the Web with Ruby

2014-06-05T00:00:00+00:00

When you run into a site that doesn’t have an API, but you’d like to use the site’s data, sometimes all you can do is scrape it! In this article, we’ll cover using Capybara and PhantomJS along with some standard libraries like CSV, GDBM, and OpenStruct to turn a website’s content into CSV data.

As an example, I’ll be scraping my own site. Keep in mind if you are planning to scrape a site, be aware of their Terms of Service. Many sites don’t allow scraping, so be a good web citizen and respect the wishes of the site’s owners.

Our goal here is to dump out a CSV file of the blog articles here on ngauthier.com that contains the titles, dates, urls, summary, and full body text. Let’s get started!

1. Scraping with Capybara

The first thing we’re going to do is get Capybara running and just dump the fields from the front page out to the screen to make sure everything’s working.

First, we’re going to load up capybara and poltergeist. Poltergeist is a ruby gem that wraps phantomjs so that we can run a real browser on the external pages.

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'

include Capybara::DSL
Capybara.default_driver = :poltergeist

Next up, we’re going to visit my site. Then we’ll iterate over every post and pull out the fields we want using css selectors and dump them to the screen.

visit "http://ngauthier.com/"

all(".posts .post").each do |post|
  title = post.find("h3 a").text
  url   = post.find("h3 a")["href"]
  date  = post.find("h3 small").text
  summary = post.find("p.preview").text

  puts title
  puts url
  puts date
  puts summary
  puts ""
end

When we run it, we can see that our scraper is loading up the homepage and finding the info:

Using Docker to Parallelize Rails Tests
/2013/10/using-docker-to-parallelize-rails-tests.html
2013-10-13
Docker is a new way to containerize services. The primary use so far has been for deploying services in a very thin container. I experimented with using it for Rails Continuous Integration so that I could run tests within a consistent environment, and then I realized that the containers provide excellent encapsulation to allow for parallelization of test suites. There are three things we'll have to do to run a Rails test suite in parallel in docker: Create a Dockerfile for a system that ca...

PostGIS and Rails: A Simple Approach
/2013/08/postgis-and-rails-a-simple-approach.html
2013-08-18
PostGIS is a geospatial extension library for PostgreSQL that allows you to perform a ton of geometric and geographic operations on your data at high speeds. For example: Compute the distance between two points Find all the points within X meters of point P Determine which points are enclosed in polygon P A million other things In Ruby land, there is a gem called RGeo that provides a ton of objects and methods for handling Geospatial objects. In Rails, there are a number of ActiveRecord ...

...

Exporting CSV

Our next goal is to export some CSV. All we’ll do is load up the csv standard library and write csv to standard out.

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'
require 'csv'

include Capybara::DSL
Capybara.default_driver = :poltergeist

visit "http://ngauthier.com/"

CSV do |csv|
  csv << ["Title", "URL", "Date", "Summary"]
  all(".posts .post").each do |post|
    title = post.find("h3 a").text
    url   = post.find("h3 a")["href"]
    date  = post.find("h3 small").text
    summary = post.find("p.preview").text

    csv << [title, url, date, summary]
  end
end

Now, our program is unixy, so we would run it and redirect its output to a csv file of our choosing. Nice!

Full Articles via Multi-Pass

So far, we’ve only pulled the summaries of the posts, but we want the whole content. For this, we’re going to need to do a two-pass scrape. First, we’ll scrape the summaries like we already are doing. Second, we’ll visit each post’s url and grab the post’s body.

To do this, we’re going to keep track of an articles array, and store the articles temporarily as hashes.

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'
require 'csv'

include Capybara::DSL
Capybara.default_driver = :poltergeist

visit "http://ngauthier.com/"

articles = []

# Pass 1: summaries and info
all(".posts .post").each do |post|
  title = post.find("h3 a").text
  url   = post.find("h3 a")["href"]
  date  = post.find("h3 small").text
  summary = post.find("p.preview").text

  articles << {
    title:   title,
    url:     url,
    date:    date,
    summary: summary
  }
end

# Pass 2: full body of article
articles.each do |article|
  visit "http://ngauthier.com#{article[:url]}"
  article[:body] = find("article").text
end

# Output CSV
CSV do |csv|
  csv << ["Title", "URL", "Date", "Summary", "Body"]
  articles.each do |article|
    csv << [
      article[:title],
      article[:url],
      article[:date],
      article[:summary],
      article[:body]
    ]
  end
end

That wasn’t so bad! The main issue now is robustness. My blog is fast (static sites with jekyll woo!) and I only have a couple posts. But imagine the scrape took an hour. If anything crashed, took too long, or the site went down, we’d waste a ton of time.

Handling Interruptions with GDBM

Enter GDBM. GDBM is a standard unix library that is a simple file-based key-value store. It’s like a hash, but you can only use strings as the values. What we’re going to do is replace our article array with a GDBM store. For the key we’ll use the url, and for the value we’ll dump the article to JSON.

Now, if the program crashes, when we resume we want to skip over any articles we’ve already processed. Let’s take a look:

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'
require 'csv'
require 'gdbm'

include Capybara::DSL
Capybara.default_driver = :poltergeist

visit "http://ngauthier.com/"

articles = GDBM.new("articles.db")

We’ve required gdbm and now articles is a GDBM store in a file in the current directory called articles.db. We can now use articles like a hash, and GDBM will sync it to the file system whenever we write to a key. Neat!

# Pass 1: summaries and info
all(".posts .post").each do |post|
  title = post.find("h3 a").text
  url   = post.find("h3 a")["href"]
  date  = post.find("h3 small").text
  summary = post.find("p.preview").text

  next if articles[url]

  articles[url] = JSON.dump(
    title:   title,
    url:     url,
    date:    date,
    summary: summary
  )
end

Now, we have a next if that checks to see if we already have the article. We don’t want to store it if we already have it, because that would overwrite the article (and we may have fetched the body).

When we store it, we JSON.dump our hash so that it’s a string.

# Pass 2: full body of article
articles.each do |url, json|
  article = JSON.load(json)
  next if article["body"]
  visit "http://ngauthier.com#{url}"
  has_content?(article["title"]) or raise "couldn't load #{url}"
  article["body"] = find("article").text
  articles[url] = JSON.dump(article)
end

When we iterate a GDBM store it gives us a key-value pair, like a hash. So, we have to load up the article from JSON before we can work with it. Our keys also become strings instead of symbols, because it was loaded from JSON.

We have another next if that skips visiting the page if we have the body already. Additionally, we check the page for the title. This gives the page time to load, as opposed to accidentally scraping the last page’s content.

Finally, we have to dump the article to JSON to store it again.

# Output CSV
CSV do |csv|
  csv << ["Title", "URL", "Date", "Summary", "Body"]
  articles.each do |url, json|
    article = JSON.load(json)
    csv << [
      article["title"],
      article["url"],
      article["date"],
      article["summary"],
      article["body"]
    ]
  end
end

When we output to CSV, we have to load up the JSON to construct our CSV output.

Now that GDBM is running, the first time I ran the scrape it took 7.3 seconds, and the second time, it took 1.7 seconds because it only hit the index and skipped scraping. I can also ctrl-c the program while it’s working, and re-run it and it will resume. Nice!

Object Oriented

OK at this point the code is pretty ugly and scripty, and some things are getting annoying. Let’s clean up this code.

First Pass: Base Class, Instance Variables and Method Splitting

First off, we’re doing everything in the global scope, and Capybara complains every time we include Capybara::DSL in the global scope because it extends it with methods like find and visit. Convenient, but messy. Let’s create a base class called NickBot and we’ll split up each phase into a method for readability.

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'
require 'csv'
require 'gdbm'

class NickBot
  include Capybara::DSL

  def initialize(io = STDOUT)
    Capybara.default_driver = :poltergeist
    @articles = GDBM.new("ngauthier.db")
    @io = io
  end

  def scrape
    get_summaries
    get_bodies
    output_csv
  end

  def get_summaries
    visit "http://ngauthier.com/"
    all(".posts .post").each do |post|
      title = post.find("h3 a").text
      url   = post.find("h3 a")["href"]
      date  = post.find("h3 small").text
      summary = post.find("p.preview").text

      next if @articles[url]

      @articles[url] = JSON.dump(
        title:   title,
        url:     url,
        date:    date,
        summary: summary
      )
    end
  end

  def get_bodies
    @articles.each do |url, json|
      article = JSON.load(json)
      next if article["body"]
      visit "http://ngauthier.com#{url}"
      has_content?(article["title"]) or raise "couldn't load #{url}"
      article["body"] = find("article").text
      @articles[url] = JSON.dump(article)
    end
  end

  def output_csv
    CSV(@io) do |csv|
      csv << ["Title", "URL", "Date", "Summary", "Body"]
      @articles.each do |url, json|
        article = JSON.load(json)
        csv << [
          article["title"],
          article["url"],
          article["date"],
          article["summary"],
          article["body"]
        ]
      end
    end
  end
end

NickBot.new(STDOUT).scrape

OK, that’s better, but really all we did was push our veggies around our plate. We didn’t eat any of them.

Second Pass: Article Class

Let’s start with an Article that will wrap up parsing a Capybara node and dumping itself automatically. GDBM calls an object’s to_str to dump it, so we can hook in there to dump ourselves to JSON:

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'
require 'csv'
require 'gdbm'

class NickBot
  include Capybara::DSL

  def initialize(io = STDOUT)
    Capybara.default_driver = :poltergeist
    @articles = GDBM.new("ngauthier.db")
    @io = io
  end

  def scrape
    get_summaries
    get_bodies
    output_csv
  end

  def get_summaries
    visit "http://ngauthier.com/"
    all(".posts .post").each do |post|
      article = Article.from_summary(post)
      next if @articles[article.url]
      @articles[article.url] = article
    end
  end

  def get_bodies
    @articles.each do |url, json|
      article = Article.new(JSON.load(json))
      next if article.body
      visit "http://ngauthier.com#{url}"
      has_content?(article.title) or raise "couldn't load #{url}"
      article.body = find("article").text
      @articles[url] = article
    end
  end

  def output_csv
    CSV(@io) do |csv|
      csv << ["Title", "URL", "Date", "Summary", "Body"]
      @articles.each do |url, json|
        article = Article.new(JSON.load(json))
        csv << [
          article.title,
          article.url,
          article.date,
          article.summary,
          article.body
        ]
      end
    end
  end

  class Article < OpenStruct
    def self.from_summary(node)
      new(
        title:   node.find("h3 a").text,
        url:     node.find("h3 a")["href"],
        date:    node.find("h3 small").text,
        summary: node.find("p.preview").text,
      )
    end

    def to_str
      to_h.to_json
    end
  end
end

NickBot.new(STDOUT).scrape

We’re using OpenStruct here to cheaply get a hash-to-object conversion so we can use dot notation to access fields. This makes it feel way more like an object. It also means we could replace the accessors later transparently.

This is better, but we still have a few issues with entanglement between NickBot and Article:

NickBot has to remember to load an Article from JSON
NickBot holds the database information where Articles are stored
NickBot has to know how GDBM works in order to iterate

Let’s clean this up next.

Third Pass: Active Record Pattern

We’re going to implement the Active Record Pattern. No, I’m not going to require 'activerecord', I’m talking about the classic Active Record Pattern from Patterns of Enterprise Application Architecture. The key here is to mix in the database behavior with the Article class so it can handle iteration, storage, and retrieval without NickBot having to understand how it works.

#!/usr/bin/env ruby

require 'capybara'
require 'capybara/poltergeist'
require 'csv'
require 'gdbm'

class NickBot
  include Capybara::DSL

  def initialize(io = STDOUT)
    Capybara.default_driver = :poltergeist
    @io = io
  end

  def scrape
    visit "http://ngauthier.com/"
    all(".posts .post").each do |post|
      article = Article.from_summary(post)
      next unless article.new_record?
      article.save
    end

    Article.each do |article|
      next if article.body
      visit "http://ngauthier.com#{article.url}"
      has_content?(article.title) or raise "couldn't load #{url}"
      article.body = find("article").text
      article.save
    end

    CSV(@io) do |csv|
      csv << ["Title", "URL", "Date", "Summary", "Body"]
      Article.each do |article|
        csv << [
          article.title,
          article.url,
          article.date,
          article.summary,
          article.body
        ]
      end
    end
  end

Let’s start looking at the usage of Article above. We’ve added Article#new_record?, Article#save, and Article.each. But also notice that NickBot no longer has an @articles instance variable. Cool! Now NickBot doesn’t care at all how Articles are persisted.

Let’s take a look at our new Article class.

  class Article < OpenStruct
    DB = GDBM.new("articles.db")

    def self.from_summary(node)
      new(
        title:   node.find("h3 a").text,
        url:     node.find("h3 a")["href"],
        date:    node.find("h3 small").text,
        summary: node.find("p.preview").text,
      )
    end

    def self.each
      DB.each do |url, json|
        yield Article.new(JSON.load(json))
      end
    end

    def save
      DB[url] = to_h.to_json
    end

    def new_record?
      DB[url].nil?
    end
  end
end

It still inherits from OpenStruct, because that gives us our initialize(attributes) and to_h for cheap. But now we have a DB constant that is set when the code loads up and connects to our GDBM file. This simple constant allows the Article class and also Article instances to access the GDBM database file.

We can now write an easy each that iterates the database and does the JSON load. We can also write our save that dumps using JSON. The new_record? is simply a check for an existing key.

The cool thing is, we could switch our GDBM implementation our for any other type of persistence. We could be using PostgreSQL if we needed more structure, or maybe Redis if we wanted to run it on Heroku with minimal changes.

Wrap-up

I’ve found GDBM to be a really useful little library, because when I am writing utility scripts I don’t want to maintain a PostgreSQL database, but I also need some persistence and reliability. It’s a great step up from writing to csv repeatedly (which is pretty slow and also error prone when the program crashes during a write!).

I continue to find wonderful nuggets of awesome in Ruby’s standard library all the time!

Final source code available as a gist.

Using Docker to Parallelize Rails Tests

2013-10-13T00:00:00+00:00

Docker is a new way to containerize services. The primary use so far has been for deploying services in a very thin container. I experimented with using it for Rails Continuous Integration so that I could run tests within a consistent environment, and then I realized that the containers provide excellent encapsulation to allow for parallelization of test suites.

There are three things we’ll have to do to run a Rails test suite in parallel in docker:

Create a Dockerfile for a system that can run our tests, and build the container
Create a script that breaks our tests across multiple containers
Create a script to run the tests within an individual container

Dockerfile and Container

Dockerfiles are quite simple. The majority of the commands are usually RUN commands that run instructions on the container. Let’s take a look:

from tianon/debian:sid
maintainer Nick Gauthier ngauthier@gmail.com

run apt-get update
run apt-get -qy dist-upgrade

run apt-get install -yq postgresql-9.3 libpq-dev nodejs ruby2.0 ruby2.0-dev build-essential
run gem install bundler --no-ri --no-rdoc --pre

The FROM line bases our container off of debian unstable. I’m using this source because it has ruby 2.0 and postgresql 9.3 right in apt, so the installation is minimal and fast.

Then, we update the system, install postgresql, node (for assets), ruby, and building libraries for gem extenions.

Finally, we install bundler.

Now, we can build our container via:

docker build -t username/appname .

That will build the current directory’s container and tag it with username/appname (so you should replace that with your name and your app’s name). I am not sure yet how to do this in a more portable and anonymous fashion.

Parallelization Script

Next, we’re going to write bin/docker-ci. The goal of this script is to split our tests across multiple containers, and ultimately call bin/ci within the containers.

bin/docker-ci

#!/usr/bin/env bash
set -e

# Make our tmp directory for gems
mkdir -p /tmp/docker

# Docker options:
# Mount the current directory to /data/code
# Mount the temp directory to /data/gems
# Set GEM_HOME to the data directory
# Set the working directory to the code directory
# Use our built container
opts="-v `readlink -f .`:/data/code
      -v /tmp/docker:/data/gems
      -e GEM_HOME=/data/gems
      -w /data/code
      username/appname"

# Bundle the gems (once, serially)
docker run $opts bundle --quiet

# Spread test files in large groups, and pass them into the
# container's bin/ci method
ls test/**/*_test.rb | parallel -X docker run $opts /data/code/bin/ci

The -v options allow us to share the current machine’s code directory with the container. One issue here is that any file system operations within the code folder could conflict across containers.

We’re using GNU parallel with the -X flag, which will spread the test files into larger chunks, as oposed to one job per test file. I don’t think this perfectly utilizes all the cores on my machine, so some more tweaking could be done here.

At this point, bin/ci is run with one or more test files as parameters.

Test running script

The bin/ci script needs to run a set of test files, and it will also need to initialize the container so that the suite can run.

#!/usr/bin/env bash
set -e
# Start postgresql
service postgresql start
# Create the db
su -c "createuser root -s" postgres
# prep the db
bundle exec rake db:test:prepare
# require test files from the arguments given to this script
ruby -I.:test -e "ARGV.each{|f| require f}" $*

We have to boot up and initialize postgresql because containers don’t preserve running services, they are simply file systems that can be booted up. We also want to do this each time because we’d rather load the db than build it into the container and have to rebuild the container when our schema changes.

I’m using ruby with require, but here you could substitute any way that says “run the following test files”. rspec’s binary would work well, and also the m binary. I just stuck a simple ruby script here that should be suite-agnostic.

Summary

And that’s it! It’s actually fairly simple, but it took me a while to stick everything together. I think there are certainly some refinements to be made to generalize it a bit better. For example, you could use any container from the docker index that provides a good rails base for your app. That way you wouldn’t have to maintain a Dockerfile in the project.

Also, I’m not currently seeing any performance improvements due to the parallelization, but that’s because it’s a very short suite, so the overhead of doing a bundle check and db initialization outweighs the savings of parallelism.

Try it on your app, I’d love to hear the results.

PostGIS and Rails: A Simple Approach

2013-08-18T00:00:00+00:00

PostGIS is a geospatial extension library for PostgreSQL that allows you to perform a ton of geometric and geographic operations on your data at high speeds. For example:

Compute the distance between two points
Find all the points within X meters of point P
Determine which points are enclosed in polygon P
A million other things

In Ruby land, there is a gem called RGeo that provides a ton of objects and methods for handling Geospatial objects. In Rails, there are a number of ActiveRecord adapters for each database driver that serialize and deserialize these objects to and from their natural types in the database and RGeo types (for example, activerecord-postgis-adapter).

They are both great and powerful gems that can handle the majority (if not all) of the kinds of things you’d like to do with a geospatial database.

However, since Rails doesn’t have hooks in place to extend the existing drivers with new ways to handle new data types, the ActiveRecord drivers are a (well maintained) collection of subclasses with key methods copied and modified from the originals.

This means that they are difficult to setup and can be hard to use. However, it certainly can be done.

A Simpler Approach

Despite the enormous power of these libraries, I tend to only need one or two features, and it’s usually the simple stuff. Like “Find all the bars near the current user”.

In this post, we’re going to explore how to use PostGIS with Rails with a few snippets of raw SQL and a couple of advanced PostgreSQL features.

1. Setup

First off, you should have Ruby 2 and Rails 4 already installed. If not, go look for guides on that first.

PostgreSQL ships with all the major linux distributions, and is available via homebrew on a Mac. However, that’s not how you want to install them.

OS X

On OS X, the best way to get PostgreSQL with PostGIS and other advanced features is with Heroku’s PostgreSQL.app.

Currently, the most recent build is having problems with PostGIS (issue #109). In the mean time, use this link to run a previous build of PostgreSQL.app:

http://postgres-app.s3.amazonaws.com/PostgresApp-9-2-2-0.zip

Once issue #109 is resolved, the current build should work.

Linux

On linux, most package managers are lagging a bit behind the current PostgreSQL version. PostgreSQL has setup repositories for BSD, the RedHat family, Debian/Ubuntu, SuSE, and more on their download page.

On Debian and Ubuntu, you’ll want to use their apt repository. Edit /etc/apt/sources.list.d/pgdg.list and paste:

deb http://apt.postgresql.org/pub/repos/apt/ YOUR_UBUNTU_VERSION_HERE-pgdg main

Then, import the repository key:

wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
  sudo apt-key add -
sudo apt-get update

Finally, install PostgreSQL 9.2, the contrib package, and PostGIS 2.0 scripts:

sudo apt-get install postgresql-9.2 postgresql-contrib-9.2 postgresql-server-dev-9.2 postgresql-9.2-postgis-2.0-scripts

(Note that if you have a previous version of PostgreSQL installed side-by-side, PostgreSQL 9.2 may run on a non-standard port so as not to conflict with the old version)

Test your installation

To test your installation, make a database and attempt to enable PostGIS:

$ psql
psql (9.2.4)
Type "help" for help.

nick=# create database postgis_test;
CREATE DATABASE
nick=# \c postgis_test
You are now connected to database "postgis_test" as user "nick".
postgis_test=# create extension postgis;
CREATE EXTENSION
postgis_test=#

If it doesn’t complain, it worked!

2. Refuelly

In order to demonstrate the steps involved, we need a demo project. We’ll be writing an app called Refuelly that will help users find the nearest place to get coffee. We won’t build the whole app, just the models necessary to ask the following question: “What are the top 10 closest places to a given location?”

Let’s make the app:

$ rails new refuelly -d postgresql
      create  
      create  README.rdoc
      create  Rakefile
      create  config.ru
      ... etc ...
$ cd refuelly
$ bundle update

At this point, I edited the config/database.yml to suit my machine. I’m in linux, so I simply removed the username and password since I have myself setup as a superuser.

Since we will be using some raw SQL, we need to switch from a schema.rb to a structure.sql. This means that in development, we’re going to save our database snapshot in raw SQL, not Rails’s Ruby representation.

Edit config/application.rb and add this line inside the class Application near the other commented out config statements:

config.active_record.schema_format = :sql

Now, create the database and a structure file should exist:

$ rake db:create db:migrate
$ ls db/structure.sql 
db/structure.sql

Great, now our app is setup with PostgreSQL. This is a good time to commit your code.

2. Cafe

We need a Cafe model to represent a coffee shop. It will have a name, latitude, and longitude. Let’s create it:

$ rails generate model cafe \
    name:string "latitude:decimal{9,6}" "longitude:decimal{9,6}" \
    --fixture false
      invoke  active_record
      create    db/migrate/20130818184035_create_caves.rb
      create    app/models/cafe.rb
      invoke    test_unit
      create      test/models/cafe_test.rb

We’re going to store latitude and longitude as decimal types with precision 9 and scale 6. That means there will be a total of 6 digits right of the decimal, and 9 total digits. PostgreSQL will secretly store this as an integer with an order of magnitude (like scientific notation). That way, there are no floating point errors. This is what the PostgreSQL money type is, under the hood. That way, it keeps the decimal point in the right place for us and doesn’t lose precision.

Hilariously, Rails pluralizes cafe to caves, so real quick edit the migration and change it to cafes (don’t forget to rename the file too). If you’re a stickler like me, you should also mark every column null: false. Also, edit app/models/cafe.rb and add:

self.table_name = "cafes"

Now, migrate:

$ rake db:migrate
==  CreateCafes: migrating ====================================================
-- create_table(:cafes)
   -> 0.0141s
==  CreateCafes: migrated (0.0143s) ===========================================

Now is another great time to commit.

3. Query on computed points

Our next goal is to be able to create some cafes and then query to find the closest ones to a given point. So, let’s write a test to help us develop the code. Inside test/models/cafe_test.rb add:

test "close cafes" do
  far_cafe = Cafe.create!(
    name:      "Far Cafe",
    latitude:   40.000000,
    longitude: -77.000000
  )

  close_cafe = Cafe.create!(
    name:      "Close Cafe",
    latitude:   39.010000,
    longitude: -75.990000
  )

  close_cafes = Cafe.close_to(39.000000, -76.000000).load

  assert_equal 1,          close_cafes.size
  assert_equal close_cafe, close_cafes.first
end

Here we’re creating two cafes and we expect the close cafe to be close_to the point, but not the far one.

(p.s., we’re calling load for prettier sql for later, normally this is not a good idea)

To solve this, we need to do two things. First, add PostGIS:

$ rails generate migration enable_postgis

Edit the migration, and write:

class EnablePostgis < ActiveRecord::Migration
  def change
    enable_extension :postgis
  end
end

enable_extension is new in Rails 4, and it abstracts PostgreSQL’s extension commands for us. Nice!

Next, we need to write our scope on Cafe:

scope :close_to, -> (latitude, longitude, distance_in_meters = 2000) {
  where(%{
    ST_DWithin(
      ST_GeographyFromText(
        'SRID=4326;POINT(' || cafes.longitude || ' ' || cafes.latitude || ')'
      ),
      ST_GeographyFromText('SRID=4326;POINT(%f %f)'),
      %d
    )
  } % [longitude, latitude, distance_in_meters])
}

This should pass our test. Let’s talk about what’s going on.

We’re making a where scope on Cafe that uses the ST_DWithin function to find all the cafes with a certain distance of a given point. The third parameter here is the distance in meters, and our default is 2km.

Then, we’re providing two point objects via ST_GeographyFromText. This function converts some text in Well Known Text (WKT) format to the binary format used by PostGIS to represent points. An example of WKT would be SRID=4326;POINT(-76.000000 39.000000). The first parameter sets the SRID to 4326 (a projection representing the whole globe) and then builds a string for a point featuring the cafe’s longitude and latitude. This builds a point on the fly for each cafe in the cafes table.

The second point object builds a point using the parameters passed in to the scope lambda, the lat and lon. We use Ruby’s built-in string interpolation to easily encode two floating point numbers. It’s safe and it let’s us avoid ActiveRecord’s quoting, which would mess up the SQL.

As you can see, I’m computing everything on the fly. That means whenever this query is run, we’re converting all the cafes’ latitudes and longitudes into points and also converting our query point into a point, then scanning every cafe and manually computing distance. This is super slow.

But it passes our test. Commit!

4. At scale

OK, so it works, but it’s probably slow (we don’t really know yet how slow, do we?). So, let’s investigate!

First, we need some test data. Open up a psql console and insert 1 million random cafes:

$ rails dbconsole
psql (9.2.4)
Type "help" for help.

refuelly_development=#
insert into cafes (name, latitude, longitude) (
  select 'Cafe ' || i as name, 39 + x.lat as latitude, -76 - x.lon as longitude
  from (
    select i, random() * 10 as lat, random() * 10 as lon
    from generate_series(1,1000000) as i
  )                                                            
as x );

For speed, we do this in PostgreSQL so that Rails doesn’t chug along making model objects. We’re making a million restaurants within 10 minutes of 39, -76.

Now, when we ran our test, you can look in the logs to find the query Rails puts together. It looks like this:

SELECT "cafes".* FROM "cafes" WHERE (
 ST_DWithin(
 ST_GeographyFromText(
 'SRID=4326;POINT(' || cafes.longitude || ' ' || cafes.latitude || ')'
 ),
 ST_GeographyFromText('SRID=4326;POINT(-76.000000 39.000000)'),
 2000
 )
 )

Simply copy and paste that into your postgresql console (and a semicolon), and it should output the result. I happened to get 8 rows, and it took my computer around 3 seconds.

Let’s use PostgreSQL’s built-in EXPLAIN and ANALYZE tools:

refuelly_development=# EXPLAIN ANALYZE SELECT "cafes".* FROM "cafes" WHERE (
 ST_DWithin(
 ST_GeographyFromText(
 'SRID=4326;POINT(' || cafes.longitude || ' ' || cafes.latitude || ')'
 ),
 ST_GeographyFromText('SRID=4326;POINT(-76.000000 39.000000)'),
 2000
 )

 Seq Scan on cafes  (cost=0.00..355927.00 rows=37036 width=47) (actual time=612.698..3435.187 rows=8 loops=1)
   Filter: ((snipped))
   Rows Removed by Filter: 999992
 Total runtime: 3435.256 ms

As you can see from the first line, we’re doing a Sequence Scan, which means we’re looking at every single cafe. Then we have a filter (I removed it from the output) which is our distance query and checks each cafe. Finally, you can see the total runtime of ~3.4 seconds.

Yup. It’s slow.

5. Indexing

So, now what? Normally, we’d store the cafe’s location as a computed point field of the latitude and longitude. However, that’s going to be really annoying because rails will get really confused trying to treat that point field like a string. This is the pain that the activerecord-postgis-adapter family of gems help fix.

But, we’re going to do something much more awesome and clever. We’re going to use PostgreSQL’s ability to index on expressions.

Unlike MySQL, PostgreSQL allows you to build an index on any expression you want. The simplest expression would simply be a column name, like this:

create index index_users_on_email on users (email)

The chunk inside the parens is the expression. So, if you always searched users by their lowercased email, you could make an optimized index just for that function, like this:

create index index_users_on_email on users (lower(email))

Now, postgresql will keep an index filled with downcased email addresses that are ready to be queried without having to perform the lowering at runtime. Sweet.

So, what we’re going to do is index on a point built from cafe’s longitude and latitude. It’s pretty easy, we just copy our SQL fragment from our query and turn it into an index:

$ rails generate migration add_point_index_to_cafes

Edit the migration and write:

class AddPointIndexToCafes < ActiveRecord::Migration
  def up
    execute %{
      create index index_on_cafes_location ON cafes using gist (
        ST_GeographyFromText(
          'SRID=4326;POINT(' || cafes.longitude || ' ' || cafes.latitude || ')'
        )
      )
    }
  end

  def down
    execute %{drop index index_on_cafes_location}
  end
end

Our index’s expression will be a point built from the latitude and longitude. Now, when our distance query asks for this built point, our index will match the query and be usable for our distance calculation.

Migrate your database (this will take a little bit because it has to index the million points already in the db, for me it took 18 seconds)

$ rake db:migrate

==  AddPointIndexToCafes: migrating ===========================================
-- execute("\n      create index index_on_cafes_location ON cafes using gist (\n        ST_GeographyFromText(\n          'SRID=4326;POINT(' || cafes.longitude || ' ' || cafes.latitude || ')'\n        )\n      )\n    ")
   -> 18.7678s
==  AddPointIndexToCafes: migrated (18.7680s) =================================

Now, let’s re-run our explained distance query:

refuelly_development=# explain analyze SELECT "cafes".* FROM "cafes" WHERE (
 ST_DWithin(
 ST_GeographyFromText(
 'SRID=4326;POINT(' || cafes.longitude || ' ' || cafes.latitude || ')'
 ),
 ST_GeographyFromText('SRID=4326;POINT(-76.000000 39.000000)'),
 2000
 )
 )

 Bitmap Heap Scan on cafes  (cost=13665.84..133655.39 rows=37036 width=47) (actual time=0.543..1.054 rows=10 loops=1)
   Recheck Cond: (cafe within distance bounding box of point)
   Filter: (cafe within distance bounding box of point and within exact distance of point)
   Rows Removed by Filter: 4
   ->  Bitmap Index Scan on index_on_cafes_location  (cost=0.00..13656.58 rows=333330 width=0) (actual time=0.119..0.119 rows=14 loops=1)
         Index Cond: (cafe within distance bounding box of point)
 Total runtime: 1.155 ms

Boom, 1ms for the query. If you look at the explain output, you’ll see we’re now doing (from the bottom up) a bitmap index scan using our index (it does a pass with just a bounding box for speed), then a filter with exact distance, then it rechecks with the bounding box.

6. Conclusion

In conclusion, there are a number of pros and cons to this solution.

It’s great that on our Cafe we simply use latitude and longitude attributes. No serializers, factories, or callbacks involved. Rails is perfectly happy.

There are no extra gems needed, which means no extra dependencies, version lag, or compatibility issues with the driver.

Indexing can be done and queries are very fast.

However, we did have to hand-write a bunch of SQL, which is always less than optimal. In this case, I’m OK with it, because PostGIS is really the best tool for the job here. We also have to keep the SQL in our query synchronized with our index, but that’s the case with any index on an expression.

All-in-all, I prefer this solution because it’s smaller, simpler, and lighter weight for accomplishing simple tasks with geospatial data on Rails.

Source Code

The source code resulting from this post is available at github.com/ngauthier/postgis-on-rails-example.

Learning Angular on Rails

2013-04-29T00:00:00+00:00

Last night I had the best idea for a JavaScript framework. It was going to use the dom with data attributes in a totally unobtrusive way. It would have global repositories for remote data, do caching, and attach controllers to the dom automatically.

This morning, I realized that’s pretty much how Angular.js rolls, so I decided to learn the basics.

1. Setup Rails

First, I setup a Rails app as the back-end. I used Rails 4.0.0.beta1. I also decided that since Angular does so much heavy data-attr (actually ng-attr) work in the view, I wanted something nicer than erb, so I added Slim for templating. Also, I like CoffeeScript, so I stuck with that.

I setup a Post model and controller. Angular behaved a little differently, so the controller looks a little different:

class PostsController < ApplicationController
  include AngularController

  def index
    @posts = Post.all
    respond_with @posts
  end

  def create
    respond_with Post.create!(params.permit(:title, :body))
  end

  def destroy
    Post.destroy(params[:id])
    head :ok
  end
end

I’m using respond_with to send the html template on index or json for the index, depending on the request. I also use it in the create action because that will send down json automatically, and I find it cleaner than render json: post.

I created a Concern called AngularController that abstracted the necessary json massaging that Angular needed:

module AngularController
  extend ActiveSupport::Concern

  included do
    respond_to :html, :json
    around_action :without_root_in_json
  end

  def without_root_in_json
    ActiveRecord::Base.include_root_in_json = false
    yield
    ActiveRecord::Base.include_root_in_json = true
  end
end

It adds the respond_to to work with respond_with. It also sets up an around_action to temporarily remove the root of json responses. I thought this was a cool way to do it, instead of doing it globally.

Now the Rails app is setup. I actually set it up while learning angular, but I thought I’d present it here first for simplicity’s sake.

2. Angular Time!

I grabbed angular.js and also angular-resource.js (RESTful requests) and dropped them into vendor/assets/javascripts, as well as loading up Twitter Bootstrap to make it look Not Terrible™.

In app/views/index.html.slim I added:

- content_for :ng_app, "blang"

.container
  .hero-unit
    h1 Posts
    p Driven by Angular.js

  div ng-controller="PostCtrl"
    .post ng-repeat="post in posts"
      h2 
        | {{ post.title }}
        button.btn.btn-danger.btn-small.pull-right ng-click="delete($index)" ×
      | {{ post.body }}

    form ng-submit="add()" action=""
      fieldset
        legend Create Post
        
        input    ng-model="post.title" type="text" id="post-title" placeholder="Title"
        br
        textarea ng-model="post.body" placeholder="Content" rows="4" columns="40"
        br

        input.btn.btn-primary type="submit" value="Create"

I use content_for :ng_app in the layout to render Angular’s ng-app="blang" so that it boots up. The page is driven by the PostCtrl controller, and it loops over all the posts in PostCtrl. It outputs their title and body, along with a delete button.

Below, there’s a form that hits the add() method of PostCtrl feeding it the title and body of the post that we’re creating.

Here’s my posts.coffee:

# Set up the module
window.Blang = angular.module("blang", ["ngResource"])

Blang.config ["$httpProvider", ($httpProvider) ->
  # Inject the CSRF token
  $httpProvider.defaults.headers.common['X-CSRF-Token'] = document.getElementsByName("csrf-token")[0].content
  # By default, angular sends "application/json, text/plain, */*" which rails
  # sees and focuses on the */* and sends html :-(
  $httpProvider.defaults.headers.common['Accept'] = "application/json"
]


# Here's our Post resource for interacting with the server
Blang.factory "Post", ($resource) -> $resource "/posts/:id", id: "@id"

# Post Controller
Blang.controller "PostCtrl", ($scope, Post) ->
  # This is the post we use for the form
  $scope.post = new Post()

  # Posts for the list
  $scope.posts = Post.query()

  # Add a new post
  $scope.add = ->
    # add to the local array and also save to the server
    $scope.posts.push Post.save title: $scope.post.title, body: $scope.post.body
    # reset the post for the form
    $scope.post = new Post()

  # Delete a post
  $scope.delete = ($index) ->
    # Yay, UX!
    if confirm("Are you sure you want to delete this post?")
      # Remove from the server
      $scope.posts[$index].$delete()
      # Remove from the local array
      $scope.posts.splice($index, 1)

All in all, it wasn’t much code, but it was very difficult to figure out what I was supposed to be doing. But that’s all part of learning a new framework. Here are a bunch of gotchas I ran into:

Providing the id: "@id" mapping in the resource url. Without this, it doesn’t automatically fill in :id with the model’s id.
I had to keep the local array of posts in sync with the server, pushing to it and splicing from it as I added and removed items.
Using $scope everywhere to expose methods in the view took me a bit to realize
I couldn’t delete a post by calling a method on the instance, because it wouldn’t remove it from the main collection

All in all, I found Angular to be pretty comprehensive when it came to the view binding and automatic dom reflection of underlying state. However I found its server synchronizing abilities to be lacking. Why is angular-resource.js a separate library anyways?

I think it would be really interesting to use Angular.js as a declarative templating and view language, but then maybe drop to backbone for models and controllers. I could expose collection.models to the scope, and it could watch that array. Then it should properly add and remove models from a collection.

I’m really curious to hear how experienced users of Angular communicate with a server and keep their data in sync.

If you’d like to mess around with the code, I put the application on GitHub.

Rails Controller Accessors

2013-04-15T00:00:00+00:00

Recently, I’ve been reading Practical Object-Oriented Design in Ruby by Sandi Metz (I highly recommend it!) and it got me thinking more about OO design in Rails. I realized that one of the patterns I’ve been using synced really well with the messages in the book, and I wanted to share it.

Model Loading Filters

One of the most common Rails controller patterns is the before filter that fetches the record being accessed during member route actions. It looks like this:

class PostController < Application Controller
  before_filter :find_post, only: [:show, :edit, :update, :destroy]

  # actions redacted

  private

  def find_post
    @post = Post.find params[:id]
  rescue ActiveRecord::RecordNotFound
    flash[:error] = "No Such Post"
    redirect_to posts_path
    return false
  end
end

<h1><%= @post.title %></h1>

This makes @post available in the show, edit, update, and destroy actions, as well as making them available in the views.

However, there are a few problems with this implementation:

In views, you access the post as @post whereas in a partial, you have to either depend on @post being available, or you reference a local called post, which is inconsistent with @post.
The before filter has two responsibilities: finding the post and handling when there is no such post. Ideally this would be extracted into two methods each with one responsibility.
Instance variable copying is The Rails Way, but it’s very different from The Ruby Way and Object Oriented best practices. In short, the view depends on the internals of the controller.

Controller Accessors

Recently, I’ve been using Controller Accessors as an alternative. Here’s what a Controller Accessor looks like:

class PostController < Application Controller
  before_filter :ensure_post, only: [:show, :edit, :update, :destroy]

  # actions redacted

  private

  def post
    @post ||= Post.find_by_id params[:id]
  end
  helper_method :post

  def ensure_post
    redirect_to posts_path, alert: "No Such Post" unless post
  end
end

<h1><%= post.title %></h1>

Here we’ve extracting the post finding to a separate accessor method with memoization. We also expose this accessor as a helper method, which is a great way to allow the view to access a controller’s properties.

Note that in the view we access post and not @post. @post is still available, but accessing it via post is the polite way to do it, and it much more resilient.

We’ve also separated loading the post and ensuring that it is available by moving our filter to ensure_post.

Lastly, I’ve opted to use find_by_id instead of find because find_by_id returns nil instead of raising ActiveRecord::RecordNotFound, and a nil is easier to test for. Also, it wouldn’t make sense for the post accessor to throw an exception (or to rescue it!) we’d rather have a nil to express that there is no such post.

WebSockets in Rails 4

2013-03-12T00:00:00+00:00

I’ve been using Rails 4 (beta) a lot recently. In a previous post we looked at how ActionController::Live can be used with Server-Sent Events, but the problem with that is that there’s no way for the client to communicate back to the web server. Enter: WebSockets.

The main issue with implementing WebSockets is that they have to keep their connections open for a long period of time, and when you’re only running a small cluster of Rails servers, it eats up potential connections fast. That’s why I was excited to hear about two important developments: concurrency in Rails 4 and Rack Hijack.

Concurrency in Rails 4

Rails 4 is now full concurrent, which means that there is no full-stack lock on a request. That means that if you use a concurrent server like Puma you can handle many requests at a time with a single process.

Even better, you can use ruby Threads inside your Rails app. That’s how ActionController::Live works (for streaming).

What this means for us is that we can use Threads to hold websocket connections open without bogging down our server.

Also, this means that our solution does not use Eventmachine, nor does it implement a reactor in any way. It’s concurrent.

Rack Hijack

Rack Hijack came with Rack 1.5.0 which was released this past January. Rack hijack allows you to access the underlying socket of a Rack connection in order to bidirectionally communicate with the client. Since Rails is built on Rack we can grab a handle to the client socket right from a Rails controller.

Tubesock

Tubesock is the gem that I made to encapsulate this functionality. It’s very small and new and untested so caveat emptor and all that. At its core it provides a module for Rails controllers and a wrapper method to hijack the rack connection. Then it wraps the ruby gem websocket to handle WebSocket handshakes and frames. Here’s an example of using it in a controller:

class ChatController < ApplicationController
  include Tubesock::Hijack

  def chat
    hijack do |tubesock|
      tubesock.onopen do
        tubesock.send_data "Hello, friend"
      end

      tubesock.onmessage do |data|
        tubesock.send_data message: "You said: #{data}"
      end
    end
  end
end

Right inside the controller action we can hijack the connection and then use some blocks to send information.

You can check out the Tubesock gem on Github for more information.

Also, there is an example chat application you can run and play with.

Happy hacking!

Rails 4 Server Sent Events with ActionController::Live and PostgreSQL NOTIFY/LISTEN

2013-02-19T00:00:00+00:00

I had a simple problem: one user takes an action, and I want it to be reflected immediately on another user’s screen.

There are lots of potential ways to solve this: polling, long polling, websockets, etc. However I had a specific goal in mind: use the stack I already had and keep complexity to a minimum. I didn’t want to use websockets because of the extra setup on the server. Nginx just got a websockets proxy patch, but I don’t feel like compiling nginx from source on my deployment machines. I wanted something evented, but I didn’t want to add another back-end service like Redis.

I decided to use Rails 4’s ActionController::Live, HTML5 Server Sent Events, and PostgreSQL’s NOTIFY/LISTEN system. The best part is that all I had to do to my stack was swap thin for puma.

Huge thanks to Aaron Patterson for his post Is it Live?. Go read that now, because that’s how I implemented ActionController::Live, SSEs, and the Javascript.

One thing I did do, though, was refactor (or unprefactor) his SSE implementation. Instead of a model I just added a controller method, so my controller action looks like:

def index
  response.headers['Content-Type'] = 'text/event-stream'
  deck.on_slide_change do |slide|
    response.stream.write(sse({slide: slide}, {event: 'slide'}))
  end
rescue IOError
  # Client Disconnected
ensure
  response.stream.close
end

private
def sse(object, options = {})
  (options.map{|k,v| "#{k}: #{v}" } << "data: #{JSON.dump object}").join("\n") + "\n\n"
end

This application is broadcasting a slide change on a deck, and every time a change occurs, it will write an SSE. My client consumes the feed just like Aaron’s, and then I also have a “broadcaster” that sends plain old ajax PUT requests to update the deck slide.

Aaron used rb-fsevent which is an evented file system watching gem for OS X. In my case, I wanted my events to come from the database, so I used PostgreSQL’s NOTIFY/LISTEN. It’s really simple pub/sub that just takes a channel and a payload, and it all operates in shared memory. So it wouldn’t be great for a resilient queue, but it’s great for messaging.

Here’s how I added that in my ActiveRecord model:

after_save :notify_slide_change
def notify_slide_change
  if current_slide_changed?
    connection.execute "NOTIFY #{channel}, #{connection.quote current_slide.to_s}"
  end
end

def on_slide_change
  connection.execute "LISTEN #{channel}"
  loop do
    connection.raw_connection.wait_for_notify do |event, pid, slide|
      yield slide
    end
  end
ensure
  connection.execute "UNLISTEN #{channel}"
end

private
def channel
  "decks_#{id}"
end

Whenever it’s saved and the slide has changed, I send a notify on the channel with the current slide as a string payload. The channel is simply the table name and id stuck together.

on_slide_change is where it starts to get interesting. I listen on the channel (with an ensure to stop listening). Then I have to get the pg gem’s raw connection so I can call wait_for_notify, which is a blocking LISTEN. It will give me an event, a pid, and my payload, which in my case is the slide. In true Ruby style, I yield the slide.

All in all, it was pretty simple, but also pretty difficult to setup. One thing that is particularly tricky is that you have to cache classes in development mode in order to multithread Rails on Puma, which means you have to reboot your server on code changes. Also, I had to make sure my DB connection pool in database.yml was as high as my Puma thread pool so everyone could have a DB connection.

Backbone Screencast 02 - Mixins

2013-02-06T00:00:00+00:00

In this screencast I cover how to extract reusable pieces of code from a Backbone.js application using Mixins. The style is similar to Ruby on Rails’s Concerns in that it encapsulates both instance and class methods.

If you like this screencast, consider buying Mobile Web Patterns with Backbone.js.

Backbone Screencast 01 - Booting your Application

2013-02-06T00:00:00+00:00

In this screencast we go over three different ways of booting your application:

jQuery onReady in the JavaScript file
Application.boot method called from html
Unobtrusive boot based on a data attribute

If you like this screencast, consider buying Mobile Web Patterns with Backbone.js.

All the code is in CoffeeScript, for more information, check out CoffeeScript’s web site.

Unsubscribe Links in Rails with ActiveSupport::MessageVerifier

2013-01-08T00:00:00+00:00

If you’re setting up an unsubscribe link for your emails in a Rails application, it’s important to make it secure and seamless. We want to have it function properly if the user is not logged in without having them log in first. We also want to make sure it’s not easy to forge. The url should be something like this:

http://example.com/unsubscribe/0938745209387452093478530938742509384752

So, we need some way to encode a user’s information securely and create a url, then we also need to decode it. Enter ActiveSupport::MessageVerifier.

MessageVerifier is initialized with a secret, and can then encode and decode messages, like this:

# make a verifier
verifier = ActiveSupport::MessageVerifier.new('secret')

# encode some data
token = verifier.generate('data')
# => "BAhJIglkYXRhBjoGRVQ=--7d0d0ec0f5572ac668afeabea7829064cc78223b"

# note: it's not encrypted!
Base64.decode64(token.split("--")[0])
# => "\x04\bI\"\tdata\x06:\x06ET"

# Get the data back out
verifier.verify(token)
# => "data"

NOTE: the data is not encrypted, just verified with a digest using a secret. The original data is present in the token, so don’t put anything secret in there.

Now, we can integrate this into our User model pretty easily. We’ll make class methods for encoding and decoding, then an instance method to make it easier to use:

class User < ActiveRecord::Base
  # Access token for a user
  def access_token
    User.create_access_token(self)
  end

  # Verifier based on our application secret
  def self.verifier
    ActiveSupport::MessageVerifier.new(MyRailsApp::Application.config.secret_token)
  end

  # Get a user from a token
  def self.read_access_token(signature)
    id = verifier.verify(signature)
    User.find_by_id id
  rescue ActiveSupport::MessageVerifier::InvalidSignature
    nil
  end

  # Class method for token generation
  def self.create_access_token(user)
    verifier.generate(user.id)
  end
end

Now all we have to do is add a route:

match '/users/unsubscribe/:signature' => 'users#unsubscribe', as: 'unsubscribe'

Include the link in our email layout:

<%= link_to "Unsubscribe", unsubscribe_url(@user.access_token) %>

And setup a controller action:

class UsersController < ApplicationController
  def unsubscribe
    if user = User.read_access_token(params[:signature])
      user.update_attribute :email_opt_in, false
      render text: "You have been unsubscribed"
    else
      render text: "Invalid Link"
    end
  end
end

This can be expanded to be used in any scenario where you need an authentication token, for example all email links could contain a token in the param, which is removed in an application-wide before_filter. That way, a user never has to log in if they’re coming from an email you sent them.

Warning: The data is encoded but since it is not (entirely) hashed it will grow in length. Be careful using variable-length or user-generated data, as you could go beyond the 2048 character limit of urls. Much like delayed jobs and cookies, it’s a good idea to store only an identifier in the token, and then look up the associated data in your database. If you really need encryption, check out ActiveSupport::MessageEncryptor.

Ruby on Rails on the Nexus 7

2012-11-28T00:00:00+00:00

This evening I decided to put Ubuntu on my Nexus 7 to see how it performs and what packages are available on ARM. I’m happy to report that Ruby 1.9.3 (p194) and Rails 3.2.9 work perfectly (albeit slowly :-D).

Step 1: Ubuntu

I followed Ubuntu’s steps for installation and it worked fine. There was an odd issue connecting via ssh, but I think it was my router. I had to ssh out of the tablet to my desktop before the desktop could ssh to the tablet. iptables was empty on both. Dunno what was up.

Step 2: Ruby

I followed my own ubuntu setup instructions, except I substituted libsqlite3-dev for the postgresql packages.

Step 3: Rails

I ran gem install bundler and gem install rails, took a minute or two, but it worked.

Step 4: A Blog!

Of course I had to make a blog, that’s the Hello World of Rails! I simply used rails g scaffold Post title:string body:text then rake db:migrate and rails server, then wrote my post.

Honestly I was very disappointed at how easy it all was. I was looking for a challenge :-D

@ngauthier

Intro to Backbone.js

2012-09-27T00:00:00+00:00

Here is a talk I gave at the Baltimore Javascript users group. It is an introduction to backbone.js, featuring about 25 minutes of slides and an hour of live coding an example application.

Live coding starts at 27:47

Check out my ebook, Recipes with Backbone!

One Click Development

2012-08-15T00:00:00+00:00

The Problem

Non-technical teammates can’t run the app. You’ve got designers, project managers, qa, and more. Whenever developers add a dependency to the project, a lot of hand-holding needs to occur to get everyone setup.

Not everyone is comfortable on the terminal, nor should they be.

go.sh

My first solution to this problem was to place go.sh in the root of the project. This command installed bundler, bundled the project, and ran the server. This worked fine, until the first dependency shift: switching from sinatra to rails with postgresql.

I’ve been doing rails for about five years now, and I got a mac a few months ago. I’m embarassed and frustrated at how long it took me to figure out xcode’s command line tools, homebrew, rbenv on OS X, starting and stopping processes, database permissions, etc. How can I ask that of someone who does not have a terminal on their dock?

Many great minds are working on making Rails setup on OS X better, and that’s great. But it’s not enough. Developers will always be one step ahead, using the latest and greatest tools, and we don’t want to wait for others to setup one-click installs for the stack we want.

Vagrant

So now, I turn to Vagrant. I’ve wanted to try it out, but I’ve had no real excuse, until now. So here’s my goal:

go.sh should download, boot, provision, and run the rails app with minimal requirements

Pre-requisites

Thanks to those working on making OS X setup easier, the OS X specific pre-requisites are:

Download and install RailsInstaller
Download and install VirtualBox

On linux, it’s:

apt-get install ruby virtualbox

We need VirtualBox for vagrant, and we need railsinstaller to get ruby 1.9 for vagrant.

Running the project

Create a symlink to go.sh called go.command for mac, so those users can simply double click.

Double click go is now the way to run the system. Then the user browses to localhost:5000

OK, so, HOW???

go.sh

#!/bin/sh

# Ensure we're in the project directory
cd `dirname $0`

# Setup gems to install to userspace
export GEM_HOME=$HOME/.gems
export PATH=$GEM_HOME/bin:$PATH

# Output debug info in case a developer needs to lend a hand
echo "Working in `pwd`"
echo "Ruby `ruby -v`"
echo "Rubygems `gem -v`"

# Install vagrant if it's not installed
vagrant -v 2>/dev/null || gem install vagrant --no-ri --no-rdoc

# Create, boot, provision vm (more later)
vagrant up

# On the vagrant box, run our custom server script
vagrant ssh -c "cd /vagrant && ./script/server"

script/server

Hey, remember this from rails 2? I wrote a custom script/server that does all the things that really need to be done to run a rails server:

#!/bin/sh

# Setup local paths
export GEM_HOME=$HOME/.gems
export PATH=$GEM_HOME/bin:$PATH

# Install bundler if not available
bundle -v 2>/dev/null || gem install bundler --no-rdoc --no-ri

# Install all gems (or update, after pulling new code)
bundle install

# Create db if it doesn't exist, migrate if we need to
bundle exec rake db:create db:migrate

# Start services
bundle exec foreman start

Also, savvy devs would be expected to provision their own system, then run script/server locally (not in a vm) for extra performance. Or, for better consistency, they could run vagrant too.

stop.sh

Lastly, there is a stop.sh. You can ctrl+c and/or close the terminal created by go.sh but that won’t halt the vm.

#!/bin/sh
cd `dirname $0`
export GEM_HOME=$HOME/.gems
export PATH=$GEM_HOME/bin:$PATH

vagrant halt

Provisioning

Now the only remaining question is how to setup vagrant for a nice rails environment. Here is my Vagrantfile:

Vagrant::Config.run do |config|
  # Use ubuntu 12.04 32 bit
  config.vm.box = "precise32"
  config.vm.box_url = "http://files.vagrantup.com/precise32.box"

  # Forward port 5000 to localhost:5000
  config.vm.forward_port 5000, 5000

  # Use chef-solo to provision
  config.vm.provision :chef_solo do |chef|
    # Use our custom recipe
    chef.cookbooks_path = "cookbooks"
    chef.add_recipe "my_application"
  end
end

Vagrant and Chef are entirely new to me (as of yesterday), so this is the area I’m not sure on. I used librarian initially, but found that I needed more control over the recipes and box setup, and that I could use a much simpler recipe and only target ubuntu. Bryan Liles pointed me at chef’s packages and scripts, and so I wrote my own simple cookbook called my_application (I actually named it the same as the Rails app).

Now, vagrant just grabs an ubuntu 12.04 base image, forwards port 5000, and runs my cookbook. The cookbook only needed two files, cookbooks/my_application/metadata.rb which was just my name and info, and then cookbooks/my_application/default.rb:

# Update repository and install any upgrades
execute "update apt" do
  command "apt-get update && apt-get upgrade -y"
end

# Install ruby and postgresql and dependencies for gems and rails
package 'libxslt1-dev'
package 'libxml2-dev'
package 'build-essential'
package 'g++'
package 'ruby1.9.1-dev'
package 'postgresql'
package 'libpq-dev'
package 'git'
package 'nodejs'

# Create a directory for us to put some lock files in so scripts run once
execute "application directory" do
  command "mkdir /var/my_application"
  creates "/var/my_application"
end

# Setup vagrant as a psql user
execute "vagrant psql role" do
  command %{sudo -u postgres createuser -s vagrant         && \
            sudo -u postgres createdb   -O vagrant vagrant && \
            touch /var/my_application/psql-vagrant.done}
  creates        "/var/my_application/psql-vagrant.done"
end

I’m used to a pretty minimal ubuntu setup like this. Much faster and easier than using rvm or rbenv. System ruby is 1.9.3p194 on ubuntu 12.04, which is rvm’s default ruby 1.9 as well. And you don’t have to compile it!

Team workflow

So now, non-technical team members have a running app on their system. Developers can add dependencies to the cookbook, and when a non-technical team member syncs their code (with Github’s apps, right?) they can just click go and vagrant will update the box based on the new provisions, and the server script will update the system’s gems and database. Sweet!

RSpec with Domino

2012-05-08T00:00:00+00:00

Using Domino with RSpec is awesome. I’ll let the code speak for itself.

We have an html page:

<!doctype html>
<html>
  <head>
    <title>Domino Rspec</title>
  </head>
  <body>
    <h1>Domino Rspec</h1>
    <ul>
      <li><span class='name'>John Doe</span> Age <span class='age'>47</span></li>
      <li><span class='name'>Jane Doe</span> Age <span class='age'>37</span></li>
      <li><span class='name'>Jim Doe</span> Age <span class='age'>27</span></li>
    </ul>
  </body>
</html>

And here is an rspec request spec to test the data:

describe :index_without_domino, :type => :request do
  before do
    visit '/'
  end

  it 'should have three people' do
    page.all('ul li').count.should == 3
  end

  context 'John Doe' do
    subject do
      page.all('ul li').find do |node|
        node.find('.name').text == 'John Doe'
      end
    end

    it 'should have an age of 47' do
      subject.find('.age').text.should == '47'
    end
  end
end

CSS selectors are brittle here, and it’s not very rubyish. It’s full of html! First, we’ll make a domino:

module Dom
  class Person < Domino
    selector 'ul li'
    attribute :name
    attribute :age
  end
end

And then update our test:

describe :index, :type => :request do
  before do
    visit '/'
  end

  it 'should have three people' do
    Dom::Person.count.should == 3
  end

  context 'John Doe' do
    subject { Dom::Person.find_by_name 'John Doe' }
    its(:age) { should == '47' }
  end
end

Because dominos are enumerable ruby objects, we can count them easily. There’s also handy attribute finders and accessors. So, a domino can be an rspec subject. Much better!

Full project source code

var self = lame

2012-04-11T00:00:00+00:00

var self = this

I see this in a lot of javascript code. The first thing it makes me think of is Avdi Grimm’s excellent talk Confident Code. I think that var self = this is an example of “timid code” in javascript. Avdi explains how timid code suffers from a “Lack of certainty” and it “lives in fear … constantly second-guessing itself … [and] imposes cognitive load on the reader”. I highly recommend you read the slides or watch the video even if you don’t code ruby.

Why?

Here’s why var self = this is helpful: callback scope. Consider this code from a JavascriptKata post:

// Create a cat
function Cat() {
    this.theCatName = "Mistigri";
}
 
// The cat will meow later
Cat.prototype.meowLater = function() {
    // I create the variable self that refers to the this (the current object)
    var self = this;
 
    // I create a timeout that calls the self.meow function within an anonymous function
    /*** NOTE : You don’t always have to create an anonymous function it’s just that in
        this case, it is required ***/
    window.setTimeout(
        function() {
            self.meow();
        }
        , 1000);
}
// The cat meows
Cat.prototype.meow = function() {
    // I can use the this expression!!!
    alert(this.theCatName + " : meow!");
}
 
// I crate an object and call the meowLater() function
var theCat = new Cat();
theCat.meowLater();

Click here to run

// The cat will meow later Cat.prototype.meowLater = function() { // I create the variable self that refers to the this (the current object) var self = this;

// I create a timeout that calls the self.meow function within an anonymous function
/*** NOTE : You don’t always have to create an anonymous function it’s just that in
    this case, it is required ***/
window.setTimeout(
    function() {
        self.meow();
    }
    , 1000); } // The cat meows Cat.prototype.meow = function() {
// I can use the this expression!!!
alert(this.theCatName + " : meow!"); } $('a[href="#meowlater"]').click(function() {   var cat = new Cat();   cat.meowLater(); }); </script>

Fix #1: Using setTimeout’s argument passing

The first way we can fix this use of timid scope passing is to use setTimeout’s ability to pass arguments:

function ApplyCat() {
  this.name = "Garbanzo"
}

ApplyCat.prototype.meowLater = function() {
  window.setTimeout(
    function(cat) {
      cat.meow()
    }, 1000, this
  )
}

ApplyCat.prototype.meow = function() {
  alert(this.name + " : meow!")
}

Click here to run

TimeoutCat.prototype.meowLater = function() { window.setTimeout( function(cat) { cat.meow() }, 1000, this ) }

TimeoutCat.prototype.meow = function() { alert(this.name + “ : meow!”) }

$(‘a[href=”#meowlatertimeout”]’).click(function() { var cat = new TimeoutCat() cat.meowLater() }) </script>

I’m passing this in as an argument for setTimeout to pass to the callback function, so that I have a reference to the cat.

So, this is better because instead of var self = this I’m passing this in as cat. This is equivalent to var self = this as far as scoping is concerned, and doesn’t buy us too much. But, one thing it does to is that it’s more explicit about why the scope is being passed, and what the scope is. But there’s a better way.

Fix #2: Use underscore.js

Why are we reinventing the wheel here? We’ve been binding scope since the dawn of time (or at least, the dawn of javascript). Why should we all repeat the same patterns over and over when smarter people have done it for us?

function BindCat() {
  this.name = "Alfonso"
}

BindCat.prototype.meowLater = function() {
  window.setTimeout(_.bind(this.meow, this), 1000)
}

BindCat.prototype.meow = function() {
  alert(this.name + " : meow!")
}

Click here to run

BindCat.prototype.meowLater = function() { window.setTimeout(_.bind(this.meow, this), 1000) }

BindCat.prototype.meow = function() { alert(this.name + “ : meow!”) }

$(‘a[href=”#meowlaterbind”]’).click(function() { var cat = new BindCat() cat.meowLater() }) </script>

This is the best solution. For four reasons:

First, _.bind(this.meow, this) very clearly states that the objective of this code is to bind the scope of the function to the current object. var self = this says “I need to use this later” and then later on when you see self you remember that self is a scope shortcut to this. It’s a minor cognitive load, but it’s not worth dismissing.

Second, you’re leaning on a well tested library like underscore to not mess things up. Whenever you can use someone else’s shared library to do something, do it. Odds are they’re covering edge cases you’re not. For example, underscore’s bind will use ECMAScript 5’s native bind if it’s available. By the way, underscore has a ton of other awesome methods you should be using.

Third, it’s a one-liner. It’s so succinct that I can put it right into the timeout call and it fits easily. var self = this is one extra line, plus the inline function definition causes three extra lines. Less is more.

Fourth, when you remove this line, it doesn’t leave any other dangling lines of code. I’ve seen plenty of var self = this followed by a bunch of code that doesn’t have any callbacks in it. There was no reason to declare self, yet it’s used in place of this out of fear.

Alternate Use of underscore

Another way you can use underscore for function binding is with _.bindAll:

function BindAllCat() {
  _.bindAll(this);
  this.name = "Fred"
}

BindAllCat.prototype.meowLater = function() {
  window.setTimeout(this.meow, 1000)
}

BindAllCat.prototype.meow = function() {
  alert(this.name + " : meow!")
}

Click here to run

BindAllCat.prototype.meowLater = function() { window.setTimeout(this.meow, 1000) }

BindAllCat.prototype.meow = function() { alert(this.name + “ : meow!”) }

$(‘a[href=”#meowlaterbindall”]’).click(function() { var cat = new BindAllCat() cat.meowLater() }) </script>

In the constructor, we use _.bindAll, which will bind every instance method to the instance. So no matter how you call the meow function, it will be bound to the cat. Now feel free to call your instance methods everywhere, and they will be scoped to the object, just like a real OO language.

Conclusion

Please stop coding out of fear. Maybe you want to keep using self anyways because you think it reads better, but at least understand that there are many ways to accomplish proper scoping in javascript, and consider all of them when you run into these kinds of issues.

xoxo @ngauthier

Pre-emptive comment responses

It’s less efficient

You’re right. And C is even faster. You know what’s the least efficient? Poorly readable and unmaintainable code.

I don’t have underscore, and I don’t want to add more dependencies

It’s 4kb. Suck it up. That’s < 1ms on broadband. The collection extensions will change the way you enumerate, and underscore alone is worth it so you can write:

var maxPrice = _(widgets).chain().map(function(widget) {
  return widget.price()
}).reduce(function(max, price) {
  return !max || price > max ? price : max
}).value();

mmmm. map/reduce.

Also, every collection method automatically takes a context as an optional final parameter. So you can do:

_(this.widgets).each(function(widget) {
  this.enable(widget);
}, this)

this on the first two lines refers to the same object, because this is passed to each as the final parameter. IMO, jQuery.on should replace the data parameter with a context parameter. It would encourage more OO-style event coding. Backbone automatically binds all view event callbacks to the view, which is awesome.

Solicitation for comments

The main reason I wrote this post is that a number of smart programmers I respect disagree with me. I would love to have a constructive and positive discussion on this topic. I never remove comments, so you can still hate on me, but everyone will know how lame you are on the internet.

Deploy Ruby as a Gem

2012-04-09T00:00:00+00:00

Background

I was looking at git-deploy and it looks really awesome. But it got me thinking: We have tools like bundler and rack that can handle gem versioning and dependencies, and creating clean interfaces for web applications. Why can’t we use those tools for deploying our own code?

I think ruby gems are a great way of packaging code up, and that any code you write should become a gem if it needs to be re-used. Gems are pretty easy to write, and they automatically help you package and version your code.

I also think we should be doing more version-based deploys of code. Even if it’s just a git tag. But it’s better if it’s actually an application version.

So, I stumbled across Running Sinatra inside a Ruby Gem by Florian Hanke, and I really liked it, but I wanted to take it one step further: deploying.

Application Setup

The first thing we have to do is setup our application:

bundle gem awesome_site

That will make a gem scaffold inside the awesome_site folder for a gem called AwesomeSite. Next, in awesome_site.gemspec add:

gem.add_dependency 'sinatra', '1.3.2'

Also, update the gem.description and gem.summary.

Next, we’re going to add a config.ru, which will help us test our application in development:

require 'rubygems'
require 'bundler'
Bundler.setup
require 'awesome_site'
run AwesomeSite::App

So, we need to setup AwesomeSite::App as a rack app. Edit lib/awesome_site.rb to autoload App:

module AwesomeSite
  autoload :App, 'awesome_site/app'
end

Then put this in lib/awesome_site/app.rb:

require 'sinatra'
module AwesomeSite
  class App < Sinatra::Base
    get '/' do
      'hello world!'
    end
  end
end

Note that I used autoload instead of require for App and I also didn’t require Sinatra until the app.rb file. This means that loading my gem will be fast, and that sinatra won’t have its code loaded and run until the App is needed. This will keep boot times down (like when unit testing code that doesn’t need App).

Now, you should be able to run rackup and visit the app!

The last thing we need to do is rake install to build and install the gem on our local system.

Server Setup

OK, so now that we have our great app as a gem (thanks Florian!) it’s time to setup the server. Make a new directory (to simulate the server’s application folder) outside of the awesome_site folder. I just called mine server. In this folder, make a Gemfile:

source :rubygems
source 'http://0.0.0.0:8808'
gem 'awesome_site'

This Gemfile shows how I would install awesome_site from a local gem server (by running gem server) and pull dependencies from rubygems.org. An alternative would be:

source :rubygems
gem 'awesome_site', :git => 'git://github.com/ngauthier/awesome_site.git'

That could pull the gem from github, but that’s suboptimal because you would have to also send a :ref in order to set the version, which would be a pain in the next section.

A third option would be to build the gem locally and scp it to the server and install, but that doesn’t use all the fun bundler stuff!

You can use geminabox to run your own gem server that you can push gems too.

OK, now we need a config.ru file for the server. It’s the same as awesome_site’s config.ru:

require 'rubygems'
require 'bundler'
Bundler.setup
require 'awesome_site'
run AwesomeSite::App

Now, we can run bundle and rackup and we’re running our server based off our gem!

Deployment

As mentioned previously, you need to push the gem to your own gem server. That means either your gem server can check out the latest code and install the gem, or simply use geminabox to let developers push a gem. So the deployment steps would be:

Update awesome_site in some way, and set AwesomeSite::VERSION to an updated number.
rake build
gem inabox ./pkg/awesome_site-1.0.0.gem (then enter the geminabox server url when prompted)
ssh production "bundle update awesome_site && sudo restart awesome_site"

This is assuming you use upstart to manage your server process (probably my next blog topic).

Now, if you want more control, put the server code into git and manually set the version number in the gemfile. That way you can roll back to a version if you have a problem with a deploy. But, you could also reverse the commit, but bump the gem version and deploy as usual. This is a more honest representation of what happened and it means the HEAD of the code is accurate.

Disclaimer

This is theoretical. The code is available at ngauthier/awesome_site and it does work. However I have never done this on a production application, so take care if you decide to go for it. I was just curious about what the process would be like.

Happy monday!

Raphael.js + Backbone.js + Traer.js

2012-02-23T00:00:00+00:00

Raphael.js is a cool vector graphic drawing library for javascript. It uses SVG (VML on IE) to draw just about anything, and provides lots of easy helper methods. The coolest thing about SVG is that since it’s XML it can be inserted directly into the dom, so every element has its own dom node.

One of my favorite things about Backbone.js is that the only assumption it makes about Views is that they will have a dom element, el. So, I thought it would be really cool if you used a Backbone.js view to render Raphael objects.

Now, to make this extra awesome, I wanted to use the physics engine Traer.js.

This was the result:

Video

Demo

Visit demo page

There are four levels here:

Particle: managed by Traer
Model: Backbone model wrapping the particle
Element: svg element managed by Raphael
View: Backbone view wrapping the model and the element

Step 1. Modding the physics engine

This first thing I had to do was make two small changes to Traer:

function Particle(mass) {
  this.position = new Vector();
  this.velocity = new Vector();
  this.force = new Vector();
  this.mass = mass;
  this.fixed = false;
  this.age = 0;
  this.dead = false;
  /* Attach Backbone's events to particles */
  _.extend(this, Backbone.Events);
}

RungeKuttaIntegrator.prototype.step = function (deltaT) {
  /* lots of physics stuff redacted */

  /* Change the conditional for skipping fixed nodes so that the
   * following runs for all particles after they've been computed:
   */
  p.trigger('change'); // p is the particle

Now, particles will fire a change event after a physics tick.

Step 2. Backbone Model based on Traer

Exobrain.Node = Backbone.Model.extend({
  // expects a particle and a size parameter
  initialize: function(attributes) {
    // when the particle changes, we fire a change
    this.get('particle').on('change', function() {
      this.trigger('change'); 
    }, this);
    // keep track of children
    this.children = new Exobrain.NodeList();
  },
  // proxy some particle attributes
  x:    function() { return this.get('particle').position.x; },
  y:    function() { return this.get('particle').position.y; },
  size: function() { return this.get('size');                },
  particle: function() { return this.get('particle'); },

  // Make a new node under this one
  createChild: function() {
    var mass = 0.4;
    // random x and y
    var x = this.x() + Math.random() * 50 - 25;
    var y = this.y() + Math.random() * 50 - 25;
    var z = 0;
  
    // make the physics particle
    var particle = Exobrain.makeParticle(mass, x, y, z);
    // make a model
    var node = new Exobrain.Node({particle: particle, size: 10});
    // add it to our child list
    this.children.add(node);

    // setup a spring between the new node and us
    var spring = 0.02;
    var damping = 0.10;
    var length = 120;
    Exobrain.makeSpring(this.particle(), node.particle(), spring, damping, length);

    // fire a child event for the node we created
    this.trigger('child', node);
    // when the node gets a child, make a link with this node
    node.on('child', this.link, this);
    // also, any child of our child is also our child, so all nodes link up the tree
    node.on('child', function(child) { this.trigger('child', child) }, this);

    return node;
  },
  // make a repulsive force with a node
  link: function(node, strength) {
    if (node === this) { return; }
    /* strength scales exponentially with distance, this keeps the system
     * from "folding up"
     */
    if (strength === undefined) {
      strength = -100.0;
    } else {
      strength = strength * 2.0;
    }
    var distanceMin = 5.0;
    Exobrain.makeAttraction(this.particle(), node.particle(), strength, distanceMin);
    this.children.each(function(child) { child.link(node, strength) }, this);
  }
});

/* Collection automatically links siblings */
Exobrain.NodeList = Backbone.Collection.extend({
  model: Node,
  initialize: function(models, options) {
    this.on('add', this.linkNodes, this);
  },
  linkNodes: function(node) {
    this.each(function(other) {
      other.link(node);
    }, this);
  }
})

Step 2. Raphael-based view

Exobrain.NodeView = Backbone.View.extend({
  // make a child on click
  events: {
    'click': 'createChild'
  },
  initialize: function() {
    // draw a node, set "element" to the raphael node
    this.element = Exobrain.drawNode(
      this.model.x(), this.model.y(), this.model.size()
    );
    // set the dom element to the raphael element's dom node
    this.setElement(this.element.node);
    this.model.on('change', this.render, this);
  },
  // render will just move the node, not re-draw it
  render: function() {
    this.element.attr('cx', this.model.x());
    this.element.attr('cy', this.model.y());
  },
  // Create a new model and a view for it
  createChild: function() {
    new Exobrain.NodeView({model: this.model.createChild()})
  }
});

var Exobrain = {
  /* redacted */

  // draw a raphael node
  drawNode: function(x,y,size) {
    // make a circle on the Raphael "paper"
    var el = this.paper.circle(x,y,size);
    // fill it with red
    el.attr({fill: 'red'});
    return el;
  }

  /* redacted */
};

Notes

This is a great example of my style of Backbone programming. Make a namespace and put class definitions in it. Then only create instances within an onReady closure. No instances on the global namespaces!

The only exception was that I have a global physics engine set to the Exobrain object as well as a Raphael paper for drawing objects. I should probably make Exobrain a class and instantiate it with a div id, and have it allocate its own engine and paper.

Playing with Ember.js

2012-02-19T00:00:00+00:00

Today I played around with Ember.js. I wanted to make my own Pomodoro timer, and I figured it would be a good way to try it out.

One of the reasons I’m really excited about Ember is that its goal is to cut down on boilerplate code especially in regards to keeping views up to date.

I’d like to mention that I have no idea whether or not I really did this right. Ember’s docs are in a state of flux as the framework has yet to hit 1.0 and so much is changing. Also, I was particularly confused by the source code, as it seems to be in packages that don’t always have a clear hierarchy. For example, the core Ember.Application class is under packages/ember-views/lib/system/application.js. So, I did dig around a while and I hope I am pretty close to the mark.

First up, Ember’s HTML

<script type="text/x-handlebars" data-template-name='timer'>
  <div class='timeleft'>{{timeLeft}}</div>
  <a href="#" class='btn btn-large btn-primary' {{action "pomodoro"}}   >Pomodoro</a>
  <a href="#" class='btn btn-large'             {{action "shortBreak"}} >Short Break</a>
  <a href="#" class='btn btn-large'             {{action "longBreak"}}  >Long Break</a>
  <a href="#" class='btn btn-large btn-danger'  {{action "stop"}}       >Stop</a>
</script>
<h1>Pomodoro</h1>
<div id='timer'></div>

We define a reusable template as a handlebars script tag and give it a name. Then lower down is a #timer div that we’ll bind to. This is not the standard Ember way of placing a template. In the examples, you’ll see:

<h1>Pomodoro</h1>
<script type="text/x-handlebars">
  {{view Todos.MainView}}
</script>

However this really rubbed me the wrong way because I hate global variables (Todos.MainView). This template itself is kind of a global since it’s directly in the dom. I prefer to give the template a global variable name (data-template-name) and have the view reference it.

My other alternative would be to define the template in the code, but that would take it farther from the html it’s being used in. I would like feedback and advice on this, please!

Next, is Ember’s Timer (note: all the JS is in a jQuery onReady closure, hence var not window., because I hate globals):

var Pomodoro = Em.Application.create();

var timer = Ember.Object.create({
  timeLeft: "25:00",
  totalTime: 25*60*1000,
  
  start: function(time) {
    var _this = this;
    this.reset(time);
    this._startedAt = new Date();
    this._intervalId = setInterval(function() { _this.updateTimeLeft.apply(_this); }, 100);
  },
  
  reset: function(time) {
    clearInterval(this._intervalId);
    if (time) {
      this.set('totalTime', time*60*1000);
    }
    this.set('timeLeft', msToString(this.get('totalTime')));
  },
  
  updateTimeLeft: function() {
    var now = new Date();
    var diff = now - this._startedAt;
    this.set('timeLeft', msToString(this.get('totalTime') - diff));
  }
});

We’re using Ember’s Object base class, and we are making use of the getters and setters, which let us bind in the view. Here’s Ember’s View:

Ember.View.create({
  templateName: 'timer',
  timer: timer,
  timeLeftBinding: 'timer.timeLeft',

  pomodoro: function(){
    this.timer.start(25);
  },
  shortBreak: function() {
    this.timer.start(5);
  },
  longBreak: function() {
    this.timer.start(15);
  },
  stop: function() {
    this.timer.reset();
  }
}).appendTo('#timer');

We’re binding the View to the template we created, we’re setting a local var timer to the timer var we instantiated when making our timer. Then we bind the view’s timeLeft attribute to the timer’s timeLeft attribute. This will automatically update the dom, when the model’s attribute changes. The special naming timeLeftBinding means that it’s a binding.

The next four methods are triggered by clicking the template links with the actions of the same name. So if you look at the template, each link has an action, and that action is a function to call on the view.

I looked for a controller to use, since the view is really being both a controller and a view here (actually, the model does some of the view work too, as it formats its timestamp). However, the only Ember controller I could find was an ArrayController, which is actually just an Array with some event bindings. So, I guess they don’t have the C of MVC yet.

All in all, there is not a lot of boilerplate code here, so nice job guys!

Now, I can’t leave it there, because I’ve done a ton of work with Backbone.js, and it seems like Ember is calling out Backbone and others like it “obvious low-level event-driven abstractions.”

So, I implemented the same timer in Backbone to see how much boilerplate I had to write:

Backbone’s HTML:

<h1>Pomodoro</h1>
<div id='backbone-timer'>
  <div class='timeleft'></div>
  <a href="#" class='btn btn-large btn-primary pomodoro'   >Pomodoro</a>
  <a href="#" class='btn btn-large             short-break'>Short Break</a>
  <a href="#" class='btn btn-large             long-break' >Long Break</a>
  <a href="#" class='btn btn-large btn-danger  stop'       >Stop</a>
</div>

Unlike Ember, this is direct html I am writing right in the dom, not script tags. If there’s a way to bind Ember to existing elements, let me know!

Next, we have Backbone’s Timer:

var timer = new (Backbone.Model.extend({
  defaults: {
    timeLeft: "25:00",
    totalTime: 25*60*1000
  },

  start: function(time) {
    this.reset(time);
    this._startedAt = new Date();
    this._intervalId = setInterval(_.bind(this.updateTimeLeft, this), 100);
  },

  reset: function(time) {
    clearInterval(this._intervalId);
    if (time) {
      this.set('totalTime', time*60*1000);
    }
    this.set('timeLeft', msToString(this.get('totalTime')));
  },
  
  updateTimeLeft: function() {
    var now = new Date();
    var diff = now - this._startedAt;
    this.set('timeLeft', msToString(this.get('totalTime') - diff));
  }
}))();

The first difference is we don’t have to init an application, because Backbone is a library, not a framework. The second difference is that Backbone has a forced separation between get/set attributes and object attributes. Thus there is a defaults hash for the get/set attributes, as opposed to declaring them on the object. I find this to be more declarative than Ember’s syntax.

The next difference is that since underscore is a dependency, I used _.bind for the timer. But we could do this in Ember too, if we brought in underscore.

Other than that, these objects are almost exactly the same.

Let’s look at Backbone’s View:

new (Backbone.View.extend({
  events: {
    'click a.pomodoro'   : 'pomodoro',
    'click a.short-break': 'shortBreak',
    'click a.long-break' : 'longBreak',
    'click a.stop'       : 'stop'
  },

  initialize: function() {
    this.model.bind('change:timeLeft', this.updateTimeLeft, this);
  },

  render: function() {
    this.updateTimeLeft();
    return this;
  },

  updateTimeLeft: function() {
    this.$('.timeleft').text(this.model.get('timeLeft'));
  },

  pomodoro: function() {
    this.model.start(25);
  },

  shortBreak: function() {
    this.model.start(5);
  },

  longBreak: function() {
    this.model.start(15);
  },

  stop: function() {
    this.model.reset();
  }
}))({el: $('#backbone-timer'), model: timer}).render();

Now we’re seeing some difference! In Backbone, we have to bind all the link actions ourselves on the view instead of on the template. This is definitely more legwork on the View’s part, but remember is means that template could drive different views with different behavior, because the behavior is defined in the view, not the template. Personally, I like to keep code out of my templates.

Next, we have to manually bind the change event on the model to a method that updates the text of that dom element. Definitely boilerplate. Ember does this by automatically creating wrapper divs like <div id="ember150" class="ember-view"> that wrap your templated attributes and auto-updates them.

After that, the methods are the same. Then when we initialize we give it a dom element to bind to, and we have to manually tell it to render the first time.

At this point, I’d like to pull out a refactoring from Recipes with Backbone from the Fill-In Views chapter. Check this out:

/* EDIT: updated to use #constructor. Thanks Tim Branyen! */
Backbone.BoundView = Backbone.View.extend({
  constructor: function(options) {
    Backbone.View.apply(this, arguments);

    this.model.bind('change', this.updateBoundAttributes, this);
    this._oldRender = this.render;
    this.render = function() { this._oldRender(); this.updateBoundAttributes(); };
  },

  updateBoundAttributes: function() {
    _(this.bindings).each( function(value, key) {
      this.$(key).html(this.model.get(value))
    }, this);
  }
});

This Backbone.BoundView is a subclass of Backbone.View and it uses a bindings object to automatically update model attributes on change. That means we can simplify our view to this:

  new (Backbone.BoundView.extend({
    events: {
      'click a.pomodoro'   : 'pomodoro',
      'click a.short-break': 'shortBreak',
      'click a.long-break' : 'longBreak',
      'click a.stop'       : 'stop'
    },

    bindings: {
      '.timeleft': 'timeLeft'
    },

    pomodoro: function() {
      this.model.start(25);
    },

    shortBreak: function() {
      this.model.start(5);
    },

    longBreak: function() {
      this.model.start(15);
    },

    stop: function() {
      this.model.reset();
    }
  }))({el: $('#backbone-timer-2'), model: timer}).render();
});

We still have the event bindings, but now all the model updating code has been reduced to a bindings object that maps dom classes to model attributes. Much better!

All-in-all, I have to say I’m not super impressed with Ember’s view and model bindings. But, I could certainly not be taking full advantage of them at this time. I did experiment with computed attributes, and those were neat. I do like the idea that functions can behave as properties as well. Also, we didn’t get into any of Ember’s other awesome features like states.

I was planning on making my Pomodoro timer in Ember, but after having some issues with the docs and the source, and not finding many satisfactory examples, I’m going to stick with Backbone for now. But you can be sure when Ember hits 1.0 I’m going to check it out!

Quick Ruby Tests with Bash

2012-02-16T00:00:00+00:00

In Ruby on Rails development, we have great gems like Guard that will re-run tests or other tasks based on changing files. I was interested in finding something more lightweight but less configurable and flexible that I could use on smaller projects.

I ended up writing this quick bash script that I put on my path called live:

#/usr/bin/env sh
# Usage "live command"
clear
$*
while inotifywait -qr -e close_write *; do clear; $*; done

This script takes a command as an argument and re-runs the command whenever any file in the current directory changes. So, you can simply re-run the tests of a project by running live rake. Or, you could re-output a project directory with live tree. It’s like watch except evented on write changes.

The next thing I noticed while using this on larger (rails) projects is that project boot is slow and the tests run in different groups that each boot the environment.

Note the raw time it takes to run the project’s tests:

$ time rake
Loaded suite /home/nick/workspace/giftsmart/.bundle/gems/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started
....
Finished in 0.588015 seconds.

4 tests, 30 assertions, 0 failures, 0 errors, 0 skips

Test run options: --seed 54473
Loaded suite /home/nick/workspace/giftsmart/.bundle/gems/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started
...............
Finished in 3.679384 seconds.

15 tests, 77 assertions, 0 failures, 0 errors, 0 skips

Test run options: --seed 45532

real  0m41.495s
user  0m38.502s
sys   0m1.928s

Seriously?! 41.5 seconds to run just over 4 seconds worth of tests? WAT? And this is using ruby-1.9.3-falcon!

So, my first step was to merge the environments and optionally skip db:reset:

# lib/tasks/testing.rake
namespace :test do
  desc 'Run tests quickly by merging all types and not resetting db'
  Rake::TestTask.new('fast') do |t|
    t.libs << 'test'
    t.pattern = "test/**/*_test.rb"
  end

  namespace :fast do
    desc 'Run tests quickly, but also reset db'
    task :db => ['db:test:prepare', 'test:fast']
  end
end

This provides rake test:fast:db which reset the db and runs the tasks merged as one, and rake test:fast which merges the tasks and doesn’t reset the db. Here’s the result:

$ time rake test:fast:db
Loaded suite /home/nick/workspace/giftsmart/.bundle/gems/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started
...................
Finished in 3.752755 seconds.

19 tests, 107 assertions, 0 failures, 0 errors, 0 skips

Test run options: --seed 57655

real  0m28.430s
user  0m26.138s
sys   0m1.216s

OK, that’s a 46% improvement. Now, without the db:

$ time rake test:fast
Loaded suite /home/nick/workspace/giftsmart/.bundle/gems/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_test_loader
Started
...................
Finished in 3.790390 seconds.

19 tests, 107 assertions, 0 failures, 0 errors, 0 skips

Test run options: --seed 53405

real  0m21.623s
user  0m20.525s
sys   0m1.020s

Better, a 92% improvement! But it’s still 3.8 seconds worth of test at 21.6 seconds. Lame.

Now it’s time for drastic measures. When I’m running tests, I see this process:

/usr/bin/ruby1.9.1 -Ilib:test /path/to/my/project/.bundle/gems/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/rake_test_loader.rb test/unit/**/*_test.rb

This is because Rake::TestTask shells out to ruby to keep the environment clean. This works great for projects with minimal boot times. But in my case, I have a much larger Rails boot I have to worry about. This means I’m still booting Rails twice!

So, I made a short ruby script that gives me direct access to the rake test loader for the current project:

#!/usr/bin/env sh
ruby -Ilib:test `bundle list rake`/lib/rake/rake_test_loader.rb $*

Now, here are my results:

$ time rtest test/**/*_test.rb
Run options:

# Running tests:

...................

Finished tests in 3.070074s, 6.1888 tests/s, 34.8526 assertions/s.

19 tests, 107 assertions, 0 failures, 0 errors, 0 skips

real  0m9.579s
user  0m9.089s
sys   0m0.324s

There we go! That’s 333% faster! Now the total time to run my tests is 1xRails and 1xTest. This is probably the minimal boot time I could get without keeping the environment hot loaded. The best part is, I can combine my two scripts:

$ live time rtest test/**/*_test.rb

Now I have a very fast and minimal live-updating setup written in 4 lines of bash that is portable across projects.

Ruby and the Web

2012-02-14T00:00:00+00:00

Here's a talk I gave at Bmore on Rails (Baltimore's Rails user group). I talk about MVC vs Model 2 and how they apply to Rails. I talk about Frameworks vs Libraries and White Box vs Black box. Also I talk about a theoretical framework I experimented with to try to implement a more object oriented system for dealing with the web in Ruby (based on Rack).

Enjoy.

Simple ruby setup on ubuntu

2012-01-30T00:00:00+00:00

Today I setup a new development machine. My preferred OS is Xubuntu, which is Ubuntu + XFCE (a light window manager). In the past I’ve used RVM and have been happy with it, except for one thing: compiling.

I used to use Gentoo, which is a linux distribution in which all software is installed by downloading the source and compiling. This is a brutal and intense introduction to linux, and I learned a hell of a lot using Gentoo in college. However, I feel like I’ve paid my dues in linux boot camp and now I want to simply install binary packages and have a ruby setup lightning-quick.

So, here were my steps to get ruby (and postgres) setup:

Ruby

Install ruby, rubygems, a few dependencies and postgres: libxslt1-dev libxml2-dev build-essential g++ ruby1.9.1-dev postgresql libpq-dev.

Bundler and gem-scoping

The only remaining issue is how to manage sets of gems between projects. I chose to do something here which leverages one of bundler’s great features to achieve a halfway gemset solution. It is not exclusive of gems installed globally, but it gives priority to local binaries over global ones.

In your .bashrc (or whatever shell init script):

alias bundle-bootstrap="bundle install --binstubs=.bundle/bin --path=.bundle/gems"
export GEM_HOME=$HOME/.gems
export PATH=.bundle/bin:$GEM_HOME/bin:$PATH

The bundler alias will put binstubs (shell scripts that run a gem’s binary) into the current directory’s .bundle/bin. It also says to store the gem sources in .bundle/gems. This means that as soon as you leave this directory, it’s like those gems aren’t even installed!

The second line puts the current directory’s .bundle/bin as the highest priority for finding binaries, followed by my home directory’s gems binary folder. This means that local gem binaries take precedent. That means no more bundle exec.

The third line sets up my gem home so if I just run gem install they go to my home. This is where I put bundler, jekyll, heroku, etc.

Setting up a project

So, when I setup a new project, it looks like:

git clone git@github.com:user/repo.git
cd repo
bundle-bootstrap

Now everything is installed and setup. Binaries are available but don’t contaminate any other projects.

Hope this helps all my fellow linux rubyists on having a fast and clean ruby install!

xoxo @ngauthier

P.S.: if you need ruby 1.8.7, check out rbenv, which is now in APT as of 12.04. You can install ruby 1.8 from APT then use rbenv to switch rubies. I haven’t used it myself but it seems to be what people prefer.

Galaxy Nexus First Impressions

2012-01-04T00:00:00+00:00

I got a Samsung Galaxy Nexus yesterday and @reillyhawk asked me to share a review. Disclaimer: I’ve had this device for 20 hours.

It’s big, but it’s the same weight as my old HTC Incredible (135g vs 130g). It has an extra inch of screen space (4.65” vs 3.7”). The feeling in your hand is like a well balanced sword. Seriously.
It’s so big my thumb can’t reach the opposite corner with the phone in the crook of my palm. The opposite corner is a common place for non-standard menus. Dear apps: use ICS menus! (bottom right instead of top right).
It’s really fast. The screen is very responsive. The throw-weight of scrolling is increased (meaning it doesn’t coast as far for small throws) and it feels good.
Swype isn’t available yet, even though I’m in the beta.
The keyboard is easy to type on because of the extra screen size.
ICS’s app folders are nice.
Calendar widget is sweet.
32G on-device memory means no sd card needed, which is nice.
I haven’t found a single piece of crapware yet.
I’ve only used wifi (at home) so I can’t comment on 4G speeds.
Browser is snappier, the zoom is much quicker. I love android’s text reflow.
I miss Cyanogen’s tethering and ability to change the number of home screens (I like having one immovable screen). I haven’t looked into root options yet, and won’t for a couple weeks to make sure I’m keeping the phone and there are no problems.
At minimum brightness when looking at solid colors or gradients, AMOLED banding is visible. I’m not too annoyed though. Really only happens and night and I should put the phone down and pick up the kindle :-)
It comes with an earbud and mic combo. I haven’t tried them out yet.
Headphone jack on the bottom is perfect, because you always put your phone into your pocket face-down, so the cables should come out the bottom.
The lock button is opposite the volume button, so when you push lock you have to be careful not to brace against the volume keys. I can’t believe how many devices do this. My timex watch has the same issue. Put buttons opposite spaces in the frame!
No hardware buttons is not a problem for me at all. I love the fact that it’s just a big hunk of screen. Looks awesome. Netflix is beautiful.

That’s it for now!

Open for Business

2012-01-02T00:00:00+00:00

I’m excited to announce that I’m now open for business!

If you don’t already know me, let me introduce myself: my name is Nick Gauthier and I am a Ruby and Javascript coder and I specialize in web development with the Rails framework. I particularly enjoy performance-tweaking applications, designing application architecture, and writing clean, object-oriented, and test-driven code. I’ve spoken at a couple of conferences including RailsConf 2011, and I recently wrote an e-book called Recipes with Backbone.

In 2012 I’m starting a new chapter in my life by striking out on my own. I plan to focus on three things:

1. Consulting and Freelance Development

Does your team need a boost? I can help out with planning, brainstorming, and kickstarting your project. Or maybe it’s getting a little clunky and needs a tune-up?

Or, do you need someone to build your prototype or MVP? Someone that can work quickly and produce high quality code that will make the next developer smile?

2. Technical Training

I’m going to be teaching a variety of courses as an instructor at Jumpstart Lab. I’m also available for private training in Ruby on Rails and Backbone.js. I’m also considering some unorthodox methods like holding open office hours, pair programming, or maybe an apprenticeship program.

3. New Projects

Last, but not least, I’m leaving some time open for new projects. I’ll be exploring some ideas of my own but would welcome collaboration. I’m especially interested in working with designers in the Baltimore area.

Get in touch

Let’s get in touch. I’m ngauthier on Twitter, Github, and Gmail, so drop me a line!

Availability

Recipes with Backbone Released!

2011-12-07T00:00:00+00:00

Chris Strom and I have finished our e-book on Backbone JS: "Recipes with Backbone". It is now available at http://recipeswithbackbone.com for $24.

The book is targeted at the intermediate to advanced level backbone developer, but that's not to say beginners won't get anything out of it. To quote the site:

This is not the definitive guide to Backbone.js.
This is not an introduction to Backbone.js.
This is the book you read after you read the tutorial.
This is the book that teaches you to kick ass with the hottest Javascript MVC framework around.
This is Recipes with Backbone
Preview the table of contents

So if you want to learn more about backbone js, buy it!

Mocking on Rails

2011-11-03T00:00:00+00:00

Gregory Moeck's awesome post Stubbing is Not Enough got my brain back on the subject of mocking. Readers of this blog may note that I had quite a rant against mocking almost a year ago and Gregory posted a response. I think the result of that post and the discussion that ensued was not that mocking and/or stubbing were bad practices, but that when they are applied inappropriately they can quickly deteriorate the tests and the design of an application.

After reading Gregory's article, I wanted to revisit the state of mocking in Rails applications. One of the things I noticed in his article was that he was addressing some ruby classes and their interactions. He enforced the OO concepts of message passing and how mocks are better than stubs at testing very OO code. While I really like his solution, something was nagging at me: how can I do this in Rails?

For example, here is the stock rails scaffold controller and functional test:

class PostsController < ApplicationController
  # GET /posts
  # GET /posts.json
  def index
    @posts = Post.all

    respond_to do |format|
      format.html # index.html.erb
      format.json { render :json => @posts }
    end
  end
end

class PostsControllerTest < ActionController::TestCase
  setup do
    @post = posts(:one)
  end

  test "should get index" do
    get :index
    assert_response :success
    assert_not_nil assigns(:posts)
  end
end

What can I mock? In Gregory Moeck's example, the ticket reservation object was passed in to the ticket machine interface (dependency injection) so we could easily mock the ticket reservation role and assert that the ticket machine interface interacted with it properly. He also intentionally doesn't touch the @current_display variable because it is internal to the system.

In our Rails controller and functional test, we can observe the following:

The actions taken by the controller are internal, not based on roles it should interact with
The functional test is testing internal state (assigns) and not messages passed by the controller

My gut at this point says I'm on an integration boundary between the user and my internal system. So that means I would have an integration test on the controller. But still, the signature of a rails controller to call activerecord and render doesn't seem to lend itself to encapsulation and mocking.

At this point I attempted to write what I thought was a change to the way controllers work using dependency injection and object composition, but I failed at it. So I'm leaving this post with some open questions:

What is the proper way to test rails requests with mocking?
What is the proper way to do an integration test, and how deep should it go (i.e. beyond the point that you have covered with unit tests)?
Are there other web frameworks out there with great encapsulation as a best-practice?

Super extra bonus points if you post links to open source projects that have a test suite that actually do these things well. Thanks.

Comments

Greg Moeck

A good number of people have asked me that question since I wrote the post. My general response is two fold.

First, if I'm just doing CRUD reading and writing, I don't really feel the need to have unit tests for the project, so long as I have end-to-end acceptance tests. The logic in Rails is simple enough that a computer could write it, so I don't really feel a high degree of risk there.

However if I'm dealing with a complex domain then I tend to separate out my domain layer from Rails, and treat my controllers as ports (from Alistair Cockburn's ports and adapters architecture) into and out of the web. I will have them talk to an adapter within my domain, which is all well encapsulated and heavily unit tested. I generally don't unit test the controller or the view layer , and just let my end-to-end acceptance tests ensure that everything is plugged together correctly. However if something is particularly hairy I will cover the rails part in an integration test.

The basic approach is similar to what Eric Evans calls the "Anticorruption Layer" in Domain Driven Design (page. 366).

I'm working on a sample application for an auction house which you can at see at https://github.com/gmoeck/auction_house. There isn't much to see in the actual domain layer yet, but it will give you a general idea.

Nick Gauthier

awesome, thank you for chiming in. One thing I've been mulling over is how much of the variability in the controller's form=>params=>domain model do you cover w/ acceptance tests?

If a form has a bunch of params, some options (radio buttons) some non-required some required, you have to make sure the fields are wired up some how to get passed in to your domain model. Do you test all the paths?

Greg Moeck

I personally tend to try and leave that stuff to the adapter to decide what to do with and think of the controller as more of just the way that I receive and send data. That's why i don't generally don't feel the need to unit test them because the controller is just getting all the data relavent to the request and handing it off to the domain to decide what it means. My adapter object will generally then read the parameters and send messages into the domain in the domain's language according to whatever the parameters mean. This isolates my actual domain objects from changes in the system and allows me to plug in the same domain into another delivery mechanism so long as I write an adapter for it.

The more complex side is when the controller then queries the "view side" of the domain to get a response object, which then passes that data to the view, or renders an error or something. This is where my integration sort of tests will sometimes come in if I want to unit test that logic because it is getting complex.

I certainly don't feel like I have this all figured out, and I'm personally excited that people are starting to think more along these lines because I feel like the Rails side of the equation is going to clean up a bit in the immediate future. Sometimes I do feel like Rails is a bit of an overkill though since I can use Rack and accomplish most of what I'm wanting to do with the "web framework" part of my application.

Nick Gauthier

cool.

So generally your domain objects play into the controller like:

if obj.create(params); head :ok; else; render :json => obj.errors

I've been looking more into goliath and it seems to strike a nice balance of rack-like directness (no magic) and also basic HTTP API niceties like content encoding.

Personally, I've been on an "acceptance test everything no unit tests" kick. It has proven to:

1) make very reliable software
2) make very slow test suites

Sam Goldman

This is perhaps an unfairly literal response to a contrived example, but if you are writing controllers like that, another option would be to use a library like inherited_resources, which is already well-tested, and just test the happy path in an integration.

Nick Gauthier

well yeah if you're using the stock scaffold you don't need to test it either, since scaffolds are well tested.

Mostly just interested because this is the standard way of writing a controller action.

As an aside, I'm not a big fan of inherited resources. I prefer scaffolding. Tracking down bugs and determining behavior w/ inherited resources is a pain. I'd rather type it out.

Alpha of "Recipes with Backbone" Released

2011-10-01T00:00:00+00:00

Chris Strom and I have released the alpha of Recipes with Backbone. It's an e-book containing intermediate to advanced design patterns and best practices for Backbone.js. Grab it now for 50% off the list price. You'll get future versions of the book for free when you buy the alpha.

http://recipeswithbackbone.com

Thanks!

Using Exceptions to manage control flow in Rails Controllers

2011-09-26T00:00:00+00:00

Ah yes, the Rails Controller, a source of much contention among Rails developers. So many different ways to manage control flow, load objects, respond in standard and erroneous ways. My opinion up until recently was "I'll just put a bunch of conditionals in there for different situations."

Recently, I've been working more on API endpoints and so responding with nice error messages has been more of a priority. I started using Exceptions more throughout my code thanks to Avdi Grimm, and I recently wrote and action that I'm particularly proud of. Check it out:

# This controller's job is to exchange twitter credentials for Shortmail credentials
class TwitterReverseAuthController < ApplicationController
  # First, let's make our own subclass of RuntimeError
  class Error < RuntimeError; end

  def api_key_exchange
    # Here are our required parameters. If any are missing we raise an error
    screen_name = params.fetch(:screen_name) { raise Error.new('screen_name required')  }
    token =       params.fetch(:oauth_token) { raise Error.new('oauth_token required')  }
    secret =      params.fetch(:oauth_secret){ raise Error.new('oauth_secret required') }

    # OK now let's authenticate that user. If we can't find a valid user, raise an error
    @user = User.by_screen_name(screen_name).where(
      :oauth_token => token,
      :oauth_secret => secret
    ).first or raise Error.new('user not found')

    # Now we'll build a device. I'm not catching an exception on create! here because
    # It should never fail. (I.e. a failure is actually a 500 because we don't expect it)
    @device = Device.find_or_create_by_token!(
      params.slice(:token, :description).merge(:user_id => @user.id)
    )

    render :json => { :api_key => @device.api_key }

  # Now I can simply catch any of my custom exceptions here
  rescue Error => e
    # And render their message back to the user
    render :json => { :error => e.message }, :status => :unprocessable_entity
  end
end

Here are the things I really like about this solution:

The happy path is really clear because there's no if/else branching
Errors are really obvious because I'm raising an exception (as opposed to "else render json that complains" which looks like a render which is not immediately apparent as a failure)
It's super easy to handle the errors in the same way. Instead of repeating the json render with a different message all throughout the method (i.e. it's DRY)

How's this look to you? How do you organize controller control flow?

Comments

Jim Gay

Interesting.
But this will only catch one error at a time, so you'd return the result when the first raise is hit even if multiple parameters are missing.

And Avdi points out that there is performance overhead with raising exceptions. Did you opt not to worry about that? Why not just collect an array of errors and test for their presence?

Nick Gauthier

I think the performance overhead isn't an issue here. I expect users only to get it wrong while they're figuring it out. If this was on a form or something, then I'd raise an exception if the user is invalid, and I'd have all the params right there. But this is an API call, so it's more of a "while I'm developing I get errors".

Also keep in mind it's more performant the sooner I can bail out of processing the request :-)

Collecting and returning all errors makes sense for fetching the params, but if I can't find a user I can't proceed with the rest of the call. So I have to stop execution anyways.

codecraig

I've used a similar approach although I was a little bit more granular in the HTTP response codes. For example I'd send back a 404 if an entity was not found.

james

- There's too much going on this method.
- From my limited reading of Avdi's blog posts / presentation slides, I don't feel that he would agree with using exceptions this way.
- If you really want to subclass RuntimeError, I would choose a more descriptive name.

An alternative: https://gist.github.com/1243758

Nick Gauthier

@codecraig great point on the 404. Could be an additional subclass that has the status set.

@james I'm looking up a user and creating a device. This is the minimum amount that can be done in a nested route on a create action. I disagree with your extraction because it will be only used in this single situation and obfuscates the method.

I might make a User.authenticate_via_oauth(screen_name, token, secret) that returns nil or a user, but creating a separate class is overkill here IMO.

Avdi Grimm

Assorted thoughts...

* So many wonderful fetches! My cup overflows.

* Nice use of 'or' as a statement modifier too, putting the error case last, where it belongs :-)

* Maybe I'm missing something... why are you explicitly instantiating the exceptions? Why not 'raise Error, "some message"'?

* Because this is in an API endpoint, it makes sense to use exceptions liberally. We expect humans to make occasional mistakes. Conversely, we expect API clients to be fixed when they make mistakes, and then to never make that mistake again. We also generally don't need to present "Here's what you said, maybe you meant something else..." type feedback to robots, so we don't need to worry about keeping context around that the exceptions might throw away.

* As james pointed out, I do think there's a lot going on in this method. E.g. I personally don't think a #where() call has any business in a controller, and then you've got a #first on the end of that, which makes it a third-order digression into querying minutia.

@Jim: Exception performance is not on the order to worry about in a case like this. It only becomes a worry inside tight loops. Here network latency is going to drown any latency the exceptions add.

Nick Gauthier

Forgot about "raise Error, "message"" :-)

Yeah the "where" on user should be "User.authenticate_via_oauth(screen_name, token, secret) => user or nil"

Thanks for the feedback everyone. Glad to know the controller is still an interesting area to experiment with.

bryanl

Why didn't you just use a goto? It would be more explicit, and the exact same thing you are trying to accomplish here.

Nick Gauthier

Lack of self confidence.

Brian Cardarella

http://i.imgur.com/HDhaa.png

Brian Cardarella

That was a reference to spaghetti code incase it was over anybody's head.

Nick Gauthier

I made a gist:

https://gist.github.com/1245259

fork it!

Brian Cardarella

Nick,

https://gist.github.com/1245327

Nick Gauthier

@Brian:

https://gist.github.com/1245342

your incorrect assumption is that every api call fails. When 1 per 1 million calls fail, exceptions are 3% slower, which is acceptable for readability purposed.

Nick Gauthier

I wanted to log this response post here for people reading the comments:

http://www.enlightsolutions.com/articles/catch-all-exception-handling-is-not-flow-control/

rubiii

Nick, your solution is pretty interesting. Thanks for posting this.

It feels like you're using Exceptions for flow control. At least I don't see missing parameters as an exceptional case. It's possible and expected to happen. That's why we test for it.

rubiii

Ok, so now that I've actually read the title of your post ... ;) Using Exceptions for control flow feels like writing goto statements again?!

Patrick

I just posted a deeper refactoring of this code and thought it might be of interest. I'm curious to get others' thoughts on this approach.

Thanks.

Nick Gauthier

@Patrick thanks! You're the third person to suggest a domain model as a solution. I like the idea of making the model encapsulate the multiple actions and have the controller simply perform the standard Create action on the domain model.

How I test EventMachine

2011-09-20T00:00:00+00:00

EventMachine's asynchronous and evented nature can be pretty tough to test. Here are some simple Test::Unit helpers I use along with a sample example:

def eventmachine(timeout = 1)
    Timeout::timeout(timeout) do
      EM.run do
        EM.epoll
        yield
      end
    end
  rescue Timeout::Error
    flunk 'Eventmachine was not stopped before the timeout expired'
  end

This is a helper that runs eventmachine in a timeout so that if it hangs the test suite flunks out after a second. Very handy.

def set_em_steps(*steps)
    @@_em_steps = *steps
  end

  def em_step_complete(step)
    @@_em_steps.delete(step)
    EM.stop if @@_em_steps.empty?
  end

This is a flow-control helper to make sure I complete all the steps I expected. Sometimes you run two chunks of EM code and then make assertions in the callbacks. Generally, you call EM.stop in your last callback, but what if they don't chain one after another? Then you have to call stop after both have finished. These two helpers just make it so that I can define my steps, then mark each as completed. They stop EM once all the steps are completed.

Example

Here is an example test from Shortmail.com's test suite for iPhone push notifications:

test 'send a push notification to the push daemon' do
  token = Factory.next :iphone_device_token
  message = 'hello world'
  # define two steps that must be completed before stopping
  set_em_steps :payload, :notification

  # run the following code, but time out after 1 second
  eventmachine do
    # MockServer is a fake iphone push server (pretending to be apple) 
    # it yields responses back to the instantiator
    Test::Helpers::PushD::MockServer.listen do |response|
      # Unpack the push
      id, exp, device, payload = PushD::Pusher.unpack(response)

      # make sure that the payload has the right into in it
      assert_equal token, device
      assert_equal message, JSON.parse(payload)['aps']['alert']

      # mark the payload step as complete, meaning we've received
      # and verified it
      em_step_complete :payload
    end
    PushD::WebServer.listen

    # This is another part of code under test.
    # This is how we send a push message
    PushNotifier.notify(token, :message => message) do |success|
      # When the push message is sent it runs the callback block with
      # a boolean and we ensure it's ok
      assert success
      # now make sure that this step is marked as finished
      em_step_complete :notification
    end
  end
end

Now, we can be sure our assertions are run, or it will time out because it won't stop.

How do you test eventmachine?

Backbone JS: View signatures to prevent repaints

2011-03-10T00:00:00+00:00

A nice thing about backbone is being able to bind a view render to new data. But sometimes you get new data, but it's not actually new, it's the same data that has not been updated. This will still cause the view to repaint because the event will fire.

To combat this, I've started putting signatures on my views.

In the initializer for the view:

this.signature = "";

Then in the render method, I do a reduction on the data that drives the view and compare signatures and decide to paint or not:


// Create a signature from the "Posts" that this view renders
var new_signature = _.reduce(posts, function(memo, post) {
  // Store the id and number of comments. This represents what a
  // "changed" post is for the view
  memo.push([post.get('id'), post.get('num_comments')].join(','));
  return memo;
}, []).join('|'); // joined with pipes for each post

// If the signature is the same, end the render
if (new_signature === this.signature) {
  this.signature = new_signature;
  return;
}
// Otherwise store the signature
this.signature = new_signature;
// Your view code down here

This is essentially caching with a custom cache key. Except instead of retrieving a cached value, we leave the dom as-is. This cuts down on repaint "flickering".

jQuery Deferred and Backbone JS

2011-03-09T00:00:00+00:00

Backbone is a really interesting framework, and my favorite part so far is the following idea:

If you use a callback, you're Doing It Wrong

This has held true for me for my development with Backbone so far. When you make data calls to the server, you let the appropriate events notify interested parties when objects are changed or updated or refreshed.

However, with non-data operations, callbacks can be really useful. Today, I needed to animate an object, using jQuery's slideUp. I wanted the slideUp to go hand-in-hand with a deletion of an object. Because slideUp is asynchronous, and the deletion action is asynchronous, I needed a callback to synchronize them. The reason I couldn't do them simultaneously is that when an object is deleted, many view elements refresh themselves, and if the slideUp wasn't finish the dom refreshing would interrupt the slideUp and it looked gross.

So, I need a callback on slideUp to call remove. Here arose an SRP problem: the view should not concern itself with the removal of the model from the collection. One solution is for the view's removal method to take a callback and pass it along to slideUp. But I wanted something more flexible. Enter jQuery Deferred.

Deferred objects let you chain callbacks and return promises as objects. For you Rails readers, imagine AREL had a hot sister written in Javascript that was into AJAX.

So here is my view method that runs slideUp and returns a deferred object:


return $(this.el).slideUp(200).promise().done(
  _.bind(function() { this.remove(); }, this
);

Now when we archive and email, we want to remove the dom element by sliding, then we want to tell the email model to archive itself. Here is the archive method:


this.remove().then(this.model.archive);

Pretty straightforward. We call remove (the previous method) then we tell the model to archive itself. This method is bound to the click event on the archive button using Backbone's view event binding methods. When the model archives itself, Backbone events automatically fire, so other Views can listen for "change" and "remove" to update their elements.

EDIT: changed code reflecting Julian's comments.

Comments

Julian

Using jQuery 1.6, you can considerably simplify your method:

remove: function() {
return
$(this.el)
.slideUp(200)
.promise()
.done(function() { this.remove(); });
}

1.6 adds jQuery.fn.promise() that returns a Promise to observe when a collection has no more animation going on. It's resolved with the collection is was called on as its context and first & only argument.

(sorry for how the code is formatted, Blogger's comments just plain suck :/)

Nick Gauthier

Cool! Yeah I noticed slideUp didn't return a promise, so I didn't know what to do with it.

I think I'd still need the _.bind within the done to bind to the object though, right?

Backbone and Rails Forgery Protection

2011-02-21T00:00:00+00:00

I just had a tough time getting Rails 3 to play nice with Backbone JS, and it turned out to be a simple problem with a simple solution. Backbone was not sending the csrf authenticity token embedded in the page when it sent create/update/delete requests, and Rails was destroying the session when it detected the invalid request.

Here is all the javascript it took to get Backbone to include the token with all requests:


/* alias away the sync method */
Backbone._sync = Backbone.sync;

/* define a new sync method */
Backbone.sync = function(method, model, success, error) {
  /* only need a token for non-get requests */
  if (method == 'create' || method == 'update' || method == 'delete') {
    /* grab the token from the meta tag rails embeds */
    var auth_options = {};
    auth_options[$("meta[name='csrf-param']").attr('content')] =
                 $("meta[name='csrf-token']").attr('content');
    /* set it as a model attribute without triggering events */
    model.set(auth_options, {silent: true});
  }
  /* proxy the call to the old sync method */
  return Backbone._sync(method, model, success, error);
}

Note that this depends on the meta tags being present, which require you to call the helper "csrf_meta_tag" in your rails view for the page (put it in the head).

Comments

rubiii

i'd love to hear more about using backbone with rails. i really like backbone, but to me it still feels like i have to write "a lot" of additional code. routes, models and validations for example.

i'm hacking on retrieving these things from rails, but maybe there's a better way to do this?!

Nick Gauthier

I'm going to put together something larger. It may fit in a blog post, but probably not. It may be a talk, I'm not sure yet.

I don't want to do a tutorial. For me, that's not very interesting. I want to write about backbone "6 months in" because that's where I think it will really shine.

So far, my initial impressions are that the code you write on the front end is code that you don't have to write on the back (especially crazy rails view code).

The most amazing part so far has been that if you wire everything up right by embracing event-driven coding, changes cascade to all the right places automatically. That is *huge* for "6 months in" when you can't afford to update an ajax call in 24 places when you change a template.

rubiii

got it. i also love the evented part of backbone!

so how do you do templating? do you load templates upfront (via something like jammit) and then request json data, do you load the templates and the json and rely on caching or do you have rails render the template and retrieve the complete template?

also, did you run any metrics comparing the performance of your app with and without backbone?

Nick Gauthier

jammit + underscore templates + json api.

Benchmarks are tough because it depends heavily on many factors. However, I can tell you that a specifically heavy page went from 10s just for the Rails request, to 0.6s according to chrome's load time. So, speed improvements are massive.

Also, the user's subjective speed is even higher, because we can manipulate elements on the front-end while doing an asynchronous call behind the scenes. So everything is instant :-)

Maciej Adwent

Heya Nick,

Thanks for this code, it really cleared up some initial confusion for me.

I've wrapped your code up in a little github project:

https://github.com/Maciek416/BackboneRailsAuthTokenAdapter

Nick Gauthier

awesome! Thanks for packaging it up nicely!

vijaydev

Thanks a lot!

View Abstraction in Integration Tests

2011-02-05T00:00:00+00:00

Goal: Make integration tests drier by adding a view abstraction layer

Ruby on Rails has a bunch of popular of test frameworks, such as:

RSpec
Cucumber
Test::Unit
Steak

But one common aspect to all of the frameworks, out of the box, is that they're very procedural. Cucumber is designed so that each scenario has a set of steps. There is a single, global, collection of steps. RSpec is global by nature, there are no test classes, just describe blocks.

There is no doubt in my mind that these frameworks have made testing easier by adding lots of common actions, and allowing you to define your own common actions.

However, there is one thing that I don't see much of in tests: Object-Oriented Patterns. Most of us use the Factory pattern through a variety of gems. Internally, many frameworks use the Visitor pattern to execute the tests. But that's all I've ever seen (disclaimer: I am young, and have much to learn).

Here are some pain points I've felt while writing integration tests in the past:

Hard to reference objects throughout Cucumber scenarios. Often resulting in global variables used between steps
Cumbersome to check the view for expected output, often resulting in lots of css selectors
Brittleness introduced by coding html and css structure into test code that prevents refactoring the view

So, I recently had to write some integration tests from scratch, and I decided to do something different. I decided to implement the Bridge Pattern in my integration tests. I was confident enough in the solution that I decided to just use Test::Unit and Capybara to write my test code.

One of the major goals of this implementation is to make it easy to interact with the UI and objects within it. I think it's time I show some code.

Post Test for a simple blog site

# Given I am on the posts page
visit posts_path
# When I create a new post
click_link 'New Post'
View::Post.create(
  :title => 'View abstraction in integration tests',
  :body => 'We must go deeper'
)
# Then I should see a success message
assert_see 'Successfully created post.'
# When I visit the posts index
visit posts_path
# Then I should see one post
assert_equal 1, View::Post.all.size
# And it should have the correct title
assert_equal 'View abstraction in integration tests',
             View::Post.all.first.title
# And it should have the correct body
assert_equal 'We must go deeper', View::Post.all.first.body

Let's take a look at a few interesting things:

I'm directly using Capybara's dsl to navigate
When I create a post, I'm using a View module so that it won't use the ActiveRecord object
When I create a post, I pass in a hash of the fields I'd like to fill in
When I check to see if the post is created, I'm using methods on View::Post that return ruby objects, like an array, and that instances of View::Post have methods like "title" and "body"
There are no css selectors or html, but I do have button test present

OK, hopefully that piqued your interest. Let's look at some of the implementation. First, let's check out the base View module and a barebones implementation of View::Post:

module View
  def self.body
    Nokogiri::HTML(Capybara.current_session.body)
  end

  class Abstract
    # Access capybara dsl in the view helper classes too
    include Capybara
    extend  Capybara

    def initialize(node)
      @id = node['id']
    end
    def self.all
      nodes.map{|node| new(node) }
    end
    private
    def id
      %{##{@id}}
    end
  end
  class Post < View::Abstract
    attr_reader :title
    attr_reader :body
    def initialize(node)
      super
      @title = node.css('.title').first.text.strip
      @body = node.css('.body').first.text.strip
    end

    private
    def self.nodes
      View.body.css('.post')
    end
  end
end

OK what's going on here? Let me step you through it:

When I call View::Post.all, that calls View::Abstract.all, which iterates over View::Post.nodes and builds instances of View::Post
View::Post.nodes runs Capybara's current page through Nokogiri, then selects all the HTML nodes with the class "post"
When a View::Post is initialized, it uses Nokogiri to set attributes on itself, from the view. Like title and body
View::Abstract always stores an object's dom id as @id so that it can be used internally, which we'll see next.

Now lets take a look at how View::Post.create works:

module View
  class Post
    def self.create(opts = Hash.new(''))
      fill_form opts
      click_button 'Create Post'
    end
    def self.fill_form(opts)
      fill_in 'Title', :with => opts[:title]
      fill_in 'Body', :with => opts[:body]
    end
  end
end

Here we show how the class method uses capybara to take care of filling in the form for us. Now, if we change how our forms are rendered, we can change them in one place. Nice and DRY.

Let's look at one of the biggest pain points for me in cucumber: deleting an object in a list of objects. Why is this a pain point? I usually have to write a custom step like "When I delete the Post 'My Post'", which will use dom_id to find the id of a Post object found in the DB with the title "My Post". I find this really roundabout, because you don't need to look in the database for an object to figure out its dom id. It's right there in the view. Any competent internet user would be able to click on the "Delete" button for the post called "My Post" if you showed them the page in a browser. Here is the test code:

# Given I made two blog posts
2.times do |i|
  visit posts_path
  click_link 'New Post'
  View::Post.create(
    :title => "Post #{i}",
    :body => "Body for #{i}"
  )
end
# When I go to the posts path
visit posts_path
# Then I should see two posts
assert_equal 2, View::Post.all.size
# When I delete Post 0
View::Post.find_by_title('Post 0').delete
# Then I should see one post
assert_equal 1, View::Post.all.size
# And I should not see Post 0
assert_nil View::Post.find_by_title('Post 0')
# And I should see Post 1
refute_nil View::Post.find_by_title('Post 1')

Notice specifically the line where the post is deleted. I grabbed the instance of View::Post corresponding to the title I wanted and called .delete on it. Then I thoroughly check that the correct post was removed. Here is the implementation:

module View
  class Post
    def delete
      within(id) { click_button 'Delete' }
    end
  end
end

Expecting more? In View::Abstract we defined the method "id" which returns the dom id of the object which was stored when we initialized it. I simply told capybara to click "Delete" inside that node's div. This was the "eureka moment" for me. Something that is frustrating and difficult in other styles of testing is just plain simple with this pattern.

There is a lot more than can be done here, and I'm just scratching the surface. If you'd like to try it out for yourself, I've created a Rails project with this environment setup on github: View Abstraction Demo. Here's how to use it:

git clone git://github.com/ngauthier/view-abstraction-demo.git
cd view-abstraction-demo
bundle
rake

Note: you must use ruby 1.9.2. Important files to look at are "test/test_helper.rb" and "test/integration/post_test.rb".

Comments

Jakub

I am sure your idea is gonna turn into a gem. It just fits so well in a view testing pattern and adds performance to it also. I don't fancy Cucumber, use Steak, but still I Repeat Myself with # and . so unql.

Gem would be nice. Generating test/views/abstract.rb and test/views/abstract/model.rb - so nice to modify you know. I'm looking forward to help out with gem development. Wish you luck.

Nick Gauthier

Hey Jakub, glad you liked the post.

I don't think this would make a good gem, because only about 20 lines would be packaged. The generator would create very bare files because it would have no understanding of your css structure.

Also, I've found that UI elements don't always map 1:1 with database elements.

Lastly, I find that the power of this pattern is simple and can be included directly in your code. Putting it in a gem would make it hard to change it on a project-by-project basis. I expect to add more helpful methods to abstract as I use this more.

Thanks,
Nick

Jakub

Thanks for the answer Nick! You can be right, maybe it's too much for a gem, however having that in a generator could be a trick.

There is no need for the gem to understand css nor thoughtlessly map db - it can generate some conventional files that can be easily modified to fit the view. Actually, modifying those files would be a first step of a view specification!

The thing about that pattern is, for me, that those methods should not be stuffed in helpers (or additional spec/test methods) which is usually done. And to not forget that, the gem hooked to a initializer would be nice.

Nick Gauthier

yeah, I think moving them to separate files is definitely a cleaner way of doing it. Feel free to make a gem / plugin with generators to make it easier for yourself.

Corey Haines

Hi, Nick,
Good thoughts.
This looks similar to the page object pattern that a lot of people are using these days in Watir and Selenium tests. Have you looked at that? It could help influence your stuff, hopefully guide you past any dark corners they've already dealt with.

Nick Gauthier

Thanks Corey, that's exactly what I'm trying to do. I figured there was no way I could be the first person to think of this :-)

Nick Gauthier

@Jakub

I've been adding more functionality, and it seems like you're right and it would make a nice gem. Mostly to make it easier to define selectors and attributes, and make a bunch of convenience methods (like the enumerable methods).

Also, I am planning on bundling some assertions with the gem, which will take advantage of capybara's automatic delay on failed assertions. I'll post on this blog soon when it is released.

-Nick

Nick Gauthier

This has been released as a Gem, and the syntax is changed a bit:

https://github.com/ngauthier/domino

My Workflow

2011-01-07T00:00:00+00:00

Stemming from a workflow discussion on twitter featuring @bryanl, @eee_c, @pjb3, @stevenhaddox, and @webandy, I decided to share my workflow. Originally, we were talking about Mac Apps (specifically the paid ones), but that turned into "how do you work". Here are my goals for a good workflow:

Goals

Easily navigate applications and contexts
Minimize distraction
Maximize focus
Get work done

Easily navigate applications and contexts

I need to be able to quickly get to certain applications in order to maximize productivity. I want to be able to swap to a browser, terminal, editor, or other application in under a second. The mouse won't cut it, I need keyboard shortcuts.

My solution here is Alt-Tab. One of the oldest and most basic ways of switching apps. Alt-Tab is very fast for me because 95% of the time I have 3 windows open on any given workspace (OS X read: spaces). This means that Alt-Tab goes back to the previous window, and Alt-Tab-Tab goes to the alternative window.

Minimize distraction

I layout my workspaces in a 2x2 layout. Here is what I put on each window:

[1,1] Chrome holding "distracting" sites like gmail, twitter, campfire, pivotal, etc
[1,2] one Vim, one Terminal, one Chrome with only development related websites open
[2,1] passive applications. Usually Xvfb and Rhythmbox. I don't go here often
[2,2] alternative dev context. Same as [1,2] but for a different project. This is only up occasionally. Sometimes I use it for transferring files to remote servers.

I spend 99% of my work day in [1,1] and [1,2].

I disable all notifications. Nothing pops up. Nothing makes noise. Nothing "pulses" in the taskbar. Nothing. My phone is on vibrate and no apps on it make notifications except calls and texts. No one ever needs your attention immediately in gmail or campfire or twitter or pivotal or basecamp. When I am at a solid stopping point in my real work (30m to 2h in between) I go check all my distraction tabs. If the server is really on fire, someone will call my phone.

Maximize focus

This is a combination of the previous two, plus I always maximize my windows so I'm only looking at one thing at a time. No distractions so nothing gets in my face while I'm working. Many people like putting up a terminal next to their editor, but they usually end up alt-tabbing anyways. I prefer larger windows so I can have more vim splits (usually 3x3 on each tab) and more terminal output.

Applications

This last section is in response to our twitter debate about paid mac apps. I'm on linux, so I have no access to any of those apps. Let's start with the apps I spend the most time in:

Edit code: Vim
Run tests, servers, and other commands: gnome-terminal
Web browser: Google Chrome
File browsing and remote FS management: nautilus w/ gvfs
Administer databases: mysql or psql console

That is where I spend the majority of my day. There are very few alternatives to the above applications that aren't very similar. Everyone has a browser and a terminal. Editors is an entirely different can of worms, so I won't get into that here. Some people use GUIs for DBs, but I prefer the console. I learned SQL before I did web development, so it's more natural for me. And if it's too hard, I write a test :-).

There are many excellent command line applications at your fingertips that I use through my terminal. Apps like git, grep, find, screen, top, ps, kill, and thousands of others. Learning the command line root of the application is usually more productive than a gui that was built on top of the command line application. The only exception I've found is when I need something shown to me graphically, like a pdf. I like to avoid the mouse, so the terminal is my friend.

Here are a bunch of Mac Apps that a couple of people brought up during our twitter discussion:

mailplane
echofon
divvy
propane
alfredapp

I think it's a nice small selection of apps that illustrate the differences in my workflow from many other developers. Mailplane, echofon, and propane are for Gmail, Twitter, and Campfire. I try to ignore those applications as much as possible. Ideally, I want to spend less time in those apps in order to get more done.

Divvy and Alfredapp are all about launching and organizing applications. I have three applications in three windows. Vim manages its own splits and tabs. In my terminal, screen or tabs works. In the browser, I use tabs. Everything is maximized to improve focus.

What it really boils down to, though, is that I don't do a lot of stuff during the work day. I write code. I avoid anything that is not essential to writing code. I want less applications and I like having my life "in the cloud" so I don't have to install a lot of stuff on a new computer. My knee jerk reaction to new applications is not "oooh shiny", it's "I don't need that". Another really nice thing about this workflow is that it's scalable. I've worked from 1024x768 to 2560x1600, and it always makes optimal use of screen real estate. When I swap to my netbook, I don't feel cramped, I just open more tabs.

Join the discussion

Please post comments about the applications you use that I haven't addressed. I'll try to reply to everyone about how I handle different scenarios. Most of the time, my response will be "I try to avoid that because I'm trying to write code here!"

Comments

pjb3

A couple of GUI tools I prefer over command line:

* GitX

I find it easier to review diffs in GitX than diff at the command line. Also easy to unstage unwanted changes before commiting

* Rubymine

Hard to explain this. I'm planning on doing a screencast soon to show how I work in Rubymine. I find it more efficient than vim + command line or emacs.

* Querious

I'm fine with writing SQL on the command line, but viewing the results, esp. when you have large columns and/or large tables, sucks. A GUI fits things in tabs, windows and grids nicely.

I agree with you about the mouse. As much as possible, I'm trying to have keyboard shortcuts for all common operations.

As for distractions, depends on the project you are working on. If you are working on a project by yourself, "leave me alone I'm coding" works really well. If you have many devs working on the same code base, near real-time communication is important. It's a balance though.

Bill Mill

pjb, check out dbext for viewing the results of queries in Vim. If you're a vimmer, it's a godsend.

Also, my workflow is nearly exactly the same as yours, but on OS X. iTerm2 is as good as Linux terminals.

bryanl

Since it appears you are critiquing that apps that I posted earlier, I now feel the need to reply.

Having unique apps which do their jobs really well is the key to helping me focus. The web browser is a lot of things to many people, but it most definitely isn't the end-all-be-all. I actually prefer to have as few browser tabs open as possible. There is nothing more frustrating to me than seeing all the wasted screen space wasted by tabs.

Workflows don't have to be dead simple to be scalable. I work on a 11.6" macbook air, a 15" macbook pro and a 27" imac, and I pretty much have the same configurations on all three workstations. I don't believe in external monitors, and as you know, the keyboard is most definitely the place to be.

For instance: I use mailplane, so I can have easy access to multiple gmail accounts. Sure the web browser works fine, but it doesn't easily scale past one account if you would like to check multiple ones easily.

You hack code 100% of the time. What you describe is ample for a person in that situation. My day is filled with much than just hacking on code, so tool sets will differ. No problems with that. There isn't one true way. The only thing you really need to work on is constant refinement on your path to happiness.

My work here is done. I've inspired yet another blog post ;-) I'll wait for you to troll me again another day.

Nick Gauthier

@pjb3

I use git-gui from time to time to review a large commit.

I'd like to see a rubymine screencast. The few times I've seen it I didn't see any advanced features. The "mining" that I saw looked the same as ctags.

As for the sql gui, I tend to write a test when stuff gets pretty complex. psql's console automatically pipes output to "less" so it's pretty easy to read.

Some projects I pay more attention to campfire, but it's never in realtime. If we really need a real time discussion, it's time for a phone call or a face-to-face.

@bryanl

I didn't mean for it to come off as a critique. I was trying to explain why I don't buy those apps, not "these apps suck because you could just do it this way".

I am pretty OCD about keeping a tidy environment, and I close tabs a lot while I browse.

I like the "single monitor" approach too. I was on a 24" for a long time, then I added a 20" that, but recently ditched both for a 30"@2560x1600.

@mailplane I only have one account. I could see needing this if I had more than one.

It's definitely a 100% hacker setup. If I had to do any graphic design or PM, it would be very different. When I do have to do PM stuff, I use another workspace and a whole new set of applications.

I don't mean to troll you. I find it very interesting that you and I have very different perspectives and setups for doing very similar work. Also, I know you can take it :-)

Elise

I use gitg as well for organizing commits, it's nearly equivalent to gitx and beats the hell out of git log :)

I also use Linux, and I found that for me tiled window managers do wonders for focus - though I understand that Compiz has that possibility now.
I use the Awesome window manager (http://awesome.naquadah.org/), which has all manner of keyboard shortcuts and can be customized using Lua. It's also ideally suited to the kind of organization you're talking about, naming tabs and allocating applications to it.

straydogsw

Nice post Nick, I've been wanting to learn some more keyboard shortcuts for moving around the linux desktop and this has been very useful.

Those windows look tight in your [1,2] screenshot. Do you have a window-placer util or something to get them so nice and even?

straydogsw

oops, nevermind. I see that they are vim-splits in a full size terminal.

Nick Gauthier

yup :-)

There are a bunch of window managers and things for placing windows in linux, but I don't know them.

If you install the ubuntu package compizconfig-settings-manager and run it you'll get the desktop effects settings, which have a keyboard shortcut for tons of stuff.

Everything that is wrong with mocking, bdd, and rspec

2010-12-02T00:00:00+00:00

Below is a public excerpt from the recently release RSpec book :

module Codebreaker
  describe Game do
    describe "#start" do
      it "sends a welcome message" do
        output = double('output')
        game = Game.new(output)
        output.should_receive(:puts).with('Welcome to Codebreaker!')
        game.start
      end
      it "prompts for the first guess"
    end
  end
end

This example illustrates what is wrong with standard BDD as practiced today . Consider the following correct implementation of the spec above:

module Codebreaker
  class Game
    def initialize(output)
      @output = output
    end
    def start
      @output.puts("Welcome to Codebreaker!")
    end
  end
end

Now consider the following refactorings of the start method:

@output.write("Welcome to Codebreaker!\n")
@output.print("Welcome to Codebreaker!\n")
> "Welcome to Codebreaker!".split.each{|c| output.write(c)}; output.write "\n"
@output 
> "Welcome to Codebreaker!\n"

All of these produce the exact same result, which is sending the string to the output stream. However, all of these refactorings will fail the test suite.

Now consider the following implementation:

@output.puts "Welcome to Codebreaker!"
@output 
> "You smell bad and are also ugly!"

That test passes the spec above, but it also insults our (paying?) users!

One of the primary reasons for having a test suite is so that I can refactor and have the tests validate my changes . If I have to change my tests and my code just to re-write the way I cause the same end result, then my test suite is useless for refactoring.

So now the obvious question is "Well Nick, how would you test it?"

require 'stringio'
output = StringIO.new
game = Game.new(output)
game.start
output.seek(0)
assert_equal "Welcome to Codebreaker!\n", output.read

This test passes for all the implementations listed initially, and will fail for the implementation in which we insult our users.

Think about the difference here. The original example is saying "Make sure the game calls puts with the following argument" whereas my test is saying "Make sure the game outputs the following string to the output object". The second case is far more useful.

Mocking is a great tool for dealing with stable APIs. However, when introduced in areas of the codebase that have just a small amount of churn, they can cripple the test suite with their brittleness. How can we expect to embrace change when our codebase can't?

RSpec has become so intertwined with mocking and stubbing practices that many people are taking mocking to the extreme and stubbing out their own code.

I will close by deferring to the wisdom of a higher power:

It is the high coupling between modules and tests that creates the need for a mocking framework. This high coupling is also the cause of the dreaded “Fragile Test” problem. How many tests break when you change a module? If the number is high, then the coupling between your modules and tests in high. Therefore, I conclude that those systems that make prolific use of mocking frameworks are likely to suffer from fragile tests.

...

Keeping middle-class test doubles (i.e. Mocks) to a minimum is another way of decoupling. Mocks, by their very nature, are coupled to mechanisms instead of outcomes. Mocks, or the setup code that builds them, have deep knowledge of the inner workings of several different classes. That knowledge is the very definition of high-coupling.
- Uncle Bob Martin on Object Mentor

Comments

dchelimsky

The test you propose will fail as soon as an additional message (praising the user?) needs to be displayed in response to start(), so it, too, is brittle due to its constraints. It's all trade-offs, my friend.

But thanks for promoting the book!

Nick Gauthier

Of course it will fail, because it is a spec on the start method, whose behavior has changed :-)

dchelimsky

It's behavior has been _extended_. If the existing message changed, that would be one thing, but the fact that there is an additional side-effect of calling start() should not cause this expectation to fail.

The issue here is that the behavior of the start() method is reflected in another object. Both examples inspect the collaborator to specify the outcome. Both have constraints that will cause future failures depending on how things change.

You could probably make something more flexible by using assert_match instead of assert_equal, but that wouldn't protect against a mean-spirited developer insulting users. Again, it's all trade-offs. Pick your poison.

Nick Gauthier

Yeah I agree. It depends on how much you want to assert. I was considering doing it as a regex match. I decided that I wanted to asset that the method output the exact message, no more and no less. It could be a looser test, and allow for other tests to check other behaviors.

I probably would do that in practice when I added more behavior to the method.

Decisions that aren't trade-offs never instigate an interesting debate.

jyurek

Extended or replaced, the behavior *changed*, and that's what tests are meant to detect, after all.

I think Nick's example highlights exactly why overmocking is a bad thing. IMO, I don't think this is a failure (per se) of BDD, RSpec, or even Mocking as a concept, it's just not the best test for the situation.

jollyjerry

I definitely agree with your point that mocking and stubbing can easily be taken too far. What I hate the most is when I see an entire chain of internal implementation stubbed out (foo.hidden.internal.method.baz). It ends up coupling your tests to your implementation, and since tests are also code, you make your code spaghetti that much more tangled. I'm just as guilty as the next guy when it comes to thinking abstractly about what the functionality should be, rather than reduplicating my code in test form. Like dchelimsky said, it's a hard problem!

Evan Light

Agreed -- including that perhaps you should be using an "assert output.read =~ //.

This is why, as a general rule, I have learned to largely avoid mocking and stubbing unless I am writing unit tests against an external system.

Yes, test doubles are great for isolating the unit under test. However, that isolation comes at a cost.

Should an external dependency change, your test doubled tests will lie to you! The test double is blissfuly unaware that the interface it has doubled has changed. Therefore, your test will inaccurately pass. Under such circumstances, I want my tests to fail loudly!

In my experience, this has resulted in significant time lost tracking down production application problems!

TL;DR White-box testing coupled with test doubles is dangerous. I strongly prefer an integration test that exercises the external dependency.

Evan Light

David: While Nick's test may have been brittle, at least the test would have failed as expected. Should the output string change, his test test fails. Should the implementation use print, his test fails.

The failure of the test is adequate to ensure that the test is revisited and updated.

Honestly, I admire so much of what you have done for the Ruby community. However, your tenacious adherence to mocking and stubbing as a routine TDD technique still utterly bewilders me. The brittleness of the resulting test hardly seems worth the benefit in almost any case except the rare external dependency.

myron.marston

Careless mocking can definitely make a test suite excessively brittle, but my overall experience with mocking has been the opposite. I used to practice a more classical TDD approach, where I used little-to-no-mocking and I wound up with bloated, slow, brittle test suites. After experimenting a bit and reading the RSpec book, I've started using mocking as a regular technique to achieve isolation in my unit tests and it's helped me create far less brittle test suites (not to mention they are much, much faster!).

It's definitely easy to shoot yourself in the foot with mocking (just like it is with ruby, git, or any of our other standard, powerful tools we use daily), but that doesn't mean it's not a useful, worthwhile tool: you just have to be aware of the dangers of over-mocking and mock with care.

Nick Gauthier

Test suite speed is not a valid argument for mocking.

http://grease-your-suite.heroku.com/

Evan Light

@Myron: How were your tests more brittle before introducing test doubles?

If you're test has clear preconditions, execution of the code under test, and clear expectations, I don't see how that leads to brittleness.

Can you give an example of one of your fragile pre-test double tests?

Jim Gay

I agree. If you're doing should_receive(:puts) then you're saying that calling "puts" is important.

All of this is behavior of the code, you just need to spec what behavior is important: the methods called, or the output. Sometimes it might be important that a method is called, but not in this situation.

Nick

I completely agree with Jim. It comes down to what behaviour is important, and thus needs to be tested.

If Codebreaker only cares that @output begins with "Welcome to Codebreaker!", then Nick Gauthier's approach is ideal.

If Codebreaker relies on #puts being called on @output , then the "bad test" is ideal.

As always, everything's contextual: what's important to you?

Nick Gauthier

@Jim @Nick

I totally agree. What I dislike is the tendency for people to write a test against methods being called, when that is almost always not what is important in the test.

Continuously asking "What am I testing and why am I testing it" is very important to writing good code and good tests.

Nick

> Continuously asking "What am I testing and why am I testing it" is very important to writing good code and good tests.

Hear hear!

tooky

If you are using mocks and stubs in your unit tests then you need to have a good set of acceptance/integration tests as well.

I like using mocks and stubs in my unit tests as they allow me to isolate the area of the code I'm working in, and I don't have to worry about side-effects or complex set-up of other objects. I also use cucumber though, this gives me great black-box coverage of the system.

So while changing the original example will make this spec break, it won't make any other specs in the system break, because no other specs will depend on its behaviour. So long as the behaviour is preserved it won't break the acceptance/integration tests either.

Pat Maddox

The mock-based example specifies the interaction with the collaborator. It demonstrates that you can pass in any object that responds to #puts(str) and expect it to work.

Your refactoring is valid -- when the output object responds to #puts, #write, #print, and #<<, and implements them all in basically the same way.

You are free to change the internals of a method however you want, but remember that certain changes will have broader-reaching effects than others. Changing how you build up the string to, say, an array of strings that then get joined, should certainly be encapsulated. Changing how you call collaborators is a totally different thing -- you're changing the contract, potentially in a way that's incompatible with existing clients.

Focused examples (or unit tests, or microtests, or whatever) help answer the question, "what do I need to know to use this code?" When there is a collaborator involved, one of the key things to know is the protocol that the collaborator must adhere to. An interaction-based example makes that protocol explicit. Moreover, it establishes a checkpoint. Should you decide to call #write instead of #puts, you have to consider the possible implications to existing code.

I understand where you're coming from. When you run the program, there is no externally visible difference in behavior. That is why we tend to separate the duties of checking program behavior into acceptance tests. Your unit tests then become a place for you to check an object's correctness, which includes its own logic and its interactions with other objects. Don't assume that because the external behavior of the program stays the same, that the behavior internal to objects has stayed the same as well.

admin

tl;dr of Pat's comment: I'mma let you finish, but the difference between unit tests and acceptance tests is one of the best differences of all time.

Nick Gauthier

haha thanks @admin.

@Pat I understand that is the case when you are unit testing between collaborators where specific method invocation is important. However in the example I used, it was clear to me that the caller was not a collaborator but a higher level controller or supervisor. Actors at a higher level of abstraction should not be concerned with the implementation details but rather the results.

To sum it up, I interpreted this as an acceptance test.

dchelimsky

Evan - there is a chapter in The RSpec Book on Mock Objects that talks about when to use them and the potential pitfalls (including the one that Nick brings up here).

Mocking external dependencies is actually considered to be a mocking no-no, because then you create a binding between your app and the external dependency's API. Instead, write a thin wrapper around that dependency that speaks in the domain of your app, and mock that as a means of isolating your app's code/specs from that dependency.

But that is only the tip of the iceberg when it comes to use cases served well by mocks. Others include: non-deterministic collaborators (intentionally random behavior) of collaborators), polymorphic collaborators (no need to have 3 examples when 1 would do), protocol definitions (where which methods are called represent adherence to a contract), collaborators that don't even exist yet, etc, etc.

Nick Gauthier

@dchelimsky

I read your slides from rubyconf and I agree with your stance on mocking. However about 95% of people that copy rspec examples don't understand the implications and often code themselves into mocking hell. They are my target audience :-)

dchelimsky

"To sum it up, I interpreted this as an acceptance test."

Part of the aim of the book is to make a distinction between application and object behavior: Cucumber is well suited to specify things at the application level (end to end tests/acceptance tests/customer tests, etc), and RSpec is well suited to specify things at the object level (unit tests, micro tests, developer tests, etc).

The example you cite comes from a chapter on describing the behavior of objects with RSpec (i.e. unit tests), after a chapter on describing the behavior of applications with Cucumber (i.e. acceptance tests), and is not at all intended to be an acceptance test. Hope that clears at least that part of this up for you.

dchelimsky

@nickgauthier - I appreciate wanting to help people understand the subtle waters of mocking. Unfortunately, the title of this post is not in any way nuanced or subtle. If your target audience is people who blindly copy examples without thinking about their implications or trying to learn to understand them, it's likely that they won't actually read this full post, much less the thoughtful discussion that follows.

Nick Gauthier

If I had to change the title, it would be expanded to "Everything that is wrong with how everyone uses mocks in rspec to do bdd"

Every time I've encountered mocking in test code (aside from 3rd party calls) I've seen it fall under the category of the content of this post.

While you describe a sound way to use mocks to test object behavior, I have never seen it executed properly. This is not to say it cannot be done, but that everyone is doing it wrong.

And yeah, you can't defeat the cargo-cult.

Evan Light

@David: I'm familiar with that practice RE: APIs. You took me a little too literally. I always abstract them a level.

I'll have to take another gander at the (finally ;-) ) completed RSpec book. I bought the beta a very long time ago and haven't read it much since. I'm curious about what you have to say about mocking.

However, I've generally found white box testing to be too implementation specific. Granted, I realize that there are almost "camps" among TDD'ers: the 'behavioral statists' that I likely fall under, the 'unit mockists' that you likely fall under, and others. I tend to prefer writing acceptance/integration tests with only supporting unit tests as necessary to help me "fill in the gaps" where the feature is too complex to develop without unit tests to handle complex logic (see http://evan.tiggerpalace.com/articles/2010/12/02/stop-writing-code-you-dont-need/).

zenspider

@ngauthier:

Via minitest (or test/unit in 1.9):

def test_start_output
assert_output "Welcome to Codebreaker!\n", "" do
Game.new.start
end
end

(blogger.... you suck. no pre tag allowed? no formatting help? bite me)

@dchelimsky: the rspec code would break too because EVERYONE uses .once. And remember, since we do test-first, it wouldn't break because we changed the output, it'd break because we added additional assertions to the test that we then needed to make pass. That isn't brittle. That's process.

myron.marston

@Evan: Re: "Can you give an example of one of your fragile pre-test double tests?"

The code base I'm thinking of was a rails app, and simple changes to models (i.e. adding a new required field or changing a validation) would cascade across the test suite and break lots of other tests because those tests depended on the existing behavior of the model when they shouldn't have. For example, a controller functional test that passed in valid attributes and would assert some result failed because the passed attributes were no longer valid. You may argue that the breaking controller test was a good thing, but I don't agree. I like to have lots of fast, isolated unit tests (which do use mocking carefully), as well as some full-stack integration tests (which don't use mocking). In this case, a change to a model my break a unit test and an integration test, but it's not going to cascade across my test suite.

Having controller tests that depended on the knowledge of what constituted a valid record was a very brittle approach. Mocking has worked well to help me unit test my controllers.

dchelimsky

@zenspider - that works assuming you have a test named test_start_output and it's the only test ever written that cares about the output of the start() method. As soon as additional context requires that start provide different output under different conditions, either this test would start to get unruly (too many states/assertions), or new tests would emerge that, in making them to pass, might cause this one to break.

re: the rspec code and "once" - in fact, it breaks just 2 or 3 pages later in the book, and a wonderful learning opportunity emerges. Hooray for examples taken out of context!

Sean DeNigris

Of course, none of these pitfalls have anything to do with BDD or RSpec. Just saying...

Pat Maddox

Can you please gist a file that demonstrates that the spec passes even when you include the insulting message to the user? Here's what I got: https://gist.github.com/730609

Nick Gauthier

@Pat ah you're right, since it's a double it will complain about extra method calls. This is why mocking becomes annoying when your method does a few things with its input. Any test will need an elaborate mocking setup.

dchelimsky

@Nick - a page or two later in the book we add "as_null_object" to the double, and the need for an elaborate mocking setup is eliminated.

That said, we've all been missing the deeper problem here, which was recognized immediately by Nat Pryce and reflected in these two tweets:

http://twitter.com/natpryce/status/12029918607048704

http://twitter.com/natpryce/status/12030541293424641

We've been talking about pain in the example, but none of us have "listened to the test". The real problem here is that there is a missing abstraction. Change "output" to "reporter" (or similar) and change "puts" to "message" (or similar) and the refactoring concern you raised goes away, everything is "speaking" in the domain, and all is right with the world.

Nick Gauthier

@dchelimsky I think the underlying concept still stands.

A mocking unit testing strategy encourages testing the implementation of a method because it's easy. My argument is that it doesn't get you very much in the way of code validation.

When I write tests, I think about the effects of my code, and then I validate that the effects occurred.

For example, consider testing user registration. Which of these scenarios below is most useful?

When I register on the site

=>

1) Then the database should be asked to insert a user
2) Then there should be 50 users
3) Then there should be an additional user
4) Then I can log in

I like #4, because it focuses on the input and the ultimately desired result and goal of a feature. This user-acceptance, and not unit testing.

But in unit testing I like to focus on the same goals and ask the same questions.

For example, if I'm implementing a "users sorted by join date" method, I can see two ways of going about it:

1) Mock the user model and ensure that the storage adapter is called with the appropriate method and parameters to return a sorted list of users.
2) Create a couple of users, call the method, and see if the results are sorted.

#1 is white-box and #2 is black box. I'll pick black box every time because it focuses on my goals, and not my process.

Regarding the tweets:

Yes, it's easy to blame the tools, but what if a particular tool can be easily misused? In the best of hands, it is revolutionary and powerful, but in most people's hands it's destructive. For example, morphine.

I think mocking is being abused and overused and is causing people pain in the long run. It's enticing because its simple to implement, but I find the assertions it makes to be weak and brittle.

I will agree, however, that in cases where you have a stable API that mocking can make your life easier (especially in large and complicated codebases). However, I've always found that it's not that hard to create helper methods to setup elaborate scenarios in which to do a real goal-based test. There are great gems out there like cucumber and factory girl that allow you to encapsulate complicated functionality in simple steps.

dchelimsky

@Nick - "but what if a particular tool can be easily misused? In the best of hands, it is revolutionary and powerful, but in most people's hands it's destructive. For example, morphine."

By that logic, we should all stop using Ruby :)

re: the "sorted users" example, I agree that is likely better specified with a more black box approach.

The distinction is that, in the sorted users example, the User class returns the object (a list of users) on which expected outcomes can be specified.

In the output example, the outcome is specified _on a collaborator_. Whether we specify that as an interaction or post-action state-based verification, the example is still bound to the collaborator object, so there is little difference in terms of dependencies and brittleness.

Using a test double for "reporter" gives us the freedom to introduce different "reporter" solutions (command line, text file, web page, etc) without these examples failing. In that context, the use of a test double and a message expectation is actually less brittle than the alternatives.

Nick Gauthier

@dchelimksy no, I'm not saying stop using ruby. I'm not saying stop using morphine in hospitals either.

It's more of an education perspective. I want to show people how they should and should not use tools so they don't dig themselves into a hole.

I think we're in agreement that when a collaborator's interface is well defined and stable that mocking is a good solution.

The problem is that whenever I see examples of mocking it's always in (what I deem to be) inappropriate scenarios.

If you showed a reporter class with a simple API of reporting messages, then showed the Codebreaker class working with a mocked Reporter, I'd be much happier with your example. I worry about a generation of rubyist who will mock their test suite to hell and then be stuck in a broken world.

Of course, anyone can write bad code in any framework, but I want to teach people the appropriate uses with solid examples.

I was very disappointed to see this example in the excerpts of the rspec book, because I've had to refactor and fix code like that and it's not fun.

dchelimsky

Well you've got me there. This is clearly a bad excerpt and I'll have it changed soon. I hope you'll consider reading the rest of the tutorial in spite of the excerpt. I'd welcome your feedback.

I think we have the same goals in terms of deepening the understanding of when/where to use different tools. I don't think we agree on everything here (I don't "always" prefer anything), but I think we probably agree on more than we disagree on.

Cheers,
David

Microgem: JSLintRB-v8

2010-11-29T00:00:00+00:00

GOAL: Provide a Ruby interface for running JSLint on javascript files using v8.

require 'jslintrb-v8'
puts JSLint.new.check("var x = 5")

Will return the string:

Error at line 1 character 1: Missing "use strict" statement.
   var x = 5
 Error at line 1 character 10: Missing semicolon.
   var x = 5

Also, all of jslint's configuration options are available in the constructor:

JSLint.new(:undef => false, :sub => true)

JSLintRB-v8 on Github
JSLintRB-v8 on Rdoc.info

Using Rails's Object#to_json to create a clean JSON Presenter

2010-11-17T00:00:00+00:00

The problem: You have an ActiveRecord model (or any object, really) and you want to output it to json, but you need some specific methods and maybe some processing of its attributes before you can call .to_json.

Let's pretend we have the following object:

class Widget
  attr_accessor :id
  attr_accessor :name
  attr_accessor :gadgets
end

class Gadget
  attr_accessor :id
  attr_accessor :name
end

And we want this json from a widget:

{
  id : 'widget-47',
  name: 'name of widget',
  children: [
    {
      id : 'gadget-63',
      name : 'name of gadget'
    }
  ]
}

Note that ID has the class name on the front, the name is passed through directly. Also, the sub-objects (Gadget) have their id processed.

First, let's make the Gadget presenter:

class GadgetPresenter
  attr_reader :id
  attr_reader :name
  def initialize(gadget)
    @id = %{gadget-#{gadget.id}}
    @name = gadget.name
  end
end

Rails provides Object extensions that give us a to_json method on every object. By default, this will call all of our attr_readers. So, all we have to do is provide attr_readers for the attributes we want in the json. Now, Gadget#to_json would give us:

{
  id: 'gadget-63',
  name: 'name of gadget'
}

Now the Widget presenter:

class WidgetPresenter
  attr_reader :id
  attr_reader :name
  attr_reader :children
  def initialize(widget)
    @id = %{widget-#{widget.id}}
    @name = widget.name
    @children = widget.gadgets.collect{|g| GadgetPresenter.new(g)}
  end
end

Pretty much the same, except for the children. This lets us call the associate "children" instead of "gadgets". And all we have to do is collect up a bunch of gadget presenters. When .to_json is called on the widget, it calls to_json on the attributes. Since children is an array, it collects .to_json from each entry in the array, which calls the GadgetPresenter's to_json.

I really like this implementation because it's really just an adapter, and it only concerns itself with the data. It is also external to the Widget and Gadget class, allowing for multiple types of presenters easily.

Here is a running example: (requires rails 3) https://gist.github.com/703747

Comments

mattsears

Nice article, thanks for posting this. Quick question...where do you recommend storing the presenter classes in you Rails project? I thought about 'app/presenters', but maybe that's overkill?

bryanl

app/presenters works well. Keeping everything in app/models makes little babies cry.

Minimal cucumber stack for testing a rich javascript application

2010-11-11T00:00:00+00:00

These are instructions for setting up a very bare cucumber stack for testing a rich javascript application.

All code available here: https://github.com/ngauthier/cucumber-webapp-demo

The following stack is used:

Cucumber
Capybara
Selenium
WEBrick

First, we make our awesome javascript powered webapp:

<!DOCTYPE html>
<html>
  <head>
    <script type='text/javascript' src='jquery-1.4.3.min.js'></script>
    <script type='text/javascript'>
      $(function() {
        $('#special-link').click(function(event) {
          $('#results').html("This is totally awesome");
          event.preventDefault();
        });
      });
    </script>
  </head>
  <body>
    <h1>My Webapp</h1>
    <p>
    <a href='#' id='special-link'>Click on this link to make cool stuff happen</a>
    </p>
    <div id="results"></div>
  </body>
</html>

All we're doing here is showing some text when we click a link. But we're using jquery's events to do it.

Now here is our feature file:

Feature: My Awesome Link
  Scenario: Clicking the awesome link shows awesome text
    Given I am on the home page
    Then  I should see "My Webapp"
    And   I should see "Click on this link to make cool stuff happen"
    And   I should not see "This is totally awesome"
    When  I click "Click on this link to make cool stuff happen"
    Then  I should see "This is totally awesome"

And here are our simple step definitions:

Given /^I am on the home page$/ do
  visit 'index.html'
end

Then /^I should see "([^"]*)"$/ do |content|
  within('body') do
    assert page.has_content?(content)
  end
end

Then /^I should not see "([^"]*)"$/ do |content|
  within('body') do
    refute page.has_content?(content)
  end
end

When /^I click "([^"]*)"$/ do |link_text|
  click_link_or_button(link_text)
end

And here is the cucumber environment file where all the magic happens. See comments for description:

require 'rubygems'
require 'cucumber'
require 'capybara/cucumber'
require 'webrick'

# setup capybara on selenium
Capybara.default_driver = :selenium

# This maps to webrick below
Capybara.app_host = 'http://127.0.0.1:3000'
Capybara.default_selector = :css

# Extend Cucumber's World with minitest assertions
World(MiniTest::Assertions)

# Launch a webrick server in a thread
AfterConfiguration do
  server_thread = Thread.new do
    project_root = File.join(File.dirname(__FILE__), '..', '..')
    WEBrick::HTTPServer.new(
      :Port => 3000,
      # TODO use the real application directory
      # Using the prototype directory for now
      :DocumentRoot => File.join(project_root, 'public'),
      :Logger => WEBrick::Log.new(File.join(project_root, 'cucumber.log')),
      :AccessLog => StringIO.new # to nowhere
    ).start
  end
  # Kill the server when cucumber is done
  at_exit do
    server_thread.exit
  end
end

The idea here is to boot webrick in a thread to serve the public directory, then connect selenium up to it. Very simple and light.

All code available here: https://github.com/ngauthier/cucumber-webapp-demo

Comments

Sharad Jain

Nice, thanks!

PRY - Please Repeat Yourself

2010-11-08T00:00:00+00:00

It's the canonical ruby metaprogramming example. You have this:

def large_image
  image_base + '-large.png'
end

def medium_image
  image_base + '-medium.png'
end

def small_image
  image_base + '-small.png'
end

And you do some dynamic programming to define the methods with similar content:

%w(large medium small).each do |size|
  class_eval %{
    def #{size}_image
      image_base + '-#{size}.png'
    end
  }
end

I have two problems with this pattern:

1) The dynamic code is far less readable and clear. It might even need a comment! The static code is very easy to read.

2) The dynamic code gives terrible stack traces

(eval):3:in `small_image': undefined local variable or method `image_base' for # (NameError)
  from test.rb:28

Versus the static stack trace:

test.rb:15:in `small_image': undefined local variable or method `image_base' for # (NameError)
  from test.rb:28

Here, test.rb:15 is the line inside "small_image" that calls "image_base".

The argument for metaprogramming is so that there is only one place to change the content. However this is easily accomplished with a helper. In fact I already had that helper in place, image_base. If you find yourself repeating the code *inside* the method, it's worth extracting into a helper.

Metaprogramming is for a dynamic system, not for enumerations. For example, Rails controllers use dynamic programming to allow you to define your own action methods. These methods are called dynamically by the router. This is a huge oversimplification, but the idea is that the Rails authors have no idea what method names could be used, so they can't type them all out. It's a fundamentally different problem.

If you find repetition in coding to be a difficult task, consider investing some time in learning a powerful editor with a regular expression matching and replace function.

So, please repeat yourself!

RESTful Many to Many Relationships in Rails

2010-11-04T00:00:00+00:00

GET    /people => List people
POST   /people => Create person
PUT    /people => Replace the collection
DELETE /people => Delete the collection

GET    /people/:id => List attribute of a person
PUT    /people/:id => update the attributes of a person
POST   /people/:id => create a new collection under a person
DELETE /people/:id => Delete a person

In rails, we use

index:   GET    /people     => List people
create:  POST   /people     => Create person
show:    GET    /people/:id => List attribute of a person
update:  PUT    /people/:id => update the attributes of a person
destroy: DELETE /people/:id => Delete a person

Now we may also have:

index:   GET    /groups     => List groups
create:  POST   /groups     => Create group
show:    GET    /groups/:id => List attribute of a group
update:  PUT    /groups/:id => update the attributes of a group
destroy: DELETE /groups/:id => Delete a group

Consider a relationship of Many To Many between users and groups.

How do I express the following?

"Add user X to group Y"

First thought may be:

POST /groups/Y/users?user_id=X

But technically, this means "Create a user whose attributes are {user_id => X} under group Y". This is wrong in two ways, the first is that we don't want to create a user inside a group. The second is that the user attributes are wrong, it's {id => X}.

The correct request would be:

POST /groups_users?user_id=X&group_id=Y

This means "Create a new GroupUser linking node whose attributes are {user_id => X, group_id => Y}".

OK now how do I express the following?

"Put user X into a new group, whose attributes are {name => 'My Group'}"

This is *two* requests:

POST /groups?name="My Group" => returns ID Z
POST /groups_users?user_id=X&group_id=Z

However, we can actually combine the requests like this:

POST /users/X/groups?name="My Group"

In a situation where a Group Belongs To a User, this would create a group under the user.

You would get in trouble if Group Belongs To a User and Group Has and Belongs To Many Users. However, in that case, you should be using another name for one of the assets. For example, Group could belong to a Creator, and a User would have a created_groups association. So the routes would actually be different:

POST /users/X/created_groups?name="My Group"

Which would mean create a group, where User X is the creator of that group.

So, how do we handle this with Rails routing and controllers?

resources :groups do
  resources :users, :controller => 'GroupsUser', :only => [:index, :create, :destroy]
end
resources :users do
  resources :groups, :controller => 'UserGroups', :only => [:index, :create, :destroy]
end

Now, for user and group resources, we would use a traditional controller. For the GroupsUser and UserGroups controller, it would be a bit different.

UserGroups controller:

index: return all the groups this user is in
create: create a new group, and add this user to that group
destroy: remove this group from the user's list of groups

Note we don't have:

show: this is redundant with groups/show.
update: this would update the attributes on a membership. If it's just a join there are no attributes, however this may be useful if the join has attributes (for example, role)

The most interesting action here is create:

@user = User.find(params[:id])
@group = Group.new(params[:group])
Group.transaction do
  if @group.save
    if GroupUser.create(params[:group_user].merge{:user => @user, :group => @group})
      # success
    else
      # Could not add user to group
    end
  else
    # Could not create group
  end
end

Note that this controller is getting a bit dangerous because there are three logic paths. Generally, controllers will only have two logic paths: success and failure.

Note also that we are merging the params[:group_user] when creating it. This is because we may want to have attributes on the GroupUser. This would all have to be in the form for posting to this action.

Technically, we are creating a group under a user. While this makes perfect sense in a belongs_to relationship, it is a little mind-bending in a many-to-many situation where the relationship is reciprocal. So, I'll end with a question. Do you think that this "double action" is a violation of REST?

Comments

pjb3

First of all, let me say that I am honored to be the first commenter on ngauthier.com :)

Second, what I would do is push the logic of creating a group down into the controller. The nested attributes feature of Rails gives you this automatically. Assuming your User model looks like this:

class User < ActiveRecord::Base
has_many :group_users
has_many :groups, :through => :group_users

accepts_nested_attributes_for :groups
end

Then the controller goes back to having just two paths, because user.save will take care of creating the group before creating the group_user. So when you do:

PUT /users/X/?group_attributes[][name]="My Group"

this would happen:

user.update_attributes :groups_attributes => [{:name => "My Group"}]
# => INSERT INTO "groups" ("name") VALUES ('My Group')
# => INSERT INTO "group_memberships" ("group_id", "user_id") VALUES (1, 1)

Nick Gauthier

You also get the honor of the first reply!

While that is convenient, it is not RESTful. The route you described is designed to modify the user resource, not any groups resources.

Also, you're going to go through "error hell" if a bunch of those groups and user attributes are invalid.

Doing nested attributes through a controller is trying to squeeze two entire controllers into one. For the sake of simplicity, testing, and maintainability, I'd avoid it.

-Nick