Skorks

Ruby & (Ampersand) Parameter Demystified

Tue, 16 Apr 2013 14:28:07 +0000

Recently I was asked a question about ‘& parameters’ when you define and/or call methods which take a block e.g.:

def blah(&block)
  yadda(block)
end

def yadda(block)
  foo(&block)
end

def foo(&block)
  block.call
end

blah do
  puts "hello"
end

As you pass this parameter around, sometimes the ampersand appears in front of it, but other times it doesn’t, seemingly with no rhyme of reason. As we dig into crazy metaprogramming or write various libraries, it is often hard to remember how confusing Ruby can be when you’re starting out. So, let’s dig into this a little more deeply and shed some light on what’s going on.

The Implicit Block

Methods in Ruby can take arguments in all sorts of interesting ways. One case that’s especially interesting is when a Ruby method takes a block.

In fact, all Ruby methods can implicitly take a block, without needing to specify this in the parameter list or having to use the block within the method body e.g.:

def hello
end

hello do
  puts "hello"
end

This will execute without any trouble but nothing will be printed out as we’re not executing the block that we’re passing in. We can – of course – easily execute the block by yielding to it:

def hello
  yield if block_given?
end

hello do
  puts "hello"
end

This time we get some output:

hello

We yielded to the block inside the method, but the fact that the method takes a block is still implicit.

It gets even more interesting since Ruby allows to pass any object to a method and have the method attempt to use this object as its block. If we put an ampersand in front of the last parameter to a method, Ruby will try to treat this parameter as the method’s block. If the parameter is already a Proc object, Ruby will simply associate it with the method as its block.

def hello
  yield if block_given?
end

blah = -&gt; {puts "lambda"}

hello(&blah)

lambda

If the parameter is not a Proc, Ruby will try to convert it into one (by calling to_proc on it) before associating it with the method as its block.

def hello
  yield if block_given?
end

class FooBar
  def to_proc
    -&gt; {puts 'converted lambda'}
  end
end

hello(&FooBar.new)

1	converted lambda

All of this seems pretty clear, but what if I want to take a block that was associated with a method and pass it to another method? We need a way to refer to our block.

The Explicit Block

When we write our method definition, we can explicitly state that we expect this method to possibly take a block. Confusingly, Ruby uses the ampersand for this as well:

def hello(&block)
  yield if block_given?
end

hello do
  puts "hello"
end

Defining our method this way, gives us a name by which we can refer to our block within the method body. And since our block is a Proc object, instead of yielding to it, we can call it:

def hello(&block)
  block.call if block_given?
end

hello do
  puts "hello"
end

I prefer block.call instead of yield, it makes things clearer. Of course, when we define our method we don’t have to use the name ‘block’, we can do:

def hello(&foo)
  foo.call if block_given?
end

hello do
  puts "hello"
end

Having said that; ‘block’ is a good convention.

So, in the context of methods and blocks, there are two ways we use the ampersand:

in the context of a method definition, putting an ampersand in front of the last parameter indicates that a method may take a block and gives us a name to refer to this block within the method body
in the context of a method call, putting an ampersand in front of the last argument tells Ruby to convert this argument to a Proc if necessary and then use the object as the method’s block

Passing Two Blocks To A Method

It is instructive to see what happens when you try to pass a both a regular block and a block argument to a method:

def hello(&block)
  block.call if block_given?
end

blah = -&gt; {puts "lambda"}

hello(&blah) do
  puts "hello"
end

You get the following error message:

1	code.rb:56: both block arg and actual block given

It is not even an exception – it’s a syntax error!

Using Another Method As A Block

It’s also interesting to note that since you can easily get a reference to a method in ruby and the Method object implements to_proc, you can easily give one method as a block to another e.g.:

def hello(&block)
  block.call if block_given?
end

def world
  puts "world"
end

method_reference = method(:world)

hello(&method_reference)

world

Passing The Block Around

We now know enough to easily understand our first example:

def blah(&block)
  yadda(block)
end

def yadda(block)
  foo(&block)
end

def foo(&block)
  block.call
end

blah do
  puts "hello"
end

we define blah to expect a block, inside blah we can refer to this block as block
we define yadda to expect one parameter, this parameter would be referred to as block inside yadda, but it is not a block in that we could not yield to it inside yadda
foo also expects a block and we can refer to this block as block inside foo
when we call yadda from within blah we pass it our block variable without the ampersand, since yadda does not a expect a block parameter, but simply a regular method argument, in our case this regular method argument will just happen to be a Proc object
when we call foo from inside yadda we pass it our block variable, but this time with an ampersand since foo actually expects a block and we want to give it a block rather than just a regular variable

It should now be much more obvious why the ampersand is used in some cases, but not in others.

The Symbol To Proc Trick

We should now also have a lot less trouble understanding the ‘symbol to proc’ trick. You’ve no doubt seen code like this:

`1`	`p ["a", "b"].map(&:upcase)`

We know that this is equivalent to:

`1`	`p ["a", "b"].map{\|string\| string.upcase}`

But now we also make an educated guess as to why they are equivalent. We have a Symbol object (’:upcase’), we put an ampersand in front of it and pass it to the map method. The map method takes a block, and by using the ampersand we’ve told Ruby that we want to convert our Symbol object to a Proc and associate it with the map method as its block. It turns out that Symbol implements to_proc in a special way, so that the resulting block becomes functionally equivalent to our second example above. Of course these days Ruby implements Symbol#to_proc using C, so it’s not quite as nice looking as the examples you’ll find around the web, but you get general idea.

Anyway, hopefully this makes blocks and ampersands a bit more friendly. It’s definitely worth it to occasionally revisit the basics, I’ll try to do it more often in the future.

Image by Martin Whitmore

Ruby – Why U No Have Nested Exceptions?

Tue, 09 Apr 2013 13:53:21 +0000

One of the things we almost always do these days when we write our libraries and apps, is use other libraries. Inevitably something will go wrong with those libraries and exceptions will be produced. Sometimes these are expected (e.g. an HTTP client that produces an exception when you encounter a 500 response or a connection timeout), sometimes they are unexpected. Either way you don’t want to allow the exceptions from these external libraries to bubble up through your code and potentially crash your application or cause other weirdness. Especially considering that many of these exceptions will be custom types from the libraries you’re using. No-one wants strange exceptions percolating through their code.

What you want to do, is ensure that all interactions with these external libraries are wrapped in a begin..rescue..end. You catch all external errors and can now decide how to handle them. You can throw your hands up in the air and just re-raise the same error:

begin
  SomeExternalLibrary.do_stuff
rescue =&gt; e
  raise
end

This doesn’t really win us anything. Better yet you would raise one of your own custom error types.

begin
  SomeExternalLibrary.do_stuff
rescue =&gt; e
  raise MyNamespace::MyError.new
end

This way you know that once you’re past your interfaces with the external libraries you can only encounter exception types that you know about.

The Need For Nested Exceptions

The problem is that by raising a custom error, we lose all the information that was contained in the original error that we rescued. This information would have potentially been of great value to help us diagnose/debug the problem (that caused the error in the first place), but it is lost with no way to get it back. In this regard it would have been better to re-raise the original error. What we want is to have the best of both worlds, raise a custom exception type, but retain the information from the original exception.

When writing escort one of the things I wanted was informative errors and stack traces. I wanted to raise errors and add information (by rescuing and re-raising) as they percolated through the code, to be handled in one place. What I needed was the ability to nest exceptions within other exceptions.

Ruby doesn’t allow us to nest exceptions. However, I remembered Avdi Grimm mentioning the nestegg gem in his excellent Exceptional Ruby book, so I decided to give it a try.

The Problems With Nestegg

Unfortunately nestegg is a bit old and a little buggy:

It would sometimes lose the error messages
Nesting more than one level deep would cause repetition in the stacktrace

I also didn’t like how it made the stack trace look non-standard when including the information from the nested errors. If we take some code similar to the following:

require 'nestegg'

class MyError &lt; StandardError
  include Nestegg::NestingException
end

begin
  1/0
rescue =&gt; e
  begin
    raise MyError.new("Number errors will be caught", e)
  rescue =&gt; e
    begin
      raise MyError.new("Don't need to let MyError bubble up")
    rescue =&gt; e
      raise MyError.new("Last one for sure!")
    end
  end
end

It would produce a stack trace like this:

examples/test1.rb:26:in `rescue in rescue in rescue in &lt;main&gt;': MyError (MyError)
	from examples/test1.rb:23:in `rescue in rescue in &lt;main&gt;'
	from examples/test1.rb:20:in `rescue in &lt;main&gt;'
	from examples/test1.rb:17:in `&lt;main&gt;'
	from cause: MyError: MyError
	from examples/test1.rb:24:in `rescue in rescue in &lt;main&gt;'
	from examples/test1.rb:20:in `rescue in &lt;main&gt;'
	from examples/test1.rb:17:in `&lt;main&gt;'
	from cause: MyError: MyError
	from examples/test1.rb:21:in `rescue in &lt;main&gt;'
	from examples/test1.rb:17:in `&lt;main&gt;'
	from cause: ZeroDivisionError: divided by 0
	from examples/test1.rb:18:in `/'
	from examples/test1.rb:18:in `&lt;main&gt;'

After looking around I found loganb-nestegg. This fixed some of the bugs, but still had the non-standard stack trace and the repetition issue.

When you’re forced to look for the 3rd library to solve a problem, it’s time to write your own.

This is exactly what I did for escort. This functionality eventually got extracted into a gem which is how we got nesty. Its stack traces look a lot like regular ones, it doesn’t lose messages and you can nest exceptions as deep as you like without ugly repetition in the stack trace. If we take the same code as above, but redefine the error to use nesty:

1
2
3

class MyError &lt; StandardError
  include Nesty::NestedError
end

Our stack trace will now be:

examples/complex.rb:20:in `rescue in rescue in rescue in &lt;main&gt;': Last one for sure! (MyError)
	from examples/complex.rb:17:in `rescue in rescue in &lt;main&gt;'
	from examples/complex.rb:18:in `rescue in rescue in &lt;main&gt;': Don't need to let MyError bubble up
	from examples/complex.rb:14:in `rescue in &lt;main&gt;'
	from examples/complex.rb:15:in `rescue in &lt;main&gt;': Number errors will be caught
	from examples/complex.rb:11:in `&lt;main&gt;'
	from examples/complex.rb:12:in `/': divided by 0
	from examples/complex.rb:12:in `&lt;main&gt;'

Definitely nicer. We simply add the messages for every nested error to the stack trace in the appropriate place (rather than giving them their own line).

How Nested Exceptions Work

The code for nesty is tiny, but there are a couple of interesting bits in it worth looking at.

One of the special variables in Ruby is $! which always contains the last exception that was raised. This way when we raise a nesty error type, we don’t have to supply the nested error as a parameter, it will just be looked up in $!.

Ruby always allows you to set a custom backtrace on any error. So, if you rescue an error you can always replace its stack trace with whatever you want e.g.:

begin
  1/0
rescue =&gt; e
  e.message = "foobar"
  e.set_backtrace(['hello', 'world'])
  raise e
end

This produces:

1 2	hello: divided by 0 (ZeroDivisionError) from world

We take advantage of this and override the set_backtrace method to take into account the stack trace of the nested error.

def set_backtrace(backtrace)
  @raw_backtrace = backtrace
  if nested
    backtrace = backtrace - nested_raw_backtrace
    backtrace += ["#{nested.backtrace.first}: #{nested.message}"]
    backtrace += nested.backtrace[1..-1] || []
  end
  super(backtrace)
end

To produce the augmented stack trace we note that the stack trace of the nested error should always be mostly a subset of the enclosing error. So, we whittle down the enclosing stack trace by taking the difference between it and the nested stack trace (I think set operations are really undervalued in Ruby, maybe a good subject for a future post). We then augment the nested stack trace with the error message and concatenate it with what was left over from the enclosing stack trace.

Anyway, if you don’t want exceptions from other libraries invading your app, but still want the ability to diagnose the cause of the exceptions easily – nested exceptions might be the way to go. And if you do decide that nested exceptions are a good fit, nesty is there for you.

Image by Samuel M. Livingston

The Best Way To Pretty Print JSON On The Command-Line

Wed, 03 Apr 2013 07:53:23 +0000

Developers tend to work with APIs a lot and these days most of these APIs are JSON. These JSON strings aren’t exactly easy to read unless they are formatted well. There are many services online that can pretty print JSON for you, but that’s annoying. I love the command-line and whenever I am playing with new APIs or writing my own I mostly use CURL which means I need a good way to pretty print my JSON on the command-line. It should be simple, quick, easy to remember, easy to install – we’re not trying to solve complex algorithms, just format some JSON.

The ‘Good’ Old Faithful

One way that is always available is the Python JSON tool. So you can always do this:

1	echo '{"b":2, "a":1}' \| python -mjson.tool

Which will give you:

{
    "a": 1,
    "b": 2
}

This is alright and, as I said, it is always available. However note that it has sorted our keys which is a major disadvantage. It is also a bit of a handful to write when you just want to pretty print some JSON. I only ever use this when I am on an unfamiliar computer and there is nothing better.

YAJL Tools

If you’re not using YAJL you should be. It is a small JSON library written in C. The parser is event driven and super fast. In the Ruby world we have a nice set of bindings and there are bindings for other languages as well. It is my go-to JSON library.

YAJL also cames with a couple of tools, json_reformat and json_verify. These are pretty self-explanatory and you can get your hands on them like this:

1	brew install yajl

1	sudo apt-get install yajl-tools

After that all you have to do is:

1	echo '{"b":2, "a":1}' \| json_reformat

Which will give you:

{
    "b": 2,
    "a": 1
}

This is pretty nice. My one tiny niggle is that json_reformat is still a bit of a mouthful, but if you just want basic JSON formatting, it’s a good solution.

Ppjson

Of course being developers, we don’t have to put up with even minor niggles since we can build ourselves exactly the tools we need (and a little bit extra to boot). I was writing a command-line framework and I just happened to need a command-line tool, which is how I ended up with ppjson. I use Ruby quite a bit, so it is a Ruby tool and even if I do say so myself, it’s pretty nice.

You just need to:

1	gem install ppjson

This will let you do:

1	echo '{"b":2, "a":1}' \| ppjson

Which will give you:

{
  "b": 2,
  "a": 1
}

It uses YAJL through multi_json under the hood, so you still get the speed of YAJL, but it can also do a few extra things for you.

You can pass or pipe it some JSON and it will pretty print it to standard output for you. But you can also pass it a file name with some JSON in it and it will pretty print the contents of the file for you:

1	ppjson -f abc123.json

You can also store the pretty printed contents back into the file:

1	ppjson -fi abc123.json

Sometimes you have saved some pretty printed JSON in a file, but now you want to use it as a body of a CURL POST request, for example. Well ppjson can uglify your JSON for you as well:

1	ppjson -fu abc123.json

This will output a minified JSON string to standard output. And of course you can also update the original file with the uglified JSON as well:

1	ppjson -fui abc123.json

It will do you basic JSON pretty printing with an easy to remember executable name, but it also has a few nice convenience features to make your life that little bit easier.

The best part is that using Escort](https://github.com/skorks/escort), it was a breeze to write. I’ll [talk about some of the other interesting projects that ‘fell out’ of Escort some other time.

Anyway, now you no longer need to remember that IDE or editor shortcut to format your JSON or look for an online tool to do the same, the command-line has you more than covered.

Image by NS Newsflash

I Was Nominated To Speak At LoneStarRuby Conference 2013

Mon, 01 Apr 2013 07:38:06 +0000

A few days ago I woke up to see this in my Twitter stream:

@indianguru Your recommendation for @skorks to speak at @lsrc is now public. Go to portal.lonestarruby.org to vote!

— Lance Vaughn (@lance_io) March 29, 2013

Looking at the list of nominated speakers, I am in extremely illustrious company which was just amazing to see. Of course I replied with my appreciation. Then a little later I saw this:

@skorks @lance_io @indianguru @lsrc I'd leave SF for atx for @skorks … And @torchytaco

— Steven G. Harms (@sgharms) March 30, 2013

Which was also very nice.

Who would have thought something like this would ever happen – thanks @IndianGuru.

Anyway, if LoneStarRuby Conference 2013 is of any interest to you and you’d like to ~~make me write a talk~~ hear me speak, you can go and vote.

A Closure Is Not Always A Closure In Ruby

Wed, 27 Mar 2013 05:53:18 +0000

Often you don’t really think about what particular language features mean, until those features come and bite you. For me one such feature was Ruby’s instance_eval. We all know how instance_eval works, it’s pretty straight forward. We pass it a block, and that block gets executed in the context of the object on which we call instance_eval e.g.:

a = Object.new
a.instance_eval do
  def hello
    puts "hello"
  end
end
a.hello

The object a will get a hello method and the subsequent a.hello call will succeed. So what’s the gotcha?

The one overwhelming use of instance_eval is for creating configuration DSLs. I won’t go into that right now – it deserves a post of its own. Suffice to say that when used for this purpose, the closure semantics of the block that you pass to instance_eval don’t matter, i.e. you don’t tend to use the variables (or call methods) from the scope where the block was created. If you do need to use the variables or methods from the scope where the block was created, you might be in for a rude surprise.

What It Means To Evaluate A Block In A Different Context

Let’s create a class with a method that takes a block and call that method from within another class:

class A
  def method1(&block)
    block.call
  end
end

class B
  def initialize
    @instance_var1 = "instance_var1"
    @a = A.new
  end

  def method2
    local1 = "local1"
    @a.method1 do
      p local1
      p @instance_var1
      p helper1
    end
  end

  def helper1
    "helper1"
  end
end

b = B.new
b.method2

As expected we get:

1
2
3

"local1"
"instance_var1"
"helper1"

What happens is that when we create the block inside method2, there is a binding that goes along with it. Inside the binding there is a self object which is our instance b of the B class. When we execute the block inside method1, it is executed within the context of this self and so our local variable, instance variable and helper method are all in scope (just as you would expect from a closure) and everything works fine. Let’s modify method1 slightly to get the self from the binding:

class A
  def method1(&block)
    puts block.binding.eval("self").object_id
    block.call
  end
end

b = B.new
p b.object_id
b.method2

We get:

70229187307420
70229187307420
"local1"
"instance_var1"
"helper1"

So both b and the self inside the binding are the same object as we expected.

But, what happens if instead of calling method1 with a block on @a we instance_eval the same block within the context of @a.

class A
end

class B
  def initialize
    @instance_var1 = "instance_var1"
    @a = A.new
  end

  def method2
    local1 = "local1"
    @a.instance_eval do
      p local1
      p @instance_var1
      p helper1
    end
  end

  def helper1
    "helper1"
  end
end

b = B.new
b.method2

We get:

1
2
3

"local1"
nil
code2.rb:19:in `block in method2': undefined local variable or method `helper1' for #&lt;A:0x007f966292d308&gt; (NameError)

This time local1 is fine, the instance variable is nil and trying to call the method raises an error. This is unexpected, the block should be a closure, but only the local variables were closed over, everything else fell apart. This is not a new issue, but it is important to understand what is happening and why.

When we call instance_eval on an object, the self within the block is set to be the object on which we called instance_eval (Yahuda has more detail and there is even more detail here). So, even though we still manage to capture the locals from the previous scope, the methods (like helper1) are no longer in scope within our new self and the instance variables will be equal to nil since we haven’t initialized them in this new scope (unless you happen to have an instance variable with the same name in which case it will shadow the one from the scope in which the block was defined – which is probably not what you want).

So, even though we know that blocks in Ruby are closures, there is an exception to every rule.

How To Overcome The Limitations

So what can we do if we want access to the instance variables and helper method from the scope where the block was defined. Well, for instance variables, one way around would be to pass them in as parameters to the instance_eval block, that way they could be treated as locals. Unfortunately instance_eval doesn’t take parameters. Luckily some time around Ruby 1.8.7 instance_exec was introduced. Since instance_eval is so much more popular we sometimes forget that instance_exec is even there, it essentially does the same thing as instance_eval, but you can pass some arguments to the block.

class B
  def initialize
    @instance_var1 = "instance_var1"
    @a = A.new
  end

  def method2
    local1 = "local1"
    @a.instance_exec(@instance_var1) do |instance_var1|
      p local1
      p instance_var1
      p helper1
    end
  end

  def helper1
    "helper1"
  end
end

b = B.new
b.method2

We get:

1
2
3

"local1"
"instance_var1"
code2.rb:15:in `block in method2': undefined local variable or method `helper1' for #&lt;A:0x007fea7912d3e0&gt; (NameError)

This time our instance variable is fine since we’ve turned it into an argument, but we still can’t call the method. Still not nice, but instance_exec is obviously somewhat more useful than instance_eval. We will come back to handling method calls shortly, but first a bit of history.

So What Did They Do Before `instance_exec` Existed

This issue has been around for ages, but instance_exec has only been around for a few years, so what did they do before that, when they wanted to pass parameters to instnace_eval?

When I decided to write Escort, I chose Trollop as the option parser. It is while reading the Trollop source that I accidentally stumbled upon the answer to the above question. The author of Trollop attributes it to _why. It’s called the cloaker and we use it as a replacement for instance_eval. Here is how it works:

class A
  def cloaker(&b)
    (class &lt;&lt; self; self; end).class_eval do
      define_method :cloaker_, &b
      meth = instance_method :cloaker_
      remove_method :cloaker_
      meth
    end
  end
end

class B
  def initialize
    @instance_var1 = "instance_var1"
    @a = A.new
  end

  def method2
    local1 = "local1"
    @a.cloaker do |instance_var1|
      p local1
      p instance_var1
      p helper1
    end.bind(@a).call(@instance_var1)
  end

  def helper1
    "helper1"
  end
end

b = B.new
b.method2

We get:

1
2
3

"local1"
"instance_var1"
code2.rb:24:in `block in method2': undefined local variable or method `helper1' for #&lt;A:0x007fb963064900&gt; (NameError)

It is somewhat more awkward to use, but the output is identical to that of instance_exec. So why does this work?

We define a method on the metaclass of A using our block as the body, we then remove it and return the UnboundMethod. We then bind the unbound method to our target object and call the method.

In order for the bind to work, the relationship between the object to which we are binding and the object on which the UnboundMethod was originally defined must be a kind_of? == true. Curiously an instance of the class A is a kind_of? metaclass of A which is why everything works.

The Many Faces Of Cloaker

Why did we have to define our UnboundMethod on the metaclass of A? Well there doesn’t seem to be any good reason really. A cloaker like this will work just fine.

class A
  def cloaker(&b)
    self.class.instance_eval do
      define_method :cloaker_, &b
      meth = instance_method :cloaker_
      remove_method :cloaker_
      meth
    end
  end
end

So will this one:

class A
  def cloaker(&b)
    self.class.class_eval do
      define_method :cloaker_, &b
      meth = instance_method :cloaker_
      remove_method :cloaker_
      meth
    end
  end
end

The kind_of? relationship will hold in both cases so everything will hang together.

Did we really need to go through all that history, we just want to call helper methods from within the scope where our block is defined?

What To Do About Method Calls

Let’s have a look at how we can get our helper methods back. One obvious way is to assign self to a local and then use this local to call our helper methods from within the instance_exec block.

class A
end

class B
  def initialize
    @instance_var1 = "instance_var1"
    @a = A.new
  end

  def method2
    local1 = "local1"
    local_self = self
    @a.instance_exec(@instance_var1) do |instance_var1|
      p local1
      p instance_var1
      p local_self.helper1
    end
  end

  def helper1
    "helper1"
  end
end

b = B.new
b.method2

We get:

1
2
3

"local1"
"instance_var1"
"helper1"

Everything works, but it is ugly and if you want to call a private helper method, you’re out of luck again since a private method can’t have an explicit receiver. This is where our cloaker comes into its own again. Using a cloaker and a bit of method_missing magic we can create our own version of instance_exec that has all the semantics we’ve come to expect from a closure, but will still execute the block within the context of the object on which it is called. We can also abstract away the ugly bind stuff while we’re at it.

class A
  def cloaker(*args, &b)
    meth = self.class.class_eval do
      define_method :cloaker_, &b
      meth = instance_method :cloaker_
      remove_method :cloaker_
      meth
    end
    with_previous_context(b.binding) {meth.bind(self).call(*args)}
  end

  def with_previous_context(binding, &block)
    @previous_context = binding.eval('self')
    result = block.call
    @previous_context = nil
    result
  end

  def method_missing(method, *args, &block)
    if @previous_context
      @previous_context.send(method, *args, &block)
    end
  end
end

class B
  def initialize
    @instance_var1 = "instance_var1"
    @a = A.new
  end

  def method2
    local1 = "local1"
    @a.cloaker(@instance_var1) do |instance_var1|
      p local1
      p instance_var1
      p helper1
    end
  end

  private
  def helper1
    "helper1"
  end
end

b = B.new
b.method2

Despite the fact that helper1 was private, we get:

1
2
3

"local1"
"instance_var1"
"helper1"

Essentially when we execute the block within the context of on object; we retain (in an instance variable) the self from the context where the block was defined (we get this self from the biding of the block). When a method is called within the block and it is not defined within the current context – method_missing will be called as per usual Ruby semantics. We hook into method_missing and send the method call to the self that we have retained from when the block was defined. This will call our method or fail if it was a genuine error.

Despite the fact that Ruby can surprise you with non-closure blocks, you can have your cake and eat it too, if you’re willing to do just a bit of work and give some new legs to an old trick while you’re at it.

Images by Jenn and Tony Bot and TarynMarie

What Is A Startup (Or A Startup Idea)?

Wed, 23 Nov 2011 00:18:05 +0000

Is a startup all about the idea? Is it a bunch of people hacking together? It is not quite a company in the traditional sense of the word (at least not yet). I like to think of a startup as a series of assumptions. Your startup idea is just one assumption or alternatively there are many assumptions inherent in your idea. Let’s take a simple one we’re all familiar with:

“People want to connect with their friends online”

The main assumption is that people actually DO want to connect with their friends online (of course we know now that they do, but a few years ago it wasn’t so obvious). Some of the inherent assumptions might be:

People want to know what their friends are doing right now
People want to see interesting links from their friends
We have a way to acquire users in the initial stages
People will want to let their friends know about our service
People will want to pay for our service
etc.

Your startup is really just a means to validate all those assumptions. If you can do it, then you may pronounce your startup a success (it’s not necessarily a success yet, but the rest is really just optimisation), if you can’t then it is a failure. Of course things are a bit more complex than that.

If you follow the Lean Startup movement at all, you’re probably aware of pirate metrics (the pirate says ‘AARRR’). We have Acquisition, Activation, Retention, Referral, Revenue. The thing about those metrics is that they are a funnel and they are all about users. We start with lots of users in the Acquisition stage and we end up with less users in the Revenue stage.

All the assumptions inherent in your idea are a funnel too (think of them as a more specific version of the pirate funnel) – you start with a bunch of potential users at the top and validating each of the assumptions will whittle away some of those users. Let’s take this simplified example:

The more people you can get through that funnel successfully the more sure you can be that you’re onto a good thing, but if you end up with zero users before you get to the end of your funnel then you have a problem. If no-one wants to click the red button, then people are unlikely to ever tell their friends or upgrade their accounts. Your assumption is flawed and you need to adjust. Perhaps it’s as simple as making the button green, or you may have to think of a way to achieve the same result without a button, or there might be a fundamental flaw in the model, in which case you might need to go back to square one, or pivot, or abandon the idea all together.

Of course your assumption funnel is not strict, you can work on any assumption at any time, but you need to be fully aware of what you’re doing. Firstly, it is easier to test the assumptions at the top of the funnel. You can figure out whether or not you can drive some initial traffic with nothing more than a dummy landing page, but to test retention you need to have built something. Secondly, there is ‘Danger, Will Robinson!’ every time you work on one of the assumptions lower down in the funnel first. You may optimise your big red button with 3 of your mates (the only users you have), but if you can never get anyone to come to the site at all, what’s the use? It can still all work out, especially if you’re flush with money and have all the time in the world, but it’s probably faster and easier to prove those assumptions in order.

Incidentally this is very similar to the problem we ran into with CrowdHired forcing us to essentially rebuild the site (in a much slimmed down fashion) and to try and bootstrap (userbase-wise) from the YOW! developer conference. Since we’re not flush with time and money it was a costly lesson, but one well learned.

The relevant concept is this, even without building anything you can easily figure out the key actions you and your potential users will have to take in order for your idea to work. You can then come up with a set of assumptions relating to each of those actions and arrange them into a funnel. You can then build the minimal amount of stuff to try and prove each assumption in turn. As you follow this process it can potentially give you some guidance regarding a number of things, such as user behaviour, where to take your idea next, when to pivot, when to get some more funding (e.g. the only way to prove this next assumption is with 10 people working for a year :)) etc.

It goes without saying that depending on the type of startup your trying to build (network effect, B2B, B2C, etc. – see what I did there, heh), the mechanics will need adjustment. It also goes without saying that you need to validate all assumptions properly, asking 2 of your friends if they would click a button you just drew on a napkin is not an indicator of anything. In-fact this is one of the things I’d like to explore further, but I guess I’ll leave it until next time.

Image by spaceninja

Why Developers Never Use State Machines

Thu, 01 Sep 2011 00:38:47 +0000

A few months ago I saw a great little blog post about state machines on the Shopify blog. The message was that state machines are great and developers should use them more – given my recent experiences with state machines at CrowdHired, I could certainly agree with that. But it got me thinking, how many times in my developer career have I actually used a state machine (either separate library or even hand-rolled abstraction)? The answer is zero times – which surprised the hell out of me since state machines really are very useful. So I decided to engage in a bit of introspection and figure out why we tend to manage our “state” and “status” fields in an ad-hoc fashion rather than doing what is clearly called for.

We Don’t Need One Until We Do

The problem is that you almost never create an object fully formed with all the behaviour it is ever going to need, rather you build it up over time. The same is true for the “states” that a state machine candidate object can be in. So, early on you don’t feel like your objects’ state machine behaviour is complex enough to warrant a “full-blown” state machine (_YAGNI and all that jazz), but later on – when it IS complex enough – you feel like you’ve invested too much time/effort to replace it with something that has equivalent functionality. It’s a bit of a <a href=“http://en.wikipedia.org/wiki/Catch-22(logic)” target=”_blank”>catch-22. It’s overkill and by the time it’s not, it’s too late.

A State Machine Is A Fluffy Bunny (Not Particularly Threatening)

Those of us who went through computer science degrees remember state machines from our computing theory subjects and the memories are often not fond ones. There are complex diagrams and math notation, determinism and non-determinism, Moore and Mealy, as well as acronyms galore (DFA, NFA, GNFA etc.). We come to believe that state machines are more complex than they actually are and it is therefore nothing but pragmatism that makes us consider a “full-blown” state machine overkill.

But most state machines you’re likely to need in your day-to-day development have nothing in common with their computing theory counterparts (except the … errr … theory). You have states which are strings, and events which are methods that cause transitions from one state to another – that’s pretty much it (at least for the state_machine gem in Ruby). The point is, even if you have two states, a state machine is not overkill, it might be easier that rolling an ad-hoc solution, as long as you have a good library to lean on.

Even A Good Tool Is Not A Good Tool

I would hazard a guess that there are decent state machine libraries for most languages that you can use (the aforementioned state_machine for Ruby is just one example). But even a fluffy bunny has a learning curve (I am stretching the metaphor well past breaking point here). That wouldn’t be such an issue if you were solving a problem, but all you’re likely doing is replacing an existing solution. Since we tend to turn to a state machine library after the fact (our ad-hoc solution is working right now). Just like with everything that has “potential future benefits” the immediate value is very hard to justify even to yourself (unless you’ve had experience with it before). The slight learning curve only tips the scale further towards the “we can live without it” side. It doesn’t matter how good a tool is if you never give it a chance.

It is really difficult to appreciate (until you’ve gone through it) – how much better life can be if you do give a good state machine library a chance. When we finally “bit the bullet” at CrowdHired and rejigged some of our core objects to use the state_machine gem, the difference was immediately apparent.

Firstly the learning curve was minor, I did spend a few hours of going through the source and documentation, but after that I had a good idea what could and couldn’t be done (I might do an in-depth look at the state_machine gem at some point).
The integration itself was almost painless, but moving all the code around to be inline with the new state machine was a big pain. In hindsight had we done this when our objects only had a couple of states it would have been a breeze.
We’re now able to easily introduce more states to give our users extra information as well as allow us to track things to a finer grain. Before it was YAGNI cause it was a pain, now we find that we “ai gonna need” after all, cause it’s so easy.
Our return values from state transitions are now 100% consistent (true/false). Before we were returning objects, arrays of objects, nil, true/false depending on who was writing it and when.
We’re now able to keep an audit trail of our state transitions simply by dropping in statemachine-audit_trail (_see that Shopify post), before it was too hard to hook it in everywhere so we had nothing.
We removed a bunch of code and improved our codebase – always worthy goals as far as I am concerned.

My gut-feel is that most people who read that Shopify post agreed with it in spirit, but did nothing about it (that’s kinda how it was with me). We seem to shy away from state machines due to misunderstanding of their complexity and/or an inability to quantify the benefits. But, there is less complexity than you would think and more benefits than you would expect as long you don’t try to retrofit a state machine after the fact. So next time you have an object that even hints at having a “status” field, just chuck a state machine in there, you’ll be glad you did. I guarantee it or your money back :).

Image by tfangel

When Developers Go To Great Length To Save Typing 4 Letters

Wed, 24 Aug 2011 07:43:19 +0000

Heroku is a great platform. Long before I joined and when I say long, I mean in startup terms (i.e. a few weeks before I joined :)) – the decision was made that CrowdHired would be hosted on Heroku. Shortly after I came on board, Heroku released their new Cedar stack and we quickly migrated across to that. I find it kinda amusing that we’re currently in alpha, deploying to a platform that’s in beta. Latest and greatest FTW. While migrating to the new stack we also settled on Thin as our web server. The Cedar stack allows you to use whatever web server you want in production and will run on Webrick by default – not ideal. Since we were going to use Thin in production it made sense that we’d also use it in development instead of Webrick.

When you’re using Rails, swapping a new web server in is pretty painless. Just include the gem and then use the rails s command to launch your new server e.g

rails s thin```

Pretty simple right? Well, it is, but I am very used to typing `rails s` to launch my server and no matter what gem you include in your project, `rails s` still starts Webrick (_this is not entirely accurate but bare with me_). Muscle memory being what it is, after typing `rails s` instead of `rails s thin` a couple of times (_and only realising this a few hours later_) I decided to see if I could make Thin the default for the `rails s` command.

## Digging Into The Rails Command Line

The key thing here was to figure out how `rails s` actually works &#8211; only way to do that is to <a href="http://www.skorks.com/2010/05/why-i-love-reading-other-peoples-code-and-you-should-too/" target="_blank">read some code</a>. We know that there is a `script/rails` executable that lives in all our Rails project so it makes sense that this would be the entry point into figuring out `rails s`, but in reality it's not (_or at least it's not that simple_). We don't actually type `script/rails s`, we do `rails s`, so **there must be an executable called `rails` within the Rails gem itself** which is declared as such in rails' gemspec. This is indeed the case, it looks like this:

```ruby
#!/usr/bin/env ruby
 
if File.exists?(File.join(File.expand_path('../..', __FILE__), '.git'))
  railties_path = File.expand_path('../../railties/lib', __FILE__)
  $:.unshift(railties_path)
end
require "rails/cli"

But even that is not the start of the story. Apparently, when you have an executable in a gem, rubygems will not use it as is, but will generate another executable which wraps your one. For the rails command it looks like this:

#!/usr/bin/env ruby
#
# This file was generated by RubyGems.
#
# The application 'rails' is installed as part of a gem, and
# this file is here to facilitate running it.
#
 
require 'rubygems'
 
version = "&gt;= 0"
 
if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
  version = $1
  ARGV.shift
end
 
gem 'rails', version
load Gem.bin_path('rails', 'rails', version)

This is the true entry point you hit when you type rails s. Of course, all this does is load/call the original executable from the Rails gem.

The Rails source is broken up into several projects such as activerecord, activesupport etc. Probably the most important one of these is railties. This is where the rails executable takes us. Of course, before it does that it needs to put the lib/ folder of the railties project on the load path, but eventually we end up in railties/lib/rails/cli.rb. Here we almost immediately execute the following:

Rails::ScriptRailsLoader.exec_script_rails!```

All this does is essentially figure out if we're inside a Rails app and if we are it executes `script/rails` passing through the command line arguments that you supply. So, we're now back in our Rails app; `script/rails` is the real entry point after all (_although we're about to be taken straight back to **railties**_). The file looks like this:

```ruby
#!/usr/bin/env ruby
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
 
APP_PATH = File.expand_path('../../config/application',  __FILE__)
require File.expand_path('../../config/boot',  __FILE__)
require 'rails/commands'

We require boot.rb so that we can hook into Bundler and make sure the relevant gems are on the load path (such as rails for example), we then jump into railties/rails/lib/commands.rb. Here everything is pretty straight forward, we have a big case statement which has a clause for “server”. We instantiate a new Rails::Server and start it, which tells us very little by itself, but if we jump into railties/rails/lib/commands/server.rb we can see that Rails::Server simply extends Rack::Server (and delegates to Rack::Server's start method from its start method) all it adds is those familiar lines we’re all used to seeing e.g.:

=&gt; Booting Thin
=&gt; Rails 3.0.7 application starting in development on http://0.0.0.0:3000
=&gt; Call with -d to detach
=&gt; Ctrl-C to shutdown server

So, if we want to change which server is picked up by default when we type rails s we need to go look in Rack.

A Quick Glance At Rack

Luckily we can easily grab it from Github and crack it open (you have to admire the Ruby open source ecosystem, it is truly awesome, largely due to Github, so big props to those guys). We of course need to check out lib/rack/server.rb where we find the following method:

1
2
3

def server
  @_server ||= Rack::Handler.get(options[:server]) || Rack::Handler.default(options)
end

So, if we don’t pass in the name of the server we want, Rack::Handler.default will try to determine it for us.

def self.default(options = {})
  # Guess.
  if ENV.include?("PHP_FCGI_CHILDREN")
    # We already speak FastCGI
    options.delete :File
    options.delete :Port
 
    Rack::Handler::FastCGI
  elsif ENV.include?("REQUEST_METHOD")
    Rack::Handler::CGI
  else
    begin
      Rack::Handler::Mongrel
    rescue LoadError
      Rack::Handler::WEBrick
    end
  end
end

As you can see, it turns out that the real default server is actually Mongrel. So if you included Mongrel in your Rails project, typing rails s would automatically pick that up without you having to do anything. Only if Mongrel fails, do we fall back to Webrick which is part of Ruby and therefore is always present. So what do we do if we want Thin to be one of the defaults? Well, first thing first, we need to check if Rack includes a handler for Thin. If we look in lib/rack/handlers/ we can see that Rack itself includes the following:

cgi.rb
evented_mongrel.rb
fastcgi.rb
lsws.rb
mongrel.rb
scgi.rb
swiftiplied_mongrel.rb
thin.rb
webrick.rb

Luckily Thin is there, so what we really want is that default method to look something like this:

def self.default(options = {})
  # Guess.
  if ENV.include?("PHP_FCGI_CHILDREN")
    # We already speak FastCGI
    options.delete :File
    options.delete :Port
 
    Rack::Handler::FastCGI
  elsif ENV.include?("REQUEST_METHOD")
    Rack::Handler::CGI
  else
    begin
      Rack::Handler::Thin
    rescue LoadError
      begin
        Rack::Handler::Mongrel
      rescue LoadError
        Rack::Handler::WEBrick
      end
    end
  end
end

This way Thin will be the first default, followed by Mongrel and only then falling through to Webrick. Luckily, since we’re using Ruby, we can reopen the class and replace the method with our version. I think this is a perfect example where the ability to reopen classes comes in extremely handy, without any need to worry about “scary consequences“.

Getting It Working

All we really need to figure out is where to put the code that reopens the class so that it gets picked up before Rails tries to launch the server. The only logical place is the script/rails executable itself, which will now look like this:

#!/usr/bin/env ruby
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
 
APP_PATH = File.expand_path('../../config/application',  __FILE__)
require File.expand_path('../../config/boot',  __FILE__)
require 'rack/handler'
Rack::Handler.class_eval do
  def self.default(options = {})
    # Guess.
    if ENV.include?("PHP_FCGI_CHILDREN")
      # We already speak FastCGI
      options.delete :File
      options.delete :Port
 
      Rack::Handler::FastCGI
    elsif ENV.include?("REQUEST_METHOD")
      Rack::Handler::CGI
    else
      begin
        Rack::Handler::Mongrel
      rescue LoadError
        begin
          Rack::Handler::Thin
        rescue LoadError
          Rack::Handler::WEBrick
        end
      end
    end
  end
end
require 'rails/commands'

This works without any problems, we type rails s and as long as Thin is in our Gemfile it starts up by default. As an aside, notice that I used class_eval to reopen Rack::Handler. Metaprogramming tricks like this should be part of every Ruby developer’s toolbox, I’ll talk more about this some other time (seeing as I am well into TL;DR land here).

Going through this exercise didn’t take long (under 30 minutes) and taught me a bit more about Rails and Rack. Shortly after doing this – in a curious case of the universe working in interesting ways – I came across a Stackoverflow question asking about this exact scenario and got an inordinate amount of satisfaction from being able to easily answer it :). In-fact, even the fact that shortly after Jason found Pow and we switched over to that, doesn’t really diminish the satisfaction of quickly solving a seemingly difficult problem in a neat way. The lesson here is this, no matter what problems you come across don’t automatically look for a library to handle it. Do spend a few minutes investigating – it might be enough to solve it (especially if you’re using Ruby) and you’ll certainly learn something.

Image by Gerard Stolk presque 64

Skorks

Ruby & (Ampersand) Parameter Demystified

The Implicit Block

The Explicit Block

Passing Two Blocks To A Method

Using Another Method As A Block

Passing The Block Around

The Symbol To Proc Trick

Ruby – Why U No Have Nested Exceptions?

The Need For Nested Exceptions

The Problems With Nestegg

How Nested Exceptions Work

The Best Way To Pretty Print JSON On The Command-Line

The ‘Good’ Old Faithful

YAJL Tools

Ppjson

I Was Nominated To Speak At LoneStarRuby Conference 2013

A Closure Is Not Always A Closure In Ruby

What It Means To Evaluate A Block In A Different Context

How To Overcome The Limitations

So What Did They Do Before instance_exec Existed

The Many Faces Of Cloaker

What To Do About Method Calls

What Is A Startup (Or A Startup Idea)?

Why Developers Never Use State Machines

We Don’t Need One Until We Do

A State Machine Is A Fluffy Bunny (Not Particularly Threatening)

Even A Good Tool Is Not A Good Tool

When Developers Go To Great Length To Save Typing 4 Letters

A Quick Glance At Rack

Getting It Working

So What Did They Do Before `instance_exec` Existed