Practical Metaprogramming with Ruby: Storing Preferences

// <![CDATA[
if (typeof window.Delicious == "undefined") window.Delicious = {}; Delicious.BLOGBADGE_GRAPH_SHOW = false; Delicious.BLOGBADGE_TAGS_SHOW = false;// ]]>

All code in this article is copyright Patrick McKenzie 2009 and released under the MIT license. Basically, you can feel free to use it for whatever, but don’t sue me.

The other day on Hacker News, commenting on a recent Yehuda Katz explanation of the nuts and bolts of metaprogramming, I mentioned that I though discussions of programming theory are improved by practical examples of how the techniques solve problems for customers.  After all, toy problems are great, but foos and bars don’t get me home from the day job quicker or convince my customers to pay me money.

My claim: Metaprogramming allows you to cut down on boilerplate code, making your programs shorter, easier to write, easier to read, and easier to test. Also, it reduces the impact of changes.

I’m going to demonstrate this using actual code from my PrintJob class, which encapsulates a single request to print out a set of bingo cards in Bingo Card Creator.  PrintJobs can have any number of properties associated with them, and every time I implement a new feature the list tends to grow.  For example, when I added in the ability to print in color, that required adding five properties. This pattern is widely applicable among many times of one-to-many relationships where you never really look at the many outside of the context of their relationship to the one — user preferences would be an obvious example.

There are a few ways you can do this in Rails. The most obvious is put each property as a separate column in your table. This means that

  1. you’d do a database migration (downtime! breakage! unnecessary work!) every week you add a new property.

If you’re getting into the associations swing of thing, you might consider creating a has_many relationship between PrintJob and PrintJobProperties, with each PrintJobProperty having a property_name and a property_value.  Swell.  Now you need to:

  1. Do twenty joins every time you inspect a single PrintJob.
  2. Add a bunch of unique constraints (in your DB or via Rails validations — I hope you have earned the favor of the concurrent modification gods) to prevent someone from assigning two properties of the same name to the same print job.
  3. Have very intensely ugly syntax for accessing the actual properties.  (Lets see, print_job.options.find_or_create_by_name(“foo”).value = “bar” ought to do it.)

Instead of either of these methods, I save the properties to an options hash, and then serialize that to JSON to save to and load from the database column.  Rails takes care of most of the details for me.

Enter The Metaprogramming Magic

However, this means I would have to write code like print_job[:options][:background_color], which is excessively verbose, and every time I referred to it I would need to possibly provide a default value in the event it was nil.  Too much work!

Instead, we’ll use this Ruby code:

#goes in /lib/current_method.rb

#Returns the method name ruby is currently executing.
#I use this just to make my code more readable to me.
module CurrentMethodName
  def this_method
    caller[0][/`([^']*)'/, 1]
  end
end

#goes in /app/models/print_job.rb
class PrintJob  5, :columns => 5, :column_headers => "BINGO", :free_space => "Free Space!", :cards_per_page => 1, :card_count => 1, :page_size => "LETTER", :title => nil, :title_size => 36, :font => "Times-Roman", :font_size => 24, :good_randomize => true, :watermark => false, :footer_text => "Omitted for length", :call_list => true, :background_color => COLOR_WHITE, :second_background_color => COLOR_GREY, :border_color => COLOR_BLACK,
:text_color => COLOR_BLACK, :color_pattern => "plain"
}

#set up accessors for options
  DEFAULT_OPTIONS.keys.each do |key|
    define_method key do
      unless options.nil?
        options[this_method.to_sym]
      else
        nil
      end
    end

    define_method "#{key}=".to_sym do |value|
      unless options.nil?
        options[this_method.to_s.sub("=","").to_sym] = value
      else
        options = {}
        options[this_method.to_s.sub("=","").to_sym] = value
      end
    end
  end

#Other stuff omitted. Sorry, I'm not OSSing my whole business today.
end

What This Code Does

OK, what does this do? Well, first I define a bunch of default options, which are later used (code not shown) to initialize PrintJobs right before they’re fed into the actual printing code. Each default option is used to create a getter/setter pair for PrintJob, so that instead of typing print_job[:options][:background_color] I can just type print_job.background_color. You’ll notice that it also note that both setters and getters pre-initialize the options array if I haven’t done it already. This saves me from accidentally forgetting to initialize it and then winding up calling nil[:some_option].

Why This Code Is Useful

Clearly this saves keystrokes for using getters/setters, but how does it actually save work? Well, because each of the properties are now methods on the ActiveRecord object (the PrintJob), all of the Rails magic which you think works on columns actually works on these options, too. This includes:

  • validations
  • form helpers
  • various pretty printing things

Since card_count is just another property on the PrintJob ActiveRecord object, Rails can validate it trivially. Try doing that within a hash — it isn’t fun. I sanity check that card_count (the number of cards printed for this print job) is an integer between 1 and 1,000, and additionally check that, for users who aren’t registered, it is between 1 and 15. (I’ve omitted the message which tells folks trying to print more to upgrade.)

  validates_numericality_of :card_count, :greater_than => 0, :less_than => 1000
  validates_numericality_of :card_count, :greater_than => 0, :less_than => 16,
:unless => Proc.new {|print_job| print_job.user && print_job.user.is_registered?}

Here’s an example of a portion of the form helper from the screen where most of these options are set:

#Just a part of the form.

   4, :title => "Total number of cards to print."%>

Ordinarily in the above code you’d expect card_count to correspond to a column in the database, and then the column would cause there to be card_count and card_count= methods on PrintJob, and this would be used to initialize the above text field. Well, Rails doesn’t really care how those methods came to be — they could be placed there by ActiveRecord magic, or attr_accessor, or defining by hand, or creative use of metaprogramming, as above. It takes about 7 lines to define a getter/setter pair in most languages. I have twenty properties listed up there. Instant savings: 140 lines of code.

Similarly, I’m saved from having to write a bunch of repetitive non-sense in the controller, too.

def some_controller_method
  @print_job = PrintJob.new
  @print_job.sensible_defaults! #initializes defaults for options not already set
  #update all parameters relating to the print job
  params[:print_job].each do |key, value|
  if @print_job.options.include? key.to_sym
    @print_job[:options][key.to_sym] = value
    end
  end
end

This walks over the param map for things being set to the PrintJob and, if they’re an option, sets them automatically. This saves about twenty lines of manual assignment. (Nota bene: PrintJob.new(param) will not work because the virtual columns are not real columns. In general, I hate mass assignment in Rails anyhow — sooner or later it will bite my hindquarters for security if I use it, so I assign only those columns which I know to be safe. Note that nothing in the options hash is sensitive — after all, they’re just user options.)

This controller is extraordinarily robust against change. When I added five extra options to the print jobs to accommodate my new features (font, color, and pattern selection), I didn’t change one single line of the associated controllers.

But wait, there’s more! You see, 200 lines of negacode (code that you don’t have to write) means 200 lines of code that you don’t have to read, test, maintain, or debug. I didn’t have to change the controller at all. I didn’t have to check to see if the new properties were automatically initialized to their starting values, since the code which performed the initialization was already known to work. I didn’t have to debug typos made in the accessors. It all just worked.

This is the power of metaprogramming. The less boilerplate code you have to write, understand, read, test, debug, and maintain, the more time you can spend creating new features for your customers, doing marketing, or doing other things of value for your business. The last three features I added caused five new properties to be added to my PrintJob model. I just diffed the pre- and post-commit code in SVN. Three features required required:

  • No change to the schema.
  • No change to the controller.
  • 35 lines to implement three features in the model. (Counting the white space.)

(The view required about 25 lines of new code, mostly inline Javascript, because a good UI for picking colors is tricky. I ended up liberally borrowing from this chap, who has an awesome color picker in Prototype that was amenable to quick adaptation to the needs of technically unsophisticated users to who wouldn’t think #F00F00 is a color.)

This is a much, much more effective use of my time than writing several hundred lines worth of boilerplate model code, then repeating much of it in XML configurations, just so that I can actually have access to the data needed to begin implementing the features. (How did you guess I program in Java?)

A Note To DBAs In The Audience

Yeah, I hear you — stuffing preferences in JSON is all well and good, but doesn’t this ruin many of the benefits of using SQL? For instance, isn’t it true that I can no longer query for PrintJobs which are colored red in any convenient manner? That is absolutely true. However, it isn’t required by my use cases, at all.

PrintJobs are only ever accessed per user and per id, and since both of those have their own columns, having all the various options be stored in an opaque blob doesn’t really hurt me. I regret not being able to do “select sum(card_count) from print_jobs;” some times, but since I don’t have to calculate “total number of cards printed since I opened” all that frequently, it is perfectly adequate to just load the entire freaking table into memory and then calculate it in Ruby. (It takes about 5 seconds to count: 217,264 cards printed to date. Thanks users!)

Note Regarding Security

Programmatically instantiating methods is extraordinarily dangerous if you don’t know what you’re doing. Note that I create the methods based on keys used in a constant hash specified in my code. You could theoretically do this from a hash created anywhere — for example, Ruby will happily let you create methods at runtime so you might decide, eh, PrintJob needs a method for everything in the params hash. DO NOT DO THIS. That would let anyone with access to your params (which is, well, anyone — just append ?foo=bar to the end of a URL and see what happens) create arbitrarily named methods on your model objects. That is just asking to be abused — setting is_admin? to 1 or adding can_launch_nuclear_weapons to role, for example.

#Fixing a WordPress bug.  Don't mind me.

No Responses to “Practical Metaprogramming with Ruby: Storing Preferences”

  1. toto November 17, 2009 at 11:09 am #

    Good read. I like to do eaxtly that for exactly that purpose.

    The accessor setup can be reduced a bit more. I find the one below a bit more readable:

    DEFAULT_OPTIONS.keys.each do |key|
    define_method key do
    return options[key.to_sym] unless options.nil?
    end

    define_method “#{key}=”.to_sym do |value|
    returning(options || {}) do |opt|
    options[key.to_sym] = value
    end
    end
    end

  2. toto November 17, 2009 at 11:10 am #

    WP killed my indetion. I put a version up as a gist: http://gist.github.com/237165

  3. Jim November 18, 2009 at 8:24 am #

    Instead of doing:

    @print_job.sensible_defaults!

    you could also do the initialization of the sensible_defaults in the after_initialize callback method, so you don’t have to do it in the controller ( which matters more if you call this a lot, and are afraid you’ll forget to do it)

  4. Arya Asemanfar November 18, 2009 at 10:09 am #

    We do something similar for extra attributes on our user model that we never need to use as query conditions. Here’s the gist: http://gist.github.com/238108

  5. Bromley November 18, 2009 at 10:57 am #

    Patrick it would have been good if you had posted this a couple of months ago… On a project I’ve been working on recently it took me about 3 major re-designs before I figured out that stuffing parameters into a blob column as JSON made the most sense for what I was trying to achieve.

    Boo to “best practices”, hooray for a healthy dose of pragmatism.

  6. David Brady November 18, 2009 at 12:29 pm #

    I call the general pattern for this “reduce relational database to key/value store”. It has pros and cons and can easily be taken too far. Probably 9 times out of 10 that I’ve seen it (or, sigh, USED it) it’s been taken too far. If you’ve considered your use cases and you’re okay, then you’re probably good to go, but do keep in mind that you need to keep an eye on this strategy in the future–if it grows, it’s probably growing out of hand.

    For example, if you find yourself needing to interact with one of those properties a lot in the code, you need to ask yourself if that property is starting to emerge as an important part of the design. The magic question is, “if I forget to put this key in the hash stored in the database, will other client code break?” (My rule of thumb: if it requires a change in the view, leave it in the hash; if it requires a change in a model, extract it to its own column.)

    Secondly, trading lines of negacode for a general solution is also a +/- proposition. I agree with your position and favor general code over specific (can you tell that I refuse to ever program in Java again?), but the counterarguments go like this: First, 200 lines of repetetive code really only as complex as the 10 lines of boilerplate code that are repeated; second, general code may give you those 20 properties, but it will also open you up to an infinite number of OTHER properties and you cannot guarantee that they will work, or worse, that they cannot be exploited maliciously.” This counterargument almost always comes from somebody who is used to programming in a more “explicitly safe” language like Java or C# or Python.

    Third, the maintainer of your code can see that whatever you put in that options hash will get extended onto your model. That’s great! But what the maintainer CAN’T see is what those options are supposed to be. She can’t look at the source and see “def color”. You might say this argument is weak in rails since you’d have to look at the schema anyway, but it’s a stronger argument than you think: the schema can be found in schema.rb. The options hash must be populated by something first before she can look at it. In other words, if she’s got an empty database she can’t save an exemplar because she doesn’t know what properties should be put into the options. She’s stranded.

    Okay, enough doom and gloom! I love this technique. It’s powerful, is all I’m saying, and care should be exercised. Powerful tools used ignorantly are merely dangerous. :-)

  7. Keith Perhac November 19, 2009 at 7:54 pm #

    Very nice idea.
    I’m going to have to try it for my user_settings section on JapaneseTesting.com.
    I was dreading having to save all of the data as either a) a separate table or b) cookies (which means I can’t look at them).

    I don’t know RoR at all, so the code was only fairly educational, but your explanation was really easy to understand. One thing I was curious about though was when you were reiterating over the Default_options array to set the user prefs. Does RoR not have an array_merge function?

  8. Patrick November 19, 2009 at 11:08 pm #

    Keith, Rails (actually, the Ruby standard library) can merge hashes, but there is a subtlety: sensible_defaults! iterates over the DEFAULT_OPTIONS hash rather than using Hash#merge because Hash#merge clobbers things in the destination hash, and the point of the exercise is to set those defaults only when the user hasn’t specified a value. (There are boring code-related reasons for not setting them prior to setting the user-defined stuff.)

  9. Sam Stokes November 21, 2009 at 12:12 pm #

    Thanks for this post! Great to see practical examples of techniques like metaprogramming. Your code also taught me something new about Ruby regexes: I blogged about it at http://bit.ly/810zXQ.

  10. pete December 26, 2009 at 12:28 pm #

    Patrick,

    “sensible_defaults! iterates over the DEFAULT_OPTIONS hash rather than using Hash#merge because Hash#merge clobbers things in the destination hash, and the point of the exercise is to set those defaults only when the user hasn’t specified a value.”

    Would ActiveSupport’s reverse_merge work better in this case?
    http://api.rubyonrails.org/classes/ActiveSupport/CoreExtensions/Hash/ReverseMerge.html

Trackbacks/Pingbacks

  1. Daily Review #28 | The Queue Blog - November 17, 2009

    [...] MetaProgramming Ruby – Patrick McKenzie gives a real life example of using Metaprogramming. [...]

  2. Ennuyer.net » Blog Archive » Rails Reading - November 24, 2009 - November 24, 2009

    [...] Practical Metaprogramming with Ruby: Storing Preferences: MicroISV on a Shoestring [...]

Loading...
Grow your software business:
(1~2 emails a week.)