Rigel Group

They shoot Yaks, don't they?

Batmanjs Binding for Executing Script Blocks in Views.

You may have noticed that if you add a <script> block to your Batmanjs views, it never gets executed. This binding will look for any <script> nodes that are immediate children and executes them.

Note that you probably should NOT be doing this, as it is certainly going against the grain of Batman, but for quickly trying out standard jquery code without any modifications, it’s nice.

Execscripts.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# Execscripts.coffee
# John Lynch / Rigel Group, LLC
# Open source under the MIT License.
#
# Create a new binding that marks a specific node as having children scripts blocks, and then executes those script
# blocks as soon as the node has been added to the DOM by Batman. (This is necessary as many jQuery-type plugins
# wont work if run on an isolated node before it has been added to the DOM.)
#
# (Thanks to SO for some of this code. http://stackoverflow.com/questions/220188/how-can-i-determine-if-a-dynamically-created-dom-element-has-been-added-to-the-d)
#
# <div data-execscripts="true">
#   <h1>Title</h1>
#   <script type="text/javascript">
#     alert("Hi Mom!");
#   </script
# </div>
#
#

Batman.DOM.readers.execscripts = (node, key, context, renderer) ->
  new Batman.DOM.ExecscriptBinding(node, key, context, renderer)
  true

class Batman.DOM.ExecscriptBinding extends Batman.DOM.AbstractBinding
  bindImmediately: false
  constructor: ->
    super
    # Only run the script blocks once this node has been added to the DOM by Batman
    @_executeOnLoad(@node, @_exec_body_scripts) if @value

  _exec_body_scripts: (elem) =>
    scripts = []

    for child in elem.childNodes
      scripts.push(child) if @_isScriptNode(child)

    for script in scripts
      script.parentNode?.removeChild(script)
      @_evalScript(script)

  _isScriptNode: (elem) =>
    elem.nodeName && elem.nodeName.toLowerCase() == "script" && (!elem.type || elem.type.toLowerCase() == "text/javascript")

  _evalScript: (elem) =>
    data = (elem.text || elem.textContent || elem.innerHTML || "" )
    head = document.getElementsByTagName("head")[0] || document.documentElement
    script = document.createElement("script")
    script.type = "text/javascript"

    try
      script.appendChild(document.createTextNode(data))
    catch e
      script.text = data

    head.insertBefore(script, head.firstChild)
    head.removeChild(script)

  _isInDOMTree: (node) =>
   !! @_findUltimateAncestor(node).body

  _findUltimateAncestor: (node) =>
   ancestor = node
   (ancestor = ancestor.parentNode) while ancestor.parentNode
   ancestor

  _executeOnLoad: (node, func) =>
    if @_isInDOMTree(node) then func(node) else setTimeout( (() => @_executeOnLoad(node, func)), 100)

Connecting to POW via SSL

If you are running your Rails apps under Pow (and really, why wouldn’t you?), and you need to connect via SSL, you may have googled around and found instructions on how to set up Nginx as an SSL proxy. While this is probably your best bet for heavy-duty usage, if you want something simpler and lighter-weight for occasional testing, check out stunnel. It will proxy SSL connections just like Nginx.

The magic incantations (for OSX at least) go something like this:

1
 brew install stunnel

Now, the default .pem file that stunnel created for me didn’t work, so let’s create our own:

1
 openssl req -new -x509 -days 9999 -nodes -out /usr/local/etc/stunnel/stunnel.pem -keyout /usr/local/etc/stunnel/stunnel.pem

Verify the .pem file with this command:

1
 openssl x509 -subject -dates -fingerprint -in /usr/local/etc/stunnel/stunnel.pem

If everything checks out, lets now create a config file for stunnel that will proxy https connections from port 443 on localhost to http connections to port 80 on localhost.

1
2
3
4
5
6
7
8
9
10
11
 tee /usr/local/etc/stunnel/stunnel.cnf <<ENDOFFILE
pid        = /tmp/stunnel.pid
setuid     = nobody
setgid     = nobody
foreground = yes
client     = no
[https]
cert = /usr/local/etc/stunnel/stunnel.pem
accept = 443
connect =  80
ENDOFFILE

Now all that is left to do is fire it up:

1
 sudo stunnel

Go here and you should see your pow server.

Simulate Poor Network Conditions on OSX

If you have OSX (Lion) and XCode installed, spotlight for Network Link Conditioner.prefPane, and install it. You will get this nifty preferences pane which lets you simulate various network speeds and latencies, which comes in really handy for testing your web app in less-than-optimal conditions.

Batman’s Secret Cache

The real Batman has caches of weapons stashed across Gotham City, but the Batman.js javascript framework could use a cache of it’s own, a Views cache. I love the fact that Batman views are all stored in separate files (a la Rails) when doing development, it makes organizing a large project very easy. But when it comes time to deploy to a production system, stock Batmanjs will happily request each view as it is first requested via AJAX, but all those individual views should really be bundled up and served in one file, for obvious performance reasons.

To accomplish this, Ryan Funduk and I recently came up with this approach. First we will leverage the Rails asset pipeline to create a json file that contains all of our app’s views.

1
 /app/assets/javascripts/all_views.json.erb
1
2
3
4
5
6
7
8
9
10
11
<%=
  prefix = "#{Rails.root}/app/assets/javascripts/views"
  paths = Dir.glob("#{prefix}/**/*").select{|f| File.file?(f) && (f =~ /\.(html|erb)$/i) }
  paths.inject({}) do |all_views, f|
    viewname = f.sub( /^#{prefix}/, '' ).sub( /\..*$/i, '' )
    view = File.read(f)
    view = ERB.new(view).result if f =~ /\.erb$/i
    all_views[viewname] = view
    all_views
  end.to_json
%>

Now, if you go to http://localhost:3000/assets/all_views.json you should see all your views in one large json object. Note that your Batman views will be processed through ERB, so if you need the server to do some processing, go right ahead. (Just remember that the ERB code will be run only once, when the views are compiled, not each time they are requested.)

Next, tell the Rails asset pipeline to precompile this file, like so:

1
 /config/environments/production.rb
1
 config.assets.precompile += %w(all_views.json)

Next, we write a Batman helper that will request the all_views.json file, and create Batman views for each of the individual views. That looks like this:

1
 /app/assets/javascripts/helpers/views_preloader.js.coffee.erb
1
2
3
4
5
6
7
8
MyApp.preloadViews = () ->
  new Batman.Request
    url: '<%= asset_path("all_views.json") %>'
    type: 'json'
    error: (response)  -> throw new Error("Could not load views")
    success: (all_views) =>
      for view of all_views
        Batman.View.store.set(view, all_views[view])

Note that we run this file through ERB first, so that we get the proper path (including the digest, if applicable) to the all_views file. The last line pre-populates the view store with each of the views. (You will need to be on the master Batmanjs branch, as the View store is a new addition since the 0.8 release).

The last thing we need to do is kick off the preloader, and a good place to do that is like so:

/app/assets/javascripts/application.js.coffee
1
2
3
4
window.MyApp = class MyApp extends Batman.App

  @on 'run', ->
    MyApp.preloadViews()

So, putting it all together, when your Rails app is deployed and the assets are precompiled, the asset pipeline will write out an all_views-1234.json file to the public/assets directory. On the client side of things, when your Batman app starts, it calls the preloader, which loads the all_views-1234.json file and creates Batman views for them all in one fell swoop. The really cool thing about this setup is that the Rails asset pipeline takes care of caching the views for you. Winning!

UPDATE: Andrew Bennett came up with an even nicer way to grab the view source using the asset pipeline, which will let you use HAML or ERB and also any Rails view helpers:

1
2
3
4
5
6
7
8
9
10
11
12
<%=
  prefix = File.expand_path('../views', __FILE__)
  paths = Dir.glob("#{prefix}/**/*").select{|f| File.file?(f) && (f =~ /\.(html|haml|erb)$/i) }
  paths.inject({}) do |all_views, f|
    viewname = f.sub( /^#{prefix}/, '' ).sub( /\..*$/i, '' )
    all_views[viewname] = environment["views#{viewname}.html"].to_s
    ## the following appear to be identical ##
    # all_views[viewname] = asset_environment["views#{viewname}.html"].to_s
    # all_views[viewname] = Rails.application.assets["views#{viewname}.html"].to_s
    all_views
  end.to_json
%>

Torquebox Really Does Go to 11!

After working with Torquebox for the last year, I couldn’t imagine writing a Rails app without it. I came for the deployment story, but I stayed because of all the fantastic tools you get for free. From diagnostic tools on the JVM, to message queueing, background tasks, the awesome Infinispan cache, etc. Do yourself a favor and check it out.

But, when doing development, I like to make sure the app will run outside of Torquebox as well, with a simple ‘rails s’, so here are a few tricks to make that happen.

When setting Torquebox-specific features, like for example setting the Rails cache store to Infinispan, do this:

1
 config.cache_store = :torque_box_store if defined?(TorqueBox)

That way, when you run outside of Torquebox, Rails will use the default file cache and all will be well.

Another great Torquebox feature is the ability to mark any method as backgroundable, and the JVM will transparently spin off that method in a new thread. You accomplish that with this code:

1
2
3
4
5
6
7
8
class Battlestar
  include Backgroundable
  always_background :fire

  def fire
    # fire something
  end
end

This of course will cause problems when running without Torquebox, so if you add this shim into your initializers, the method will just run in the foreground like normal.

1
2
3
4
5
6
7
8
9
unless defined?(Torquebox)
  module Backgroundable
    extend ActiveSupport::Concern
    module ClassMethods
      def always_background(method)
      end
    end
  end
end

Another technique I use, especially for accessing the Torquebox message queues, is make sure you use your own class as a central point of control, like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
class QueueManager
  include TorqueBox::Injectors if defined?(TorqueBox)

  def publish_weapon_event(event)
    @weapon_queue ||= inject("/queues/myapp/weapon_events") if defined?(TorqueBox)

    if @weapon_queue
      @weapon_queue.publish(event)
    else
      processor = WeaponEventProcessor.new
      processor.on_message(event)
    end
  end

So, when running inside Torquebox, the message gets sent to the queue, which would then at some point in the future execute the event processor, but when running under Webrick, it calls the event processor directly.

Render Rails3 Views Outside of Your Controllers

If you ever had the need to break out of the MVC box and render a view template outside of your controller, the old-style Rails2 version went something like this:

1
body = ActionView::Base.new(Rails::Configuration.new.view_path).render(:file => "/orders/receipt.html.erb",:layout => false,:locals=>{:order=>order})

In Rails3, the same thing can be accomplished with this incantation:

1
2
3
4
av = ActionView::Base.new()
av.view_paths = ActionController::Base.view_paths
av.extend ApplicationHelper #or any other helpers your template may need
body = av.render(:template => "orders/receipt.html.erb",:locals => {:order => order})

Transparently Cache Network Calls in Titanium

Typically, when you are building a mobile app with Titanium, you are grabbing data from the web via RSS/Atom/etc or various APIs, and displaying them to the user. And of course, when developing for mobile clients, you can never take the network for granted. So a common pattern we use is to always cache the results of network requests locally, so in the event of a network glitch or even “airplane mode”, we still have some (stale) data to show to the user. We have packaged this functionality up into a CoffeeScript snippet here.

Basically, it is a wrapper around the standard Titanium.Network.HTTPClient API, except that we cache the results of each request in a local SQLite database table, and use that for subsequent requests until the cached record expires. The cache key is a hash of the full URL (plus any data parameters for POSTs).

For an example (written in CoffeeScript), we can create a new instance of the class to connect to the CNN RSS news feeds:

1
2
3
4
5
6
7
8
9
10
  Ti.include("HTTPClientWithCache.js")
  cnn = new root.HTTPClientWithCache({
    baseURL: "http://rss.cnn.com/rss",
    retryCount: 2,
    cacheSeconds: 60,
    onload: (response) ->
      Ti.API.debug("Response Data: #{response.responseText}")
      Ti.API.debug("Is this cached data?: #{response.cached}")
      Ti.API.debug("Cached at: #{response.cached_at}")
  })

Each time we want to make the request, we say:

1
  cnn.get({url: "/cnn_topstories.rss"})

The response will either come from the cache, if it has not expired, or from cnn.com. Because we set the retryCount property to 2, if the first attempt to fetch the data over the network fails (meaning any HTTP status code > 400) it will automatically try again a second time. If that fails, then it will respond with the most recent version available in the cache, or null if nothing exists in the cache. The response object also has a cached property of true for anything served out of the cache, and a cached_at property of the timestamp the record was cached.

You can use the same object to make calls to other URLs as well, like so:

1
  cnn.get({url: "/cnn_us.rss"})

If you need to pass parameters (like for pagination) you can say:

1
  cnn.get({url: "/cnn_us.rss?page=1"})

To POST data to a URL, do this:

1
  cnn.post({url: "/story/19912/edit", data: {param1: "value1", param2: "value2"}})

To manually prune the cache, you can call the prune_cache method, and anything older than seconds will be deleted from the cache. For example, to remove anything older than 1 day (86,400 seconds) you would say this:

1
  cnn.prune_cache(86400)

To completely clear the cache of every single entry, you can do this:

1
  cnn.prune_cache(0)

The code is on GitHub as a gist here. For pointers on how to build Titanium apps using the incredible CoffeeScript instead of Javascript, see our previous blog post here.

Building iPhone Apps Using Titanium and CoffeeScript

We have been using Titanium to build mobile apps for some time now. It is truly amazing how much more productive we are as opposed to when we use POOC (Plain Old Obj-C). Granted, Titanium is not a great fit for some applications, but for your standard network and data-heavy line of business applications, it rocks.

But, Titanium does not give you a lot of guidance on how you should write your app. They provide all the pieces you need, but folks are still figuring out the best way to structure things. So, we thought we would show you our particular technique for putting together a Titanium mobile app.

The first thing we did, was ditch Javascript. To be honest, something about it just makes our eyes bleed. Fortunately, jashkenas has come to our rescue with a Ruby-like language that compiles down to Javascript, called CoffeeScript. CoffeeScript makes Javascript a joy to use, and because it “compiles” to plain ole’ Javascript, you can use it anywhere, including Titanium. (Basically, you run the command coffee -w -c *.coffee in whatever directory your files are in, and it continually watches for any changes and immediately compiles the *.coffee into a *.js file.) So to be clear, you are only ever referencing the .js files in your Titanium project, Titanium doesn’t know anything about CoffeeScript or how to parse/use it.

One interesting thing CoffeeScript does, is compile each CoffeeScript file into its own namespace inside an anonymous Javascript function. This is great, because it isolates each file and prevents you from polluting the global namespace. For example, this CoffeeScript code:

1
2
show_flag = true
alert "Hello World!" if show_flag

gets translated into this Javascript code:

1
2
3
4
5
6
7
(function() {
  var show_flag;
  show_flag = true;
  if (show_flag) {
    alert("Hello World!");
  }
})();

Another thing CoffeeScript enables, is an easier-on-the-eyes way to creates “classes”, for example (from the CoffeeScript docs):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
class Animal
  constructor: (@name) ->

  move: (meters) ->
    alert @name + " moved " + meters + "m."

class Snake extends Animal
  move: ->
    alert "Slithering..."
    super 5

class Horse extends Animal
  move: ->
    alert "Galloping..."
    super 45

sam = new Snake "Sammy the Python"
tom = new Horse "Tommy the Palomino"

sam.move()
tom.move()

(If you really want to see what that gets translated to, put on some shades and click here)

So, when using CoffeeScript in Titanium apps, we can use these two features to help us structure things in a way that keeps each window isolated in its own namespace, yet allows us easy access to global objects as well. To start, we set up our app.js file, which Titanium runs on app startup. We keep this file as a plain Javascript file, and define a variable called root, to which we will attach anything that we want access to globally. We also include each window’s CoffeeScript file (well, technically we include the Javascript file that the CoffeeScript file was translated into). Finally, we include a main file, which sets up the main tab group and kicks everything off.

app.js
1
2
3
var root = {};
Ti.include('js/generic_window.js');
Ti.include('js/main.js');

In the generic_window.coffee file, we create a class definition for a generic window, and actually create a Titanium window in the constructor. We then attach the class definition to the root object so we can get at it from anywhere.

generic_window.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class GenericWindow
  constructor: (theTitle, theText) ->
    @win = Ti.UI.createWindow({title:theTitle,backgroundColor:'#fff'})
    label = Titanium.UI.createLabel({
      color: '#999',
      text: theText,
      font: {
        fontSize: 20,
        fontFamily: 'Helvetica Neue'
      },
      textAlign: 'center',
      width: 'auto'
    })
    @win.add(label);

root.GenericWindow = GenericWindow

Now the main file sets up the tabgroup, and actually instantiates some windows from the above generic window class…

main.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
tabGroup = Titanium.UI.createTabGroup({ barColor:'#336699'})
Titanium.UI.setBackgroundColor('#000')

# Attach the window instance to root so we can get to it from anywhere
root.Win1 = new root.GenericWindow('Win1','I am Window 1')

tab1 = Titanium.UI.createTab({
  icon:'KS_nav_views.png',
  title:'Win1',
  window: root.Win1.win
})
tabGroup.addTab(tab1)

# Attach the window instance to root so we can get to it from anywhere
root.Win2 = new root.GenericWindow('Win2','I am Window 2')

tab2 = Titanium.UI.createTab({
  icon:'KS_nav_views.png',
  title:'Win2',
  window: root.Win2.win
})
tabGroup.addTab(tab2)

tabGroup.open({transition:Titanium.UI.iPhone.AnimationStyle.FLIP_FROM_LEFT})

Now, let’s add a custom property to our generic window class, so add this line to the constructor of generic_window.coffee:

1
@custom1 = "Default Value"

This creates a property and gives it a default value. To access it, you could say root.Win1.custom1. Now let’s add an event handler that displays an alert box containing the value of @custom1:

generic_window.coffee
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class GenericWindow
  constructor: (theTitle, theText) ->
    # Save off the context of this object to a local var 'self'
    self = this
    @custom1 = "Default Value"
    @win = Ti.UI.createWindow({title:theTitle,backgroundColor:'#fff'})
    label = Titanium.UI.createLabel({
      color: '#999',
      text: theText,
      font: {
        fontSize: 20,
        fontFamily: 'Helvetica Neue'
      },
      textAlign: 'center',
      width: 'auto'
    })
    @win.add(label);

    @win.addEventListener('click', () ->
      alert(self.custom1)
    )

root.GenericWindow = GenericWindow

If you would like to see how it all hangs together, clone the GitHub repo here and try it out. The bottom line is that before CoffeeScript, our Titanium apps tended to resemble spaghetti, but now we are able to encapsulate functionality and just generally make things look much cleaner. Sure, you don’t need CoffeeScript to do this, but it sure makes it a lot more fun!

Tee With Sinatra

OK, so I wanted to take all of the JSON data that we were stuffing into our Riak cluster, and send a copy of it to our ElasticSearch cluster as well, so that we could, you know, actually find the data later. We could have done this by modifying one of the Riak client libraries, but then any data that got uploaded through a different client would be missed. So as an experiment, we turned to our new favorite tool, Sinatra, and hacked up a Rack proxy app, that will intercept the incoming HTTP requests, send them on to Riak and also send a copy to the ElasticSearch cluster. We used Typhoeus as the HTTP client to do this, so that we could concurrently execute the 2 requests in the interests of speed.

Here is the proof-of-concept:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
require 'rubygems'
require 'sinatra'
require 'typhoeus'

OPTIONS = {}
OPTIONS[:riak_host] = "localhost"
OPTIONS[:riak_port] = "8098"
OPTIONS[:es_host] = "localhost"
OPTIONS[:es_port] = "9200"
OPTIONS[:riak_timeout] = 5000 # milliseconds
OPTIONS[:es_timeout] = 5000 # milliseconds

class Rack::Proxy

  def initialize(app)
    @app = app
    @hydra = Typhoeus::Hydra.new
  end

  def call(env)
    req = Rack::Request.new(env)
    # We need to use it twice, so read in the stream. This is an obvious problem with large bodies, so beware.
    req_body = req.body.read if req.body

    riak_url = "http://#{OPTIONS[:riak_host]}:#{OPTIONS[:riak_port]}#{req.fullpath}"

    opts = {:timeout => OPTIONS[:riak_timeout]}
    opts.merge!(:method => req.request_method.downcase.to_sym)
    opts.merge!(:headers => {"Content-type" => req.content_type}) if req.content_type
    opts.merge!(:body => req_body) if req_body && req_body.length > 0

    riak_req = Typhoeus::Request.new(riak_url, opts)
    riak_response = {}
    riak_req.on_complete do |response|
      riak_response[:code] = response.code
      riak_response[:headers] = response.headers_hash
      riak_response[:body] = response.body
    end
    @hydra.queue riak_req

    # If we are putting or posting JSON, send a copy to the ElasticSearch index named "riak"
    if (req.put? || req.post?) && req.content_type == "application/json"
      req.path =~ %r{^/riak/([^/]+)/([^/]+)}
      bucket, key = $1, $2
      es_url = "http://#{OPTIONS[:es_host]}:#{OPTIONS[:es_port]}/riak/#{bucket}/#{key}"
      opts = {:timeout => OPTIONS[:es_timeout]}
      opts.merge!(:method => req.request_method.downcase.to_sym)
      opts.merge!(:body => req_body) if req_body  && req_body.length > 0
      es_req = Typhoeus::Request.new(es_url, opts)
      es_response = {}
      es_req.on_complete do |response|
        es_response[:code] = response.code
        es_response[:headers] = response.headers_hash
        es_response[:body] = response.body
      end
      @hydra.queue es_req
    end

    # Concurrently executes both HTTP requests, blocks until they both finish
    @hydra.run

    #If we wrote to ES add a custom header
    riak_response[:headers].merge!("X-ElasticSearch-ResCode" => es_response[:code].to_s) if es_response && es_response[:code]

    #Typhoeus can add nil headers, lets get rid of them
    riak_response[:headers].delete_if {|k,v| v == nil}

    # Return original Riak response to client
    [riak_response[:code], riak_response[:headers], riak_response[:body]]
  end
end

use Rack::Proxy

(Gist here)

Execute the script, and it will listen on port 4567, so point your Riak client of choice there and start PUTing data, which will be seamlessly replicated into the ElasticSearch cluster. If we were really going to use this in anger, there is a lot of work yet to be done, but as a skeleton of how to use Sinatra (Rack, really) to quickly whip up custom proxys, and tee HTTP requests, I thought it might be useful.

Light a FUSE Under Your Riak Cluster

As an experiment, we hacked together a FUSE driver in Ruby that lets you mount your Riak cluster as a file system, and browse around. Not sure how really useful it is, but was fun to do nonetheless.

If your Riak keys look like /foo/bar/logo.png, you will be able to ls and cd around the (simulated) directory structure, and cat files (keys).

Things get even more interesting if you hook this up with jsawk, then you can do things like this if your keys contain JSON values:

1
 cat /mnt/riak/users/* | jsawk 'if (this.city != "Paris") return null'

The current implementation does a list keys function when you access a bucket, which of course is slow. We put all the keys into a tree structure and cache that for performance. Another fun project would be to rewrite it using Riak’s Links to simulate the directory structure, and then run interactive map/reduce jobs to navigate the hierarchy. That will have to wait for another time…

Anyway, check it out yourself at Github, just keep in mind it is a toy and should not be pointed at a cluster you care about.