Rigel Group

They shoot Yaks, don't they?

Hydra Attacks Riak Cluster

The great thing about querying a Riak cluster is that it’s all just HTTP, so the huge universe of tools that are available to work with HTTP resources are at your disposal. For example, lets say we want to quickly request 20 keys from a Riak cluster. We could use curl or curb and request them one at a time, or we could use the multi-headed hydra that is Typhoeus to get them concurrently! Let’s see this monster in action with a little Ruby code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
require 'rubygems'
require 'typhoeus'
require 'yajl'

HOST = "riak.cluster.com"
PORT = "8098"

# bucket is the name of a Riak bucket
# keys is an array of keys to get
# returns array of response hashes with the keys, :key :code :headers :body
def multi_get(bucket, keys)
  hydra = Typhoeus::Hydra.new
  @responses = []
  keys.each do |key|
    url = "http://#{HOST}:#{PORT}/riak/#{bucket}/#{key}"
    request = Typhoeus::Request.new(url)
    # When the request completes, this block gets run
    request.on_complete do |response|
      result = {}
      result[:key] = key
      result[:code] = response.code
      result[:headers] = response.headers
      result[:body] = response.body
      @responses << result
    end
    # queue up the request to run later
    hydra.queue request
  end
  # This is a blocking call that executes all queued requests concurrently,
  # and returns when all requests have completed
  hydra.run
  @responses
end

# OK, lets try this baby out now,
# first, set up the list of keys we want to get
keys = YAML::load(&lt;&lt;EOT)
---
- 1244idS1NaricUO2RtXJrjcfzr8
- 12ktMhh8KZOCYzMIRTLlqf5JeGA
- 129Izkjd6Fh2i1zqxCE2acT6iju
- 129AomedyZIa3gjudjCxpke5kU9
- 12BEfyiKWPqwZqiqcKGmizVN34i
- 12EZagKQHnakkIChE3ruLUu4TrA
- 12RBVtZV0EwQyTixXFLwHHqwLuK
EOT

# Now let the Hydra out!
rs = multi_get("my_bucket", keys)

# Or, if you are storing JSON objects, get the results as JSON using the speedy Yajl gem...
rs = multi_get("my_bucket", keys).map{|r| Yajl::Parser.parse(r[:body])}

As you can see, it’s pretty darn easy to leverage all the great work by the Typhoeus folks to speed up your queries.