ShardTheLove

ShardTheLove is a Rails and Merb plugin for horizontal scaling of databases It is built with ActiveRecord- A version somewhere in the git history is compatible with Rails 2.1 and Merb 0.9. The most up-to-date version has been ported to use connection handlers, a feature introduced in Rails and ActiveRecord 2.3. The codebase has been used in a production environment for 18 months on MySQL.

Sharding, federation, and partitioning all describe a similar horizontal scaling strategy (check out this discussion on exactly what means what). The recommended setup for ShardTheLove is several shards with a directory and system database in addition.

 -----------   --------   ---------   ---------
|           | |        | |         | |         |
| Directory | | System | | Shard 1 | | Shard 2 | etc..
|           | |        | |         | |         |
 -----------   --------   ---------   ---------

The biggest benefit of this architecture is the ability to easily add new shards. Each server type has a specific function:

• Directory - Speedy lookups telling you where to find sharded data.
• System - Non-sharded data. For instance if you shard by users, any data not user specific goes here.
• Shards - Sharded data. If you are sharding by users, user data goes here. All shards have the same schema.

Some example models using ShardTheLove to partition by users would be:

class User < ActiveRecord::Base
  # This model is on the directory server.
  #
  acts_as_directory

  has_many :pets
  has_many :cellphones
  has_many :subscriptions
  has_one :accounts_pricing_plan, :through => :subscriptions
end

class Pet < ActiveRecord::Base
  # This model is on shards.
  #
  acts_as_shard

  belongs_to :user
end

class Cellphone < ActiveRecord::Base
  # This model is on shards.
  #
  acts_as_shard

  belongs_to :user
end

class Subscription < ActiveRecord::Base
  # This model is on shards.
  #
  acts_as_shard

  belongs_to :user
  belongs_to :accounts_pricing_plan
end

class AccountPricingPlan < ActiveRecord::Base
  # This model is not user specific, so it's on the system.
  #

  has_many :subscriptions
  has_many :users, :through => :subscriptions
end

Keep in mind this is just one way to use ShardTheLove. Because the default database is the system DB, you can start implementing sharding on an exiting codebase without a complete refactor right away.

ShardTheLove does not provide explicit support for any specific strategy of partitions, but instead aims to provide the building blocks for any strategy. Some situations call for the dismissal of a directory and use hashed user id's for determining the shard. For more detail on other approaches:

• http://blog.maxindelicato.com/2008/12/scalability-strategies-primer-database-sharding.html
• http://www.startuplessonslearned.com/2009/01/sharding-for-startups.html

Configuring Databases

There isn't much magic here, configuration is pretty straight ahead. The labels of each database connection correspond to it's use.

• "#{Rails.env}" - System database
• "#{Rails.env}_directory" - Directory database
• "#{Rails.env}_myshard" - A shard database you can reference as "myshard"

Here is a sample configuration. In development mode, there are two shards labeled hewey and dewey. In test mode, there is only one shard, labeled shard.

development: &defaults
  reconnect: true
  adapter: mysql
  encoding: utf8
  database: coolapp_development
  username: root
  password:
  pool: 15 # Increase this, we will need them. 

development_directory:
  <<: *defaults
  database: coolapp_development_directory
  # The literal db name can be anything

development_hewey:
  <<: *defaults
  database: coolapp_development_hewey

development_dewey:
  <<: *defaults
  database: coolapp_development_dewey

test:
  <<: *defaults
  database: coolapp_test

test_directory:
  <<: *defaults
  database: coolapp_test_directory

test_shard:
  <<: *defaults
  database: coolapp_test_shard

Using ShardTheLove

Use of ShardTheLove itself is straight forward. Models declaring acts_as_shard will look on the database specified in ShardTheLove.with blocks. An example with the models and database configuration above:

# Usage of ShardTheLove

ShardTheLove.with('hewey') {
  Pet.all # Only pets on hewey
}

ShardTheLove.with('hewey') {
  ShardTheLove.current_shard # is 'hewey'

  ShardTheLove.with('dewey') {
    ShardTheLove.current_shard # is 'dewey'

    Pet.all # Only pets on hewey

    User.all # All users from the directory

    plan = AccountPricingPlan.first # A plan from the system db
    
    plan.subscriptions # Only subscriptions from dewey!
  }
}

# Walking shards is highly discouraged.  It means you make a connection
# to every shard.  At scale (lots of shards) this would not work smoothly.

ShardTheLove.shards # is ['hewey', 'dewey']

ShardTheLove.shards { |shard|
  shard == ShardTheLove.current_shard # is true
  
  Cellphone.all # gets called twice, once on each shard.
}

# Let's presume the model User has a method or column named
# "shard" that tells us what shard user data is on.

@user = User.find( '[email protected]' )

ShardTheLove.with(@user.shard) {
  # Now we can access that user's data
  @user.pets
  @user.cellphones
}

Setup on Rails

ShardTheLove is a Rails plugin. It provides only a basic framework for partitioning, and does not prescribe any specific pattern. Strategies like sharding by user, using a lightweight directory, using a DEFAULT_SHARD, and set_a_shard_by_current_user around_filter are recommended, but not required. Check out the examples directory for options to evaluate.

After installing the plugin, you will need three directories for migrations.

• db/migrate - System DB migrations.
• db/migrate_shards - Shards DB migrations.
• db/migrate_directory - Directory DB migrations.

rake db:all:migrate will migrate all databases. Run a rake -T to see all the options, or look in lib/tasks/databases.rake. Shard migrations run once on all databases, once in the scope of each shard.

Setup on Merb

Setup on Merb is the same as Rails, except the migration directories are moved:

• schema/migrate - System DB migrations.
• schema/migrate_shards - Shards DB migrations.
• schema/migrate_directory - Directory DB migrations.

Using RSpec with ShardTheLove

ShardTheLove has not been used extensively with fixtures, but it has been used heavily with Mocha and Rspec.

Add the following to your spec/spec_helper.rb in Merb or Rails:

Spec::Runner.configure do |config|

  # Your RSpec config here...
  #

  # Transaction setup and teardown for ShardTheLove
  #
  config.before(:all) do
    ShardTheLove.with(DEFAULT_SHARD)

    # The following is only needed in Rails- Merb loads all
    # the model files without this.
    # 
    ar_classes = []
    Dir.foreach("#{RAILS_ROOT}/app/models/") do |file|
      next if file =~ /^\./ || !(file =~ /\.rb$/)
      ar_classes << file.sub(".rb", "").camelize.constantize
    end
    ar_classes.each do |ar_class|
      next unless ar_class.ancestors.include?(ActiveRecord::Base)
      ar_class.connection_handler
    end
  end

  config.before(:each) do
    (
      [ ActiveRecord::Base.connection_handler,
        ShardTheLove.directory_handler ] +
      ShardTheLove.shard_handlers.collect {|name, handle| handle }
    ).each do |handler|
      next unless handler
      handler.connection_pools.each do |name, pool|
        pool.connection.begin_db_transaction
        pool.connection.increment_open_transactions
      end
    end
  end

  config.after(:each) do
    (
      [ ActiveRecord::Base.connection_handler,
        ShardTheLove.directory_handler ] +
      ShardTheLove.shard_handlers.collect {|name, handle| handle }
    ).each do |handler|
      next unless handler
      handler.connection_pools.each do |name, pool|
        if pool.connection.open_transactions != 0
          pool.connection.rollback_db_transaction
          pool.connection.decrement_open_transactions
        end
      end
    end
  end

end

Note that this assumes you are using the initializer to set DEFAULT_SHARD. Alternatively you can hardcode a shard matching you database configuration. A default shard is not required, but it simplifies most tests where you're concerned about the app logic and not the sharding logic. Of course you should test sharding logic in important places:

it "should shard with a users shard" do
  User.stubs(:find).returns(stub(:shard => 'the right one'))
  ShardTheLove.expects(:with).with('the right one')
  # now do something...
end

Authors

ShardTheLove was written at Give Real by Matthew Beale (mixonic) with input by Frank Mashraqi, Patrick Ledbetter, and Frank Macreery. Further development was undertaken by both Frank Macreery and Joseph Aghion. Of course, they wouldn't have done anything this cool without all the other great people working there.

ShardTheLove is available under the MIT License, the same license Rails is available under.