Skip to main content

Hibernate Shards for data partitioning across databases

Google has donated Hibernate Shards - a Java 5 add-on to Hibernate that allows application driven data partitioning with your custom or their pre-build configurations. One thing that caught my eye was that when they covered partitioned Native ID generation (aka Identity), they mentioned having Database A use a range 0-200000, Database B use range 200001 - 400000, etc. I know that MySQL cluster suggests using something like this:

Database A: Starting ID 1, Increment ID by 3 (number of databases)
Database B: Starting ID 2, Increment ID by 3
Database C: Starting ID 3, Increment ID by 3

This allows a hands-off approach, and easily lets you divide ID by 3 and use the remainder to map to a "Shard". Of course, if your data changes significantly, you may have to dump and reload to add another database. One way to avoid a dump and reload is to pick an increment that is larger than your actual database count, and just drop an extra database in an the available Starting ID. This can eat up your "keyspace" faster, but if you don't have a huge amount of data, you don't need partitioning that bad anyway.

If you have been reading my Blog, you know that I use NHibernate, but I am confident that the techniques that they are using are portable to .Net. I can see some value in using this for year-based partitioning, where archives are made available as read-only data, and the new database is created with the next available ID.
Labels:

Comments

Max said…
You make an excellent point Roy, and if you've read the section on virtual shards you saw that in order to use this feature we require you to determine the maximum number of physical shards your app will ever require up front. We really do think this is doable for applications that require sharding, since there's no penalty for overshooting. In the technique you describe there is, as you said, a penalty for overshooting (your keyspace gets eaten into), but if this is acceptable your technique sounds great.

Max (Hibernate Shards developer)
March 24, 2007 9:29 PM
Post a Comment

Popular posts from this blog

How does Rails scaffolding select HTML input tags?

Recently, a reader saw my fix for SQL Server booleans, and asked me a followup question: why does Rails display a yes/no selection instead of a checkbox? The short answer is look in {RUBY_HOME} /lib/ruby/gems/1.8 /gems/actionpack-1.10.2 /lib/action_view/helpers, but your path may vary depending on whether you are using gem, "edge rails", etc. Anyway, look in the file "active_record_helper.rb" for a method called "all_input_tags", and notice that it calls "default_input_block" if you don't supply an input_block. Now notice that "default_input_block" creates a label and calls "input(record, column.name)" which in turn calls "InstanceTag#to_tag" which finally looks at the datatype and maps boolean to a select tag. Perhaps a wiser Rails explorer can provide us with the rationale for this, but I guess we could add a MixIn for InstanceTag that redefines the to_tag() method, or just do a dirty and unmaintainable hack l...
2 comments
Read more

Castle ActiveRecord with DetachedCriteria

My current development environment is Visual Studio Express C# Edition (read that as free ), Castle ActiveRecord's latest svn trunk(usually within a few days), and NHibernate svn trunk. As of NHibernate version 1.2.0, there is a very cool new class out there ... DetachedCriteria. This class lets you set all of your Castle relational attributes like BelongsTo, HasMany, etc. as lazy fetch, and over-ride this for searches, reports, or anytime you know ahead of time that you will be touching the related classes by calling detachedCriteria.SetFetchMode(..., FetchEnum.Eager). As a good netizen, I have tried to contribute to NHibernate and Castle ActiveRecord even if only in the smallest of ways . Oh yeah, I tried mapping to a SQL VIEW, and it worked GREAT! I received a comment after my last post, indicating that there is a better way, and I am sure of it, but the view guaranteed that I only have one database request for my dataset. NHibernate was wanting to re-fetch my missing as...
1 comment
Read more

Destiny 2 - The Corrupted Nightfall

 == Overview == This Dreaming City Nightfall is already considered one of the most irritating strikes, in part because of bugs that sometimes prevent players from proceeding to the next encounter.  Notable bugged areas are the transition from the Elevator to the Knights Steps, and after the passage, the temple sometimes doesn't reveal a way to the shadow realm where you will take your leap. Important Note: After a "de-buff" from Particle Deconstruction, shoot the boss in the head!  It is quite effective, and can melt her with some coordination. Champions: Unstoppable and Overload Shields: Arc, Solar and Void (many shields!) Don't place your well in the open where boss can snipe the sword!  She will break your super. Pass the "Relic" (Orb) twice to maximize damage. == Loadouts == General: * Add Taken spec to every weapon Titans: * Bottom Tree Solar + Path of the Burning Steps or Phoenix Cradle * Sentinal Shield + Ursa, Aeon Gauntlets Warlocks: * Stasis + Blea...
Post a Comment
Read more