NewThe Learning module is now liveSee what's new
Advanced
Database Sharding and Partitioning
30 min readChapter 5Core
1Database sharding and partitioningChapter 1

Picture a group-chat app with conversations, members, and a messages table that grows by billions of rows. It all sits on one Postgres primary, which has run out of disk and write headroom. Read replicas do not help, because they copy every write rather than share it, and moving other tables to their own database only postpones the problem, since messages alone is the load. Sharding is the fix: you split one table's rows across several independent database nodes, called shards, and a function of a column called the shard key decides which shard holds each row. Each shard stores and serves only its slice, so writes and storage grow with the number of nodes.

Each shard is usually its own replicated group, which is the subject of Database Replication. "Partitioning" also has a second meaning. A database can split one table into partitions on the same machine, as Postgres declarative partitioning does, so queries skip irrelevant partitions and old data can be dropped a whole partition at a time. That helps pruning and retention, but it adds no write capacity, because every partition still lives on one node. Sharding puts the partitions on separate nodes, and that is the case this tutorial teaches.

Sharding comes up in almost every scale-out design round (chat, feeds, URL shorteners, ride-hailing) once the estimates say the data will not fit on one box, and it usually gets three to eight minutes of the deep dive. It is also a last resort.

Before you shard, say the cheaper steps first: a bigger machine, read replicas for read load, archiving or expiring cold rows, and moving other tables to their own database. Shard only when one table's writes or size still outgrow the largest primary you can run, because every cross-shard query, transaction, and rebalance afterwards is a cost you keep paying. In the chat app those steps are already spent. At mid-level, a sensible key with one reason and one named hotspot is enough. At senior level, expect to defend the key against the hottest query, handle one enormous conversation, cost a query that crosses shards, and explain how you would add capacity later without downtime.

Your progressLogin to save progress
Spot an issue or have a suggestion?
Availability and Failover Patterns  |

Discussion

Log in to join the discussion.