Can Bitcoin Scale? or, ‘How I Learned to Stop Worrying and Love The Hard Fork’

Changes to the Bitcoin protocol are inherently contentious. When you update normal software, you just tell the hardware what to do, and the hardware does it. In the case of Bitcoin, the hardware can argue with you. Bitcoin is implemented on a distributed cluster of unruly sociopaths (miners), in a constant Mexican standoff amongst themselves, who have to be sweet-talked into every change.

Because of the decentralized nature of Bitcoin, fundamental changes (so-called “hard forks”) to the operation of the protocol can only be made by majority consent, and if that consent isn’t close to unanimous, the result is a split userbase and permanent damage to the integrity of the currency. Unfortunately, these hard forks are also vital if the currency is to grow and change and scale.

Recently, we reported that Gavin Andresen, one of the most important Bitcoin core developers, has proposed a hard fork to increase the possible block size from 1 megabyte to at least 20 (plus a protocol to allow the number to grow over time). This has raised some hackles in the community, so today we’re going to sit down and look at, in more detail, what the fork accomplishes, what negative consequences it might have, and what future changes will be needed as Bitcoin scales.

The Status Quo

First off, let’s look at what happens if no change is made: Bitcoin presumably continues to grow in usage, and the number of transactions naturally increases. At some point, the 1MB transaction limit will be reached (when the transaction rate is about 5555 transactions per block). Suddenly, miners will have a problem: if they include all of the transactions they receive, their blocks will be rejected by the network for being too big. So they’ll probably start culling all of the lowest-fee transactions until they get down to the 1MB limit. That means that if your transactions aren’t in the top 5555 for transaction fees, they’ll be ignored, probably forever. Bitcoin will never be able to sustain transactions rates above 5555 per fifteen-minute block interval (just under 7 per second). If there’s demand above that level, fees will rise to squash it back down.

If Bitcoin grows beyond this point, it’ll be in the form of off-block chain transactions, with Bitcoin becoming merely the underlying infrastructure for a whole new generation of (probably regulated) electronic banks that use conventional centralized networks to move money between users. The fees for “normal” Bitcoin transfers will be much too high for conventional users. To me, this strikes as a nightmare scenario, and one worth avoiding.

The Proposed Change

The change is not without its downsides. The biggest concern in terms of the change itself is that as the network scales, if the growth of the network outpaces growth of hard drive storage per dollar, the total storage demand of the network could balloon. Already, downloading several gigabytes of network history is a significant barrier to launching a Bitcoin full node. If transaction volume continues to grow unbounded, that number could rise to terabytes, which would increase the incentive towards pool mining and raise the risk of 51% attacks.

There are some other security-related objections being passed around, but this seems to be primarily the work of a small community of fairly kooky people who call themselves “The Bitcoin Lordship.” Basically, their complaint is that increasing block sizes means that mining clients will have to expend more CPU power building the Merkle trees for individual blocks. This is, to the best of my knowledge, not a particularly serious objection: building the Merkle trees is extremely computationally inexpensive, and processors continue to get faster, quicker than other potential bottlenecks like network bandwidth. On the whole, I think it’s safe to write that concern off entirely.

The increased hard drive footprint is a serious concern, but one that’s likely to be addressed through the use of ‘pruned’ block chain storage protocols, which allow most nodes to store only the trailing few blocks of the block chain in a secure fashion, capping the maximum memory footprint of individual mining nodes and fighting the tendency towards centralization. This protocol change is already in the works.

Maybe the most tangible objection to the change is simply that hard forks are scary and costly to the network as a whole. However, there are good ways to minimize these costs: Bitcoin has had compatibility-breaking changes made before, and these changes have gone by largely unnoticed, because they were built into the clients months or years before they were actually deployed, ensuring that by the time the update actually goes into effect, most miners are running the latest (post-update) client, because the non-updated clients haven’t been available from any reputable download provider for months. Mining hardware goes obsolete quickly, and it’s often necessary to download a fresh copy of the mining client when replacing hardware, particularly since performance often improves from version to version. Hard forks, delayed in this manner, can be made relatively bloodless.

That is, of course, if the network consents to the change. If there’s a widespread miner revolt, the fork can fail. Or, worse, the network can get relatively evenly (and permanently) split between the alternate versions of the block chain ideologically – which would be extremely bad and do enormous damage to Bitcoin’s integrity and credibility in the public eye. The good news is that this fork looks relatively uncontroversial: most miners want their Bitcoins to be worth more, and that isn’t going to happen if Bitcoin remains at its current scale. This change needs to happen for Bitcoin to go global. The mining community largely isn’t crazy enough to think that Bitcoin can continue with no change forever and remain even slightly relevant in the world stage. If Bitcoin can never change, it’ll be replaced by an altcoin that can.

Future Changes

This block-size problem is just the first and most trivial change needed to allow Bitcoin to scale. We’ve touched on some other changes (block chain pruning), but, in reality, that’s just the start. The Bitcoin of five years from now will look very, very different from the Bitcoin of today. Bitcoin is still in a lively state of development, and a number of critical changes are needed if it’s going to operate at the scale of PayPal or Visa.

Imagine a possible future where both block size limits and block chain storage limits have been fixed for miners. We aren’t out of the woods yet! The next big scale issue that miners run into has to do with block races: when two miners mine blocks at about the same time, which happen periodically, a race ensues to see which one can propagate their block to the entire network first. Because smaller blocks can be spread faster, this creates an incentive for miners to throw out low-fee transactions – which is bad.

The solution for this, which is still being investigated, is called “invertible bloom filters.” The algorithm here is incredibly clever, but beyond the scope of this article. Here’s what they can do:

Let’s say two miners want to share a block that’s being propagated across the network. When miner one transmits it to miner two, most of the bandwidth is wasted, because miner two already has a copy of most of the data present in the block. They’ve already seen almost all of the transactions present in the block independently, on their own quest to mine a block.

Invertible bloom filters let you get most of that wasted bandwidth back. Instead of transmitting the whole block, invertible bloom filters let you take the expected difference between your data sets and transmit a small chunk of compressed data, equivalent to the size of that expected difference, plus a small overhead, and then decompress that chunk of data into whatever pieces of data you’re missing, regardless of which pieces they are.

Think of an invertible bloom filter as being like a painting you can look at from different angles and see different things. You can use the data you have to disambiguate its structure and uncover the data you don’t have. It’s lossy and has errors, but in practical applications, these errors can be made arbitrarily low. The main impact is that it makes block propagation time fixed, no matter how many transactions are included – the amount of data you need to transmit depends only on how similar your set of transactions is to your neighbors, not how many transactions you mined overall. As a nice side effect, this also creates an incentive to broadcast transactions to your neighbors as you receive them (so that future block broadcasts can happen faster), rather than hoarding transactions to yourself, which is what current incentives dictate. This effectively fixes a minor miner perverse incentive that’s been built into Bitcoin since the beginning.

Once that protocol is implemented, the last major problem you run into is the need for individual miners to keep up with all of the transactions made in the network. It’s unclear how serious a problem this is going to wind up being (it’s possible that available bandwidth will continue to grow ahead of Bitcoin usage). If it does become a problem, though, some sort of more exotic block chain structure, like side chains or tree chains will need to be implemented to make the structure of the network more manageable.

Once you’ve got all of this infrastructure in place, the road does, finally, seem clear for Bitcoin to truly scale. Here it is from Andresen’s roadmap blog post:

“There is a clear path to scaling up the network to handle several thousand transactions per second (‘Visa scale’). […] According to my rough back-of-the-envelope calculations, my above-average home Internet connection and above-average home computer could easily support 5,000 transactions per second today.  That works out to 400 million transactions per day. Pretty good; every person in the US could make one Bitcoin transaction per day and I’d still be able to keep up.

After 12 years of bandwidth growth that becomes 56 billion transactions per day on my home network connection — enough for every single person in the world to make five or six bitcoin transactions every single day. It is hard to imagine that not being enough; according the the Boston Federal Reserve, the average US consumer makes just over two payments per day.

So even if everybody in the world switched entirely from cash to Bitcoin in twenty years, broadcasting every transaction to every fully-validating node won’t be a problem.”

There’s still a lot of discussion left before all of these changes are made. Even the simple increase in block size is considered to be too aggressive by some of the core developers. Nothing in Bitcoin happens overnight, and it’ll be years before some of these changes are rolled out. Either way, it’s an exciting time to be interested in Bitcoin, and there’s good reason to believe that Bitcoin ultimately will be able to reach the scope and impact that we all want it to.