A few weeks ago, Facebook rolled out a new real-time commenting feature. But what the average user doesn’t realize is that it took some serious manpower to get the real-time commenting system right.
Facebook receives around 100 million pieces of content (links, posts, likes, etc.) and around 650,000 comments every minute. To make sure these pieces of content get to the correct person, Facebook had to test a few different approaches before it got the process right.
First off, Facebook used a poll-based approach, which meant that for every comment-able piece of content posted, the system would send a request every few minutes to see if a comment was made.
If there was indeed a comment, the user was alerted when the system crawled for comments. Unfortunately, this didn’t create the real-time experience Facebook hoped to achieve, or as one of its engineers called it in a blog post, a “truly serendipitous commenting experience.”
The poll-based approach meant a lag in communication and the potential to overload the servers if lots of comments were made in between the time the system last crawled for comments.
To fix the lag and prevent overload, Facebook moved towards a push-based system, meaning the servers push information about comments to viewers instantaneously.
The push system can keep track of “who’s looking at what” information and handle a higher rate of information change.
Facebook points out that storing one-on-one comments in a database is relatively easy, but that keeping up with 16 million new comments and pieces of information per second is not. Clearly, Facebook was built upon the idea that more people are reading the site rather than contributing.
Nowadays, the game has changed and more people are writing and commenting on Facebook rather than simply reading what’s going on.
In a blog post, Facebook said “Each write of a comment requires a read (to figure out the recipients of an update). We realized that we were building something that was fundamentally backwards from most of our other systems.”
In the past, each “write” (comment or other piece of information) would be applied to one database and asynchronously replicated to databases across all information.
Now, Facebook has been forced to adopt a “write locally, read globally” mindset. This means that the site uses distributed storage ties that only handle writes locally.
Once logged on the local level, the information is collected across the board to produce the final result: a site with information available to everyone across the board if they choose to access it (and of course based on a security preferences).
For example, if I write a comment in New York City, the system writes to a storage tier in the same data center, recording the fact that I commented on a piece of information and therefore pushing the information to everyone involved.
Eventually, the system reads the comments globally and saves Facebook from having to replicate a high volume of writes across data centers, saving expensive, long-distance bandwidth.
The result? A more real-time experience for everyone involved based on commenting and collaboration rather than shouting non-interactive information.
