Facebook Architecture From 2005
Early architecture of Facebook
The architecture evolves over time and the idea should always be we’re able to iterate over it.
It’s nice when you get stuff right the first time but we don’t need to
As long as you get the architecture as right as possible; the implementation stuff isn’t going to be big of a deal
This article captures the very early architecture of Facebook, how it started and challenges faced with scaling.
Initial Architecture
Facebook was launched in February 2004 with focus on connecting people for one school
Tech Stack
Programming Language: PHP
Database: MySQL
Apache Web server
Figure 1 Early Facebook Architecture
The simple architecture served well for single school, but soon there were challenges with more interest and increasing scale.
Scaling Challenges
Let’s discuss few of the challenges faced due to architecture limitations:
Privacy: Data and feed separation between the schools
Traffic Distribution:
How the data should be distributed for different schools
Some schools have heavier usage as compared to others.
Figure out friends connections with higher degree (friend-of-friend or friend-of-friend-of-friend)
Let’s discuss on the improvements in the architecture to handle the bottlenecks.
Improving Architecture
The key aspect is hiring more and more smart people and give them independence to work on the improvements.
Onboarding More Schools
The first thing that comes to mind for handling more traffic is adding more machines.
We mentioned web server and MySQL coupled on same machine, so the idea is to decouple the storage from application server and introduce load balancing between the machines.
This also removed single point of failure. Consistent database layer ensures traffic can be served from any operational machine.
Handling Friend-Of-Friend
If you wish to figure out friend of a person, it’s a direct query to database. For friend-of-friend:
Retrieve friend list of a person
Iterate over this list and get friend list of all the items
This becomes n*n operation.
For friend-of-friend-of-friend, it becomes n*n*n operation; so it’s compute-intensive. The current architecture uses dedicated graph database, but back then:
Split the database and create one instance of mysql database for each school in the network
How does this help? Now you’ve to compute n*n*n for 10k (approximate number of people signed up from one school) instead of n*n*n over 6 million (all people across the schools)
The above is in addition to decoupling application servers and database compute. This helped a lot but further the traffic was reaching 100 Million pages/day, a query approximately takes 2-4 milliseconds. The problem lies in each page repeating on the same queries.
Improving Database Performance
What comes to mind to solves the same query problem? Cache the results to avoid hitting the database. Facebook introduced Memcached as caching layer; this brought down query performance to 0.3-0.5 milliseconds.
There were couple of issues though with the distributed architecture of memcached instances; If a memcached instances go down:
There is no redundancy on information.
Cache miss occurs; and all the traffic routes to database.
Facebook introduced photo sharing feature, let’s discuss on how it’s implemented.
Photo Storage
From early stage, Facebook chose to smaller machines instead of one big machine. The disk is relatively slow but stable, to overcome the slowness:
High RAM Caching machines to hold most of the thumbnails and frequently accessed images.
The space consumed by photos is on higher side, some issues uncovered:
If you wish to upload a photo album with 30 photos (3 MB each), then 90MB is total size; Networking issues with slow internet.
Router used was built to support 1GB at a time.
Solution:
Built Java applet and ActiveX control to couple the choosing of photos
Compression at client side.
It helps:
Upload photos quickly
Resource optimisation - less server CPU utilisation
Even with caching machines, transferring images to users takes time. So the solution is to move caching near to users, aka, Content Delivery Networks (CDNs).
Conclusion
Follow Iterative Design Process; Remember you don’t have built it right (or perfect) the first time. Happy Building!