The primary motivation for writing a technical book should not be money. If it is, you should probably reconsider. Technical books have a highly niche audience. With the rise of AI tools, fewer people rely on books for technical answers.

Financial gain was never the motivation for me, so what inspired me to write “System Design on AWS”?

• Personal Growth: Taking a complex idea and translating it for beginners forces you to deepen your own understanding.

• The Communication Challenge: Having deep technical knowledge is common, but delivering it effectively is rare. Writing the book was a test of communication and documentation skills. I credit much of this to the writing culture at Amazon.

• Building Trust: Being an author with a respected publisher like O’Reilly acts as a powerful credential. Much like having a top-tier university or top company on your resume.

Is It Really Worth It?

Absolutely, but the ROI is in your network, not your bank account. As the saying goes, “Your network is your net worth.” Publishing the book opened doors to connect with industry leaders, CTOs, and senior engineers. I’ve been invited to multiple conferences to share my insights so it has been my way of giving back to the community. The connections allow you to learn from other leaders’ failures, successes, and decision-making processes.

Adding to above, it’s not like you’ll make nothing. The idea is not setting high expectations on monetary terms.

Project Initiation

Me and my co-author Jayanth Kumar took close to 2.5 years to finish the project. The details outlined as part of this blog are specific to O’Reilly but it’s similar with other technical publishers as well. I’ll divide the process into multiple phases from initial thought process to hard copies in my hands.

Proposal Submission

Starting out is always hard. Jayanth shared this idea with me and had some details already put up in the proposal. We brainstormed among ourselves on the structure of the book so as readers are hooked to the content throughout.

You can find all the details on the O’Reilly Work With Us page regarding different ways you can contribute.

The Proposal Details

Here is the high level description of what you need to fill in the proposal (make a copy of proposal from here).

• Book title and subtitle: The title of the book. For subtitles, you can also suggest multiple subtitles and decide later what works best.

• About the Author: This is basically an author biography and why do you feel you’re the best person to write the book on a particular topic. You can include your past work such as your industry experience, previous technical books/blogs/videos, speaking at conferences, etc.

• Marketing Description: This is something you find on the back cover of the book. You’re basically pitching to buy the book to a potential reader.

• About the topic: This section explains the book title in detail and why it’s important.

• Audience: Who will read this book?

• What the reader will learn: There is always some end goal if you’re selling a product. If the reader finishes the book, what kind of knowledge gain happens and how/where the reader will apply this knowledge.

• Competing titles: If there are similar titles in the market, how this book differs from them and what new values this book specifically offers.

• Book outline: Here you include a detailed chapter-wise table of contents.

• Schedule: How much time will you take to finish the first two chapters, half manuscript and full manuscript. Keep in mind that life happens and if you’re working on the book writing along with your full-time job then keep a buffer.

Balancing the content takes effort, what you go into the book and what should be skipped is the decision you’ve to make. Be ready to spend so much time on choosing topics to be included as part of the book and then again striking them off because you found other things which are more important. Once the draft was ready, we directly emailed one the Content Acquisition Editors from O’Reilly.

Proposal Submission Timeline

We were hopeful of the proposal approval as we firmly believed this is a unique book and solves a lot of gaps in the content already available in the market. Good things happen if you’re hopeful, and we received a positive response to the proposal. At this point, it’s not 100% approved but the acquisitions editor will work with you to modify some details as needed. For example:

• We had a lot of discussion if Part II of the book (AWS specific content) should be included or not. The point being made was there is already a lot of AWS documentation available on the internet, so what new value does the book bring? We finally agreed to keep it so readers can find everything at single place and they are able to follow along from Chapter 1 to Chapter 21. You don’t want to ask the reader to go read on internet about a specific AWS service and then come back to the book on how it’s used practically.

• We had 15 chapters in Part III of the book in the initial proposal which was trimmed down to 8 later. As I was saying earlier, you’ll have to balance the content.

TIP: We also wrote one chapter from Part III and shared after proposal submission on how we’re envisioning the content. I later scrapped it off and wrote it from scratch but if you can, it’s a good idea to have some content ready from the chapters.

After all this communications, the proposal was officially approved and we were presented with a contract to sign. Once signed, the actual work started.

Book Writing Process

You’re assigned a development editor to work with you throughout the book after contract signing. In the first meeting, the development editor shares details like:

• Writing tool: You can either write in Google Docs or O’Reilly has an in-house tool called Atlas.

• Development Schedule: You agree on a chapter-wise schedule on when the chapters will be ready.

• Writing Style: Details around illustrations, writing style, content structuring, etc.

Once you finish a chapter, you share the chapter with the development editor. The role of development editor is to share feedback on the first draft. You work on the shared feedback and it becomes a continuous loop for all the chapters.

After development review is done and feedback is addressed, the chapters are published to O’Reilly platform in early release form. The good thing is people can read chapters early and share reviews on the platform. It’s much similar launching a product MVP, gathering feedback and improving on it before the final launch of the product.

In parallel, the book chapters also goes through technical reviews. For technical reviews, O’Reilly has their own reviewers and you can also suggest from your network. The technical reviews are phased into generally 3 parts, first two chapters, then half manuscript and then full manuscript.

After development and technical review completion of the chapter, the next step is the production phase.

Book Production Phase

A production editor walks you through the book production timelines and different stages as part of this process. Here are few things taken care of as part of this phase:

• Copy Editing: A copy editor goes through the entire manuscript to make sure it matches the O’Reilly style guidelines. This stage generally doesn’t require too much input from your end. This is divided into two parts, QC1 and QC2. You can suggest anything as part of QC1 and that is addressed as part of QC2.

• Illustrations: In parallel to copy editing, an illustrator works through the images and recreates them as per O’Reilly style. You can work through the illustrations for any suggestions from your end.

• Index: An indexer is assigned to the project responsible to create the final index of the book.

The book is ready to go to print after all these steps as part of the production phase. The final version of the book is available within a week on O’Reilly platform and it generally takes 3-4 weeks for physical copies to be printed out.

Let’s talk a little about the book I wrote with O’Reilly - System Design on AWS.

System Design on AWS

System Design on AWS was written with a clear philosophy: to be useful both “during the interview and after the interview.” It is designed to take your knowledge beyond basic whiteboard theory so you can make effective long-term architectural decisions when building actual products.

Book Content

• Part 1 - Fundamentals: Covers core distributed building blocks like load balancers, caching layers, databases, network protocols, containerisation and general architecture patterns.

• Part 2 - AWS Services: Maps out the specific AWS services available across different operational domains (compute, databases, networking, event-driven, observability, big data, machine learning etc.).

• Part 3 - Practical Implementation: Focuses on real-world system design philosophy. It walks through building complex applications from scratch such as a stock broker platform or a hotel reservation system. From gathering initial requirements to deploying the actual AWS components. It emphasizes that drawing boxes on a whiteboard is not enough; you need to know how to actually run the system.

You can find everything about the book on https://msdeepsingh.com/books/

Get System Design on AWS

📕 O’Reilly: https://learning.oreilly.com/library/view/system-design-on/9781098146887/

📕 Amazon.in: https://amzn.to/4jExkte

📕 Amazon.com: https://amzn.to/3D3BIBJ

📕 Amazon.co.uk: https://amzn.to/3QW6Fvf

📕 Shroff Publishers (Indian Subcontinent): https://www.shroffpublishers.com/books/9789355428035/

Before closing, I want to share my opinions on the choice of publisher.

Self Publishing or Publishing House?

Self publishing means you’re already well-known in the industry and people will buy the book looking at your name as author. If you’re a first time author like me, you’ve to give some reason to readers so they can trust you. O’Reilly solved this problem for me. O’Reilly is a trusted name and they don’t publish random jargon, the book goes through a strict process so readers clear the first barrier of buying the book.

Also, we were not aware of the overall process. You’ve to spend some additional time on marketing, content reviews, copy editing, etc which are taken care of by O’Reilly in-house. Given we both had full time jobs, O’Reilly was the preferred choice and I’m very glad we pursued this project.

On a closing note, writing a technical book may not make you a millionaire, but it will sharpen your communication, expand your professional network and strengthen your domain expertise.

Best of Luck!!!

You can reach out to me on LinkedIn for specific queries.

Decision Flowchart

Let me take a reference from my O’Reilly book “System Design on AWS” Chapter 3. Figure 1 shows a flow chart on how can you navigate through the choice of database over multiple options available in the market.

Figure 1 Decision Flowchart for Database Choice

The question that comes to your mind, can I follow this flowchart blindly to answer database choice questions? The answer is NO. In Software Architecture and design decisions, there are always trade-offs with any direction you go with. So saying, a particular database is perfect is wrong. In addition to the above flowchart, database choice should be combination of below factors:

Business Requirements

The most important decision criteria is that the database choice should meet the requirements. All other things comes later.

You can start by looking at data access patterns. The databases such as DynamoDB only makes sense if you know the access patterns. If there so many joins required, and you don’t what kind of queries will be there, relational database is better choice. This can very much happen for a new project or a startup; relational database is good choice to start with.

Don’t forget to checkout the “System Design on AWS” book for detailed dive deep on relational, non-relational databases and how best to use them.

Everything about "System Design on AWS"

Previous Expertise

If you already have expertise with some technology then you know how to use it in the best way possible. If something meets the requirements and you also have previous working experience, nothing beats this combination.

In case you’re new to databases or your use case is totally new, try to stick with battle-tested software. If a lot of people have used something at scale already, you’ll get ton of support from community.

Cost

You should be able to afford something to use it. Cost is important consideration, this is mostly applicable to managed offerings. When it comes to cost, I recommend to look at total cost of operations and not just the monthly bill paid to the provider.

See,

• How much time you’re saving in launching your product by leveraging managed offering. This is also applicable to other solutions as well and not just databases.

• How much operational burden it takes off from your plate.

With this, let’s conclude the discussion.

Conclusion

Always consider trade-offs for any decision you make, what are the pros and cons for any choice being made. Also, there is thin line between database features these days. There are a lot of managed relational database offerings with horizontal scaling support so you don’t have to implement this in-house. There are a lot of non-relational database offerings with transactional support.

I’ll recommend to choose non-relational only if you know you need it. Else stick to relational databases, they solves most of the requirements. Happy Building!

Check out the YouTube video for the discussion with the community -

As I was going through the book, I created chapter-wise short notes and YouTube videos as a summary.

DDIA Chapter 1-12 Notes

70.1MB ∙ PDF file

Download

Chapter-wise summary of Designing Data-Intensive Applications book

Download

Watch on YouTube

Should you read the book?

An excerpt from the O’Reilly around usefulness of the book

• Peer under the hood of the systems you already use, and learn how to use and operate them more effectively

• Make informed decisions by identifying the strengths and weaknesses of different tools

• Navigate the trade-offs around consistency, scalability, fault tolerance, and complexity

• Understand the distributed systems research upon which modern databases are built

• Peek behind the scenes of major online services, and learn from their architectures

Where can I get a copy?

• amazon.com

• amazon.in

• O’Reilly online

• ScyllaDB sponsored chapters for free

Figure 1. Train booking application

There was something interesting about the project. There was no underlying database. The data was directly dumped to files, including login details, train details, booking details, etc. Example, train details for sun stored in sun.txt. Checkout code on Github - https://github.com/msdeep14/irctc

1 2 12956 jp-pune_exp 3:20PM 51 587
2 1 12957 jp-pune_exp 9:15AM 87 501
20 21 19876 goa_exp 5:00PM 54 709
3 2 10786 jhelum 5:25PM 112 654
1 9 11760 jp-mumbai-exp 2:00PM 67 543
1 2 34561 premium 11:00AM 45 1700

So I think I’m eligible to answer the question; why do we need database if you can store data in files?

Bottlenecks

If there is already classic file storage available, what’s the need for all kind of databases? Let’s take an quick example, you wish to get all the trains from Jaipur to Pune:

• The code snippet above shares trains availability as per source to destination. The beginning two numbers are keys mapped to locations.

• Essentially, you’ll pick the file for a day and find the train.

• What if there is no direct trains? It constructs a graph from file data and finds the trains matching to source and destination.

It’s works all great for college project because there is not much data but soon it will face problems:

• Increased latency

• Supporting different query patterns

Introducing Databases

At a very high level, databases offers structured way of accessing the data. Depending on the use case, the data is stored in a way that it’s retrieval is fast. There are different kind of databases depending on the requirements. Chapter 2, 3 & 10 of O’Reilly title “System Design on AWS” explores multiple databases with their architectures.

Here is a diagrammatic representation:

Figure 2 Decision Flowchart for Database selection

Let’s take an example from train ticket booking. Instead of storing the data in the files, if it’s stored in relational database such as PostgreSQL then it makes easier. For the query pattern, get all the trains from Jaipur to Pune. The point of consideration was how can we skip the reading all the records. The database indexes can be created location and time wise, reducing the data footprint.

Figure 2 offers some clarity on decisions can be made. Further, database choice is not just the use case, it’s combination of all below factors. This is also covered in YouTube channel video.

• Solves the use case (read/write access patterns)

• Previous Expertise with the database

• Cost

• Battle-tested software

• Cloud Managed vs Self Managed (Operational Overhead)

• Time to Market

Watch on YouTube

Subscribe now

The architecture evolves over time and the idea should always be we’re able to iterate over it.

• It’s nice when you get stuff right the first time but we don’t need to

• As long as you get the architecture as right as possible; the implementation stuff isn’t going to be big of a deal

This article captures the very early architecture of Facebook, how it started and challenges faced with scaling.

Initial Architecture

Facebook was launched in February 2004 with focus on connecting people for one school

Tech Stack

• Programming Language: PHP

• Database: MySQL

• Apache Web server

Figure 1 Early Facebook Architecture

The simple architecture served well for single school, but soon there were challenges with more interest and increasing scale.

Scaling Challenges

Let’s discuss few of the challenges faced due to architecture limitations:

• Privacy: Data and feed separation between the schools

• Traffic Distribution:

  • How the data should be distributed for different schools

  • Some schools have heavier usage as compared to others.

• Figure out friends connections with higher degree (friend-of-friend or friend-of-friend-of-friend)

Let’s discuss on the improvements in the architecture to handle the bottlenecks.

Improving Architecture

The key aspect is hiring more and more smart people and give them independence to work on the improvements.

Onboarding More Schools

The first thing that comes to mind for handling more traffic is adding more machines.

• We mentioned web server and MySQL coupled on same machine, so the idea is to decouple the storage from application server and introduce load balancing between the machines.

• This also removed single point of failure. Consistent database layer ensures traffic can be served from any operational machine.

Handling Friend-Of-Friend

If you wish to figure out friend of a person, it’s a direct query to database. For friend-of-friend:

• Retrieve friend list of a person

• Iterate over this list and get friend list of all the items

• This becomes n*n operation.

For friend-of-friend-of-friend, it becomes n*n*n operation; so it’s compute-intensive. The current architecture uses dedicated graph database, but back then:

• Split the database and create one instance of mysql database for each school in the network

• How does this help? Now you’ve to compute n*n*n for 10k (approximate number of people signed up from one school) instead of n*n*n over 6 million (all people across the schools)

The above is in addition to decoupling application servers and database compute. This helped a lot but further the traffic was reaching 100 Million pages/day, a query approximately takes 2-4 milliseconds. The problem lies in each page repeating on the same queries.

Improving Database Performance

What comes to mind to solves the same query problem? Cache the results to avoid hitting the database. Facebook introduced Memcached as caching layer; this brought down query performance to 0.3-0.5 milliseconds.

There were couple of issues though with the distributed architecture of memcached instances; If a memcached instances go down:

• There is no redundancy on information.

• Cache miss occurs; and all the traffic routes to database.

Facebook introduced photo sharing feature, let’s discuss on how it’s implemented.

Photo Storage

From early stage, Facebook chose to smaller machines instead of one big machine. The disk is relatively slow but stable, to overcome the slowness:

• High RAM Caching machines to hold most of the thumbnails and frequently accessed images.

The space consumed by photos is on higher side, some issues uncovered:

• If you wish to upload a photo album with 30 photos (3 MB each), then 90MB is total size; Networking issues with slow internet.

• Router used was built to support 1GB at a time.

Solution:

• Built Java applet and ActiveX control to couple the choosing of photos

• Compression at client side.

It helps:

• Upload photos quickly

• Resource optimisation - less server CPU utilisation

Even with caching machines, transferring images to users takes time. So the solution is to move caching near to users, aka, Content Delivery Networks (CDNs).

Conclusion

Follow Iterative Design Process; Remember you don’t have built it right (or perfect) the first time. Happy Building!

Subscribe now