• Few of the concepts are specific to JAVA. I used to write a lot of Java and Kotlin code earlier :)

• The information is compiled here basis my understanding by reading multiple blogs/oracle docs/videos and is mostly suitable for interview preparation. You can explore more on internet (some references are also included) for further deep dive.

• These are self-written notes without AI help, so don’t expect nice and fancy English. Written in minimal terms for quick revision and reference for interviews.

• There are additional code samples in GitHub repo: msdeep14/getAheadWithMe.

Let’s start the discussion with the most basic term, process.

Process

You can think of process as a sequence of steps to be performed to achieve a task. Here are some key details on how processes work on an Operating System (OS).

• Process has separate memory allocated space. A process doesn’t share memory with other processes. This ensures complete isolation of processes from each other. In summary, one process’s state don’t impact the state of another process.

• Processes interact with each other via Inter process communication (IPC) mechanisms such as Shared Memory or Message passing.

• Processes take more time in creation, termination, context switching (in comparison to threads, discussed shortly). Why? Because process has its own Process Control Block(PCB), stack and address space. Separate data segment and code segment is created at each process level.

• A process goes through multiple states. These states include NEW, READY, RUNNING, WAITING, TERMINATED and SUSPENDED.

• Process creation requires OS API call - fork(). For example, java.lang.Process for JAVA.

• Processes don’t require synchronization between them unlike threads where synchronization needs to explicitly managed to avoid unwanted issues.

There is mention of threads at couple of places above, let’s discuss threads next.

Threads

Think of thread as sub step of a process. A thread can be referred to as a light-weight process. Multiple threads can be created inside a process to carry out specific actions. Here are couple of characteristics associated to a thread.

• Threads share the memory space which makes them less latent.

• If one thread is blocked within a process, it will also impact other threads in the same process.

• Each thread in process has their own stack and register.

A thread can be of multiple types, let’s discuss the thread types next.

Thread Types

A thread can be of two types, user level and kernel level.

User Level Thread: Threads managed by users and created via user-level libraries such as JAVA Thread library.

Kernel Level Thread: Threads managed by kernel. Kernel-level threads are created via OS system calls.

Java Virtual Threads

Light-weight implementation of Java threads. 1 Java Thread == 1 OS thread, which is resource intensive. Virtual threads store their stack frames in Java’s garbage-collected heap rather than in monolithic blocks of memory allocated by the operating system.

A good read - infoQ article. I earlier mentioned there is no need of synchronization for processes but it is for threads. Let’s discuss what I mean from thread-safe.

Thread-safe

Thread-safe class/method/codeBlock means a class/method/codeBlock that can used by multiple threads concurrently without any issues. The output of threads should be same as if they were processed sequentially.

Singleton is class used by multiple threads as a single object exists. The singleton class should be thread-safe. This is also referred to as Double-Checked locking with Singleton. Refer ThreadSafeSingleton class for code implementation.

If the code is not written properly, e.g. one thread block another thread for execution, the situation can lead to deadlock. Let’s discuss deadlock next.

Deadlock

A process/thread is in waiting state because it is waiting for a resource that is held by another process P2. P2 is waiting on another resource held by P3. P3 is waiting for another resource held by P1. All the processes are waiting on each other without being progressing eventually leading to deadlock situation.

Deadlock conditions: All four conditions should satisfy for system to be in deadlock state -

• Mutual Exclusion: resources can’t be shared by multiple processes at the same time.

• Hold and Wait: holding resource and waiting for other resource.

• No Preemption: resources acquired by process can only be released by itself once execution completes.

• Circular Wait: processes waiting on each other.

There are ways to avoid being in deadlock situation. If deadlock occurs, the system should get out of it. Let’s discuss them next.

Deadlock Handling

We’ll discuss three techniques for deadlock handling: prevention, avoidance and detection.

Deadlock Prevention: Don't let system enter into deadlock state by avoiding any of the four conditions specified in above section.

Deadlock Avoidance: The resource request is analyzed before-hand to ensure if it can cause any deadlock situations in the future. To perform this analysis, it requires all the information in advance around how resources will be requested/used in the future. Deadlock avoidance algorithm - Banker's algorithm.

Deadlock Detection:

1. Check process state for deadlock in system.

2. If isDeadlock == true, apply algorithm for deadlock resolution.

   1. Process Termination: Abort the processes involved in deadlock. This might result in program failures as process might be partially completed. Instead of all process abort, system can look for processes which are of low priority or are running for too long in the system.

   2. Resource Preemption: resources are preempted and allocated to other processes until deadlock is resolved.

Moving on to a similar concept to deadlock, livelock.

LiveLock

The threads/processes are not blocked as in deadlock situation but the state keep on changing from one to another and not making any further progress.

Example:

Queue Message processing: Queue consumer failed with exception and it puts back to queue in hope that it will processed next time but it keeps on failing again and again with same exception. A solution could be to keep a upper limit on number of retries for specific failure if msgs are put into dead-letter queue. Failed msgs should anyway never be put in the same queue again for re-processing.

Starvation

Process in on hold because resources are not allocated to it. This could happen if resources keep getting allocated to high priority threads and low priority threads keep on waiting.

Some ways to avoid starvation situation -

1. Age of process - If process is too old, it should be given priority for resource allocation even though it is low priority task.

2. Multi-level feedback queue - geeksforgeeks article

Mutex vs Semaphore

Semaphore

Semaphores can be used to manage access of limited resources(via counting semaphores) or to resolve critical section problem(via binary semaphores).

Refer SemaphoreImpl class for sample example.

Mutex

Mutex = Mutual Exclusion. It is similar to binary semaphore which ensures only one thread can acquire it and access the critical section.

1. Mutex is a locking mechanism whereas semaphore is signalling mechanism. The context which acquired the mutex lock can only release it but this is not the case with semaphores.

2. The #1 can prove to be downside if mutex lock is not released as it can’t be unlocked from different context than who acquired it leading to starvation.

I/O Bound vs CPU Bound

I/O Bound: Application is referred as I/O bound if it involves reading/writing from/to input/output system and waiting for information.

The main disadvantage of such applications could be the application can be spending too much time in I/O operations without doing any actual operations. Example - upload/download files.

Share System Design From Scratch

CPU Bound: Task/process execution is dependent on CPU without requiring any I/O operations. Example - High Performance computing

Blocking I/O vs Non-Blocking I/O

Blocking I/O

The request waits for server to complete its processing. In general terms, the request will wait for read or write before returning back meaning the thread will be blocked state until there is data available for read or ongoing write operation is fully completed.

Blocking I/O operations are available with java.io package.

1. One byte data at a time - InputSteam and OutputStream

2. Wrapper for streams - Reader and Writer.

Non-Blocking I/O

The request is executed immediately. There could be possibility that request is not processed as soon as it is received, if there are no available threads to process the request, the request is queued(to be processed at later point of time) and response is returned. Other way to look at is if there is ongoing write operation, the thread can perform any other operation.

In analogy to event-driven architecture, you can think of this scenario as communication between service A and service B. service A invokes service B API for some processing and gets a response that request is received by service B. Once the request processing completes by service B, it can publish an event to SNS topic(or Kafka topic) to which service A is subscribed. service A can then perform any additional required processing basis Service B response.

Non-Blocking I/O operations are available with java.nio package.

1. Read chunks of data at a time - Buffer.

2. Mapping raw bytes to/from readable characters - CharsetDecoder

3. Outside world communication - Channel

4. Enable multiplexing on a SelectableChannel and provide access to any Channels which are ready for I/O - Selector.

Volatile/ Atomic Integer/ Synchronized

Volatile Keyword

Volatile keyword is used inside a class to make it thread-safe. It can be used for any variable such as value of it can be modified by different threads at same time without any problem.

1. It can be used for primitive or object types though it can’t be used with classes/methods.

2. Volatile keyword enforces to always read variable value from main memory and it is not cached. Both reads and writes will be atomic.

3. Compiler don’t perform any optimization and order of instructions is maintained.

4. It can be used as replacement of synchronized in specific cases.

NOTE 1 - Visibility problem is created without the usage of volatile keyword. How? Any modification performed by a thread A is updated thread A’s local cache so thread B will still assume previous value of any variable. This can be solved by volatile keyword as volatile enforces to read always from main memory and not from local thread cache.

NOTE 2- volatile can be used to resolve visibility practice, prefer to use it for setting flag variables. For compound operations or counters, prefer AtomicInteger/AtomicLong/AtomicReference.

Atomic Integer

java.util.concurrent.atomic.AtomicInteger. Refer the sample example and comparison of volatile, AtomicInteger and synchronized keyword in class VolatileKeywordImpl.

AtomicInteger count = new AtomicInteger(1);

Operations available with AtomicInteger -

1. incrementAndGet

2. decrementAndGet

3. addAndGet(int delta)

4. compareAndSet(int expectedValue, int newValue) - Type of optimistic locking.

NOTE - AtomicReference provides atomic operations on underlying object reference variable.

Synchronized

Synchronized keyword over a method or block of code ensures only single threads access it at a time avoiding race conditions.

Refer SynchronizeSample class for example.

1. Any object in synchronized block is placed inside a lock(lock acquired on start of block and released at end of the block). The lock is internal monitor which is responsible for checking synchronization.

2. Synchronized keyword should not be used with constructors/variables.

Locks

Locks can be used to ensure single thread access the critical section at a time. Think of lock as a guard to critical section and this guard provides key to enter the critical section and while leaving critical section, handover this key back to guard. Java Lock is similar to synchronized block discussed in above section.

Reentrant Locks

private static Lock lock = new ReentrantLock().

1. synchronized is implicit lock and locks are explicit. We need to specifically define variable to create a lock and use it in the code block.

2. You can acquire the lock in one method and release in another method. There is no such flexibility in synchronized keyword.

3. tryLock() method with Locks which can be used to check if lock is available.

4. ReentrantLocks allows to call lock.lock() multiple times without calling unlock(). Finally you can unlock() for the number of times lock() was called. getHoldCount() can be used to check how many times to lock() is called.

5. To make the lock fair, use lock = new ReentrantLock(true). This ensures that longest waiting thread gets the lock. Fair locks can be little slower as an extra queue needs to be maintained based ageing. Unfair lock could be faster, but it can potentially lead to thread starvation in some cases.

6. tryLock() doesn’t honor lock fairness. To make it fair, use lock.tryLock(0, TimeUnit.SECONDS).

Refer LockSample class.

ReadWrite Locks

Reentrant locks can be inefficient if there are threads blocked which only want to read some data. ReadWriteLocks helps to optimize this kind of use-case.

1. Reentrant lock - one thread at a time.

2. ReadWrite lock - one writer thread at a time OR multiple reader threads at a time. Key Point - Both read and write threads will not be given permission simultaneously, either one of them can execute at a time.

3. ReadWrite lock can be efficient for use-cases which requires high number of reads and less writes in general.

Refer ReadWriteLock class.

Optimistic and Pessimistic Locking

Pessimistic Locking - Lock the object beforehand performing any operation. Java synchronized keyword is example of pessimistic locking. It has extra performance bottleneck as threads have to wait unless permitted to execute specific operation.

Optimistic Locking - As name suggests, in this approach you can go ahead and execute the operation with assumption that no other thread will interfere. It relies on collision detection and handle if detected via retry/failure. Java CompareAndSwap is example of optimistic locking.

Two-Phase Locking

Locking and unlocking has two phases -

1. Growing Phase - Locks are obtained but not released.

2. Shrinking Phase - Locks are released but no new locks are obtained.

This ensures serializability but can cause deadlock, starvation and cascading rollback.

The last lock acquired in growing phase is referred as Lock Point. Lock Points are used to define serializability, example basis lock point you can identify which transaction comes first in series of transactions.

Phaser, CountDownLatch and CyclicBarrier

Read more on oracle docs. Here is the summary of these terms.

CountDownLatch

This is helpful in scenarios if there is requirement to initialize some piece of code before executing further system logic or other threads. Think of a scenario, where database connection to be established before other threads can actually carry out the db query operations.

Refer CountDownLatchSample class.

CyclicBarrier

Set of threads wait for each other to reach a common barrier point. and once common point is reached, then all the threads can carry forward their execution.

Why cyclic? The same barrier can be used again and again among the threads.

Phaser

It is similar to CountDownLatch with some added functionalities. Phaser allows dynamic number of threads that needs to wait for continuing execution, in countDownLatch, we specific fixed number of parties upon initialization.

1. register() - party can register itself

2. bulkRegister() - extra parties register in bulk

3. arrive() - parties can arrive and continue

4. arriveAndAwaitAdvance() - parties can arrive and await for other parties.

5. arriveAndDeregister() - Parties can arrive, deregister and continue.

Conditions

Take an example of Thread-1 and Thread-2. Thread-1 complete execution in two stages. Stage-1 and Stage-2. Stage-1 is executed and the execution of State-2 is dependent on completion of Thread-2. In such scenarios, Thread-1 will put an additional condition condition.await() to put thread in blocked state. Once Thread-2 execution is completed, it will run condition.signal() or condition.signalAll() and with this signal all the threads which are blocked on this condition will carry forward their execution.

Subscribe now

This is similar to monitor.wait() and monitor.notify() in synchronized code block.

Refer ConditionSample class.

wait(), notify() and notifyAll()

wait(), notify() and notifyAll() can be used in replacement of conditions. They can be applied on any java object.

1. wait() - similar to myCondition.await()

2. notify() - similar to myCondition.signal()

3. notifyAll() - similar to myCondition.signalAll()

Refer MyBlockingQueue class and MyBlockingQueue2 class for sample examples.

ExecutorService

1 Java Thread == 1 OS Thread

How to run tasks in parallel? via using Thread class.

for(int i=0;i<1000;i++) {
   Thread thread = new Thread(some Runnable);
   thread.start();
}

The above operation will be too expensive as 1000 threads are being created at the same time which could lead to system crash.

To solve this problem, you can create a pool of threads and submits the large list of tasks to them. Now the tasks can be picked up as there are free threads present in the thread pool which are being maintained in a BlockingQueue.

Example - ThreadPool size = 10, even though submitted task size = 1000, the system will execute max 10 tasks at a time.

   ExecutorService service = Executors.newFixedThreadPool(10);
   for(int i=0;i<1000;i++) {
      service.execute(Runnable task);
   }

submit() vs execute()

submit()execute()return type is Futurereturn type is voidBoth Runnable and Callable can be passedRunnable task can be passed as input

ThreadPool Types

1. FixedThreadPool - initialize thread pool with fixed number of threads.

2. CachedThreadPool - tasks are held in synchronous queue. It will check if there are already available free threads, if yes then use one from available else create a new thread and assign task to it. This threadPool kills the idle threads with lifecycle more than 60 seconds.

3. ScheduledThreadPool - tasks that are required to be submitted after certain delay or require scheduling. It stores the tasks in delayQueue which stores tasks basis when these tasks should be executed(priority order is basis that).

   1. service.schedule(Runnable, delay:10, SECONDS). Task will be enabled after 10 seconds.

   2. service.scheduleAtFixedRate(Runnable, initialDelay:15, period:10, SECONDS). First task will start after 15 seconds. Subsequent tasks will execute at initialDelay+period, then initialDelay+2*period, and so on...

   3. service.scheduleAtFixedDelay(Runnable, initialDelay:15, delay:10, TimeUnit.SECONDS). First task will start after 15 seconds. Once this is completed, second will start after 10 seconds of 1st task completion.

4. SingleThreadedExecutor - similar to FixedThreadPool with numThreads=1

Future

1. Placeholder for value that will arrive sometime in the future.

2. To get actual response of submit(), use future.get().

3. get() is blocking operation - it will wait until the thread result is returned. We can use future.get(1000, TimeUnit.MILLISECONDS) to specify a timeout value for it to be blocked state if result is not available.

Runnable vs Callable

Runnable return type is void whereas Callable can return a value.

Refer RunnableVsCallable class for example.

CompletableFuture

1. Assume you’ve used a for loop to gather all the results from callables.

2. There is possibility that as your iterate over future list, not all futures are computed at a moment.

3. In such scenario, the thread will blocked for time being until the future result arrives for specific thread.

4. There can be a case in which you’re waiting at 4th thread whose result is not computed yet but results for 10th and 11th thread are already in.

5. But since you’re iterating in for loop, you’ll still be blocked on 4th thread.

6. How can you overcome this?

   1. use timeout value on future.get(1000, TimeUnit.MILLISECONDS)

   2. use CompletableFuture to run threads independently.

   3. Use ExecutorCompletionService - helps in processing the threads as their execution completes.

CompletableFuture helps to run a series of tasks on a separate thread than the main thread. Consider an example of uploading a youtube video on @MsDeepSingh channel, what are the steps.

1. upload video file from local store.

2. run validation checks

3. upload in SD format

4. upload in HD format.

Share

Now these steps should be executed one after another and also there could be multiple users who are uploading videos on their YT channels. CompletableFuture makes it easier to code such pattern with proper exception handling.

ExecutorCompletionService

ExecutorCompletionService helps to retrieve response as Futures finish. Refer ExecutorCompletionServiceSample class for example.

ForkJoinPool

Achieve similar functionality to ExecutorService. ForkJoinPool is optimized for use-cases where tasks creates sub-tasks on the go. Example - mergesort. As the name suggests, ForkJoin consists of two operations -

1. fork - divide task into sub-tasks

2. Join - aggregate the result of sub-tasks.

Per-thread queuing - Each thread has their own deque, so all the sub-tasks will be part of this deque.

Work-Stealing algorithm Take a scenario of two threads, Thread-1 and Thread-2. In a scenario where Thread-1 created large number of sub-tasks and Thread-2 has very less number of sub-tasks. Once Thread-2 completes its subtasks, it will subtasks from Thread-1(from other end of the queue) to help out in processing of subtasks. In simple terms, available threads take up tasks from busy threads.

Synchronous Queue

Synchronous queue is similar to blocking queue with size=1. Refer producer-consumer example to understand about blocking queue.

so why do we need a queue with size=1?

There is direct handoff between producer and consumer. The item is not moved to queue. The producer will not be allowed to put item into queue even though the queue is empty. The producer is only allowed to put item to queue once consumer requests for item. Well, technically item is not put into the queue. The item is passed on once consumer thread requests for it.

ConcurrentModificationException

When collection is modified when already an operation is going on. Example, iterating over the list and at the same time removing of an element.

Monitors

Monitors are nothing but implementation of synchronized keyword in JAVA. Monitor is an abstraction that helps to achieve mutual exclusion between multiple threads trying to access a resource concurrently.

Additionally in JAVA, any object can act as a monitor utilizing wait() and notify() methods. Refer MyBlockingQueue2 class for example.

Practice

Producer Consumer Problem

1. Multiple producers and consumers.

2. Producer is responsible for generation of new tasks.

3. Consumer is responsible for executing the tasks created by producer.

Clarifying questions -

1. What’s the maximum number of tasks that producers can produce? Is there any upper cap? For this problem, assume producers will produce max n tasks and then wait for consumer to consume.

2. The scale of operation, how many producers? how many consumers?

Approach - Producer

1. Create a FIFO queue with max size = n

2. Producer threads will put tasks to queue until full.

3. Block producer threads if queue capacity is full.

4. Publish a signal that queue is not empty.

Approach - Consumer

1. Get tasks from queue if queue is not empty.

2. Block consumer threads if queue is empty.

3. Publish a signal that queue is not full anymore once consumer fetches a task.

Dining Philosopher’s Problem

Round dining table where philosophers are sat. There can be two states for them -

1. Eating

2. Thinking

They need to make use of forks, philosopher need two forks adjacent to him and starts eating. Problem? The forks are limited, with 5 philosophers there are only 5 forks but philosopher need two forks to start eating, so all philosophers can’t each at the same time.

1. No philosophers who are adjacent to each other should not eat at the same time.

2. A philosopher may pick up only one fork at a time.

3. Philosopher can’t pick fork from neighbour’s hand.

4. Once philosopher is done eating, he puts back the forks on the table.

Why solve such problem? - Problem of resource utilization and how limited set of resources can be allocated to bunch of processes.

Solution Approach

1. Each fork is represented by semaphore.

2. Philosopher picks up fork and put corresponding semaphore to await() state.

3. Once done eating, philosopher releases both left and right fork with signal() operation.

//binary semaphore, initialized to 0
Semaphore forks[5];

for philosopher i
do {
    //change semaphore value to 1 ---> java: semaphore[i].acquire();
    await(fork[i])
    await(fork[(i+1)%5])
    //eat
    
    //change semaphore value back to 0 ---> java: semaphore[i].release();
    signal(fork[i])
    signal(fork[(i+1)%5]);
    
    //think
} while(true);

Any issues with above approach? Deadlock situation - All the philosophers get hungry at the same time and pick up the left fork. Now all the philosophers are waiting for fork in the right and none of them is actually eating.

Potential solutions -

1. Allow only 4 philosophers at the table for the table of 5.

2. Allow philosopher to pick up a fork only if both forks are available. - execute code in critical section - Refer wikipedia blog for Dijkstra’s solution.

3. Asymmetric solution - odd philosopher picks up first left fork and right fork; even philosopher picks up first right fork and then left fork. This can also be achieved by - All the philosophers pick up their left fork except the last one, so essentially the 5th philosopher will pick right fork first and then the left fork. Refer this implementation in DiningPhilosophers class.

<image generated using Grok>

Practice

1. Leetcode problemset - https://leetcode.com/problemset/concurrency/

2. https://www.careercup.com/page?pid=threads-interview-questions

3. Refer to src/practice folder for problem set.

What’s early release? Well you get hands on the content before it’s edited or technically reviewed. It helps to gather feedback from larger audience and incorporate it before hard copies are out.

About the Book

As a product grows from a proof of concept to a large-scale distributed system, its design has to evolve with it. System Design from Scratch follows that process, evaluating trade-offs involving scale, cost, expertise, time to market, resilience, and operational overhead at each stage.

Software engineer Mandeep Singh takes an iterative approach to system design, showing you how choices around compute, databases, load balancing, caching, deployment, and software engineering principles change as systems scale. As you follow along, you’ll learn how to weigh technical and operational considerations and make choices that fit your organization’s needs.

• Understand how core system design concepts work together

• Apply a decision framework to evaluate trade-offs involving scale, cost, expertise, time to market, and resilience

• Make technical choices based on your organization’s needs

• Design systems that are practical to run and maintain over time

• Use AI as an assistant in system design or as a component of the system

What’s Unique?

System Design content, wherever it’s available is theoretical and heavily influenced by the interview preparation. System Design From Scratch takes entirely a different approach, an approach of how you work in actual life. A Software Engineering journey.

The role of this book is to not act as a consultant giving recommendations, but as a fellow engineer working on the problem statement piece by piece and developing an understanding of system design from scratch.

System Design is not theoretical and it should not taught via theoretical means drawing boxes. Don't make system design decisions because you read on some blog. Experience the trade-offs first hand with my new O'Reilly book, ‘System Design From Scratch’.

Practical, How?

Practical means actually thinking, designing and building and deploying the systems. The journey mentioned earlier is followed via the GitHub repository - msdeep14/systemdesign-from-scratch.

This is public GitHub repository, refer README.md to make best use of the repository. Feel free to contribute in any way possible :)

Coming up, NEXT…

I’ll use this newsletter to publish the Software Engineering journey for the chapters published as part of the book. Stay Tuned and subscribed for new updates.

Happy Building!

Cache doesn’t just mean Redis. Redis has an early advantage of market adoption but that doesn’t mean Redis is only solution. Caching is a general concept and today we’re going to uncover if adding a cache really helps. If it does, then at which layer (yes, there could be multiple layers) a cache should be added.

Need For Caching

A same request can be made multiple times by a single user or multiple users depending on the application behaviour. Processing the request and generating response takes time as well as processing power. Caching helps remove this bottleneck.

The response once computed are stored in cache store, so user queries can be served from the cache instead of processing the request again and again. You can very well use any datastore as a caching solution such as a relational database (e.g. PostgreSQL, MySQL) and in-memory data stores (e.g. Redis, valkey, dragonflydb).

Read a detailed chapter ‘Caching Solutions and Strategies’ in my O’Reilly book System Design on AWS.

System Design Resources

Today’s blog talks about different caching layers and how a generalized solution can be implemented across the teams. It references DoorDash internal implementation. Let’s start with the problem statement.

Problem Statement

• Each team follows a different approach for caching implementation. Mostly Caffeine is used for local caching (instance level) and Redis Lettuce for distributed caching.

• Each team is responsible to manage their data. The data is exposed to other teams via gRPC APIs.

Every team is spending time to manage the similar kind of infrastructure and posses multiple challenges. Let’s discuss challenges with current implementation.

Bottlenecks with Independent Implementations

Here are few challenges. :

• The cache should not lag behind (by larger margin). Cache staleness should be handled appropriately.

• Single point of failure on Redis. If Redis is unavailable, the whole caching infrastructure falls apart.

• For introducing a new cache (or caching layer), new deployment and rollback strategies are needed. There is no control at runtime.

• There is no standard schema to store the data in cache. Each team follows their own standards for caching keys.

• Observability is big pain point. The metrics like cache hit, error rates are not consistent for every team to monitor the cache success.

• Multi-layer caching support is not there. For teams, cache means Redis in current architecture.

Each team is spending the time to fix these challenges on their own so there is duplication of effort. The better idea could be a centralised solution, let’s discuss the solution in details.

Solution

The implementation abstraction can help teams to focus on the business logic instead of spending time on managing the caching infrastructure. The solution is to create a single interface for client’s interaction, depicted below.

single_interface_cache.get(cache_name, cache_key)

The interface handles querying for the data across multi-layered cache and return once found, mentioned in the sequence below (subscribe for free to continue reading on the architecture details).

Let’s conclude on today’s discussion.

Conclusion

Every system design decision has trade-offs. It’s always recommended to do proper analysis if you think caching can solve a problem. If it does, which caching layer (or combination of layers) should be included as part of constructing request’s response.

You can watch complimentary video on YouTube for the analysis on DoorDash architecture -

Happy Building!

What are some of the things you can do so you can stand out from rest of the crowd and show a clear impact. In short, “Make Interviewer believe that you’re the one”.

Approaching System Design Problems

Every problem can be approached in a certain way and you’ve to time it to 45-60 minutes. Time allocation is equally important because you don’t want to miss critical considerations for a system.

SD interviews are generally timed interviews but I’ve seen instances where you’re given a full fledged problem statement, you brainstorm on it for a week or so and then present a solution.

We’ll discuss the step-wise structure for approaching a system design problem. For each step, we’ll also focus on important details you should keep in mind. Let’s start with first step, system requirements.

System Requirements

The first part of solving something is identifying the requirements and the same is true for system design problem statement as well. Requirements means what a system is expected to do (and not do). The requirements can be classified into two categories, functional requirements (FR) and non-functional requirements (NFR).

As part of requirement gathering, you should be clear on system expectations and don’t leave any ambiguity on the table. A recommendation for interviews is keeping a limited scope. Always try to limit the requirements, keep in mind you’ll have to fulfil those requirements in next 30 minutes.

Let’s take an example of designing Facebook. You can’t design Facebook in one hour or a week or even a year, there are a lot (a lot) of points to consider. So the next step is figuring out what part Facebook should be considered? This could be focused on photos feed, video management, handling ads, etc.

The intention of interviews is never to design full-fledged system. It’s the thought process and how you tackle an ambiguous problem. You’re not being hired to design next Facebook (this could be the case as well) but the idea is if you’re hired, can you come up with solutions to problems. And more importantly, can the interviewer work with you during the job, which mostly depends on how well you give and take feedback throughout the discussion.

A subpart of requirements is NFR. Don’t throw random words for sake of completion (because you’ve seen bunch of videos on SD interviews). The NFRs such as highly available, scalable, reliable and fault tolerant are included into the discussion without much thought.

If you say the system should be highly available, then be ready to answer:

• What does available mean?

• Why do you need system to be available?

• How much availability?

• What considerations you’ll take later to make sure the system is actually available?

This set of questions applies to every NFR you suggest. Most people can’t differentiate available and reliable, don’t be that person. If you say the application is fault tolerant, make a point during architecture discussion for what components are included (or decisions are taken) in the system ensuring fault tolerance.

Ok, moving on the next step once you’re all cleared up with the requirements; Scale.

Scale

Scale is again similar to NFR discussion. Most people include scale because they have seen a few videos or read some blog posts with mention of scale in system design interviews. What is scale and why does it matter?

Scale is one of the term, another popular term used is Back-of-the-envelope Estimation. In this step, you determine various aspects such as traffic numbers, network bandwidth, storage calculations, etc.

The numbers can’t be perfect and could be based on assumptions. The recommendation is clarify with the interviewer and what kind of scale we’re looking at? Scale matters because the system design depends on the scale numbers. The design varies by a magnitude if a system needs to support 1 million users vs 100 million users.

If you agreed on certain scale numbers in initial discussion, bring them up again during design decision making for why particular choice makes sense.

Expanding on the Facebook example from previous section, don’t just say you’ll build newsfeed and then discuss those two approaches of push vs pull published on every system design blog post. Clarity is very important, which approach works and which not depends on the scale.

Share System Design From Scratch

System requirements and scale discussion both are part of requirements gathering. Try to limit the discuss under 10 minutes for both of these steps. After scope clarity, we move on to actual design of the system.

Start with the Design

I generally start on my system design thinking from how customers access the system. What’s the first point of interactions for the customers? From there take a very high level picture of entire system (10000 ft) and then break into components as needed.

Don’t forget to check out my books with O’Reilly for building system design thought process.

• System Design on AWS

• System Design From Scratch (in progress)

and subscribe to YouTube @MsDeepSingh for detailed discussions.

Once you’ve a high level picture, how should you proceed further? The SD interview is a live discussion so there is no strict format. The interview can take different directions based on communication between interviewer and interviewee. Some of the pointers include:

• API design

• Monolithic setup or Microservices

• If Microservices, what all services and how the boundary is defined?

• Database choices and schema

• Networking protocols

• Design choices such as sync vs async

• Failure handling, etc.

Sometimes discussing algorithms is important such as how will you shorten the URL and other times the algorithms could be too low level detail. As I said, this is live discussion so there is no strict format.

Let’s take an example of another pitfall which I’ve seen during interviews. To resolve any scaling bottleneck, people right away suggest a new micro-service without clarifying how it actually solves the problem. For any new feature introduction, always ask yourself.

• Do you really need a new service?

• Can the existing service be extended?

Trade-off analysis is key for system design discussion. So there is no right answer or an answer with 100% upsides. The decision will depend on the requirements, scale and multiple other factors. So proper upsides and downsides for each approach helps to figure out which fits best for the given use case.

The next recommendation is to avoid memorising the solutions. Example, I’ve heard multiple times if it’s latency intensive system, you can’t use queues, it has to be synchronous system and vice versa. For all the readings you do on internet or watch videos, always try to think on your own, why particular decisions are being made. Recommendation is to read company blogs for insights on actual production systems.

A general discussion that comes up during interviews is database choices, let’s discuss it in more details.

Database Choice

Database choice discussion is among the popular ones in SD interviews. The low level schema is mostly deferred for high level design interviews but are important for low level design interviews. How will you choose a database for give problem statement?

The recommendation is to avoid phrases like,

• because there are 100 million customers or huge dataset, I’ll use non-relational database.

• because there is transactional data storage requirements, I’ll use relational databases.

The choice of database should be derived based on similar factors as how we gather requirements for overall systems. There is a more detailed blog on database choices, please read -

The choice of database is highly dependent on the query patterns. You store the data in such a way such that it’s retrieval is faster. Other factors include cost (operations and infrastructure), ACID compliance, previous expertise, etc. The discussion could further move to database scaling, so you should be prepared to answer accordingly.

Relational databases such as MySQL or PostgreSQL solves most of the use cases. If you don’t know which database to pick, start with relational databases.

System health is another important discussion which holds good value during the system design interviews. Let’s focus on system health next.

System Health

System health means system is working as per the expectations and there is no degradation. Here are a couple of things to consider.

• Failures and faults happen in live systems. This could be API failures or system in whole. The system should be able to handle these scenarios gracefully.

• What if certain components in the system break, ways for graceful degradation. It involves discussion around if the components are decoupled enough.

• Monitoring, logging and alerts are important to monitor the health of the system. The combination of three is often referred to as Observability. This helps you to figure out if something goes wrong so never skip this part.

This is all for today’s discussion. Let’s summarise.

Closing Thoughts…

Remember, no architecture is final or best architecture. Every decision in designing systems holds trade-offs. You should clearly specify why a particular decision makes sense for a specific scenario. Think before you answer to certain question, your capability to ask right questions matters a lot. If you’re asked to improved database performance, then don’t directly say I’ll use cache. First analyse why the bottleneck exist, introducing a cache might help or not.

If possible, take examples from your previous projects if any trade-off analysis matches the situation. Personal experience adds a lot of value. There are some lesser focused upon topics in SD interviews due to time constraints and you can consider depending on time and problem statements. The topics include security, application deployments and managing operational overhead (in addition to observability).

Here is the complimentary video on YouTube discussing “How to win System Design interviews” :

With all these TIPS, I’ll leave you with your preparation. Best of luck :) Apart from self-preparation, try mock interviews with your friends (or anyone) and analyse on what went wrong (and right).

For any questions, you can comment here or DM on LinkedIn.

Assume you’ve to build a payments application supporting multiple payment options such as UPI, credit card. Each of these payment option state is synced using callbacks and synchronous APIs with the payment providers.

Guideline of Isolation: Build it Modularly

The system should be built as modules with each module being independent on its own. This helps with maintainability and reusability of modules.

Example includes keeping independent modules for different payment methods and separate module for callback processing. You can keep the common stuff in common module which can be imported as needed.

Guideline of Simplicity: Keep it Simple, Silly

You should always avoid unnecessary components. Only build what’s really needed. Iterate over the implementation - always test and improve.

Examples include trying to support a lot of payment methods in the beginning or thinking of going global on day zero. This could create extra churn and gets the project delayed.

** The NOTES in this blog are taken from Chapter 01 of System Design on AWS. Explore the book:

📕 O’Reilly: https://oreil.ly/ruQbc

📕 Amazon.in: https://amzn.to/4jExkte

📕 Amazon.com: https://amzn.to/3D3BIBJ

Guideline of Performance: Metrics Don’t Lie

Metrics (observability) are really important. They helps in debugging as well as how well your system is doing.

Example metrics include time taken for payment processing and payment failure rates.

Guideline of Trade-offs: There is no such thing as Free Lunch

Every choice has trade-offs. There are pros and cons with every approach. Choose what makes sense for your business use case and to your product’s scale. Take inspiration from others but don’t follow blindly.

Example, thinking of independent micro-services from day one because you can scale them independently. Well yes, but who will manage this?

Guideline of Use-cases: It Always Depends

The design of system can depend on a lot of factors. Some of the factors include system requirements, user needs, expertise, budget and regulations.

Reiterating from previous point, choose what works for you. Here is summary image (generated with Grok AI)

Happy Building!

General Thoughts on Compute Selection

• If you’re working in a company, the selection of compute is generally influence by what’s being already being used in other projects/services. This is because the team already has experience managing certain technology. Introducing new services brings in learning curve and extra operational burden.

• For reference, the comparison of services is corresponding to AWS cloud but they hold across the cloud providers.

Let’s start the discussion, starting with virtual machines.

Elastic Compute Cloud (EC2)

There are two options with Amazon EC2. A shared tenancy or dedicated tenancy. The idea is you wish to buy the entire physical server or just a part of it. Most of needs are served with shared tenancy, don’t worry the customers workloads are properly isolated.

Once you decided on the tenancy (default is shared), the second choice you’ve to make is instance type. This depends on a variety of factors such as:

• Hardware and performance characteristics of an instance such as CPU, disk, memory and network.

• Type of workloads such as huge AI training workflows vs running a web application.

The third selection choice you’ve to make is billing model. There are three types, reserved, on-demand and spot instances. The default selection is an on-demand instance.

• Reserved: If you’re aware of scaling needs, you can pre-order the number of instances needed. The reserved instances are available at a discounted price.

• On-Demand: You spawn the instances when needed and terminate once done. This is default pricing model.

• Spot: For less critical workloads where uptime is not a hard requirement, spot instances could be chosen. You can bring up an instance but AWS can terminate it at anytime they wish.

The content of this blog is influenced by Chapter 11 of O’Reilly book System Design on AWS. Checkout the book at https://msdeepsingh.com/books/system-design-on-aws/.

Let’s move to second compute option, containers.

Containerisation Services

The major problem that occurs in development is “it works on my machine”. The development environment could vary from engineer to engineer, from local setup to production environment and so on. This problem is solved via containers.

Another problem addressed via containers is optimising the resource usage on EC2 machines.

You package the application binary with all the dependencies and the same container can run on any machine. You can deploy these container images and manage them on your own but it’s hard job. So there are multiple orchestration tools built on top to manages the containers at scale, such as Kubernetes.

AWS offers two container orchestration services, namely Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS).

Both of these services comes with two forms of deployment, either we can choose to deploy via EC2 or let AWS manage everything behind the scenes (serverless), called as Fargate. Let’s move on to next choice, Function-as-a-Service.

Serverless Functions

You just write functions and don’t worry about underlying infrastructure at all, everything is managed by the cloud provider. For example, AWS Lambda.

The code you write in functions can be invoked with the help of a trigger. For example, whenever there is new object uploaded to S3 bucket, read the object and perform some analysis.

Decision Framework

From earlier blog on this newsletter on choosing a database, the factors are applicable for compute selection as well.

The decision factors include:

• Solves the use case

• Previous Expertise

• Cost

• Battle-tested software

• Cloud Managed vs Self Managed (Operational Overhead)

• Time to Market

With all this information, let’s conclude our discussion.

Closing Remarks…

Use the below table as reference for selection of compute along with some decision factors discussed in previous section.

In summary:

• All compute options can work at high scale. Evaluate on what works for you. Do PoC if you’ve money and resources.

Iterate on design if you require more control over time.

• Example started with EKS Fargate

• Move to EKS EC2

• Migrate away from managed offering and deploy your own EKS cluster to manage containers EC2 instances.

Taking an example of Amazon Prime video monitoring architecture. The distributed architecture involved multiple AWS Lambda functions orchestrated via AWS Step Functions. This led to lot of operational overhead with extra dollars to spend. The team rearchitected the system by migrating to ECS. This saved cost plus network latency between the components of distributed system.

See you next week! Don’t forget to subscribe :)

Subscribe now

Problem Statement

Now think of a same scenario for a system or a component of a system. If a large number of requests arrives at once trying to acquire a resource, the system will be overloaded if not handled well. The sudden traffic spike can cause issues like high CPU utilisation, memory leaks, etc.

Thundering Herd is often used in-relation to cache and databases. For a cache-miss scenario, all the requests trying to access same item will be routed to the database. This overloads the database if database is not scaled to handle this load. This causes a cascading failure in the system bringing down the entire system.

Resolution

The resolution has two directions, one is mitigation (come out of situation if thundering herd occurs) and other is prevention (system is designed in a way so thundering herd doesn’t occur). Let’s explore both of these directions.

Thundering Herd Mitigation

It’s really hard to come out of Thundering Herd situation. This can cause cascading failures and bring the entire infrastructure down. The best is dropping the user requests. This could be for all the users or a percentage of users depending on the situation.

The database might have queued up all the requests, so restart the database. Same applies to other infra components.

Thundering Herd Prevention

The system can be designed in a way ensuring it can come out of these situations easily if they happen and enhancements ensuring this situation doesn’t happen.

Discard Requests

The first solution is load shedding. Serving some requests is always better than serving none of them. This is the idea behind load shedding because an overloaded system can cause cascading failures.

Here are few suggestions on discarding requests:

• Discard lower priority requests. This could be some back-office jobs that can be scheduled for later point of time.

• Discard % of user requests. If above doesn’t help, the system should drop some percentage of user requests.

• Keep a request buffer. The requests are put in a queue for certain time and served if capacity is available (or system recovers).

Watch the dive deep on Uber’s Load Shedding Framework QALM (QoS Load Management Framework) and Cinnamon architecture (Part I, Part II, Part III) for discussion on how these systems are built.

Jitter in Retries

If a request fails, what you do? Hit the refresh button. The retry could be user enforced (such as refresh) or system invoked on request failures. Retries are great to recover from transient failures but this can overload the system in case of thundering herd scenarios.

If the system is not able to process the requests, and all the users retried at the same time. This makes the situation even worse. Introducing Jitter means there is random lag between the retries ensuring not all requests arrive at the same time.

Exponential backoff in retries is only good if applied with Jitter else it doesn’t solve the problem it is supposed to. In summary, don’t retry at fixed intervals. Jitter is must, exponential backoff is good to have.

What’s Important Should Stay Alive?

Big software architecture has a lot of components (or services). Each component does some job and might depend on other components for part of the job. There are also some good have components. Take an example of video feed generated by Netflix, it’s customised for an user.

In overloaded system scenarios, it’s ok to reflect same video feed to all users and stop the work from recommendation system. This is good to have and not critical. The critical component is video play. The video should start once user clicks on play button.

The recommendation is clear division between the services. General criteria followed by many companies is tier strategy. Tier 1 service is most critical, Tier 2 is less critical and so on.

The chaos and stress testing should focus majorly on Tier 1 services ensuring system operations in unknown scenarios.

Scaling

Automated Scaling is one of suggestions that floats around to unburden the servers from this load. For thundering herd, it doesn’t help much. Bringing up a server takes some time (even with light-weight application binaries and low startup time).

Pre-scaling is one recommendation if you’re aware of some upcoming event and loosening up the auto-scaling thresholds so the system doesn’t wait for hit upper bounds can help.

Request Coalescing

We discussed thundering herd occurs if a huge number of requests are trying to access the same resource. Can we deploy a mechanism so as only one request is processed and others wait for for its results, see Figure 1.

Figure 1 Request Coalescing

All the requests are trying get(x), only one request actually fetch the result and rest of them use the same. This shields the backend systems, resolving thundering herd issue. At the caching layer, this problem is solved with a cache promise.

Cache Promise works on the similar concept, it stores a promise till the results are retried by the primary query and other requests wait for this promise. Once the promise is fulfilled, all requests are served.

Interested in practical implementation of these concepts, STAY TUNED. Something exciting is coming — https://msdeepsingh.com/books/

Further…

• How to avoid thundering Herd?

• Instagram Architecture: Solving Thundering Herd with Cache Promise

The primary motivation for writing a technical book should not be money. If it is, you should probably reconsider. Technical books have a highly niche audience. With the rise of AI tools, fewer people rely on books for technical answers.

Financial gain was never the motivation for me, so what inspired me to write “System Design on AWS”?

• Personal Growth: Taking a complex idea and translating it for beginners forces you to deepen your own understanding.

• The Communication Challenge: Having deep technical knowledge is common, but delivering it effectively is rare. Writing the book was a test of communication and documentation skills. I credit much of this to the writing culture at Amazon.

• Building Trust: Being an author with a respected publisher like O’Reilly acts as a powerful credential. Much like having a top-tier university or top company on your resume.

Is It Really Worth It?

Absolutely, but the ROI is in your network, not your bank account. As the saying goes, “Your network is your net worth.” Publishing the book opened doors to connect with industry leaders, CTOs, and senior engineers. I’ve been invited to multiple conferences to share my insights so it has been my way of giving back to the community. The connections allow you to learn from other leaders’ failures, successes, and decision-making processes.

Adding to above, it’s not like you’ll make nothing. The idea is not setting high expectations on monetary terms.

Project Initiation

Me and my co-author Jayanth Kumar took close to 2.5 years to finish the project. The details outlined as part of this blog are specific to O’Reilly but it’s similar with other technical publishers as well. I’ll divide the process into multiple phases from initial thought process to hard copies in my hands.

Proposal Submission

Starting out is always hard. Jayanth shared this idea with me and had some details already put up in the proposal. We brainstormed among ourselves on the structure of the book so as readers are hooked to the content throughout.

You can find all the details on the O’Reilly Work With Us page regarding different ways you can contribute.

The Proposal Details

Here is the high level description of what you need to fill in the proposal (make a copy of proposal from here).

• Book title and subtitle: The title of the book. For subtitles, you can also suggest multiple subtitles and decide later what works best.

• About the Author: This is basically an author biography and why do you feel you’re the best person to write the book on a particular topic. You can include your past work such as your industry experience, previous technical books/blogs/videos, speaking at conferences, etc.

• Marketing Description: This is something you find on the back cover of the book. You’re basically pitching to buy the book to a potential reader.

• About the topic: This section explains the book title in detail and why it’s important.

• Audience: Who will read this book?

• What the reader will learn: There is always some end goal if you’re selling a product. If the reader finishes the book, what kind of knowledge gain happens and how/where the reader will apply this knowledge.

• Competing titles: If there are similar titles in the market, how this book differs from them and what new values this book specifically offers.

• Book outline: Here you include a detailed chapter-wise table of contents.

• Schedule: How much time will you take to finish the first two chapters, half manuscript and full manuscript. Keep in mind that life happens and if you’re working on the book writing along with your full-time job then keep a buffer.

Balancing the content takes effort, what you go into the book and what should be skipped is the decision you’ve to make. Be ready to spend so much time on choosing topics to be included as part of the book and then again striking them off because you found other things which are more important. Once the draft was ready, we directly emailed one the Content Acquisition Editors from O’Reilly.

Proposal Submission Timeline

We were hopeful of the proposal approval as we firmly believed this is a unique book and solves a lot of gaps in the content already available in the market. Good things happen if you’re hopeful, and we received a positive response to the proposal. At this point, it’s not 100% approved but the acquisitions editor will work with you to modify some details as needed. For example:

• We had a lot of discussion if Part II of the book (AWS specific content) should be included or not. The point being made was there is already a lot of AWS documentation available on the internet, so what new value does the book bring? We finally agreed to keep it so readers can find everything at single place and they are able to follow along from Chapter 1 to Chapter 21. You don’t want to ask the reader to go read on internet about a specific AWS service and then come back to the book on how it’s used practically.

• We had 15 chapters in Part III of the book in the initial proposal which was trimmed down to 8 later. As I was saying earlier, you’ll have to balance the content.

TIP: We also wrote one chapter from Part III and shared after proposal submission on how we’re envisioning the content. I later scrapped it off and wrote it from scratch but if you can, it’s a good idea to have some content ready from the chapters.

After all this communications, the proposal was officially approved and we were presented with a contract to sign. Once signed, the actual work started.

Book Writing Process

You’re assigned a development editor to work with you throughout the book after contract signing. In the first meeting, the development editor shares details like:

• Writing tool: You can either write in Google Docs or O’Reilly has an in-house tool called Atlas.

• Development Schedule: You agree on a chapter-wise schedule on when the chapters will be ready.

• Writing Style: Details around illustrations, writing style, content structuring, etc.

Once you finish a chapter, you share the chapter with the development editor. The role of development editor is to share feedback on the first draft. You work on the shared feedback and it becomes a continuous loop for all the chapters.

After development review is done and feedback is addressed, the chapters are published to O’Reilly platform in early release form. The good thing is people can read chapters early and share reviews on the platform. It’s much similar launching a product MVP, gathering feedback and improving on it before the final launch of the product.

In parallel, the book chapters also goes through technical reviews. For technical reviews, O’Reilly has their own reviewers and you can also suggest from your network. The technical reviews are phased into generally 3 parts, first two chapters, then half manuscript and then full manuscript.

After development and technical review completion of the chapter, the next step is the production phase.

Book Production Phase

A production editor walks you through the book production timelines and different stages as part of this process. Here are few things taken care of as part of this phase:

• Copy Editing: A copy editor goes through the entire manuscript to make sure it matches the O’Reilly style guidelines. This stage generally doesn’t require too much input from your end. This is divided into two parts, QC1 and QC2. You can suggest anything as part of QC1 and that is addressed as part of QC2.

• Illustrations: In parallel to copy editing, an illustrator works through the images and recreates them as per O’Reilly style. You can work through the illustrations for any suggestions from your end.

• Index: An indexer is assigned to the project responsible to create the final index of the book.

The book is ready to go to print after all these steps as part of the production phase. The final version of the book is available within a week on O’Reilly platform and it generally takes 3-4 weeks for physical copies to be printed out.

Let’s talk a little about the book I wrote with O’Reilly - System Design on AWS.

System Design on AWS

System Design on AWS was written with a clear philosophy: to be useful both “during the interview and after the interview.” It is designed to take your knowledge beyond basic whiteboard theory so you can make effective long-term architectural decisions when building actual products.

Book Content

• Part 1 - Fundamentals: Covers core distributed building blocks like load balancers, caching layers, databases, network protocols, containerisation and general architecture patterns.

• Part 2 - AWS Services: Maps out the specific AWS services available across different operational domains (compute, databases, networking, event-driven, observability, big data, machine learning etc.).

• Part 3 - Practical Implementation: Focuses on real-world system design philosophy. It walks through building complex applications from scratch such as a stock broker platform or a hotel reservation system. From gathering initial requirements to deploying the actual AWS components. It emphasizes that drawing boxes on a whiteboard is not enough; you need to know how to actually run the system.

You can find everything about the book on https://msdeepsingh.com/books/

Get System Design on AWS

📕 O’Reilly: https://learning.oreilly.com/library/view/system-design-on/9781098146887/

📕 Amazon.in: https://amzn.to/4jExkte

📕 Amazon.com: https://amzn.to/3D3BIBJ

📕 Amazon.co.uk: https://amzn.to/3QW6Fvf

📕 Shroff Publishers (Indian Subcontinent): https://www.shroffpublishers.com/books/9789355428035/

Before closing, I want to share my opinions on the choice of publisher.

Self Publishing or Publishing House?

Self publishing means you’re already well-known in the industry and people will buy the book looking at your name as author. If you’re a first time author like me, you’ve to give some reason to readers so they can trust you. O’Reilly solved this problem for me. O’Reilly is a trusted name and they don’t publish random jargon, the book goes through a strict process so readers clear the first barrier of buying the book.

Also, we were not aware of the overall process. You’ve to spend some additional time on marketing, content reviews, copy editing, etc which are taken care of by O’Reilly in-house. Given we both had full time jobs, O’Reilly was the preferred choice and I’m very glad we pursued this project.

On a closing note, writing a technical book may not make you a millionaire, but it will sharpen your communication, expand your professional network and strengthen your domain expertise.

Best of Luck!!!

You can reach out to me on LinkedIn for specific queries.

Decision Flowchart

Let me take a reference from my O’Reilly book “System Design on AWS” Chapter 3. Figure 1 shows a flow chart on how can you navigate through the choice of database over multiple options available in the market.

Figure 1 Decision Flowchart for Database Choice

The question that comes to your mind, can I follow this flowchart blindly to answer database choice questions? The answer is NO. In Software Architecture and design decisions, there are always trade-offs with any direction you go with. So saying, a particular database is perfect is wrong. In addition to the above flowchart, database choice should be combination of below factors:

Business Requirements

The most important decision criteria is that the database choice should meet the requirements. All other things comes later.

You can start by looking at data access patterns. The databases such as DynamoDB only makes sense if you know the access patterns. If there so many joins required, and you don’t what kind of queries will be there, relational database is better choice. This can very much happen for a new project or a startup; relational database is good choice to start with.

Don’t forget to checkout the “System Design on AWS” book for detailed dive deep on relational, non-relational databases and how best to use them.

Everything about "System Design on AWS"

Previous Expertise

If you already have expertise with some technology then you know how to use it in the best way possible. If something meets the requirements and you also have previous working experience, nothing beats this combination.

In case you’re new to databases or your use case is totally new, try to stick with battle-tested software. If a lot of people have used something at scale already, you’ll get ton of support from community.

Cost

You should be able to afford something to use it. Cost is important consideration, this is mostly applicable to managed offerings. When it comes to cost, I recommend to look at total cost of operations and not just the monthly bill paid to the provider.

See,

• How much time you’re saving in launching your product by leveraging managed offering. This is also applicable to other solutions as well and not just databases.

• How much operational burden it takes off from your plate.

With this, let’s conclude the discussion.

Conclusion

Always consider trade-offs for any decision you make, what are the pros and cons for any choice being made. Also, there is thin line between database features these days. There are a lot of managed relational database offerings with horizontal scaling support so you don’t have to implement this in-house. There are a lot of non-relational database offerings with transactional support.

I’ll recommend to choose non-relational only if you know you need it. Else stick to relational databases, they solves most of the requirements. Happy Building!

Check out the YouTube video for the discussion with the community -

As I was going through the book, I created chapter-wise short notes and YouTube videos as a summary.

DDIA Chapter 1-12 Notes

70.1MB ∙ PDF file

Download

Chapter-wise summary of Designing Data-Intensive Applications book

Download

Watch on YouTube

Should you read the book?

An excerpt from the O’Reilly around usefulness of the book

• Peer under the hood of the systems you already use, and learn how to use and operate them more effectively

• Make informed decisions by identifying the strengths and weaknesses of different tools

• Navigate the trade-offs around consistency, scalability, fault tolerance, and complexity

• Understand the distributed systems research upon which modern databases are built

• Peek behind the scenes of major online services, and learn from their architectures

Where can I get a copy?

• amazon.com

• amazon.in

• O’Reilly online

• ScyllaDB sponsored chapters for free

Figure 1. Train booking application

There was something interesting about the project. There was no underlying database. The data was directly dumped to files, including login details, train details, booking details, etc. Example, train details for sun stored in sun.txt. Checkout code on Github - https://github.com/msdeep14/irctc

1 2 12956 jp-pune_exp 3:20PM 51 587
2 1 12957 jp-pune_exp 9:15AM 87 501
20 21 19876 goa_exp 5:00PM 54 709
3 2 10786 jhelum 5:25PM 112 654
1 9 11760 jp-mumbai-exp 2:00PM 67 543
1 2 34561 premium 11:00AM 45 1700

So I think I’m eligible to answer the question; why do we need database if you can store data in files?

Bottlenecks

If there is already classic file storage available, what’s the need for all kind of databases? Let’s take an quick example, you wish to get all the trains from Jaipur to Pune:

• The code snippet above shares trains availability as per source to destination. The beginning two numbers are keys mapped to locations.

• Essentially, you’ll pick the file for a day and find the train.

• What if there is no direct trains? It constructs a graph from file data and finds the trains matching to source and destination.

It’s works all great for college project because there is not much data but soon it will face problems:

• Increased latency

• Supporting different query patterns

Introducing Databases

At a very high level, databases offers structured way of accessing the data. Depending on the use case, the data is stored in a way that it’s retrieval is fast. There are different kind of databases depending on the requirements. Chapter 2, 3 & 10 of O’Reilly title “System Design on AWS” explores multiple databases with their architectures.

Here is a diagrammatic representation:

Figure 2 Decision Flowchart for Database selection

Let’s take an example from train ticket booking. Instead of storing the data in the files, if it’s stored in relational database such as PostgreSQL then it makes easier. For the query pattern, get all the trains from Jaipur to Pune. The point of consideration was how can we skip the reading all the records. The database indexes can be created location and time wise, reducing the data footprint.

Figure 2 offers some clarity on decisions can be made. Further, database choice is not just the use case, it’s combination of all below factors. This is also covered in YouTube channel video.

• Solves the use case (read/write access patterns)

• Previous Expertise with the database

• Cost

• Battle-tested software

• Cloud Managed vs Self Managed (Operational Overhead)

• Time to Market

Watch on YouTube

Subscribe now

The architecture evolves over time and the idea should always be we’re able to iterate over it.

• It’s nice when you get stuff right the first time but we don’t need to

• As long as you get the architecture as right as possible; the implementation stuff isn’t going to be big of a deal

This article captures the very early architecture of Facebook, how it started and challenges faced with scaling.

Initial Architecture

Facebook was launched in February 2004 with focus on connecting people for one school

Tech Stack

• Programming Language: PHP

• Database: MySQL

• Apache Web server

Figure 1 Early Facebook Architecture

The simple architecture served well for single school, but soon there were challenges with more interest and increasing scale.

Scaling Challenges

Let’s discuss few of the challenges faced due to architecture limitations:

• Privacy: Data and feed separation between the schools

• Traffic Distribution:

  • How the data should be distributed for different schools

  • Some schools have heavier usage as compared to others.

• Figure out friends connections with higher degree (friend-of-friend or friend-of-friend-of-friend)

Let’s discuss on the improvements in the architecture to handle the bottlenecks.

Improving Architecture

The key aspect is hiring more and more smart people and give them independence to work on the improvements.

Onboarding More Schools

The first thing that comes to mind for handling more traffic is adding more machines.

• We mentioned web server and MySQL coupled on same machine, so the idea is to decouple the storage from application server and introduce load balancing between the machines.

• This also removed single point of failure. Consistent database layer ensures traffic can be served from any operational machine.

Handling Friend-Of-Friend

If you wish to figure out friend of a person, it’s a direct query to database. For friend-of-friend:

• Retrieve friend list of a person

• Iterate over this list and get friend list of all the items

• This becomes n*n operation.

For friend-of-friend-of-friend, it becomes n*n*n operation; so it’s compute-intensive. The current architecture uses dedicated graph database, but back then:

• Split the database and create one instance of mysql database for each school in the network

• How does this help? Now you’ve to compute n*n*n for 10k (approximate number of people signed up from one school) instead of n*n*n over 6 million (all people across the schools)

The above is in addition to decoupling application servers and database compute. This helped a lot but further the traffic was reaching 100 Million pages/day, a query approximately takes 2-4 milliseconds. The problem lies in each page repeating on the same queries.

Improving Database Performance

What comes to mind to solves the same query problem? Cache the results to avoid hitting the database. Facebook introduced Memcached as caching layer; this brought down query performance to 0.3-0.5 milliseconds.

There were couple of issues though with the distributed architecture of memcached instances; If a memcached instances go down:

• There is no redundancy on information.

• Cache miss occurs; and all the traffic routes to database.

Facebook introduced photo sharing feature, let’s discuss on how it’s implemented.

Photo Storage

From early stage, Facebook chose to smaller machines instead of one big machine. The disk is relatively slow but stable, to overcome the slowness:

• High RAM Caching machines to hold most of the thumbnails and frequently accessed images.

The space consumed by photos is on higher side, some issues uncovered:

• If you wish to upload a photo album with 30 photos (3 MB each), then 90MB is total size; Networking issues with slow internet.

• Router used was built to support 1GB at a time.

Solution:

• Built Java applet and ActiveX control to couple the choosing of photos

• Compression at client side.

It helps:

• Upload photos quickly

• Resource optimisation - less server CPU utilisation

Even with caching machines, transferring images to users takes time. So the solution is to move caching near to users, aka, Content Delivery Networks (CDNs).

Conclusion

Follow Iterative Design Process; Remember you don’t have built it right (or perfect) the first time. Happy Building!

Subscribe now