The WHY of Databases?
Why do I need a Database if I can directly store data in files?
Back in 2nd year of my college, I built a train-ticket booking application (somewhat similar to IRCTC in India). The application was built in C++ with Qt framework. Not sure if you heard about this framework but it’s good and get’s the job done. The below image is the landing page:
Figure 1. Train booking application
There was something interesting about the project. There was no underlying database. The data was directly dumped to files, including login details, train details, booking details, etc. Example, train details for sun stored in sun.txt. Checkout code on Github - https://github.com/msdeep14/irctc
1 2 12956 jp-pune_exp 3:20PM 51 587
2 1 12957 jp-pune_exp 9:15AM 87 501
20 21 19876 goa_exp 5:00PM 54 709
3 2 10786 jhelum 5:25PM 112 654
1 9 11760 jp-mumbai-exp 2:00PM 67 543
1 2 34561 premium 11:00AM 45 1700So I think I’m eligible to answer the question; why do we need database if you can store data in files?
Bottlenecks
If there is already classic file storage available, what’s the need for all kind of databases? Let’s take an quick example, you wish to get all the trains from Jaipur to Pune:
The code snippet above shares trains availability as per source to destination. The beginning two numbers are keys mapped to locations.
Essentially, you’ll pick the file for a day and find the train.
What if there is no direct trains? It constructs a graph from file data and finds the trains matching to source and destination.
It’s works all great for college project because there is not much data but soon it will face problems:
Increased latency
Supporting different query patterns
Introducing Databases
At a very high level, databases offers structured way of accessing the data. Depending on the use case, the data is stored in a way that it’s retrieval is fast. There are different kind of databases depending on the requirements. Chapter 2, 3 & 10 of O’Reilly title “System Design on AWS” explores multiple databases with their architectures.
Here is a diagrammatic representation:
Figure 2 Decision Flowchart for Database selection
Let’s take an example from train ticket booking. Instead of storing the data in the files, if it’s stored in relational database such as PostgreSQL then it makes easier. For the query pattern, get all the trains from Jaipur to Pune. The point of consideration was how can we skip the reading all the records. The database indexes can be created location and time wise, reducing the data footprint.
Figure 2 offers some clarity on decisions can be made. Further, database choice is not just the use case, it’s combination of all below factors. This is also covered in YouTube channel video.
Solves the use case (read/write access patterns)
Previous Expertise with the database
Cost
Battle-tested software
Cloud Managed vs Self Managed (Operational Overhead)
Time to Market