What’s the first thing that comes to mind while thinking of designing a chat application or live streaming application? WebSockets.
We talk about one such architecture today built with WebSockets to meet the near-real time communication needs. The architecture is from Recall.AI, offering meeting bots. You can integrate the recall.ai with meeting softwares such as Zoom or Google Meet. It joins as a meeting bot and offers details such as meeting summary, transcript, meeting recording, etc.
The article heading may sound fancy, but to prove ‘WebSockets are costly’, we’ll dig deeper into the existing architecture and then look for optimizations.
Architecture with WebSockets
The current architecture involves a video processing pipeline responsible for serving all the features.
Video processing on CPU.
A bot requires 4 CPU cores to run efficiently.
This is how it works. Figure 1 shows the design.
Headless chromium joins the meeting, aka meeting bot.
Real time video processing over websocket connections.
Media ingestion.
The cost of managing the system increased by huge margin with increasing scale and is not sustainable for growth. So there is a requirement to optimise the system architecture.
New Requirements
The new requirements are aligned towards reducing the costs.
Reduce AWS cloud bill atleast by 50%.
Reduce from 4 CPU cores to 2.
Chapter 6 and Chapter 9 from my O’Reilly book ‘System Design on AWS’ captures in-depth knowledge on networking protocols and concepts. Chapter 20 digs into design of video processing pipeline including concepts such as encoding. Check out the book for end-to-end understanding and detailed trade-off analysis.
Since the reducing cost revolve around optimising workloads on CPU, let’s understand what CPU is actually doing.
What CPU is Doing?
Video processing is computationally heavy and you can very well assume that the video encoding and decoding tasks consumes the resources. So to verify this assumption, we ran profiling on the meeting bots.
The observation shows majority of CPU time being spent on two below functions.
__memmove_avx_unaligned_erms__memcpy_avx_unaligned_erms
Both of these methods are used for copying memory blocks. _avx_unaligned_erms suffix means optimised for specific processor. The methods are called from.
Python WebSockets to receive the data.
Chromium WebSocket implementation to send the data.
Instead of video encoding and decoding, it’s looks like the processing time is spent by WebSockets data sent/receive. Let’s take a closer look on WebSockets.




