How to Build a Scalable Live Streaming Interactive Service - Part II

2022-03-27

Background

Because of the networking improvement and the influent of the COVID-19, live streaming has become the hottest technology on the Internet, again.

In the past article we have discussed signal modeling and interactive modeling, this time we will discuss the scale methods of interactive service.

Limitations

As we all know, there are simply two kinds of services in the backend, called stateless service and stateful service.

Stateless service means that the service processes requests based only on the information relayed with each request and doesn’t rely on information from earlier requests.

Stateful service means that the service processes requests based on the information relayed with each request and information stored from earlier requests.

The biggest difference between stateless service and stateful service is that stateless service can route requests to different instances easily and stateful service can’t.

The stateless services are easy to scale so that the bottleneck of the entire system commonly comes from the stateful parts.

The interactive service of live streaming is composed of a few parts as I mention below.

Storages
- Relational Databases
Caches
- Structured Collections
- K-V Caches
Calculating Services
- Scheduled Tasks
Paying Service
- Account Transporting
HTTP Servers
Signaling Servers

And today the scaling methods of interactive service are aiming at these parts, too.

Scaling Methods

The distributed systems are often constructed by several small instances, we can increase the capacity and performance by scaling the instance up. But before scaling it up we have to make our system scalable.

Relational Databases

As an internet service, we always use a relational database for data storage. We can split our data into different parts so that we can store them in different databases so that we can scale our database up.

We can treat our live streaming databases in this way, too. But the data of live streaming has some special characteristics.

Timeliness. For now, the longest duration of live streaming is less than 480 hours, the average duration is less than 5 minutes. Mostly we only care about the ongoing live streaming rooms, so we can just store the recent 4 weeks’ records for online service.
Geographically. People would like to consume the live streaming related to them. So we can simply split the live streaming records by the region of the anchor.

Firstly, we generate the live streaming room id according to the anchor id, which is generated by the region where they signup their accounts.

And then we insert the record into the database which the live streaming room id belongs to.

When we are querying the database by ongoing live streaming room id, we can route the request to the actual database by the id.

When we are querying the database by offline live streaming room id, we can just route it into archiving database which is append-only and can be easily extended by adding nodes.

Structured Collections

We use Redis to implement the structured collections, so the method to scale the structured collections is to scale the Redis servers.

We never use Redis Cluster technology as our Redis Cluster solution. Conversely, we just use twemproxy to reroute the write and query. So the Redis Cluster we talk about below is not as same as the official Redis Cluster but the cluster constructed by several Twemproxy, Redis-Master, Redis-Slave, and Redis-Sentinal.

On the other hand, we often build two same Redis Cluster in different AZ(Available Zone) and assign one of them as the main cluster by config.

The write operation of the main cluster would replica to the secondary cluster by Kafka Message.

In the image above, We can notice that the Twemproxy is a stateless service so it’s scalable and will not be the bottleneck of the Redis Cluster.

In fact, the key of Redis storage is often constructed by the room id, after scaling the Redis servers up, all we have to do is prevent building big-key and hot-key of Redis.

Big-key often comes from the write operation of sorted set keys and hash keys. In the scenario of live streaming, there are two effective methods to avoid big-key.

Trim the collection to a fixed length when the collection is bigger than the threshold;
Limit the writing speed by rate-limiter or some other method.

Hot-Key often comes from the read operation, in order to prevent it, we can use these ways below.

Use another cache to store the result temporarily. For example, we can build a local cache in our service and this local cache will serve all the requests before expired;
Storage the key redundantly and choose a key randomly to read. In the general Redis Cluster solution, the different keys will locate in a different Redis-Server, so that all the requests will not be routed in a certain Redis-Server, too.

Also, in the social platform, detecting hotspots and using a separate policy to limit writing and reading rate is a common way to be scalable.

K-V Caches

We use Memcached as our K-V Caches. Which “K-V” of K-V Caches always comes from the databases.

Different from Redis, we use no proxy in Memcached. All the read is routed by the client.

There are no master and slave in Memcached, so we built several clusters for the same usage. All the clusters are equal.

When the client tries to read a cache from Memcached, it will have a few steps as below.

Choose a cluster of its AZ randomly, and then calculate the hash number and route to a Memcached Node by this hash number;
Try to read this Memcached Node, return immediately if success;
Choose another cluster of its AZ randomly, and route to another Memcached Node by another hash number;
Try to read this Memcached Node. Firstly write back to the Memcached Node in step.2 and then return;
Call a service named DbReader to read the database and then write back to the Memcached Nodes of step.4 and step.2.

One more thing worth mentioning is that the hash salts of different Memcached Cluster are different so that the same index Memacached Node of different Memcached Cluster will not store the values of the same keys.

In this way cache, we will have the highest availability.

If one Memcached node is down, no cached will be lost because clients will read the lost keys from another Memcached Cluster and write back;
If two Memcached nodes of different Memcached Clusters are down, only 1/M * 1/N data will be lost. (M, N means the nodes count of the Memcached Clusters)

Scheduled Tasks

Processing some business periodically is a very common live streaming backend. Most of the process tasks do have these properties.

They should know which rooms are ongoing;
They would do the logic separate by the live streaming room.

As we know, the ongoing live streaming rooms are stored in the online databases, we can only query the ongoing live streaming room ids by scanning these tables, it would be an extravagant operation, especially for lots of businesses and processes scan concurrently.

We can use a service named “OngoingQuery” to protect the databases.

The OngoingQuery service will store all the room ids of ongoing live streaming.

They scan the databases periodically to update the entire cache of themselves, and they the binlog of databases for instantly update. In this way, the cache of the OngoingQuery service will be the same as the data of the database eventually.

When we deploy a number of shard tasks for processing some business data, the shard tasks will register themselves to ZooKeeper at first, and then run periodically to do these things.

Query the ZooKeeper to know how many process instances at this moment;
Find out the index itself of all the process instances;
Request Ongoing Query service to get the part of the partition result;
Process the business of the part.

With these partition abstractions, we can separate the process task into several process instances and they can run concurrently.

Account Transporting

Almost every live streaming platform will support gift features.

The key content of the live streaming gift is account transporting. When an audience tries to pay a gift for the anchor, there will be an account transporting between the account of the audience and the account of the anchor.

As we know, platforms often store the balance of different users in different databases for scalability. So the account transporting in live streaming will always be a distributed transaction, which is very expensive.

We can use a virtual account to improve the transporting performance.

When the live streaming room begins, we will create a virtual account for this live streaming room in each database of user balance.

The gifting operation during the live streaming will be transportation between the audience account and the virtual account of the database which the audience account belongs to.

When the live stream room end, the service named “Settler” will collect all of the virtual accounts and do the distributed transaction between virtual accounts and the anchor account.

HTTP Service

There are lots of load-balancing strategies for HTTP services, so I will not talk about this topic a lot.

The only thing I want to mention is that we can use a route policy in Nginx, for routing the same room id to the same group of service nodes, which would help us to increase the hit rate of local-cache.

Signaling Servers

We have talked about this in the How to Build a Scalable Live Streaming Interactive Service - Part I.

Summary

In this article, we have discussed the scale strategies of the stateful services of live streaming interactive services. Some people also called these strategies “sharding strategy”.

All of these things have a core concept commonly – “Split the big thing into small things”, oh we have learned it in our university, isn’t it?

Next time I will share my thoughts on building a multi-region or cross-region live streaming platform, hope you will like it.

A place to talk trash

Backend