HOQU TECH DIGEST #5. Handling decentralized data in a fast way

Anyone who has ever developed a decentralized app using blockchain technology knows that the latency of all communications is so high that it is almost impossible to build a fast and responsive application. The fact that any changes in the blockchain can happen in a minute or even in 10 minutes contradicts the idea of having smooth and progressive UI applications. To overcome this limitation, we have to consider the blockchain as a secondary decentralized storage providing transparency and truthfulness with simultaneous usage of another kind of storage with fast access to data and responding with a moderate latency.

In this article, the HOQU development team would like to share its approach to addressing this problem.

First things first. Let us see how the client side applications access the blockchain. The back-end of the application can directly connect to the blockchain RPC. In that case, the back-end developers have to implement the logic of storing data in hot storage (a fast and reliable database) and maintain the sync logic with the blockchain. Moreover, if multiple applications are created, this behavior should be implemented on all of them. This creates an unnecessary overhead to the development team and raises the initial knowledge level to start developing the application.

On the other hand, each back-end application should have access credentials to write data to the blockchain to access the blockchain smart contracts. This creates an additional security breach without any control of access and tracing of the data flow. From the future perspective, it can become a huge security problem, especially considering the fact that one cannot simply change the blockchain access credentials (you have to create a new account and redeploy all the smart contracts).

Taking into consideration all of the above, the idea of creating a middleware service between the blockchain and all of the back-end applications seems very good. This service can handle all the requests to the blockchain, store the data on hot storage and keep the blockchain credentials safe and secure while providing a convenient REST JSON API to other applications. The HOQU development team took this approach from the beginning and improved it over time.

Let us have a closer look at the next picture.

Decentralized storage scheme

Here we have a number of blockchain API instances, let us name them dAPI. Each instance has a connection to the blockchain RPC and uses it as a secondary decentralized storage to achieve transparency and consensus. Along with the blockchain, the dAPI instances have their own hot storages that are used to work with data in a fast way. Technically speaking, each dAPI service has a separate stack of storages (linked list of the storages) for each package. The list of available storages and its order defined in the config of a particular dAPI package. Thus, we can define the use of Redis, then MySQL, and then Ethereum blockchain storages for the lead package and define Postgres, then the Ethereum blockchain for user package. Moreover, one instance of dAPI can use Memcache storage as a hot memory cache, while another instance of dAPI can use Redis storage for the same purpose.

During the request to the particular package of a dAPI, it calls the stack of storages by its order and first storage, which can resolve the request return the response. Thus, if particular dAPI instance has a data in its memory storage, it will fetch it extremely quickly — in 3ms. If the data is not available in the memory storage but present in the shared MySql database, then it will fetch it from the database. And only if the data is not available in two previous storages, it will be fetched from the blockchain.

As far as dAPI applications are written in the go language, it is possible to respond to the customer and continue to work on the request. It means that after the data was fetched from the blockchain, it will be stored back to the fast storages including Memcache, Redis, MySql and so on. The next time someone requests the same data, it will be fetched from the fast storage and returned extremely fast.

The challenging part here is the data synchronisation. Imagine the case when one instance of the dAPI store some data locally, then created the transaction to store this data to the blockchain. While this blockchain transaction is not mined only that particular instance of dAPI will know about a new portion of the data and only that particular dAPI can reply with the new data. But while this data is not mined in the blockchain, it has an unconfirmed status. During mining, this transaction can be failed, and in that case, the data in the hot and fast storages will be marked as failed as well. Only when the data is mined, the blockchain smart contracts create an appropriate event signaling that a new portion of data is stored. It will dispatch an action on all instances of dAPI and this new portion of the data will be fetched from the blockchain and stored locally in the fast storages of all dAPI. Since there any instance of dAPI, it can respond with a new data very quickly (as long as it was fetched on the smart contract event).

As we can see, this approach provides simple and confident access to blockchain data in a fast and reliable way. The front-end applications are not restricted anymore by the slow processing of the data in the blockchain.

Moreover, the dAPI service provides another useful feature — logging and tracing all of the requests. By using OpenTracing as a basis, we can trace the requests through multiple boundaries of the services. The trace span can be started even on the front-end application, then flow through the back-end application to the dAPI service. Then it handles the trace flow until the end of each storage used in the processing. On the example below, you can see the trace created during fetching the offer data.

The trace of the first request

As we can see on the initial fetch, there was an attempt to fetch the data from fast storage (MySql in that package). The data was not found and the request to the blockchain was performed. After the blockchain returned the data, it was stored to the local fast storage. Next time, the same data will be fetched and the return can be done in 1.14ms (instead of 482ms from the blockchain).

HOQU platform development is currently in its active stage. We are constantly improving the processes within the system and trying to find proper solutions to using the best practices. We appreciate your support and are delighted to answer your questions. Stay tuned for more news and updates.

P.S. The source code of the dAPI can be found on our GitHub.

Regards,

The HOQU team.