What is this ?

I am trying probably something new -- sharing initial versions of the documents / notes on the projects I am working on. The reasoning is,

1. sometimes the ideas are good enough to be shared
2. Many times we need inputs from people who have already done it to make the design better
3. Obviously it will also act like a advertisement to attract quality people :-)

So this series of "thoughts" are open to criticisms, suggestions, questions and Criticisms are more important!

Streaming the blockchain and ingestion

To provide the middle-ware its important that we have a near-real time copy of the data from the blockchain in a database. This feature will also provide the foundations for establishing a snap shot of the database and thus reducing the RAM needs.

The conceptual architecture

---

From the blockchain we will streaming the data using the RPC (gRPC etc can be considered later). As the first step this will be stored in a in-memory cache (Redis) for the further processing. This in-memory caching is used to provide real time capture of data and if required expose the data to other "north bound" DApps in "near real time".

[]-[]-[]-[]-[] ------> * (stream logic) -----> mDB

In the step2, in-memory DB data will be stored to MongoDB.

• We will consider the blockchain as the producer and the streaming logic as the consumer

• create two thred pools one consuming the data from the genesis block and the other consuming the data from the last irriversible block in the chain

• The thread / worker pools can be implemented in Java or in Python (though we will have to fight the GIL limitations)

• Since the data is not going to be the order to Tera bytes like in a typical Big data scenario, we will not be using a Message Queue / AMQT & considering this in a later stage.

• The project also has wallet, historical information etc and these will be taken from the micro services APIs exposed using the streaming + ingestion component

• real time data from the blockchain is not the scope of this project

• In the context of STEEM blockchain SBDS and chainsync are two interesting projects with similar goals

• STEEM blockchain also has many MongoBD based (third party) data providers

Databases

For now we will be proceeding with MongoDB. In the future multiple sinks like Elastic Search or RethinkDB will be selected based on the requirements

Caching

We will soon have to provide in memory caching and as of now Redis is selected for the purpose.