Repository

https://github.com/evildido/dtube-community-support

Introduction

Dtube is a video plateform based on the steem blockchain that store videos on IPFS. Because most of users doesn't use ipfs. Dtube is a central point that will store content on IPFS node.

Dtube community support is a bot that will "listen" for new dtube content for a specific steem tag and store the video on a local IPFS node. Plus it stores metadata about dtube content like title (steem post) / video size / permlink / author. The main goal is to persist dtube content of your favorite community and or authors on the IPFS network.

DCS will retrieve 480p videos. if not available, it will retrieve the source file.

New Features

Fetch the blockchain block by block

As previously said in my last contribution, I have modified the streamOps function by fetching blocks instead of using streamOperation of steem-js.

In the previous version, streamOps() was implemented in index.js. This function had two roles :

• stream the blockchain
• search dtube content

To enhance visibility, I have splited it in two functions :

• stream the blockchain : catchup() in utils/utils.js
• search for dtube content and pin it on the ipfs node : streamOps() in actions/stream.js

Stream

catchup() uses LightRPC to fetch blocks.

LightRPC is a minimal library for interacting with JSON RPC.

Take a quick look on the code (complete code : utils/utils.js#L89)

function catchup(blockNumber) {
  lightrpc.sendAsync('get_ops_in_block', [blockNumber, false]).then(ops => {
    if (!ops.length) {
        // Blocks does not exists or is empty
        // If does not exist -> restart catchup() with the same block number
        // If exists -> restart() with blockup number + 1
    } else {
      // Block exists and not empty
      // Save blocknumber in DB (to resume)
        saveBlockState(blockNumber);
      // Process the block(operations) by calling streamOps() 
      stream.streamOps(ops);
    // retart the catchup() with the next block
      return catchup(blockNumber + 1);
    }
  }).catch(err => {
   // try another node and restart the function
  });
};

If a block is not empty, catchup() calls streamOps().

Search dtube content and pin it

streamOps() the file has not been modified except for quick fix (will be talked later) but streamOperations() has been removed.

Replay the blockchain

As we previously saw, the catchup() function takes block number in argument. I’ve added the possibility to start the bot by specified the first block to fetch instead of the last one.

The bot is able to replay partially the blockchain.

To do that, we need to specify block number by using '-b' argument.

npm start -- b=21803547

The following script point on start() in index.js.

Resume the bot

In the same logic, we are able to save the last block number so we can restart the bot to the last block.

When starting the bot (npm start), there are three cases :

• No block number specified in argument and no block number saved : The bot will fetch the block number with get_dynamic_global_properties
• No block number specified in argument but a block number has been saved : The bot starts from the last block saved
• Block is specified in argument : The blot starts from this block

Take a quick look

function start() {
  if(args.blockNumber!=undefined) {
    // A block number passed thought argument
    utils.catchup(Number(args.blockNumber));
  }
  else
  {
    var state = db.getAsync("block_state");
    state.then(result=> {
      // Start from the last block number stored in JFS DB
      utils.catchup(Number(result.blockNumber));
    }).catch(err => {
      // Start to the last block
      lightrpc.sendAsync('get_dynamic_global_properties', []).then(result => {
        utils.catchup(result.head_block_number);
      }).catch(err => {
        // retry with another node
    failover();
        return start();
       })
      });
    });
  }
}

Before stopped the bot

And after restart, the bot resumes from the last processed block.

Bug fix

Disable logs archiving

Because an issue on Winstan-dail-rotate (issue 125), old zipped log file cannot be deleted during the rotation. I've disabled this options (commit 6d1f5b8 and fea7e32).

fix "check if enought space"

In the last relase, I've added a test to ckeck if the IPFS repository has enough space before pinning a new content. This test consists by comparing content size and repository usage PLUS content size being pinned.

'size_tmp' is used to handle content being pinned. In the release, this variable was not properly updated. Commit 0855211 fix that.

Retrieve created date

Dtube Community Support stores dtube content metadata. "Created" stores the publication date.
By using streamOperations, the publication date given by the system (date()).

Because the bot doesn't use StreamOperations anymore and can replay the blockchain, I've added the commit 21953d2.

The commit consists to fetch the the created date by using steem.api.getContent().

Next step

With the ability to replay the blockchain, the bot t is facing another problem. If we replays too faar, we can try to pin content which has no seed. In that case, the bot continues to run but gets stuck with instances of streamOps.

I have asked here how to handle this point. dht.findprovs(hash) seems to do the job. findprovs finds peer for a specific pinset (hash).

I've tried it on my a ipfs node (go-ipfs).

With an old dtube content (no seed).

With a recent dtube content.

As we can see, findprovs() should do the job. The next step will be to use it for automatic and manual pinning.

Proof of Work Done

https://github.com/evildido

Pull Request

https://github.com/evildido/dtube-community-support/pull/17