@ro-witness - Witness Update 10th of July '18

ro-witness (57)in #witness • 6 years ago

Hello, fellow Steemians,

here is the weekly witness update from @ro-witness. You can read about us in our introduction post.

Weekly Freeroll on Lucksacks Poker League

This weeks winner was @hopfarmnc. Congratulations on the win!

Visit https://www.lucksacks.com and follow @tuck-fheman and @spl for the poker-related news.

Weekly Delegation Project

The winner from last week was @spl, where the community has shown again that it is one of the best on Steem.

Technical Update

Monitoring your witness is of paramount importance. We talked in the post from two weeks ago about a method to send an SMS with various system parameters, in case of failure, through a 3G modem, to the backup witness server which should react to the failure messages and start the broadcasting of keys.

As our servers are bare-metal, privately held locations, we need a way to offset the advantages of hosting them in a dedicated data center, thus creating a fail-safe mechanism which would react in time in case of:

power outage longer than 45 minutes - which for our APC UPSes is the maximum at a standard load of 45% of its 2kW battery.
sudden spike in local load.
ISP related failures.

Power outage - monitor UPS - issue commands

For monitoring the UPS status, I use nut. I won't go into the details of installing the package, as it is pretty straightforward. So I will use the AT commands in upssched.conf to issue, based on UPS status, various actions, steps to be taken.

One example of upssched.conf would be:

CMDSCRIPT /bin/witnessmon.sh AT ONBATT * EXECUTE upsonbatt AT ONBATT * START-TIMER disableandupdate 1800 AT ONLINE * EXECUTE upsonline AT ONLINE * CANCEL-TIMER disableandupdate

Now, the first line from this configuration of upssched.conf use the AT command CMDSCRIPT to run a private script which has 2 variables wrapped in a case logic. Then, it continues with AT commands based on UPS status ONBATT or ONLINE to EXECUTE commands from the scrips. The command of disableanduptate uses a timer, as I don't want to rush to disable the witness, sends SMS to the backup and update keys if the power mainlines resumes in 30 minutes. When power resumes, if <30 minutes since the ONBATT status, the script will be aborted.

Now for the script part: nano witnessmon.sh and paste the following and adapt to your own coordinates - email and phone number of the receiving 3G modem attached to the backup witness server.

#!/bin/bash case $1 in upsonbatt) mail -s "UPS on battery power" [email protected] echo "Primary UPS on battery" | gsmsendsms -d /dev/ttyUSB0 -b 19200 sysadminphonenumber ;; upsonline) mail -s "UPS on line power" [email protected] echo "Primary UPS online" | gsmsendsms -d /dev/ttyUSB0 -b 19200 sysadminphonenumber ;; disableandupdate ) ./pathtorunsh/run.sh stop echo "Primary down" | gsmsendsms -d /dev/ttyUSB0 -b 19200 backupsrvnumber;sysadminphonenumber mail -s "Primary disabled" [email protected] ;; esac

So, in case of a power outage, that goes longer than 30 minutes, our primary witness server is disabled and our backup witness server is enabled.

Next week we will be covering other situations and parameters to monitor, and also how the backup witness server will interpret these messages received by SMS.

Maintenance

We will run a maintenance tomorrow from 09:00 am UTC, so if you see the server disconnected tomorrow is because of this.

During last weeks downtime, we lost only 1 block and managed to be back fats online with the version 0.19.5

Rank

At this moment of writing, we are on position 78 in the witness ranking. 74 for the active ones. We maintained our position in the active ones and have lost one in general. We produce at the moment 6 blocks a day.

Thank you for the support given and helping us reach this spot on the witness ranking.