Developing Steemfiles : Journal Entry for 2018 07(July) 09
July 9th, 2018
In this developer's journal of Steemfiles:
- Validation from Javascript transactions still an enigma in Python code
- Public Key Cacheing in Steemfiles
- Master passwords converted into Private Posting Keys
- Recovering a Public Key
Validation from Javascript transactions still an enigma in Python code
Javascript signed transactions when sent to the server fail to validate with any Python library. This is strange to me because I had used code like this in the past and it is Javascript that is broadcast to the blockchain and we know that works. Somehow the Python generated transactions validate in Python and also do most of the validated blockchain transactions. Is it possible I am creating the transactions wrong in Javascript? Or are the python generated signatures somehow specific to the verification algorithm? It seems to me the former seems more likely.
The CGI script doesn't keep the key. Now, you only really have the word of those running Steemit or SteemConnect because after all the javascript is obfuscated in those sites.
wget https://www.steemit.com
mv index.html index.html.gz
gzip -d index.html.gz || mv index.html.gz index.html
wc index.html
The word count program in UNIX reveals the whole webpage is one line. The source code is certianly not meant to be readable by humans.
I downloaded one of the two linked javascript files from steemit.com in the page and found it didn't contain any new line characters at all but was over a million bytes long
Public Key caching in Steemfiles
Steemfiles [https://test.steemfiles.com](test site) now caches the public keys. So, if the blockchain freezes or no public remote procedure call (RPC) nodes are available for any reason, you will still be able to login. Whenever possible, keys are fetched from the RPC nodes in order to make sure your password changes will be registered with Steemfiles.
Master password converts to Private Posting Key
You cannot convert master or active keys into lower powered keys because of the way these keys are normally derived but you can convert a master password into any key. So now in Steemfiles a master password entered will be converted into a private posting key before being received by the server. This does make things easier to the user, whose password might be the only thing he or she has. It also makes things easier on the server, because it doesn't need to convert a password into a key. It's a win-win situation.
Public Key recovery implemented in Beem
I posted serveral times asking how to recover a public key. The routine is low level so the required arguments are not like most of Beem and the returned type is not an address format at all but a lower level representation.
Note: Just because you can call this and get a public address returned doesn't mean the signature is valid. You should verify the signature after. It should save a great deal of time from comparing many different keys.
import beembase.signedtransactions
import beemgraphenebase.account
SECP256K1_MODULE = None
SECP256K1_AVAILABLE = False
CRYPTOGRAPHY_AVAILABLE = False
GMPY2_MODULE = False
if not SECP256K1_MODULE:
try:
import secp256k1
SECP256K1_MODULE = "secp256k1"
SECP256K1_AVAILABLE = True
except ImportError:
try:
import cryptography
SECP256K1_MODULE = "cryptography"
CRYPTOGRAPHY_AVAILABLE = True
except ImportError:
SECP256K1_MODULE = "ecdsa"
try:
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.asymmetric.utils \
import decode_dss_signature, encode_dss_signature
from cryptography.exceptions import InvalidSignature
CRYPTOGRAPHY_AVAILABLE = True
except ImportError:
CRYPTOGRAPHY_AVAILABLE = False
log.debug("Cryptography not available")
def filter_transaction(tx):
for key in ['transaction_id', 'trx_in_block', 'block_num', 'transaction_num']:
try:
del tx[key]
except KeyError:
pass
return tx
def recoverPublicKey(tx, network_name):
import beembase.signedtransactions
import beemgraphenebase.ecdsasig
import binascii
import beemgraphenebase.account
tx = filter_transaction(tx)
txs = beembase.signedtransactions.Signed_Transaction(**tx)
txs.deriveDigest(network_name)
i = binascii.unhexlify(tx['signatures'][0][:2])[0] - 31
signature = binascii.unhexlify(tx['signatures'][0][2:])
if SECP256K1_MODULE == "secp256k1":
sig = pubkey.ecdsa_recoverable_deserialize(signature, i)
p = secp256k1.PublicKey(pubkey.ecdsa_recover(txs.message, sig))
p_comp = p.serialize()
else:
try:
p = beemgraphenebase.ecdsasig.recover_public_key(txs.digest, signature, i, txs.message)
except:
p = beemgraphenebase.ecdsasig.recover_public_key(txs.digest, signature, i)
p_comp = beemgraphenebase.ecdsasig.compressedPubkey(p)
beemPublicKey = beemgraphenebase.account.PublicKey(binascii.hexlify(p_comp).decode("ascii"))
# Use format(beemPublicKey, address_prefix) to get the address
return beemPublicKey
However, this works well enough for some cases but it has to work for all cases.
Next week, I'll reveal the bug in the code above and give the next implementation. I will be going away for a week you know and I will be away from the keyboard.
Stream TX : Stream your transactions in Python
I wrote this SteemTx.py program about a year ago, from the piston lib examples, into steem-python and now it uses Beem. What it does is it streams transactions from the blockchain in Python. Now, I decided I would add a validation step -- You know? Verify all of those transactions off the blockchain.
Here is Streamtx: It streams all the transactions as it comes in.
Use: python3 streamtx.py -m (run on mainnet)
Use: python3 streamtx.py -t (run on testnet (very quiet))
Use: python3 streamtx.py -mV (runs on mainnet and validates the transactions)
from beem.blockchain import Blockchain
import mysql.connector as mc
from simple_config import SimpleConfig
from getpass import getpass
import mysql.connector.errors
import sys, traceback
from optparse import OptionParser
import beem.steem
import beemapi.exceptions
import beembase.signedtransactions
import time
import datetime
config = SimpleConfig('streamtx', {})
db_password = config.get('db_password')
if db_password is None:
db_password = getpass(prompt='Enter the database password for the janitor on \'steemfiles\' : ', stream=None)
config.set_key('db_password', db_password)
config.save_user_config()
parser = OptionParser()
parser.add_option("-t", "--testnet", action="store_true", dest="testnet", default=config.get('testnet', False))
parser.add_option("-m", "--mainnet", action="store_false", dest="testnet", default=config.get('testnet', False))
parser.add_option('-V', '--verify', action='store_true', dest='verify', default=False)
(options, arg) = parser.parse_args()
testnet = options.testnet
config.set_key('testnet', testnet)
config.save_user_config()
if testnet:
print("Using Testnet!")
def get_chain_name():
if testnet:
return 'TESTNET'
else:
return 'STEEM'
def get_chain_prefix():
if testnet:
return 'STX'
else:
return 'STM'
mysql_connection = mc.connect (host="localhost",
user='root',
passwd = db_password,
db = "steemrpc")
if mysql_connection is None:
print("Mysql connection value ie None")
def get_best_node(db_connection, testnet):
db_cursor = db_connection.cursor(dictionary=True)
chain_id = '79276aea5d4877d9a25892eaa01b0adf019d3e5cb12a97478df3298ccdd01673' if testnet else '0000000000000000000000000000000000000000000000000000000000000000'
db_cursor.execute("select url from node where chain_id=%s order by success_rate desc limit 1", (chain_id,))
dat = db_cursor.fetchone()
return dat
def lower_node_score(db_connection, node):
db_cursor = db_connection.cursor(dictionary=True)
db_cursor.execute("update node set failures=failures+1, success_rate=successes/(successes+failures) where url=%s", (node,))
def raise_node_score(db_connection, node):
db_cursor = db_connection.cursor(dictionary=True)
db_cursor.execute("update node set successes=successes+1, success_rate=successes/(successes+failures) where url=%s", (node,))
SECP256K1_MODULE = None
SECP256K1_AVAILABLE = False
CRYPTOGRAPHY_AVAILABLE = False
GMPY2_MODULE = False
if not SECP256K1_MODULE:
try:
import secp256k1
SECP256K1_MODULE = "secp256k1"
SECP256K1_AVAILABLE = True
except ImportError:
try:
import cryptography
SECP256K1_MODULE = "cryptography"
CRYPTOGRAPHY_AVAILABLE = True
except ImportError:
SECP256K1_MODULE = "ecdsa"
try:
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.asymmetric.utils \
import decode_dss_signature, encode_dss_signature
from cryptography.exceptions import InvalidSignature
CRYPTOGRAPHY_AVAILABLE = True
except ImportError:
CRYPTOGRAPHY_AVAILABLE = False
log.debug("Cryptography not available")
def filter_transaction(tx):
for key in ['transaction_id', 'trx_in_block', 'block_num', 'transaction_num']:
try:
del tx[key]
except KeyError:
pass
return tx
def recoverPublicKey(tx, network_name):
import beembase.signedtransactions
import beemgraphenebase.ecdsasig
import binascii
import beemgraphenebase.account
tx = filter_transaction(tx)
txs = beembase.signedtransactions.Signed_Transaction(**tx)
txs.deriveDigest(network_name)
i = binascii.unhexlify(tx['signatures'][0][:2])[0] - 27
if i > 3:
i -= 4
signature = binascii.unhexlify(tx['signatures'][0][2:])
if SECP256K1_MODULE == "secp256k1":
sig = pubkey.ecdsa_recoverable_deserialize(signature, i)
p = secp256k1.PublicKey(pubkey.ecdsa_recover(txs.message, sig))
p_comp = p.serialize()
else:
try:
p = beemgraphenebase.ecdsasig.recover_public_key(txs.digest, signature, i, txs.message)
except:
p = beemgraphenebase.ecdsasig.recover_public_key(txs.digest, signature, i)
p_comp = beemgraphenebase.ecdsasig.compressedPubkey(p)
beemPublicKey = beemgraphenebase.account.PublicKey(binascii.hexlify(p_comp).decode("ascii"))
# Use format(beemPublicKey, address_prefix) to get the address
return beemPublicKey
verified_count = 0
import decimal
# Prepares a transaction for the 'Signed_Transaction' constructor by converting the new amount types into the old strings.
def convert_amount_types(d):
if type(d) is dict:
if 'nai' in d:
symbol = None
if d['nai'] == '@@000000013':
symbol = 'SBD'
elif d['nai'] == '@@000000021':
symbol = 'STEEM'
elif d['nai'] == '@@000000037':
# VESTS?
symbol = 'VESTS'
else:
print("Unknown symbol " + d['nai'] + ' converting ' + str(d) + " to older format...")
return d
new_format = str(decimal.Decimal(d['amount'])/10**d['precision']) + ' ' + symbol
return new_format
else:
for y in d:
d[y] = convert_amount_types(d[y])
return d
convert_amounts_flag = False
best_node = None
try:
while True:
best_node = get_best_node(mysql_connection, testnet)
url = best_node['url']
print(f"Attempting to connect to {url}")
try:
then = time.perf_counter()
steem = beem.steem.Steem(node=url, num_retries=1, appbase=True)
now = time.perf_counter()
mysql_connection.cursor().execute("update node set delay=%s where url = %s", (now-then, best_node['url'],))
raise_node_score(mysql_connection, best_node['url'])
break
except:
lower_node_score(mysql_connection, best_node['url'])
bc = Blockchain(steem_instance=steem, mode='head')
raise_node_score(mysql_connection, best_node['url'])
print('Loaded blockchain')
for block in bc.blocks():
print('loaded block %s %d transactions' % (str(block["block_id"]), len(block['transactions']),))
# members of this block: previous,timestamp,witness,transaction_merkle_root,extensions,witness_signature,transactions,block_id,signing_key,transaction_ids,block_num
if len(block['transactions']):
for tx in block['transactions']:
tx['id'] = block['transaction_ids'][0]
tx['block_id'] = block['block_id']
if options.verify:
for op in tx['operations']:
if op['type'] == 'transfer_operation':
op['value']['amount'] = convert_amount_types(op['value']['amount'])
else:
op['value'] = convert_amount_types(op['value'])
try:
public_key = recoverPublicKey(tx, get_chain_name())
txsb = beembase.signedtransactions.Signed_Transaction(**tx)
txsb.deriveDigest(get_chain_name())
txsb.verify([public_key], get_chain_name())
verified_count += 1
except:
print("Verification failed.")
print(tx)
traceback.print_exc()
sys.exit(1)
except mysql.connector.errors.ProgrammingError as e:
config.set_key('db_password', None)
config.save_user_config()
print(e)
except beemapi.exceptions.RPCError:
print("Storing a lower score")
lower_node_score(mysql_connection, best_node['url'])
except beemapi.exceptions.NumRetriesReached:
print("Storing a lower score")
lower_node_score(mysql_connection, best_node['url'])
except beemapi.exceptions.UnhandledRPCError:
print("Storing a lower score")
lower_node_score(mysql_connection, best_node['url'])
except KeyboardInterrupt:
pass
finally:
mysql_connection.commit()
print("Verified %d transactions" % (verified_count,))
The program relies on a database. Here is SQL code for creating it:
create database steemrpc;
connect steemrpc;
create table node ( url char(64) unique not null, witness char(16), successes int(10) unsigned default 0, failures int(10) unsigned default 0, success_rate float default 1, delay float default 0, chain_id char(64) default '0000000000000000000000000000000000000000000000000000000000000000' , prefix char(3) default 'STM');
insert into node (url, witness) values ('wss://rpc.steemviz.com', 'ausbitbank'), ('wss://steemd.minnowsupportproject.org', 'followbtcnews'), ('wss://steemd.privex.io', 'privex'), ('wss://steemd.steemit.com', 'steemit'), ('wss://steemd.pevo.science', 'pharesim'), ('wss://appbasetest.timcliff.com', 'timcliff'), ('https://seed.bitcoiner.me', 'bitcoiner'), ('https://steemd.steemitstage.com', 'steemit'), ('wss://api.steemit.com', 'steemit'), ('https://rpc.buildteam.io', 'themarkymark');
insert into node (url, witness, chain_id, prefix) values ('https://testnet.steem.vc', 'almost-digital', '79276aea5d4877d9a25892eaa01b0adf019d3e5cb12a97478df3298ccdd01673', 'STX')
Now, the program had to go through several changes before you see it here. As I was running it, I immediately found cases of transactions that wouldn't validate with the original implementation and looking at the signatures helped me to find the defect. It was a matter of recovering the hint from the signature to recreate the public key. After making those corrections, I found attribute errors deep in Beem inside the Signed_Transaction() constructor. It seems Signed_Transaction()'s constructor was not apt to take as input the transactions that StreamTx was streaming in.
The amounts in Steem RPC nodes have changed from things like '0.283 SBD' to ['283', 3, '@@000000013'] and now it's {'amount': '283', 'precision': 3, 'nai': '@@000000013'}. More and more easier for software to use but more and more cryptic for human beings. So, I simply adopt the newest format to the oldest format and Signed_Transaction() works well like that. It's more of a stop-gap measure as the creator of beem adapts the constructor to use the new format. I hope he keeps compatibility with the old format as well though!
Now, the program validates hundreds of transactions without an error.
Previous Journal Entries:
- June 23rd Signing vs. Shared Secrets
- March 16th Developing Steemfiles, History in Javascript
- March 15th Developing Keybox, a signer, and a broadcaster
- March 14th (Working on a new Wallet
- March 13th (importance of source control, showing Public Keys, Politics Utopian)
- March 12th (keyring package for python) listing)
- March 10th (Keybox full working source code listing)
March 9th
March 8th
Dash XjzjT4mr4f7T3E8G9jQQzozTgA2J1ehMkV
LTC LLXj1ZPQPaBA1LFtoU1Gkvu5ZrxYzeLGKt
BitcoinCash 1KVqnW7wZwn2cWbrXmSxsrzqYVC5Wj836u
Bitcoin 1Q1WX5gVPKxJKoQXF6pNNZmstWLR87ityw (too expensive to use for tips)
See also
