Merge pull request #1 from KomodoPlatform/master

.

README.md

@@ -1 +1,32 @@
-Toolbox for managing Komodo and Iguana nodes
+## Komodotools repository
+Purpose of this repository is to collect scripts/tools from Notary Nodes operators.
+We would like to get rid of scripts in [komodo](https://github.com/jl777/komodo) and [iguana](https://github.com/jl777/SuperNET) repositories. These supposed to serve as examples in the past. 
+
+Because we did not decide what the best management tooling is, we will keep different ideas separated in directories until we fix problems which will converge to common tools.
+
+## Contribution
+- normal Github workflow is followed (fork this repo, create branch, make changes and push back to your repo, create PR, review, merge, repeat)
+- if you see similar script to one which exists in repo, try to implement your changes against it
+- each directory must contain README.md file with description how to install/use your scripts
+- update this README.md and add brief description what your script/tool does
+- try to keep only one version of file which would work in test and prod environments
+- once your scripts are merged into this repository, refer to it and push all updates here 
+- try to separate data/configs from code, if the data/configs will change, your scripts should still work without rewriting them
+
+## Directories content
+### kolo
+It was Kolo's idea to create this repository and here is his concept how it should work.
+
+### dragonriders
+ - dokomodo - script written in Python [Click](http://click.pocoo.org) framework. 
+   - write funtcions which will become commands
+   - supports dev and prod environments
+   - assetchains data stored in yaml data file and ini config for configuration (you can enable mining for specific chains) 
+   - there is Config Class which takes care of parsing config/data files so no need to do it in you functions
+   - many other features supported by Click framework
+
+### a-team 
+ - complete step-by-step guide with bash installation scripts
+
+## Contacts
+Ideas can be discussed in [#notarynode](https://komodo-platform.slack.com) Slack channel, but please all code proposals discuss via PRs on Github.

dragonriders/.gitignore

@@ -0,0 +1,7 @@
+*.swp
+assetchains
+komodotools_venv/
+docker-compose_assets_development.yml
+docker-compose_assets_production.yml
+__pycache__
+dokomodo.egg-info

dragonriders/README.md

@@ -0,0 +1,84 @@
+# DOkomodo - swiss army knife for Notary Nodes
+
+Current komodo/iguana management scripts are not ideal and this is idea how to replace 
+existing komodo/iguana scripts with universal framework. Python [Click](http://click.pocoo.org) 
+framework was used which is ideal for creating CLI applications. Main idea of Click is that your
+functions will become excutable as commands.
+
+Features of DOkomodo:
+- single data file in yaml format (easy to edit), sections for prod and dev environments - all data
+ like list of assetchains and their ports will come here.
+- repeating functions like e.g. loading list of assetchains do not have to be written again and again
+- data file loaded only once and data accessed via instance attributes
+- configuration in config.ini file allows you to say for which coins mining is enabled, you can disable
+ minining completely or you can enable mining on randomly selected assetchains
+- currently only 2 commands available
+
+## Installation steps
+### Enable python3 virtualenv
+Virtualenv is directory which contains all python packages you need for your project.
+
+Install virtualenv on Centos (as root user):
+```
+# yum install python34 python-pip
+```
+
+Install virtualenv on Ubuntu (as root):
+```
+# apt-get install python-pip
+```
+
+Now install virtualenv package:
+```
+# pip install -U virtualenv
+
+```
+
+Create our virtualenv
+```
+cd ~/venv_projects
+virtualenv -p python3 komodotools_venv
+```
+
+This is how we activate our virtualenvironment. You can deactivate it with `deactivate` command:
+```
+source ~/venv_projects/komodotools_venv/bin/activate
+```
+
+### Clone this repository
+Make sure you have activated your virtualenv before this step. 
+```
+cd ~/git_projects/
+git clone <this_repo>.git && cd komodotools
+pip install -Ur requirements.txt
+pip install --editable .
+```
+
+requirements.txt file contains list of all python packages we need for this project. All python packages will be installed into virtualenv folder you created. They won't collide with your system python packages.
+
+## How-to use it
+Once your virtualenv is activated, type `dokomodo`:
+```
+$ dokomodo
+Usage: dokomodo [OPTIONS] COMMAND [ARGS]...
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  assetchains              Replacement for assetchains script
+  generate_docker_compose  Generates docker-compose file with all assetchains
+```
+
+Then you can continue specify subcommands:
+```
+$ dokomodo assetchains --help
+Usage: dokomodo assetchains [OPTIONS]
+
+Options:
+  -b, --branch [development|production]
+                                  [required]
+  --help                          Show this message and exit.
+```
+
+

dragonriders/dokomodo/cli.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+from jinja2 import Environment, FileSystemLoader
+from ruamel.yaml import YAML
+import socket
+import shlex
+import subprocess
+import time
+from configparser import ConfigParser
+import click
+import py
+import colorama
+import sys
+
+# asset_data_url = ("https://raw.githubusercontent.com/patchkez/kmdplatform/"
+#    "master/yaml/data.yaml")
+
+env = Environment(loader=FileSystemLoader('./dokomodo/templates/'), trim_blocks=True,
+    lstrip_blocks=True)
+
+config_dir = py.path.local('dokomodo/yaml')
+
+
+class Config(object):
+    def __init__(self, *args, **kwargs):
+        # self.config = py.path.local('dokomodo').join('yaml').join('data.yaml')
+        self.config = config_dir.join('data.yaml')
+        # self.config_ini = py.path.local('dokomodo').join('yaml').join('config.ini')
+        self.config_ini = config_dir.join('config.ini')
+
+        super(Config, self).__init__(*args, **kwargs)
+
+    def load(self):
+        """Try to load the yaml"""
+        yaml = YAML(typ='safe', pure=True)
+        yaml.default_flow_style = True
+        self.config_data = yaml.load(self.config.read())
+        self.branches = self.config_data['assetchains']
+        self.seed_ip = socket.gethostbyname(self.config_data['seed_host'])
+
+    def load_ini(self):
+        ini_parser = ConfigParser()
+        ini_parser.read(str(self.config_ini))
+        self.assetchains = ini_parser['ASSETCHAINS']
+        self.mined_coins = self.assetchains['mined_coins'].split()
+        self.delay_asset = float(self.assetchains['delay_asset'])
+
+
+# This is click thing, it will create decorator named pass_config from our Config class
+# This decorator is then passed to every function which needs to access attributes from Config class
+pass_config = click.make_pass_decorator(Config, ensure=True)
+
+
+@click.group()
+@pass_config
+def cli(config):
+    config.load()
+    config.load_ini()
+
+
+@click.command('generate_docker_compose',
+    short_help='Generates docker-compose file with all assetchains')
+@click.option('-b', '--branch', required=True, type=click.Choice(['development', 'production']),
+    prompt=True)
+@pass_config
+def generate_docker_compose(ctx, branch):
+    """ TODO """
+    filename = 'docker-compose_assets_' + branch + '.yml'
+    click.echo('Writing new docker compose file into: %s' % filename)
+    template = env.get_template('docker-compose-template.conf.j2')
+    templatized_config = template.render(items=ctx.config_data['assetchains'][branch],
+        seed_ip=ctx.seed_ip, mined=ctx.mined_coins)
+
+    fo = open(filename, 'w')
+    fo.write(templatized_config)
+    fo.close()
+
+
+@click.command('assetchains', short_help='Replacement for assetchains script')
+@click.option('-b', '--branch', required=True, type=click.Choice(['development', 'production']),
+    prompt=True)
+@pass_config
+def assetchains(ctx, branch):
+    bash_template = env.get_template('assetchains.j2')
+    bash_templatized_config = bash_template.render(items=ctx.config_data['assetchains'][branch],
+        seed_ip=ctx.seed_ip, mined=ctx.mined_coins)
+
+    # fa = open('assetchains', 'w')
+    # fa.write(bash_templatized_config)
+    # fa.close()
+
+    # Remove empty strings
+    assetchains = list(filter(None, bash_templatized_config.split("\n")))
+    # Executed komodod commands with predefined sleep time
+    for assetchain_command in assetchains:
+        args = shlex.split(assetchain_command)
+        try:
+            subprocess.Popen(args)
+        except OSError as exception:
+            click.echo(exception)
+            sys.exit(1)
+        time.sleep(ctx.delay_asset)
+
+
+# Add functions into cli() function which is main group for all commands
+cli.add_command(generate_docker_compose)
+cli.add_command(assetchains)
+
+
+if __name__ == "__main__":
+    cli()

dragonriders/dokomodo/templates/assetchains.j2

@@ -0,0 +1,11 @@
+{% for assetname, asset in items.items() %}
+komodod -pubkey=${pubkey} -ac_name={{ assetname }} -ac_supply={{ asset.amount 
+  }} -addnode={{ seed_ip }}{% for mine in mined %}
+      {% if assetname in mine %} -gen{% endif %}
+      {% if 'random' in mine %}
+      {% if range(0, 32767) | random % 10 == 1 %}
+      -gen{% endif %}
+      {% endif %}
+      {% endfor %}
+
+{% endfor %}

dragonriders/dokomodo/templates/docker-compose-template.conf.j2

@@ -0,0 +1,46 @@
+# vim: ts=2 sw=2 et
+version: '3'
+
+services:
+  {% for assetname, asset in items.items() %}
+  {{ assetname }}:
+    image: kmdplatform_komodo
+    network_mode: "host" 
+    ports:
+      {% if asset.rpc_port %}
+      - "127.0.0.1:{{ asset.rpc_port }}:{{ asset.rpc_port }}"
+      {% endif %}
+      {% if asset.p2p_port %}
+      - "{{ asset.p2p_port }}:{{ asset.p2p_port }}"
+      {% endif %}
+    volumes:
+      - komodo-data:/home/komodo/.komodo
+      - shared-data:/home/komodo/.shared:ro
+    environment:
+      - RPC_USER=${RPC_USER}
+      - RPC_PASSWORD=${RPC_PASSWORD}
+    command: > 
+      bash -c "komodod -pubkey=$$pubkey 
+      -ac_name={{ assetname }} 
+      -ac_supply={{ asset.amount }} 
+      -addnode={{ seed_ip }}{% for mine in mined %}
+      {% if assetname in mine %} -gen{% endif %}
+      {% if 'random' in mine %}
+      {% if range(0, 32767) | random % 10 == 1 %} 
+      -gen{% endif %}
+      {% endif %}"
+      {% endfor %}
+  
+  {% endfor %}
+
+volumes:
+  komodo-data:
+    driver_opts:
+      type: none
+      device: ${KOMODO_DATA}
+      o: bind
+  shared-data:
+    driver_opts:
+      type: none
+      device: ${SHARED_DATA}
+      o: bind

dragonriders/dokomodo/yaml/config.ini

@@ -0,0 +1,19 @@
+[DEFAULT]
+
+[ASSETCHAINS]
+# mined_coins is list of assetchains for which mining will be enabled
+# When the list is empty, no -gen option will be addded to assetchain in question
+# When the list contains only 'random' lowercase word, -gen param will be added to assetchains randomly
+# Example 1:
+# mined_coins = random 
+
+# Example 2:
+# mined_coins = SUPERNET, BEER, PIZZA
+
+# Example 3:
+# mined_coins = 
+
+mined_coins =
+
+# Delay in seconds between each assetchain is started
+delay_asset = 20

dragonriders/dokomodo/yaml/data.yaml

@@ -0,0 +1,109 @@
+assetchains:
+  production:
+    REVS:
+      ammount: 1300000
+      p2p_port: 10195
+      rpc_port: 10196
+    SUPERNET:
+      amount: 816061
+      p2p_port: 11340
+      rpc_port: 11341
+    DEX:
+      amount: 999999
+      p2p_port: 11889
+      rpc_port: 11890
+    PANGEA:
+      amount:  999999
+      p2p_port: 14067
+      rpc_port: 14068
+    JUMBLR:
+      amount: 999999
+      p2p_port: 15105
+      rpc_port: 15106
+    BET:
+      amount: 999999
+      p2p_port: 14249
+      rpc_port: 14250
+    CRYPTO:
+      amount: 999999
+      p2p_port: 12947
+      rpc_port: 12948
+    HODL: 
+      amount: 9999999
+      p2p_port: 14430
+      rpc_port: 14431
+    MSHARK:
+      amount: 1400000
+      p2p_port: 10113
+      rpc_port: 10114
+    BOTS:
+      amount: 999999
+      p2p_port: 11363 
+      rpc_port: 11364
+    MGW:
+      amount: 999999
+      p2p_port: 12385
+      rpc_port: 12386
+    COQUI:
+      amount: 72000000
+      p2p_port: 14275 
+      rpc_port: 14276
+    WLC:
+      amount: 210000000
+      p2p_port: 12166
+      rpc_port: 12167
+    KV:
+      amount: 1000000
+      p2p_port: 8298
+      rpc_port: 8299
+    CEAL:
+      amount: 366666666
+      p2p_port: 11115
+      rpc_port: 11116
+    MESH:
+      amount: 1000007
+      p2p_port: 9454
+      rpc_port: 9455
+    MNZ:
+      amount: 257142858
+      p2p_port: 14336
+      rpc_port: 14337
+    AXO:
+      amount: 200000000
+      p2p_port: 12926
+      rpc_port: 12927
+    ETOMIC:
+      amount: 100000000
+      p2p_port: 10270
+      rpc_port: 10271
+    BTCH:
+      amount: 20998641
+      p2p_port: 8799
+      rpc_port: 8800
+    VOTE2018:
+      amount: 600000000
+      p2p_port: 10316
+      rpc_port: 10317
+    PIZZA:
+      amount: 100000000
+      p2p_port: 
+      rpc_port: 11116
+    BEER:
+      amount: 100000000
+      p2p_port: 
+      rpc_port: 8923
+    NINJA:
+      amount: 100000000
+      p2p_port: 15430
+      rpc_port: 15431
+  development:
+    PIZZA:
+      amount: 100000000
+      p2p_port: 11607 
+      rpc_port: 11608
+    BEER:
+      amount: 100000000
+      p2p_port: 8922
+      rpc_port: 8923  
+
+seed_host: zero.kolo.supernet.org

dragonriders/requirements.txt

@@ -0,0 +1,8 @@
+ruamel.yaml
+Jinja2
+requests
+click
+py
+ipython
+configparser
+colorama

dragonriders/setup.py

@@ -0,0 +1,16 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="dokomodo",
+    version='0.1',
+    py_modules=['dokomodo'],
+    # packages=find_packages(),
+    packages=['dokomodo'],
+    install_requires=[
+        'Click',
+    ],
+    entry_points='''
+        [console_scripts]
+        dokomodo=dokomodo.cli:cli
+    ''',
+)

kolo/README.md

@@ -0,0 +1,2 @@
+## Do script - original idea
+This was initial idea of Kolo about how he would like to manage Notary Nodes. There would be one central command which would fire another commands.