Skip to content

Latest commit

 

History

History
111 lines (78 loc) · 4.77 KB

README.md

File metadata and controls

111 lines (78 loc) · 4.77 KB
Raw

Helium Blockchain ETL Lite

A "Light" ETL for the Helium blockchain. A lower resource-intensive option to Helium's full ETL

This tool is best suited for collecting new data on the blockchain as it's happening versus historical data. You should probably use the full ETL if you need historical data.

ETL Lite relies on Helium's blockchain-node to extract its data.

NOTE: ETL Lite will only load data as old as the height of the snapshot loaded in blockchain-node.
For more historical data please use Helium's Full ETL.

Get Started

For a more in-depth example, see the quick start guide in docs

  1. Follow these steps to build and run Helium's blockchain-node. NOTE You must change {store_json, false} to {store_json, true} in blockchain_node's sys.config file. Run this command before make release

sed -i "/{blockchain,/{N;s/$/\n {store_json, true},/}" config/dev.config

  1. Install cargo + rust

    curl https://sh.rustup.rs -sSf | sh

  2. Install postgresql

    sudo apt get install postgresql
    or for MacOS
    brew install postgresql

  3. Clone this repo

    git clone https://github.com/dewi-alliance/helium-etl-lite.git

  4. Build from source

    cargo build --release

  5. Update settings.toml. node_addr is your blockchain-node and database_url should be your postgres db

  6. Run target/release/helium_etl_lite migrate to run migrations then target/release/helium_etl_lite start

Settings

Settings are found in the settings.toml file in the config directory.

log_dir : location to create log files. Default is log in the parent directory.
mode : Choose which mode to run ETL Lite in. Currently (phase 1) only rewards is supported.
node_addr : Ip address for blockchain-node.
database_url : Url to postgresql server.

Mode Options

ETL Lite is currently in Phase 3

Phase Modes Available
1 Rewards
2 Rewards, Full
3 Rewards, Full, Filters

Rewards: Load all rewards only
Full: Load all transactions including rewards
Filters: Filter what is loaded by either gateway or account

Filters

Filters must be added to the filters table. Rewards can be filtered by account or gateway.

example:

INSERT INTO filters (type, value) VALUES ('account', '13oNZxczcP2urLzQTGQVFpezg4C3EADqjcTmDDH5yrpAzi389HL')

Database schemas

               Table "public.rewards"
Column Type Collation Nullable Default
block bigint not null
transaction_hash text not null
time bigint not null
account text not null
gateway text not null
amount bigint not null
type text not null 
         Table "public.follower_info"
Column Type Collation Nullable Default
height bigint not null
first_block bigint not null 
            Table "public.transactions"
Column Type Collation Nullable Default
block bigint not null
hash text not null
type transaction_type not null
fields jsonb not null
Indexes: 
"transactions_pkey" PRIMARY KEY, btree (hash)
"transaction_block_idx" btree (block)
"transaction_type_idx" btree (type)

                Table "public.filters"
Column Type Collation Nullable Default
type filter_type not null
value text not null

Rewards Data Note

Because of the way blockchain-node stores rewards info, the first ~300 blocks after the snapshot height won't incldue gateway or type information for specific rewards. All rewards with type = 'rewards_v2' are the total rewards paid to that account vs individual rewards that you will see being loaded into the rewards db after the first ~300 blocks after the snapshot height of the node.

Please note: anytime you see 1Wh4bh as a value, this is the hash for null. You will see this in rewards of type securities and rewards_v2 in the gateway field and for type overages in the account field.