Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Balancer Balances View to Incremental Strategy #39

Closed
wants to merge 22 commits into from

Conversation

jacek0x
Copy link
Collaborator

@jacek0x jacek0x commented Feb 2, 2023

Brief comments on the purpose of your changes:

Changed the Balancer balances view to an incremental strategy.

The default DBT strategy creates a view that has a very long query execution time.

cc: @Stefenon

For Dune Engine V2

I've checked that:

General checks:

  • I tested the query on dune.com after compiling the model with dbt compile (compiled queries are written to the target directory)
  • I used "refs" to reference other models in this repo and "sources" to reference raw or decoded tables
  • if adding a new model, I added a test
  • the filename is unique and ends with .sql
  • each sql file is a select statement and has only one view, table or function defined
  • column names are lowercase_snake_cased
  • if adding a new model, I edited the dbt project YAML file with new directory path for both models and seeds (if applicable)
  • if wanting to expose a model in the UI (Dune data explorer), I added a post-hook in the JINJA config to add metadata (blockchains, sector/project, name and contributor Dune usernames)

Pricing checks:

  • coin_id represents the ID of the coin on coinpaprika.com
  • all the coins are active on coinpaprika.com (please remove inactive ones)

Join logic:

  • if joining to base table (i.e. ethereum transactions or traces), I looked to make it an inner join if possible

Incremental logic:

  • I used is_incremental & not is_incremental jinja block filters on both base tables and decoded tables
    • where block_time >= date_trunc("day", now() - interval '1 week')
  • if joining to base table (i.e. ethereum transactions or traces), I applied join condition where block_time >= date_trunc("day", now() - interval '1 week')
  • if joining to prices view, I applied join condition where minute >= date_trunc("day", now() - interval '1 week')

MSilb7 and others added 22 commits January 30, 2023 11:20
* fix Aave

* revert bnb aave change
* join to self to grab factories

* fix th field name

* handle incremental

* add uni pools from ovm1

* fix uni pool ref

* Add cast()

---------

Co-authored-by: Huang Geyang <[email protected]>
* Add balancer_vebal_slopes to schema yml file

* Add veBAL related sources to sources yml file

* Add balancer_vebal_slopes spell

* Reorder sources

* Create balancer_vebal_votes spell

* Add balancer_vebal_votes to schema file

* Add GaugeController_evt_VoteForGauge to balancer source file

* Rename vebal_votes model file

* Try to modify to incremental view

* Revert "Try to modify to incremental view"

This reverts commit e28d024.

* Update sql

* Update config block

* update model naming to include blockchain

---------

Co-authored-by: Stefenon <[email protected]>
Co-authored-by: Huang Geyang <[email protected]>
* Add AXL price data

* Add axlUSDC price data

* Add axlUSDT price data

* Add axlATOM price data

* Remove AXL, fix details for axlUSDC and axlUSDT
* Update prices_tokens.sql

* Fix token-id and address

---------

Co-authored-by: Huang Geyang <[email protected]>
* adding 32 mainnet tokens and 1 Gnosis chain

* so it appears that it is commonly accepted that this ancient token has 16 decimals and is called TheDAO
Add new token VRA-Verasity into prices_tokens.
* Update prices_tokens.sql

* Update prices_tokens.sql

* Fix comma

* Rearrange records

* lowercase

---------

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: jeff-dude <[email protected]>
…cs#2579)

* Fix: cast traces.value and evt_Transfer.value and as double

* fix:typo
* deploy dagster on merge w env var
* add okx nft

* add op grants wallets

* fix wallet

* pushes

* labels edit

* make chain an array
* adding mistX on dex trades

* updated project name to mistX

* updated the comments

* updated changes for mstable

* Delete insert_mistx.sql

deleted

* updated the syntax for trades.sql

* Update dex_trades.sql

* Update dex_trades.sql

* Update mstable_ethereum_trades.sql

* fix syntax

* rebase

* Revert "rebase"

This reverts commit eb673bd.

* update dex.trades file

* Update dex_trades.sql

* updated the weth contract address

* clean up & formatting

* fix select statement

* array fix

* finished v1 and v2 factory trades

* fix block_time naming

* add readme

* fix pool_address join

* oops wrong join again

* ordering for unions

* meta case when was swapped

* dealing with weird USD values and missing bought amounts

* try isn't in spark sql lol

* fix brackets

* wrong order of columns gdi

* remove coaslece since it's crashing the PR

* remove extra

* see if fixes on combining all trades works

* update readme

* move comma

* missed

* lowercase to standardize

* more fixes

* remove duplicate hooks

* save as md

* jinja syntax & formatting

* move lower() to base model

* more fixes

* change to fix CI order?

---------

Co-authored-by: Don Bustos <[email protected]>
Co-authored-by: ripples3 <[email protected]>
Co-authored-by: Boxer <[email protected]>
Co-authored-by: 0xBoxer <[email protected]>
Co-authored-by: jeff-dude <[email protected]>
* add blur trades

* unnomalise % value

* fixes

* fixes

* remove wrong file changes

* temporarily double runtime - REMEMBER TO REMOVE

* fix

* fix

* ci timeout, formatting, filters

---------

Co-authored-by: jeff-dude <[email protected]>
* create files

* add hubble exchange perps

* fix synthax

* fix project start date and add model to perp model

* fix naming

* adjust trade categorization

* fix schema and remove quotes

* fix uniqueness test

* update schema, add seeds and smol fix

* Reformat sql

---------

Co-authored-by: Huang Geyang <[email protected]>
* add emdx files

* FIX SYNTHAX

* fix schema and remove quotes

* Fix uniqueness test

* Fix project name

* add seeds

* Reformat sql

---------

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: jeff-dude <[email protected]>
)

* add wigoswap fantom to dex trades

* update project start date

* fix incremental sql

* rerun gh

* Update sql

---------

Co-authored-by: Huang Geyang <[email protected]>
…nalytics#2536)

* create files

* add seeds and add model to dex trades

* fix incremental sql

* rerun gh

* Update sql

---------

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: jeff-dude <[email protected]>
* upload manifest 
---------

Co-authored-by: André Monteiro <[email protected]>
@jacek0x jacek0x requested a review from mendesfabio February 2, 2023 16:03
@jacek0x jacek0x self-assigned this Feb 2, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Nov 28, 2023
@mendesfabio mendesfabio deleted the bug/fix-balancer-balance-query-performance branch April 12, 2024 09:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.