Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Add experimental BalancedParquetEngine implementation #80

Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

[WIP] Add experimental BalancedParquetEngine implementation #80

wants to merge 8 commits into from

Commits on May 2, 2022

  1. initial BalancedParquetEngine implementation

    @rjzamora
    rjzamora committed May 2, 2022
    Configuration menu
    Copy the full SHA
    6499c29 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2022

  1. use fsspec.parquet for remote storage

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    9514210 View commit details
    Browse the repository at this point in the history

  2. align partition sizes with batch_size

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    9a7ff2c View commit details
    Browse the repository at this point in the history

  3. fix fsspec_parquet bug

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    e95947a View commit details
    Browse the repository at this point in the history

  4. remove from_map usage from now (its not even merged upstream yet)

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    6f84530 View commit details
    Browse the repository at this point in the history

  5. add basic test coverage

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    ca473f6 View commit details
    Browse the repository at this point in the history

  6. add warning message

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    60130ec View commit details
    Browse the repository at this point in the history

  7. cover num_workers argument

    @rjzamora
    rjzamora committed May 3, 2022
    Configuration menu
    Copy the full SHA
    a0d7e07 View commit details
    Browse the repository at this point in the history