Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chore: Remove legacy CI/CD workflow #58

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 0 additions & 27 deletions .github/workflows/ci.yml

This file was deleted.

9 changes: 2 additions & 7 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
ci:
autofix_commit_msg: "Chore: pre-commit autoupdate"
skip:
# pre-commit.ci cannot install WGET, so tomlint must be disabled
- tomllint

exclude: |
Expand Down Expand Up @@ -96,6 +95,7 @@ repos:
rev: 6.3.0
hooks:
- id: pydocstyle
additional_dependencies: ["tomli"]

- repo: https://github.com/PyCQA/isort
rev: 5.13.2
Expand All @@ -113,12 +113,7 @@ repos:
hooks:
- id: flake8
# Ignore all format-related checks as Black takes care of those.
args:
- --ignore=E2, W5, F401, E401
- --select=E, W, F, N
- --max-line-length=120
additional_dependencies:
- pep8-naming
args: ["--ignore=E2, W5, F401, E401", "--select=E, W, F, N", "--max-line-length=120"]

- repo: https://github.com/pre-commit/mirrors-mypy
rev: "v1.10.0"
Expand Down
26 changes: 17 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# osc-ingest-tools

python tools to assist with standardized data ingestion workflows

### Install from PyPi
Expand Down Expand Up @@ -42,10 +43,10 @@ pip install osc-ingest-tools
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 first_name 3 non-null string
1 age_in_years 3 non-null Int64
1 age_in_years 3 non-null Int64
dtypes: Int64(1), string(1)
memory usage: 179.0 bytes

Expand All @@ -55,10 +56,11 @@ memory usage: 179.0 bytes
first_name varchar,
age_in_years bigint

>>>
>>>
```

#### Adding custom type mappings to `create_table_schema_pairs`

```python
>>> df = pd.DataFrame(data, columns = ['First Name', 'Age In Years'])

Expand All @@ -68,10 +70,10 @@ memory usage: 179.0 bytes
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 2 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 first_name 3 non-null object
1 age_in_years 3 non-null int64
1 age_in_years 3 non-null int64
dtypes: int64(1), object(1)
memory usage: 176.0+ bytes

Expand All @@ -87,7 +89,7 @@ memory usage: 176.0+ bytes
### Development

Patches may be contributed via pull requests to
https://github.com/os-climate/osc-ingest-tools.
<https://github.com/os-climate/osc-ingest-tools>.

All changes must pass the automated test suite, along with various static
checks.
Expand All @@ -97,12 +99,14 @@ checks.

Enabling automatic formatting via [pre-commit](https://pre-commit.com/) is
recommended:

```
pip install black isort pre-commit
pre-commit install
```

To ensure compliance with static check tools, developers may wish to run;

```
pip install black isort
# auto-sort imports
Expand All @@ -112,6 +116,7 @@ black .
```

Code can then be tested using tox.

```
# run static checks and tests
tox
Expand All @@ -124,13 +129,16 @@ tox -e cov
```

### Releasing

To release a new version of this library, authorized developers should;

- Prepare a signed release commit updating `version` in setup.py
- Tag the commit using [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
prepended with "v"
prepended with "v"
- Push the tag

E.g.,

```
git commit -sm "Release v0.3.4"
git tag v0.3.4
Expand Down
13 changes: 11 additions & 2 deletions osc_ingest_trino/__init__.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
from .boto3_utils import attach_s3_bucket, upload_directory_to_s3
from .dotenv_utils import load_credentials_dotenv
from .sqlcols import enforce_partition_column_order, enforce_sql_column_names, sql_compliant_name
from .sqlcols import (
enforce_partition_column_order,
enforce_sql_column_names,
sql_compliant_name,
)
from .sqltypes import create_table_schema_pairs, pandas_type_to_sql
from .trino_utils import TrinoBatchInsert, _do_sql, attach_trino_engine, fast_pandas_ingest_via_hive
from .trino_utils import (
TrinoBatchInsert,
_do_sql,
attach_trino_engine,
fast_pandas_ingest_via_hive,
)

__all__ = [
"sql_compliant_name",
Expand Down
Loading
Loading