Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Architecture refactoring #61

Merged
merged 153 commits into from
Oct 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
ae2c676
chore(ci): remove the limitation on branches for automatic testing
raf-nr Sep 12, 2024
bbdd4ea
chore(domain): replace optional model to new domain module
raf-nr Sep 12, 2024
20fa0c1
feat(tests): add tests for optional model
raf-nr Sep 12, 2024
f801d15
feat(domain): add file entity
raf-nr Sep 12, 2024
9f7757e
feat(tests): add tests for file entity
raf-nr Sep 12, 2024
464bf2c
chore(domain): replace and restructure task entities to new domain mo…
raf-nr Sep 12, 2024
3fb52eb
chore: change imports to absolute ones
raf-nr Sep 12, 2024
2c397d8
chore(domain): add empty user submodule to domain module
raf-nr Sep 17, 2024
9fbfed1
fix(domain): add handling of incorrect case in optional model
raf-nr Sep 17, 2024
4e929ea
chore(domain): add code documentation for tasks and file
raf-nr Sep 17, 2024
bac652e
chore(domain): add README files for domain modules
raf-nr Sep 17, 2024
11319b7
chore(use case): init use case module, exception and dto submodules
raf-nr Sep 17, 2024
19b2f3f
chore(use case): init file, task and user use case modules
raf-nr Sep 17, 2024
9f60d7b
chore: add pytest mock, pytest asyncio and cfgv to dependencies
raf-nr Sep 17, 2024
ffd25d1
fix: replace dto module to internal
raf-nr Sep 17, 2024
a983a54
feat(dto): add base schema for dto
raf-nr Sep 17, 2024
5cac8bb
feat(dto): add dto schemas for file
raf-nr Sep 17, 2024
f086724
feat(dto): add dto schemas for file metadata
raf-nr Sep 17, 2024
e0abbe2
feat(dto): add dto schemas for dataset
raf-nr Sep 17, 2024
7dce251
feat(dto): add dto schemas for task
raf-nr Sep 17, 2024
876314f
chore: init repository exceptions file
raf-nr Sep 17, 2024
77f4814
feat(dto): add dto schemas for task worker
raf-nr Sep 17, 2024
c48e450
chore: add readme file for dto module
raf-nr Sep 17, 2024
095054b
feat: add file and dataset exceptions
raf-nr Sep 17, 2024
01c264e
fix: delete old use case exception module
raf-nr Sep 17, 2024
1aea56a
feat(use case): add use case for file's content type checking
raf-nr Sep 18, 2024
d35b28f
feat(tests): add tests for file's content type checking use case
raf-nr Sep 18, 2024
adf1394
feat: add Unit Of Work pattern implementation
raf-nr Sep 18, 2024
4dccf1c
feat(tests): add tests for unit of work
raf-nr Sep 18, 2024
549c905
feat(use case): add use case for saving datasets
raf-nr Sep 18, 2024
04d67bc
feat(tests): add tests for save dataset use case
raf-nr Sep 18, 2024
9bacfc0
feat(use case): add use case for file saving
raf-nr Sep 18, 2024
5054d11
feat(tests): add tests for save file use case
raf-nr Sep 18, 2024
f6eae29
feat: add task use cases' exceptions
raf-nr Sep 18, 2024
3f6a6cd
feat(use case): add use case for retrieving task
raf-nr Sep 18, 2024
c1a5bb6
feat(tests): add tests for retrieve task use case
raf-nr Sep 18, 2024
933edc1
chore: add missing imports
raf-nr Sep 18, 2024
3aeadd7
feat(use case): add use case for setting task
raf-nr Sep 18, 2024
0b3ddff
feat(tests): add tests for set task use case
raf-nr Sep 18, 2024
1ac699e
chore: add file for user's use case exceptions
raf-nr Sep 18, 2024
ddf134d
chore: add readme file for usecase module
raf-nr Sep 18, 2024
6973494
chore: init infrastructure, repository and worker modules
raf-nr Sep 18, 2024
ba6974f
chore(infra): init data storage module
raf-nr Sep 18, 2024
6c05e60
feat(infra): add settings for data storages
raf-nr Sep 18, 2024
68bd15e
feat(infra): add context implementation for postgres
raf-nr Sep 18, 2024
77cb1ff
feat(infra): add base orm model for relational data bases
raf-nr Sep 18, 2024
e904208
feat(infra): add base orm model for relational data bases
raf-nr Sep 18, 2024
6883f13
feat(infra): add file metadata model for relational data bases
raf-nr Sep 18, 2024
2bdaec4
feat(infra): add dataset model for relational data bases
raf-nr Sep 18, 2024
174bb42
feat(infra): add task model for relational data bases
raf-nr Sep 18, 2024
906ad41
chore(infra): add empty module for user relational models
raf-nr Sep 18, 2024
8192e40
chore: set new directories for test command in the makefile
raf-nr Sep 18, 2024
fbe79a1
chore: set new alembic.ini file's path for postgres migrations comman…
raf-nr Sep 18, 2024
45988c0
feat(infra): add first empty postgres migration
raf-nr Sep 18, 2024
becf941
feat(infra): add postgres db migration that creates file and dataset…
raf-nr Sep 18, 2024
1cbe946
feat(infra): add postgres db migration that creates task table
raf-nr Sep 18, 2024
97f1ca4
feat(repo): add file repository implementation
raf-nr Sep 18, 2024
1ad0cd3
chore(infra): rename and replace relational storage context
raf-nr Sep 19, 2024
c53091d
chore(repo): format code
raf-nr Sep 19, 2024
439e6cc
fix(infra): remove typo in dataset relational model's fields
raf-nr Sep 20, 2024
67e8f94
fix(uow): change the way to return a context from uow
raf-nr Sep 20, 2024
c74ca72
feat(use case): add file metadata not found exception
raf-nr Sep 20, 2024
806b6d9
feat(use case): add use case for retrieving datasets
raf-nr Sep 20, 2024
606b8e4
feat(tests): add tests for retrieve dataset use case
raf-nr Sep 20, 2024
69cdbc6
fix(use case): remove unnecessery fields in profiling task worker int…
raf-nr Sep 20, 2024
307f1ad
chore(use case): make usecase arguments callable only by key-value
raf-nr Sep 20, 2024
e35f077
feat(use case): add use case for updating task info
raf-nr Sep 20, 2024
383fdf1
chore(dto): update task update schema
raf-nr Sep 20, 2024
cf6ac02
feat(tests): add tests for updating task info use case
raf-nr Sep 20, 2024
b2a654e
feat(dto): add dataset and file metadata repositories' exceptions
raf-nr Sep 20, 2024
cf8ba5b
feat(dto): add find and response schemas for csv files
raf-nr Sep 20, 2024
efb4913
feat(domain): add a function to get a primitive by its name
raf-nr Sep 20, 2024
c2c1701
feat(use case): add use case for task profiling
raf-nr Sep 20, 2024
e010653
feat(tests): add tests for profilie task use case
raf-nr Sep 20, 2024
cab295a
chore(infra): change name of relational context type
raf-nr Sep 20, 2024
7e52f89
feat(infra): add postgres context and context maker
raf-nr Sep 20, 2024
27124ea
feat(repo): add find(read) csv file method to the file repository
raf-nr Sep 20, 2024
6cf24b6
feat(repo): add class with crud operations for relational repositories
raf-nr Sep 20, 2024
2f1c506
feat(repo): add file metadata repository implementation
raf-nr Sep 20, 2024
efdc4a8
fix(tests): update conftest file, delete alembic tests
raf-nr Sep 20, 2024
b963293
feat(infra): add necessary imports for migrations
raf-nr Sep 20, 2024
73fcab2
feat(repo): add dataset repository implementation
raf-nr Sep 20, 2024
62c021b
feat(tests): update conftest file with database preparing fixture
raf-nr Sep 20, 2024
4f205d3
feat(repo): add find with join for dataset repository
raf-nr Sep 20, 2024
812b523
feat(repo): add task repository implementation
raf-nr Sep 20, 2024
e69eb79
feat(infra): init background task and celery modules
raf-nr Sep 20, 2024
11e4773
feat(infra): add settings for background tasks
raf-nr Sep 20, 2024
7e3a337
feat(infra): add celery config
raf-nr Sep 20, 2024
d5c290e
feat(infra): add resource intensive celery task class
raf-nr Sep 20, 2024
32af375
feat(infra): add dependencies for injection in celery tasks
raf-nr Sep 20, 2024
a73b900
feat(infra): add data profiling celery task
raf-nr Sep 20, 2024
5ba0b87
feat(infra): add the main celery worker
raf-nr Sep 20, 2024
1fd1fbb
feat(worker): add data profiling worker implementation (for celery)
raf-nr Sep 20, 2024
cefb554
feat(rest): init rest module and its submodules
raf-nr Sep 20, 2024
15b4bfc
feat(test): add ping endpoint, common router
raf-nr Sep 20, 2024
a47646f
feat(rest): add dependencies for injection in endpoints
raf-nr Sep 20, 2024
38f5963
feat(rest): add dependencies for injection in file/dataset endpoints
raf-nr Sep 20, 2024
a917b95
feat(rest): add endpoint for upload csv datasets
raf-nr Sep 20, 2024
d3bd6a4
feat(rest): add endpoint for retrieve datasets
raf-nr Sep 20, 2024
6b31383
feat(rest): add dependencies for injection in task endpoints
raf-nr Sep 20, 2024
809a8d0
feat(rest): add endpoint for task setting
raf-nr Sep 20, 2024
f0f2210
feat(rest): add endpoint for task retrieving
raf-nr Sep 20, 2024
65c57a1
feat(rest): add di module for user
raf-nr Sep 20, 2024
7c120a7
feat(rest): add main api router
raf-nr Sep 20, 2024
7807ba9
feat: add main application (fastapi) object
raf-nr Sep 20, 2024
6209e71
feat: change path to app object and celery worker in makefile
raf-nr Sep 20, 2024
80eb264
fix(repo): typo in imports
raf-nr Sep 20, 2024
4f26968
fix(tests): remove typo in fixture name
raf-nr Sep 20, 2024
f19e995
chore: change name for celery worker in makefile
raf-nr Sep 20, 2024
55ad7b6
fix(infra): make the correct traceback processing
raf-nr Sep 21, 2024
35b8306
feat(rest): add use case's exceptions handling
raf-nr Sep 21, 2024
d53540a
chore(domain): add code documentation for math function
raf-nr Sep 21, 2024
bfef666
chore: add failed file read exception and its handler
raf-nr Sep 21, 2024
c5ba4ca
chore: delete old app module
raf-nr Sep 21, 2024
2623cd5
chore: format code
raf-nr Sep 21, 2024
82cce96
chore: change old module path in all makefile commands
raf-nr Sep 21, 2024
27c4184
fix(tests): remove typo in exception name and expande uow mocks
raf-nr Sep 21, 2024
0574ed8
feat(infra): add flat file storage context and update use cases
raf-nr Sep 21, 2024
3032413
feat(rest): update di after add new flat context
raf-nr Sep 21, 2024
0f65293
fix: typo in file metadata repository name
raf-nr Sep 21, 2024
57e0320
feat(tests): add tests for file repository
raf-nr Sep 21, 2024
abbe5a2
feat(tests): add tests for file metadata repository
raf-nr Sep 21, 2024
d8bc2ee
feat(tests): add tests for dataset repository
raf-nr Sep 21, 2024
c0b3c13
feat(tests): add tests for task repository
raf-nr Sep 21, 2024
315f6bb
fix(tests): add converter from UUID to string for file name
raf-nr Sep 21, 2024
facf649
chore(domain): move task status and task failure reason classes in se…
raf-nr Sep 21, 2024
23e7c58
fix(dto): remove unnecessery generic in update schemas
raf-nr Sep 21, 2024
d6a3e5f
chore: add and fix documentation for repo and use case exceptions
raf-nr Sep 21, 2024
1ee1d93
chore(infra): add readme file for data storage module
raf-nr Sep 21, 2024
35e27a9
chore(infra): correct typing
raf-nr Sep 21, 2024
2115922
chore: update new directories path in Dockerfile
raf-nr Sep 21, 2024
f4af48a
chore(infra): add missed imports to postgres env.py file
raf-nr Sep 21, 2024
eeb12b2
chore: fix format errors
raf-nr Sep 21, 2024
d36ab13
chore: delete sqlalchemy mixins from project dependencies
raf-nr Sep 25, 2024
636488c
fix: type errors
raf-nr Sep 27, 2024
835272c
chore: add command for type checking in makefile and rename old targe…
raf-nr Sep 27, 2024
e5e1c21
chore: add type checking to ci
raf-nr Sep 27, 2024
cad5d30
chore(infra): add relational/postgres context maker types
raf-nr Sep 30, 2024
459d797
chore(infra): add helper functions for getting postgres context
raf-nr Sep 30, 2024
79ebcce
feat(infra): implement an analogue of a transaction for local storage
raf-nr Oct 1, 2024
72910be
feat(tests): add tests for new flat context
raf-nr Oct 1, 2024
eeff5b9
feat(infra): add relational add/delete models
raf-nr Oct 1, 2024
af7aeb9
chore(tests): divide the test directory into subdirectories for unit …
raf-nr Oct 1, 2024
9025b96
feat(infra): implement sync flush method for flat context
raf-nr Oct 1, 2024
6550cee
feat(infra): add universal context for storages
raf-nr Oct 1, 2024
07857a3
feat(tests): add tests for universal context
raf-nr Oct 1, 2024
d26bec2
chore(usecase): add missed task usecases imports
raf-nr Oct 1, 2024
b2ffbe3
feat(usecase): make a single unit of work in each use case and its us…
raf-nr Oct 2, 2024
673611c
feat(repo): add processing for case when value is not found when upda…
raf-nr Oct 2, 2024
4b1fcfa
feat(tests): add tests that check dependencies between modules
raf-nr Oct 2, 2024
e787967
feat(tests): set up the correct system of fixtures with access to sto…
raf-nr Oct 2, 2024
00d24e8
feat(tests): add integration tests for all api usage scenarios
raf-nr Oct 2, 2024
b19ed84
chore: format code, and some documentation and readme files
raf-nr Oct 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions .github/workflows/run-linter-and-tests.yaml
Original file line number Diff line number Diff line change
@@ -1,11 +1,7 @@
name: Lint
on:
pull_request:
branches:
- main
push:
branches:
- main
jobs:
run-linters-and-tests:
runs-on: ubuntu-latest
Expand All @@ -24,6 +20,9 @@ jobs:
- name: Run all linters and formatters
run: make lint

- name: Run type check
run: make check-types

- name: Up all containers
run: docker compose -f dev-docker-compose.yaml up --build --force-recreate --remove-orphans -d

Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ EXPOSE ${APPLICATION_SERVER_PORT}
USER 1001

# Run the uvicorn application server.
CMD exec uvicorn --workers 1 --host 0.0.0.0 --port $APPLICATION_SERVER_PORT app.main:app
CMD exec uvicorn --workers 1 --host 0.0.0.0 --port $APPLICATION_SERVER_PORT internal:app

FROM server-setup-build-stage as install-dependencies-build-stage
# install [tool.poetry.dependencies]
Expand All @@ -93,4 +93,4 @@ FROM server-setup-build-stage as production-image
COPY --chown=python_application:python_application --from=install-dependencies-build-stage /application_root/.venv /application_root/.venv

# Copy application files
COPY --chown=python_application:python_application /app /application_root/app/
COPY --chown=python_application:python_application /internal /application_root/internal/
35 changes: 19 additions & 16 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: env install-deps up open-db revision migrate downgrade worker app init lint test
.PHONY: env install-deps up open-db pg-revision pg-migrate pg-downgrade celery-worker app init lint test check-types

ifeq ($(shell test -e '.env' && echo -n yes), yes)
include .env
Expand Down Expand Up @@ -27,43 +27,46 @@ open-db:
docker exec -it desbordante-postgres psql -d $(POSTGRES_DB) -U $(POSTGRES_USER)

## Create new revision file automatically
revision:
poetry run alembic -c app/settings/alembic.ini revision --autogenerate $(args)
pg-revision:
poetry run alembic -c internal/infrastructure/data_storage/relational/postgres/migrations/alembic.ini revision --autogenerate $(args)

## Make migrations in database
migrate:
poetry run alembic -c app/settings/alembic.ini upgrade $(args)
pg-migrate:
poetry run alembic -c internal/infrastructure/data_storage/relational/postgres/migrations/alembic.ini upgrade $(args)

## Downgrade database
downgrade:
poetry run alembic -c app/settings/alembic.ini downgrade $(args)
pg-downgrade:
poetry run alembic -c internal/infrastructure/data_storage/relational/postgres/migrations/alembic.ini downgrade $(args)

## Run celery worker in watch mode
worker:
watchmedo auto-restart --directory=./ --pattern='*.py' --recursive -- celery -A app.worker worker --loglevel=info --concurrency=1
celery-worker:
watchmedo auto-restart --directory=./ --pattern='*.py' --recursive -- celery -A internal.infrastructure.background_task.celery worker --loglevel=info --concurrency=1

## Run application server in watch mode
app:
poetry run uvicorn --port 8000 app.main:app --reload
poetry run uvicorn --port 8000 internal:app --reload

## Initiate repository
init:
make env install-deps

## Run all formatters and linters in project
lint:
poetry run ruff check tests app \
& poetry run ruff format --check tests app \
& poetry run black --check tests app
poetry run ruff check tests internal \
& poetry run ruff format --check tests internal \
& poetry run black --check tests internal

## Reformat code
format:
poetry run ruff format tests app & poetry run ruff check --fix & poetry run black tests app

poetry run ruff format tests internal & poetry run ruff check --fix & poetry run black tests internal

## Run all tests in project
test:
poetry run pytest -o log_cli=true --verbosity=2 --showlocals --log-cli-level=INFO --test-alembic --cov=app --cov-report term
poetry run pytest -o log_cli=true --verbosity=2 --showlocals --log-cli-level=INFO --cov=internal --cov-report term

## Check all types
check-types:
poetry run pyright .

.DEFAULT_GOAL := help
# See <https://gist.github.com/klmr/575726c7e05d8780505a> for explanation.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Execute `make` to see all available rules with documentation
1. Activate virtual environment: `source .venv/bin/activate`
2. Don't forget to change values in .env
3. Run **development-only** containers: `make up`
4. Run celery worker in watch mode: `make worker`
4. Run celery worker in watch mode: `make celery-worker`
5. Run application in watch mode: `make app`

## Docs
Expand Down
9 changes: 0 additions & 9 deletions app/api/__init__.py

This file was deleted.

49 changes: 0 additions & 49 deletions app/api/dataset.py

This file was deleted.

44 changes: 0 additions & 44 deletions app/api/task.py

This file was deleted.

2 changes: 0 additions & 2 deletions app/db/__init__.py

This file was deleted.

70 changes: 0 additions & 70 deletions app/db/session.py

This file was deleted.

1 change: 0 additions & 1 deletion app/domain/common/README.md

This file was deleted.

18 changes: 0 additions & 18 deletions app/domain/common/optional_model.py

This file was deleted.

1 change: 0 additions & 1 deletion app/domain/file/README.md

This file was deleted.

36 changes: 0 additions & 36 deletions app/domain/file/dataset.py

This file was deleted.

1 change: 0 additions & 1 deletion app/domain/task/README.md

This file was deleted.

32 changes: 0 additions & 32 deletions app/domain/task/__init__.py

This file was deleted.

Loading