Feature: Private Datasets, Vol. 1 #1000

s373r · 2024-12-19T13:54:10Z

Description

Changed

Private Datasets:
- OSO: using user actors / dateset resources that come from the database
  - Thus, any access check relies on real entities
- GQL, added Dataset.visibility() to get the current visibility value
- GQL, added DatasetMut.setVisibility() to be able to change the dataset visibility after it has been created
- Deletion of previously created (and unused) ReBAC-properties and reindexing
- OSO: updating the schema to use identifiers instead of names
- OSO: added resource storage for access speed
- E2E: Using the correct account in multi-tenant mode
  - And also the possibility of set it up
- DatasetOwnershipService: moved to the kamu-dataset crate area & implemented via DatasetEntryServiceImpl
- GQL, DatasetMetadata.currentUpstreamDependencies: indication if datasets not found/not accessed
- GQL, DatasetMetadata.currentDownstreamDependencies: exclude datasets that cannot be accessed
- E2E: added the ability to create an account using CLI

Checklist before requesting a review

* Changes before rebasing * from_catalog_n: add clippy warnings suppression * kamu-adapter-auth-oso: add TODOs * Migrations: re-index ReBAC properties * test_oso: update imports * KamuAuthOso: add TODOs * DatasetActionAuthorizer::check_action_allowed(): add a TODO * DatasetEntryServiceHarness: update for tests * RebacService::{get_account_properties(),get_dataset_properties()}: return idempotency * DatasetEntryRepository::get_dataset_entries(): implement for SQLite & Postgres * RebacRepository::properties_count(): implement for SQLite & Postgres * AccountRepository::get_accounts(): implement for SQLite & Postgres * OsoResourceServiceInMem: handle DatasetLifecycleMessage's * OsoResourceServiceInMem::initialize(): update types * Split OsoResourceHolder to OsoResourceServiceInMem & OsoResourceServiceInitializator * OsoResourceHolder: remove dependency to JOB_KAMU_DATASETS_DATASET_ENTRY_INDEXER * kamu-cli: register DatasetEntryIndexer even if not in workspace * Tests stabilization activities * RebacIndexer: add missed #[interface(dyn InitOnStartup)] * kamu-cli: kamu_auth_rebac_services::register_dependencies() * kamu-cli: kamu_adapter_auth_oso::register_dependencies() * OsoDatasetAuthorizer: integrate OsoResourceHolder * OsoResourceHolder: introduce * DatasetEntryIndexer::index_datasets(): increase log severity * RebacIndexer: introduce * kamu-adapter-auth-oso: update description * RebacServiceImpl: dataset_id_entity -> dataset_entity * test_multi_tenant_rebac_dataset_lifecycle_message_consumer: actualize tests * kamu-adapter-auth-oso: add anonymous() helper * kamu-adapter-auth-oso: use MockDatasetRepositoryWriter * kamu-adapter-auth-oso: actualize tests * DatasetActionAuthorizer, DatasetAction: add oso-related impls * OsoDatasetAuthorizer::get_allowed_actions(): return <HashSet<DatasetAction>, InternalError> * GQL, Dataset::properties(): use kamu_auth_rebac::DatasetProperties * kamu-adapter-auth{,-rebac}: remove experimental crates * OsoDatasetAuthorizer: initial RebacService integration * #[allow(unused_variables)] -> #[expect(unused_variables)] * kamu-adapter-auth: extract * kamu-adapter-rebac: initial * kamu-adapter-oauth, AggregatingDatasetActionAuthorizer: initial * kamu-adapter-graphql, from_catalog_n!(): introduce * test_multi_tenant_rebac_dataset_lifecycle_message_consumer: stabilize tests * SmTP, AxumServerPushProtocolInstance::push_main_flow(): remove extra allocations * Tests, test_gql_datasets: use macros for tests * Tests, test_gql_datasets: expected first * Fixes after rebasing * Tests: update dataset_create_empty_*() * RunInDatabaseTransactionLayer: remove unused * GQL, Datasets: use pretty_assertions::assert_eq!() * GQL, DatasetPropertyName: remove outdated scalar * MultiTenantRebacDatasetLifecycleMessageConsumer::handle_dataset_lifecycle_created_message(): add "allows_anonymous_read" property as well * GQL, Dataset::properties(): return flags for simplicity * DatasetMut::set_visibility(): stabilize * Preparations - DependencyGraphServiceInMemory: remove extra .int_err() calls - Dataset::rebac_properties(): introduce - RebacService::get_dataset_properties(): use DatasetPropertyName instead of PropertyName - kamu-auth-rebac: extract value constants - DatasetMut::{set_publicly_available(),set_anonymous_available()}: ensure account owns dataset - DatasetMut: move to own directory - DatasetMut::{set_publicly_available(),set_anonymous_available()}: hide methods behind logging guards - DatasetMut::set_property(): extract method - DatasetMut::set_anonymous_available(): implement - DatasetMut::set_publicly_available(): implement - RevokeResultSuccess::message(): fix typo * Fixes after rebasing on 0.208.* * Tests, kamu-cli: auto-register e2e-user for the e2e mode * OSO: replace names with IDs in schema * Tests stabilization * sqlx: add cached queries * Build speed-ups: remove unused deps * test_pull_derivative_mt: correct running * CHANGELOG: add some entries * DatasetEntryRepository: simplify lifetimes * kamu-adapter-auth-oso-rebac: add "-rebac" suffix * Remove several TODOs * CHANGELOG.md: add several entries * OsoDatasetAuthorizer: revisit implementation * Review 1: GQL: remove Dataset.properties * Review 1: OsoDatasetAuthorizer::ctor(): fix param name * database-common, EntityStreamer: introduce * DatasetEntryServiceImpl: use EntityStreamer * RebacServiceImpl::get_dataset_properties_by_ids(): add * PaginationOpts::safe_limit(): add * Tests, EntityStreamer: add tests with input data * RebacService::get_dataset_properties_by_ids(): update interface * DatasetEntryServiceImpl: use EntityStreamer [2] * OsoResourceServiceInMem: rewrite to use streamed pages * OsoDatasetAuthorizer: use get_multiple_dataset_resources() * OsoResourceServiceInitializator: remove * query_handler_post(): add a comma in doc * DatasetActionAuthorizer: add TODOs * test_flow_event_store: fix typos * OsoDatasetAuthorizer::user_dataset_pair(): remove * RebacIndexer::index_dataset_entries(): iterate over a stream * EntityStreamer: remove extra int_err() & resort declarations * AccountRepository::get_accounts(): streamed version * RebacIndexer::index_accounts(): use iterate over a stream * Test fixes * RebacRepository::get_entity_properties_by_ids(): implementations * Remove extra as_did_str() call * RebacRepository::get_entity_properties_by_ids(): implementations[2] * AccountRepository::accounts_count(): implementations * PostgresAccountRepository::get_accounts(): implementation * sqlx: update cached queries * RebacRepository::get_entity_properties_by_ids(): implementations[3] * DatasetEntryServiceImpl: use tokio::sync::RwLock * PostgresDatasetEntryRepository: tweaks * EntityStreamer -> EntityPageStreamer * sqlite_generate_placeholders_list: extract & use * OsoResourceServiceInMem: add a TODO about state * Search::query(): use from_catalog_n!() * OsoResourceServiceInMem -> OsoResourceServiceImpl * KamuAuthOso: impl Deref to Arc<Oso> * OsoResourceServiceImpl: concrete error types * kamu-adapter-auth-oso-rebac: remove extra dep * DatasetEntryRepository: use odf namespace * DatasetEntryServiceImpl: use odf namespace * DatasetEntryService::list_entries_owned_by(): do not clone owner_id * DatasetEntryRepository::get_dataset_entries(): update ORDER BY column * EntityListing -> EntityPageListing * Tweaks before merging

#997)

* Do not show usage error for --all flag (#960) * Do not show usage error for --all flag When --all flag is set for the `repo delete` command, and there are no repositories to delete, do not shoow usage error. * Improve args validation * Improve args validation, e2e tests * Typo corrected in feature flags (#974) * Images, kamu-base-git: fix collision of executable files (#975) * 868 api server provide feature flags for UI (#976) Separated runtime and UI configuration flags. UI config is provided by API server too. * Release v0.210.0 + minor deps * 854 persistent storage of dataset dependencies graph (#973) Dependency graph service moved to 'datasets' domain. Defined dataset dependency repository interface and created 3 implementations. No more postponed initialization, organized initial setup in the form of an indexer. Added telemetry extensions on the way. Tests for repositories, stabilized other tests. Cascading effect on delete within the dataset entry domain. * v0.211.0 + minor deps * Fixed image building (#977) Replaced cascade delete of dataset entries in graph with more explicit events to allow orphan upstream dependencies where only ID is given * Upgrade to datafusion 43 * Use thiserror v2 throughout * trust-dns-resolver => hickory-resolver + minor deps * Fix non-sequential offsets on ingest * 0.212.0 * Use KAMU_CONTAINER_RUNTIME_TYPE env var in Makefile (#991) * Use KAMU_CONTAINER_RUNTIME_TYPE env var in Makefile * Make podman default engine for e2e tests * Backporting changes from Private Datasets feature branch (#992) * Backport tweaks * Add doc strings * Remove unused deps * Remove unactual test * CHANGELOG.md: update * Tips after self-review * Delete env var on dataset delete (#993) * Delete env var on dataset delete * 984 refactoring separate planning and execution phases in key dataset manipulation services (#994) * Draft split of `CompactionService` into planner and execution parts * Compaction cleanups * Compacting more cleanups * Compacting: read old HEAD on planning phase * Reset service split on planner and execution * Extracted `MetadataQueryService` - to query polling, push sources and set transform, instead of ingest/transform planners * DataWriterMetadataState became part of polling ingest item at the planning phase * Setting watermark : separate planner and execution service * Push ingest service prepared for split * Push ingest split on planning and executing * Made some order in infra/core services * {Flow,Task,Outbox}Executor=>Agent * Unified naming of planners and executors * Revised telemetry in refactored components * Review: DataWriterDataFusionBuilder flattened * changelog * v0.123.0 + minor deps * kamu-dev-base: include short commit hash as well (#995) * v0.213.1: less agressive telemetry with `DataWriterMetadataState` --------- Co-authored-by: Andrii Demus <[email protected]> Co-authored-by: Sergei Zaychenko <[email protected]> Co-authored-by: Sergii Mikhtoniuk <[email protected]> Co-authored-by: Roman Boiko <[email protected]>

…nted via `DatasetEntryServiceImpl` (#1004) * DatasetOwnershipService: use odf namespace * DatasetEntryServiceImpl: impl DatasetOwnershipService * DatasetOwnershipService: move to kamu-datasets scope * CHANGELOG.md: update

…at are not found (#1013)

* E2E: added the ability to create an account using CLI * OutboxImmediateImpl::post_message_as_json(): return a dispatch error, if present

# Conflicts: # CHANGELOG.md # Cargo.lock # Cargo.toml # LICENSE.txt # resources/openapi-mt.json # resources/openapi.json # src/adapter/auth-oso/Cargo.toml # src/adapter/flight-sql/Cargo.toml # src/adapter/flight-sql/src/lib.rs # src/adapter/flight-sql/src/session_factory.rs # src/adapter/graphql/Cargo.toml # src/adapter/graphql/src/queries/datasets/dataset.rs # src/adapter/graphql/src/queries/datasets/dataset_metadata.rs # src/adapter/graphql/tests/tests/test_gql_account_flow_configs.rs # src/adapter/graphql/tests/tests/test_gql_dataset_flow_runs.rs # src/adapter/http/Cargo.toml # src/adapter/oauth/Cargo.toml # src/adapter/odata/Cargo.toml # src/app/cli/src/app.rs # src/domain/accounts/services/Cargo.toml # src/domain/core/src/services/mod.rs # src/domain/datasets/domain/src/repos/dataset_entry_repository.rs # src/domain/datasets/domain/src/services/dataset_entry_service.rs # src/domain/datasets/services/src/dataset_entry_service_impl.rs # src/domain/flow-system/services/tests/tests/utils/flow_harness_shared.rs # src/infra/accounts/inmem/Cargo.toml # src/infra/accounts/mysql/Cargo.toml # src/infra/accounts/postgres/Cargo.toml # src/infra/accounts/sqlite/Cargo.toml # src/infra/auth-rebac/repo-tests/Cargo.toml # src/infra/core/src/services/dataset_ownership_service_inmem.rs # src/infra/core/src/services/mod.rs # src/infra/core/tests/tests/test_dataset_ownership_service_inmem.rs # src/infra/datasets/inmem/Cargo.toml # src/infra/datasets/inmem/src/repos/inmem_dateset_entry_repository.rs # src/infra/datasets/postgres/.sqlx/query-30c92efe33072f0b9fa446ea3255ffca15f34c2af9aaeb8d31453ab364f97495.json # src/infra/datasets/postgres/Cargo.toml # src/infra/datasets/repo-tests/Cargo.toml # src/infra/datasets/sqlite/Cargo.toml # src/infra/messaging-outbox/inmem/Cargo.toml # src/infra/messaging-outbox/postgres/Cargo.toml # src/infra/messaging-outbox/sqlite/Cargo.toml # src/infra/task-system/inmem/Cargo.toml # src/infra/task-system/postgres/Cargo.toml # src/utils/database-common/Cargo.toml # src/utils/event-sourcing/Cargo.toml # src/utils/init-on-startup/Cargo.toml

# Conflicts: # CHANGELOG.md # Cargo.lock # Cargo.toml # resources/schema.gql # src/adapter/graphql/src/mutations/flows_mut/account_flow_triggers_mut.rs # src/adapter/graphql/src/queries/accounts/account_flow_triggers.rs # src/adapter/graphql/tests/tests/test_gql_datasets.rs # src/app/cli/Cargo.toml # src/domain/core/Cargo.toml # src/domain/flow-system/services/src/flow/flow_scheduling_helper.rs # src/domain/flow-system/services/tests/tests/utils/flow_harness_shared.rs # src/e2e/app/cli/repo-tests/src/test_smart_transfer_protocol.rs

src/adapter/auth-oso-rebac/Cargo.toml

src/adapter/auth-oso-rebac/src/oso_dataset_authorizer.rs

src/adapter/auth-oso-rebac/src/oso_resource_service_impl.rs

src/adapter/graphql/src/mutations/dataset_mut/dataset_mut_utils.rs

src/adapter/graphql/src/queries/datasets/dataset_metadata.rs

src/adapter/graphql/src/queries/datasets/datasets.rs

src/domain/accounts/domain/src/repos/account_repository.rs

src/domain/accounts/domain/src/services/account_service.rs

src/domain/auth-rebac/services/src/rebac_indexer.rs

src/domain/core/src/auth/dataset_action_authorizer.rs

src/domain/core/src/services/dataset_registry.rs

src/domain/datasets/domain/src/services/dataset_entry_service.rs

src/infra/auth-rebac/postgres/src/repos/postgres_rebac_repository.rs

src/infra/datasets/postgres/src/repos/postgres_dataset_entry_repository.rs

* KamuApiServerClient::graphql_api_call_assert_with_token(): remove extra method * KamuApiServerClient: introduce GraphQLResponse type * E2E: DatasetMut::set_visibility()

* DatasetEntryServiceExt: absorb DatasetOwnershipService::get_owned_datasets() * DatasetEntryServiceExt: absorb all rest DatasetOwnershipService * kamu-adapter-auth-oso-rebac: remove duplicate dep * DatasetActionAuthorizer: classify_datasets_by_allowance() -> classify_dataset_handles_by_allowance() * DatasetRegistry: remove an TODO * DatasetEntryRepository::get_dataset_entries(): use dataset_name column for sorting in implementations (as it was) * OsoResourceServiceImpl: state extraction to singleton component * DatasetActionAuthorizer::check_action_allowed(): use DatasetID instead of DatasetHandle * DatasetActionAuthorizer::is_action_allowed(): use DatasetID instead of DatasetHandle * DatasetActionAuthorizer::get_allowed_actions(): use DatasetID instead of DatasetHandle * DatasetActionAuthorizer: finalization * ODataServiceContext::list_collections(): use DatasetActionAuthorizer::filtered_datasets_stream() * Datasets::by_account_impl(): use DatasetActionAuthorizer::filtered_datasets_stream() * Search::query(): use DatasetActionAuthorizer::filtered_datasets_stream() * GetDatasetDownstreamDependenciesUseCase: extract * GetDatasetUpstreamDependenciesUseCase: extract * AccountServiceImpl::all_accounts(): absorb list_all_accounts() method * ExpensiveAccountRepository: extract trait * RebacService::properties_count(): implement * DatasetEntryService: move list-* operations within an implementation

src/domain/auth-rebac/domain/src/services/rebac_service.rs

src/domain/auth-rebac/services/src/rebac_service_impl.rs

src/adapter/graphql/src/mutations/dataset_mut/dataset_mut_utils.rs

src/infra/core/src/use_cases/create_dataset_from_snapshot_use_case_impl.rs

* ensure_account_owns_dataset() -> ensure_account_is_owner_or_admin() * {Account,Dataset}Properties::apply(): add * RebacServiceImpl: inject default properties

zaychenko-sergei

LGTM, assuming:

SM's notes are fixed
latest master merged in
UI is finally re-tested

# Conflicts: # CHANGELOG.md # Cargo.lock # Cargo.toml # src/adapter/auth-oso/Cargo.toml # src/adapter/graphql/Cargo.toml # src/domain/accounts/services/Cargo.toml # src/domain/datasets/services/src/dataset_entry_service_impl.rs # src/domain/datasets/services/tests/tests/test_dataset_entry_service.rs # src/infra/accounts/inmem/Cargo.toml # src/infra/accounts/mysql/Cargo.toml # src/infra/accounts/postgres/Cargo.toml # src/infra/accounts/sqlite/Cargo.toml

s373r and others added 4 commits December 16, 2024 17:49

GQL: Dataset.visibility(): return back, after being deleted by mistake (

99251e7

#997)

Fixes after merging (#999)

34217e9

s373r self-assigned this Dec 19, 2024

s373r added 10 commits December 20, 2024 22:57

GQL, DatasetMut::set_visibility(): correct return type (#1007)

301b5c1

GQL, DatasetMetadata: be prepared for not accessed datasets (#1011)

e1b8804

GQL, DatasetMetadata: correct processing of dataset's dependencies th…

ac122df

…at are not found (#1013)

GQL, DatasetMetadata: update dataset's dependencies types (#1014)

607894d

Private Datasets: absorb helpful commits from command updates (#1016)

4524c6b

* E2E: added the ability to create an account using CLI * OutboxImmediateImpl::post_message_as_json(): return a dispatch error, if present

Fixes after merge

5ada448

GQL: Datasets: auth checks (#1017)

4417cd7

s373r changed the title ~~Feature/private datasets~~ Feature: Private Datasets, Vol. 1 Jan 13, 2025

s373r requested a review from zaychenko-sergei January 13, 2025 16:01