Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Databricks SDK to 0.30 #2429

Closed
wants to merge 4 commits into from
Closed

Upgrade Databricks SDK to 0.30 #2429

wants to merge 4 commits into from

Conversation

HariGS-DB
Copy link
Contributor

Changes

Databricks SDK version 0.30 has fixed the bug due to incorrectly identifying cloud value when using :443 (databricks/databricks-sdk-py#656) or trailing slash (databricks/databricks-sdk-py#661) in the profile.
Upgrading SDK also had cascading effect on other API calls in UCX mainly on redash API. This PR also updates the relevant code.

Resolves #1757

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs ucx ...
  • added a new workflow
  • modified existing workflow: ...
  • added a new table
  • modified existing table: ...

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • verified on staging environment (screenshot attached)

@HariGS-DB HariGS-DB marked this pull request as ready for review August 15, 2024 16:13
@HariGS-DB HariGS-DB requested review from a team and larsgeorge-db August 15, 2024 16:13
@HariGS-DB HariGS-DB requested review from nfx and removed request for larsgeorge-db August 15, 2024 16:13
@HariGS-DB
Copy link
Contributor Author

HariGS-DB commented Aug 15, 2024

Dabricks Profile

With training slash:

[hsdb]
host = adb-2381667944289056.16.azuredatabricks.net/
auth_type = databricks-cli

With token and port appended

[hsdb2]
host = https://adb-2381667944289056.16.azuredatabricks.net:443
token = supersecrettoken

Without slash

[hsdbuc]
host = adb-2381667944289056.16.azuredatabricks.net
auth_type = databricks-cli

Test with SDK 0.29
First 2 profiles fail and the third one succeeds
image

Test with SDK 0.30
All 3 profiles succeed

image

Copy link

github-actions bot commented Aug 15, 2024

❌ 105/108 passed, 3 failed, 20 skipped, 1h24m13s total

❌ test_running_real_workflow_linter_job: databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ValueError: Invalid semantic version: 0.33.1+520240815173351 (3m4.032s)
databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ValueError: Invalid semantic version: 0.33.1+520240815173351
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f/notebook.ipynb
17:32 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/313968170637392
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=313968170637392)
17:32 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_scctb: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_scctb
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_scctb', metastore_id=None, name='ucx_scctb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:32 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.A81q/config.yml) doesn't exist.
17:32 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:32 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:32 INFO [databricks.labs.ucx.install] Fetching installations...
17:32 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:32 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:33 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:33 INFO [databricks.labs.ucx.install] Installing UCX v0.33.1+520240815173351
17:33 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:34 INFO [databricks.labs.ucx.install] Creating dashboards...
17:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
17:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
17:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.A81q/README for the next steps.
17:34 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/219456137051031
17:34 INFO [databricks.labs.ucx.installer.workflows] Started experimental-workflow-linter job: https://DATABRICKS_HOST#job/219456137051031/runs/897590161879203
17:34 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of experimental-workflow-linter job: https://DATABRICKS_HOST#job/219456137051031/runs/897590161879203
17:35 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
17:35 WARNING [databricks.labs.ucx.installer.workflows] Cannot fetch logs as folder /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.A81q/logs/experimental-workflow-linter does not exist
17:35 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ----------
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f/notebook.ipynb
17:32 INFO [databricks.labs.ucx.mixins.fixtures] Job: https://DATABRICKS_HOST#job/313968170637392
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added job fixture: CreateResponse(job_id=313968170637392)
17:32 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_scctb: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_scctb
17:32 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_scctb', metastore_id=None, name='ucx_scctb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:32 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.A81q/config.yml) doesn't exist.
17:32 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:32 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:32 INFO [databricks.labs.ucx.install] Fetching installations...
17:32 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:32 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:33 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:33 INFO [databricks.labs.ucx.install] Installing UCX v0.33.1+520240815173351
17:33 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:33 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:34 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:34 INFO [databricks.labs.ucx.install] Creating dashboards...
17:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
17:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
17:34 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
17:34 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:34 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.A81q/README for the next steps.
17:34 DEBUG [databricks.labs.ucx.installer.workflows] starting experimental-workflow-linter job: https://DATABRICKS_HOST#job/219456137051031
17:34 INFO [databricks.labs.ucx.installer.workflows] Started experimental-workflow-linter job: https://DATABRICKS_HOST#job/219456137051031/runs/897590161879203
17:34 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of experimental-workflow-linter job: https://DATABRICKS_HOST#job/219456137051031/runs/897590161879203
17:35 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
17:35 WARNING [databricks.labs.ucx.installer.workflows] Cannot fetch logs as folder /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.A81q/logs/experimental-workflow-linter does not exist
17:35 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ----------
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 job fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing job fixture: CreateResponse(job_id=313968170637392)
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 directory fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing directory fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 notebook fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing notebook fixture: /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f/notebook.ipynb
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] ignoring error while notebook /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f/notebook.ipynb teardown: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/sdk-WdcJ-ra78a5086f/notebook.ipynb) doesn't exist.
17:35 INFO [databricks.labs.ucx.install] Deleting UCX v0.33.1+520240815173351 from https://DATABRICKS_HOST
17:35 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_scctb
17:35 INFO [databricks.labs.ucx.install] Deleting jobs
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=319440639517984.
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=1063135301695377.
17:35 INFO [databricks.labs.ucx.install] Deleting failing job_id=120519212299081.
17:35 INFO [databricks.labs.ucx.install] Deleting assessment job_id=869737981971918.
17:35 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=344451196894791.
17:35 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=2308339361040.
17:35 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=623071080840829.
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=46295709921641.
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=1011469164260079.
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=513862177827952.
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=781808998984918.
17:35 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=219456137051031.
17:35 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=18654061707357.
17:35 INFO [databricks.labs.ucx.install] Deleting cluster policy
17:35 INFO [databricks.labs.ucx.install] Deleting secret scope
17:35 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace user fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 account group fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 workspace group fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 schema fixtures
17:35 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_scctb', metastore_id=None, name='ucx_scctb', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw9] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_running_real_validate_groups_permissions_job: databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ValueError: Invalid semantic version: 0.33.1+520240815173902 (2m52.407s)
databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ValueError: Invalid semantic version: 0.33.1+520240815173902
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5032049297002924', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-3KK2-ra78a5086f: https://DATABRICKS_HOST#setting/accounts/groups/79540435631632
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-3KK2-ra78a5086f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='79540435631632', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5032049297002924', type=None, value='5032049297002924')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-3KK2-ra78a5086f: https://accounts.CLOUD_ENVdatabricks.net/users/groups/862692540603865/members
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-3KK2-ra78a5086f', entitlements=[], external_id=None, groups=[], id='862692540603865', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5032049297002924', type=None, value='5032049297002924')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swwc4: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swwc4
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swwc4', metastore_id=None, name='ucx_swwc4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_swwc4.ucx_t9hye: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swwc4/ucx_t9hye
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_swwc4.ucx_t9hye', metastore_id=None, name='ucx_t9hye', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_swwc4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_swwc4/ucx_t9hye', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Query Created ucx_query_QIWve: https://DATABRICKS_HOST/sql/editor/175b3498-5fe1-46c8-ae2c-13eb1ef3423f
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added query fixture: LegacyQuery(can_edit=None, created_at='2024-08-15T17:38:49Z', data_source_id=None, description='TEST QUERY FOR UCX', id='175b3498-5fe1-46c8-ae2c-13eb1ef3423f', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QIWve', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/4279257340449065', permission_tier=None, query='SELECT * FROM ucx_swwc4.ucx_t9hye', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-08-15T17:38:49Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[LegacyVisualization(created_at='2024-08-15T17:38:49Z', description='', id='80f23f93-5f34-46fe-aace-30465cd605eb', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-08-15T17:38:49Z')])
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added query permissions fixture: 175b3498-5fe1-46c8-ae2c-13eb1ef3423f [user_name 0a330eb5-dd51-4d97-b6e4-c474356b1d5d CAN_MANAGE, group_name admins CAN_MANAGE] -> [group_name ucx-3KK2-ra78a5086f CAN_EDIT]
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000021495CD37E23
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000021495CD37E23')
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000021495CD37E23 [group_name admins CAN_USE] -> [group_name ucx-3KK2-ra78a5086f CAN_USE]
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sm2g9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm2g9
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm2g9', metastore_id=None, name='ucx_sm2g9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sm2g9.ucx_tztkt: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm2g9/ucx_tztkt
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sm2g9.ucx_tztkt', metastore_id=None, name='ucx_tztkt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_sm2g9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_sm2g9/ucx_tztkt', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-N8qw-ra78a5086f
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-N8qw-ra78a5086f', 'ucx-3KK2-ra78a5086f')
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_ssszx: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_ssszx
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_ssszx', metastore_id=None, name='ucx_ssszx', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:38 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2bZB/config.yml) doesn't exist.
17:38 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:38 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:38 INFO [databricks.labs.ucx.install] Fetching installations...
17:39 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:39 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:39 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:39 INFO [databricks.labs.ucx.install] Installing UCX v0.33.1+520240815173902
17:39 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:39 INFO [databricks.labs.ucx.install] Creating dashboards...
17:39 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
17:39 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
17:39 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2bZB/README for the next steps.
17:39 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.grants] fetching grants inventory
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.grants] crawling new batch for grants
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.tables] fetching tables inventory
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.tables] crawling new batch for tables
17:39 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sm2g9] listing tables and views
17:39 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sm2g9.ucx_tztkt] fetching table metadata
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.tables] found 1 new records for tables
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.udfs] fetching udfs inventory
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.udfs] crawling new batch for udfs
17:39 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_sm2g9] listing udfs
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.udfs] found 0 new records for udfs
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.grants] found 3 new records for grants
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 5
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 5
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.permissions] found 5 new records for permissions
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Saved 5 to hive_metastore.ucx_ssszx.permissions
17:39 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/295304222080452
17:39 INFO [databricks.labs.ucx.installer.workflows] Started validate-groups-permissions job: https://DATABRICKS_HOST#job/295304222080452/runs/832895943963020
17:39 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of validate-groups-permissions job: https://DATABRICKS_HOST#job/295304222080452/runs/832895943963020
17:41 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
17:41 WARNING [databricks.labs.ucx.installer.workflows] Cannot fetch logs as folder /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2bZB/logs/validate-groups-permissions does not exist
17:41 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ----------
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5032049297002924', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-3KK2-ra78a5086f: https://DATABRICKS_HOST#setting/accounts/groups/79540435631632
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-3KK2-ra78a5086f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='79540435631632', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5032049297002924', type=None, value='5032049297002924')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-3KK2-ra78a5086f: https://accounts.CLOUD_ENVdatabricks.net/users/groups/862692540603865/members
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-3KK2-ra78a5086f', entitlements=[], external_id=None, groups=[], id='862692540603865', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5032049297002924', type=None, value='5032049297002924')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_swwc4: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swwc4
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swwc4', metastore_id=None, name='ucx_swwc4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_swwc4.ucx_t9hye: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_swwc4/ucx_t9hye
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_swwc4.ucx_t9hye', metastore_id=None, name='ucx_t9hye', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_swwc4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_swwc4/ucx_t9hye', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Query Created ucx_query_QIWve: https://DATABRICKS_HOST/sql/editor/175b3498-5fe1-46c8-ae2c-13eb1ef3423f
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added query fixture: LegacyQuery(can_edit=None, created_at='2024-08-15T17:38:49Z', data_source_id=None, description='TEST QUERY FOR UCX', id='175b3498-5fe1-46c8-ae2c-13eb1ef3423f', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QIWve', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/4279257340449065', permission_tier=None, query='SELECT * FROM ucx_swwc4.ucx_t9hye', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-08-15T17:38:49Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[LegacyVisualization(created_at='2024-08-15T17:38:49Z', description='', id='80f23f93-5f34-46fe-aace-30465cd605eb', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-08-15T17:38:49Z')])
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added query permissions fixture: 175b3498-5fe1-46c8-ae2c-13eb1ef3423f [user_name 0a330eb5-dd51-4d97-b6e4-c474356b1d5d CAN_MANAGE, group_name admins CAN_MANAGE] -> [group_name ucx-3KK2-ra78a5086f CAN_EDIT]
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/000021495CD37E23
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='000021495CD37E23')
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 000021495CD37E23 [group_name admins CAN_USE] -> [group_name ucx-3KK2-ra78a5086f CAN_USE]
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_sm2g9: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm2g9
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm2g9', metastore_id=None, name='ucx_sm2g9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_sm2g9.ucx_tztkt: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_sm2g9/ucx_tztkt
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sm2g9.ucx_tztkt', metastore_id=None, name='ucx_tztkt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_sm2g9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_sm2g9/ucx_tztkt', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-N8qw-ra78a5086f
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-N8qw-ra78a5086f', 'ucx-3KK2-ra78a5086f')
17:38 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_ssszx: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_ssszx
17:38 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_ssszx', metastore_id=None, name='ucx_ssszx', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:38 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2bZB/config.yml) doesn't exist.
17:38 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:38 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:38 INFO [databricks.labs.ucx.install] Fetching installations...
17:39 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:39 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:39 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:39 INFO [databricks.labs.ucx.install] Installing UCX v0.33.1+520240815173902
17:39 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:39 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:39 INFO [databricks.labs.ucx.install] Creating dashboards...
17:39 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
17:39 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
17:39 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
17:39 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:39 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2bZB/README for the next steps.
17:39 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.grants] fetching grants inventory
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.grants] crawling new batch for grants
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.tables] fetching tables inventory
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.tables] crawling new batch for tables
17:39 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sm2g9] listing tables and views
17:39 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_sm2g9.ucx_tztkt] fetching table metadata
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.tables] found 1 new records for tables
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.udfs] fetching udfs inventory
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.udfs] crawling new batch for udfs
17:39 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_sm2g9] listing udfs
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.udfs] found 0 new records for udfs
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.grants] found 3 new records for grants
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 5
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 5
17:39 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_ssszx.permissions] found 5 new records for permissions
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table
17:39 INFO [databricks.labs.ucx.workspace_access.manager] Saved 5 to hive_metastore.ucx_ssszx.permissions
17:39 DEBUG [databricks.labs.ucx.installer.workflows] starting validate-groups-permissions job: https://DATABRICKS_HOST#job/295304222080452
17:39 INFO [databricks.labs.ucx.installer.workflows] Started validate-groups-permissions job: https://DATABRICKS_HOST#job/295304222080452/runs/832895943963020
17:39 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of validate-groups-permissions job: https://DATABRICKS_HOST#job/295304222080452/runs/832895943963020
17:41 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
17:41 WARNING [databricks.labs.ucx.installer.workflows] Cannot fetch logs as folder /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.2bZB/logs/validate-groups-permissions does not exist
17:41 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ----------
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope acl fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope acl fixture: ('sdk-N8qw-ra78a5086f', 'ucx-3KK2-ra78a5086f')
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope fixture: sdk-N8qw-ra78a5086f
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 000021495CD37E23 [group_name admins CAN_USE] -> [group_name ucx-3KK2-ra78a5086f CAN_USE]
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='000021495CD37E23')
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 query permissions fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing query permissions fixture: 175b3498-5fe1-46c8-ae2c-13eb1ef3423f [user_name 0a330eb5-dd51-4d97-b6e4-c474356b1d5d CAN_MANAGE, group_name admins CAN_MANAGE] -> [group_name ucx-3KK2-ra78a5086f CAN_EDIT]
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 query fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing query fixture: LegacyQuery(can_edit=None, created_at='2024-08-15T17:38:49Z', data_source_id=None, description='TEST QUERY FOR UCX', id='175b3498-5fe1-46c8-ae2c-13eb1ef3423f', is_archived=False, is_draft=False, is_favorite=False, is_safe=True, last_modified_by=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), last_modified_by_id=None, latest_query_data_id=None, name='ucx_query_QIWve', options=QueryOptions(catalog=None, moved_to_trash_at=None, parameters=[], schema=None), parent='folders/4279257340449065', permission_tier=None, query='SELECT * FROM ucx_swwc4.ucx_t9hye', query_hash=None, run_as_role=<RunAsRole.OWNER: 'owner'>, tags=['original_query_tag'], updated_at='2024-08-15T17:38:49Z', user=User(email='0a330eb5-dd51-4d97-b6e4-c474356b1d5d', id=481119220561874, name='labs-account-admin-identity'), user_id=481119220561874, visualizations=[LegacyVisualization(created_at='2024-08-15T17:38:49Z', description='', id='80f23f93-5f34-46fe-aace-30465cd605eb', name='Results', options={'version': 2}, query=None, type='TABLE', updated_at='2024-08-15T17:38:49Z')])
17:41 INFO [databricks.labs.ucx.install] Deleting UCX v0.33.1+520240815173902 from https://DATABRICKS_HOST
17:41 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_ssszx
17:41 INFO [databricks.labs.ucx.install] Deleting jobs
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=827707672310799.
17:41 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=45269814426528.
17:41 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=979815240193798.
17:41 INFO [databricks.labs.ucx.install] Deleting assessment job_id=328181669458800.
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=426710741684864.
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=735658951598954.
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=855343672048462.
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=1055325054919203.
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=805324467290334.
17:41 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=516187731343809.
17:41 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=983710815995026.
17:41 INFO [databricks.labs.ucx.install] Deleting failing job_id=504324774826543.
17:41 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=295304222080452.
17:41 INFO [databricks.labs.ucx.install] Deleting cluster policy
17:41 INFO [databricks.labs.ucx.install] Deleting secret scope
17:41 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='5032049297002924', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx-3KK2-ra78a5086f', entitlements=[], external_id=None, groups=[], id='862692540603865', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5032049297002924', type=None, value='5032049297002924')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx-3KK2-ra78a5086f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='79540435631632', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/5032049297002924', type=None, value='5032049297002924')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 2 table fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_swwc4.ucx_t9hye', metastore_id=None, name='ucx_t9hye', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_swwc4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_swwc4/ucx_t9hye', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_sm2g9.ucx_tztkt', metastore_id=None, name='ucx_tztkt', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_sm2g9', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_sm2g9/ucx_tztkt', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 3 schema fixtures
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_swwc4', metastore_id=None, name='ucx_swwc4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_sm2g9', metastore_id=None, name='ucx_sm2g9', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:41 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_ssszx', metastore_id=None, name='ucx_ssszx', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw7] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
❌ test_running_real_migrate_groups_job: databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ValueError: Invalid semantic version: 0.33.1+520240815174015 (2m23.834s)
databricks.labs.blueprint.parallel.ManyError: Detected 2 failures: ValueError: Invalid semantic version: 0.33.1+520240815174015
17:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python
17:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='8473558542457423', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-B4xt-ra78a5086f: https://DATABRICKS_HOST#setting/accounts/groups/262451636434034
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-B4xt-ra78a5086f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='262451636434034', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/8473558542457423', type=None, value='8473558542457423')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-B4xt-ra78a5086f: https://accounts.CLOUD_ENVdatabricks.net/users/groups/1010137272930156/members
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-B4xt-ra78a5086f', entitlements=[], external_id=None, groups=[], id='1010137272930156', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/8473558542457423', type=None, value='8473558542457423')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001B20EB6D82B6EF
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001B20EB6D82B6EF')
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001B20EB6D82B6EF [group_name admins CAN_USE] -> [group_name ucx-B4xt-ra78a5086f CAN_USE]
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svqe4: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svqe4
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svqe4', metastore_id=None, name='ucx_svqe4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_svqe4.ucx_tflle: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svqe4/ucx_tflle
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_svqe4.ucx_tflle', metastore_id=None, name='ucx_tflle', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_svqe4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_svqe4/ucx_tflle', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-eBfN-ra78a5086f
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-eBfN-ra78a5086f', 'ucx-B4xt-ra78a5086f')
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s6kyn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s6kyn
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s6kyn', metastore_id=None, name='ucx_s6kyn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:40 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/config.yml) doesn't exist.
17:40 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Inventory Database stored in hive_metastore
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Log level
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Number of threads
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Backup prefix
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: �[1mChoose how to map the workspace groups:�[0m
�[1m[0]�[0m �[36mMatch by Name�[0m
�[1m[1]�[0m �[36mApply a Prefix�[0m
�[1m[2]�[0m �[36mApply a Suffix�[0m
�[1m[3]�[0m �[36mMatch by External ID�[0m
�[1m[4]�[0m �[36mRegex Substitution�[0m
�[1m[5]�[0m �[36mRegex Matching�[0m
Enter a number between 0 and 5
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Enter a prefix to add to the workspace group name
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Comma-separated list of workspace group names to migrate. If not specified, we'll use all account-level groups with matching names to workspace-level groups
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Comma-separated list of databases to migrate. If not specified, we'll use all databases in hive_metastore
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Does given workspace 414838440637730 block Internet access?
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Do you want to trigger assessment job after installation?
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Reconciliation threshold, in percentage
17:40 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: No HMS lineage collection init script exists, do you want to create one?
17:40 INFO [databricks.labs.ucx.install] Fetching installations...
17:40 INFO [databricks.labs.blueprint.parallel] finding ll3h installations 100/106, rps: 88.886/sec
17:40 INFO [databricks.labs.blueprint.parallel] finding ll3h installations 106/106, rps: 87.297/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'finding ll3h installations' tasks: 0% results available (0/106). Took 0:00:01.214906
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: �[1mSelect PRO or SERVERLESS SQL warehouse to run assessment dashboards on�[0m
�[1m[0]�[0m �[36m[Create new PRO SQL warehouse]�[0m
�[1m[1]�[0m �[36mDEFAULT Test Warehouse (TEST_DEFAULT_WAREHOUSE_ID, SERVERLESS, RUNNING)�[0m
Enter a number between 0 and 1
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Instance pool id to be set in cluster policy for all workflow clusters
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: We have identified one or more cluster policies set up for an external metastore. Would you like to set UCX to connect to the external metastore?
17:40 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Parallelism for migrating dbfs root delta tables with deep clone
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Min workers for auto-scale job cluster for table migration
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Max workers for auto-scale job cluster for table migration
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Open config file in the browser and continue installing? https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/config.yml
17:40 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:40 INFO [databricks.labs.blueprint.parallel] ensure clusters running 3/3, rps: 25.031/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'ensure clusters running' tasks: 0% results available (0/3). Took 0:00:00.120091
17:40 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:40 INFO [databricks.labs.ucx.install] Installing UCX v0.33.1+520240815174015
17:40 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn database exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.CLOUD_ENV_service_principals table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.clusters table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.global_init_scripts table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.jobs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.pipelines table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.external_locations table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.mounts table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.grants table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.groups table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.tables table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.table_size table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.table_failures table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.workspace_objects table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.permissions table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.submit_runs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.policies table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.migration_status table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.workflow_problems table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.udfs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.logs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.recon_results table exists
17:40 INFO [databricks.labs.blueprint.parallel] deploy tables 10/21, rps: 5.502/sec
17:40 INFO [databricks.labs.blueprint.parallel] deploy tables 20/21, rps: 7.068/sec
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:40 INFO [databricks.labs.blueprint.parallel] deploy tables 21/21, rps: 5.516/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'deploy tables' tasks: 0% results available (0/21). Took 0:00:03.807547
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.grant_detail view matches queries/views/grant_detail.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.objects view matches queries/views/objects.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.table_estimates view matches queries/views/table_estimates.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.misc_patterns view matches queries/views/misc_patterns.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.code_patterns view matches queries/views/code_patterns.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.reconciliation_results view matches queries/views/reconciliation_results.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:40 INFO [databricks.labs.ucx.install] Creating dashboards...
17:40 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
17:40 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
17:40 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.blueprint.parallel] installing dashboards 6/6, rps: 1.059/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'installing dashboards' tasks: 0% results available (0/6). Took 0:00:05.664688
17:40 INFO [databricks.labs.blueprint.parallel] installing components 2/2, rps: 0.128/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'installing components' tasks: 0% results available (0/2). Took 0:00:15.595448
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Open job overview in your browser? https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/README
17:40 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/README for the next steps.
17:40 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.grants] fetching grants inventory
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.grants] crawling new batch for grants
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.tables] fetching tables inventory
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.tables] crawling new batch for tables
17:40 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svqe4] listing tables and views
17:40 INFO [databricks.labs.blueprint.parallel] listing tables 1/1, rps: 1.615/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'listing tables' tasks: 100% results available (1/1). Took 0:00:00.619678
17:40 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svqe4.ucx_tflle] fetching table metadata
17:40 INFO [databricks.labs.blueprint.parallel] describing tables in hive_metastore 1/1, rps: 1.771/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'describing tables in hive_metastore' tasks: 100% results available (1/1). Took 0:00:00.564833
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.tables] found 1 new records for tables
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.udfs] fetching udfs inventory
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.udfs] crawling new batch for udfs
17:40 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_svqe4] listing udfs
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.udfs] found 0 new records for udfs
17:41 INFO [databricks.labs.blueprint.parallel] listing grants for hive_metastore 5/5, rps: 0.293/sec
17:41 INFO [databricks.labs.blueprint.parallel] Finished 'listing grants for hive_metastore' tasks: 100% results available (5/5). Took 0:00:17.086134
17:41 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.grants] found 3 new records for grants
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 4
17:41 INFO [databricks.labs.blueprint.parallel] crawl permissions 4/4, rps: 13.704/sec
17:41 INFO [databricks.labs.blueprint.parallel] Finished 'crawl permissions' tasks: 100% results available (4/4). Took 0:00:00.292250
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 4
17:41 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.permissions] found 4 new records for permissions
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Saved 4 to hive_metastore.ucx_s6kyn.permissions
17:41 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups job: https://DATABRICKS_HOST#job/361746715374039
17:41 INFO [databricks.labs.ucx.installer.workflows] Started migrate-groups job: https://DATABRICKS_HOST#job/361746715374039/runs/1081351938261402
17:41 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of migrate-groups job: https://DATABRICKS_HOST#job/361746715374039/runs/1081351938261402
17:42 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
17:42 WARNING [databricks.labs.ucx.installer.workflows] Cannot fetch logs as folder /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/logs/migrate-groups does not exist
17:42 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ----------
17:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:40 INFO [databricks.sdk] Using Databricks Metadata Service authentication
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='8473558542457423', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Workspace group ucx-B4xt-ra78a5086f: https://DATABRICKS_HOST#setting/accounts/groups/262451636434034
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added workspace group fixture: Group(display_name='ucx-B4xt-ra78a5086f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='262451636434034', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/8473558542457423', type=None, value='8473558542457423')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Account group ucx-B4xt-ra78a5086f: https://accounts.CLOUD_ENVdatabricks.net/users/groups/1010137272930156/members
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added account group fixture: Group(display_name='ucx-B4xt-ra78a5086f', entitlements=[], external_id=None, groups=[], id='1010137272930156', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/8473558542457423', type=None, value='8473558542457423')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Cluster policy: https://DATABRICKS_HOST#setting/clusters/cluster-policies/view/001B20EB6D82B6EF
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster policy fixture: CreatePolicyResponse(policy_id='001B20EB6D82B6EF')
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added cluster_policy permissions fixture: 001B20EB6D82B6EF [group_name admins CAN_USE] -> [group_name ucx-B4xt-ra78a5086f CAN_USE]
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_svqe4: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svqe4
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svqe4', metastore_id=None, name='ucx_svqe4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Table hive_metastore.ucx_svqe4.ucx_tflle: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_svqe4/ucx_tflle
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_svqe4.ucx_tflle', metastore_id=None, name='ucx_tflle', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_svqe4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_svqe4/ucx_tflle', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope fixture: sdk-eBfN-ra78a5086f
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added secret scope acl fixture: ('sdk-eBfN-ra78a5086f', 'ucx-B4xt-ra78a5086f')
17:40 INFO [databricks.labs.ucx.mixins.fixtures] Schema hive_metastore.ucx_s6kyn: https://DATABRICKS_HOST/explore/data/hive_metastore/ucx_s6kyn
17:40 DEBUG [databricks.labs.ucx.mixins.fixtures] added schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s6kyn', metastore_id=None, name='ucx_s6kyn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:40 DEBUG [databricks.labs.ucx.install] Cannot find previous installation: Path (/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/config.yml) doesn't exist.
17:40 INFO [databricks.labs.ucx.install] Please answer a couple of questions to configure Unity Catalog migration
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Inventory Database stored in hive_metastore
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Log level
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Number of threads
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Backup prefix
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: �[1mChoose how to map the workspace groups:�[0m
�[1m[0]�[0m �[36mMatch by Name�[0m
�[1m[1]�[0m �[36mApply a Prefix�[0m
�[1m[2]�[0m �[36mApply a Suffix�[0m
�[1m[3]�[0m �[36mMatch by External ID�[0m
�[1m[4]�[0m �[36mRegex Substitution�[0m
�[1m[5]�[0m �[36mRegex Matching�[0m
Enter a number between 0 and 5
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Enter a prefix to add to the workspace group name
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Comma-separated list of workspace group names to migrate. If not specified, we'll use all account-level groups with matching names to workspace-level groups
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Comma-separated list of databases to migrate. If not specified, we'll use all databases in hive_metastore
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Does given workspace 414838440637730 block Internet access?
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Do you want to trigger assessment job after installation?
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Reconciliation threshold, in percentage
17:40 INFO [databricks.labs.ucx.installer.hms_lineage] HMS Lineage feature creates one system table named system.hms_to_uc_migration.table_access and helps in your migration process from HMS to UC by allowing you to programmatically query HMS lineage data.
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: No HMS lineage collection init script exists, do you want to create one?
17:40 INFO [databricks.labs.ucx.install] Fetching installations...
17:40 INFO [databricks.labs.blueprint.parallel] finding ll3h installations 100/106, rps: 88.886/sec
17:40 INFO [databricks.labs.blueprint.parallel] finding ll3h installations 106/106, rps: 87.297/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'finding ll3h installations' tasks: 0% results available (0/106). Took 0:00:01.214906
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: �[1mSelect PRO or SERVERLESS SQL warehouse to run assessment dashboards on�[0m
�[1m[0]�[0m �[36m[Create new PRO SQL warehouse]�[0m
�[1m[1]�[0m �[36mDEFAULT Test Warehouse (TEST_DEFAULT_WAREHOUSE_ID, SERVERLESS, RUNNING)�[0m
Enter a number between 0 and 1
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Instance pool id to be set in cluster policy for all workflow clusters
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: We have identified one or more cluster policies set up for an external metastore. Would you like to set UCX to connect to the external metastore?
17:40 INFO [databricks.labs.ucx.installer.policy] Creating UCX cluster policy.
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Parallelism for migrating dbfs root delta tables with deep clone
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Min workers for auto-scale job cluster for table migration
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Max workers for auto-scale job cluster for table migration
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Open config file in the browser and continue installing? https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/config.yml
17:40 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:40 INFO [databricks.labs.blueprint.parallel] ensure clusters running 3/3, rps: 25.031/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'ensure clusters running' tasks: 0% results available (0/3). Took 0:00:00.120091
17:40 DEBUG [tests.integration.conftest] Waiting for clusters to start...
17:40 INFO [databricks.labs.ucx.install] Installing UCX v0.33.1+520240815174015
17:40 INFO [databricks.labs.ucx.install] Creating ucx schemas...
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn database exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.CLOUD_ENV_service_principals table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.clusters table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.global_init_scripts table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.jobs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.pipelines table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.external_locations table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.mounts table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.grants table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.groups table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.tables table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.table_size table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.table_failures table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.workspace_objects table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.permissions table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.submit_runs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.policies table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.migration_status table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.workflow_problems table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.udfs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.logs table exists
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.recon_results table exists
17:40 INFO [databricks.labs.blueprint.parallel] deploy tables 10/21, rps: 5.502/sec
17:40 INFO [databricks.labs.blueprint.parallel] deploy tables 20/21, rps: 7.068/sec
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=validate-groups-permissions
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups
17:40 INFO [databricks.labs.blueprint.parallel] deploy tables 21/21, rps: 5.516/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'deploy tables' tasks: 0% results available (0/21). Took 0:00:03.807547
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.grant_detail view matches queries/views/grant_detail.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.objects view matches queries/views/objects.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=remove-workspace-local-backup-groups
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-tables-ctas
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.table_estimates view matches queries/views/table_estimates.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=experimental-workflow-linter
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-data-reconciliation
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.misc_patterns view matches queries/views/misc_patterns.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=scan-tables-in-mounts-experimental
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-groups-experimental
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.code_patterns view matches queries/views/code_patterns.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-external-hiveserde-tables-in-place-experimental
17:40 INFO [databricks.labs.lsql.deployment] Ensuring ucx_s6kyn.reconciliation_results view matches queries/views/reconciliation_results.sql contents
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=migrate-tables-in-mounts-experimental
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=assessment
17:40 INFO [databricks.labs.ucx.install] Creating dashboards...
17:40 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment...
17:40 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration...
17:40 DEBUG [databricks.labs.ucx.install] Reading step folder /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/views...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/CLOUD_ENV...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/interactive...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/estimates...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/assessment/main...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/groups...
17:40 INFO [databricks.labs.ucx.install] Creating dashboard in /home/runner/work/ucx/ucx/src/databricks/labs/ucx/queries/migration/main...
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.workflows] Creating new job configuration for step=failing
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.ucx.installer.mixins] Fetching warehouse_id from a config
17:40 INFO [databricks.labs.blueprint.parallel] installing dashboards 6/6, rps: 1.059/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'installing dashboards' tasks: 0% results available (0/6). Took 0:00:05.664688
17:40 INFO [databricks.labs.blueprint.parallel] installing components 2/2, rps: 0.128/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'installing components' tasks: 0% results available (0/2). Took 0:00:15.595448
17:40 INFO [databricks.labs.blueprint.tui] Asking prompt: Open job overview in your browser? https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/README
17:40 INFO [databricks.labs.ucx.install] Installation completed successfully! Please refer to the https://DATABRICKS_HOST/#workspace/Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/README for the next steps.
17:40 DEBUG [databricks.labs.ucx.workspace_access.manager] Crawling permissions
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.grants] fetching grants inventory
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.grants] crawling new batch for grants
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.tables] fetching tables inventory
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.tables] crawling new batch for tables
17:40 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svqe4] listing tables and views
17:40 INFO [databricks.labs.blueprint.parallel] listing tables 1/1, rps: 1.615/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'listing tables' tasks: 100% results available (1/1). Took 0:00:00.619678
17:40 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.ucx_svqe4.ucx_tflle] fetching table metadata
17:40 INFO [databricks.labs.blueprint.parallel] describing tables in hive_metastore 1/1, rps: 1.771/sec
17:40 INFO [databricks.labs.blueprint.parallel] Finished 'describing tables in hive_metastore' tasks: 100% results available (1/1). Took 0:00:00.564833
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.tables] found 1 new records for tables
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.udfs] fetching udfs inventory
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.udfs] crawling new batch for udfs
17:40 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.ucx_svqe4] listing udfs
17:40 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.udfs] found 0 new records for udfs
17:41 INFO [databricks.labs.blueprint.parallel] listing grants for hive_metastore 5/5, rps: 0.293/sec
17:41 INFO [databricks.labs.blueprint.parallel] Finished 'listing grants for hive_metastore' tasks: 100% results available (5/5). Took 0:00:17.086134
17:41 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.grants] found 3 new records for grants
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Starting to crawl permissions. Total tasks: 4
17:41 INFO [databricks.labs.blueprint.parallel] crawl permissions 4/4, rps: 13.704/sec
17:41 INFO [databricks.labs.blueprint.parallel] Finished 'crawl permissions' tasks: 100% results available (4/4). Took 0:00:00.292250
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Total crawled permissions: 4
17:41 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.ucx_s6kyn.permissions] found 4 new records for permissions
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Successfully saved the items to inventory table
17:41 INFO [databricks.labs.ucx.workspace_access.manager] Saved 4 to hive_metastore.ucx_s6kyn.permissions
17:41 DEBUG [databricks.labs.ucx.installer.workflows] starting migrate-groups job: https://DATABRICKS_HOST#job/361746715374039
17:41 INFO [databricks.labs.ucx.installer.workflows] Started migrate-groups job: https://DATABRICKS_HOST#job/361746715374039/runs/1081351938261402
17:41 DEBUG [databricks.labs.ucx.installer.workflows] Waiting for completion of migrate-groups job: https://DATABRICKS_HOST#job/361746715374039/runs/1081351938261402
17:42 INFO [databricks.labs.ucx.installer.workflows] ---------- REMOTE LOGS --------------
17:42 WARNING [databricks.labs.ucx.installer.workflows] Cannot fetch logs as folder /Users/0a330eb5-dd51-4d97-b6e4-c474356b1d5d/.ll3h/logs/migrate-groups does not exist
17:42 INFO [databricks.labs.ucx.installer.workflows] ---------- END REMOTE LOGS ----------
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope acl fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope acl fixture: ('sdk-eBfN-ra78a5086f', 'ucx-B4xt-ra78a5086f')
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 secret scope fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing secret scope fixture: sdk-eBfN-ra78a5086f
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster_policy permissions fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster_policy permissions fixture: 001B20EB6D82B6EF [group_name admins CAN_USE] -> [group_name ucx-B4xt-ra78a5086f CAN_USE]
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 cluster policy fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing cluster policy fixture: CreatePolicyResponse(policy_id='001B20EB6D82B6EF')
17:42 INFO [databricks.labs.blueprint.tui] Asking prompt: Do you want to uninstall ucx from the workspace too, this would remove ucx project folder, dashboards, queries and jobs
17:42 INFO [databricks.labs.ucx.install] Deleting UCX v0.33.1+520240815174015 from https://DATABRICKS_HOST
17:42 INFO [databricks.labs.blueprint.tui] Asking prompt: Do you want to delete the inventory database ucx_s6kyn too?
17:42 INFO [databricks.labs.ucx.install] Deleting inventory database ucx_s6kyn
17:42 INFO [databricks.labs.lsql.deployment] deleting ucx_s6kyn database
17:42 INFO [databricks.labs.ucx.install] Deleting jobs
17:42 INFO [databricks.labs.ucx.install] Deleting validate-groups-permissions job_id=514416723054201.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-groups job_id=361746715374039.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-tables job_id=556306115331532.
17:42 INFO [databricks.labs.ucx.install] Deleting remove-workspace-local-backup-groups job_id=736235127100038.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-external-tables-ctas job_id=615351235620266.
17:42 INFO [databricks.labs.ucx.install] Deleting experimental-workflow-linter job_id=151084152455259.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-data-reconciliation job_id=960914579213389.
17:42 INFO [databricks.labs.ucx.install] Deleting scan-tables-in-mounts-experimental job_id=741033002938362.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-groups-experimental job_id=305664556188867.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-external-hiveserde-tables-in-place-experimental job_id=555729987731534.
17:42 INFO [databricks.labs.ucx.install] Deleting migrate-tables-in-mounts-experimental job_id=949048338510418.
17:42 INFO [databricks.labs.ucx.install] Deleting assessment job_id=920449804759317.
17:42 INFO [databricks.labs.ucx.install] Deleting failing job_id=670991966632101.
17:42 INFO [databricks.labs.ucx.install] Deleting cluster policy
17:42 INFO [databricks.labs.ucx.install] Deleting secret scope
17:42 INFO [databricks.labs.ucx.install] UnInstalling UCX complete
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace user fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace user fixture: User(active=True, display_name='[email protected]', emails=[ComplexValue(display=None, primary=True, ref=None, type='work', value='[email protected]')], entitlements=[], external_id=None, groups=[], id='8473558542457423', name=Name(family_name=None, given_name='[email protected]'), roles=[], schemas=[<UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_USER: 'urn:ietf:params:scim:schemas:core:2.0:User'>, <UserSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_EXTENSION_WORKSPACE_2_0_USER: 'urn:ietf:params:scim:schemas:extension:workspace:2.0:User'>], user_name='[email protected]')
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 account group fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing account group fixture: Group(display_name='ucx-B4xt-ra78a5086f', entitlements=[], external_id=None, groups=[], id='1010137272930156', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/8473558542457423', type=None, value='8473558542457423')], meta=None, roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 workspace group fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing workspace group fixture: Group(display_name='ucx-B4xt-ra78a5086f', entitlements=[ComplexValue(display=None, primary=None, ref=None, type=None, value='allow-cluster-create')], external_id=None, groups=[], id='262451636434034', members=[ComplexValue(display='[email protected]', primary=None, ref='Users/8473558542457423', type=None, value='8473558542457423')], meta=ResourceMeta(resource_type='WorkspaceGroup'), roles=[], schemas=[<GroupSchema.URN_IETF_PARAMS_SCIM_SCHEMAS_CORE_2_0_GROUP: 'urn:ietf:params:scim:schemas:core:2.0:Group'>])
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 0 table fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 1 table fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing table fixture: TableInfo(access_point=None, browse_only=None, catalog_name='hive_metastore', columns=None, comment=None, created_at=None, created_by=None, data_access_configuration_id=None, data_source_format=<DataSourceFormat.DELTA: 'DELTA'>, deleted_at=None, delta_runtime_properties_kvpairs=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, encryption_details=None, full_name='hive_metastore.ucx_svqe4.ucx_tflle', metastore_id=None, name='ucx_tflle', owner=None, pipeline_id=None, properties={'RemoveAfter': '2024081519'}, row_filter=None, schema_name='ucx_svqe4', sql_path=None, storage_credential_name=None, storage_location='dbfs:/user/hive/warehouse/ucx_svqe4/ucx_tflle', table_constraints=None, table_id=None, table_type=<TableType.MANAGED: 'MANAGED'>, updated_at=None, updated_by=None, view_definition=None, view_dependencies=None)
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] clearing 2 schema fixtures
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_svqe4', metastore_id=None, name='ucx_svqe4', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
17:42 DEBUG [databricks.labs.ucx.mixins.fixtures] removing schema fixture: SchemaInfo(browse_only=None, catalog_name='hive_metastore', catalog_type=None, comment=None, created_at=None, created_by=None, effective_predictive_optimization_flag=None, enable_predictive_optimization=None, full_name='hive_metastore.ucx_s6kyn', metastore_id=None, name='ucx_s6kyn', owner=None, properties=None, schema_id=None, storage_location=None, storage_root=None, updated_at=None, updated_by=None)
[gw3] linux -- Python 3.10.14 /home/runner/work/ucx/ucx/.venv/bin/python

Running from acceptance #5232

@nfx
Copy link
Collaborator

nfx commented Aug 16, 2024

can you finish this PR instead - #2417

@nfx nfx closed this Aug 16, 2024
@nfx nfx deleted the bug/1757 branch October 16, 2024 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG]: ucx install Fails on Azure with AWS-linked validation error
2 participants