-
Notifications
You must be signed in to change notification settings - Fork 127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make a best effort attempt to initialise all Databricks globals #562
Conversation
7ed6cd4
to
ebe0fc0
Compare
1cd1c48
to
7296d83
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #562 +/- ##
==========================================
+ Coverage 57.91% 58.05% +0.14%
==========================================
Files 45 44 -1
Lines 28291 29738 +1447
==========================================
+ Hits 16385 17265 +880
- Misses 11906 12473 +567 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for this! Few suggestions, but looks good overall. Can merge once these comments are addressed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really like this change.
This means I can always just call from databricks.sdk.runtime import *
from any code and it will work identical in the IDE and in notebooks.
Co-authored-by: Miles Yucht <[email protected]> Signed-off-by: Kartik Gupta <[email protected]>
Co-authored-by: Miles Yucht <[email protected]> Signed-off-by: Kartik Gupta <[email protected]>
…icks-sdk-py into all-databricks-globals
Co-authored-by: Miles Yucht <[email protected]> Signed-off-by: Kartik Gupta <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, LGTM with two small nits.
* Fix get_workspace_client in GCP ([#532](#532)). * Fix integer deserialization for headers ([#553](#553)). * Add Files API docs to the SDK Documentation ([#556](#556)). * Add back enums to docs ([#557](#557)). * Sort index pages by name in docs ([#560](#560)). * Add integration tests for Files API ([#552](#552)). * Distinguish between empty types and fields that can take any value ([#561](#561)). * Support subservices ([#559](#559)). * Use all-apis scope with external-browser ([#563](#563)). * Make a best effort attempt to initialise all Databricks globals ([#562](#562)). * Added tokei.rs badge ([#567](#567)). * New example to list compute resource for SUBMIT_RUN job runs ([#572](#572)). * Update SDK to latest OpenAPI spec ([#576](#576)). API Changes: * Added [w.permission_migration](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/permission_migration.html) workspace-level service. * Added `databricks.sdk.service.iam.PermissionMigrationRequest` dataclass. * Added `databricks.sdk.service.iam.PermissionMigrationResponse` dataclass. * Changed `version` field for `databricks.sdk.service.serving.AppManifest` to `databricks.sdk.service.serving.AnyValue` dataclass. * Removed `delete_personal_compute_setting()` method for [a.account_settings](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_settings.html) account-level service. * Removed `get_personal_compute_setting()` method for [a.account_settings](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_settings.html) account-level service. * Removed `update_personal_compute_setting()` method for [a.account_settings](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_settings.html) account-level service. * Removed `delete_default_namespace_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. * Removed `delete_restrict_workspace_admins_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. * Removed `get_default_namespace_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. * Removed `get_restrict_workspace_admins_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. * Removed `update_default_namespace_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. * Removed `update_restrict_workspace_admins_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. * Added [w.automatic_cluster_update](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/automatic_cluster_update.html) workspace-level service. * Added [w.csp_enablement](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/csp_enablement.html) workspace-level service. * Added [a.csp_enablement_account](https://databricks-sdk-py.readthedocs.io/en/latest/account/csp_enablement_account.html) account-level service. * Added [w.default_namespace](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/default_namespace.html) workspace-level service. * Added [w.esm_enablement](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/esm_enablement.html) workspace-level service. * Added [a.esm_enablement_account](https://databricks-sdk-py.readthedocs.io/en/latest/account/esm_enablement_account.html) account-level service. * Added [a.personal_compute](https://databricks-sdk-py.readthedocs.io/en/latest/account/personal_compute.html) account-level service. * Added [w.restrict_workspace_admins](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/restrict_workspace_admins.html) workspace-level service. * Removed `databricks.sdk.service.settings.DeleteDefaultNamespaceSettingRequest` dataclass. * Removed `databricks.sdk.service.settings.DeletePersonalComputeSettingRequest` dataclass. * Removed `databricks.sdk.service.settings.DeleteRestrictWorkspaceAdminsSettingRequest` dataclass. * Removed `databricks.sdk.service.settings.GetDefaultNamespaceSettingRequest` dataclass. * Removed `databricks.sdk.service.settings.GetPersonalComputeSettingRequest` dataclass. * Removed `databricks.sdk.service.settings.GetRestrictWorkspaceAdminsSettingRequest` dataclass. * Added `aws_stable_ip_rule` field for `databricks.sdk.service.settings.NccEgressDefaultRules`. * Added `databricks.sdk.service.settings.AutomaticClusterUpdateSetting` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessage` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessageEnablementDetails` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindow` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowDayOfWeek` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayBasedSchedule` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayFrequency` dataclass. * Added `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWindowStartTime` dataclass. * Added `databricks.sdk.service.settings.ComplianceStandard` dataclass. * Added `databricks.sdk.service.settings.CspEnablement` dataclass. * Added `databricks.sdk.service.settings.CspEnablementAccount` dataclass. * Added `databricks.sdk.service.settings.CspEnablementAccountSetting` dataclass. * Added `databricks.sdk.service.settings.CspEnablementSetting` dataclass. * Added `databricks.sdk.service.settings.DeleteDefaultNamespaceRequest` dataclass. * Added `databricks.sdk.service.settings.DeletePersonalComputeRequest` dataclass. * Added `databricks.sdk.service.settings.DeleteRestrictWorkspaceAdminRequest` dataclass. * Added `databricks.sdk.service.settings.EsmEnablement` dataclass. * Added `databricks.sdk.service.settings.EsmEnablementAccount` dataclass. * Added `databricks.sdk.service.settings.EsmEnablementAccountSetting` dataclass. * Added `databricks.sdk.service.settings.EsmEnablementSetting` dataclass. * Added `databricks.sdk.service.settings.GetAutomaticClusterUpdateRequest` dataclass. * Added `databricks.sdk.service.settings.GetCspEnablementAccountRequest` dataclass. * Added `databricks.sdk.service.settings.GetCspEnablementRequest` dataclass. * Added `databricks.sdk.service.settings.GetDefaultNamespaceRequest` dataclass. * Added `databricks.sdk.service.settings.GetEsmEnablementAccountRequest` dataclass. * Added `databricks.sdk.service.settings.GetEsmEnablementRequest` dataclass. * Added `databricks.sdk.service.settings.GetPersonalComputeRequest` dataclass. * Added `databricks.sdk.service.settings.GetRestrictWorkspaceAdminRequest` dataclass. * Added `databricks.sdk.service.settings.NccAwsStableIpRule` dataclass. * Added `databricks.sdk.service.settings.UpdateAutomaticClusterUpdateSettingRequest` dataclass. * Added `databricks.sdk.service.settings.UpdateCspEnablementAccountSettingRequest` dataclass. * Added `databricks.sdk.service.settings.UpdateCspEnablementSettingRequest` dataclass. * Added `databricks.sdk.service.settings.UpdateEsmEnablementAccountSettingRequest` dataclass. * Added `databricks.sdk.service.settings.UpdateEsmEnablementSettingRequest` dataclass. * Removed `planning_phases` field for `databricks.sdk.service.sql.QueryMetrics`. * Changed `delete_endpoint()` method for [w.vector_search_endpoints](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_endpoints.html) workspace-level service with new required argument order. * Changed `create_index()` method for [w.vector_search_indexes](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_indexes.html) workspace-level service with new required argument order. * Changed `delete_data_vector_index()` method for [w.vector_search_indexes](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_indexes.html) workspace-level service with new required argument order. * Changed `upsert_data_vector_index()` method for [w.vector_search_indexes](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_indexes.html) workspace-level service with new required argument order. * Changed `endpoint_name` field for `databricks.sdk.service.vectorsearch.CreateVectorIndexRequest` to be required. * Removed `name` field for `databricks.sdk.service.vectorsearch.DeleteDataVectorIndexRequest`. * Added `index_name` field for `databricks.sdk.service.vectorsearch.DeleteDataVectorIndexRequest`. * Removed `name` field for `databricks.sdk.service.vectorsearch.DeleteEndpointRequest`. * Removed `databricks.sdk.service.vectorsearch.EmbeddingConfig` dataclass. * Removed `embedding_config` field for `databricks.sdk.service.vectorsearch.EmbeddingSourceColumn`. * Added `embedding_model_endpoint_name` field for `databricks.sdk.service.vectorsearch.EmbeddingSourceColumn`. * Added `score_threshold` field for `databricks.sdk.service.vectorsearch.QueryVectorIndexRequest`. * Removed `name` field for `databricks.sdk.service.vectorsearch.UpsertDataVectorIndexRequest`. * Added `index_name` field for `databricks.sdk.service.vectorsearch.UpsertDataVectorIndexRequest`. * Removed `delta_sync_vector_index_spec` field for `databricks.sdk.service.vectorsearch.VectorIndex`. * Removed `direct_access_vector_index_spec` field for `databricks.sdk.service.vectorsearch.VectorIndex`. * Added `delta_sync_index_spec` field for `databricks.sdk.service.vectorsearch.VectorIndex`. * Added `direct_access_index_spec` field for `databricks.sdk.service.vectorsearch.VectorIndex`. OpenAPI SHA: 1026b998b14fba1b8317528f47778240dc4e9a5d, Date: 2024-03-06
### New Features and Improvements * Fixed get_workspace_client in GCP ([#532](#532)). * Use all-apis scope with external-browser ([#563](#563)). * Make a best effort attempt to initialise all Databricks globals ([#562](#562)). * Fixed type issue with widgets.getArgument ([#581](#581)) * Note: Backwards incompatible changes - Settings are now nested, please see the API changes below. ### Documentation * Added Files API docs to the SDK Documentation ([#556](#556)). * Added new example to list compute resource for SUBMIT_RUN job runs ([#572](#572)). * Sorted index pages by name in docs ([#560](#560)). * Added back enums to docs ([#557](#557)). ### API Changes #### Added Services: - [w.permission_migration](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/permission_migration.html) workspace-level service. - [w.settings.automatic_cluster_update](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/automatic_cluster_update.html) workspace-level service. - [w.settings.csp_enablement](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/csp_enablement.html) workspace-level service. - [a.settings.csp_enablement_account](https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/csp_enablement_account.html) account-level service. - [w.settings.default_namespace](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/default_namespace.html) workspace-level service. - [w.settings.esm_enablement](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/esm_enablement.html) workspace-level service. - [a.settings.esm_enablement_account](https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/esm_enablement_account.html) account-level service. - [a.settings.personal_compute](https://databricks-sdk-py.readthedocs.io/en/latest/account/settings/personal_compute.html) account-level service. - [w.settings.restrict_workspace_admins](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings/restrict_workspace_admins.html) workspace-level service. Dataclasses: - `databricks.sdk.service.settings.AutomaticClusterUpdateSetting` - `databricks.sdk.service.settings.ClusterAutoRestartMessage` - `databricks.sdk.service.settings.ClusterAutoRestartMessageEnablementDetails` - `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindow` - `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowDayOfWeek` - `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayBasedSchedule` - `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWeekDayFrequency` - `databricks.sdk.service.settings.ClusterAutoRestartMessageMaintenanceWindowWindowStartTime` - `databricks.sdk.service.settings.ComplianceStandard` - `databricks.sdk.service.settings.CspEnablement` - `databricks.sdk.service.settings.CspEnablementAccount` - `databricks.sdk.service.settings.CspEnablementAccountSetting` - `databricks.sdk.service.settings.CspEnablementSetting` - `databricks.sdk.service.settings.DeleteDefaultNamespaceRequest` - `databricks.sdk.service.settings.DeletePersonalComputeRequest` - `databricks.sdk.service.settings.DeleteRestrictWorkspaceAdminRequest` - `databricks.sdk.service.settings.EsmEnablement` - `databricks.sdk.service.settings.EsmEnablementAccount` - `databricks.sdk.service.settings.EsmEnablementAccountSetting` - `databricks.sdk.service.settings.EsmEnablementSetting` - `databricks.sdk.service.settings.GetAutomaticClusterUpdateRequest` - `databricks.sdk.service.settings.GetCspEnablementAccountRequest` - `databricks.sdk.service.settings.GetCspEnablementRequest` - `databricks.sdk.service.settings.GetDefaultNamespaceRequest` - `databricks.sdk.service.settings.GetEsmEnablementAccountRequest` - `databricks.sdk.service.settings.GetEsmEnablementRequest` - `databricks.sdk.service.settings.GetPersonalComputeRequest` - `databricks.sdk.service.settings.GetRestrictWorkspaceAdminRequest` - `databricks.sdk.service.settings.NccAwsStableIpRule` - `databricks.sdk.service.settings.UpdateAutomaticClusterUpdateSettingRequest` - `databricks.sdk.service.settings.UpdateCspEnablementAccountSettingRequest` - `databricks.sdk.service.settings.UpdateCspEnablementSettingRequest` - `databricks.sdk.service.settings.UpdateEsmEnablementAccountSettingRequest` - `databricks.sdk.service.settings.UpdateEsmEnablementSettingRequest` - `databricks.sdk.service.vectorsearch.ClusterAutoRestartMessageMaintenanceWindow` - `databricks.sdk.service.vectorsearch.ClusterAutoRestartMessageMaintenanceWindowDayOfWeek` - `databricks.sdk.service.vectorsearch.ClusterAutoRestartMessageMaintenanceWindowWeekDayBasedSchedule` - `databricks.sdk.service.vectorsearch.ClusterAutoRestartMessageMaintenanceWindowWeekDayFrequency` - `databricks.sdk.service.vectorsearch.ClusterAutoRestartMessageMaintenanceWindowWindowStartTime` - `databricks.sdk.service.vectorsearch.ComplianceStandard` - `databricks.sdk.service.vectorsearch.CspEnablement` - `databricks.sdk.service.vectorsearch.CspEnablementAccount` - `databricks.sdk.service.vectorsearch.CspEnablementAccountSetting` - `databricks.sdk.service.vectorsearch.CspEnablementSetting` - `databricks.sdk.service.vectorsearch.DeleteDefaultNamespaceRequest` - `databricks.sdk.service.vectorsearch.DeletePersonalComputeRequest` - `databricks.sdk.service.vectorsearch.DeleteRestrictWorkspaceAdminRequest` - `databricks.sdk.service.vectorsearch.EsmEnablement` - `databricks.sdk.service.vectorsearch.EsmEnablementAccount` - `databricks.sdk.service.vectorsearch.EsmEnablementAccountSetting` - `databricks.sdk.service.vectorsearch.EsmEnablementSetting` - `databricks.sdk.service.vectorsearch.GetAutomaticClusterUpdateRequest` - `databricks.sdk.service.vectorsearch.GetCspEnablementAccountRequest` - `databricks.sdk.service.vectorsearch.GetCspEnablementRequest` - `databricks.sdk.service.vectorsearch.GetDefaultNamespaceRequest` - `databricks.sdk.service.vectorsearch.GetEsmEnablementAccountRequest` - `databricks.sdk.service.vectorsearch.GetEsmEnablementRequest` - `databricks.sdk.service.vectorsearch.GetPersonalComputeRequest` - `databricks.sdk.service.vectorsearch.GetRestrictWorkspaceAdminRequest` - `databricks.sdk.service.vectorsearch.NccAwsStableIpRule` - `databricks.sdk.service.vectorsearch.UpdateAutomaticClusterUpdateSettingRequest` - `databricks.sdk.service.vectorsearch.UpdateCspEnablementAccountSettingRequest` - `databricks.sdk.service.vectorsearch.UpdateCspEnablementSettingRequest` - `databricks.sdk.service.vectorsearch.UpdateEsmEnablementAccountSettingRequest` - `databricks.sdk.service.vectorsearch.UpdateEsmEnablementSettingRequest` - `databricks.sdk.service.iam.PermissionMigrationRequest` - `databricks.sdk.service.iam.PermissionMigrationResponse` #### Changed - `version` field for `databricks.sdk.service.serving.AppManifest` to `databricks.sdk.service.serving.AnyValue` dataclass. - `delete_endpoint()` method for [w.vector_search_endpoints](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_endpoints.html) workspace-level service with new required argument order. - `create_index()` method for [w.vector_search_indexes](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_indexes.html) workspace-level service with new required argument order. - `delete_data_vector_index()` method for [w.vector_search_indexes](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_indexes.html) workspace-level service with new required argument order. - `upsert_data_vector_index()` method for [w.vector_search_indexes](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/vector_search_indexes.html) workspace-level service with new required argument order. - `endpoint_name` field for `databricks.sdk.service.vectorsearch.CreateVectorIndexRequest` to be required. #### Removed - `delete_personal_compute_setting()` method for [a.account_settings](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_settings.html) account-level service. - `get_personal_compute_setting()` method for [a.account_settings](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_settings.html) account-level service. - `update_personal_compute_setting()` method for [a.account_settings](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_settings.html) account-level service. - `delete_default_namespace_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. - `delete_restrict_workspace_admins_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. - `get_default_namespace_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. - `get_restrict_workspace_admins_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. - `update_default_namespace_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. - `update_restrict_workspace_admins_setting()` method for [w.settings](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/settings.html) workspace-level service. - `databricks.sdk.service.settings.DeleteDefaultNamespaceSettingRequest` dataclass. - `databricks.sdk.service.settings.DeletePersonalComputeSettingRequest` dataclass. - `databricks.sdk.service.settings.DeleteRestrictWorkspaceAdminsSettingRequest` dataclass. - `databricks.sdk.service.settings.GetDefaultNamespaceSettingRequest` dataclass. - `databricks.sdk.service.settings.GetPersonalComputeSettingRequest` dataclass. - `databricks.sdk.service.settings.GetRestrictWorkspaceAdminsSettingRequest` dataclass. - `databricks.sdk.service.vectorsearch.EmbeddingConfig` dataclass. - `embedding_config` field for `databricks.sdk.service.vectorsearch.EmbeddingSourceColumn`. - `name` field for `databricks.sdk.service.vectorsearch.DeleteDataVectorIndexRequest`. - `name` field for `databricks.sdk.service.vectorsearch.DeleteEndpointRequest`. - `planning_phases` field for `databricks.sdk.service.sql.QueryMetrics`. - `delta_sync_vector_index_spec` field for `databricks.sdk.service.vectorsearch.VectorIndex`. - `direct_access_vector_index_spec` field for `databricks.sdk.service.vectorsearch.VectorIndex`. ### Internal Changes * Added tokei.rs badge ([#567](#567)). * Update SDK to latest OpenAPI spec ([#576](#576)). * Add integration tests for Files API ([#552](#552)). * Fix integer deserialization for headers ([#553](#553)). * Support subservices ([#559](#559)). * Distinguish between empty types and fields that can take any value ([#561](#561)). OpenAPI SHA: 1026b998b14fba1b8317528f47778240dc4e9a5d, Date: 2024-03-06
Changes
We only initialise dbutils locally when using
from databricks.sdk.runtime import *
. Users in the webui are guided to use this import in all library code.The local (for people outside DBR) solution so far was to initialise spark manually. But this can be tedious for deeply nested libraries (which is the reason this import was introduced in the first place).
Now, we make a best effort attempt to initialise maximum number of globals locally, so that users can build and debug libraries using databricks connect.
Tests
make test
run locallymake fmt
applied