Welcome to Autobricks
Python Azure databricks API wrapper for CI/CD. For when the CLI just doesn't cut it this allows high level python scripting of databricks assets deployments.
The library is a collection of modules functions of similar topology to the databricks API version 2.0
Installation
pip install autobricks
Configuration
Azure databricks allows the following authentication methods:
- Databricks User PAT Token
- Azure Service Principal
- Azure Service Principal over the Management End Point
This library supports all by simply setting the following environment variables as required for each method. The AUTH_TYPE sets the authorisation mode and therefore what configuration to expect. Ensure that sensitive values are managed using secret redaction e.g. key vault or some other method.
Variable | User PAT | SP | SP on Mgmt Endpoint |
---|---|---|---|
AUTH_TYPE | USER | SERVICE_PRINCIPAL | SERVICE_PRINCIPAL_MGMT_ENDPOINT |
DBUTILSTOKEN | ✓ | ✓ | ✓ |
TENANT_ID | ✓ | ✓ | |
SP_CLIENT_ID | ✓ | ✓ | |
SP_CLIENT_SECRET | ✓ | ✓ | |
AD_RESOURCE | ✓ | ✓ | |
MGMT_RESOURCE_ENDPOINT | ✓ | ||
WORKSPACE_NAME | ✓ | ||
RESOURCE_GROUP | ✓ | ||
SUBSCRIPTION_ID | ✓ | ||
DATABRICKS_API_HOST | ✓ | ✓ | ✓ |
Also note that library will use simple rest calls to retrieve tokens. An alternative approach is to leverage the adal library to authenticate. If you wish to authenticate over adal please refer to following settings:
Variable | SP | SP on Mgmt Endpoint |
---|---|---|
AUTH_TYPE | SERVICE_PRINCIPAL_ADAL | SERVICE_PRINCIPAL_MGMT_ENDPOINT_ADAL |
DBUTILSTOKEN | ✓ | ✓ |
TENANT_ID | ✓ | ✓ |
SP_CLIENT_ID | ✓ | ✓ |
SP_CLIENT_SECRET | ✓ | ✓ |
AD_RESOURCE | ✓ | ✓ |
MGMT_RESOURCE_ENDPOINT | ✓ | |
WORKSPACE_NAME | ✓ | |
RESOURCE_GROUP | ✓ | |
SUBSCRIPTION_ID | ✓ | |
DATABRICKS_API_HOST | ✓ | ✓ |
When deciding between adal and rest calls the following is relevant: - adal may not work in Azure DevOps in highly restricted private network setups - Rest calls are more simple however if MS makes any changes to the API's the adal isn't there to abstract those changes.
The following variables will default.
Variable | Default |
---|---|
AUTH_TYPE | SERVICE_PRINCIPALs |
AD_RESOURCE | 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d |
MGMT_RESOURCE_ENDPOINT | https://management.core.windows.net/ |
Modules
The following modules provide high level functions that can be used to python script management and deploy assets to databricks workspaces:
- autobricks.Cluster
- autobricks.Dbfs
- autobricks.Job
- autobricks.Library
- autobricks.Workspace