Manage and automate databricks deployment using Terraform in Azure databricks

[@RajashekharKandurke]

please can you just tell me in which application your writing terraform code for azure databricks jobs creation

[@divyaburri-z5j]

HI ,I have followed your video for creating databricks workspace and resources ,but I am facing issue like provider depends on workspace ID ,and resources depends on databricks provider ,how can I achieve all in a single terraform apply ,can please help me here❗important

[@chandu1740]

Hello, We have create databricks workspace, cluster and metastore/unity Catalogue in Dev subscription, we are planning to proceed with test, uat and prod. I understand that we can create only one metastore per region which I have done in Dev subscription, May i know how to use/call the existing metastore in Test, uat and prod subscriptions please. Thanks

[@ankitapundhir1675]

how you are able to fetch all values like instance pool and all ? how these intellisense are working ? did you install anything for databricks also?

[@Infrasity]

hey, just get the terraform plugin for IntelliJ, that should be all you need.

[@JuliaShub]

Can you please add an example using VNet injection? for instance if I want the Databricks workspace to run on an existing resource group within an existing VNET?

[@Infrasity]

hey,
Assuming you already have an Azure Virtual Network (VNet) and a resource group set up, follow these steps to create a Databricks workspace using VNet injection:
Prepare the existing VNet:
Make sure your existing VNet is properly configured and contains the necessary subnets for Databricks. You will need at least two subnets: one for the control plane (called "Control Plane Subnet") and another for the data plane (called "Data Plane Subnet"). Ensure that these subnets do not have any overlapping CIDR ranges.
Create a new Databricks workspace:
In the Azure portal, navigate to your desired resource group (the existing one) where you want to create the Databricks workspace. Click on "+ Add" to create a new resource and search for "Databricks" in the marketplace. Select "Azure Databricks" from the results and click "Create."
Configure the workspace:
During the workspace creation process, you will come across the "Networking" section. Here, you will need to select "VNet Injection" as the workspace network type.
Choose the existing VNet and subnets:
Once you select "VNet Injection," you will be prompted to choose the existing VNet and its associated subnets. Select the control plane and data plane subnets you prepared earlier. This will integrate your Databricks workspace into your existing VNet.
Configure workspace settings:
Continue with the workspace creation process, specifying other necessary settings like workspace name, subscription, pricing tier, etc.
Review and create:
Review your settings, and if everything looks good, click "Create" to provision the Databricks workspace.

[@akashkumar-gh3ri]

Apart the Service principal, could you please share any other authentication option

[@Infrasity]

Hey, you may visit infrasity.com/p/azure-databricks-administration-etl-workflow for other authentication methods.

[@justgamesone]

Thanks Bhai

[@shivamgovekar1757]

can you please provide the terraform code for reference

[@Infrasity]

Here is an example of Terraform code for provisioning an Azure Databricks workspace using Terraform:
provider "azurerm" {
version = "2.30.0"
}
locals {
resource_group_name = "databricks-rg"
workspace_name = "databricks-workspace"
}
resource "azurerm_resource_group" "databricks" {
name = local.resource_group_name
location = "westus2"
}
resource "azurerm_databricks_workspace" "databricks" {
name = local.workspace_name
resource_group_name = azurerm_resource_group.databricks.name
location = azurerm_resource_group.databricks.location
sku {
name = "standard"
}
}
This Terraform code creates a resource group named "databricks-rg" in the "westus2" region, and a Databricks workspace named "databricks-workspace" within the resource group. The workspace is created using the "standard" SKU.
You can modify this code as needed to meet the specific requirements of your environment and use case. For example, you can change the resource group name, workspace name, location, and SKU as needed.
Note: Make sure you have the necessary permissions and have configured the Azure CLI or have the necessary environment variables set before running this Terraform code.

[@shivamgovekar1757]

@@Infrasity thank you :)