Відео

Yes, but for deploying DABs to Staging/Prod you want to use the same user every time so they are consistently the owner. For Github Actions I use a token in a secret. I think you could pull from key vault in dev ops pipeline, not positive on the best practice there.

Yes, editing that one to release in January

So you need to have a databricks.yml file in the root folder and it has to have some workflows or pipelines defined. Check out my videos on intro to Databricks Asset Bundles if you aren't sure how to get jobs created. The short answer is you can create a starting project using `databricks bundle init` or find examples like what I have in the resources folder and modify as needed for your project.

The general guidance is if you want to deploy together and code can be versioned together, then put it in the same bundle (all using same databricks.yml). If you want to keep things separate then its fine to have separate bundles and you can either deploy in separate CD pipelines or the same one by calling `databricks bundle deploy` multiple times, once from each directory with a databricks.yml. For making it more dyanmic I suggest variables, especially complex variables, but usually that is just to change values based on the target environment. Using SDK to create workflows is an alternative to DABs and other things have been discussed which might be more of a blend between the two options eventually.

@ Thank you very much!

Yes, meaning not a job cluster but an all-purpose cluster? You can either reference one with existing_cluster_id or define one in resources under the `clusters` section. docs.databricks.com/en/dev-tools/bundles/settings.html#specification

The idea makes sense but the way environments interact with workflows is still different depending on what task type you use. Plus you can't use them with standard clusters at this point. So it depends on how much variety you have in your jobs which is why I don't really include that in my repo yet.

@@derkachm No, I am not doing enough with Fabric yet to add anything new there.

@@thevision-y1b yes, it’s not ideal. Indicates I either want to 1) change to a worker VM type with more memory per core or 2) split into more tasks since the Input Size for my median and max tasks is a bit too high. By the way, these days that input size is usually ok for me but I use different node types now.

You can deploy schemas within unity catalog but for external storage locations or volumes I would expect those to either happen from Terraform or as notebooks/scripts that you run in the deploy pipeline. Jobs to populate the storage would be defined in DABs, but not the creation of storage itself unless it's built into a job you trigger with bundle run.

If you view the event log you might see some things. Sometimes a message will show that compute couldn't be retrieved from Azure which may be a quota limit (very common in trial accounts). If you added init scripts or libraries that can slow it down. Otherwise you can try posting more details (like event log info) in Databricks community. If you are really stuck and that doesn't help, message me more details through LinkedIn.

Meaning AWS account with GitHub Actions? If not, what combo of tools are you curious about for the deployment?

Not really, using databricks bundle validate is best way to see things. There are some options to view as debug but I haven't found something that works quite like Terraform plan. When you try to run destroy it does show what will be destroyed before you confirm.

For number 2, I agree. For number 3, I disagree, I think using git provider to push/pull from various environments is the right way to handle it. This is based on my belief its too confusing to sync two ways without git and often a team of people may be working together anyway. For number 4, if you append a incrementing version number or timestamp it will update on all-purpose cluster that already has it installed. Not really an IDE thing but it is all sort of related.

blog.jetbrains.com/pycharm/2024/08/introducing-the-pycharm-databricks-integration/

Change from mode: development to mode: production (or just remove that line). This will remove prefix and change default destination. However, for dev target I recommend you keep the prefix if multiple developers will be working in the same workspace. Production target is best deployed as a service principal from CICD pipeline (like Azure DevOps Pipeline) to avoid different people deploying the same bundle and having conflicts with resource owner and code version.

@@DustinVannoy Thank you Vannoy!! Worked fine now !!

The things that are logged are different. I've never written it up but we had some logs enabled that way plus we used this. There are other options to get logs, of course, but I found this one to be useful in the past for Azure focused environments.

You would need to define the job in the targets section of only the targets you want it in. If it needs to go to more than one environment, use YAML anchor to avoid code deuplication. I would normally just let a testing job get deployed to prod without a schedule, but others can't allow that or prefer not to do it that way.

Yes, it is possible. It looks something like this: job_clusters: - job_cluster_key: job_cluster new_cluster: spark_version: 14.3.x-scala2.12 node_type_id: m6gd.xlarge num_workers: 0 data_security_mode: SINGLE_USER spark_conf: spark.master: local[*, 4] spark.databricks.cluster.profile: singleNode custom_tags: {"ResourceClass": "SingleNode"}

@@DustinVannoy Thanks a lot! This cannot be found in the Databricks documentation.

Not that I'm aware of.

Using Databricks CLI path, you can have command that copies data up to volume. Replace all the curly brace { } parts with your own values. databricks fs cp --overwrite {local_path} dbfs:/Volumes/{catalog}/{schema}/{volume_name}/{filename}

nice observation . i am also facing this issue. can u figure out soln ? how do u know "merging the data into a table based on a condition" is issue?

I don't think I follow. Is there a code example you can send along? For Databricks I sometimes just set this up as separate parallel workflow tasks but you may be describing other challenges. If there is error message you encounter please share.

The part that references variables group PrdVariables shows how you set different variables and values depending on target environment. - stage: toProduction variables: - group: PrdVariables condition: | eq(variables['Build.SourceBranch'], 'refs/heads/main') In the part where you deploy the bundle, you can pass in variable values. See the docs for how that can be set. docs.databricks.com/en/dev-tools/bundles/settings.html#set-a-variables-value

Yes, ask away. I'll answer what I can.

Thank you, @@DustinVannoy. I wonder whether the following development progress does make sence. And if there any thing we could improve it. Background: (1) We have two Azure Databricks workspaces, one is for development, one is for production. (2) I am the only Data Engineer in our team, and we don't have dedicate QA. I am responsible to development and test. Who consume the data will do UAT. (3) We use Azure DevOps (repository and pipelines). Process: (1) Initialization (1.1) Create a new project by using `databricks bundle init` (1.2) Push the new project to Azure DevOps (1.3) On development DBR workspace, create a GIT Folder under `/Users/myname/` and link to the Azure DevOps repository (2) Development (2.1) Create a feature branch on DBR workspace (2.2) Do my development and hand test (2.3) Create a unit test job and the scheduled daily job (2.4) Create a pull request from the feature branch to the main branch on DBR workspace (3) CI (3.1) An Azure CI pipeline (build pipeline) will be trigerred after the pull request is created (3.2) The CI pipeline will check out the feature branch, and do `databricks bundle deploy` and `databricks bundle run --job the_unit_test_job` on the development DBR workspace by using Service Principal. (3.3) The test result will show on the pull request (4) CD (4.1) If everything looks good, the pull request will be approved (4.2) Manually trigger an Azure CD pipeline (release pipeline). Checkout the main branch, do `databricks bundle deploy` to the production DBR workspace by using Service Principal Explanation: (1) Because we are a small team and I am the only person who works on this, we do not have a `release` branch to simply the process (2) Due to the same reason, we also do not have a staging DBR workspace

Overall process is good. It’s typical not to have a separate QA person. I try to use yaml pipeline for the release step so code would look pretty similar to what you use to automate deploy to dev. I recommend having unit tests you can easily run as you build which is why I try to use Databricks-connect to run a few specific unit tests at a time. But, running workflows on all-purpose or serverless isn’t too bad an option for quick testing as you develop.

🤣🤣 it may happen one day, but not today. I would probably need help from build5nines.com

Are you trying to install the wheel in a notebook task, so you are required to install with %pip install? If you include the artifact section it should build and upload the wheel regardless of usage in a taks. You can predict the path within the .bundle deploy if you aren't setting mode: development, but I've been uploading it to a specific workspace or volume location. As environments for serverless evolve I may come back wtih more examples of how those should be used.

I would use a bundle variable and set it in the target overrides, then reference it anywhere you need it.

You should export as source files instead of dbc files if you want to view and edit in VS Code.