Відео

It is just a label for the object. You can call anything you want. Usually if you only have one instance of the object in the module you just call it this.

Thanks 🙏🏻. Please subscribe and smash 💥 the like button 😃

There is no question :). I have figured it out. My connection was connection="BlobStorageConnectionString" and i did not add it to Environments variables. Stupid error.

Awesome. Great it worked

Great to hear! Below you sorted out. Cheers

Have you done a databricks auth profiles ? Do you get a green tick ✅?

@@pytalista Yes, I have used debug and I found the issue. I have used the command databricks --debug bundle init to find the issue. Then it has thrown an error to choose the profile among default and new created by providing --profile flag. Then I used the databricks --debug bundle init --profile "new profile".

You mean that now is working ?

@@pytalista Yes

🙌🏻👏🏻

Hi, do not intend to go on this topic. This is a way to bring data to the lake form transactional systems. then from there I would ingest in delta tables.

Thanks 🙏🏻. I mostly use spark on the cloud which is already implemented. You focus on running code.

Thanks 🙏🏻

Hi good point. Certainly delta live table simplify the code and the developer experience in a more declarative way. This is a decision engineer need to make. DLT cost is a bit higher and some prefer to code more declarative. I would say that the simpler the transformation and more "cookie cutter" I would go for DLT otherwise better to have CDF.

Thanks 🙏🏻. Make sure the Databricks cli version is 0.218.0 or higher. docs.databricks.com/en/dev-tools/bundles/index.html

Yes

Yes in this case for speed you can still develop using the UI and when done export the notebook and deploy in a DAB. Also you can run the notebook in VS code by attaching cluster in VS code. But still UI offers a better experience while developing.

🤣

Is the same user in Fabric been granted the permission in lay vault ?

Hi I think is a valid approach. Test in data engineering is a bit different. I would separate the test into 2 categories. Unit Test which would be test of the functions that is used in your code. Then use pytest for that. Then tests of data that you can use Delta Live tables expectations or libraries like great expectations. This because in data engineer data is stateful.

@@pytalista Good point. So basicaly, you're suggesting that queries are basically tested by the data quality checks you apply in the result, and not testing the query itself using mock data and expected dataframes?

@SergioPolimante on data quality you can test things like. Not nulls, unique, data ranges, valid keys etc. that is the part of your data is statefull. About testing the logic of your query with mock data is also a valid approach maybe it is a lower return on time investment than quality expectations checks. Then there is the unit test of your functions you may have in your pipeline in this case you would do classic unit tests.

Glad it helped

follow this then it will make sense learn.microsoft.com/en-us/azure/azure-functions/durable/quickstart-python-vscode?tabs=linux%2Cazure-cli-set-indexing-flag&pivots=python-mode-decorators

I ran into the same issue, I just did it via the command line. Use 'func templates list' to get the list of templates you want to use.

@carlosr204 good one thanks 🙏🏻

ua-cam.com/video/lYYIFRaY8Tk/v-deo.htmlsi=RiYQqRffGBTVkYZR

No worries Thanks. Hard to tell without more context. 404 is a not found error. If you follow the tutorial should work exact the same.

@@pytalista Oh yes, I had to implement custom logic for my use case, so the code was quite different. The storage account I was trying to access did not allow anonymous access and an incorrect request URL was being generated. All in all, Blob was not found. I was able to debug it. Your video set a great precedent, thank you for replying.

@diyakhajuria2478 great work. Happy it helped.

Thanks

Thanks 🙏🏻

still can use it. only classic will be retired. Use a workspace based one learn.microsoft.com/en-gb/azure/azure-monitor/app/create-workspace-resource?tabs=bicep

Generate a PAT to the service principal. Need to do via databricks cli to generate a PAT for SP. do a M2M authentication for that.

dbt only support connecting to warehouse but your warehouse can do cross query. This is possible. What you cannot do is to query the lakehouse directly from dbt.

@@pytalistaYou mean, what is possible to create a "warehouse" from your lakehouse and then target another real Warehouse? Is this what cross-query across Lakehouses/warehouses from a single SQL Endpoint mean? Thanks for this feedback!

@@susanODilla No. I said you can query a lakehouse from a warehouse. This is what cross query mean.

make sure you all the requisite covered. All the installations and your environment is set correctly.

@@pytalista I am working on codespace is that making difference??

@manavkumar1071 i don’t know what is code space. Probably it is. They to configure on your local computer.

Hi Ajmal, Just install WSL then on the terminal type code . to open VS that is already installed on windows. To make your terminal pretty watch this video ua-cam.com/video/ehM3vgPcTc8/v-deo.html

Hard without context. But I would watch the video and follow exactly. Looks like you are not inserting your connection string in the code. Read this question that may point you on the right direction. stackoverflow.com/questions/2338650/connection-string-to-an-azure-cloud-storage-account

@@pytalista I'm trying to test the queue trigger fucntionapp locally. Then I do not want to use a queue storage that is in portal azure but one for develpment. For example with the eventgrid function in the settings I have this "AzureWebStorage"=UseDevelopmentStorage=true". I'm asking if there is anything similar for the connection string

@@pytalista Ok, I solved. If you want to use Azurite and therefore run it locally the connection string on the function app should be "UseDevelopmentStorage=true". Then when u want to send a message the connection string for the queue storage only is DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;QueueEndpoint=127.0.0.1:10001/devstoreaccount1;

Make sure you follow all pre requisites. Installations and etc.

Thanks 🙏🏻

No worries 😉

Thanks 🙏🏻

Glad it was helpful! 😀

Noted

Thanks 🙏🏻

Not a data scientist 😀. But I am doing on building a Python Wheel and another on a GitHub Actions CI/CD

What is the error ?

@@pytalista When I try to activate the Virtual environment using Source it is saying that Source is not the CMD word

@Vatsav04 on my video I am using Linux. Looks like you are on a Windows machine. It is a bit different. myenv\Scripts\activate

Hard to tell without seeing the error. One possible thing is if you are using any environment variables. Then you need to have on your cloud environment to work.

Glad you liked 👍🏻

A blob trigger is an event driven action. So whenever a blob arrive at a particular location a function will be triggered. Service principle are to grant access to resources. So a blob trigger is independent of service principle. Seu you have a service principal that you want access to your storage account you can do that for sure. 👍🏻

Sure 👍🏻 I am working on a version without music.

No problem here is a version without music. Please like and subscribe. ua-cam.com/video/wUqohFhYHl0/v-deo.html

Glad it helped.

@@pytalista Please also upload a video of how to store API keys and scerets in a secret manager and use that in Databricks notebook!

If you have an azure account you can use this trick ua-cam.com/video/alZWTehsTcg/v-deo.htmlsi=KGdbe5Y_8igGtDKS

Have tested locally ? Did it work?

Create a parameter in DF

Welcome 🙏🏻

Glad you liked 😀

Thanks 🤩

Thanks for the feedback. Please like and subscribe

Thanks 🙏🏻

Cool. Thanks for the tip 😉

Hi, if you want explicitly is easy. Just put the full blob name. I uploaded the code in the repository github.com/pedrojunqueira/PytalistaYT/blob/master/Python/storage_trigger/function_app.py

@@pytalista i do not want to explicitly memtion it. Currently I am using environmental variables.

@@SAHITHTHATIPALLI Ok good.

Well done Roxane. Great it worked in the end. Life of a programmer 🧑‍💻😀