What I’m struggling to understand is how it’s even possible to make notebooks work in production such that there is an entire faction of people shilling for it. Like how are their companies even functioning when 9.999999/10 cases the notebook breaks because the cells are ran out of order or something
You can run notebooks (running the cells in order) in Databricks, as if it's production. I imagine they're nice prototypes, but manually tested code is just a long-term nono.
Databricks sort of encourages you to use notebooks all the way. Orquestrating them through workflows solves the cell execution order issue. But maintaining them is insane as the Workspace scales up.
I'm extremely inexperienced and have only used notebooks in the cloud (so for programs that need to run continiously). Is the alternative to notebooks just docker containers? How does a company generally run python code in production? I work at a startup and I'm the only technical guy.
"Is the alternative to notebooks just docker containers?" Oh man. No. Sorry, it's not you. It's just that this is a common question and shows how far away so many analyst-types are. The alternative to notebooks is small, modular .py files organized into something called a "package". Docker helps both notebooks AND packages be reproducible (if you leave and come back to your code 1 month from now, there's still a good chance you can install and run everything without fighting "Dependency hell")
@@Eriddoch yes obviously my python project makes use of packages. I'm more wondering how I run these continuously in the cloud. Do I let Microsoft host the docker container for me, or are there other ways of running my python project in the cloud?
@@Eriddoch Could you have a look at my previous reply? I'm still struggling with finding how you would run a python project in the cloud without docker containers.
@@matthiaswarlop2316 You can still use Databricks to run Python packages as workflows, for example. Instead of the workflows pointing to notebooks in a Git repo, they can point to a package. An alternative, if you're not using Databricks, is indeed to run containers, in Kubernetes clusters, for example.
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
Pure gold here 😄😄
That blog got you a subcriber
thanky kindly
What I’m struggling to understand is how it’s even possible to make notebooks work in production such that there is an entire faction of people shilling for it. Like how are their companies even functioning when 9.999999/10 cases the notebook breaks because the cells are ran out of order or something
You can run notebooks (running the cells in order) in Databricks, as if it's production. I imagine they're nice prototypes, but manually tested code is just a long-term nono.
Databricks sort of encourages you to use notebooks all the way. Orquestrating them through workflows solves the cell execution order issue.
But maintaining them is insane as the Workspace scales up.
@@apollon456 how do you run python code in your company?
I'm extremely inexperienced and have only used notebooks in the cloud (so for programs that need to run continiously). Is the alternative to notebooks just docker containers? How does a company generally run python code in production? I work at a startup and I'm the only technical guy.
"Is the alternative to notebooks just docker containers?"
Oh man. No. Sorry, it's not you. It's just that this is a common question and shows how far away so many analyst-types are.
The alternative to notebooks is small, modular .py files organized into something called a "package".
Docker helps both notebooks AND packages be reproducible (if you leave and come back to your code 1 month from now, there's still a good chance you can install and run everything without fighting "Dependency hell")
@@Eriddoch yes obviously my python project makes use of packages. I'm more wondering how I run these continuously in the cloud. Do I let Microsoft host the docker container for me, or are there other ways of running my python project in the cloud?
@@Eriddoch Could you have a look at my previous reply?
I'm still struggling with finding how you would run a python project in the cloud without docker containers.
@@matthiaswarlop2316 You can still use Databricks to run Python packages as workflows, for example. Instead of the workflows pointing to notebooks in a Git repo, they can point to a package.
An alternative, if you're not using Databricks, is indeed to run containers, in Kubernetes clusters, for example.
Skill issue 😂