In a small team, business analysts are also data engineer, architect, etc etc. Wear all the hats. Reuse is huge for me and for supporting my team as all our data comes from soloed systems via manual exports. This can take me out of the loop/future proof!
Again, thank you very much for bringing light and to dare with this poliedric issue. You can study it or draw conclusions from very different angles... As far as me I believe i will use it for our employee table. Many people try to ETL (to cook) a new employee table every month for their KPIs and for further reports in Power BI. With Datflows this table can be cooked only once and all the people who build reports from employees have a common place to take this file. Before dataflows each one had to cook the file more or less in the same way... and besides, if the boss decides to cook it in a different way then it will be inmediatelly available for the rest of the people. The advantage is just to cook just once in a higher level of the tree and not many times on the leaves!
Thanks for sharing your point of view and with a specific example, it helps me get a better understanding of use cases for dataflows and I am sure will help others! Awesome and thank you! /Ruth
First time i actually went through a white paper (directly/indirectly), thanks for this, you made an interesting video out of a white paper. you did quite an ETL :)
Nice video Ruth! Agree with your comment on Business Analyst........the BA's that I know don't want to touch anything remotely technical!!! For some reason a lot of BI tools (Power BI, Qlik) are now calling developers as business analyst....not sure why though!
Funny you say that as I have been wondering the same thing. I dislike the term developer as it somehow implies that you need to have an it background to do it ... I wonder why that is too.. /Ruth
Thank you for breaking it down for us, Ruth! I WOULD like to see a video demo of setting up and consuming dataflows entities. The process appears to be pretty much identical to PBI/Excel Power Query process, but I have not been successful connecting to any data sources. I consider myself more of an analyst, but having no one else to rely on to create/manage usable datasets (i.e. data warehouses), I see a HUGE potential in data flows for my quest for a wider adoption of Power BI by my employer (who is shockingly primitive despite its size). I think it can make things less intimidating (and more self-service-y) if users don't have to feel like they must learn M or DAX. I will just have to learn how it works first. lol
I will, it is on my to-do list. I will try to put it for tomorrow’s video, but as I am travelling today, I might not make it. But you will see one for sure :) And thanks for adding your point of view, I have a hard time to see dataflows as a self-service tool for business users or analysts, but then again, I don’t think I have seen a need for it on the .org I have worked with so hearing your perspective is useful for me and others. Thanks! /Ruth
@@CurbalENmadam, It's possible to create report directly from dataflow not in power BI desktop in power BI service?? if possible please explain the steps?
Excelente Ruth , ya me baje tu paper de apoyo, ya lo traduje y muy buena explicacion , ahora juntar todos esos tipo de datos y hacer magia como dicen los amigos de PowerPivotPro , aparte de aprender todo este mundo facinante de excel, dax ,pq y power bi. Tambien aprendo ingles contigo , ya que aprendo palabras que no se , ejemplo tu video tuvo 4538 palabras , solo 37 no sabia , osea tengo menos del 1% de palabras que desconocia, esa es mi manera de aprender ingles ,ver videos y anotar las que no se y voy mejorando .Siempre me preguntan donde aprendiste tanto ingles escrito y es solo perseverancia con ganas gracias Ruth.
Great video Ruth, im still trying to learn about dataflows and how it can help me in my deployments. The point at 9:50 about data locked in Power BI, I thought it meant that the data you clean and analyse in Power BI can't be used as a data source for other apps like Power Apps, Azure MI, etc. NOT about the capacity of data. What about using dataflows as a way to standardise data source structures? For example Microsoft has a CDM for different sectors and I was hoping to use the standard data structure in there to create a "plug-and-play" system, where I create a standard template for a report, and then connect my clients data to these reports. Is this industry-standard? I'm not quite sure if it is.
Fantastic video...and yes I do not see the average Joe or Jane self-service Power Bi user evolving to PQ M (propriety) language rather than evolving to either Data factory v2 or a Databricks notebook running in SQL, Python or R. I agree 100% with you let’s see some movement on Data Catalogue rather than this slightly weird investment. 🙃
Ruth, Excellent Video - Very helpful learning on how the Dataflow is connected to its CDM Folder tables with the M expression. Also, is there a way we can restore a dataflow (with the history) in case of a disaster?
When I try to do a "FuzzyNestedJoin" I get this error in Data Flows: "The name 'Table.FuzzyNestedJoin' wasn't recognized. Make sure it's spelled correctly" Here is what I am doing......... Table.FuzzyNestedJoin(#"Expanded Exclude", {"Account_Name"}, HPG_Roster, {"Name2"}, "HPG_Roster", JoinKind.LeftOuter, [IgnoreCase = true, IgnoreSpace = true, NumberOfMatches = 1])................ Any help would be amazing!
Hola Ruth. Espero que estés muy bien!!! ¿Es normal que al conectarme a un Flujo de Datos, no se vean datos en las Tablas en el Desktop? He creado el flujo de datos correctamente en el Servicio Power BI. He abierto Power BI desktop; lo conecto correctamente con el flujo de datos. Pero... No se ven datos en las tablas de Power BI. En Power Query (desktop) tampoco se ven datos. Al actualizar, pone: "La Tabla está vacía". Sin embargo, en el Flujo de Datos del Servicio Power BI sí puedo ver los datos... Es muy extraño... ¿no? ¡Muchas gracias!!!
So if a user wanted to keep the ETL and the data in the cloud why not use Data Factory and Azure SQL/Data Warehouse? Can't see how this adds anything. Unless this is meant to be ETL light? Friendly and less intimidating for Data Analysts (not DataWarehouse/BI Developers)? What about failure logging, restarts, success and failure notifications that you would need with ETL?
Hello Ruth ! Do you know any workaround to use excel get data from a dataflow ? I know the feature is supposed to be release this year but nothing yet and it is a really needed feature. Please help !
One quick question - If I use dataflow in Power BI Desktop, will it store data or data will be available on cloud? Will the size of Power BI Desktop workbook increase?
I am just hesitant managing Data Flows, the referencing can build layers of complexities, also validating the data can be challenging. Bottom line, it may take away from the advantage of the 'Single Source of Truth' for developers.
Hi Ruth.. thankyou for providing such beautiful explanations. One issue that i found using Data flows is that the time for refresh is huge compared to datasources directly connected to pbix file. I guess the underlying reason is compression (vertipaq vs csv files). Is there any way to compress the data in dataflow itself? Thanks in advance!
Hi Ruth. Great video. Thanks a lot Trying to summarize about costs, if I understood correctly, it is something like: - using Data flows “seriously “ needs Premium license ($$$) - Data flows with Pro license ($) doesn’t make too much sense since it is limited to 10 GB and almost all new features need Premium - PowerQuery is (by now) free and it is almost same thing as Data Flows with Pro license but running locally instead of web based Am I right?
Yes, dataflows are Power query online with the difference that dataflows store the data in a datalake so it can be reused in other systems while power query stores it on power bi or Powerpivot. The only remark is that the cost is not for dataflows but for the data lake storage, otherwise bullseye 🎯, as I understand it at least! /Ruth
Hi Ruth, great video. Thank you so much. What hardware and software did you use for the notes? Would love to use it for our Internal Training Videos...
@@CurbalEN actually, we are having crazy discussion on what to use for report to improve performance from Dataflow or Dataset. And end end with Dataflow. What you think?
Can try out Premium features using Power BI Embedded "A" SKUs: christopherfinlan.com/2018/11/18/how-to-use-a-skus-to-try-out-paginated-reports-in-power-bi-without-upfront-cost-or-long-term-commitment/comment-page-1/ Also, to handle the Dataflows version of Power Query being limited to GUI features, many people having success copying and pasting M code from Power BI into Dataflows, and having them run okay.
There are for sure advantages and disadvantages with dataflows, but it will come in handy for some users. Let’s see how it develops, it is at an early stage at the moment. /Ruth
In a small team, business analysts are also data engineer, architect, etc etc. Wear all the hats. Reuse is huge for me and for supporting my team as all our data comes from soloed systems via manual exports. This can take me out of the loop/future proof!
Very true :)
Thanks for this overview. 1.5yrs later and it's still very relevant.
Glad to hear :)
/Ruth
Again, thank you very much for bringing light and to dare with this poliedric issue. You can study it or draw conclusions from very different angles...
As far as me I believe i will use it for our employee table. Many people try to ETL (to cook) a new employee table every month for their KPIs and for further reports in Power BI. With Datflows this table can be cooked only once and all the people who build reports from employees have a common place to take this file. Before dataflows each one had to cook the file more or less in the same way... and besides, if the boss decides to cook it in a different way then it will be inmediatelly available for the rest of the people. The advantage is just to cook just once in a higher level of the tree and not many times on the leaves!
Thanks for sharing your point of view and with a specific example, it helps me get a better understanding of use cases for dataflows and I am sure will help others!
Awesome and thank you!
/Ruth
Your first 5 minutes clarified a lot for me. Thank you!
Wonderful to hear :)
/Ruth
First time i actually went through a white paper (directly/indirectly), thanks for this, you made an interesting video out of a white paper. you did quite an ETL :)
Hi Ruth, thanks for your perspective and helping breakdown Dataflows!
My pleasure!! I should do un updated video. My views have not changed a lot but they have indeed changed :)
Nice video Ruth! Agree with your comment on Business Analyst........the BA's that I know don't want to touch anything remotely technical!!! For some reason a lot of BI tools (Power BI, Qlik) are now calling developers as business analyst....not sure why though!
Funny you say that as I have been wondering the same thing. I dislike the term developer as it somehow implies that you need to have an it background to do it ... I wonder why that is too..
/Ruth
Hi, Very useful video but may I know what happen about data privacy policies in Dataflows?
Thank you for breaking it down for us, Ruth! I WOULD like to see a video demo of setting up and consuming dataflows entities. The process appears to be pretty much identical to PBI/Excel Power Query process, but I have not been successful connecting to any data sources. I consider myself more of an analyst, but having no one else to rely on to create/manage usable datasets (i.e. data warehouses), I see a HUGE potential in data flows for my quest for a wider adoption of Power BI by my employer (who is shockingly primitive despite its size). I think it can make things less intimidating (and more self-service-y) if users don't have to feel like they must learn M or DAX. I will just have to learn how it works first. lol
I will, it is on my to-do list. I will try to put it for tomorrow’s video, but as I am travelling today, I might not make it.
But you will see one for sure :)
And thanks for adding your point of view, I have a hard time to see dataflows as a self-service tool for business users or analysts, but then again, I don’t think I have seen a need for it on the .org I have worked with so hearing your perspective is useful for me and others.
Thanks!
/Ruth
@@CurbalENmadam,
It's possible to create report directly from dataflow not in power BI desktop in power BI service?? if possible please explain the steps?
This is such a solid video. Keep em coming, Curbal! Can @guyinacube give this channel a shout out! You deserve it!
Thanks!! He does feature me know and then :)
/Ruth
Very informative. If only more people were lovely as you... : )
Excelente Ruth , ya me baje tu paper de apoyo, ya lo traduje y muy buena explicacion , ahora juntar todos esos tipo de datos y hacer magia como dicen los amigos de PowerPivotPro , aparte de aprender todo este mundo facinante de excel, dax ,pq y power bi.
Tambien aprendo ingles contigo , ya que aprendo palabras que no se , ejemplo tu video tuvo 4538 palabras , solo 37 no sabia , osea tengo menos del 1% de palabras que desconocia, esa es mi manera de aprender ingles ,ver videos y anotar las que no se y voy mejorando .Siempre me preguntan donde aprendiste tanto ingles escrito y es solo perseverancia con ganas gracias Ruth.
Genial! Me alegro que mis vídeos te sean doble útiles y enhorabuena por tu progreso!
/Ruth
Informative stuff. Keep going Ruth
Thank you for the video! I like how you provide the information and add your interpretation and comments on top of it :)
Very effective! Thanks for sharing knowledge. And quintessentially yourself, your casual conversation style is very effective. Keep going Ruth.. :-)
This was another useful explanation on your channel - great content!
Best explanation on Dataflows. Thanks
for sharing.
Thanks Ruth, as always! You put it all in a nutshell! Cheers!
Thanks for the feedback ! 😊
/Ruth
Great video Ruth, im still trying to learn about dataflows and how it can help me in my deployments.
The point at 9:50 about data locked in Power BI, I thought it meant that the data you clean and analyse in Power BI can't be used as a data source for other apps like Power Apps, Azure MI, etc. NOT about the capacity of data.
What about using dataflows as a way to standardise data source structures? For example Microsoft has a CDM for different sectors and I was hoping to use the standard data structure in there to create a "plug-and-play" system, where I create a standard template for a report, and then connect my clients data to these reports.
Is this industry-standard? I'm not quite sure if it is.
Not sure if it is an industry standard but you can use it either way.
Fantastic video...and yes I do not see the average Joe or Jane self-service Power Bi user evolving to PQ M (propriety) language rather than evolving to either Data factory v2 or a Databricks notebook running in SQL, Python or R. I agree 100% with you let’s see some movement on Data Catalogue rather than this slightly weird investment. 🙃
I would love a proper data catalogue, that would be amazing...
/Ruth
Nice Lamp! And great video! I've been trying to figure out what dataflows are all about, and this was helpful.
Great, a practical video Will follow and hopefully that will make it all clear :)
/Ruth
thanks for sharing
My pleasure!
So valuable to where I’m at w PQ. This will get me rolling today in trying this out. Btw, comment at :35 😆
Oh dear! What did I say? 🙈
@@CurbalEN followup, I used it today and feel like I gained a new super power. Thanks again!
You definitely leveled up , congrats!
Ruth, Excellent Video - Very helpful learning on how the Dataflow is connected to its CDM Folder tables with the M expression. Also, is there a way we can restore a dataflow (with the history) in case of a disaster?
When I try to do a "FuzzyNestedJoin" I get this error in Data Flows: "The name 'Table.FuzzyNestedJoin' wasn't recognized. Make sure it's spelled correctly" Here is what I am doing......... Table.FuzzyNestedJoin(#"Expanded Exclude", {"Account_Name"}, HPG_Roster, {"Name2"}, "HPG_Roster", JoinKind.LeftOuter, [IgnoreCase = true, IgnoreSpace = true, NumberOfMatches = 1])................ Any help would be amazing!
Great explanations!
10x
What do you mean in minute 24 forget to mention...
I am not sure I understood how the cost can be huge?
Many thanks.
Dataflows store the data in a data lake, if you have a pro account you have only 10GB of storage for free, the rest you need to pay.
/Ruth
Brilliant very well done. Thank you kindly.
Hola Ruth. Espero que estés muy bien!!!
¿Es normal que al conectarme a un Flujo de Datos, no se vean datos en las Tablas en el Desktop? He creado el flujo de datos correctamente en el Servicio Power BI. He abierto Power BI desktop; lo conecto correctamente con el flujo de datos. Pero... No se ven datos en las tablas de Power BI. En Power Query (desktop) tampoco se ven datos. Al actualizar, pone: "La Tabla está vacía". Sin embargo, en el Flujo de Datos del Servicio Power BI sí puedo ver los datos... Es muy extraño... ¿no?
¡Muchas gracias!!!
So if a user wanted to keep the ETL and the data in the cloud why not use Data Factory and Azure SQL/Data Warehouse? Can't see how this adds anything. Unless this is meant to be ETL light? Friendly and less intimidating for Data Analysts (not DataWarehouse/BI Developers)? What about failure logging, restarts, success and failure notifications that you would need with ETL?
Hi, is it possible to use dax tables created in power bi desktop as data source for data flows ?
Ruth what software did you use for this. Especially the writing on the pad to display on the TV
Camtasia :)
@@CurbalEN wow. This is great. Any link to help me learn
Thank you Ruth.
Great explanation
Perfect! I drove me nuts not to understand, so I thought there might be others!
Glad it was useful :)
/Ruth
Hello Ruth ! Do you know any workaround to use excel get data from a dataflow ? I know the feature is supposed to be release this year but nothing yet and it is a really needed feature. Please help !
Maybe Analyze in excel? Workaround but it should work!
very nice video. so helpful and very good explanation of Dataflow. Thanks a lot
Glad you liked it and thanks for the feedback!
/Ruth
Dear Ruth,
Please cover paginated reports.
Regards
M Osama
Thanks for the explanation. I think I've learned enough to know that dataflows are not for me. I am not going swimming in the data lake.
If you want to do autoML I am afraid you need those :(
/Ruth
@@CurbalEN I went back and looked at this again. It makes sense now.
I found some useful use cases too. Still think it is not for the business user, but to build data storages. :)
Wow--can I be you when I grow up? Very well done!
😊
One quick question - If I use dataflow in Power BI Desktop, will it store data or data will be available on cloud? Will the size of Power BI Desktop workbook increase?
Connections to a dataflow are always live, in other words, nothing gets stored in power bi desktop.
/Ruth
I am just hesitant managing Data Flows, the referencing can build layers of complexities, also validating the data can be challenging. Bottom line, it may take away from the advantage of the 'Single Source of Truth' for developers.
Hi Ruth.. thankyou for providing such beautiful explanations. One issue that i found using Data flows is that the time for refresh is huge compared to datasources directly connected to pbix file. I guess the underlying reason is compression (vertipaq vs csv files). Is there any way to compress the data in dataflow itself?
Thanks in advance!
I dont know, let me see if I can find someone that knows
/Ruth
Great video!
Thanks Thomas !
/Ruth
Hi Ruth. Great video. Thanks a lot
Trying to summarize about costs, if I understood correctly, it is something like:
- using Data flows “seriously “ needs Premium license ($$$)
- Data flows with Pro license ($) doesn’t make too much sense since it is limited to 10 GB and almost all new features need Premium
- PowerQuery is (by now) free and it is almost same thing as Data Flows with Pro license but running locally instead of web based
Am I right?
Yes, dataflows are Power query online with the difference that dataflows store the data in a datalake so it can be reused in other systems while power query stores it on power bi or Powerpivot.
The only remark is that the cost is not for dataflows but for the data lake storage, otherwise bullseye 🎯, as I understand it at least!
/Ruth
Hi.... good morning 🌞
How to create line graph inside metrix in power bi..?plz help
Hi Shyam, please ask in the power bi community and give details to get help quickly.
Happy new year!
/Ruth
Hi Ruth, great video. Thank you so much. What hardware and software did you use for the notes? Would love to use it for our Internal Training Videos...
My apologies. I did not see the link in the description.
Not everything is there:
I used a MS surface book 2 (any pc with a touch screen will work) a surface pen and the software is called epic pen :)
/Ruth
@@CurbalEN Excellent :-) This will help us so much. Thanks Ruth.
Always a pleasure to help a fellow creator :)
/Ruth
What is difference between Dataflow and Dataset?
Very simplified a dataflow is Power Query online.
/Ruth
@@CurbalEN actually, we are having crazy discussion on what to use for report to improve performance from Dataflow or Dataset.
And end end with Dataflow. What you think?
Great video. May I know what does self service mean?
Sure, it means: do it yourself , in this case business users can grab data without the help of IT.
Hope the explanation helps :)
/Ruth
Understood. What makes it unique compared to desktop? How about refresh?
I explain that in the video, check it out!
/Ruth
Can try out Premium features using Power BI Embedded "A" SKUs: christopherfinlan.com/2018/11/18/how-to-use-a-skus-to-try-out-paginated-reports-in-power-bi-without-upfront-cost-or-long-term-commitment/comment-page-1/ Also, to handle the Dataflows version of Power Query being limited to GUI features, many people having success copying and pasting M code from Power BI into Dataflows, and having them run okay.
Will do!
/Ruth
👍👍❤❤
❤️
Seems like a lot of work just to put data into a silo.
There are for sure advantages and disadvantages with dataflows, but it will come in handy for some users.
Let’s see how it develops, it is at an early stage at the moment.
/Ruth
Hi Ruth look like Princesses
Delightful
:)
/Ruth