Create custom keys for your Power BI relationships
Вставка
- Опубліковано 31 лип 2024
- Enough of these single table models! Model your data and create custom keys within Power BI to make your reports more efficient!
⏬ Download sample: guyinacu.be/createkeyssample
📢 Become a member: guyinacu.be/membership
*******************
Want to take your Power BI skills to the next level? We have training courses available to help you with your journey.
🎓 Guy in a Cube courses: guyinacu.be/courses
*******************
LET'S CONNECT!
*******************
-- / guyinacube
-- / awsaxton
-- / patrickdba
-- / guyinacube
-- / guyinacube
-- guyinacube.com
**Gear**
Check out my Tools page - guyinacube.com/tools/
#PowerBI #GuyInACube - Наука та технологія
Note that doing this approach could result in the underlying table being queried multiple times on your data source to satisfy the multiple Power Query queries.
Could you create one query of the source in power query and then "refer" to it multiple times rather than copy? Would this reduce the number of times the source is queried? I've done this many times for consistency sake, but am not sure if it results in the source being hit multiple times.
I usually then disable load of the "source" query so it does not become a table in my data model.
@@danneubauer6474That won't help. Each query is evaluated in isolation. Also for cleaner reference to just a single column use: MyTable[[MyColumn]]. This way you don't have to convert list to table vs MyTable[MyColumn]
From what I can tell the only way to truly avoid this is with premium capacity or premium per user dataflows which then do allow you to do true query chaining with the results of each query stored to disk for use as a source by downstream queries.
Rather than creating a new item "Not Supplied", I use "" or "". That way those items appear first in an ascending sorted list or a slicer. Plus they stand out when looking at many rows in a table. Now that it is clear they exist in my data, I can take steps to address them.
Thanks for sharing that John! nice trick to get them to the top.
AA😂😂😊😅😊😅,p
S
,,,zx,,,,,zzzz**
Dude I can't tell you how much your videos have helped me. I inherited a mess of a database in my new position and had no one to really learn from. You rock at teaching
That's awesome to hear. Thanks for watching! 👊
I’m just learning Power BI and your videos are so helpful and fun to watch! Thank you so much!
Excellent video! Exactly what I needed with no unnecessary filler. As a budding data engineer, this was a huge help! You are both a scholar and a gentleman! :-D
This is great! Thank you so much! You guys make this fun to learn. Keep up the good work!
so thank you for this video. there was not too much talking, but only the amount to provide support for the steps. i love how you broke the task down into easy to follow steps and explaining why it was done that way. 🙂
Nice video. I like these quick an useful data wrangling type videos. Please keep them up.
Dude, you just saved my life! I was looking for it and all the results I got were like "how to replace ID with the name", and I wanted the opposite! You just got a subscriber from Brazil! Great content! Cheers!
Hey Patrick, your videos are just awesome. Thank you so much for such easy to understand and accurate explanations ! Great Job
Appreciate that! Thanks for watching 👊
Awesome!! I am going to try this method! Thank you for walking through it.
I have been searching for you. Great video🥰🥰 thanks a lot
Love every your video, huge help to me. Thank you so much!
I can’t thank you enough for this video! I think this will solve our problem at work! 🙌🏻🙌🏻
Awesome! 👊
What I like is how the star schema changes how you look at you data. It organizes your thought. Segments your perspective.
You don't even imagine how I learn looking your videos, thousands thanks for your great job
This video helped me a LOT, thank you so much
Thank you so much ! This helped me a lot.
Just what I needed. Thank you
Thank you for giving good information
Great video, thanks for all the help
Thanks, I actually used this yesterday and your steps worked like a champ, YA HIGH FIVE! Thanks for explaining what trim does too, that tool is very helpful. Going to put my gloves back on, clean clean clean data haha good thing I have a janitorial degree from the Corp. haha
This was perfect. Thank you!!
Thanks for the video. It was really helpful.
Thanks, Patrick. Good stuff as always!
Thank you! 👊
Brilliant way to generate look up tables. Thank you
That is seriously good stuff! I've been thinking about something similar and now I have the ultimate solution to make this work. My only more-burning question at the moment is how do I get one of those Power BI coffee mugs... lusting after that!
Patrick, very cool video.. Normaly you receive multiple tables and have to do something with it, now you give an example if you receive one big (=wide) table. Thank you for this interesting example.
This is incredibly helpful. Thank you so much!
Happy to help. Thanks for watching! 👊
Great video, Patrick!
Very helpful - thank you!
Really useful tutorial for messy data. Thanks!
Thanks ! Excellent advice at 6:45 !
Came looking for exactly this. Great stuff!!
looking for what?
Surrogate key?
This video helped me a lot thanks was getting low percentages merging tables
Very dope video. It really helped me
Again, I went back here to refreshen up the ideas you got here Patrick. It really helped me a lot with my stuff! How about using this method in Import mode of connection then the data is updated, Does the other table will also be updated together with your other keys? Thanks a lot man!
Thanks man, one hour looking for this
Great video! thanks!
Thanks dude. I didn't know about this method which doesn't use the "duplicate function". Much easier!
Love it! 👊
thanks this helped a lot!
Thanx Patrick, when comparing this method with using Dax combine values is there a performance difference?
Thanks for confirming I wasn't that nuts when doing exactly this. The merge step can take a LONG time for bigger tables though. Nice video!
Indeed
That is why I doubt if it really useful to use that technique. I see huge cons (long merging in PQ) and tiny pros (a little bit more usability).
If we speak about a small model, there should not be any noticable difference in productivity, but if we consider a big one, we could merging in PQ would be painful.
So why should we do all of that and pay more then receive? Or when this approach really matters and helps?
@@hukumka2601 I have the same struggle currently, as I need to decide which approach to take for generating relationship keys (create integers vs concatenated keys). I like integer approach however it slows down refresh time significantly. Luckily I have a premium PBI capacity so I am considering moving most of the data transformations to data flows.
This was so helpful! Now I’m trying to add more columns from my two fact tables to the new tables 😅 without my PK’s yet.
Another great video!
Hi Patrick! I had to create IDs and I did a very similar process but instead of 'right click -> add as a new query', I've duplicated the entire table ('right click on main table-> duplicate') and from them I've performed exact same steps that you, what is the difference? Thanks for creating such great videos!
Espectacular!! Muchas gracias!
Appreciate it, thanks! 👊
Your instructions are certainly stepping stones towards becoming "a big deal", keep 'em comin'
BAM! 👊 Thanks Alexius!
Great one Patrick..I'd add one more step to yours and hide the airline id from the transactions table
Yep. All tallies with what I do ! Thanks for confirming !
Thanks for watching! 👊
I wish that I had watched this video last week! I did this in a much more manual way.
Excelente, Just saved me hours of Work!!
That's awesome! 👊
Great video! If you get additional data, let’s say, with a new airline, will the refresh process take care of everything? Meaning add the ID to your airline table?
Great job, Patrick. This is helpful. I am going to use this technique in my dataflows so that it doesn't slow down the refresh. My question is about CamelCase. I heard (from the Tabular Editor Best Practices Analyzer) that CamelCase is not best practice. Why do people say that and what do you think?
Thank you so much for the video.
How to connect Dynamic folder(File name changes day to day) ? Extracting the data from new file through refresh is getting failed?
Thanks alot
Hey Patrick, great video! Do the Airline query update the names when new ones is added in your ERP system?
Thanks for this, now I know how to narrow my fact tables down
Nice video, do we have other methods to remove many to many in power bi?
Hi Patrick ! love the video ! Just one question that I couldn't find answer to : does matching on Id's rather than Airplane name improves the performance of the model ? Thanks à lot
Good Stuff Patrick!!
Appreciate that Nelson! 👊
Come back to the UK Nelson 😊
Wolfstar eheh I will eventually! But for now I’m enjoying this February’s - almost summer weather - in Lisbon :)
This is beautiful
Hey Patrick, I want to use a parameter to filter Top N output of a matrix ; using the parameter slicer. Could you please show me how? Thanks
You must be a mind reading Jedi. I needed this as I am currently doing a similar approach through much more convoluted methods of duplicating tables and removing columns to get down to a basic ID Table. Your method will save me much time and my mind and emotional state are very appreciative. Have a question of how I can possibly have my company's IT department give me access to view the relationships they have built and had created through our data warehouse. Currently, they have many many tables with similar or exact names of different columns and attributes. It's making me go through a process of trying out different combinations figure out where they have pulled the data from and what relationships they have built between the two.I relate it to shooting a target with an arrow, in the dark, blindfolded, and with one arm tied behind my back. I'm not lazy, I just want to be efficient and not waste my time with the guessing game approach I find myself in.
ask them. Develop a relationship with someone in ICT.
Cool T-Shirt Patrick !!
Thanks. 👊
I use this method all the time, works very well! Cool to see you guys use the same methods!
What's the best practice for troubleshooting the data once you've broken everything out?
For example, if you need to sift through that fact table by airline name, it become rather tedious to go back and forth between the tables matching keys. Worse yet, if you have multiple dimensions that are filtering the fact table, it can be difficult to identify the proper keys to look through the fact table.
If the source is a relational database, this could be done in the database, but in this situation, the source is a CSV or other file, so that type of out-of-Power BI querying is not possible.
Thanks!
I usually create a table visual in report space with the columns I need, and just add some slicers for the dimensions I care about QCing. Then I browse the data in report space rather than in query space.
Amazing video
Cool video, thanks! Do you have a video about caveats of joining on strings? Tnx!
We do not. We should definitely do something about strings. Lots of things to consider.
thank u
so cool!
I really liked the way of subtly taking performance into consideration (Look for int), is there a visual form of execution plan? What would be the equivalent of SQL Execution Plan to use with Power BI? Exists?
Cool! Tks.
Most welcome!
Hi, Patrick, if new Airline is added to original table will be it auto added to new query
Thanks
John
Great!!!
Hi Patrick can please do a video on aggregation , i have created an aggregated table using fax and want to create a dynamic filter for a column not included in the aggregation table
Hi Patrick , I love your explanation very much , actually im beginner , pls help me below , i want to lookup one particular product in another table , but that product was booked by two different customer , finally it was sold to one customer , how to create a relationship for this from one table to another table
Best explanation ever.
Thanks for watching! 👊
Is having the relation on ID giving a better performance than just having it on airlinename in the airline helper table? Is the performance gain of this worth the performance overheat you mention for generating the keys? I would (until I saw this video) just have made the link on airline name....
Thanks in my real world with no clean data warehouse the data modelling and joining is the biggest hurdle to using power bi
Superbbbb bro
Thanks! 👊
Patrick, I am using your solution but I am facing performance problems when I join (inner join) both tables by the text key column (number of rows 1.000.000 aprox). Thanks
What if we have to consider multiple columns for this approach?
Also, if we have to use a table as global filters across different sheets developed using different table. I.e. star schema.
First of all, mapping tables are cool:)
When your end user is someone who knows how to use PBI, then this method may come in handy to clean up the main table, however most people on the end of the chain possibly just know the frontend and only things they will change will be the filters. Therefore there is no need to create an artificial (in this particular example) mapping table.
Nevertheless great video, I really enjoy your content
I get what you say. But will this improve performance or size of the data? Will that be a valid reason to create this kind of tables?
@@impala4641 I find star schemas useful for speeding up report performance, however when one needs to build your star schema from your fact table, this can really reduce refresh performance, so one needs to balance these two points. If you're scheduling refreshes you might be able to offload refresh performance to off peak hours. Win win.
@@impala4641 it’ll reduce the RAM required to hold the data model, and also make some DAX calculations easier. Power BI is designed to use Star Schemas.
Hello All, Is there a method of automating the process of Merging two tabular model ? I am using manual method in BISM normalizer
Great video Patrick. Such a clean way to create lookup table and join. I have 2 related questions.
1. If I need to join 2 tables on multiple columns (e.g. composite keys), do I create a lookup table with those columns from the 1 side of the 1:N relationship?
2. If I need to join on BETWEEN clause, e.g. table1.date between table2.startdt and table2.enddt, what would be the best approach?
Thanks in advance.
@ANIRBAN PAL, did you solve your two challenges?
How do you add the ID back into the fact table if you want to avoid (merging for query load time reasons)?
Maybe a newbie question, but still. I come from SAP BW world. How to ensure a new index will be automatically created and a new entry will be automatically added into this Airline dimension table when a new unique Airline name appears in source data (excel, csv, table,...).?
hey Patrick... would about joining on alphanumeric keys with Tpye as "any" ?
The most beautiful part is that it makes that column disappear from original table!
I do this also, but instead of merge I do a transform with my buffered table. So if I have multiple columns, it's one step. I usually do
TblID[ID]{List.PositionOf(TblID[Element], [ThingToReplace]}
I don't know if the merges would be faster
I love this t shirt
Can you still use this method if the incoming values for Airline are constantly changing? (e.g., new airlines are regularly being added to your original table)
Great video, I do this when I want to split up a column that has multiple values, such as a tags column that would have a list of tags delimited by semi-colon. That way the user can select a single tag and see all matching rows that have that tag.
Question: Why duplicate instead of reference if you are doing multiple columns?
Hi Zoe, as far as I know, you cannot merge referenced queries, only duplicated ones
Thank you Luis! That makes sense
Master!
How would this work with multiple columns. Columns example: cost center, cost center mapping and period.
Is there a way to automate this , i have 40 tables I need to move from the flat-file
Would be REALLY cool if power query had an automated way of doing this. Right click on column -> "Create Unique Dim Table" and it does this all automatically.
This is a beautiful and relevant video Patrick. I've often found myself thinking about this business case where the dimension data is long text strings and doing joins on such dimensions is fraught with uncertainty at the best of times. There is one use case I find myself thinking about, which the video does not address so I'm pinging to understand how you have thought about this. Imagine the flat file had different values for "TWA", "Transworld Airlines", "Trans World Airlines". This technique would create a different custom key for each of these entries - in reality however, these should point to the same key. Therefore, using this technique in power query will not cover this particular use case. Up in my head, the only way to do this is through manual intervention where the key is inserted through a manual scan of the table to ensure that "TWA", "Transworld Airlines" and "Trans World Airlines" all point to the same key. Short question - is there a way to reject this "lazy" technique and become more "efficient"??!!!?!
I'm sure you've already found your answer, but a creating a Transformation table would solve that problem. Power Query's documentation would show you how to do that.
Amazing!
Glad you think so. Thanks for watching! 👊
Is there any way i can have only one APPLY button for multiple Slicers/filters? Such that my visualisations don't get updated every time i select a filter.
Thanks
I don't know if there's a way for that. But you could add an apply button for each slicer.
Go to File --> Options and Settings --> Options --> scroll down to Current File --> Query reduction and voila.
Hope this helps