Tim Higgins
Tim Higgins
  • 143
  • 28 508
Medallion Architecture - Video 13 -Talk about the next steps for Gold by using Aws ETL to move data
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena
Video 13 Talk about the next steps for Gold by using Aws ETL to move data
I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session... Lots of videos but it really didn’t take long, concerning we started from scratch.
dataengineering.wiki/Concepts/Medallion+Architecture
Переглядів: 6

Відео

Medallion Architecture - Video 12 Gold Copy - Aws Crawler/ Aws Athena / Aws ETL / Aws QuickSight
Переглядів 1519 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 12 Look at our data in Athena/ looking at an Aws ETL tool and introducing Aws Quicksight for dashboards I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all ...
Medallion Architecture - Video 11 Look at and update our crawler/ Introducing Aws Athena
Переглядів 319 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 11 Look at and update our crawler/ Introducing Aws Athena I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session... Lots of vid...
Medallion Architecture - Video 10 Setting up our Aws Glue Crawler and using the Aws Data Catalog
Переглядів 1119 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 10. Setting up our Aws Glue Crawler and using the Aws Data Catalog I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session... Lo...
Medallion Architecture - Video 9 finishing our ETL/Silver by getting our data ready for Aws Glue
Переглядів 1319 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 9. Finishing our ETL to Silver layer by getting our data ready for Aws Glue I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working sess...
Medallion Architecture - video 8 Continuing our ETL to Silver by building the Merge layer to Aws S3
Переглядів 1619 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 8. Continuing our ETL to Silver by building the Merge layer with Python Pandas to Aws S3 Buckets I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a re...
Medallion Architecture - Video 7 Starting the ETL to Silver
Переглядів 3319 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena 7. Starting our Silver Layer by building an ETL with Python, Aws Boto3, and Aws S3 Buckets I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world wor...
Medallion Architecture - Video 4 Building our Bronze layer by starting our code in Python and Boto3
Переглядів 6319 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 4. Building our Bronze layer by starting our code in Python and Boto3 I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session......
Medallion Architecture - Video 5 Sending a bulk amount of data to Aws S3 using Python/Boto3
Переглядів 5419 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 5. Coding our Bronze layer by continuing with Bulk upload to Aws s3 with Python and Boto3 I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-worl...
Medallion Architecture - Video 6 Coding our Bronze layer by refracting (cleaning) our code
Переглядів 1219 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena Video 6. Coding our Bronze layer by refracting (cleaning) our code I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session... Lots of ...
Medallion Architecture - Video 3 Start our Bronze layer by looking at our data
Переглядів 4919 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena 3. Start our Bronze layer by looking at our data I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session... Lots of videos but it real...
Medallion Architecture - Video 2 Create a bulk amount of realistic data with Broadcom TDM for our LZ
Переглядів 6319 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena 2. Create a build amount of realistic data with Broadcom TDM for our LZ I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working session... Lot...
Medallion Architecture - Video 1 Building out a Medallion Architecture from scratch w/ Python & Aws
Переглядів 6219 годин тому
Building out a Medallion Architecture from scratch with Python, Aws Boto3, Aws S3, Aws Glue and Aws Athena. Video 1. What is the Medallion Architecture design and how should we get started? I had a use case for handling a large amount of data for querying. I used the Medallion Architecture design to build this out. I thought it would be fun to share this with you all in a real-world working ses...
Using Broadcom OI APIs to Export Metrics
Переглядів 53День тому
This is for learning the process, I don't actually recommend this because OI has the power to show you everything and then some. knowledge.broadcom.com/external/article/206461/aiops-how-to-query-for-metrics-and-top.html#mcetoc_1fic5bbldv
aws S3 buckets from the cli (command line)
Переглядів 55День тому
aws S3 buckets from the cli (command line)
A quick video for setting up JetBrains with Aws and run the LLM demos locally
Переглядів 75Місяць тому
A quick video for setting up JetBrains with Aws and run the LLM demos locally
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Переглядів 36Місяць тому
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Переглядів 72Місяць тому
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Переглядів 32Місяць тому
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Переглядів 22Місяць тому
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Переглядів 19Місяць тому
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Переглядів 36Місяць тому
Aws Serverless architect with Aws API Gateway, Foundation Models, Lambda and S3 buckets series
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 9-9 Final Thoughts
Переглядів 132 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 9-9 Final Thoughts
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 8-9 deploying
Переглядів 132 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 8-9 deploying
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 6-9
Переглядів 102 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 6-9
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 5-9
Переглядів 102 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 5-9
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 7-9
Переглядів 62 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 7-9
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 2-9
Переглядів 182 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 2-9
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 4-9
Переглядів 82 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 4-9
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 3-9
Переглядів 202 місяці тому
Get data from Broadcom Spectrum and PM/DA with Python into Json/CSV files part 3-9

КОМЕНТАРІ

  • @MSLifeExperiences
    @MSLifeExperiences 2 місяці тому

    You didn't get a single like in 4 years, but your one point in this whole video helped me a lot, thanks.

    • @timhiggins4440
      @timhiggins4440 2 місяці тому

      thank you very much!!! I am glad it helped!

  • @ShivaKumar-ix8xg
    @ShivaKumar-ix8xg 4 місяці тому

    Hi Tim, its a great video, but I would like to know more about Pre-Scripts & Post Response Scripting. I tried with Jscript, but looks its still failing with attributes. If you could help me with your vide if you made it already or any document where i can refer them. It would be a great help & thanks in advance.

  • @suhailNXT
    @suhailNXT 5 місяців тому

    Hi Tim, Have a question. How do we create relationships between 2 or more tables in the process of synthetic data creation.Thanks in advance

    • @timhiggins4440
      @timhiggins4440 5 місяців тому

      Hi, I can show you that but give me a few days. If you are doing Data Gen, in the middle of the data painter you can choose columns form other tables. They are other ways but this is a quick answer. Let me know if this helps or I can explain it more if needed.

  • @utubeasif
    @utubeasif 7 місяців тому

    Thanks for this wonderful video, it really helped, I have used Postman API testing tool extensively, this just looks so similar to me.

  • @W4EMB
    @W4EMB 8 місяців тому

    Tim, Great stuff. Loving your audio quality and the interviews... A whole other world out there beyond our own circles... Thanks for sharing!!

  • @richardhallebeek
    @richardhallebeek 9 місяців тому

    Thanks for this great opportunity to talk tech, gear and history Tim! I really enjoyed it and thanks for coming up with the cool questions! ;-)

    • @timhiggins4440
      @timhiggins4440 9 місяців тому

      Thanks man... the honor was all mine. I truly enjoyed it and we must do it again, soon.

  • @W4EMB
    @W4EMB 10 місяців тому

    Thanks Tim! How awesome. Thanks Joe. Hey Tim, great "studio" set up. Wokld live to know more about your recording process... Id like to improve my audio and video from my "almost studio"... Ok, not studio... My man room. Thanks!

    • @timhiggins4440
      @timhiggins4440 10 місяців тому

      yea man... 1st thank you!!! 2nd, lets get together and I will be glad to show you!!!

  • @bannochang
    @bannochang Рік тому

    is this tool opensource or Licensed ?

    • @timhiggins4440
      @timhiggins4440 Рік тому

      Sorry its Licensed through Broadcom but a GREAT TOOL

  • @bannochang
    @bannochang Рік тому

    is this tool opensource or Licensed ?

  • @uttamanand6364
    @uttamanand6364 Рік тому

    Do you have any plan to post a video of DevTest being deployed on OpenShift?

    • @timhiggins4440
      @timhiggins4440 Рік тому

      let me see if i can get one together... if not i'll get you to the right place for instructions

  • @abdoanahr4288
    @abdoanahr4288 Рік тому

    ي❤❤»❤

  • @DamelaMicha-se4ye
    @DamelaMicha-se4ye Рік тому

    Muy buen video

  • @W4EMB
    @W4EMB Рік тому

    Sweet!

  • @biirajenepher7921
    @biirajenepher7921 Рік тому

    How can eastern countries get you

    • @TimHGuitar-l8v
      @TimHGuitar-l8v Рік тому

      can you send me a private message. I am not sure if youtube will do that

  • @biirajenepher7921
    @biirajenepher7921 Рік тому

    I need also to join you because I have liked your business

  • @W4EMB
    @W4EMB Рік тому

    Sweet !

  • @jagsalariahomes
    @jagsalariahomes Рік тому

    At what point of time we create Travel_E schema? Is this one of the pre-requisites before we could start creating even the Project- Dataset - Pool in Data Maker? This means we require DBA's support and expose PII data before we could do subsetting and on the fly masking? Great Video by the way!

    • @timhiggins4440
      @timhiggins4440 Рік тому

      Yes when you download TDM one of the options is to download the demo databases. Sadly TDM is a database tool so you will need to interact with the database sometimes... but you could also use files if you are just learning. You can do fly masking with Subsetting, I have a video on masking from subsetting. see if you can find it.

  • @gopi929020
    @gopi929020 Рік тому

    Thank you for explaining each and every concept in the portal. I have one doubt.. how to run pre or post SQL in the portal data masking

    • @timhiggins4440
      @timhiggins4440 Рік тому

      Hello Sorry for the late response but you can run a Javelin in your publish in a pre or post publish Look at the Javelin video

  • @danielthirumalai
    @danielthirumalai Рік тому

    Thanks for sharing your knowledge. helped me a lot in my company, hope you put more videos thanks

    • @timhiggins4440
      @timhiggins4440 Рік тому

      thank you Daniel. I will put more videos up soon.. Im glad to hear this helps.

    • @ArunKumar-ox3qm
      @ArunKumar-ox3qm Рік тому

      I have 2.5 years experience in devtest and now I'm searching for job could you refer me

  • @W4EMB
    @W4EMB 2 роки тому

    Great stuff Tim. Thanks for the video!

  • @W4EMB
    @W4EMB 2 роки тому

    Who knew ??... Awesome video. Thanks Tim!

  • @W4EMB
    @W4EMB 2 роки тому

    Short vids help close the loops! Thanks Tim!

  • @W4EMB
    @W4EMB 2 роки тому

    Now the advanced stuff! Digging deep now! Keep the videos coming

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      thanks man... I thought of you while I was doing this :)

  • @W4EMB
    @W4EMB 2 роки тому

    Such a great video.

  • @sree7200
    @sree7200 2 роки тому

    How can i add data rules on my table column in generator?

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      can you give me an example of what you are looking for? If you are talking about if user = tim then do this... then you can use the data painter for that. I have some videos on that.

    • @sree7200
      @sree7200 2 роки тому

      @@timhiggins4440 yes exactly.my requirement is i have a table with no data.i need to create some test datas for that table and export to excel.but i need to do some data rules like for the coloumn SSN,FName,last name.the challenge is i need to take some value in a seed table and need to use that in rules.. Could you give your vedio link for data rules

    • @sree7200
      @sree7200 2 роки тому

      @@timhiggins4440 exmple validation for SSN columns

  • @W4EMB
    @W4EMB 2 роки тому

    Nobody gets it done like Tim. Great stuff!

  • @W4EMB
    @W4EMB 2 роки тому

    Now we are getting to the good stuff!

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      well thank you sir!!!! we need to have game night soon!!

  • @W4EMB
    @W4EMB 2 роки тому

    Ahhh 64 bit!........ Now it all makes sense. Tim, you're the man!

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      Glade I can help you out my friend.... ha ha.. you can always count on me ...

  • @W4EMB
    @W4EMB 2 роки тому

    Nice!

  • @W4EMB
    @W4EMB 2 роки тому

    Awesome follow up video on part 1!....

  • @W4EMB
    @W4EMB 2 роки тому

    Finally! Great info!

  • @ark5048
    @ark5048 2 роки тому

    Hi Tim, Can you create a video on how to configure GT maker with my tdm repository

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      Yes I will be glad to, I am making a GT Datamaker set of videos today. Tell me if this helps, or I can give more info.

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      see if this helps ua-cam.com/video/g67UqyvBFyU/v-deo.html

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      Sorry I had to reupload the video... try this one ua-cam.com/video/-QOvNvdlehM/v-deo.html

    • @ark5048
      @ark5048 2 роки тому

      @@timhiggins4440 unfortunately I delete the gtrep connection in GTmaker. Let me try to connect with 64bits odbc with mentioned configuration. I appreciate your effort on help make configuration. I'll update you the result shortly

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      @@ark5048 good luck.. let me know if you get it working

  • @chadhudson2046
    @chadhudson2046 2 роки тому

    I bet you can do a great Eddie Van Halen impersonation. You sound just like him.

  • @ss-ss6qt
    @ss-ss6qt 2 роки тому

    ​ @Tim Higgins Hello, Can you please provide your emai id ? I want to reach out to you ?

  • @W4EMB
    @W4EMB 2 роки тому

    Thank goodness! .... Soooo been waiting for this!.

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      Ha ha.. I've missed you man... how are you?

  • @sudhakargopal0
    @sudhakargopal0 2 роки тому

    Hi Tim.. Can you help me that i need to get trained in ARD

    • @timhiggins4440
      @timhiggins4440 2 роки тому

      Sadly I am not sure, Broadcom did have some sites for training. My videos should help a lot. If you need more, I would try Broadcom.

  • @praveen123ist
    @praveen123ist 2 роки тому

    Hey Tim, great video .. thanks so much! appreciate it ..

  • @Suman-pc9kr
    @Suman-pc9kr 3 роки тому

    Can we re run the masking job if the test DB is refreshed with the production data ? It happens in regular interval in my organization

    • @timhiggins4440
      @timhiggins4440 3 роки тому

      you can save your masking and run it when ever you need it. That way if production data comes down, you can re run the masking

  • @W4EMB
    @W4EMB 3 роки тому

    Finally! We've been waiting on this one!

  • @W4EMB
    @W4EMB 3 роки тому

    Great stuff!

  • @snehalkhot5937
    @snehalkhot5937 3 роки тому

    Very good video. But a humble request can you please show like how can we take api key, entity from newrelic to integrate blazmeter to it. I tried it but am not able to get entity in blazmeter.

    • @timhiggins4440
      @timhiggins4440 3 роки тому

      sorry I have been busy but yes I will be glad to. are you still having issues with this

    • @snehalkhot5937
      @snehalkhot5937 3 роки тому

      @@timhiggins4440 It will be good for all of us if you add that video aswell

  • @W4EMB
    @W4EMB 3 роки тому

    That filled in the blanks!

  • @NaveedAshraf1
    @NaveedAshraf1 3 роки тому

    Nice!!

  • @W4EMB
    @W4EMB 3 роки тому

    Finally! The one I was waiting for!.. Great Information Tim.

  • @Suman-pc9kr
    @Suman-pc9kr 3 роки тому

    Appreciate your efforts Tim. Is there anyway I can learn all the components of TDM ? Complete course or something ?

    • @timhiggins4440
      @timhiggins4440 3 роки тому

      sadly Broadcom has training but its not free. If you watch all my TDM videos I cover them all. My company can also provide training.

    • @prabhaek4908
      @prabhaek4908 Рік тому

      Hi

    • @prabhaek4908
      @prabhaek4908 Рік тому

      Can I know how to enroll for training ?

  • @arijitghosh7742
    @arijitghosh7742 3 роки тому

    Is there any trial version available?please give the link

    • @timhiggins4440
      @timhiggins4440 3 роки тому

      Hey man, sorry I am the architect and I don't do sales. If your company is looking to try TDM, email me and i can get you in touch with my company.

  • @W4EMB
    @W4EMB 3 роки тому

    Probably the best 22 minute instructional video I've seen today! Great stuff. Keep it coming. Thumbs up from me!

  • @ramalakshmisugavanam6936
    @ramalakshmisugavanam6936 4 роки тому

    Hi Tim, Are you with Broadcom? we are looking for corporate license. Do you have any clue how to get it?

    • @timhiggins4440
      @timhiggins4440 4 роки тому

      Sorry for the late response. We are a partner with Broadcom; you company should have a sales rep. If this isnt to late send me a private message with your email and I can get someone to help you.

  • @prasadraghu2008
    @prasadraghu2008 4 роки тому

    Hi Tim, how are you doing? suppose you have a use case where we need to pull and display columns from more than one table after joining the tables using relationship, how would you do that in Find and reserve? I tried but I am not able to design Find and reserve request such that columns across 2 to 3 tables are populated in the results table after you click on Find Data from self service.

    • @timhiggins4440
      @timhiggins4440 4 роки тому

      When doing Find and Reserve, you want to create your model 1st. You can create relationships in the model that you want to use. Try that... that should work for you. Let me know if that helps but spend more time creating your relationships with the data model

  • @sathiyalr
    @sathiyalr 4 роки тому

    Thanks Tim. Thank you so much for the detailed Demo

    • @timhiggins4440
      @timhiggins4440 4 роки тому

      Thank you very much. Please let me know if you have any questions.