![Shipyard](/img/default-banner.jpg)
- 159
- 95 127
Shipyard
United States
Приєднався 26 січ 2021
We envision a world where solving problems with data is effortless.
Data teams are often overwhelmed, drowning in a sea of ad hoc requests and siloed data. This chaos leads to inefficiencies, frustration, and mistrust, impacting not just data engineers but also data analysts, data scientists, and ultimately, business stakeholders.
Our mission is to pioneer a world where hybrid code and custom code work seamlessly together, guaranteeing that each new data request is no longer a herculean task and allowing data to be used to its fullest potential.
We believe data can have the greatest impact when everyone is on board. That's why we prioritize unified accessibility, making it easy to move, manipulate, and monitor data, whether you prefer to write complex code or simply drag and drop.
This relentless focus on accessibility streamlines the time to impact, empowering data teams to deliver results quickly and build solutions that meet organizational needs now and in the future.
Data teams are often overwhelmed, drowning in a sea of ad hoc requests and siloed data. This chaos leads to inefficiencies, frustration, and mistrust, impacting not just data engineers but also data analysts, data scientists, and ultimately, business stakeholders.
Our mission is to pioneer a world where hybrid code and custom code work seamlessly together, guaranteeing that each new data request is no longer a herculean task and allowing data to be used to its fullest potential.
We believe data can have the greatest impact when everyone is on board. That's why we prioritize unified accessibility, making it easy to move, manipulate, and monitor data, whether you prefer to write complex code or simply drag and drop.
This relentless focus on accessibility streamlines the time to impact, empowering data teams to deliver results quickly and build solutions that meet organizational needs now and in the future.
Hakkōda and Shipyard Partnership: What It Means For You
Exciting news! We are thrilled to announce our partnership with the Hakkoda team, the Snowflake experts.
This video discusses how this collaboration will revolutionize data operations and make it easier than ever to create a data-drive culture.
Visit our blog to learn more about this game-changing partnership
#snowflakes #dataoperations #dataorchestration #datamanagement #data #datanengineering
Blog announcement:
Website: Shipyardapp.com
Hakkoda's Website:
Get started with Shipyard (it's free!): app.shipyardapp.com/auth/signup
Follow us on LinkedIn: www.linkedin.com/company/shipyard/
This video discusses how this collaboration will revolutionize data operations and make it easier than ever to create a data-drive culture.
Visit our blog to learn more about this game-changing partnership
#snowflakes #dataoperations #dataorchestration #datamanagement #data #datanengineering
Blog announcement:
Website: Shipyardapp.com
Hakkoda's Website:
Get started with Shipyard (it's free!): app.shipyardapp.com/auth/signup
Follow us on LinkedIn: www.linkedin.com/company/shipyard/
Переглядів: 39
Відео
How to Export Your Logs to Snowflake and Tableau
Переглядів 374 місяці тому
While you can monitor your processes in Shipyard's holistic data operations platform, sometimes you may want to have a dashboard in your own tool to monitor your Fleets. With the Shipyard API Blueprint, you can do just that! In this video, we walk you through how you can export logs from Shipyard into Snowflake and Tableau.
Inside the Life of a Data Consultant
Переглядів 334 місяці тому
Ever wondered what it's like to be a data consultant? In this video, get an inside look at the day-to-day life of a data consultant working with Fortune 500 companies. Plus, get some interview tips for breaking into this exciting field! Timestamps: Introduction and Overview 00:00 Introduction to the Captain's Compass series and today's focus on a data consultant's life. 00:15 Reminder to check ...
Data Science Project: BMI Index Predictions Using a Python Machine Learning Model
Переглядів 2035 місяців тому
Looking for a data science project idea? Watch this video on predicting BMI index using a Python machine learning model. Get insights on healthcare data science projects and learn more about Python and machine learning. 00:00 - Introduction and Project Overview: Celebrating HIPAA Compliance 04:15 - Setting Up: Importing and Preparing the BMI Dataset from Kaggle 08:00 - Data Cleaning and Initial...
The Life of a Data Engineer: What You Didn't Know
Переглядів 1375 місяців тому
In this video, we'll take a closer look at the exciting world of data engineering. From designing and building data pipelines to working with cutting-edge AI technology, we'll give you a glimpse into the day-to-day life of a data engineer. If you're curious about the field of data engineering, or looking to pursue it as a career, this video is for you! More Captain's Compass episodes Learn Juli...
Learn Tips for a Career in Analytics Engineering
Переглядів 1976 місяців тому
Learn Tips for a Career in Analytics Engineering
Unlocking Business Success: Choosing Dashboards or Spreadsheets
Переглядів 436 місяців тому
Unlocking Business Success: Choosing Dashboards or Spreadsheets
How to Start or Advance Your Data Career
Переглядів 336 місяців тому
How to Start or Advance Your Data Career
Revealing the Insider Tips for Business Intelligence Analysts
Переглядів 1727 місяців тому
Revealing the Insider Tips for Business Intelligence Analysts
Secrets of Shipyard Crew's New Year Goals
Переглядів 218 місяців тому
Secrets of Shipyard Crew's New Year Goals
Inside Look: Shipyard Crew's Favorite 2023 Projects
Переглядів 328 місяців тому
Inside Look: Shipyard Crew's Favorite 2023 Projects
Escaping dbt Cloud: Move to dbt Core
Переглядів 7098 місяців тому
Escaping dbt Cloud: Move to dbt Core
Data Engineering Project: Quickly Build a Complete Pipeline
Переглядів 47010 місяців тому
Data Engineering Project: Quickly Build a Complete Pipeline
Orchestrate Your Data Tools Free: Get the Details
Переглядів 9610 місяців тому
Orchestrate Your Data Tools Free: Get the Details
Data Engineering Project: Quickly Build a Complex Data Pipeline
Переглядів 10 тис.Рік тому
Data Engineering Project: Quickly Build a Complex Data Pipeline
Generative AI, BI and Analytics: Where It Is, Where It's Heading with Sonny Rivera
Переглядів 95Рік тому
Generative AI, BI and Analytics: Where It Is, Where It's Heading with Sonny Rivera
Data Orchestration Demystified: A Beginner's Guide
Переглядів 472Рік тому
Data Orchestration Demystified: A Beginner's Guide
Master Two Workflows, Fast: Snowflake and External Tools
Переглядів 43Рік тому
Master Two Workflows, Fast: Snowflake and External Tools
What are Ephemeral Files, Otherwise Known as Temporary Files?
Переглядів 72Рік тому
What are Ephemeral Files, Otherwise Known as Temporary Files?
What is Snowflake Data Unload and Why You Should Care
Переглядів 123Рік тому
What is Snowflake Data Unload and Why You Should Care
Fivetran and Jira Integration: How to Avoid Failure
Переглядів 109Рік тому
Fivetran and Jira Integration: How to Avoid Failure
What is Snowpark and Why Should I Care?
Переглядів 157Рік тому
What is Snowpark and Why Should I Care?
What Problems does Data Orchestration Solve?
Переглядів 220Рік тому
What Problems does Data Orchestration Solve?
Automate Transcribing your YouTube Videos
Переглядів 87Рік тому
Automate Transcribing your UA-cam Videos
Airflow Alternatives: Find the Right Data Orchestration Tool
Переглядів 157Рік тому
Airflow Alternatives: Find the Right Data Orchestration Tool
Improve Your Efficiency with Open-Source Low-Code Orchestration
Переглядів 86Рік тому
Improve Your Efficiency with Open-Source Low-Code Orchestration
Data Orchestration vs. Data Observability: Learn the Differences
Переглядів 69Рік тому
Data Orchestration vs. Data Observability: Learn the Differences
great video!
Such an informative video. I found it very useful. Just got a new job where i need to migrate entire Hadoop (oozie) workflows in Snowflake using dbt, with all 4 technologies new to me. This type of videos are extremely helpful. If you can make more on the modelling part with dbt, would be great
Thanks for the comment! What type of modeling content would you like to see from dbt?
Why not create an out of the box vessel for it with parameters to fill out for GitHub details and command to run (etc)? It sounded like you were saying there's some technical challenge... But there's a ton of other tools out there integrating dbt core from a GitHub repo, with a nice wysiwyg form. I think it would go a long way for you guys... To honestly say you have a dbt core solution instead of a python tool.
Great point! We have thought about making low-code templates for these, but we decided to give the users the ability to create this for themselves. There are small changes needed for each individual database that the user needs to take care of on their end.
Glad to see this!
Thanks so much! Any other content you'd like to see from us? BTW, we really like your channel!
Dear author, thank you, very interesting! Can you tell me how to download a ready-made file with subtitles directly? Without unnecessary steps, such as uploading to the cloud, etc.
We currently do not have a way to download directly to your PC. However, you could email it to yourself and download it that way.
Can this flow be revered. extract data from SqlServer in CSV file to google Storage
It absolutely can. You would use the Download Query as a CSV Blueprint with SQL Server then use the Upload File Blueprint with Google Cloud Storage
Full end-to-end project: ua-cam.com/video/IQxhutCbN7A/v-deo.html
How to do incremental load for a table while transferring data from snowflake to Big Query.
Essentially you would need to setup your query to do the incremental load. If the query runs in Snowflake, it will run in Shipyard
its given the error "pytube.exceptions.RegexMatchError: get_throttling_function_name: could not find match for multiple " when download the video
Hello! The package that we use for downloading UA-cam videos is having issues. We are working to rectify this as we speak. Sorry for the issues!
Any solution to do this with twitch livestream in real-time?
Hey Jeff! Shipyard isn't made for real-time data processing. I have built some Twitch integrations that work every 30 minutes or so to produce clips and things of that nature. You would need to use a product that can handle real-time processes to complete that unfortunately.
Beautiful and intutive! This is such an incredible demo. Thanks for showcasing our transcription models. We can't wait to see what you have in store next!
Set all this up and hope I can get it to work but getting an error from Whisper in the logs that says: openai.error.APIError: Maximum content size limit (26214400) exceeded (26416764 bytes read) To be fair the video is 1h 50m long so may simply be too big. I wonder if we could set an in/out point on videos by timecode so Whisper only transcribes shorter segments.
I got it to work using this video (ID Yu-gWepOiS4). Looks like limit is 25MB for the audio file in Whisper. Instead of translating to another language however I'm just using a prompt to OpenAI to 'Summarize the text' (of the transcribed video). BUT this is what it provided as a summary for this video... weird 🤔 "The text talks about the benefits and drawbacks of working from home during the COVID-19 pandemic. The advantages of remote work include less commuting, increased flexibility, and improved work-life balance. However, remote work can also negatively impact mental health due to isolation and lack of social interaction. Additionally, working from home can blur the boundaries between work and personal life, leading to increased stress and burnout. Finally, the text concludes that a hybrid model of in-person and remote work may be the best solution post-pandemic."
Super interesting. Unfortunately, we can only work in the confines of the APIs provided. You could definitely write a Python script to chop the video into pieces and roll from there.
Hey, I'm interested in exactly this setup! Especially with webhook and a variable; Reached out to your support
Just got over a response! Look forward to chatting with you!
Interesting - Bit of a clunky / slow method using shipyard for me though... you could create an OpenAI API token, ask ChatGPT for the Python code to write a small program that takes the file as an input to perform an API request chatGPT asking to comment the code. Then send it to postfix or whatever the Windows land equiv is and you could do this from the terminal / cmd in one line :)
No doubt about that! We will never claim that we are the only solution to any of the use-cases we provide. We just want to make sure that people have the option to do those things in the platform if they want!
@@shipyardapp It wasn't a dig, just an observation - Seems as though Shipyard might be more of a user friendly Jenkins style pipeline for your cloud needs?
Absolutely right! We are trying to take some tasks that were seen as too technical and make them available to the masses!
Very important question sir , How can I translate a .txt file from a specific language to an another specific language (not English)? How do I specify the second language please ? test.txt --model medium --language Spanish --task translate Spanish --> French
Hello Pierre! We currently do not have any translation services in Shipyard natively, however you could write a custom Python script to do that translation in Shipyard. Feel free to reach out in Intercom in the application if you need help with that script!
I was having issues setting up python / whisper and everything else that goes along with it on my computer. I am not expert or even an armature when it comes to this type of stuff, but shipyard is pretty amazing. it makes things 100x less complicated. thanks
That's great to hear DJ! I'm glad you have been able to get some processes running in Shipyard!
Can you schedule this to be sent on a monthly basis?
Absolutely. We have scheduling and webhook triggers to start Fleet runs. You can read more about them here: www.shipyardapp.com/docs/reference/triggers/schedule-triggers/
Can you help me to get the slack token for my channel?
Sure! If you check out our documentation here: www.shipyardapp.com/docs/blueprint-library/slack/slack-authorization/. We walk through the steps to get the Slack token. If you have any other questions, feel free to reach out to us using intercom on our website.
Hi, i have an error in sending email, can you help me with this?
Sure! Feel free to send us a message through Intercom on our website, and we would be happy to help!