Hi This is the best video on duckdb atm. How do you get resources on it? I heard you can connect to a postgres db using a Postgres scanner. How does that work?
for example i don't think it supports the nice extended Syntax like EXCLUTE. but the main reason is speed. duckDB will execute way faster, pandas SQL is still pandas in the end. (correct me if i'm wrong)
i'm sorta stuck... your VSCode is running like a Jupiter notebook... how so... what did i miss... your screen shows a run cell run above debug above... where does that come from
Thanks as always for the great content Marc! Just as a sanity check for myself, OLAP Databases don't necessarily need to be columnar based correct? Recent big data databases use columnar based storage, but any database that processes analytical workloads is considered an OLAP database even if it is row based correct?
Thanks, great lesson! Could you help to fix an error, it occurs when I try to run the sales table that we`ve created. The error is "Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds" Don`t understand why it`s trying to get nanoseconds with the date format. Thanks!
I am getting this error Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds and run conn.execute("from sales").df() when i create a sales table. Not sure if you have seen it?
I was able to solve it by dropping the non date values df = df.loc[df['Order Date'] != 'Order Date'] df['Order Date'] =pd.to_datetime(df['Order Date'],format='%m/%d/%y %H:%M') In the sql block i made this change to match df format strptime("Order Date", '%Y-%m-%d %H:%M:%S') Looks like the trycast is not working or some bug
@@MarcLamberti In order to work with this tutorial, you provided the sales dataset in the materials. There is a link in the description. On this page, there is a link to the kaggle dataset that you used in the video. However, this one got removed. So it is not possible anymore to follow this tutorial :(
Here is important information. Jupyter notebook extension needs to be added to VSCode. How about using the StackOverflow principle and avoid the worthless thank you comments.
Oh man, oooofff! I tried following your video here and your blog with the stock analysis data. I've run into too many errors unfortunately. That's the thing about these dependencies - there's too many moving parts where errors are persistent. I can sort through them, but it would take up too much time. We all got work to do :-). For data analysis work, my go to still at this point is using an AWS S3 bucket to load the raw data, use an AWS Glue to create a database and an AWS Glue Crawler to upload the data while creating a table within that database. I can also use AWS Glue Studio to convert the data from .cvs format to .parquet format. From there, I can use AWS Athena to query the data.
When I ran this Python/Duckdb it only shape returned (186862, 6): # with duckdb cur_time = time.time() df = conn.execute(""" SELECT * FROM '/kaggle/input/sales-product-data/*.csv' """).df() print(f"time: {(time.time() - cur_time)}") print(df.shape)
Would love a full Udemy Course on DuckDB & dbt from you!
I would buy it
That's game changing, very nice video, Marc, Thank you.
This is a great intro video, thank you.
Thank you ❤️
great tutorial, thanks for inspiration and your interstings lessons
Dude.....I like your style! Subscribed and hit the bell....
Thank you 🙏
Brilliant tutorial, appreciate your work.
I did smashed the like "booton", thanks brother, very helpful content!
Thank you 🙏
Thank you!
Please make more videos
Thank you! Great content as always!
Thanks Marc
thx a lot, I learned a lot
Awesome content! Thanks a lot!
The link in the description only provides one dataset. How do I get all the CSVs?
do you have a video on your vscode config setup?
Great, thanks!
You’re welcome ☺️
Thanks.
Awesome ❤
Thank you 🙏
Hi This is the best video on duckdb atm. How do you get resources on it? I heard you can connect to a postgres db using a Postgres scanner. How does that work?
Why not use the pandas SQL functions?
for example i don't think it supports the nice extended Syntax like EXCLUTE. but the main reason is speed. duckDB will execute way faster, pandas SQL is still pandas in the end. (correct me if i'm wrong)
i'm sorta stuck... your VSCode is running like a Jupiter notebook... how so... what did i miss... your screen shows a run cell run above debug above... where does that come from
Thanks as always for the great content Marc! Just as a sanity check for myself, OLAP Databases don't necessarily need to be columnar based correct? Recent big data databases use columnar based storage, but any database that processes analytical workloads is considered an OLAP database even if it is row based correct?
Thanks, great lesson! Could you help to fix an error, it occurs when I try to run the sales table that we`ve created. The error is "Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds" Don`t understand why it`s trying to get nanoseconds with the date format. Thanks!
like always... way to few likes for such a great video!🎉
Would love an dbt+ duckdb course from u on Udemy. Big fan of yours 🎉
I am getting this error
Conversion Error: Could not convert DATE (0019-04-19) to nanoseconds and run conn.execute("from sales").df()
when i create a sales table. Not sure if you have seen it?
Let me double check that
I'm getting this error too!
I was able to solve it by dropping the non date values
df = df.loc[df['Order Date'] != 'Order Date']
df['Order Date'] =pd.to_datetime(df['Order Date'],format='%m/%d/%y %H:%M')
In the sql block i made this change to match df format strptime("Order Date", '%Y-%m-%d %H:%M:%S')
Looks like the trycast is not working or some bug
4:28 it is written redhshit :D Please don't mind. Thanks for the video.
🫣🥹🥹🥹🥹
The files are not available anymore :(
What files?
@@MarcLamberti In order to work with this tutorial, you provided the sales dataset in the materials. There is a link in the description. On this page, there is a link to the kaggle dataset that you used in the video. However, this one got removed. So it is not possible anymore to follow this tutorial :(
oh oh, let me check if I can fix that
@@MarcLamberti Thank you! Let us know if it works again
Here we go: www.kaggle.com/datasets/kushagra1211/usa-sales-product-datasetcleaned
Here is important information. Jupyter notebook extension needs to be added to VSCode. How about using the StackOverflow principle and avoid the worthless thank you comments.
Oh man, oooofff! I tried following your video here and your blog with the stock analysis data. I've run into too many errors unfortunately. That's the thing about these dependencies - there's too many moving parts where errors are persistent. I can sort through them, but it would take up too much time. We all got work to do :-).
For data analysis work, my go to still at this point is using an AWS S3 bucket to load the raw data, use an AWS Glue to create a database and an AWS Glue Crawler to upload the data while creating a table within that database. I can also use AWS Glue Studio to convert the data from .cvs format to .parquet format. From there, I can use AWS Athena to query the data.
Except that baziz, everything is perfectly fine.😀
When I ran this Python/Duckdb it only shape returned (186862, 6):
# with duckdb
cur_time = time.time()
df = conn.execute("""
SELECT *
FROM '/kaggle/input/sales-product-data/*.csv'
""").df()
print(f"time: {(time.time() - cur_time)}")
print(df.shape)