Відео

I thought the same thing when I saw Daft. There was a great demo of it at the Ray conference earlier this year

Some obvious remarks with a lot of rage bait on top. What a BS

you are rigth why i wanna to be in the office if even that all my team is in another office in another country i am remote employee since pandemic really who likes to go to the office i think that the answer is the people who dose,nt have family , friends or don,t have hobbies who likes to spend 3 or 4 hours driving

Fair enough comments. If analytical processing risks interfering with transaction processing, it does not really help to do full table scans. Having said that: transactional processing is brilliant if you can drip-feed changes to a duckdb database sitting elsewhere. That may be faster than using materialized views with aggregate-refreshes. Something like BemiDB which fills a duckdb database from postgress can be handy. Maybe compare the speed between that extract and postgres.

Thank you for making this! I combed through all the pg_duckdb promotional material trying to figure out what amazing innovation made pg_duckdb analytical queries so much faster than their pure Postgres equivalents. After all, the main thing that's supposed to make most OLAP databases (including DuckDB) so much faster is compressed columnar storage, and pg_duckdb doesn't change the row-oriented storage method of Postgres. I couldn't find an answer better than "it's magic." Glad to know that the "magic" is just to not use the main optimization tool that Postgres has. I will say that it's still impressive that the pg_duckdb query performs at least on the same order of magnitude as Postgres (with indexes). I think this should have been the main selling point. If you are an analyst who has to write a bunch of ad-hoc queries, it can be really painful and time-consuming to have to think through the indexes at every turn. An index that works well for some analytical query might be useless against a tweaked version. Indexes are a resource that take time to build, especially for large data sets. If you can get close to index-based performance but eliminate the need for index building altogether, that usually amounts to a substantial net gain, especially for ad-hoc work.

TLDR(?) : a programmer makes a very good system but he is being a perfectionist. But other people say his current craftwork is fine. They have disagreement. More people know and take sides.

For an opinion, this one's pretty spot on. I would say that linting/formatting errors shouldn't wait for CI/CD, and should instead be trapped with pre-commit hooks so it never gets to the place where another person needs to witness it.

I do system migrations for a living, this involves a lot of data. I'm a developer that uses python code to do all transformations and loading. There are also analysts that work on the projects, who use SQL to do their analysis, so that they can check the data looks like it should, and what outscoping should be done. This makes it necessary for the tables I'm saving supports querying wtih SQL. Is there a datalake solution that has full SQL support? Or should I adopt a hybrid approach where I save json and mapping tables (.csv) to the datalake, but save all tables to e.g. Microsoft SQL Server? Thanks.

Nice overview. I wasn't clear on what DuckDB was about but it's clearer now. I think I prefer the polars snippet due to the better modularity. It's closer to the unix philosophy: do one thing and do it well. Integrating with cloud storage providers in a database engine seems a bit weird to me. I do agree that the appeal is obvious and my preference isn't a strong one. The DuckDB way is also way more portable, as you can just copy the SQL code and run it wherever.

But traditional RDMS will continue to be used to serve data to applications, right ? Because applications require fast response times and RDBMS are still faster to serve data for small to medium size data volumes.

SQL programmers are bad programmers? It's nothing but your ignorance. SQL can do it in one SQL what you writing pages of crazy code that is converted to SQL altimately. Stop your BS advice. People are more knowledgable than you.

wow, all the information in 2 minutes. Thank you so much

Who does RTO serve? For some, WFH is the right path - it avoids the commute (unpaid time, travel costs, harm to environment and health), reduces the inevitable distractions in open floor plan workplaces, and allows many people to work where they thrive. It can be counterproductive for some, and for jobs with poor ability to assess employee performance it's also a great way to hide incompetence. In-office can be useful for social types (though that socialization might come at the cost of others' productivity) or for work that simply can't be effectively done remotely. It's also a performative way for C-level to pretend that they're doing something meaningful to address financial woes, and gives managers a reason to exists (after all, you can't manage people when you can't hover over their shoulders, apparently).

I've been remote since COVID, and full time in office before thet. I miss the whiteboarding sessions the most. There's a lot less idea sharing remote because you're chatting less. Also makes my days blur together. The last four years went by in an instant. Startup offices aren't beige hell holes. I worked on site at AWS and it wasn't either.

It would be helpful to see more of the code, while you are talking about the code.

I got an offer recently for 'Junior Data Software Engineer'. Do you think Is it somehow way of naming one of these group?

How would Group 3 folks still be called Data Engineers?

Link to full article dataengineeringcentral.substack.com/p/there-are-3-types-of-data-engineers

RE: os.environ.get Why not use pydantic to load your env vars? It'll ensure the types are correctly casted, and create a "define once, use anywhere kind of deal. Unless os.environ.get was only for demonstration purposes, then I've said nothing.

Yeah for the most part, the syntax is better than pandas

Are the databrick jobs you are talking about, notebooks? If not could you describe what kind of jobs you are using to presumably run python (pyspark) code? I'm not yet familiar with the capabilities of databricks

How are you hosting the apache airflow? You guys have a Linux server at the office? I work at a small startup, and don't feel like this is something I would want to be responsible for (I'm talking setting up a self hosted server).

Very Helpful, Please use OBS or something to record screen and your face at one place.

Thank you for sharing this. really useful. Audio can be little louder.

I wish you would’ve asked how many work with streaming data (near real time and/or real time data) pipelines.

I'm extremely inexperienced and have only used notebooks in the cloud (so for programs that need to run continiously). Is the alternative to notebooks just docker containers? How does a company generally run python code in production? I work at a startup and I'm the only technical guy.

Skill issue 😂

Pure gold here 😄😄

What I’m struggling to understand is how it’s even possible to make notebooks work in production such that there is an entire faction of people shilling for it. Like how are their companies even functioning when 9.999999/10 cases the notebook breaks because the cells are ran out of order or something

That blog got you a subcriber

If you had a post every day, I’d read every one. Your posts are excellent.

Interesting, thanks for sharing!

Not everyone a christian

very good advice ;)

Polars also has a lot of issues reading and writing parquet files. Recently logged a bug, and got fixed a week after, but the update broke an other part of writing parquet files in my code, so I just convert all my dataframes to pandas before writing them away

Lots of words but saying nothing

dataengineeringcentral.substack.com/p/snowflake-is-dying-on-the-vine

I don’t think most places will leave Snowflake because all these databases migrations cost A LOT of money. Especially for a place that uses snowflake as a Data Warehouse. The people who need it more for AI/ML & Data Warehousing those people will probably be more likely to leave snowflake to databricks. Also like you said, people on old tech that actually want to or need to move to the cloud will probably go to Snowflake first if they don’t have enough people skilled with python & spark to move to databricks.

I think snowpipe, snowspark and dynamic tables in snowflake are great. New features doesn’t mean great software services. It could mean confusion

Weirdly, you are like invisible on google. I can’t find your blog after finding it once.

The substack link to full article. dataengineeringcentral.substack.com/p/cicd-for-data-engineers

Can you post the substack link

What data did you say you used, I didn’t catch the name

Resorting to setting up a raspberry pi cluster in my room so I can peacefully learn Spark

I for one am happy they're using Ray instead of spinning off with their own software. Though I agree on the (slight) bait-and-switch of their claims.

Also, please normalize the sound in your videos. Your voice is a lot more silent than your outro music 😅

Speaking of S3 - did AWS fix the bug where if you know the bucketname of some one else, you can spam that bucket with requests that end up with 404s, racking up your bill, because a call was made?

Good video Daniel, ran to your substack to check out if you had made a new post but didn't find any lol, i enjoy all the rust DE content you have made

To be clear you are running databricks notebook with airflow, what is your view with using pulumi to push your spark scripts to databricks for running your pipeline. Do you use the approach too

Heard it takes a year to learn the build system. Phew! As for "search" , it is just a catalog.