Im currently making a datavalidation script using pydantic. Combining custom constrained annotations with the dynamic model creation per variable works like a charm!
By utilizing `BeforeValidator`, `AfterValidator`, and `WrapValidator`, you can construct an Extract, Transform, and Load (ETL) pipeline directly within an 'Annotated' type hint. Additionally, you can achieve aggregated entity resolution by mapping known keys to a JSON field using `AliasChoices`, and serialize the parsed response within a single model from as many data sources as you need. Create trigger conditions using discriminated unions and fully automate processing procedures like OpenAI Vector Stores. 🐍🐍
Arjan, as a Dutchy but also as a human being: thank you for all of these videos, including this one. However, I am still looking for a video where you properly explain how to use the debug tools in an IDE properly rather than just printing everything and using "if self.debug=true: print(f"Beautiful variable in this method"{variable}"). Do you perhaps have something in the pipeline for this?
@wesleydevries7415 check out mCoding's recent video on debugging python in IDE's. I would specifically recommend using Pycharm for advanced debugging, it's unparalleled imho. Groeten!
If talking about python, why not to use built-in IDE debugger. For instance, in vs code you'll find debugger shipped with python microsoft extension. I believe it's there. Which is triggered via F5 key. If there's no option to run debugger or you prefer to use prints, logging can be the solution. The minimal logging level to print during runtime (debug, info, warning) can be set by its config. IMO the loguru lib is the easiest solution to use for logging
Wow 😲 my comment referring to a video from another youtuber (mCoding) that has an excellent introduction to IDE python debugging got removed (by you(r team), @ArjanCodes? That doesn't sound like you at all somehow...) Anyway, make sure to check that out! And use Pycharm, it's awesome 😎
I know I sound like a broken record in these comments but I really wish msgspec would gain the same traction. Does everything pydantic does but better.
Thanks arjan! Again a great video. I didn't knew about pydantic before and after watching this video, i will surely use this in my projects and make this an addon to my dev toolkit. Again thanks for this awesome knowledge! 😀
Thanks was having a problem upgrading from pydantic 1.9 to 2.8 seeminly some of the modules i wanted to use needed 2.8 - the headache was i could not understand why optional no longer worked - now i get it i had to specifically set the Field Value of all Optional Fields to None
I actually also miss doing those myself. We’re making a few changes at the moment to (partly) return to the style of the older videos. So stay tuned 😊.
Thank you Arjan ! I love pydantic with FatsApi, my only fault was putting too much business logic into it and not knowing where the data transformation was taking place :)
Pydantic would be better, if it would have 1) proper serialising and deserialising and b) If it would have a proper way to convert to and from snake_case and CamelCase for python/JSON stuff.
Hi Arjan I have a question regarding python processes. I am diving in to learn elixir. But I am wonderung if the similar functionality we can achieve from python? Especially all processes are like having individual dashboard to monitor the performance and self healing concept. Thanks in advance.
For some reason making compositions and accessing attributes from nested models just doesn’t seem to work the way it’s supposed to. I know my models work because I can query them in any other validation library.
Do you have an example of after creation there is a change in the value? To Validate or change the information. Example: set a new password in example2
Can you create your own types, like the EmailStr, so that they can be used in multiple classes? I feel like this would help reduce the massive class bloat for the final data container and help reusability. (Particularly my use case which would be laden with numpy fields!) Also, is pydantic still really slow when compared like for like against dataclasses and attrs?
That's what the BaseModel stands for, doesn't it ? You can create your own custom string type, or int type, or anything else you wish, and change every behaviour of the type (and also its validation, thanks to pydantic)
@@juanjosefarina ah so you'd basically nest types and it's solved recursively?? Ty! Definitely going to take a look and see how it runs on my embedded stuff. I'm using dataclasses at the moment which are nice and quick, even though I'm locked on 3.8 so no slots.
Hello, @ArjanCodes, can I ask you use, classmethod decorator in validator methods even if you don't need need class instance in code ("cls")?? Wouldn't it be better to use staticmethod?
With lots of models and some even nested ones pedantic is extremely slowing down our cold start time in the lambdas. Also with 3.12 + pydantic2 + fastapi and debug mode it sometimes takes up to a minute to start the application:/
If you use data from mongodb is there better way to create instances of pydatic classes then getting binary json as dictionary and dictionary unpack into pydantic instance? I looked into creating pandas dataframes with PyMongoArrow but I'm not sure I need that clunk.
💡 Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
Im currently making a datavalidation script using pydantic. Combining custom constrained annotations with the dynamic model creation per variable works like a charm!
Pydantic is a lifesaver when dealing with messy/nested JSON data. I love it.
Does dataclass provide same functionality with json?
@@coladockuse sqlmodel. Thank me later
Totally agree!!
Every time I see your video, I get new ideas about the code structure. Thanks a lot for your great job.
Thank you for the kind comment, Michael :)
By utilizing `BeforeValidator`, `AfterValidator`, and `WrapValidator`, you can construct an Extract, Transform, and Load (ETL) pipeline directly within an 'Annotated' type hint. Additionally, you can achieve aggregated entity resolution by mapping known keys to a JSON field using `AliasChoices`, and serialize the parsed response within a single model from as many data sources as you need. Create trigger conditions using discriminated unions and fully automate processing procedures like OpenAI Vector Stores. 🐍🐍
Arjan, as a Dutchy but also as a human being: thank you for all of these videos, including this one.
However, I am still looking for a video where you properly explain how to use the debug tools in an IDE properly rather than just printing everything and using "if self.debug=true: print(f"Beautiful variable in this method"{variable}").
Do you perhaps have something in the pipeline for this?
@wesleydevries7415 check out mCoding's recent video on debugging python in IDE's. I would specifically recommend using Pycharm for advanced debugging, it's unparalleled imho. Groeten!
😊
If talking about python, why not to use built-in IDE debugger. For instance, in vs code you'll find debugger shipped with python microsoft extension. I believe it's there. Which is triggered via F5 key. If there's no option to run debugger or you prefer to use prints, logging can be the solution. The minimal logging level to print during runtime (debug, info, warning) can be set by its config. IMO the loguru lib is the easiest solution to use for logging
Wow 😲 my comment referring to a video from another youtuber (mCoding) that has an excellent introduction to IDE python debugging got removed (by you(r team), @ArjanCodes? That doesn't sound like you at all somehow...)
Anyway, make sure to check that out! And use Pycharm, it's awesome 😎
Using breakpoints and the debug console of VS Code ? That gives you pretty much full control over runtime code and what is going on
I eagerly await the release of a new upload from Arjan every week. Never disappoints me.
pydantic is one of the best Python packages I've ever used.
I know I sound like a broken record in these comments but I really wish msgspec would gain the same traction. Does everything pydantic does but better.
Love pydantic, makes life much easier
Thanks arjan! Again a great video. I didn't knew about pydantic before and after watching this video, i will surely use this in my projects and make this an addon to my dev toolkit.
Again thanks for this awesome knowledge! 😀
I'm really glad to hear the video had a positive impact on your journey, Devan! Best wishes :)
Just saw for the first time that it's possible to use python primitive types in the dictionary as keys... Learning never ends. That's neat, thanks!
Glad you found it helpful!
Thanks was having a problem upgrading from pydantic 1.9 to 2.8 seeminly some of the modules i wanted to use needed 2.8 - the headache was i could not understand why optional no longer worked - now i get it i had to specifically set the Field Value of all Optional Fields to None
Ninja is also interesting to know about, in my opinion. It really is a life-saver for those of us who live in the Django world.
Is django a right tool for making web APIs?
@@matis9783 I recommend you use FastAPI instead of Django for APIs
@@matis9783 if you ask me: with Ninja it is.
Awesome explanation as always Arjan! Thank you!!!
Thank you for the kind comment, Bruno! Glad you liked it :)
Miss your more long form, typing the code included videos 😊. The click clack is unbeatable.
But it feels like those videos stick way more!
I actually also miss doing those myself. We’re making a few changes at the moment to (partly) return to the style of the older videos. So stay tuned 😊.
Pydantic 2 is much easier and cleaner than version 1, and thanks for clarifying this
Thank you Arjan ! I love pydantic with FatsApi, my only fault was putting too much business logic into it and not knowing where the data transformation was taking place :)
I'm glad to hear that the video was useful, David!
Pydantic would be better, if it would have 1) proper serialising and deserialising and b) If it would have a proper way to convert to and from snake_case and CamelCase for python/JSON stuff.
You can use the field method alias parameter to assign the different field name types
@@santoshmagadum7086 I know, but that's a workaround, not a good design.
Very informative. Can you/did you use pydantic for validation of sql queries?
Hi Arjan I have a question regarding python processes. I am diving in to learn elixir. But I am wonderung if the similar functionality we can achieve from python? Especially all processes are like having individual dashboard to monitor the performance and self healing concept. Thanks in advance.
7:02 my teamlead will "kill" me for code like on lines 53-55, it's unreadable. Why do not you use a match statement?
Fantastic video. Thank you.
Glad you liked it, Juan!
For some reason making compositions and accessing attributes from nested models just doesn’t seem to work the way it’s supposed to. I know my models work because I can query them in any other validation library.
6:43 line 58 why use a list comprehension when you can just provide a generator expression to join()?
Do you have an example of after creation there is a change in the value? To Validate or change the information. Example: set a new password in example2
nice video as always👍🏻
Thank you so much!
Can you create your own types, like the EmailStr, so that they can be used in multiple classes? I feel like this would help reduce the massive class bloat for the final data container and help reusability. (Particularly my use case which would be laden with numpy fields!)
Also, is pydantic still really slow when compared like for like against dataclasses and attrs?
That's what the BaseModel stands for, doesn't it ? You can create your own custom string type, or int type, or anything else you wish, and change every behaviour of the type (and also its validation, thanks to pydantic)
@@juanjosefarina ah so you'd basically nest types and it's solved recursively?? Ty! Definitely going to take a look and see how it runs on my embedded stuff. I'm using dataclasses at the moment which are nice and quick, even though I'm locked on 3.8 so no slots.
I think you should make a video about django-ninja
It is django with pydantic and it's mind-blowing
used it recently to get environment variables.
What about the Marshmallow library In python, can you please post a video about that!!..
The new pydantic version is deprecating example field, now you need to use json_schema_extra={"example": "123"}
Good to know!
Hello, @ArjanCodes,
can I ask you use, classmethod decorator in validator methods even if you don't need need class instance in code ("cls")?? Wouldn't it be better to use staticmethod?
Can a field be a descriptor?
How you handle uniques validations in combination with database in fastapi?
Thank you....
i use attrs !
What about schematics for python
you should had made analogy with zod
Miller Kenneth Jackson Brenda Brown Timothy
With lots of models and some even nested ones pedantic is extremely slowing down our cold start time in the lambdas. Also with 3.12 + pydantic2 + fastapi and debug mode it sometimes takes up to a minute to start the application:/
12:11 Oh and ellipses are no longer necessary for required fields. I think they're even discouraged
i suggest compiled fastjsonschema for deeply nested messy data
I would like to know your opinion about SQLmodel for fastapi
I like the idea of the library, but I’m waiting to do a video about it until it catches up to the latest version of SQLAlchemy and Pydantic.
your discord is not loading
Pydantic is epic, if it was fighting elections. He would win single handedly, cause there's no opposition.
why use pydantic when you can... just check the types with the isinstance() function?
Jackson Timothy White Gary Taylor Deborah
how you gonna write the test cases for them ?
What do you mean how ? One way was pretty much everything he showed in the video, but you should be using pytest or other testing library.
Williams Shirley Harris Melissa Davis Sarah
Young Brenda Brown Melissa Lopez Jennifer
Good video! The North face is racist
But you shouldn't be using poetry.
Why?
If you use data from mongodb is there better way to create instances of pydatic classes then getting binary json as dictionary and dictionary unpack into pydantic instance? I looked into creating pandas dataframes with PyMongoArrow but I'm not sure I need that clunk.
you can try beanie-odm☺