That buffering gotcha tho. I think the reasoning is that it wants to give you the greatest control of syscalls possible. Buffering in some cases would not be zero cost (and it needs heap allocations).
I think the API is designed to separate concerns. The raw stream is intentionally lacking buffering to make its API simple and orthogonal to the buffer layer.
Buffering is not necessarily a sensible default. Interactive applications behave really strangely with buffering. In Python it takes some reffort to disable buffering on IO streams every time you do interactive stuff.
I think the cynical frowning upon Rust's "defaults" is really misplaced because the examples you gave are also not the defaults in the C APIs; if you did fopen & fread surely you're also not surprised that it's not buffering for you behind the scenes? :thonk:
If you code in C you obviously know exactly what every stdlib function is doing. Also, since C programmers are naturally smarter than all other developers, this is not an issue:)
I would suppress the noise from the beginning, that would reduce the stored data too since indexed data will be eventually stored to a DB. Nice content of course.
1:12:00 - this feeling is precisely what is at the heart of mathematics and why it is studied in its pure form at all. i think itd be really cool to do a stream where you explore a higher level pure mathematics concept and work on implementing it, the same way you often do for a lot of higher level development ideas. i believe there's a lot of value in seeing how the (now) two different studies, really arent as different as they look
I do use LSP for highlighting issues with my code, and I really like it, although knowing this universal way of compiler driven refactoring is also nice, at least I now know what to do if I have to program just using a notepad and a command prompt lolz.
@@javierflores09 Oh is it already? Sry I assumed from this that it's just direct matching between search terms and the index. I must have missed him implementing fuzzy term selection in a previous stream.
@@naturallyinterested7569 ah, no it wasn't streamed yet I believe, he implemented stemming off camera or so it seems. I too thought I missed it when I looked at the repo but after watching this stream I see that it isn't the case
Patterns are symmetry; you want to continue the pattern, because it's symmetrical and being symmetrical is easy. The edge of symmetry and breaking symmetry leads to new discoveries. Physicians do that all the time.
I was so proud of myself being a rust beginner only starting 2 weeks ago ,when he did a count for an iterator, around 21:13 and I was screaming how is he making it work without defining as mutable the compiler's thugs beat me up when I do that but then he compiled and got an error . I now know I am not alone in this world
Oh wow, someone soured on Rust. I mean, I get that this situation is suboptimal. But just having buffering always enabled also is suboptimal, I've seen that cause a lot of issues in other situations. Which is probably why it's off by default, also finding a way, that doesn't end up causing issues, to opt out would not be easy or even possible. Its also not as if other languages are always buffered. They buffer in some instances, don't in others, good luck finding out when what is happening. Rusts design isn't intended to make someone feel stupid. It is meant to make you aware of what is happening, to be unsurprising and very tunable, and generally it does succeed with that. In many instances other than the one here it'll find a way to make you aware of these gotchas. In this case it failed, I'll give you that, and yes, in MANY cases buffered IO will work better. But saying that the reasons it was designed as it was is to make one feel stupid is, sorry, but just wrong and IMO not a very good approach. Rather say that you are fine with not being able to tune some things in the exceptions, that the added ease of use in most other situations is worth it, and that thus you prefer language X that "just does the right thing" in most cases.
@@javierflores09 that's because that isn't what I intend to say. Rather this is a specialized version of "blindly being rude ain't productive". Rust is indeed my favorite language, but it sure aint perfect. The root cause behind what he's saying is indeed problematic, but sadly hard or perhaps impossible to avoid, and what he's saying isn't exactly actionable or close to the root issue.
I like your approach. I'm just starting to write Rust code and I will use seroost for other docs. I'm pretty sure of that. Maybe the search page should get some love... Awesome stuff and you've got a new subscription to your channel.
Why use `PathBuf` as the key for the hashmap in the TermFreqPerDoc? Why not use the inode instead. The inode is just a u64. Then store a inode -> path separately and retrieve the relevant data when needed.
"pub" as default would be really bad. Requiring the developer to be explicit about everything instead of lots of "magic" is one of the big advantages of Rust. If that's not what you want, Python or something like that is probably a better choice.
@@gzoechi It is just a matter of default option. You be explicit on public or private, just a matter of default. It doesn't make one more explicit than the other.
You are inserting a 1 in the global frequency table when the term does not exist; this is wrong I believe, instead it should be the term count from the given document which can exceed 1.
I may be wrong but I think the right thing is to use a 0 in the unwrap_to and resolve the division by 0 issue with the log rule log(n/m) = -log(m/n) as n is always > 0. I think this makes compute_tf and compute_idf can be easly factorized.
better to index all the words and calculate tf-idf for each word with meta data. the words and their metadata could be hashed and retrieved in O(1) time during a search.
Crate devs love feature switching way too much. ifdef compilation switches are bad practice in C for a reason, giving them a different syntax doesn't make them a good idea.
Serde serialized a struct as an array? wtf? Is that an option you can change in the derive macro? I guess an array is more efficient but it's throwing away information about the name of the struct fields.
Fewer timestamps today because I'm starting to hit the description size limit of 5000 characters.
Don't worry tsoding, thanks for time-stamping at all
Insane effort
crazy effort
That buffering gotcha tho. I think the reasoning is that it wants to give you the greatest control of syscalls possible. Buffering in some cases would not be zero cost (and it needs heap allocations).
I think the API is designed to separate concerns. The raw stream is intentionally lacking buffering to make its API simple and orthogonal to the buffer layer.
i think because rust creator was interested in embedded systems maybe the unbuffered readers/writers got added first.
more rust sessions please...
Buffering is not necessarily a sensible default. Interactive applications behave really strangely with buffering. In Python it takes some reffort to disable buffering on IO streams every time you do interactive stuff.
I like that you take the time to index your streams. Mostly i watch them in one step but every now and then it helps to find an important part. 👍🙏
0:53
Тудей-сюдей, today, сегодня))
I think the cynical frowning upon Rust's "defaults" is really misplaced because the examples you gave are also not the defaults in the C APIs; if you did fopen & fread surely you're also not surprised that it's not buffering for you behind the scenes? :thonk:
If you code in C you obviously know exactly what every stdlib function is doing. Also, since C programmers are naturally smarter than all other developers, this is not an issue:)
@@alexpyattaev shiiiiii, do I take the bait?
agreed!
Forcing the program to go a certain path is usually called a unit test
I would suppress the noise from the beginning, that would reduce the stored data too since indexed data will be eventually stored to a DB. Nice content of course.
1:12:00 - this feeling is precisely what is at the heart of mathematics and why it is studied in its pure form at all. i think itd be really cool to do a stream where you explore a higher level pure mathematics concept and work on implementing it, the same way you often do for a lot of higher level development ideas. i believe there's a lot of value in seeing how the (now) two different studies, really arent as different as they look
Thanks for the timestamps mr.tsoding
4ms + all the http overhead is impressive indeed
Actually I found that random perdoc is kinda funny word for me too. Thank you for making me a little laugh.
I do use LSP for highlighting issues with my code, and I really like it, although knowing this universal way of compiler driven refactoring is also nice, at least I now know what to do if I have to program just using a notepad and a command prompt lolz.
Fuzzy, live searching would definitely be interesting. Although for fuzzy you'd probably need a separate list of all terms.
this is already fuzzy search what do you mean
@@javierflores09 Oh is it already? Sry I assumed from this that it's just direct matching between search terms and the index. I must have missed him implementing fuzzy term selection in a previous stream.
@@naturallyinterested7569 ah, no it wasn't streamed yet I believe, he implemented stemming off camera or so it seems. I too thought I missed it when I looked at the repo but after watching this stream I see that it isn't the case
13:45 I think this monologue should be put on the main page of the rust website
Patterns are symmetry; you want to continue the pattern, because it's symmetrical and being symmetrical is easy. The edge of symmetry and breaking symmetry leads to new discoveries. Physicians do that all the time.
Physicians are medical experts, actually. Shout out to Group Theory and Category Theory tho!
@@ezg5221 I think he meant physicists :)
I was so proud of myself being a rust beginner only starting 2 weeks ago ,when he did a count for an iterator, around 21:13 and I was screaming how is he making it work without defining as mutable the compiler's thugs beat me up when I do that but then he compiled and got an error . I now know I am not alone in this world
Oh wow, someone soured on Rust.
I mean, I get that this situation is suboptimal. But just having buffering always enabled also is suboptimal, I've seen that cause a lot of issues in other situations. Which is probably why it's off by default, also finding a way, that doesn't end up causing issues, to opt out would not be easy or even possible.
Its also not as if other languages are always buffered. They buffer in some instances, don't in others, good luck finding out when what is happening.
Rusts design isn't intended to make someone feel stupid. It is meant to make you aware of what is happening, to be unsurprising and very tunable, and generally it does succeed with that. In many instances other than the one here it'll find a way to make you aware of these gotchas. In this case it failed, I'll give you that, and yes, in MANY cases buffered IO will work better.
But saying that the reasons it was designed as it was is to make one feel stupid is, sorry, but just wrong and IMO not a very good approach. Rather say that you are fine with not being able to tune some things in the exceptions, that the added ease of use in most other situations is worth it, and that thus you prefer language X that "just does the right thing" in most cases.
That's a lot of words just to say you are being rude to my favorite language
@@javierflores09 that's because that isn't what I intend to say.
Rather this is a specialized version of "blindly being rude ain't productive".
Rust is indeed my favorite language, but it sure aint perfect. The root cause behind what he's saying is indeed problematic, but sadly hard or perhaps impossible to avoid, and what he's saying isn't exactly actionable or close to the root issue.
@@javierflores09 oh wow, someone soured on someone soured on someone soured on Rust. Better go pretend they're a zealot!
13:45 edge lord mode activated ⚡
I like your approach. I'm just starting to write Rust code and I will use seroost for other docs. I'm pretty sure of that. Maybe the search page should get some love... Awesome stuff and you've got a new subscription to your channel.
Looks like we're live! Hello TsCoding!
Could you just move search token loop higher than document loop to calc idf once per search token?
can your google do that?
14:00 what a guy
Why use `PathBuf` as the key for the hashmap in the TermFreqPerDoc? Why not use the inode instead. The inode is just a u64. Then store a inode -> path separately and retrieve the relevant data when needed.
Well for one, using inodes would make the code non-cross-platform.
The god of debagging
wasn't pagerank basically tf-idf over a graph of tf-idfs?
simple & easy = просто и легко. С различиями ты загнул жестко.
Hey man did you do any stop words and word stemming algorithm?
Keep it up 👍
My goal is to generate as many compilation errors as possible. (c) Tsoding
lets play RUST the videogame
"pub" as default would be really bad.
Requiring the developer to be explicit about everything instead of lots of "magic" is one of the big advantages of Rust. If that's not what you want, Python or something like that is probably a better choice.
What you are saying doesn't make sense.
It literally has no connection between those two things.
@@nataestanislaubastos7637 That you do not understand it, doesn't mean it makes no sense.
are you the guy that i often see on flutter stackoverflow answers?
@@gzoechi It is just a matter of default option. You be explicit on public or private, just a matter of default. It doesn't make one more explicit than the other.
You are inserting a 1 in the global frequency table when the term does not exist; this is wrong I believe, instead it should be the term count from the given document which can exceed 1.
I may be wrong but I think the right thing is to use a 0 in the unwrap_to and resolve the division by 0 issue with the log rule log(n/m) = -log(m/n) as n is always > 0. I think this makes compute_tf and compute_idf can be easly factorized.
better to index all the words and calculate tf-idf for each word with meta data. the words and their metadata could be hashed and retrieved in O(1) time during a search.
00:13:45 so damn true!
13:25 If Stockholm Syndrome were a language...
Lmao per-doc!!
Crate devs love feature switching way too much. ifdef compilation switches are bad practice in C for a reason, giving them a different syntax doesn't make them a good idea.
Serde serialized a struct as an array? wtf? Is that an option you can change in the derive macro? I guess an array is more efficient but it's throwing away information about the name of the struct fields.
No, it didn't. It serialized the tuple as an array, which doesn't have field names anyway. It made the struct into an object.
What's up! Build a package managar for C or C++.
PerDoc 🤣🤣
The Rust Language behaves to a programmer like its creators ... 😉