Understanding HDFS using Legos
Вставка
- Опубліковано 25 сер 2024
- You've been hearing about Hadoop and HDFS. How does it work?
In this video, we use an innovative method to show how HDFS works with Legos. Jesse Anderson shows how HDFS handles files and replicates the data, then covers the read and write paths for the data. Finally, he talks about how HDFS handles failure scenarios and the importance of data locality.
** Hadoop & Apache Spark training from NewCircle: newcircle.com/...
![АЛАУДИНОВ у Скабеевой: эти их ВСУ нас НЕ ДОГОНЯТ 😁 [Пародия]](http://i.ytimg.com/vi/an0anqU4WGQ/mqdefault.jpg)
This video was uploaded 5 years ago and it still one of the best explanations I've ever seen.
the first explanation at all that could tell me what Hadoop has to do with MapReduce
Nice explanation man, if I could, I would buy you an air conditioner.
You deserve it.
may the force be with you
sorry to be offtopic but does anyone know a trick to log back into an Instagram account?
I was stupid forgot my account password. I would love any tricks you can give me.
@Billy Dariel it worked and I actually got access to my account again. Im so happy!
Thank you so much you really help me out!
@Miguel Chris Happy to help :D
@@billydariel9140 nice ad
You really have a " hands-on" approach of teaching this. : )
This is a best explanation I have seen yet! It's a lot more engaging and informative than the traditional PowerPoint! Thank You!!
11:06 R.I.P. Data Node 3
He was a good node, admired by his node friends.
I'm still crying, can't get over it, such a good node dude rippp
He is happy in the Node Heaven and is saving all the replicated data it ever wanted to save.
Nice job Jesse. Very informative and creative. Thank you.
TeraTom
Excellent video. I actually understand Hadoop somewhat after watching this video.
I love learning with legos, even watching it at 1.5x, I was able to follow along easily. Well Done.
Wow man, you explain it in a creative way.
The explanation is so clear I understood everything
Excellent teaching. Conceptually things are very clear now. Thank you.
Thank you for such a good explanation!
This video just made my day! Thank you New Circle Training! And thank you Doug Cutting for sharing this video on Twitter.
Really nice and innovative way to teach hdfs concept. loved it and understood it very clearly. Thank you.
Superb, I had it running at 1.5x and it was still easy to follow! Thanks :)
Can you explain every CS concept with Lego? - that would be amazing
thanks dude lol
Amazing explanation !! one of the best videos I've seen about HDFS
Jesse! Great explanation as always.
Great introduction to HDFS!
very nice teaching methodology jesse, thanks for sharing
Nice work. Really help me to understand how HDFS works.
Great explanation! Voted!
nice illustrative way of teaching HDFS. Would have been wonderful if some more information was given about fallback mechanism of Name node or coordinator
gorgeous presentation, thanks
Incredible illustration! Would love to see more on this topic in this form :-)
Nice clear explanation
thanks for the explanation
Very innovative presentation thanks a lot!
Thanks for your efforts! Smart display!
Very good work for begginers - THANKS A LOT!
Good Presentation, however I do have question. Since the file is split to other nodes, doesn't replication also take places so that if a node does go down, then you can retrieve. Node 3 went down, but I would have figured I could get it from another node. Does master save a copy of all files as well? I didn't see how that works on the video.
This was great, thanks.
I'm just now learning about the methods used in distributed file systems. I'm an application developer and it's a bit difficult to wrap my head around the lower level storage systems like HDFS.
This video explained replication in an easy to understand way. Now only if I could have one other BIG question answered.
What kind of file system would we use if we want an application to use a virtualized file system stored over many nodes? For example, we want each node to add additional storage capacity.
To the application layer, this would look like a single big storage drive but to the lower level facilities this would use network coordination to serve the files to the application.
ĺm ĺ
looks like we have a perfect explaination!
Excellent video, thanks!
This was great, thank you! Should have more views.
Great Job Jesse! Nicely done! :-)
Excellent, thank you. A serious question: What would happen if two of the four nodes crash?
very informative, thank you
Great explanation!
Thanks 😊
Question: when we have to read from let's say the red file. Would all three nodes be processing simultaneously different data (chunks) and give us an combined output or would only one node process the complete the data processing alone?
Awesome, Thank you for the explanation, Could you please make videos of Spark with Yarn and how the communication is handled?
Very nice video! Only one question: If node 3 is down, how can the data from node 3 be moved to the other nodes?
@@brianbitchballs3902 thanks for the great explanation BrianBitchBalls
Thanks a lot
Perfect! Thanks so much
Did we find who broke node 3 yet?
Amazing !!! great Job
If a node goes down, why does it need 3 replications to pull the data, could it not just read from the existing two?
Nice Job. Brazil thanks you!
what does happen when you add a new and empty node?
Thank You
For the red file, are EACH of the replicas still 64 mb in size?
if a data node malfunctions, then name node instructs remaining data nodes to create replicas of files that malfunctioned data node held. why create another replica when there are two others already? I mean they are there for the backup, right? Do correct if I am missing something. Thanks.
Brilliant!
Nice job, it is helping getting familiar with HDFS. I'm new to Data Engineering and so on.. Could you please explain to me what is a cluster.
Thanks;
Great explanation! I assume that another benefit of the HDFS is that reading large files will be quicker, because you would be able to effectively "BitTorrent" from your cluster. Does HDFS do that?
feel like buying Legos already!
What if hulk smashes the naming node?
yoou haveabsolutely no need to be nervous! Doing a great job here
Very creative, indeed.
Interesting, it's very similar to how Elasticsearch works
Hi,
While setting up pseudo or full cluster, do we need to format data node with HDFS file system or we only have to format namenode.
I am asking this because, I have read this on many blogs that, HDFS stores the data in sequential order on the hard disk and it is an abstract layer which stores data on big blocks rather than default block size storage provided host file system.
If we are not formatting datanode than the powerful feature of HDFS will come to toss.
Hello, nice job,
I have a question, as a file system how hadoop can manage a database file(like mssql or mysql) file? how can it devide an .MDF file to other separeted files to store them on different machines? Because such files, have meta data and overhead and they are not like a basic txt file. Does hadoop have special system to treat each file type differently? Regards.
why s3 service streaming with embedded link is slow (get stock every 2-3 second) when embedded on wp site? after understanding what you saying how can i improve the speed? i read something about the hdfs but i don't understand how it's related to s3 if at all. thanks in advance
tnx
thanx!
The fact that hbase write on a node as you say is the reason why it corrupts the HDFS filesystem so easily?
What will happen once Data Node 3 is alive again? will the Name Node asks other nodes to copy some data to Data Node 3?
did you get a AC by now ?
Cool
I don't understand. If each set of blocks is "one" file (red, yellow, blue) why he says that blocks are replicated? He should say "distributed", not replicated! Replicating involve data resiliency, erasure code and so on. Is it?
Then he put the case that one of the cluster's node crash. So in this case replications come on play. And, of course, he should not call the set of blocks "one" file. But it should say there's one file composed of "n" chunks that are replicated among nodes of cluster.
The way I understood it, = .
I suppose (if my understanding is correct of course) the confusion then comes from the fact that at the beginning of the video he said that a = .
He really should have started with just one row of lego bricks for each file, and just explained that a data node sends a copy of the block it just wrote to one its data peers as directed by the control node.
Now I like Legos :)
狡兔三窟说的就是这个意思
SUPER informative, but also i hear "Hadoop" and just think "Hadooken" just me? yeah okay hahaha
play in 1.25x
英语不太好，但是觉得很棒
Nican
its lego, not legos... but nice vid.
It started off pretty well but then it got very confusing. He didn't seem so sure of what he was saying. It didn't work for me. Thank you, anyway.
Why is he so sweaty?
Yeah, it was the lighting. We tried moving the lights around, but I didn't have any makeup on which mitigates the lights.
You should not answer this moron. u doing a good job for free
Looks like you are a moron!
He replied so kindly and genuinely, you are calling him moron.
Pretty sure he was referring to the initial commenter, not Jesse. Rude remark nevertheless.
Why do Americans call Lego bricks Legos? Lego is a company name and small building bricks is what they make, they don't make Legos! You can't implicitly type Lego bricks as Legos it sounds silly.
we do the same in Europe. Where are you from?
I've been thinking deeply about this and decided I'm the worst person to be dictating English.
Briton mate.
cafecapes Every nation has it's own way of speaking. There's a very elaborate section on the Stack Exchange website that goes into great depth on the difference in languages, pronunciation, and word usage. If this is a serious inquiry, I'd suggest you go there. It's a very informative place to learn anything you want - and if the question hasn't already been asked, you can ask it yourself.
One example is that there is a topic on how Americans pronounce the word solder as "sodder" while other countries pronounce it as "sold-er" and where this distinction came from. You might be surprised to know that the language variations have a lot to do with heritage dating way way back. Every language and dialect, regardless of what it is has become bastardized - and it's just a part of how languages evolve.
For what it's worth, I used to pronounce it as "sold-er" until I got tired of being corrected - and I have no indication of where I learned this from. I now colloquially pronounce it as "sodder" just because of tradition and geographical linguistics. Also, pronunciations and accent vary widely in the United States from coast to coast also.
Apologize for the long comment.
It's been quite a challenge to hear this video. Headphones are in but it doesn't seem to help.
audio volume is fine to me, must be an issue on your machine
Freddy yes, that’s what it was. Guess my Mac needed a reboot. Worked fine afterwards.