This is a video that should go to a reference library about Elasticsearch. Thank you for putting such a good, clear and methodical overview of ES. Just brilliant!
Holy sh*t! This must've been the best tutorial I've ever seen on UA-cam. High production, clear presentation and well thought through. +1 subscriber for sure!
I started going through a Udemy course on Elasticsearch and came to a section about Shards. When the chapter was complete I still didn't understand fully what a shard is. I searched on UA-cam and ended up on this video. You do a great job of explaining it so I (as a complete beginner) have a better understanding. Kudos to you for providing this video :)
I really wish this was the 1st elasticsearch video that I had watched, so as to save me so much time watching other video which could not teach me the same level of information that I need. Thank you very much, and I shall check out your courses.
This is the best explanation of elasticsearch I've ever seen. So many videos skip over the details, and it's been making it difficult to understand what elasticsearch is doing under the hood. I normally don't comment on videos, but this is too high quality not to. Please continue to put out content!
Done! New video just posted! There will be more. I have lots of ideas but I'm really trying to get this my Elasticsearch course finished, and work the day job.
Hi thanks for a video. For example we have: "unassigned_shards" : 40, When we run: GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.* { "index": "elastalert_past", "shard": 0, "primary": false } We reiceve next answer: "explanation" : "a copy of this shard is already allocated to this node [[elastalert_past][0], node[JaLzrdasdajQ], [P], s[STARTED], a[id=OmY9kwpHTlybJfSrWvdsadada6g]]" We have only one node and what we can do in this situation ? Also we have "number_of_replicas" : "0", "auto_expand_replicas" : "false", what we can do in this situation ? GET /.kibana/_settings { ".kibana_2" : { "settings" : { "index" : { "number_of_shards" : "1", "auto_expand_replicas" : "false", "provided_name" : ".kibana_2", "creation_date" : "1601664093", "number_of_replicas" : "0", "uuid" : "WKdIpzLFSP-ydObLw", "version" : { "created" : "7090299" } } } } }
Isn't the cluster the server (i.e. AWS EC2 instance) itself? To my mind, a node is not a server because you can create several nodes in the same machine. I was expecting to see MyCluster1 and MyCluster2 each having a single node, hence, high availability via cross-cluster communication. 11:11 node = server in his example 15:05 node = process
A node is an Elasticsearch process running on a host. You're right that you can run multiple nodes on the same host (even not containerised), but it's not recommended and it's widely accepted that you only run a single node on a host. If you did run two nodes on a single host, you could have either one or two clusters on that host. The node is configured with the cluster name it's expected to join, so you could configure each node with a different cluster name and have two clusters on that host! Cluster formation can get quite involved. There are configuration settings that need to be applied specifically at the formation stage. I can do a video on how that works at some stage.
Sorry for the very late response. I've had feedback from a couple of people using Firefox, who worked around it by using a different browser. I'm not sure if that's your issue but thought I'd mention it. Let me know if you're still having issues and I'll try responding quicker this time!
Is it fair to say you could build ES from dynamoDB? I'm trying to compare the two. I would love a video on the query language, does it have a mathematical basis like sql does to sets? It goes without saying, but I'm say it, thanks for making this clear, concise, focused high level content.
Hi there. I'm really pleased you enjoyed the video. I'm not sure you could build an equivalent of Elasticsearch using DynamoDB. There's a *lot* more to Elasticsearch than I talked about in this video! There's more content coming, including an introduction to the query language. The in-depth content will be in a training course instead of UA-cam, though. I've never considered if there's a mathematical basis to the query language. I doubt there is in terms of what Elasticsearch offers, but all Elasticsearch queries are converted to Lucene query language, which may be more thoroughly researched. Interesting question!
If elasticsearch distributes the data between the shards of an index such that each lucene store roughly holds the same number of documents, when you run a search query, elasticsearch, despite the inter-node communication, only knows which shards hold that index and not which particular shard will have that document? So it has to run the query against all the shards and merge results, it cannot just search the one shard that contains that document? It does not know beforehand based on how documents are distributed among shards.
Hands down the best explanation and introduction to Elasticsearch. Can't thank enough for making this video.
Thank you for your lovely comment. I'm so pleased you enjoyed the video!
@@GeorgeBridgemanData⁷77777777777777777777777777777777777777777777777777777777777777777777😅😮
Man! You are not from this planet! You deserve a thousand thumbs up.
This is a video that should go to a reference library about Elasticsearch. Thank you for putting such a good, clear and methodical overview of ES. Just brilliant!
Such a kind comment. Thank you so much!
I crawled thru many ES 101 videos that explain “index”and “shard”. This one did the best job.
Holy sh*t! This must've been the best tutorial I've ever seen on UA-cam. High production, clear presentation and well thought through. +1 subscriber for sure!
That's very kind of you to say. Thanks so much!
I started going through a Udemy course on Elasticsearch and came to a section about Shards. When the chapter was complete I still didn't understand fully what a shard is.
I searched on UA-cam and ended up on this video. You do a great job of explaining it so I (as a complete beginner) have a better understanding.
Kudos to you for providing this video :)
Agree. Same for me.
I really wish this was the 1st elasticsearch video that I had watched, so as to save me so much time watching other video which could not teach me the same level of information that I need. Thank you very much, and I shall check out your courses.
That's the reaction I was hoping for - a useful first video on Elasticsearch. Thank you so much for posting!
This is really one of the most useful videos that introduced ES to me.
Watched this 5 times, rewound several times and i understand it FULLY. Thanks so much for such clear explaining
You're welcome! I'm so pleased you understood everything. I hope it helps!
This video changed my life. No exaggeration.
I just started learning Elastic search and this is the best an clear information on Elastic search architecture. Thanks for sharing!
This is the best explanation of elasticsearch I've ever seen. So many videos skip over the details, and it's been making it difficult to understand what elasticsearch is doing under the hood. I normally don't comment on videos, but this is too high quality not to. Please continue to put out content!
What a lovely comment! Thank you so much. It means a lot!
What a clear , and progressively explained architecture. Thank you so much
Please post more videos…ur videos are easy to understand and quite informative….please carry on the good work
Done! New video just posted!
There will be more. I have lots of ideas but I'm really trying to get this my Elasticsearch course finished, and work the day job.
Awesome explanation, I love your narrative style, it really underlines the why and how of the current ecosystem!
Thank you. It feels like my brain is getting clearer.
Very clearly explained. Thanks
Thanks - the perfect introduction to Elasticsearch architecture.
Thank you!
Awesome explanation about Elasticsearch!!!
Thank you!
This is fantastic! Bricks till walls in a nutshell! Thanks much for this great presentation.
You're welcome! I'm really pleased you enjoyed it.
Very clear explanation!
thanks for the clear and awesome explanation to Elasticsearch and Lucene. really appreciate this useful content
I'm so pleased you appreciated it. It was great to make it!
Wonderful. Looking forward to more courses from you.
Thanks so much! Elasticsearch Engineer Essentials is in the works, and I'll be posting shorter content on here as well.
wonderful technical story.
Thank you!
Thank you very much sir for this information. Awesome people like you are what's good about this world!!! Much appreciated!!!
You're very welcome! Thanks for your kind words.
Thank You for making this fantastic video !
You're more than welcome. I'm glad you enjoyed it!
Many thanks for your work! It's awesome video!
Great content, thanks for sharing!
My pleasure!
Absolutely fantastic!
Thank you!
Thanks so much for this great presentation.
You're welcome. I'm glad you enjoyed it!
Its two year late but the lesson is extremely value
where was thsi video 1 month ago. you should be paid by these software companies... bless your heart
this is great content. thank you.
I think that shard in ES has the same concept with partition in Kafka when they have all partition replicas in different nodes
Premium content, thank you!
Hi thanks for a video.
For example we have: "unassigned_shards" : 40,
When we run:
GET _cluster/allocation/explain?filter_path=index,node_allocation_decisions.node_name,node_allocation_decisions.deciders.*
{
"index": "elastalert_past",
"shard": 0,
"primary": false
}
We reiceve next answer:
"explanation" : "a copy of this shard is already allocated to this node [[elastalert_past][0], node[JaLzrdasdajQ], [P], s[STARTED], a[id=OmY9kwpHTlybJfSrWvdsadada6g]]"
We have only one node and what we can do in this situation ?
Also we have "number_of_replicas" : "0", "auto_expand_replicas" : "false", what we can do in this situation ?
GET /.kibana/_settings
{
".kibana_2" : {
"settings" : {
"index" : {
"number_of_shards" : "1",
"auto_expand_replicas" : "false",
"provided_name" : ".kibana_2",
"creation_date" : "1601664093",
"number_of_replicas" : "0",
"uuid" : "WKdIpzLFSP-ydObLw",
"version" : {
"created" : "7090299"
}
}
}
}
}
Wonderful. Thank you so much.
You're very welcome!
What if i have 2 primary shards and 1 replica shard. Will that replica store all docs from both primary shard?
This is so good.
Thanks very much!
Isn't the cluster the server (i.e. AWS EC2 instance) itself? To my mind, a node is not a server because you can create several nodes in the same machine. I was expecting to see MyCluster1 and MyCluster2 each having a single node, hence, high availability via cross-cluster communication.
11:11 node = server in his example
15:05 node = process
A node is an Elasticsearch process running on a host. You're right that you can run multiple nodes on the same host (even not containerised), but it's not recommended and it's widely accepted that you only run a single node on a host.
If you did run two nodes on a single host, you could have either one or two clusters on that host. The node is configured with the cluster name it's expected to join, so you could configure each node with a different cluster name and have two clusters on that host!
Cluster formation can get quite involved. There are configuration settings that need to be applied specifically at the formation stage. I can do a video on how that works at some stage.
I am not able to enroll for your course, tried with 2 different emails, please have a look into this.
Sorry for the very late response. I've had feedback from a couple of people using Firefox, who worked around it by using a different browser. I'm not sure if that's your issue but thought I'd mention it. Let me know if you're still having issues and I'll try responding quicker this time!
Thank you so much :)
Is it fair to say you could build ES from dynamoDB? I'm trying to compare the two.
I would love a video on the query language, does it have a mathematical basis like sql does to sets?
It goes without saying, but I'm say it, thanks for making this clear, concise, focused high level content.
Hi there. I'm really pleased you enjoyed the video.
I'm not sure you could build an equivalent of Elasticsearch using DynamoDB. There's a *lot* more to Elasticsearch than I talked about in this video!
There's more content coming, including an introduction to the query language. The in-depth content will be in a training course instead of UA-cam, though. I've never considered if there's a mathematical basis to the query language. I doubt there is in terms of what Elasticsearch offers, but all Elasticsearch queries are converted to Lucene query language, which may be more thoroughly researched. Interesting question!
If elasticsearch distributes the data between the shards of an index such that each lucene store roughly holds the same number of documents, when you run a search query, elasticsearch, despite the inter-node communication, only knows which shards hold that index and not which particular shard will have that document? So it has to run the query against all the shards and merge results, it cannot just search the one shard that contains that document? It does not know beforehand based on how documents are distributed among shards.
No one talks of indexing live updates from Relational Databases
1000th like 😅