Written Summary Here: elasticsearch.evermight.com/setup-elasticsearch-cluster Tutorial to Setup Cluster with Docker: ua-cam.com/video/5Bi8TMAJzfY/v-deo.html UPDATES: since the release of this video , elasticsearch has released upgrades with some breaking changes. See the description of our video for any updates/suggestions/work-arounds that we discover. And if you learn of anything, please share in the comments below to help everyone else out. Thank you in advance!
Thank you very much , your video help me add elastic node to an existing cluster (I am using ES 8..11.3 and it worked very well) , thanks a ton , the ES people should really update their docs regarding using elasticsearch-reconfigure-node --enrollment-token instead of elasticsearch --enrollment-token .
Hi John ! First off, thank you so much for this very precious and useful walkthrough. I have been carefully watching it every day for more than a week now, to reproduce every step and understand everything thoroughly. You give a lot of insight and that is so valuable. I do have my let's encrypt certs, but for the sake of completeness, I would like to use the self signed certificates to connect from Kibana to the cluster as well. I want to be able to do both dev and production setups. And honestly, I am drowning here, as there does not seem to be a unique way to tackle things.there are so many moving pieces. My cluster is working fine with default self signed certs, and I do not whish to disturb it, I just want to plug Kibana to it using self signed certs. I read many documenation pages but as I said, the all assume different cluster setups and mess around with what is already configured properly in your video. If you have any ressources on a step by step basis on how to achieve that, that would be great. Once again, thanks for posting this, it helps a lot !
Thanks for your message! Let me know if this answer is what you're looking for: If you look in my video description , I posted a NOTE 2, which shows how you can have elasticsearch use self signed certificates for the http protocol and Kibana do the same. Is that what you're looking for?
Great job!. This is a perfect tutorial. Split-brain problem is something that is missing in most of the other tutorials. Is it possible for a cassandra and redis tutorial in cluster mode?
Great video, detailed and helpful! I have one question. You've mentioned that joining additional node with enrollment-token won't be possible if the configuration of certificates is different than auto-generated. How can we add another node to the cluster with publicly signed certificates already set? Thank you in advance!
Yes, you should try temporarily swapping back to the self signed ones on the machine creating the nerollment tokens. Enrol the new nodes. Then swap back over to your publicly signed certs
Hello I am a fresher that joined a company and I was given the role of managing the elastic stack because I am a fresher your video helped me a lot and I am very thankful for your great videos but can you make a video on how to make an elastic search cluster in kubernetes
Yes this is on my task list, but just a lot of deadlines these past few months, so might need until end of year to get back to making these elasticsearch guides
Thank you so much your video help me to know clearly how to setup Elastic cluster for production env , and next can you share another tutorial video How to connect Spirng-boot project with Elastic cluster that we already setup and Log spring-boot project on Kibana, Thank you
Thanks for the tutorial. Would you mind creating another video using public certificates ( instead of self-signed )? Please feel free to point me to the documentation if there is one.
Sorry for the long delay i replying to your request about using public certs for the transport protocol. I just released it as a members video here: ua-cam.com/video/X9R7ocI2mWs/v-deo.html
Hi John Before say anything your video has been one of the best best elasticsearch cluster videos i have seen. so i am creating a gui interface for the implementation of elasticsearch cluster on single and multinode servers. for test purposes, how can i eliminate the whole ssl certificates, because it is just a project , and can i add the elasticsearch on my virtual machine to the cluster
Hey there. I haven't tried to disable SSL for the API port 9200 or the TCP port 9300. You can easily disable SSL for Kibana through the kibana.yml file. I recommend you continue to use the self signed certificates or auto-generated certificates just to avoid headaches of trying to configure things in ways that elasticsearch doesn't document too much on. Other then that, I don't have enough experience to help you on that front!
Thank you for your tutorials, they are very good. Congratulations. I would like a recommendation for a video or tutorial on monitoring postgres logs (.log or .csv) using elastic-agent and fleetserver (versions 12 to 15). I tried to implement it, but it doesn't seem to work. I'm confused about whether I need to install and configure filebeat or just use integrations via elastic-agent.
Thanks I recommend for now that you try metricbeat or filebeat (or both together). Those should have postgres modules ready for you. I'll add to my to-do list on how to use elastic agent and fleet server for postgres ingestion, but it might be a while before I get to it!
Hello @evermightsystems I have configured Elasticsearch as per this tutorial. also configured Kibana. But kibana now wirking. kibana sayes "Kibana server is not ready yet." need help plz
You should look in the /var/logs/kibana/kibana.log file to see what the specific issue is. If you run into trouble, use the contact us form in our website to schedule a meeting with us and we'll try to fix
Sorry for delay but I jsut got around to making this video for ELK with Docker Compose: ua-cam.com/video/FYr7HVLlvcs/v-deo.html I think from here, it should be pretty easy for you to start a swarm!
Hi dear. Thank you for video. I have a question. I installed 2 nodes. and first of them the master. Then I stop the first node , the cluster also down, and not passed to second node as cluster. How can I solve this problem?
Thanks for your message. From the elasticsearch documentation: "To be sure that the cluster remains available you must not stop half or more of the nodes in the voting configuration at the same time." If you really want to have the final remaining node running, you could try to set `minimum_master_nodes` equal to one in your elasticsearch yml file. But I haven't tried this before so don't know if it will work .
Thanks you for this video, it is very useful. But I have a problem with cluster connection to kibana with self signed certs. Do you have more info about it?
@dv thanks for your message. I just got things to work and added a NOTE 2>option 1 to my video description. But I'll mention here for your convenience. Try adding this line to your kibana.yml elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ] Where the http_ca.crt is a copy of /etc/elasticsearch/certs/http_ca.crt from node1
Hello, I first formed one node, after a few days, I am trying to add more nodes into the existing cluster, but I am not able to do so, can i know the process how to do.
Apologies for delayed replies. Got so busy I wasn't able to check these messages. You can email us through our website if you still have problems. As far as I am aware, you can keep adding nodes at anytime. If you email us your error logs, we can take a quick look
I don't know the answer to this question! Normally you would have many nodes in a cluster so that when some our unavailable, you would have other master nodes take over. The only time when all nodes are unavailable is if you intentionally shutdwn the whole cluster, and in such an event, you would carefully take note of which were the last master nodes during your shutdown process.
Thanks for your message! Yes alot of th se videos are still work-in-progrees. Sorry for delay, just a lot of deadlines and other priorities keep coming up!
thanks for the sharing , 1 more question ,please. I generated the elastic/kibana service account token but this token only worked on node where it been created , could you help share some idea , thanks in advance
On the node that created the token, I think there is a /etc/elasticsearch/service_token file or something similar that stores the encrypted token. You can copy this file to any other elastic node you want Kibana to connect to.
@@evermightsystems thanks for quick reply , yes , I found /etc/elasticsearch/service_token and I tried copy and fixed the issue, but what I mean is it's cluster , should it be this token work for all elasticsearch nodes , just worried missing setting on elasticsearch
thank you so much for valuable configurations , l have been asked to configure public signed cert. if you can make a video about that, it would be great!
I set up a single node cluster in this video using entirely public certs: ua-cam.com/video/HOmq3STxAKg/v-deo.html If you go to 6 minute or so, you see how we install the public certs. For a cluster u you basically use the same lines. But be aware that if you use public certs, any node by any one else in the world can join your cluster. Elastic effectively uses self signed certificates as access tokens. The root of public certs are known by everyone.
Hello John, your video help me a lots. Can you make another one like this using Docker Compose on each node or you have note about it, please share with me. Thanks!
I don't have enough experience to speak to this. I've used CloudFlare for ssl termination in load balancing. My first naive attempt would be to do the same for the elastic cluster. Meaning I'd set up load balancing with CloudFlare then forward traffic to my elastic cluster. Then my elastic cluster is set up using the instructions in the video. That would be my initial thought, but I got you get more experienced answers from someone else!
@@evermightsystems Thank you for looking into my request. I am not sure if we should use our own SSL or self-signed SSL by Elasticsearch. I am working on a SOC setup and I wanted to make sure all the traffic is encrypted. But was not able to find much support online. But your videos have solved a few of my issues. Appreciate your efforts in making these videos.
@@mahesh_9567 ok yup. Self signed certificates will do encryption. The only difference between public certs and self signed ones is that most clients won't automatically trust the certificate chain of a self signed certificate, which in the scenario of an elastic cluster is arguably more secure than a public certificate. I forget if cloudflare certs is just a sectigo certificate or some other well known signing authority... because if it is, using cloud flare certs will actually make your set up less secure, because the root cert of a publicly signed certificate is known by everyone
After doing all the setup, i am getting error "[INFO ][node] Kibana process configured with roles: [background_tasks, ui]" Not sure what is the issue, can you help on this
@@mahesh_9567 ok i recommend you post the full details of your issue to the elasticsearch slack channel or the discuss.elastic.co website so someone can followup. Or if you need more direct support and want to hire myself , just contact us through our website on our contact page. Hopefully that helps!
Hi SonaliDive, i also got your email, so i will continue my conversations with you there. IN the meantime, here is a video of how I use Let's Encrypt to make the publicly signed certs: ua-cam.com/video/Kk9kuf6D8so/v-deo.html
For the public cert, we used: ua-cam.com/video/Kk9kuf6D8so/v-deo.html . THe rest of the time, we used the self signed certs that were auto-generated by elastic
I will like to add at 12:35 you should run ES in an Internal network so that it shouldn't be publicly accessible unless you need to collect data form the internet i think using a fleet will be better. Still Learning Elasticsearch as I am running it on my Tailscale network. If anyone is interested i can tell them how to set it up. Also if i am wrong about the fleet server please let me know thanks.
![[Elasticsearch] Как собрать кластер Elasticsearch. Какие есть роли у нод Elasticsearch.](http://i.ytimg.com/vi/-e0M9X6uD_4/mqdefault.jpg)
Written Summary Here: elasticsearch.evermight.com/setup-elasticsearch-cluster
Tutorial to Setup Cluster with Docker: ua-cam.com/video/5Bi8TMAJzfY/v-deo.html
UPDATES: since the release of this video , elasticsearch has released upgrades with some breaking changes. See the description of our video for any updates/suggestions/work-arounds that we discover. And if you learn of anything, please share in the comments below to help everyone else out. Thank you in advance!
This is a blueprint of what a Tutorial should be.
I love this channel. hope evermight system can be a benefit company
Appreciate your hard work. It was detailed on every step, which is rare to find. Thank you.
Thank you very much , your video help me add elastic node to an existing cluster (I am using ES 8..11.3 and it worked very well) , thanks a ton , the ES people should really update their docs regarding using elasticsearch-reconfigure-node --enrollment-token instead of elasticsearch --enrollment-token .
Hi John ! First off, thank you so much for this very precious and useful walkthrough. I have been carefully watching it every day for more than a week now, to reproduce every step and understand everything thoroughly. You give a lot of insight and that is so valuable. I do have my let's encrypt certs, but for the sake of completeness, I would like to use the self signed certificates to connect from Kibana to the cluster as well. I want to be able to do both dev and production setups. And honestly, I am drowning here, as there does not seem to be a unique way to tackle things.there are so many moving pieces. My cluster is working fine with default self signed certs, and I do not whish to disturb it, I just want to plug Kibana to it using self signed certs. I read many documenation pages but as I said, the all assume different cluster setups and mess around with what is already configured properly in your video. If you have any ressources on a step by step basis on how to achieve that, that would be great. Once again, thanks for posting this, it helps a lot !
Thanks for your message! Let me know if this answer is what you're looking for:
If you look in my video description , I posted a NOTE 2, which shows how you can have elasticsearch use self signed certificates for the http protocol and Kibana do the same. Is that what you're looking for?
@@evermightsystems I tried to answ3r but it kept getting del3ted for some r3ason so I post3d as a com
Great job!. This is a perfect tutorial. Split-brain problem is something that is missing in most of the other tutorials. Is it possible for a cassandra and redis tutorial in cluster mode?
Thank you for comment! I had plans for redis but it could be quite a while before I get to it. But thanks for the support and suggestions!
This is really good stuff. I really liked the way you explained everything.
Great video, detailed and helpful! I have one question. You've mentioned that joining additional node with enrollment-token won't be possible if the configuration of certificates is different than auto-generated. How can we add another node to the cluster with publicly signed certificates already set? Thank you in advance!
Yes, you should try temporarily swapping back to the self signed ones on the machine creating the nerollment tokens. Enrol the new nodes. Then swap back over to your publicly signed certs
Hello I am a fresher that joined a company and I was given the role of managing the elastic stack because I am a fresher your video helped me a lot and I am very thankful for your great videos but can you make a video on how to make an elastic search cluster in kubernetes
Yes this is on my task list, but just a lot of deadlines these past few months, so might need until end of year to get back to making these elasticsearch guides
Thank you so much your video help me to know clearly how to setup Elastic cluster for production env , and next can you share another tutorial video How to connect Spirng-boot project with Elastic cluster that we already setup and Log spring-boot project on Kibana, Thank you
thanks for your message! This wil take time for us to research, but we will add it to our task list.
I really liked the way you explained and hope to see that video soon . Thank you sir@@evermightsystems
Thanks for the tutorial. Would you mind creating another video using public certificates ( instead of self-signed )? Please feel free to point me to the documentation if there is one.
Sorry for the long delay i replying to your request about using public certs for the transport protocol. I just released it as a members video here: ua-cam.com/video/X9R7ocI2mWs/v-deo.html
Hi John
Before say anything your video has been one of the best best elasticsearch cluster videos i have seen. so i am creating a gui interface for the implementation of elasticsearch cluster on single and multinode servers. for test purposes, how can i eliminate the whole ssl certificates, because it is just a project , and can i add the elasticsearch on my virtual machine to the cluster
Hey there. I haven't tried to disable SSL for the API port 9200 or the TCP port 9300. You can easily disable SSL for Kibana through the kibana.yml file.
I recommend you continue to use the self signed certificates or auto-generated certificates just to avoid headaches of trying to configure things in ways that elasticsearch doesn't document too much on.
Other then that, I don't have enough experience to help you on that front!
Thank you for your tutorials, they are very good. Congratulations.
I would like a recommendation for a video or tutorial on monitoring postgres logs (.log or .csv) using elastic-agent and fleetserver (versions 12 to 15).
I tried to implement it, but it doesn't seem to work. I'm confused about whether I need to install and configure filebeat or just use integrations via elastic-agent.
Thanks I recommend for now that you try metricbeat or filebeat (or both together). Those should have postgres modules ready for you.
I'll add to my to-do list on how to use elastic agent and fleet server for postgres ingestion, but it might be a while before I get to it!
Hello @evermightsystems
I have configured Elasticsearch as per this tutorial. also configured Kibana. But kibana now wirking. kibana sayes "Kibana server is not ready yet." need help plz
You should look in the /var/logs/kibana/kibana.log file to see what the specific issue is. If you run into trouble, use the contact us form in our website to schedule a meeting with us and we'll try to fix
Exactly what I was looking for. Thank you :)
Have you had an opportunity to do a tutorial on how to add an Elasticsearch server to a cluster using publicly signed certs?
Haven't had time yet! Will let you know once we do
Hi Jhon Thank you for your tutorials, very good and detail, can you make another tutorial for installing ELK like this using Docker Swarm Please 😇😇
Sorry for delay but I jsut got around to making this video for ELK with Docker Compose: ua-cam.com/video/FYr7HVLlvcs/v-deo.html
I think from here, it should be pretty easy for you to start a swarm!
Hi dear. Thank you for video. I have a question. I installed 2 nodes. and first of them the master. Then I stop the first node , the cluster also down, and not passed to second node as cluster. How can I solve this problem?
Thanks for your message. From the elasticsearch documentation: "To be sure that the cluster remains available you must not stop half or more of the nodes in the voting configuration at the same time." If you really want to have the final remaining node running, you could try to set `minimum_master_nodes` equal to one in your elasticsearch yml file. But I haven't tried this before so don't know if it will work .
Thanks you for this video, it is very useful. But I have a problem with cluster connection to kibana with self signed certs. Do you have more info about it?
@dv thanks for your message. I just got things to work and added a NOTE 2>option 1 to my video description. But I'll mention here for your convenience. Try adding this line to your kibana.yml
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ]
Where the http_ca.crt is a copy of /etc/elasticsearch/certs/http_ca.crt from node1
thanks for your amazing video
Fantastic video; When will you release the 10:22 SSL trust chain video
Thanks for your message. I released the video here: ua-cam.com/video/zPkSiojo7rA/v-deo.html
And if you haven't seen this already, it is my fix for the cert authority issue I encountered in video
ua-cam.com/video/reupgV-A3NA/v-deo.html
Hello, I first formed one node, after a few days, I am trying to add more nodes into the existing cluster, but I am not able to do so, can i know the process how to do.
Apologies for delayed replies. Got so busy I wasn't able to check these messages. You can email us through our website if you still have problems. As far as I am aware, you can keep adding nodes at anytime. If you email us your error logs, we can take a quick look
how about we know that node2 or node1 or another node is the last master node ? if we notice after all node down or cluster down ?
I don't know the answer to this question! Normally you would have many nodes in a cluster so that when some our unavailable, you would have other master nodes take over. The only time when all nodes are unavailable is if you intentionally shutdwn the whole cluster, and in such an event, you would carefully take note of which were the last master nodes during your shutdown process.
Thank you for this video, it is very useful.
halo sir, great video ? hmm are the video about SSL Trust chain have done sir ?
and the others video that you mention how to manually reconfigure another nodes to join cluster if we use publicly certificate ?
Thanks for your message! Yes alot of th se videos are still work-in-progrees. Sorry for delay, just a lot of deadlines and other priorities keep coming up!
@@evermightsystems 🙏🙏
thanks for the sharing , 1 more question ,please. I generated the elastic/kibana service account token but this token only worked on node where it been created , could you help share some idea , thanks in advance
On the node that created the token, I think there is a /etc/elasticsearch/service_token file or something similar that stores the encrypted token. You can copy this file to any other elastic node you want Kibana to connect to.
@@evermightsystems thanks for quick reply , yes , I found /etc/elasticsearch/service_token and I tried copy and fixed the issue, but what I mean is it's cluster , should it be this token work for all elasticsearch nodes , just worried missing setting on elasticsearch
It helped me a lot thank you so much!
thank you so much for valuable configurations , l have been asked to configure public signed cert. if you can make a video about that, it would be great!
I set up a single node cluster in this video using entirely public certs: ua-cam.com/video/HOmq3STxAKg/v-deo.html
If you go to 6 minute or so, you see how we install the public certs. For a cluster u you basically use the same lines.
But be aware that if you use public certs, any node by any one else in the world can join your cluster.
Elastic effectively uses self signed certificates as access tokens. The root of public certs are known by everyone.
Hello John, your video help me a lots. Can you make another one like this using Docker Compose on each node or you have note about it, please share with me. Thanks!
Apologies for late reply! Yes this video is on our task list!
@@evermightsystemsI am excited. How is the docker compose version coming along?
Hello , I want to install the Cloudflare-provided SSL in the elastic search do you have any recommendation
I don't have enough experience to speak to this. I've used CloudFlare for ssl termination in load balancing. My first naive attempt would be to do the same for the elastic cluster. Meaning I'd set up load balancing with CloudFlare then forward traffic to my elastic cluster. Then my elastic cluster is set up using the instructions in the video.
That would be my initial thought, but I got you get more experienced answers from someone else!
@@evermightsystems Thank you for looking into my request. I am not sure if we should use our own SSL or self-signed SSL by Elasticsearch. I am working on a SOC setup and I wanted to make sure all the traffic is encrypted. But was not able to find much support online. But your videos have solved a few of my issues. Appreciate your efforts in making these videos.
@@mahesh_9567 ok yup. Self signed certificates will do encryption. The only difference between public certs and self signed ones is that most clients won't automatically trust the certificate chain of a self signed certificate, which in the scenario of an elastic cluster is arguably more secure than a public certificate. I forget if cloudflare certs is just a sectigo certificate or some other well known signing authority... because if it is, using cloud flare certs will actually make your set up less secure, because the root cert of a publicly signed certificate is known by everyone
Excellent Explaination
After doing all the setup, i am getting error "[INFO ][node] Kibana process configured with roles: [background_tasks, ui]"
Not sure what is the issue, can you help on this
The [INFO] is just a notice and doesn't mean an error. Actual errors have the [ERROR] flag. If you can share any errors, then I can comment
@@evermightsystems i am not getting errors. Page is not loading
@@mahesh_9567 maybe it's your firewall that's preventing the page from loading?
@@evermightsystems UFW is disabled
@@mahesh_9567 ok i recommend you post the full details of your issue to the elasticsearch slack channel or the discuss.elastic.co website so someone can followup. Or if you need more direct support and want to hire myself , just contact us through our website on our contact page. Hopefully that helps!
Great tutorial👏👏
i want to know how to creare public self sign certificate
Hi SonaliDive, i also got your email, so i will continue my conversations with you there. IN the meantime, here is a video of how I use Let's Encrypt to make the publicly signed certs: ua-cam.com/video/Kk9kuf6D8so/v-deo.html
thanks for the video
how you have created certificate
For the public cert, we used: ua-cam.com/video/Kk9kuf6D8so/v-deo.html . THe rest of the time, we used the self signed certs that were auto-generated by elastic
I will like to add at 12:35 you should run ES in an Internal network so that it shouldn't be publicly accessible unless you need to collect data form the internet i think using a fleet will be better. Still Learning Elasticsearch as I am running it on my Tailscale network. If anyone is interested i can tell them how to set it up. Also if i am wrong about the fleet server please let me know thanks.
you have ansible for this here ?
Not at the moment, sorry. But if you make one, feel free to share with the community!
Thanks😀
Please share Letsencrypt certificate creation vedio
ua-cam.com/video/pZldq3Bz1oo/v-deo.html in this video we needed to set up a fleet server with let's encypt SSL to do APM
ua-cam.com/video/Kk9kuf6D8so/v-deo.html and this video is how we make our let's encypt ssl
Thanks