PostgreSQL Clustering the Hard Way...

Поділитися
Вставка
  • Опубліковано 9 січ 2025

КОМЕНТАРІ •

  • @mirceanton
    @mirceanton Місяць тому +40

    Missed opportunity to call this video "installing Postgres the hard way" as a throwback to Kubernetes install tutorials

    • @TechnoTim
      @TechnoTim  Місяць тому +7

      That was 100% going to be the title but I didn't want to scare anyone!

    • @TechnoTim
      @TechnoTim  Місяць тому +13

      OK, I decided to rename it. Thanks for the nudge!

    • @akhilleusuggo
      @akhilleusuggo Місяць тому +7

      @@TechnoTim I prefer the hard way, as it teaches a lot. For a tutorial is perfect

    • @TheKnightArch
      @TheKnightArch Місяць тому

      @@TechnoTim I found the video because I was searching for a production ready environment. UA-cam notified nothing about the new video release, as always.

    • @BrantScalan
      @BrantScalan 21 день тому

      @@TechnoTim Whats the easy way to do this??

  • @Jims-Garage
    @Jims-Garage Місяць тому +26

    Awesome video, I'll definitely be checking this out in my lab.

  • @organon69
    @organon69 Місяць тому +10

    I'd give a few props to the Zalando team who created Patroni way back now. As you cited, many ways to do the cert thing, but to slicken up the process, maybe bake into your docs a note or two about creating a StepCA instance (and enable ACME on it), and then just use 'certbot' on the HAproxy, etcd, and Postgres server nodes to allow the pull/refresh of needed certs automatically. The StepCA client auto-adds the rootCA of the StepCA server you're talking to into the local trusted cert store, and that prevent all the squawking about untrusted CA this/that/the-other. Great vid.

    • @menruletheworld
      @menruletheworld Місяць тому

      That’s a lot better solution than manually creating and managing certificate, but why run certbot on every node? Store your certificates in Hashicorp vault and use consul-template to deploy them, simpler, less moving parts and more elegant solution.

    • @organon69
      @organon69 Місяць тому

      @@menruletheworld - lots of good choices just as you say. Whether Vault or OpenBao and their CA backend are all viable. I'll +1 Tim's work talking through the OpenSSL's .cnf config options, however, because - though a little lumpy - it's always useful knowledge.

  • @marcomorozzi4501
    @marcomorozzi4501 Місяць тому +3

    Finally someone wich doesn't do only hello world tutorials. If I had this content when I was starting I would have skipped many headaches. Please provide more content like this 🙏

  • @RicardoWagner
    @RicardoWagner Місяць тому +2

    I went backwards. Started with Zalando in Kube. Way too old and is good for k8s only. then gave Crunchy a try.; not good for production.. then thought about streight Patroni, found hudreds of people doing their own way. I will give Tim`s a try... seems very complete and straight forward.Thanks buddy.

  • @Phaxmohdem
    @Phaxmohdem Місяць тому +4

    Amazing and very timely for me! Thank you! Heads up: The Video notes URL in the description has a typo and returns 404 currently. Had to go directly to your root address and get to it from the homepage.

    • @TechnoTim
      @TechnoTim  Місяць тому +5

      Thanks you! I changed the URL and you beat me to it. Just refresh and the link should be updated!

  • @jeffherdzina6716
    @jeffherdzina6716 Місяць тому

    What a great video ! And what great timing. Because, guess what I was going to be tasked with at work. Thank you!!!!!
    I would like to give you a huge thanks, for just how much hard work you have put into all of this, and the coding done on the website.

  • @WarLordOmar
    @WarLordOmar 28 днів тому

    Damn man thats amazing, and extremely useful, i love how you always explain indl details the important things and choose the most important topics, looking forward for the automation tutorial and the kubernetes one❤

  • @cxl520
    @cxl520 Місяць тому

    Fantastic video! I love to see how it run under the hook, not just a automate process.

  • @jasonperry6046
    @jasonperry6046 Місяць тому

    I am super excited you are sharing this with us.
    I am also super excited for your next hardware update, not that you have been living with your co-lo for a while.

  • @medazz2847
    @medazz2847 Місяць тому +2

    Great video ! I hope you'll have the time one day and make a video demonstrating how to include Traefik Proxy in our Docker Compose file and use it in our architecture with PostgreSQL and pgAdmin service containers. I really learned too much from your course.

  • @joggienl
    @joggienl Місяць тому

    This is an awesome rundown on how to install it "the hard way". This is the way. 🚀

  • @linusthorsell437
    @linusthorsell437 Місяць тому

    I did this exact setup a while back. Was super tough. Good luck to everyone, probably going to be easier for you all with this video :)

  • @carstenr.1682
    @carstenr.1682 Місяць тому

    Awesome!

  • @alexanderg9106
    @alexanderg9106 Місяць тому

    Great Video! Some stuff to add for keepalived. It usaly use multicast, the traffic will be send from the interface in the config. With vrrp3 you can swithc to unicast and then need to define a unicast source and the destionations. Also in the vrrp VIP you could add more than one ip and after the ip add the interface the ip is added to. The unicast checks are done by the interface defined in the top. This is very important for overlay networks since multicast can generate there much traffic.

  • @da1985B
    @da1985B Місяць тому

    thank you, just at the right time while my company is about to transition from a patroni-managed pg cluster in k8s towards managed postgres hosting - as i need to setup a test-system in-house for our development team ❣

  • @otisrancko
    @otisrancko Місяць тому

    Can't wait for the automated version of this with IaC...
    this is awesome content!!!

  • @АртёмЕфимов-о6н
    @АртёмЕфимов-о6н 26 днів тому

    Thank you for the video. It's amazing!

  • @TomSwartz
    @TomSwartz Місяць тому +4

    Crunchy Data has a pretty good Kubernetes Operator that makes deployment easy

  • @DudeItsDallyBoy
    @DudeItsDallyBoy 28 днів тому

    Hey Tim. Followed along and got it working. Any chance you could make a followup tutorial on how to use this with my existing docker containers. I can't seem to figure out how to get the SSL certs to work on the the containers. I know that the server.crt and key have to be provided to the containers but i can't seem to figure out how to get them trusted.

  • @marianoloo
    @marianoloo Місяць тому +1

    Great video! I would love to see similar content for MariaDB severs.

  • @Standbackforscience
    @Standbackforscience Місяць тому

    Learned a lot about HA in general with this tutorial. Please please _please_ do one for ElasticSearch, because for the life of me I cannot figure out how to self-host an ES cluster for longer then 2-3 days before it grinds to a screeching halt, despite claiming everything is healthy.

  • @eattofuholmes
    @eattofuholmes 25 днів тому

    Absolutely amazing content! This is exactly what I was planning on doing at my new job and I think you just saved me about a million headaches! I am curious though, how would you implement a flask api in this setup to also be high availabilty?

  • @vvshvvv
    @vvshvvv Місяць тому

    It is better to have separate LBs for primary and secondary nodes. Usually applications would have separate connection pools for master and replicas, so read only transactions would go to replicas while read/write will go master. Great guide nevertheless!

  • @MrEtoel
    @MrEtoel Місяць тому

    You need a coffee after this.

    • @TechnoTim
      @TechnoTim  Місяць тому

      @@MrEtoel thank you!!!

  • @RyanOHaganWA
    @RyanOHaganWA Місяць тому

    Awesome Tutorial!

  • @FireWyvern870
    @FireWyvern870 Місяць тому +1

    1. How does it handle transactions?
    2. How do i remove or add node into the cluster?
    3. How does it handle wal archiving? Is there any way I can restore point in time?

  • @DDD-d1g
    @DDD-d1g 29 днів тому

    Very impressing! I am sure that for this almost an hour of content, you had to work for many more hours to polish the content and give it to us clean and clear. Thanks for that!
    Now for a question: if the replicas are able to get READ queries, how do you configure the HAProxy to balance the reads all over the cluster? And how do you control (in the Postgres replication process) if the data is eventually consistent (faster writes) or immediately consistent (slower writes, waiting for an acknowledge from other replicas) across all nodes?

  • @ClayBellBrews
    @ClayBellBrews Місяць тому

    Man i got all excited when you said stacks, then poof, FU it’s vm’s. :) really nice breakdown. I look forward to watching your brain melt when you do this under kube.

    • @TechnoTim
      @TechnoTim  Місяць тому

      @@ClayBellBrews already did it and switched back 😅

    • @ClayBellBrews
      @ClayBellBrews Місяць тому

      @@TechnoTim haha; you still need to add pgbouncer and pgpools as well. I think percona did a video presentation on the kube stuff. I’ve not watched it yet but should be current. Great presentation btw, pretty complete but way faster than most people get it done.

  • @TheKnightArch
    @TheKnightArch Місяць тому

    My eternal dilema leveled up to another level: docker container or hosted at the SO as showed by you? I would like to know how you set backups for this kind of environment of the video. Thank you very much, Tim!

  • @johnholland2575
    @johnholland2575 Місяць тому

    Fantastic job, Tim. Earned you a sub and 👍

  • @mra2202
    @mra2202 Місяць тому

    Really great tutorial. Would really love a follow-up video for Kubernetes.

  • @mrgingerpapa
    @mrgingerpapa Місяць тому

    Great video!

  • @fbifido2
    @fbifido2 Місяць тому

    @40:10 - could you put the root-ca cert on the proxy VM to verify the PostgreSQL certificates?

    • @TechnoTim
      @TechnoTim  Місяць тому

      Yes, you absolutley could!

  • @arsalan1377
    @arsalan1377 Місяць тому

    Awesome! Please make content on MySQL InnoDB cluster too

  • @BrantScalan
    @BrantScalan 21 день тому

    This was a great video. But it got me thinking about two things. The first is could you have 2 of the Postgres nodes being receiving query requests from the HAProxy while the 3rd is just a replication node? The first two would be the DR/HA nodes while the 3rd was the "backup". Was thinking about a cloud provider and putting two nodes in one region and the 3rd in another region in case of outage? Any thoughts? Then the second thought was about just having two HAProxy machines? Do we need the 3 and maybe just build a check_script and let Keepalived verify it can reach the VIP through one of the HAProxies? Just wondering if you could save a VM on the front-end and still have enough availability and reachability? Would love to hear your thoughts and maybe a follow up video if my ideas make sense??? Great content - Thank you

  • @Wheels35
    @Wheels35 Місяць тому +7

    I hate that this is posted right now, right as I am actively planning an application that will be relying on Postgres...absolutely worst timing!! :D

  • @blakehamm9014
    @blakehamm9014 25 днів тому

    Would love a video on the zalando k8s operator!

  • @tekjoey
    @tekjoey 29 днів тому

    @40:57 For anyone who sees this in the future...
    When I applied the configuration and went to view the syslog, the syslog file wasn't found. I then ran 'sudo systemctl status haproxy' and got the following message: "Server postgres_backend/postgresql-02 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable"" I got this for both postgress-02 and postgres-03.
    This is normal. For some reason, the patroni instance on the non-master nodes will report as Unavailable, even though you can curl the patroni endpoint and connect to the postgresql port. If you continue following Tim's instructions, keepalived will work just fine and the cluster will be created. If you then shut down one of the db nodes and 'sudo systemctl status haproxy', you'll see that the new master node magically became available.
    I'm running brand new Debian 12 VMs on Proxmox.

  • @floriantthebault521
    @floriantthebault521 Місяць тому

    Well, I was wondering about "enterprisify" my kubernetes cluster with a distributed database. That's... coming at the right time!

  • @phillipmelvin4756
    @phillipmelvin4756 Місяць тому

    ClusterSSH would a real asset for videos like this lol

  • @ButchHammer
    @ButchHammer Місяць тому

    Hello guys,
    Trying to follow here :)
    Always have hard time understanding when it comes to certificates... Here the etcd nodes got certificates signed by local CA and this local CA will have to be in the trusted certs of etcd ? And for the postgres server certificate, we create only one cert but self-signed (without any CA involved) ?
    Thank for the free share of knowledge as usual !

  • @FlorianTHEBAULT-h3w
    @FlorianTHEBAULT-h3w Місяць тому

    What would be the added value of having a HA postgres cluster compared to a HA VM on which Postgres is installed ?

  • @justinknash
    @justinknash 28 днів тому

    Have you used open source repmgr before? It removes the requirement for HA proxy and Keepalived.

  • @djordje1999
    @djordje1999 Місяць тому

    30:21 why patroni need to have etc's private key?

  • @MrEtoel
    @MrEtoel Місяць тому

    Great video! A lot of moving parts (so a bit intimidating) :o If I already have HAproxy running on a pfSense box it will double as a LB. Can you use the health checks there to achieve the same/similar thing (using the patroni API to get the master)? Pro/cons?

  • @Hermanverschooten
    @Hermanverschooten 23 дні тому

    Nice video, was waiting for this for a long time. Question about the keepalive config, is the authentication necessary, because mine logs that "VRRP version 3 does not support authentication. Ignoring." Oh and I did mine with all IPv6, was a but of a challenge but it works.

  • @nicholasl.4330
    @nicholasl.4330 Місяць тому +1

    This video makes me feel even better about using ClickHouse instead of Postgres or CockroachDB

  • @tylersmith8662
    @tylersmith8662 10 днів тому

    I've probably spent 10+ hours on this project. I am stuck on the etcd cluster. It just wont work. Restarted from scratch a couple of times certs are in place, data is in them, but this damn cluster just refuses to come up. I can manually connect the servers with other commands and using passed variables to the etcdctl command, and see healthy member lists, but these just refuse to actually connect using the env file. despite all the key/values being combed over and correct from the start. Also pg1 seems to just not want to listen on the peer port for some reason. troubleshooting that just lead me down a dead end rabbit hole. Maybe I'll give this another shot in a month or 2. Great video though. Love your content.
    Edit: The issue was the documentation here. I pulled up the video and slowly followed along while you skipped over the commands that you put in the documentation. You then ran the commands following the init of the etcd cluster showing that the cluster was online. When I skipped those commands I got the same results showing that my cluster was also online. I love the work you do here. It's great, but things like that just send people down rabbit holes wasting their time. I'd consider removing the 2 lines from your documentation telling people to run
    etcdctl endpoint health
    etcdctl member list

  • @fbifido2
    @fbifido2 Місяць тому

    @5:50 - can all these be setup on only 3 VM?

    • @fbifido2
      @fbifido2 Місяць тому

      can't the proxy & keepalive be on the same SQL VM?

  • @fbifido2
    @fbifido2 Місяць тому

    @16:00 - why did you not use the node fqdn in the cert?

    • @TechnoTim
      @TechnoTim  Місяць тому

      Great question! In general you probably should, I just didn't set up DNS for this cluster, which looking back I should have which would also illustrate better why having a SAN on the cert helps!

  • @ws_stelzi79
    @ws_stelzi79 Місяць тому

    Do I get the correct impression that you basically can use this HA cluster config with way more stuff to cluster than "just" PostgreSQL? Would be interesting to automate that also for other stuff. 🤔

  • @tomrogo87
    @tomrogo87 Місяць тому

    that was awesome!

  • @abetechtips
    @abetechtips Місяць тому

    Hey Tim, While watching your video a bulb sparked and wondered the following:
    Since you're configuring etcd in HA to be able to talk to each other host, how can this be used to configure a K3S cluster without using Ansible?
    Any way we can get an updated video of K3S since its bee a few years?

    • @TechnoTim
      @TechnoTim  Місяць тому

      @@abetechtips hi! I am not aware of a way to connect to an external etcd with k3s however it does support external mysql. I have videos on both without Ansible!

    • @abetechtips
      @abetechtips Місяць тому

      @ right, like you mentioned in your ansible video, who wants another VM to maintain 🥲

  • @geoDunkleAura
    @geoDunkleAura Місяць тому +1

    A Kubernetes "the hard way" would be nice

  • @RyanPhillipsATX
    @RyanPhillipsATX Місяць тому +2

    Etcd needs HA for its own three nodes to have quorum. So running etcd on the same nodes as postgresql would not be advisable.

    • @TomSwartz
      @TomSwartz Місяць тому

      This! Also, having more nodes is good for 'production' sites, but you'll always need an odd number of nodes, I've seen some sites do 5 or 7 nodes, just because of the spread across availability zones

    • @TechnoTim
      @TechnoTim  Місяць тому +1

      For sure, I talked about quorum quite a bit and there is a minimum of 3 nodes. When you add more nodes, you also install and configure etcd which means etcd will scale with postgres. Otherwise, you can host etcd youself. All explained in here.

  • @walideshtiwi6303
    @walideshtiwi6303 29 днів тому

    i keep getting TLS error when creating ETCD cluster the error is "rejected connection on client endpoint","remote-addr

  • @FullStackDevSecOps
    @FullStackDevSecOps Місяць тому

    Great video! Can you do it for mongodb ?

  • @munroegarrett
    @munroegarrett Місяць тому +1

    Postgres 17 supports node-based replication. A multi- master approach would be more robust and performant.

    • @TechnoTim
      @TechnoTim  Місяць тому

      Thank you! I'll check it out in the future! I am a fan of this because complex reads can be offloaded to replicas. I think both have their pros and cons but will look into this config too!

    • @BertPdeboy
      @BertPdeboy 26 днів тому

      can you give more concrete information about how this works (and pointers to documentation)?
      The way I understand it is; the existing logical replication setup can now be extended natively (without add-ons) to replicate bi-directionally resulting in 2 master nodes, so the "restricted" relationship master -> slave is lifted?

  • @fbifido2
    @fbifido2 Місяць тому

    can you know how to do this using microsoft SQL server for linux/docker ?

  • @devilondeath
    @devilondeath Місяць тому

    Will not be possible and easy to do it with kubernetes, instead 6 machine host?

  • @ascario
    @ascario Місяць тому

    Awesome video Tim! I might have missed it in this video or a previous one, but what are the specs of those xing proxmox nodes you used?

    • @TechnoTim
      @TechnoTim  Місяць тому +1

      Hey! They are Intel NUCs 11th gen i7 with 64GB of RAM.

    • @ascario
      @ascario Місяць тому

      @TechnoTim great, found the details of those on your recommended hardware page, thanks!

  • @travispulley5288
    @travispulley5288 Місяць тому

    Thanks for the useful tutorial! btw, I just noticed a typo "HA Porxy" in the description

    • @TechnoTim
      @TechnoTim  Місяць тому +1

      Thank you! Good catch, I just fixed that!

  • @demorez5
    @demorez5 Місяць тому

    i wanted to set up a "simple" posgtres replication for quite some time... i cannot express how overkill your solution feels :D

  • @SB-qm5wg
    @SB-qm5wg 5 днів тому

    This reminds me of MySQL Cluster/NDB.

  • @cheebadigga4092
    @cheebadigga4092 Місяць тому

    Ah yes "the hard way" of doing things is still the most fun sometimes. Getting your hands dirty with config files while any change gets applied immediately to production is wild xD but I have to do it in some old legacy systems we have at work so it's always a game of "did I ruin everything just now or did I fix it?" :D

  • @LampJustin
    @LampJustin Місяць тому +2

    Next video, a follow up on how to do it on kubernetes. Two steps, install Operator, let operator deploy HA setup 🤣

    • @smajl159
      @smajl159 27 днів тому

      Sure. Right after you deploy k8s cluster using one Azure node, one AWS node and one GCP mode.

  • @lsalgueiro
    @lsalgueiro Місяць тому

    Wait, in which machine should I generate the certificates?

    • @TechnoTim
      @TechnoTim  Місяць тому

      THIS MACHINE. 😂. I had to make it very clear.

  • @cxl520
    @cxl520 Місяць тому

    Was wondering, can you do it with MariaDB? WordPress site?

    • @ychto
      @ychto Місяць тому

      Galera is what you are looking for with MariaDB

  • @demanuDJ
    @demanuDJ Місяць тому

    That is why I prefer pacemaker way, I dont need haproxy for managing vip address. Of course I thinkt that patroni is still a great tool but still pacemaker wins for HA in PostgreSQL

  • @RonaldChmara
    @RonaldChmara Місяць тому

    LOL @ how the title and dialog both have "Postgres SQL" (pronounced Post Gres See Quell), when it actually has one less 'S' than that, PostgreSQL, pronounced Post-GreS-Queue-Ell, or simply PostgreS (with the 'QL' silent).
    It's a silly thing we've been (not really, but sometimes) arguing about in nerd-fight fashion for the last 35 years, it's even in the PostgreSQL FAQ.

  • @houdini4673
    @houdini4673 Місяць тому

    like before watching!

  • @geogmz8277
    @geogmz8277 Місяць тому

    54 mins! 😂 Holy baby Jesus!

  • @romanrm1
    @romanrm1 8 днів тому

    Now when you are 50 years old, in addition to all the other issues, you will have all the damn certs suddenly expire. :))

    • @TechnoTim
      @TechnoTim  8 днів тому

      @@romanrm1 i better up it to 100!

  • @BrantScalan
    @BrantScalan 12 днів тому

    whats the easy??

  • @Jaabaa_Prime
    @Jaabaa_Prime Місяць тому

    Great video. Yeah, don't bring the ludicrous KubeCtl pronunciations (even if a commiter says it is cuttle, as in fish?) to the rest of the sane world ;-) ctl = ConTroL , Systemctl)(ConTroL!), timedatectl)(ConTroL!), journnal(ctl)(ConTroL!), etcd(ctl)(ConTroL!), the only confusion is in Kubernetes, the rest of the world is fine with "control" ;-)

    • @TechnoTim
      @TechnoTim  Місяць тому +2

      I am a fan of saying "control" for ctl :). I never know how things cross over from k8s pronounciations :)

  • @dyto2287
    @dyto2287 Місяць тому

    What about ClouNativePG?

  • @SankalpSaxena99
    @SankalpSaxena99 Місяць тому

    Just one thing, if possible please use dark mode for explaining diagrams.
    PS. Amazing video, Thanks for sharing.

  • @djordje1999
    @djordje1999 Місяць тому

    Citus Data in Docker is way to go...

  • @tonyd6853
    @tonyd6853 Місяць тому

    open relational😂

  • @candoom
    @candoom Місяць тому +1

    I'd rather take a small 3-node Kubernetes cluster and put cnpg in there.

    • @TechnoTim
      @TechnoTim  Місяць тому +4

      I've since moved postgres out of my k8s cluster due to performance and continued issues with k8s storage (pods sometimes do not reconnect to their storage and takes manual intervention). Neither are inherent issues with postgres or cpng however it still resulted in more issues and downtime than hosted in VMs/LXC (0 downtime so far in over a month). I agree, if I do go back to k8s, a dedicated cluster with a dedicated node pool is the way to go to avoid performance issues you get with shared environments, but that still doesn't address the bigger issue I had that was PVCs hanging or not reconnecting to pods.

    • @ralumbur
      @ralumbur Місяць тому

      I am currently trying to migrate our production environment to kubernetes. Still testing at the moment. And I would also like to have postgresql in Kubernetes via an operator like cnpg, stackgres or zalando. Would you advise against something like that?

    • @TechnoTim
      @TechnoTim  Місяць тому +1

      @@ralumbur See my comment above for some of my issues and how I would do it different in the future.

    • @Kank2888u
      @Kank2888u Місяць тому

      @@TechnoTim Were you experiencing those issues with k3s/rke2 or on native k8s? I have run into PVC mounting issues on rke2 clusters on Harvester HCI before which is why I moved away at some point from Harvester and rke2/k3s.

    • @john-sea
      @john-sea Місяць тому

      @@TechnoTim Were you using local storage or network storage? I am pretty sure CNPG recommends local storage given the HA is covered by replicas.