Just a quick note, the zpool clear command is used to clear the error counts on drives back to zero, it does not clear data off the drive. zfs was asking you to do this so you could see if any new errors occurred on the drive you pulled and reinserted. so you can check if it actually needs to be replaced because errors are increasing or if all is OK and can be left alone. zpool status with high error counts can often be a sign of a drive in a prefail state that SMART has not yet picked upon, but zfs has due to all the checksums zfs uses. hope that helps and saves on future heart attacks 😊
Thanks for this thorough overview on how to have visibility on my proxmox server, right after i did this i discovered 8.1 includes some native GUI for notifications. Might be worth a new video lol
1. I love all these recommendations/ alternatives in the comments section. That's one of favorite part about home-labbing! 2. You could also use n8n. You can send the data via curl as a response to a listening webhook on the n8n server. I'm not sure if you could pull the error message from the log (i watched the video but came back for the microcenter link lol). N8n could then send out the alert via any of their options like slack, discord, mattermost, etc.
If I had a homelab with as many devices as you have, I would monitor the environment with Icinga, Nagios or CheckMK and use their notifications. Much simpler and easier to maintain than configuring alerts on each device.
you can send metrics to an influxdb or graphite endpoint too but the email alerts are nice and be harder to setup and tune with other monitoring systems
@@justwhyamerica I've build influxdb+Grafana to monitor Proxmox. But, error like that (zfs fail, disk fail or backup fail), can it do that? Where I can improve? Any reference would be appreciated, thank you.
Same story here. Didn't have alerts setup and randomly decided to check the status of my drives and saw that I was one drive away from losing all my data. Thanks for the great tutorial!
What perfect timing for this guide! The "old_age" flag on several of the drives in my oldest ZFS array just got triggered a few weeks ago so it will be nice to have notification as soon as one, or more, of them throw a more urgent SMART flag. Thanks for another great guide!
absolutely perfect timing! this is exactly what I'll be spending my last day in the office before the holidays doing. it always irked me that there wasn't more notification control in Prox. I have been relying on the ceph dashboard and cluster health to know when a disk goes down and it is far from ideal. And my Zabbix dashboard is not fully configured yet, so this is a perfect fix in the meantime.
I kinda did something similar, except i make a postfix mail relay vm so all my things on the subnet could relay without auth to it, and from there it could forward out to external mailbox. Just makes mail setup on new/additional systems easier/faster.
Nice way to get rid of some extra layers of complexity. Some labor involved but for us homelabbers it’s cool - we are eager to spend hours to save money. A very nice video as usually.
18:50 - Something you may want to mention is the difference between a RAID card and HBA. Most RAID cards won't give you SMART data from the drives even in JBOD mode so that may be a reason SMART may not be working.
Thanks for the video, it was very helpful. I just got my proxmox servers set up using this approach, and it worked great!
Рік тому
Hey Tim, recently found your channel and this is my first comment. Great fucking work! As a SysAdmin, this video is high quality! All the world's success to you in 2023.
Thanks a lot Tim for sharing your knowledge - works like a charm! I was able to setup email notifications on 3 PVE nodes (one Cluster) and my two PBS Servers. Not sure if I am really looking forward to receiving emails now ;-)
@TechnoTim Nice how to. I was able to fast forward (thanks for section labels) to the alerts setup. if you or anyone else is interested, I have a mailserver setup that uses mailjet, so 2 factor isnt required, and the emails are not marked as spam thanks to mailjet. All I had to do was configure proxmox postfix as a relay to my mailserver then turn on notifications, alerts, etc So a mailserver is an option especially if you have multiple systems you would like to have alerts setup and not having to setup each one like the how to provided. Setup one mailserver, then configure postfix on the others as a relay. Have a great day and thanks for sharing!
Leaving this comment for the algorithm and to reiterate my appreciation for all that you give to the community. This worked flawlessly for me. I have 4 proxmox nodes, so I ran most of the commands with adhoc ansible commands against the nodes. Super happy with having this working now. Before I was sporadically getting emails from proxmox, but gmail would often block them. This is a huge improvement. Thanks, again.
This video was very interesting. I had the exact same issue recently, a drive bricked itself and I noticed it later. I chose a different route to notifications: a selfhosted ntfy instance. I also ended up touching Zed, but I had to write a script to send the POST messages since ntfy is not one of the preconfigured services. I agree that Proxmox should include better alert management tools, although webhooks would not be my first choice. My ideal system would be similar to Zed: execute command X when event Y is registered. This could be adapted to any notification method and even for automating complex tasks like starting a failover in case of degraded zpool. And finally, if you want to test Zed it is a lot safer to start a scrub or a resilver manually instead of removing a drive, but you probably already knew that.
I set up mailjet through my domain on CloudFlare. Way better then using a Google email. Other fun things to do, if you have home assistant, there is a smpt2mqtt unofficial add-on. Then you can do whatever you want with notification
I like this, I've had alerts going since the start but I've had to test them several times because I never get any so I always get worried it's stopped working :D
@TechnoTim - Just a heads up, Proxmox is actively implementing webhooks, and there is a draft of the functionality already pushed to their mailing list. Sounds like it'll be coming soon!
What I find amusing is proxmox has the frontend UI for email alerts but not the backend out of the box, a bit half assed if you ask me. It should be ready to go out of the box.
Isn't there a way to forward an email to an webhook? Could be actually made with a pretty simple script. Edit: Forgot to say: Thank you for the tip!!! Something i absolute need either and didn't even thought about and anyway didnt knew how to setup!!! So yes! Amazing Video! Thanks for making such simple but very specific video's! While other UA-camrs make another 100 videos about setting up failover dual wan with pf or opnsense, where 99% of home peoples don't even have an second isp 😂 So thanks for making really good videos and not boring crap that everyone knows! Thing is, that there are a lot of your videos that are pretty amazing, or at least were at the time when you published them! You know after some time everyone knows something, like "authelia" for example, where 1 year later your video is useless, but at the time you published and not many knew about it, it was amazing.... So however, great Videos man! I mean i know a lot either, but some ideas i got from you and wanted to say thanks once! Cheers 😂👌
I think you should look at Zapier to finish this off and send all alerts to Slack. You create "Zaps" which are custom email addresses, and that then is able to send to a webhook, then to Slack. I use this for any alerts that are not webhook enabled and it allows me to send ALL alerts to Slack. And it's free!
Great video Tim. Out of interest (and this may already have been asked below), couldn't you use the likes of Promtail / Grafana Loki to aggregate the logged activities from Proxmox / smartctl (etc.) and push an alert via email / webhook from there?
one tip: it's nice that you provide an echo command for simplicity to put the password in the file, however that command will stay with the app password in your bash history file, which I believe it's not the nicest place to have sensitive data, isn't it?
Regarding Webhooks - and to be fair to the Proxmox team - they are just packaging Smartctl and ZED. They do not control these projects. The only one they totally control is their backup notifications. Absolutely yes, they can (and should) build a way to fully configure email via their GUI rather than the pain of fixing up the various config files. That is exactly what they have done for so many other aspects of configuring the tools they build upon. It should do this gracefully across the whole cluster (cluster level mailer configuration, per-host "from" config, etc). And they should push this config into the mail conigs of related tools like ZED and SmartCTL. Bush asking them to do the webhooks for those other projects is probably too big an ask.
It would be nice if the alerts could be posted via xmpp or element protocols instead. With the amount of spam and undelivered mail I get these days, I'm far more likely to spot an IM alert.
Oh, probably I missed something, but looks like you have auto dark theme reinstall on each apt install as trigger? Do you have this explained somewhere? 😄
I setup the Postfix Docker from juanluisbaptiste/postfix:latest and a local internal Postfix SMTP server. That way I can use it with any service internally, not just Proxmox.
I'd be interested in the pricing of the server and components. The Rack itself is quite expensive IMO, so i figured i get the profiles myself for cheap and build a case just by myself - since it is trivial, unless there is something i dont know about. Also, the servers themselves are very expensive too, so i would wonder how does someone goes about it - i got an Proliant G9 for less than a thousand, relatively happy about it, since it is relatively modern - but asked vendors for "basic but new ones" - and 10k didnt get me anywhere close to the used server i got. Another thing, would be UPS, i am kinda sick of hearing from them., since the upkeep is something you have to buy into as well - batteries turn bad. Thus, i have decided to get a laptop and let run proxmox etc. since it comes with its own UPS, basically - plus peripherals. With 2 drives in raid, it also has redundancy there and is about the same pricepoint as a PSU alone, which could not handle the load of the server(s) anyway for a meaningful time, other to power down. I'd love a video about the concept of how to go about getting a meaningful homelab - maybe it is me, as an European who has to deal with such high prices or just earn shit compared to the US, but spending 15-30k on a single server, really is not in my budget.
I am on proxmox 8.2. I had to change compatibility in order to make it work. It is in configuration file. Convert "compatibility_level = 2" => "compatibility_level = 3.6"
The stress caused by deliberately pulling a functional drive!!! Yeah, we all know it should be fine. We all know we should have engineered the system correctly. But we ALSO know, it is only 94% guaranteed to work. If it did, KINGS OF THE UNIVERSE! If it didn't: Well, I guess it was a learning experience.
Regarding the FROM part, have you seen Datacenter -> Options -> Email from address? That probably could have saved you the additional replace config in postfix :)
after switching to spectrum in 2105 Proxmox wold always send n email to the email i set during setup. errors like when my last 2 drives started failing to pass smart testing i would get a email at 5 am every day. back in the day you could send an email from any ip but until it was abused isps block the email out port. i may only get the email as my google account is tied to spectrum. im setting up asterisk again soon as i would prefer a call if i have a drive failure, water flood or carbon monoxide buildup. great video alot of it people still use pagers in my area.
Question: you chmod the sasl_passwd file to 600, but not the sasl_passwd.db. And when you catted the file, you can see the password is still readable there. Presumably you would want to chmod it too. But does postfix run as root or does it run as its own user that you'd have to chmod the file to?
23:25 - Yes, it can resilver that fast given you put the same disk back in. One of the things you do when removing a disk is to clear the partitions / ZFS metadata so it can't recognize the drive. I would guess it recognized the disk and then at worst would have had to write the data that changed since you unplugged it (I would guess it could pick up the outstanding changes via the SLOG but don't quote me on that). That resilvering process only having to touch actual data vs a RAID which operates at the block level of the disk is one of the major optimizations in ZFS. Its supposed to at worst need to scan the tree (ZFS's equivalent of the master file table in windows) looking for differences which is exactly what it does during a scrub just in this case it knows which drive not to trust.
Is there a story behind the reason why Tim is Sign-Language "Thanks for watching" at the end of every video? Just wondering... Thanks for sharing the video Tim. Followed and implemented the postfix alerts in my homelab pve.
Hey! Thanks! I worked on a team that focused on accessibility and worked with someone who was hearing impaired. So one day I added it to the end and just kept doing it :)
Awesome video. Followed your steps and got this setup on my PVE box. Everything is working but when I checked my drives as you pointed out at 18:20 it shows "SMART support is: Unavailable - device lacks smart capabilities". This happens on sda1 which is a SATA SSD where the OS is installed and sdb1 & sdc1 which are SAS HDDs used for backups. nvme0n1 does show as being supported; this is where all my VMs are installed.
While the major point is totally fair - that webhooking should be integrated into Prox itself - an alternative solution could be to webhook from existing monitoring solutions - e.g., Zabbix or Grafana. But of course that comes with a lot of added complexity and configuring to perform something that should be as easy as possible by nature.
is it possible to setup something like prometheus, snmp server, zabbix, graphite, influxdb, grafana to aggregate all the alerts from various VM hosts, VM guests, containers... Then configure alert policies and send emails / webhooks / smoke signals and what not? Surely proxmox should have some sort of native methods to report errors right? maybe at least syslog?
Curious - Why would this be better than letting Proxmox use its default SMTP relay (built in)? If I enter alert email addresses on a newly-implemented Proxmox server, they send out just fine. Also, it's worth noting that this configuration is very specific to Gmail. If using an alternative port with another service, complexities ensue.
How can I setup smart alerts just for the device, not for a ZFS pool? I am passing my raw disk to a truenas VM and the ZFS is managed within that VM. Truenas can warn me of the degraded pool, but I want to know about smart errors early and also.
Would say webhook or SNMP trap (I would much prefer webhooks) and should be a cluster-wide setting. I'm guessing the concern is depending on the failure there is risk that the alert message could be lost and that could be even worse. I'm thinking something like a node being offline and the alerts being sent from the cluster master means they aren't sent out till a new master is elected. What if something is stopping the election? I guess simplest way would be each host handles its own system / node level alerts and cluster master sends out cluster level alerts? Should be possible to push that config out via the etcd synced config and they already have the templates to change the message based on node.
Is there a way to get an alert when a VM starts up or shuts down? A Windows VM decided it was done for the day and shut down. Nothing in the event viewer explains why, just that it did it and it was unexpected. I didn't notice until I tried to use a service hosted on the VM and started getting error messages. An e-mail that my VM had shut down would have been extremely useful.
I am still little confused .. Need to know if the below scenario will work. I will install proxmox in my server .. install portainer and in each container inside portainer I will install wordpress .. will it work .. Please guide
I never set up any email stuff with my install and I get the backup alerts to my email with no issues. not sure what kind of magic proxmox is doing but it does seem to have a built in mail server that works out of the box. Would be nice to get web hooks though!
Hello!Thanks for everything!I would like to install nextcloud in proxmox for pictures like google photos. Is it ok with 2 hdd of 2 terabytes with zfs storage for data? Can you please tell me how to do it? Thanks in advance.its is ok hdd WD PURPLE SURVEILLANCE HDD?
Love watching your videos - although you talk faster than I do.... 😀 I cannot seem to find this APP PASSWORD option. It does not exist on this newly create account following your video.
maybe its because LTT hacking incident. "To help keep your account secure, from May 30, 2022, Google no longer supports the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password."
This is helpful. However, I utilize a custom email domain through iCloud. So the email address to authenticate is different than the email I will be sending from. Can't seem to find any Proxmox documentation on this. Any suggestions?
I saw this video and thought, "Cool, I really should set this up before it's too late." I went to log into my webshell for Proxmox and it 404'd. I hooked a monitor up, did a hard reboot, and on POST it threw an error for a fried RAM stick. So I guess it already WAS too late, for me...
What's your favorite thing about Proxmox?
Simple
Container Support and low overhead for management.
Scalability
That it is based on Debian and I could install mdadm and samba natively on it and set it up as a file server too. That way all the RAM cash is used :)
As @abdraoufx said it is simple, yet it also is powerful and lets you shape it to suit your needs if you know how to
Just a quick note, the zpool clear command is used to clear the error counts on drives back to zero, it does not clear data off the drive. zfs was asking you to do this so you could see if any new errors occurred on the drive you pulled and reinserted. so you can check if it actually needs to be replaced because errors are increasing or if all is OK and can be left alone. zpool status with high error counts can often be a sign of a drive in a prefail state that SMART has not yet picked upon, but zfs has due to all the checksums zfs uses. hope that helps and saves on future heart attacks 😊
Thank you!!!
Thanks for this thorough overview on how to have visibility on my proxmox server, right after i did this i discovered 8.1 includes some native GUI for notifications. Might be worth a new video lol
1. I love all these recommendations/ alternatives in the comments section. That's one of favorite part about home-labbing!
2. You could also use n8n. You can send the data via curl as a response to a listening webhook on the n8n server. I'm not sure if you could pull the error message from the log (i watched the video but came back for the microcenter link lol). N8n could then send out the alert via any of their options like slack, discord, mattermost, etc.
If I had a homelab with as many devices as you have, I would monitor the environment with Icinga, Nagios or CheckMK and use their notifications. Much simpler and easier to maintain than configuring alerts on each device.
There's a really good nagios/icinga plugin called check_pve . They have a gitlab repo easy to find!
you can send metrics to an influxdb or graphite endpoint too but the email alerts are nice and be harder to setup and tune with other monitoring systems
I'd be using something more modern like Prometheus but yes all of the above are better than configuring each host.
@@justwhyamerica I've build influxdb+Grafana to monitor Proxmox.
But, error like that (zfs fail, disk fail or backup fail), can it do that?
Where I can improve? Any reference would be appreciated, thank you.
Ah, I need to mention.
Can we monitor Proxmox temperature?
Please kindly give any suggestion or tutorial online would be nice.
Same story here. Didn't have alerts setup and randomly decided to check the status of my drives and saw that I was one drive away from losing all my data. Thanks for the great tutorial!
postfix setup is horrible on proxmox, there should be an easier way in these days of notifications, or the ability to set it up via the GUI.
Still horrible. Truenas is super easy
They added smtp via gui
What perfect timing for this guide! The "old_age" flag on several of the drives in my oldest ZFS array just got triggered a few weeks ago so it will be nice to have notification as soon as one, or more, of them throw a more urgent SMART flag.
Thanks for another great guide!
New version of ProxMox has a GUI finally! But always nice to know it via CLI.
absolutely perfect timing! this is exactly what I'll be spending my last day in the office before the holidays doing. it always irked me that there wasn't more notification control in Prox. I have been relying on the ceph dashboard and cluster health to know when a disk goes down and it is far from ideal. And my Zabbix dashboard is not fully configured yet, so this is a perfect fix in the meantime.
I kinda did something similar, except i make a postfix mail relay vm so all my things on the subnet could relay without auth to it, and from there it could forward out to external mailbox. Just makes mail setup on new/additional systems easier/faster.
Nice way to get rid of some extra layers of complexity. Some labor involved but for us homelabbers it’s cool - we are eager to spend hours to save money. A very nice video as usually.
18:50 - Something you may want to mention is the difference between a RAID card and HBA. Most RAID cards won't give you SMART data from the drives even in JBOD mode so that may be a reason SMART may not be working.
Thanks for the video, it was very helpful. I just got my proxmox servers set up using this approach, and it worked great!
Hey Tim, recently found your channel and this is my first comment. Great fucking work! As a SysAdmin, this video is high quality! All the world's success to you in 2023.
Thanks a lot Tim for sharing your knowledge - works like a charm! I was able to setup email notifications on 3 PVE nodes (one Cluster) and my two PBS Servers. Not sure if I am really looking forward to receiving emails now ;-)
@TechnoTim Nice how to. I was able to fast forward (thanks for section labels) to the alerts setup. if you or anyone else is interested, I have a mailserver setup that uses mailjet, so 2 factor isnt required, and the emails are not marked as spam thanks to mailjet. All I had to do was configure proxmox postfix as a relay to my mailserver then turn on notifications, alerts, etc So a mailserver is an option especially if you have multiple systems you would like to have alerts setup and not having to setup each one like the how to provided. Setup one mailserver, then configure postfix on the others as a relay. Have a great day and thanks for sharing!
Really handy video guide, and very timely for me. Glad you were balanced in reporting the notification system's shortcomings also.
One thing that came to mind is that postfix has a mechanism to retry delivery until it succeeds, a simple webhook script probably won't have that
Leaving this comment for the algorithm and to reiterate my appreciation for all that you give to the community. This worked flawlessly for me. I have 4 proxmox nodes, so I ran most of the commands with adhoc ansible commands against the nodes. Super happy with having this working now. Before I was sporadically getting emails from proxmox, but gmail would often block them. This is a huge improvement. Thanks, again.
Thanks for the algo boost and the kind words!
This video was very interesting. I had the exact same issue recently, a drive bricked itself and I noticed it later.
I chose a different route to notifications: a selfhosted ntfy instance. I also ended up touching Zed, but I had to write a script to send the POST messages since ntfy is not one of the preconfigured services.
I agree that Proxmox should include better alert management tools, although webhooks would not be my first choice. My ideal system would be similar to Zed: execute command X when event Y is registered. This could be adapted to any notification method and even for automating complex tasks like starting a failover in case of degraded zpool.
And finally, if you want to test Zed it is a lot safer to start a scrub or a resilver manually instead of removing a drive, but you probably already knew that.
I set up mailjet through my domain on CloudFlare. Way better then using a Google email.
Other fun things to do, if you have home assistant, there is a smpt2mqtt unofficial add-on. Then you can do whatever you want with notification
I like this, I've had alerts going since the start but I've had to test them several times because I never get any so I always get worried it's stopped working :D
Pro-tip: Connect your email alerts to Pushover for even more control! :D
What a great tutorial. My alerts are now configured, thanks.
Amazing, thank you. 1 quick note in your documentation under the "add the module to our postfix config" you have the word "nano" twice in the command.
Thank you, I can fix it or you can open a PR! It's open source!
Another great tutorial Tim, thank for sharing!!!
Glad you enjoyed it
Thanks for that tutorial. I configured it for Proxmox and also for PBS :D
@TechnoTim - Just a heads up, Proxmox is actively implementing webhooks, and there is a draft of the functionality already pushed to their mailing list. Sounds like it'll be coming soon!
Logging in to set this up now! Once again thank you for the video!!
What I find amusing is proxmox has the frontend UI for email alerts but not the backend out of the box, a bit half assed if you ask me. It should be ready to go out of the box.
Most useful tutorial so far IMO! Thank you Tim! :)
Isn't there a way to forward an email to an webhook?
Could be actually made with a pretty simple script.
Edit: Forgot to say:
Thank you for the tip!!! Something i absolute need either and didn't even thought about and anyway didnt knew how to setup!!!
So yes! Amazing Video!
Thanks for making such simple but very specific video's!
While other UA-camrs make another 100 videos about setting up failover dual wan with pf or opnsense, where 99% of home peoples don't even have an second isp 😂
So thanks for making really good videos and not boring crap that everyone knows!
Thing is, that there are a lot of your videos that are pretty amazing, or at least were at the time when you published them!
You know after some time everyone knows something, like "authelia" for example, where 1 year later your video is useless, but at the time you published and not many knew about it, it was amazing....
So however, great Videos man! I mean i know a lot either, but some ideas i got from you and wanted to say thanks once!
Cheers 😂👌
This is pretty cool. I wondered how this would be set up for email alerts.
It works fine Tim, thanks for the help!
I set mine up when I setup the hypervisor. Good vid ,tim
Thank you!
I think you should look at Zapier to finish this off and send all alerts to Slack. You create "Zaps" which are custom email addresses, and that then is able to send to a webhook, then to Slack. I use this for any alerts that are not webhook enabled and it allows me to send ALL alerts to Slack. And it's free!
Great video Tim. Out of interest (and this may already have been asked below), couldn't you use the likes of Promtail / Grafana Loki to aggregate the logged activities from Proxmox / smartctl (etc.) and push an alert via email / webhook from there?
one tip: it's nice that you provide an echo command for simplicity to put the password in the file, however that command will stay with the app password in your bash history file, which I believe it's not the nicest place to have sensitive data, isn't it?
Regarding Webhooks - and to be fair to the Proxmox team - they are just packaging Smartctl and ZED. They do not control these projects. The only one they totally control is their backup notifications.
Absolutely yes, they can (and should) build a way to fully configure email via their GUI rather than the pain of fixing up the various config files. That is exactly what they have done for so many other aspects of configuring the tools they build upon. It should do this gracefully across the whole cluster (cluster level mailer configuration, per-host "from" config, etc). And they should push this config into the mail conigs of related tools like ZED and SmartCTL. Bush asking them to do the webhooks for those other projects is probably too big an ask.
Webhooks will definitely make me try Proxmox as I'm still more in Qemu. I'll be using a Discord Server to monitor the services and servers running.
It would be nice if the alerts could be posted via xmpp or element protocols instead. With the amount of spam and undelivered mail I get these days, I'm far more likely to spot an IM alert.
Great video Tim. I wish esxi had alerts like this
Oh, probably I missed something, but looks like you have auto dark theme reinstall on each apt install as trigger? Do you have this explained somewhere? 😄
iLude in our discord created it! Hop in and grab it! It's great!
@@TechnoTim Thank you, just found in ilude gist
Thanks for this very useful tutorial.
I setup the Postfix Docker from juanluisbaptiste/postfix:latest and a local internal Postfix SMTP server. That way I can use it with any service internally, not just Proxmox.
Super helpful and informative video. Thank you Sir.
Great ! Keep going with your Videos - like your way to do it ! Greetings from Germany!
I'd be interested in the pricing of the server and components.
The Rack itself is quite expensive IMO, so i figured i get the profiles myself for cheap and build a case just by myself - since it is trivial, unless there is something i dont know about.
Also, the servers themselves are very expensive too, so i would wonder how does someone goes about it - i got an Proliant G9 for less than a thousand, relatively happy about it, since it is relatively modern - but asked vendors for "basic but new ones" - and 10k didnt get me anywhere close to the used server i got.
Another thing, would be UPS, i am kinda sick of hearing from them., since the upkeep is something you have to buy into as well - batteries turn bad. Thus, i have decided to get a laptop and let run proxmox etc. since it comes with its own UPS, basically - plus peripherals. With 2 drives in raid, it also has redundancy there and is about the same pricepoint as a PSU alone, which could not handle the load of the server(s) anyway for a meaningful time, other to power down.
I'd love a video about the concept of how to go about getting a meaningful homelab - maybe it is me, as an European who has to deal with such high prices or just earn shit compared to the US, but spending 15-30k on a single server, really is not in my budget.
I am on proxmox 8.2. I had to change compatibility in order to make it work. It is in configuration file. Convert "compatibility_level = 2" => "compatibility_level = 3.6"
Awesome tutorial Tim !!!
Nice work !
Would be good to get this to work with Telegram too 😍
The stress caused by deliberately pulling a functional drive!!! Yeah, we all know it should be fine. We all know we should have engineered the system correctly. But we ALSO know, it is only 94% guaranteed to work. If it did, KINGS OF THE UNIVERSE! If it didn't: Well, I guess it was a learning experience.
Haha! Yeah, I was super stressed out doing it if you couldn't tell in my voice!
Regarding the FROM part, have you seen Datacenter -> Options -> Email from address? That probably could have saved you the additional replace config in postfix :)
I have and covered it but that doesn't fix the "name" of root, which I cover :)
after switching to spectrum in 2105 Proxmox wold always send n email to the email i set during setup. errors like when my last 2 drives started failing to pass smart testing i would get a email at 5 am every day. back in the day you could send an email from any ip but until it was abused isps block the email out port. i may only get the email as my google account is tied to spectrum. im setting up asterisk again soon as i would prefer a call if i have a drive failure, water flood or carbon monoxide buildup. great video alot of it people still use pagers in my area.
Question: you chmod the sasl_passwd file to 600, but not the sasl_passwd.db. And when you catted the file, you can see the password is still readable there. Presumably you would want to chmod it too. But does postfix run as root or does it run as its own user that you'd have to chmod the file to?
23:25 - Yes, it can resilver that fast given you put the same disk back in. One of the things you do when removing a disk is to clear the partitions / ZFS metadata so it can't recognize the drive. I would guess it recognized the disk and then at worst would have had to write the data that changed since you unplugged it (I would guess it could pick up the outstanding changes via the SLOG but don't quote me on that). That resilvering process only having to touch actual data vs a RAID which operates at the block level of the disk is one of the major optimizations in ZFS. Its supposed to at worst need to scan the tree (ZFS's equivalent of the master file table in windows) looking for differences which is exactly what it does during a scrub just in this case it knows which drive not to trust.
Is there a story behind the reason why Tim is Sign-Language "Thanks for watching" at the end of every video? Just wondering...
Thanks for sharing the video Tim. Followed and implemented the postfix alerts in my homelab pve.
Hey! Thanks! I worked on a team that focused on accessibility and worked with someone who was hearing impaired. So one day I added it to the end and just kept doing it :)
This was so helpful! Thank you!
Thanks, will give this a go. As others have said, it would be cool if this could be done via the web gui
THX for the guide ! Keep up !
Thanks! That's really helpful.
What do you think about self-hosting a SMTP server for that purpose?
yeah don't set this to send all backup notifications, just got my google account locked for the "spam" policy (when it was 1 email a day)
Spinup your ouw SMTP server and redirect all incoming mails to webhooks
Fantastic! Thnx!
Glad you liked it!
Awesome video. Followed your steps and got this setup on my PVE box. Everything is working but when I checked my drives as you pointed out at 18:20 it shows "SMART support is: Unavailable - device lacks smart capabilities". This happens on sda1 which is a SATA SSD where the OS is installed and sdb1 & sdc1 which are SAS HDDs used for backups. nvme0n1 does show as being supported; this is where all my VMs are installed.
While the major point is totally fair - that webhooking should be integrated into Prox itself - an alternative solution could be to webhook from existing monitoring solutions - e.g., Zabbix or Grafana. But of course that comes with a lot of added complexity and configuring to perform something that should be as easy as possible by nature.
Grazie TechnoTim
Thanks Tim
I definitely won't look any further then nearest micro center
It would be just few thousands miles
I love in UK
is it possible to setup something like prometheus, snmp server, zabbix, graphite, influxdb, grafana to aggregate all the alerts from various VM hosts, VM guests, containers...
Then configure alert policies and send emails / webhooks / smoke signals and what not? Surely proxmox should have some sort of native methods to report errors right? maybe at least syslog?
Curious - Why would this be better than letting Proxmox use its default SMTP relay (built in)? If I enter alert email addresses on a newly-implemented Proxmox server, they send out just fine. Also, it's worth noting that this configuration is very specific to Gmail. If using an alternative port with another service, complexities ensue.
It's pretty wild that Proxmox still doesn't have a way to do this through the UI.
Agreed!
Hi, great content! What about proxmox-mail-gateway instead of a third party (that we have to pu our thrust in)?
How can I setup smart alerts just for the device, not for a ZFS pool? I am passing my raw disk to a truenas VM and the ZFS is managed within that VM. Truenas can warn me of the degraded pool, but I want to know about smart errors early and also.
I just came here quickly to like for the thumbnail
Would say webhook or SNMP trap (I would much prefer webhooks) and should be a cluster-wide setting. I'm guessing the concern is depending on the failure there is risk that the alert message could be lost and that could be even worse. I'm thinking something like a node being offline and the alerts being sent from the cluster master means they aren't sent out till a new master is elected. What if something is stopping the election? I guess simplest way would be each host handles its own system / node level alerts and cluster master sends out cluster level alerts? Should be possible to push that config out via the etcd synced config and they already have the templates to change the message based on node.
Thanks!
Good to see you
Can you do a tutorial on OVS in proxmox?
Is there a way to get an alert when a VM starts up or shuts down? A Windows VM decided it was done for the day and shut down. Nothing in the event viewer explains why, just that it did it and it was unexpected. I didn't notice until I tried to use a service hosted on the VM and started getting error messages. An e-mail that my VM had shut down would have been extremely useful.
I am still little confused ..
Need to know if the below scenario will work.
I will install proxmox in my server .. install portainer and in each container inside portainer I will install wordpress .. will it work ..
Please guide
I never set up any email stuff with my install and I get the backup alerts to my email with no issues. not sure what kind of magic proxmox is doing but it does seem to have a built in mail server that works out of the box.
Would be nice to get web hooks though!
Hello!Thanks for everything!I would like to install nextcloud in proxmox for pictures like google photos. Is it ok with 2 hdd of 2 terabytes with zfs storage for data? Can you please tell me how to do it? Thanks in advance.its is ok hdd WD PURPLE SURVEILLANCE HDD?
my isp doesnt block port 25 and i get emails form smart monitoring already from the default setup
Great tutorial! Do you know a way to use Telegram for the same purpose?
Thanks got email going!!!!
But cron is emailing on the hour is there a fix?
Is there a write up somewhere we can copy and paste the text you are putting in? I don't see the link your description.
Love watching your videos - although you talk faster than I do.... 😀 I cannot seem to find this APP PASSWORD option. It does not exist on this newly create account following your video.
Hey Tim, can proxmox send alerts to more than one email address?
How timely. Just noticed today that a drive in my raid-z1 pool has been dead for a few months. 😓💦
3:34 Checkpoint
webhooks when used correctly can save a lot of frustration.
I checked my google account under "Security" --> "Signing into Google" there is no section for "App Password" :(
maybe its because LTT hacking incident.
"To help keep your account secure, from May 30, 2022, Google no longer supports the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password."
can we get the google mail config lines? your came was blocking a good portion of it.
Thanks!
hmm, what's that saying? Oh yeah... "Raid is NOT a backup".
I had to install postfix-pcre for the header rewrites or I just got an error.
This is helpful. However, I utilize a custom email domain through iCloud. So the email address to authenticate is different than the email I will be sending from.
Can't seem to find any Proxmox documentation on this. Any suggestions?
Does icloud support SMTP?
Yes! @@TechnoTim
Too old method , PVE 8 add a notifications center with STMP etc, no ?
I saw this video and thought, "Cool, I really should set this up before it's too late." I went to log into my webshell for Proxmox and it 404'd. I hooked a monitor up, did a hard reboot, and on POST it threw an error for a fried RAM stick. So I guess it already WAS too late, for me...