Our pfsense tutorials are here lawrence.technology/pfsense/ HAProxy Videos mentioned How To Setup ACME, Let's Encrypt, and HAProxy HTTPS offloading on pfsense ua-cam.com/video/gVOEdt-BHDY/v-deo.html How To Create pfsense Let's Encrypt Wildcard Certificates using HAProxy ua-cam.com/video/jpyUm53we-Y/v-deo.html ⏱️ Timestamps ⏱️ 0:00 HA Proxy Troubleshooting 1:26 Change Port and Disable webConfigurator redirect 2:23 Backend Encrypt SSL & SSL Checks 5:06 Front End and Firewall Rules 6:34 Front End ACL Rules 8:04 DNS Troubleshooting 9:23 testing certificate responses
Thank you so much for this video! I've struggled with the concepts of configuring an HTTP site mapped to an HTTPS site using Squid Reverse Proxy, then HAProxy, but this morning, after watching this video, the light bulb glowed bright and I am ready to host websites with confidence! You are the man! Never forget that!
This is a great reference video for trouble shooting. Help me solved many issues. They included: - my internal URL/DNS entry was pointing to the server rather than HA proxy - tools (dig and openssl) to verify what certificate is being returned
I realize this is old, but... At 2:40, I wonder if you are using the "Client certificate" option wrong when trying to configure "Encrypt(SSL)" on the backend? You appear to have a LetsEncrypt server cert on your XenOrchestra server. But the "Client certificate" field is for when the "certificate will be sent if the server send a client certificate request". Unless you require a client SSL certificate to access your XenOrchestra server, the "Client certificate" field is not needed. All that is needed for the SSL encryption to occur is for your HAProxy instance to trust the CA of the XenOrchestra server cert. You do this implicitly by setting CA = "None" on the backend. Therefore, the "Client certificate" is not needed.
Love the way you explained all the small intricacies of HAProxy that might be difficult for a beginner to understand! One thing I would like to add seeing as I misunderstood the concept at first: When traffic enters HAProxy it will evaluate ALL rules found on ALL frontends even though this might not be wanted. Some ACLs are better off being tied to a specific backend. I wanted to create a rule to block access to a group of local services of mine (netdata, xenorchestra, portainer, dozzle etc.) on my publicly exposed HAProxy when the source IP is neither my public IP nor from my LAN network. At first I made the mistake of adding the "http-request deny" ACL to the frontend and that then cut off public access to absolutely everything... even the services, that I actually did want exposed on the internet.
Thanks for the video. I was trying to point to a Nextcloud instance on 80 without SSL but as soon as I changed it to 443 and ticked the encrypt box it worked.
Great video, It helped me with the issue that once HAproxy is enabled and the front end configured, the webconfig page is not accessable any more. It's because of port conflication.
I spent an entire weekend troubleshooting. When I figured out that my firewall wasn't working as expected, I called up my ISP. I was actually behind a CGNAT. it wasn't until I opted out of this that my firewall rules actually worked correctly and suddenly HAProxy was working perfectly.
Yeah this helped me out. The part I missed was I tuned out when you got to the DNS part for the back end servers. I have the names assigned when I give them assign them static ip addresses in the DHCP part. So I thought I was done. Obviously I missed the part where you have to have the name point to the HAProxy instead...
Still having issues. When I point the FrontEnd to my LAN address , everything works fine. Once I switch it to my DMZ address everything breaks. LAN to DMZ rules are open permitting anything. DNS entry has been modified to have everything point to my DMZ address. Its not a FW rules issue or DNS. Suggestions?
I have a question on redirect to a subdomain... eg. I used synology drive. and have already setup HAProxy to access my nas using nas.domain.com... From there I can also access to Synology Drive via nas.domain.com/drive... I would like though to setup drive.domain.com to redirect to nas.domain.com/drive.... Can you please help with how this is done via HAProxy.
I have DuckDNS setup to resolve to my external IP and use Lets Encrypt with DuckDNS domain to get valid certs for use in HAProxy. HAProxy listens on WAN, but I dont portforward 443 to the Internet. So if I'm on my internal network, the DuckDNS domain resolves and hits my HAProxy. I can then add any backend to proxy all my services from pfsense itself, to plex to grafana. It's great for homelab to not remember IPs and dismiss self-signed warnings. Ever since lets encrypt supports wildcard certs its been great 1 cert to rule them all.
I used to be with a domain provider that didn't offer wildcard dns entries even though I had the beautiful LE wildcard cert. Those were some dark times...
Hi Lawrence! Any idea how to set PFSense's HA_Proxy to send email notification alerts when Backend is down for example ? I know it can be done with .lua scripts but do you know the exact order how things need to be configured? Looking forward hearing from you.
Not something I have ever tested, I send all my logs to Graylog and create triggers in Graylog to notify me when things go wrong. We also use UptimeKuma to keep let us know when things go down.
@@LAWRENCESYSTEMS Lawrence, another question. How do you configure GrayLog to send you Windows logs. I am using NXLog to send the Logs from the Windows machine to GrayLog. After that I want to create a email trigger, when the local admin password is changed that event I see in graylog, but don't know how exactly to send it over to come as a Email Alert ? Can I email you ? Thanks in advance! Appreciate it in a million!
Most of the issues people are having such as DNS, firewall rules, have pfsnese on the same port, ACL, and Port Forward all need to be done before there are any real logs to dive into. Also had to cut out some things or the video would be substantially longer. Maybe I will make an HAProxy Log troubleshooting video if needed.
@@LAWRENCESYSTEMS in general the pfsense haproxy should only be used internally and for external facing reverse proxies its better to be a standalone one so it doesn't pose the risk of compromising your firewall. At least from my understanding of how you should separate your network? So with internal use cases, if you're digging into logs to troubleshoot its probably something advanced that the pfsense GUI doesnt support. Might as well have a dedicated HAProxy or nginx instance at that point?
Thank you for wonderfully video, i am facing issue i want to use my domain without 'www' i tried but not resolved and shows (503 Service Unavailable No server is available to handle this request.) i need help in this with Haproxy and domain configuration, once again thank you
Hi Lawrence, i am struggling with setup of 2 pfsense one is Master and second beck up node with CARP and HA Sync. Everything is sync with backup node like firewall rules, HA proxy fronted and backend, user except ACME certificates. I have create create ACME certificate on Master after successful creation i can't see or didn't sync newly successful created cert on backup node. Both nodes have HaProxy and ACME package installed. I have tried to copy files /conf/acme to backup node but on web GUI i can't see certificate listed.
@@LAWRENCESYSTEMS thanks for your prompt reply! I am getting following issue while issuing ACME certificate : acme_issuecert.log: response='{ "type": "urn:ietf:params:acme:error:malformed", "detail": "Unable to update challenge :: authorization must be pending", "status": 400 }' On Web GUI: Service - Acme - Certificate Getting webroot for domain='mydomainname.network' Verifying: devop-testing2wtls.r3c.network Standalone mode server Pending Pending Pending mydomainnamenetwork:Verify error:Fetching mydomainname.network/.well-known/acme-challenge/5cec_IyG_JqMoGGEy-D8iO0QJBhPhexzdLCePw9BiHg: Timeout during connect (likely firewall problem) Please check log file for more details: /tmp/acme/devop-testing2wtls/acme_issuecert.log
Hi lawrence can i access my pfsense server outside the network without port forwarding? example tunneling. because my ISP doesn't provide bridge modem or port forwarding
Hi, I have a pfsense with haproxy and I've experienced some times and in a random way that, when I make a change in some ACL, haproxy configuration has broken when i apply it. Apparently the configuration is applied normally, but if I check backend status, the backend servers disappear. I can see that in haproxy config file the lines are missing, but in pfsense gui I have the related entries properly configured. It happens also in backends not related to ACL modified. Do you have any ideas about this bug ? Thanks!
@Lawrence Systems , just to share with you and with this channel. The problem was a random missing DNS resolution of backend hostname by pfsense resolver. PHP code doesn't catch this type of error and, in case of missing resolution, doesn't generate server line in haproxy.conf. The workaround is to configure "global resolver" in haproxy settings or, I suppose, using BIND in pfsense to replicate local zones (DNS Resolver in pfSense sometimes has a strange behaviour). We have understood this just reading pfsense PHP code with the help of one of our developers.
Can someone help me? I wasn't able to fix the dns part. I have configured DNS Resolver and added the ip of the domain's IP for the local server but when I dig into dns on the pfsense box it does not show the local ip. It shows public IP. Besides DNS, I have configured backend and front end the front ends passed the traffic to back end however backend gives 404 on the domain. Thank you.
I have the same issue, I dig my domain and comes back with public IP, Tom mentioned this in the video but didnt say how to fix this. Did you get it fixed?
@@brianjensen2923 As much as I remember, I have fixed this issue by enabling DNS Resolver and adding proper records. Then it showed the local ip however after that I didn't need that configuration on the future installations...
@@magfelsic4909 Nothing at the moment. I was using an 2 Nginx LBs, and a database for a k3s Rancher, 2 server nodes, and 2 worker node deployment. Ended up tearing the whole thing down, too much maintenance and tinkering. I'm liking LXC for persistent shared services like a database
hi, nice video, I have a working pfsence + haproxy working fine, but when I download a file through https, look like the file download through http, the google chrome browser make a warnig telling the conexion is no secure, how can I do to force the download through https?
You are an absolute blessing my man! Hope you can help me out as I am having a world of problems trying to figure out NextCloud (running in Docker on UnRAID server) communication externally through pfSense with HAProxy. I am getting an error "400 Bad Request The plain HTTP request was sent to HTTPS port"
Yes, it works similar. i recommend running Nginx Reverse on a Separate VM and run the Web Server itself on another VM then use ProxyPass with the IP Address
Thanks for this video. It helped find some things I had wrong, but I'm still not there. I'm using CloudFlare, and I set up the DNS records on their site, but when I run 'dig' (from Windows WSL), I get two records for an answer, and neither of them match my home IP. I don't know where to go from there. I also got lost toward the end of the video because it shows what you should see when the certificate is set up correctly, but I just got two messages: "...No route to host:../crypto/bio/b_sock2.c:110:" and "...BIO_connect:connect error:../crypto/bio/b_sock2.c:111:" The last line says "connect:errno=113".
Our pfsense tutorials are here
lawrence.technology/pfsense/
HAProxy Videos mentioned
How To Setup ACME, Let's Encrypt, and HAProxy HTTPS offloading on pfsense
ua-cam.com/video/gVOEdt-BHDY/v-deo.html
How To Create pfsense Let's Encrypt Wildcard Certificates using HAProxy
ua-cam.com/video/jpyUm53we-Y/v-deo.html
⏱️ Timestamps ⏱️
0:00 HA Proxy Troubleshooting
1:26 Change Port and Disable webConfigurator redirect
2:23 Backend Encrypt SSL & SSL Checks
5:06 Front End and Firewall Rules
6:34 Front End ACL Rules
8:04 DNS Troubleshooting
9:23 testing certificate responses
Thanks for clarifying the resolved IP needs to be active network HA-proxy is listening to. Easy mistake to make!
Thank you so much for this video! I've struggled with the concepts of configuring an HTTP site mapped to an HTTPS site using Squid Reverse Proxy, then HAProxy, but this morning, after watching this video, the light bulb glowed bright and I am ready to host websites with confidence! You are the man! Never forget that!
This is a great reference video for trouble shooting. Help me solved many issues. They included:
- my internal URL/DNS entry was pointing to the server rather than HA proxy
- tools (dig and openssl) to verify what certificate is being returned
I realize this is old, but... At 2:40, I wonder if you are using the "Client certificate" option wrong when trying to configure "Encrypt(SSL)" on the backend? You appear to have a LetsEncrypt server cert on your XenOrchestra server. But the "Client certificate" field is for when the "certificate will be sent if the server send a client certificate request". Unless you require a client SSL certificate to access your XenOrchestra server, the "Client certificate" field is not needed. All that is needed for the SSL encryption to occur is for your HAProxy instance to trust the CA of the XenOrchestra server cert. You do this implicitly by setting CA = "None" on the backend. Therefore, the "Client certificate" is not needed.
Love the way you explained all the small intricacies of HAProxy that might be difficult for a beginner to understand!
One thing I would like to add seeing as I misunderstood the concept at first:
When traffic enters HAProxy it will evaluate ALL rules found on ALL frontends even though this might not be wanted. Some ACLs are better off being tied to a specific backend.
I wanted to create a rule to block access to a group of local services of mine (netdata, xenorchestra, portainer, dozzle etc.) on my publicly exposed HAProxy when the source IP is neither my public IP nor from my LAN network. At first I made the mistake of adding the "http-request deny" ACL to the frontend and that then cut off public access to absolutely everything... even the services, that I actually did want exposed on the internet.
Thanks for the video. I was trying to point to a Nextcloud instance on 80 without SSL but as soon as I changed it to 443 and ticked the encrypt box it worked.
A very comprehensive overview as usual Laurence , cheers.
Ha ha so surreal, UA-cam lists "lets hack your home network " on the right side menu.
Great video, It helped me with the issue that once HAproxy is enabled and the front end configured, the webconfig page is not accessable any more. It's because of port conflication.
I spent an entire weekend troubleshooting. When I figured out that my firewall wasn't working as expected, I called up my ISP. I was actually behind a CGNAT.
it wasn't until I opted out of this that my firewall rules actually worked correctly and suddenly HAProxy was working perfectly.
DNS got me - wasn't pointed to HAProxy. Thanks for this.
Tom, I've followed the guide but I keep getting my self signed cert? If I do the openssl test with the correct it I get the right one though?
Thanks for that - but how do you get your server Ips to resolve to HAProxy IP?
Great video as always. I had no issues, but still missed the first step with the redirect rule checkbox. Thnx Lawrence!
Yeah this helped me out. The part I missed was I tuned out when you got to the DNS part for the back end servers. I have the names assigned when I give them assign them static ip addresses in the DHCP part. So I thought I was done. Obviously I missed the part where you have to have the name point to the HAProxy instead...
Thank you for making this video it fixed baisally every issue i had and mistake I made from watching the first 2 videos
one thing i missed was i needed to set the dns entries to the pfsense and not to the servers themselves
you already encounter a watchdog error in pfsense?
Did you have an advise how to fix that?
Thank you
Good video but I was expected something about backend down... like one is up and the others down!
Still having issues. When I point the FrontEnd to my LAN address , everything works fine. Once I switch it to my DMZ address everything breaks. LAN to DMZ rules are open permitting anything. DNS entry has been modified to have everything point to my DMZ address. Its not a FW rules issue or DNS. Suggestions?
I have a question on redirect to a subdomain... eg. I used synology drive. and have already setup HAProxy to access my nas using nas.domain.com... From there I can also access to Synology Drive via nas.domain.com/drive...
I would like though to setup drive.domain.com to redirect to nas.domain.com/drive.... Can you please help with how this is done via HAProxy.
I have DuckDNS setup to resolve to my external IP and use Lets Encrypt with DuckDNS domain to get valid certs for use in HAProxy. HAProxy listens on WAN, but I dont portforward 443 to the Internet. So if I'm on my internal network, the DuckDNS domain resolves and hits my HAProxy. I can then add any backend to proxy all my services from pfsense itself, to plex to grafana. It's great for homelab to not remember IPs and dismiss self-signed warnings. Ever since lets encrypt supports wildcard certs its been great 1 cert to rule them all.
Yeah, I do love LE wild card certs!
I used to be with a domain provider that didn't offer wildcard dns entries even though I had the beautiful LE wildcard cert.
Those were some dark times...
One other thing - make sure haproxy is enabled! I had to enter the number of connections for the process manually for some reason.
OMG thank you so much! Literally 10 seconds in and you solved my problem :) (web redirect checkbox!) ❤❤❤
Hi Lawrence! Any idea how to set PFSense's HA_Proxy to send email notification alerts when Backend is down for example ? I know it can be done with .lua scripts but do you know the exact order how things need to be configured? Looking forward hearing from you.
Not something I have ever tested, I send all my logs to Graylog and create triggers in Graylog to notify me when things go wrong. We also use UptimeKuma to keep let us know when things go down.
@@LAWRENCESYSTEMS Lawrence, another question. How do you configure GrayLog to send you Windows logs. I am using NXLog to send the Logs from the Windows machine to GrayLog. After that I want to create a email trigger, when the local admin password is changed that event I see in graylog, but don't know how exactly to send it over to come as a Email Alert ? Can I email you ? Thanks in advance! Appreciate it in a million!
Should troubleshooting for ha proxy not include enable and analyzing logs?
Most of the issues people are having such as DNS, firewall rules, have pfsnese on the same port, ACL, and Port Forward all need to be done before there are any real logs to dive into. Also had to cut out some things or the video would be substantially longer. Maybe I will make an HAProxy Log troubleshooting video if needed.
@@LAWRENCESYSTEMS in general the pfsense haproxy should only be used internally and for external facing reverse proxies its better to be a standalone one so it doesn't pose the risk of compromising your firewall. At least from my understanding of how you should separate your network? So with internal use cases, if you're digging into logs to troubleshoot its probably something advanced that the pfsense GUI doesnt support. Might as well have a dedicated HAProxy or nginx instance at that point?
As will all things, it depends. There are valid reasons to use it inside or externally to pfsense depending on needs.
@@LAWRENCESYSTEMS being flexible is what is so great with these open-source software
Thank you for wonderfully video, i am facing issue i want to use my domain without 'www' i tried but not resolved and shows (503 Service Unavailable
No server is available to handle this request.) i need help in this with Haproxy and domain configuration, once again thank you
Hi Lawrence, i am struggling with setup of 2 pfsense one is Master and second beck up node with CARP and HA Sync. Everything is sync with backup node like firewall rules, HA proxy fronted and backend, user except ACME certificates. I have create create ACME certificate on Master after successful creation i can't see or didn't sync newly successful created cert on backup node. Both nodes have HaProxy and ACME package installed. I have tried to copy files /conf/acme to backup node but on web GUI i can't see certificate listed.
Never tried that configuration.
@@LAWRENCESYSTEMS thanks for your prompt reply! I am getting following issue while issuing ACME certificate :
acme_issuecert.log:
response='{
"type": "urn:ietf:params:acme:error:malformed",
"detail": "Unable to update challenge :: authorization must be pending",
"status": 400
}'
On Web GUI:
Service - Acme - Certificate
Getting webroot for domain='mydomainname.network' Verifying: devop-testing2wtls.r3c.network Standalone mode server Pending Pending Pending mydomainnamenetwork:Verify error:Fetching mydomainname.network/.well-known/acme-challenge/5cec_IyG_JqMoGGEy-D8iO0QJBhPhexzdLCePw9BiHg: Timeout during connect (likely firewall problem) Please check log file for more details: /tmp/acme/devop-testing2wtls/acme_issuecert.log
Hi lawrence can i access my pfsense server outside the network without port forwarding? example tunneling. because my ISP doesn't provide bridge modem or port forwarding
is there any video tutorial how to access pfsense server outiside the office?
You could use this ua-cam.com/video/ZShna7v77xc/v-deo.html
Hi, I have a pfsense with haproxy and I've experienced some times and in a random way that, when I make a change in some ACL, haproxy configuration has broken when i apply it. Apparently the configuration is applied normally, but if I check backend status, the backend servers disappear. I can see that in haproxy config file the lines are missing, but in pfsense gui I have the related entries properly configured. It happens also in backends not related to ACL modified. Do you have any ideas about this bug ? Thanks!
Nope, if you can reproduce the bug then report it to the developers.
@Lawrence Systems , just to share with you and with this channel. The problem was a random missing DNS resolution of backend hostname by pfsense resolver. PHP code doesn't catch this type of error and, in case of missing resolution, doesn't generate server line in haproxy.conf. The workaround is to configure "global resolver" in haproxy settings or, I suppose, using BIND in pfsense to replicate local zones (DNS Resolver in pfSense sometimes has a strange behaviour). We have understood this just reading pfsense PHP code with the help of one of our developers.
Hello,
I have an issue with HAproxy, the service stops each 15-20 min and I have to start it manually each time ??
any Idea?
1. Read the logs
2. Find the error
3. Google the error
4. Start testing solutions
@@LAWRENCESYSTEMS unfortunately, the system logs shows nothing related to HAproxy
Can someone help me? I wasn't able to fix the dns part. I have configured DNS Resolver and added the ip of the domain's IP for the local server but when I dig into dns on the pfsense box it does not show the local ip. It shows public IP.
Besides DNS, I have configured backend and front end the front ends passed the traffic to back end however backend gives 404 on the domain.
Thank you.
I have the same issue, I dig my domain and comes back with public IP, Tom mentioned this in the video but didnt say how to fix this. Did you get it fixed?
@@brianjensen2923 As much as I remember, I have fixed this issue by enabling DNS Resolver and adding proper records. Then it showed the local ip however after that I didn't need that configuration on the future installations...
Only used ha proxy via terminal, need to checkout this pfsense app.
Currently using nginx proxy manger for all web, ssl needs.
Thinking this is the way going forward, separate lxc container
@@TerrellR are you using Proxmox
@@magfelsic4909 That's right using Proxmox for VMs & containers. I'm not entirely sold on using pfSense for managing SSL termination.
@@TerrellRI do think the same.
I haven't started using LXC in proxmox, what things are you running on LXC.
@@magfelsic4909 Nothing at the moment. I was using an 2 Nginx LBs, and a database for a k3s Rancher, 2 server nodes, and 2 worker node deployment. Ended up tearing the whole thing down, too much maintenance and tinkering. I'm liking LXC for persistent shared services like a database
hi, nice video, I have a working pfsence + haproxy working fine, but when I download a file through https, look like the file download through http, the google chrome browser make a warnig telling the conexion is no secure, how can I do to force the download through https?
That does not sound like an HAProxy issue
I don't know why but it works only when I'm inside my network. Maybe a wrong configuration.
You are an absolute blessing my man! Hope you can help me out as I am having a world of problems trying to figure out NextCloud (running in Docker on UnRAID server) communication externally through pfSense with HAProxy. I am getting an error "400 Bad Request The plain HTTP request was sent to HTTPS port"
I have the same issue, did you solve it?
Have you ever configured a production nginx reverse proxy?
nope
Yes, it works similar. i recommend running Nginx Reverse on a Separate VM and run the Web Server itself on another VM then use ProxyPass with the IP Address
Forgot: make sure HAProxy is enabled. Just spent 2 days troubleshooting that one... doh!
Right on time I've borked my install 🥺, many times
In summary - WTFV
(and in other situations, RTFM)
I always have to laugh, Tom starts his videos saying 'Tom here from Lawrence Systems', yet he still gets called by his last name. lol....
I'm used to it 😃
@@LAWRENCESYSTEMS for giggles you should start a video "Lawrence here from Tom Systems..."
I like that idea.
I need a more simplified video cause I'm doing something wrong. Don't explain, just do x + y = z
haha, may skip some steps...
Thanks for this video. It helped find some things I had wrong, but I'm still not there. I'm using CloudFlare, and I set up the DNS records on their site, but when I run 'dig' (from Windows WSL), I get two records for an answer, and neither of them match my home IP. I don't know where to go from there. I also got lost toward the end of the video because it shows what you should see when the certificate is set up correctly, but I just got two messages: "...No route to host:../crypto/bio/b_sock2.c:110:" and "...BIO_connect:connect error:../crypto/bio/b_sock2.c:111:" The last line says "connect:errno=113".