Thanks for that. Very useful. Would love to see more on energy plotting (for example stacking tariffs in kWh column graphs and calculating costs and so) if you get around to it. Good luck to all you guys in Melbourne with the lock downs you are experiencing down there. Take care and thanks for the videos.
I broke my InfluxDB and Grafana recently so I'll take a look and see if I can get that added to the graphing. It's a bit harder in my circumstances as my Tariff changes based on the wholesale price (I'm with Amber Electric). While my retailer has an API to pull the pricing data to assist calculation, the HACS plugin that worked with it recently broke and I've had no time to troubleshoot/investigate.
Thanks a lot Stuey, i really soak up your videos and by the way your presentation mode is getting better and better. Nice exaple to use InfluxDB and GRAFANA in a meaningful way.
G'day mate, just starting to get back into home assistant after my raspberry pi died 5 years ago. Planning on using this so that i use all the solar i produce and minimal goes back to the grid.
Hahaha, I forgot about that line. I recently created a new Dashboard to track my cryptocurrency portfolio in Grafana. It was lots of fun figuring out how to push data into InfluxDB.
Hi really enjoyed your video. Are you able to help me. I want to extract data for a certain time range ie from 21:00 to 24:00 every day but I am having trouble creating the query in Grafana. Can you help me with the syntax to get that time range
It's probably not ideal to handle this as part of your import query. The benefit to Grafana is the flexibility of the display. Once you've got your query setup to pull all the data for the value you want to graph, you can easily then choose to visualise only the required time period when you need to using the time-range in the top-right corner.
Works great after some coding fails by myself. How can I add grid +solar production to display current net household usage in the graph? If I'm producing 6kw(shown as +6000) and exporting 5kw(shown as -5000) I should have a +1000 usage Thank you.
You can do math inside the Grafana Dashboards, so you should be able to plot a line that is something along the lines of Solar Production - Energy Exports. Not knowing anything about your specific setup and the sensors you're passing through to Grafana, I can't give you the exact config but I'm sure the documentation for Grafana can help you to answer that question.
Very cool! Thanks for Sharing. I'll definitely give this a try after I get my solar system installed early next month. I bit the bullet and went for the Fronius smart meter, so I believe I'll be able to do some interesting things like you have. By the way, I reckon you can bring that dashboard into Lovelace by creating a new view, adding a webpage card, and setting the view to Panel mode. That way, you have the Grafana dashboard within HA itself! :)
@@HiveMindAutomation Will share a video once I have everything setup. Do you think my RaspberryPi 3B+ server will be able to handle InfluxDB and Grafana? My home automation system is relatively simple at the moment with 20 or so devices, but the memory usage is always around 45% and CPU 3%. I do have Node-RED running for all my automations. Just wondering what sort of overhead you think it will add, and if you think it will impact overall performance. Thanks!
Honestly the biggest issue with using a Pi is the I/O speed and longevity of the SD Cards. If you can avoid putting extra load on the Pi3, I'd do so. In my production environment, I have both InfluxDB and Grafana running on my Docker Host, which doubles as my File Server. InfluxDB uses ~ 300MB RAM at idle. Grafana seems to only use resources when you're looking at visualisations. Bear in mind Pi 3 only has 1GB RAM. CPU won't be a problem but RAM might be, and InfluxDB can write a fair bit of data too. Frequent Writes kill SD Cards quicker. If you're booting from a USB Flash Drive/SSD, your writes wont be a problem, but keep an eye on RAM usage. Of course, your environment may not be tracking many sensors and may not require 300MB RAM usage, so you might be okay.
@@HiveMindAutomation Thanks for that. Agree on the SD cards. After having reliability issues with them, I decided to swap over to an SSD connected to the Pi3 via USB. That significantly improved performance. If InfluxDB uses 300MB of RAM, I guess I better hold off until I move everything to a Pi4 - maybe get the 8GB version like you have. Thanks again!
Thanks for the video. How do you manage the amount of data stored? Is there some way of offloading some of the data like on a yearly basis or something to keep from filling the hard drive?
You can set the Influx Database up with a retention policy to flush data once it's past a certain age. Mine is normally set to indefinite, however if your storage is limited, you're better off offloading some of it
@@HiveMindAutomation Thanks for the reply. I do have the default retention policy set and have changed the granularity of data to something less data hungry. That will work for some time. I am wondering though how to offload the data when the time comes. Right now, i plan to manually download CSV data via Grafana every 6 months and have the database trimmed down automatically at 56 week intervals by setting the RP to 56 weeks. I guess I was wondering if there was a better way to do it within InfluxDb. I don't think additional RPs will work once the data has already been collected for some time. i.e. the RPs and Continuous Queries should be setup initially for the aggregate capabilities of InfluxDB to work properly. Thanks again...
With Grafana you can change the viweing period in the top right corner, I've not figured a way to do it interactively like scrubbnig through the data. the refresh period is configurable down to every 5 seconds, with the polling interval of most of the sensors, it's unlikely that's really required.
Thanks for explaining more about grafana, makes working with it a lot easier and helpful as I'm a visual learner. With your daily energy and stuff, did you have some type of calculation for that or is that an entity you can pull from fronius? Thanks!
Hi Stuey. Great videos! Really enjoying them. I’ve followed this Grafana set up. One question. What is providing the sun elevation data? Is this from your inverter? My inverter doesn’t integrate so I just use Shelly meters to track incoming from inverter and main feed to the grid. I was hoping I could use the default HA Sun Integration for elevation but doesn’t seem to populate in Grafana. Any thoughts? Thanks Owen
You can pull it from the home assistant Sun integration but you need to create a template sensor for it. Check out my Template Sensors video. I don’t cover it explicitly in that vid, but you’ll see it as one of the sensors I’ve created. You can also check out the home assistant template sensors documentation and it’s one of the examples given.
Amazing video Stuey! Thanks a lot. I am stuck though... After creating a database on InfluxDB and restarting Home Assistant, I tried to visualize the measurements on the InfluxDB "explore" panel and i get an error message saying that "Your query or command is syntactically correct but returned no results. Would you know how to get this fixed? Thanks a lot!!!
You'll want to check your Home Assistant Influx DB Configuration as it seems like your data isn't passing through to it yet. If you run the 'SHOW MEASUREMENTS on "home_assistant" ' query and it doesn't produce a result, your Home Assistant hasn't pushed any data into Influx just yet.
I had the same problem. Ended up adding these two lines to the configuration.yaml in addition to the other influxDB configuration details: ssl: false verify_ssl: false After that the database connected to HA and everything worked.
Thanks for all the very detailed and informative videos, I have learnt a lot for a total noob. I know this is an old video but with the latest version of HA I have got this working partially well, but the struggle is trying to get the sun angle part working, I have followed this video to create a template ( ua-cam.com/video/Zd7tUchYjM0/v-deo.html) I have got the script in config.yaml all working but when I go to setup the overlay panel for the sun angle, I can select FROM>default>state>WHERE>entity_id=sun or sunrise nothing displays. if I select FROM>default>o>WHERE>entity_id= there is no selection for sun or sunrise, would you please be able to assist? Thanks in advance
I've moved onto a newer version of Grafana and InfluxDB, so I'll be doing a new video about this topic in the (hopefully) not too distant future. I vaguely recall having to setup a Template Sensor for the Sun Angle to make this work so you might benefit from ua-cam.com/video/Zd7tUchYjM0/v-deo.html
That's what I needed. Great display of the setup and different options!
Glad it helped. Thanks for watching.
I probably need to re-do this cause there's new ways to do it.
Thanks for that. Very useful. Would love to see more on energy plotting (for example stacking tariffs in kWh column graphs and calculating costs and so) if you get around to it. Good luck to all you guys in Melbourne with the lock downs you are experiencing down there. Take care and thanks for the videos.
I broke my InfluxDB and Grafana recently so I'll take a look and see if I can get that added to the graphing.
It's a bit harder in my circumstances as my Tariff changes based on the wholesale price (I'm with Amber Electric).
While my retailer has an API to pull the pricing data to assist calculation, the HACS plugin that worked with it recently broke and I've had no time to troubleshoot/investigate.
Nice visualisation of information. I also love the timestamped video, will make it easy to refer back to.
The time stamps were easier to add than I’d initially thought, still refining the workflows around all these enhancements.
Thanks a lot Stuey, i really soak up your videos and by the way your presentation mode is getting better and better. Nice exaple to use InfluxDB and GRAFANA in a meaningful way.
Thanks for the feedback. I'm slowly getting more comfortable on Camera but I know I've got a long way to go.
Glad you enjoyed the video!
Brilliant! Thank you
Glad it helped! I've noticed some significant changes in newer versions of InfluxDB so I might need to revisit this video
G'day mate, just starting to get back into home assistant after my raspberry pi died 5 years ago. Planning on using this so that i use all the solar i produce and minimal goes back to the grid.
There's a bunch of new stuff in Home Assistant now with the Energy Dashboard as well that might be helpful.
@HiveMindAutomation are you in SE suburbs of Vic?
Thanks for this, very inspiring. And also, as you said, a great way to proof my nerdyness to my friends ;)
Hahaha, I forgot about that line. I recently created a new Dashboard to track my cryptocurrency portfolio in Grafana. It was lots of fun figuring out how to push data into InfluxDB.
@@HiveMindAutomation I was just thinking about that yesterday. Especially in this bull run that been going on ;)
Good job ! Thank you !
Thanks for watching! Glad you liked it
Hi really enjoyed your video. Are you able to help me. I want to extract data for a certain time range ie from 21:00 to 24:00 every day but I am having trouble creating the query in Grafana. Can you help me with the syntax to get that time range
It's probably not ideal to handle this as part of your import query.
The benefit to Grafana is the flexibility of the display.
Once you've got your query setup to pull all the data for the value you want to graph, you can easily then choose to visualise only the required time period when you need to using the time-range in the top-right corner.
Works great after some coding fails by myself.
How can I add grid +solar production to display current net household usage in the graph?
If I'm producing 6kw(shown as +6000) and exporting 5kw(shown as -5000) I should have a +1000 usage
Thank you.
You can do math inside the Grafana Dashboards, so you should be able to plot a line that is something along the lines of Solar Production - Energy Exports.
Not knowing anything about your specific setup and the sensors you're passing through to Grafana, I can't give you the exact config but I'm sure the documentation for Grafana can help you to answer that question.
Very cool! Thanks for Sharing. I'll definitely give this a try after I get my solar system installed early next month. I bit the bullet and went for the Fronius smart meter, so I believe I'll be able to do some interesting things like you have.
By the way, I reckon you can bring that dashboard into Lovelace by creating a new view, adding a webpage card, and setting the view to Panel mode. That way, you have the Grafana dashboard within HA itself! :)
Good Idea, I'll be interested to see how your Dashboard looks when you get your Solar installed.
@@HiveMindAutomation Will share a video once I have everything setup.
Do you think my RaspberryPi 3B+ server will be able to handle InfluxDB and Grafana? My home automation system is relatively simple at the moment with 20 or so devices, but the memory usage is always around 45% and CPU 3%. I do have Node-RED running for all my automations. Just wondering what sort of overhead you think it will add, and if you think it will impact overall performance. Thanks!
Honestly the biggest issue with using a Pi is the I/O speed and longevity of the SD Cards.
If you can avoid putting extra load on the Pi3, I'd do so.
In my production environment, I have both InfluxDB and Grafana running on my Docker Host, which doubles as my File Server. InfluxDB uses ~ 300MB RAM at idle. Grafana seems to only use resources when you're looking at visualisations.
Bear in mind Pi 3 only has 1GB RAM.
CPU won't be a problem but RAM might be, and InfluxDB can write a fair bit of data too. Frequent Writes kill SD Cards quicker. If you're booting from a USB Flash Drive/SSD, your writes wont be a problem, but keep an eye on RAM usage.
Of course, your environment may not be tracking many sensors and may not require 300MB RAM usage, so you might be okay.
@@HiveMindAutomation Thanks for that. Agree on the SD cards. After having reliability issues with them, I decided to swap over to an SSD connected to the Pi3 via USB. That significantly improved performance.
If InfluxDB uses 300MB of RAM, I guess I better hold off until I move everything to a Pi4 - maybe get the 8GB version like you have.
Thanks again!
Thanks for the video. How do you manage the amount of data stored? Is there some way of offloading some of the data like on a yearly basis or something to keep from filling the hard drive?
You can set the Influx Database up with a retention policy to flush data once it's past a certain age. Mine is normally set to indefinite, however if your storage is limited, you're better off offloading some of it
@@HiveMindAutomation Thanks for the reply. I do have the default retention policy set and have changed the granularity of data to something less data hungry. That will work for some time. I am wondering though how to offload the data when the time comes. Right now, i plan to manually download CSV data via Grafana every 6 months and have the database trimmed down automatically at 56 week intervals by setting the RP to 56 weeks. I guess I was wondering if there was a better way to do it within InfluxDb. I don't think additional RPs will work once the data has already been collected for some time. i.e. the RPs and Continuous Queries should be setup initially for the aggregate capabilities of InfluxDB to work properly. Thanks again...
Is it possible to change the viewing period directly from the dashboard? In this case, change the graphics interactively in real time.
With Grafana you can change the viweing period in the top right corner, I've not figured a way to do it interactively like scrubbnig through the data.
the refresh period is configurable down to every 5 seconds, with the polling interval of most of the sensors, it's unlikely that's really required.
Got it. Thanks for the answer.
Great tutorial! Thanks!
Glad you enjoyed it!
Thanks for explaining more about grafana, makes working with it a lot easier and helpful as I'm a visual learner.
With your daily energy and stuff, did you have some type of calculation for that or is that an entity you can pull from fronius?
Thanks!
The Fronius inverter has a Sensor for Daily Energy, Yearly Energy, Total Energy etc.
Hi Stuey.
Great videos! Really enjoying them. I’ve followed this Grafana set up. One question. What is providing the sun elevation data? Is this from your inverter? My inverter doesn’t integrate so I just use Shelly meters to track incoming from inverter and main feed to the grid. I was hoping I could use the default HA Sun Integration for elevation but doesn’t seem to populate in Grafana. Any thoughts? Thanks Owen
You can pull it from the home assistant Sun integration but you need to create a template sensor for it. Check out my Template Sensors video. I don’t cover it explicitly in that vid, but you’ll see it as one of the sensors I’ve created. You can also check out the home assistant template sensors documentation and it’s one of the examples given.
@@HiveMindAutomation Great! That did the trick. Hadn’t got to that video yet.
Thanks. Owen
Amazing video Stuey! Thanks a lot. I am stuck though... After creating a database on InfluxDB and restarting Home Assistant, I tried to visualize the measurements on the InfluxDB "explore" panel and i get an error message saying that "Your query or command is syntactically correct but returned no results. Would you know how to get this fixed? Thanks a lot!!!
You'll want to check your Home Assistant Influx DB Configuration as it seems like your data isn't passing through to it yet.
If you run the 'SHOW MEASUREMENTS on "home_assistant" ' query and it doesn't produce a result, your Home Assistant hasn't pushed any data into Influx just yet.
I had the same problem. Ended up adding these two lines to the configuration.yaml in addition to the other influxDB configuration details:
ssl: false
verify_ssl: false
After that the database connected to HA and everything worked.
Thanks for all the very detailed and informative videos, I have learnt a lot for a total noob. I know this is an old video but with the latest version of HA I have got this working partially well, but the struggle is trying to get the sun angle part working, I have followed this video to create a template ( ua-cam.com/video/Zd7tUchYjM0/v-deo.html) I have got the script in config.yaml all working but when I go to setup the overlay panel for the sun angle, I can select FROM>default>state>WHERE>entity_id=sun or sunrise nothing displays. if I select FROM>default>o>WHERE>entity_id= there is no selection for sun or sunrise, would you please be able to assist? Thanks in advance
I've moved onto a newer version of Grafana and InfluxDB, so I'll be doing a new video about this topic in the (hopefully) not too distant future.
I vaguely recall having to setup a Template Sensor for the Sun Angle to make this work so you might benefit from ua-cam.com/video/Zd7tUchYjM0/v-deo.html