SuperHouse #41: Datalogging with MQTT, Node-RED, InfluxDB, and Grafana
Вставка
- Опубліковано 11 гру 2020
- * www.superhouse.tv/41-datalogg...
* www.superhouse.tv/episodes
Recording sensor data and then reporting it is a very important element of many automation systems. By setting up a datalogging system using Mosquitto, Node-RED, InfluxDB, and Grafana you can record data from many different sources and then show critical information and trends over time.
See the link above for a step by step guide, including all the commands so that you can easily copy and paste them.
SuperHouse:
- www.superhouse.tv
- / superhousetv
- / superhousetv
Jonathan Oxer:
- / jonoxer
- / jonoxer
Support me on GitHub Sponsors or Patreon to receive exclusive discount codes, access to exclusive sections of the SuperHouse Discord, and early access to new episodes: www.superhouse.tv/support - Наука та технологія
I was waiting for someone to make a video like this a long time ago! With exactly all this elements, thank you so much!!
This is an excellent, detailed yet understandable walk through of this whole setup. Absolutely awesome work!
Great video Jon, thanks for filling in a lot of gaps for me in the whole sensor to presentation stream.
Node Red and Grafana look like really useful tools for this application. Much appreciated.
Dear Jonathan. Thank you for making this video and making it available for us. I had been looking around for a informative video on the topic, since past few days. You have made it easy for us to understand the flow and enabled us to implement it ourselves.
I wish this video existed months ago when I was trying to figure this out. Great job! Thank you!!
Thank you this !!! This is really great. Explanations are clear, easy to follow, and the seed is great. You explained so much in a single video. Please keep posting!
Oh man, you saved my day! Formating the data with json in Nodered was exactly was i was looking for - THE WHOLE DAY! Now my Jelly switch transfers numbers and not strings anymore. Thank you SOOO MUCH!
Nice tutorial. I’m a big fan of mqtt, node red, influxdb and Grafana and wild agree that this is the way to go to support future projects. 100k tantalisingly close - well done Jon, well deserved.
Thanks Adam!
Great! You did an awesome job explaining everything in a way anyone can follow!
Excellent! Thoughtful balance of flexibility vs complexity. Can’t wait to build my own stack on my RPi. Well done, great demo!
Having played with Node-RED a few years back and all that, this was a beautiful refresher and extension adding InfluxDB and Grafana. Thanks!
This makes me want to have another go at Node-Red. Thanks for the great information.
This is by far THE BEST video on "how to use" .... all these appts !!!
Clean and simple demonstration of all the essential tools importantly with good documentation to allow me to follow it - Fantastic kick start for automation and data recording - Thanks
Great tutorial! Thank you! I was struggling with Nr and Grafana.
That is by far the best video I have seen on Node Red! Most “How to” videos are so quick that it is really hard to follow, yours is perfectly paced and beautifully explained. I now feel confident in setting up InfluxdDB and Grafana. Thank you so much
Stumbled on this video just now… It’s one of the most helpfull vids i’ve seen on this topic! My compliments and thanks! .. subscribed immediately!!
Great tutorial! I've always wondered about Node Red and now this makes sense on how to set it up. I'm probably going to do this as a container in ProxMox as am concerned about the excessive writing to the SD card on the Raspberry Pi.
*Really good and comprehensive tutorial. Thank You very much!*
Glad to have found your channel Johnathon, I really enjoyed this and hope to put it into practice.
Great work Jon!! I have seen performance issue using Nodered. Rest of the components works flawless. I had switched to telegraf and the performance was far better.
Nice walk through!
I am just starting with node red. You can set the format on the mqtt input node to JSON, so you don't need the JSON node.
I just wanted to say thank you for this very in-depth and great explanation! definitely gained a new subscriber.
Absolutely fantastic video as everyone else is saying. Fun, inspiring and useful!
this is like a one stop shop tutorial. Brilliant!
Super video... yes, long but worthwhile, educational and tells the story perfectly
Fantastic walk-through - exactly what I have been looking for. Thanks!
Thank you for that great tutorial! I am sure I can use the information directly as I am redesigning my smart home.
Great video, just what I was looking for, Thanks very much Jonathan!
Fantastic video. I wish I had know more about grafana two years ago.
I spent years creating a new concept for my company adding embedded “senors” to monitor the operation of a real-time core 5G network component and the only missing element was a presentation layer (GUI).
All the data (sensors) were in a database yet we couldn’t get funding for the final missing GUI/presentation layer.
I’m actually gutted I couldn’t complete my concept that would have revolutionised the monitoring of this critical product ☹️
Open Source is frankly amazing! 🥰
SPLENDID, JON! THANK-YOU SO MUCH!
I very much needed a detailed walk-through like this to get from zero to first-working. Trying to do this kind of first-time integration on my own is beyond painful, especially given how little spare time I have after my day-job. It takes weeks instead of hours for me to get things done starting from scratch, so not very many things get done!
My wish-list item would be for one more level of time savings: Automation to create database entries on the fly for each mqtt data item the first time it is seen, then add line charts to a dashboard to display them. I'd very much rather prune unwanted items than add the wanted ones one at a time! I've never used NodeRED, but I suspect it should be able to handle this task.
NodeRED looks to be a powerful OpenSource alternative to some of the National Instruments LabVIEW tools I've used in the past (and learned to love to hate). And I see PyNodeRED (pypi.org/project/pynodered/) will let me create and import modules written in my language of choice, Python, which I use at work for data transformation and analysis (and everything Bash won't do).
Again, this video is awesome. It feels as if it were created just for me, to greatly accelerate my learning curve and make me instantly productive. I didn't even know the questions to ask, and you answered them all. THANKS!
Next step: On to the web page for this episode!
Edit: Really, this combined with your Tasmota walk-throughs gives me a soup-to-nuts recipe for data acquisition, transmission, archive and display. Thanks!
Hi BobC, thanks for your positive comments! Being able to automate the end-to-end setup of a new data point would be fantastic. Since recording this video I've been told about a way to simplify the JSON-to-InfluxDB step, so I'm going to add that info to the tutorial page as I'm finishing it off. It doesn't automate the process of creating dashboard widgets, but at least it could save some time setting up Node-RED for each individual measurement.
Thanks for this informative video! Double thumbs up!
Excellent video/ tutorial as always Jonathan. Thank you for all the time you spend creating them. I'm just about to teach my students IOT, what a better way of backing up my teaching by pointing them at your videos... Hope life is getting easier in Sunny Australia..
That's fantastic, thanks John! I'm glad the videos are useful. Since you teach this sort of thing professionally (I'm just an amateur who makes it up as I go along) if there are specific topics or ideas that would be useful for your students, please let me know.
Thanks for the video. I heard about Node-Red, InfluxDB and Grafana combination and never explore it. This video make the process easy to understand.
This is my reference video, very well made and even suitable for beginners!
Thank you so much for putting this together!
Excellent tutorial. Brilliantly explained. You should be a teacher.
Very well explained. Thank you
Excellent video. Yesterday I added InfluxDB, Grafana and Telegram to my OpenHab + NodeRed + Zigbee2mqtt setup and I was interested to see if I missed anything. I didn't, but it this tutorial would have definitely saved me time in working out how to set it all up. I'm using node-red as the data bridge and rules engine with telegram only for alerts (and, later, maybe specific controls) - mainly because node-red is very powerful and local, whereas telegram relies on the internet. So for my water leak sensors and mains water valve cutoff system, I want that to operate simply and locally.
Super Video from Superhouse....big fan 👍👍👍
Wow. It finaly makes sense :) Succinct and clear explanations. I'm in! Thank you.
Great video, I love the considerations for re-use, the versatility of the system and the excellent choice of software.
Hi, I really like your videos and have followed them for years. I always learn a lot from you. On this one i will suggest that you use a SSD drive in stead of a SD card for the system and the database. in this way you increase the reliability of the system over time.
Bravo
You are a very good 'teacher' also for non uk native language like me.
Thanks
Very good explanation of how to setup an extensive logging and visualization platform. Thanks!
I indeed came here for informations on telegraf, but I'm also relieved that one does not have to use telegraf.
You really made a very good job, in all of your video. It's very interesting and very useful. Many thanks !
Congrats my friend! Very nice 👍🏼
Great detailed video! uUst what I needed to know.
Bravo, Got me thinking again
Pretty god damn awesome. If you didn't do code reviews as in the previous video in this series I wouldn't have been as interested. I think the format of doing code walk throughs of microcontroller projects is one of the best ways to teach your viewers what's happening! One tip in general for the code you do as a good habit to develop. Instead of calling a function again and again and again in your main loop that always returns again and again the same return call data value such as when you are doing all the sprintf function calls to get the ESP ID move the function call out of the main execution loop() and up into the one time setup section. Then you only need one single call to that function for the entire time the microcontroller project is running -- assign the return value of the single ESP ID function call in setup and assign the data value in the return call to a variable -- or the hexadecimal equivalent that you want for this particular example. Then in your main loop() just reference the variable directly to save repetitive calls which when you design all your code this way can lead to faster execution times, faster time to reach that critical next sleep call and therefore more power savings -- less critical when powered by a wall wart power supply but very critical and good habit for when you start to build battery powered microcontroller solutions where every ounce of runtime life you can squeeze out per battery charge counts for something.
Amazing video, you describe it very well! ✌️
Super cool tutorial. Pulling data from a Sensirion SCD30 into a 3 gauge cluster for CO2, RH, and temperature. I need to learn how to combine outputs on Arduino to create one json instead of three. Thanks again.
nice overview and video... now I want to build an air-quality monitor as well!
Excellent explanation 👍
Very Good presentation. Thank you for this, so useful to my application.
I love you. Thats exactly what I was looking for.
Or in German: Danke bester, genau nach dem Zeug mit deinem Vorgehen habe ich gesucht! :D
Superb video and information!!!
Great video really good explanation 👍
As always well presented!
Excellent video ! Great Job !
Really helpful and easy to follow tutorial!!! 😃😃
Excellent content and well presented
Excellent video.
Thank you...
It’s very useful information
Well done, Very good video!
Wonderful, sir!
Much appreciated. Thanks.
I did this. Excellent video mate. After all, I needed to know my beer fridg ewas always at the right temp.HeHe. Not joking about that either. Grafana says my beer is -0.69C right this second.
Thank you for your video.
Very good explanation for the combined software projects. It helped me a lot.
Thanx for your work!!!
You can also take the json object and send to influx. If you do that you dont have to split each value Up. And if your sensors follow a Common system in terms of json data vs topic then you can have 1 single Line that takes all data from your sensors to the influx. Then you dont have to care when you add more sensors :) i do that for both tasmota, My own sensors and others. The data ends Up tagged and with proper series names IF done properly :)
That's a fantastic tip, thanks!
@@SuperHouseTV also use tags/Keys in influx. Makes it easier to Group and show data without having to do queries for each sensor but instead use filters :) i have way more than 1000 data sources above is a must :) you could argue to send the data to influx directly or use telegraf but i like having all My data distributed via mqtt :) since i also use alot of the data for automations where they dont exist in ha. (I have used node red longer than ha)
awesome video !
Exactly what I was after and a great pace and level thank you. My next task is how to store pairs of JSON data in influx (lat and long) to track and chart stuff - wish me luck
Great video!!!!! Thanks!!!!
Got it working very nice instructions. The only issue is that when you share Grafana, any random user can edit the Grafana dashboard. I could share just the plots (Kiosk mode), but when you hit Esc you go back to dashboard.
Great work. Thanks.
Well done 👍
This is outstanding
Great tutorial!
Hi, I came across your channel today. I've to ask a favor: in which video do you explain the json setup for your sensor? I like to do that for my BME280. THX for that super video. Now I now where and how my weather station project goes.
Nice video! For step 7, is there a way to automatically split all the JSON items and store each one in the DB without having to manually copy/paste/change each entry?
brilliant thank you.
Very nice video...
Wery useful video, thank you Jon!
I would like to noticed: if someone wanna use the authentication from the version of mosquitto 1.5 , then need to write an additional row into the mosquitto.conf:
per_listener_settings true
If this row is missing, than the authentication will not work (actually without password will work the mosquitto_sub example (at 4:13).
thank you so much !!
Very nice video. Thanks. It's exactly like my setup. I hope there will be a sequel.
I have been trying for a long time to figure out how to handle long-term data. For example, when I want to draw an annual graph, Raspberry is weak, it takes a very long time and it heats up a lot. So it is necessary to downsample the data. But this downsampled data goes into new metrics, and that means a new graph for long-term data. That seems silly to me.
Have you looked into InfluxDB retention policies? As part of the retention policy you average older data points so to minimise space at the expenses of accuracy
Yes, retention policies are something I should cover in a follow-up. It's very important on any datalogging system, but particularly on low-end devices like a Pi.
Hi thanks a lot for the video. Also what model is your air quality sensor please
Great video. What Air Quality Sensor are you using? I'm looking for something similar with a MQTT output
Nice video! I actually did connect home assistant to influxdb but node red is also a very good option!
The only problem I've come across is that after a certain period of time tje data in influxdb gets corrupted. Long story short: Rpi OS actually works natively in 32bits while influxdb is actually 100% 64bits software. Since then I actually use Ubuntu Server instead of rpi os, which does work in 64bits.
I also did make a docker-compose file so that I can easily recreate the stack whenever I need, "the guy with a swiss accent" (andreas spiess) has some pretty good videos on that.
Good points, thanks. I personally run these software components in Docker on a Synology NAS, but for this tutorial I wanted to provide a basic step-by-step guide that would show a system that didn't introduce more things to explain. I'd like to do another guide which is showing the same 4 software components, but installed using Docker instead and using external mount points to retain data.
Great ! ... Thank you.
Hi Jonathan thanks for this great tutorial .I've achieved to send the data from my Higrow sensor to Influx and Grafana.Now I need your help .How can we send the data to a sensor in Homeassistant and see it in Lovelace.
Thank you for sharing . I am working on a project related the internet of farmingbased on what we call a '' Plant Speaking Approach" and I find your video as great plus . I am using (influx db and grafana) home assistant.
As I just bought a dragino gateway with LoRa. Definitely , I will get in touch with you for more advises . I have a question regarding the storage capacity.Can we use an external storage to store the data as I am looking to use for statistical modeling in the future.
Thank you once again for sharing and believing that the open source is great way to shape the world in a sustainable way.
Thanks LiveArt! For long term storage a very important thing to set up is InfluxDB data retention policies. By default, InfluxDB just keeps everything you throw at it. Over time the amount of data can become unworkable. Retention policies allow you to specify things like averaging data across periods (effectively "thinning it out") so that you have detailed data for recent events, and for older periods of time you have less detail. For example, you might have 1 minute samples for the last month, 5 minute samples from 1 month to 12 months, and then hourly samples beyond 12 months. This prevents your storage getting out of control and keeps queries running fast.
@@SuperHouseTV Thank you for your reply and I appreciate your effort of sharing and spreading knowledge . Thank you once again. Happy Holidays to you and family .Stay safe.Cheers.
You have a link for the mods for the ikea tables for that enclosure? :)
Hi, excellent video 👍
One question:
If I wanted to add actuators, what would I have to add to that architecture?
What software or technology would I be missing?
I would appreciate your answer.
👍🇵🇪
Very interesting, loved it. Can you make a graph using take data from a raspberry pi (ex. temperature, CPU load, ram uses, and so on) in Grafana
Excellent video! Thanks a lot! I'm actually working on something just like this...but I'm getting an error with influxdb (i guess). After installing infkuxdb and adding some sensors to store data (from PLC) the Raspberry Pi disconnects from internet and stops storing data.
Thanks a lot
Excellents video. Quick question, what would be the procedure if I wanted to access Grafana from a website instead of locally?
Danke!
awesome!