Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun. Thanks for another great video and the idea...
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you. I have three point about your setup: - InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination - This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this) - All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot. 2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...) Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) ) Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them. I like music, but in videos like that, I hate it myself. This is why I do not use it...
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto. For the installation of Grafana and InfluxDB I found some instructions on the internet. I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files. Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on UA-cam. Your viewers might find the comparison interesting.
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Nice video Andreas! In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before). Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting. I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon. Point is that any additional knowledge nicely fits in picture. Constant evolution...
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info. GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
... Yes I am looking deep in Hass.io ... but ... before go in Strong IOT every one should read 'Abusing the Internet of Things'. For now l prefer to stay with a well lock down reporting tools.
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search. As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data. I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)? Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible. The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ). Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure. If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ... I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
"use a dedicated field to indicate the type of the value" My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch. Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter. Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it. The built-in capabilities is the other important point: does Elasticsearch support querying time series data? Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana. And all these thanks to you again :)
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB. I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
Hi! Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb. I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years! While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
Very interesting and easy to jump in. Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
I do not see that this is a topic with lots of interested people. I think you will find some information on the net. Your situation is perfect as you can play with your data and find out which policy fits your needs
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
What do you think about using the SD card long term with the amount of writing influx will be doing? Would you use an SSD if you were planning on keeping your setup long term?
Did you set your retention policy for the weather station to do hourly averages after one year? If so, how? I am seeing it is ideally done during the database creation but can be done after. Eitherway the syntax isn't so clear to me, and I like the retention policy you mention.
Thanks for suggesting the script from Peter Scargill. However the script did not install properly and the pi4 seems to be slow now. Is there any way to roll back the script? I rather install the tools one by one.
Great video thank to this i have a working weather station for a month now, i have some questions about the RETENTION POLICY, is the one you explain the default? do you know any source where i can find more about this? Great work on youtube I´m a long time fan.
I love it, now we really talk IoT. One more step Andreas, automated MQTT to influx converter. Let me know if you'd like to have the links. In my porject (google "home smart mesh"), I have such a python converter. It subscribes to a certain MQTT topic pattern, and posts jsons as is into influxdb. Everyone is already very familiar with MQTT directly produced from ESP and others, with an automated converter, you would not need to edit a custom translator for every sensor you add. Yeah and I forgot something, once MQTT is plugged to influxdb and grafana, you enpower it with something like zigbee2mqtt and have awesome cheap market products on you dashboard !
I would be interested in the links. Did you do a tutorial? So far I always thought that ZigBee is quite expensive for makers and not well accepted in our community. Maybe I am wrong...
Hello, I spent a quick 30 min digging into your “home smart mesh” with the objective of understanding just the mqtt to influx converter. I feel close to getting it but honestly got a little lost in trying to peel the mesh application out of it. In my case I have at least a dozen esp and other devices all speaking to mqtt over Wi-Fi to node red then over to RRD databases. I want to explore updating this to mqtt to influx to grafana directly. Examples: /sonoff/1/temperature/ or /base/55/humidity/ or /1wire/“serialnumber”/“englishID”/temperature/ Do I configure all these in JSON or just create rules for each generic example? And Bravo to Peter and team for that script which I have used several times on Pi and other Linux installs!!
@@AndreasSpiess for MQTT to influx with python please find a direct link to the github influx python directory here github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx I do not have a special tutorial for the MQTT to influx, but to be honest I struggle to make a documentation or tutorials as clear as you do, you are better at that Andreas :) Nevertheless I spent effort to document the whole "Home Smart Mesh" which has a lot of custom RF in it, and it is not easy to isolate the raspberry pi scripts from the rest. I tried to document on Hackaday hackaday.io/project/20388-home-smart-mesh and on Github which is more focused on the nRF52 Hardware github.com/nRFMesh/nRF52_Mesh When it comes to Zigbee, I can confidently say that it became cheap enough now with all sort of ready sensors ~< 10 € from Ali, and an official nRF52840-Dongle < 10€ from Mouser. Now I would agree with you if you say nRF52 Thread or Zigbee is not well supported by Arduino, but Andreas, we have to make it happen, by selecting best HW and pushing it to be come easy and popular. I started wrapping the NRF SDK official from Nordic around a c++ API with classes that make it Arduino like, my latest progress on that here github.com/Roblibs/rover_firmware/blob/master/applications/03_sensored_bldc/main.cpp but far from being a real arduino core. The nRF52840 is I believe a real challenger for the ESP32, as the nRF52 has native USB, an Automated Programmable Peripheral Interconnect, and with its custom radio I achieve latency ~ 1 ms. I'll be glad to share any required further info and details.
@@DavidL3430 please find a direct link to the python script that converts MQTT to influx, it should be possible to take that directory away from all the rest github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx But I admit, it is not packaged to be used without Python knowledge, as you notice, even if I have the main config in the config.json, a lot of the logic is coded in the def mqtt_on_message, for example the number of words in the topic e.g. mesh/node5/temperature and also I force some types to be float or int in lines 36 to 43. But on the good side, I even prepared scripts for backup and export, and the note that the config can be different depending on the hstname so that you have many raspberry pi and save all config in the same directory with names config_rasphostname1.json, config_rasphostname2.json, ... I will gladly provide you any further help if you want.
I suppose all the credentials should be changed to use the SD card image securely? What kind of changes should be done? Paswords for the Raspberry Pi, Node_red, influxDB and Grafana should be changed? It would be nice to have a guide how to do this and how changing the password of one app affects to another app.. in other words, how to keep the system working after passwords are changed.
can you make a video on the topic of retention strategies, it seems to me to be a little more complicated? Would you use this configuration, shown in the video, for everyday use?
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red? Thank you - great video!
A verry good video! Thanks. I all ready got influxdb and node red on my server. But i never used the influxdb node in node red befor. It would be great if your node red flows where published somewhere!!
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
I do not know if it works on the 2W. Two things to consider: 1. The memory of the 2W is very low for such applications. I would go for at least 2G 2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest? thank you in advance!
Hello i install successfully nodered, influxdb but i have problem in grafana. Fail to start grafana instance. I am using raspberry Pi zero w(raspberry pi os). Help me if possible... (I didn't use peter's script, i do it manually)
Thank you for the video, otherwise I have a question is it possible to integrate Grafana in a php page or is it an open source that can be easily integrated into a web page in your site?
Great video, as usual. But you should start to use the ip(8) command and not ifconfig(8), as that is obsolete and are going to be removed. And it is much more flexible and useful, as you don't need the arp(8), route(8) etc as that is all merged into the ip(8) command. And it is shorter to write. :-) ip -4 address show # can be shorted down to 'ip -4 a s' ip -6 address show ip route show # can be shorten down to 'ip r l' ip neighbour show # can be 'ip n s', also try 'ip -6 n s'
Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff. The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.
Another great video. By coincidence I was looking at just these products yesterday. Thank you for such a detailed but concise summary. I am left with one question though, since InfluxDB automatically timestamps data is there any way of moving already existing historical data into an InfluxDB without it all being given the current time?
Hi @Andreas, thanks for this video. Definitely got me excited to revise my home setup. Unfortunately, I'm running into a lot of issues. I had two different RPis to run PiHole and Home Assistant. I wanted to move to an SSD to store and analyze data locally (influx, grafana). I had a MQTT broker set up in HA already and was interested to migrate this to a container. Unfortunately I can't get the MQTT addon or integration working in HA. I tried to contact the HA discord support and they were very clear this is not a supported installation method. It seems to mix the supervised/Haasio and core/docker methods. So...I'm concerned about maintainability of this method. Hoping you have some thoughts on viability here. Thanks!
@@AndreasSpiess apologies, looks like I commented on the wrong video. Yes, I am using IOTstack. The Home Assistant community was clear this is not a supported way of installing HA. Seems a big downside to not be able to leverage such a well documented and supported community. Hoping it can evolve.
Seems retention policy only tells the data base what old data to delete. Achieving the reduced resolution for older data requires a lot more work and down samples it in separate place. Then it is not possible to have a single graph in graphana. Or am I missing something?
@@AndreasSpiess Indeed, this seems to be complicated, but there is a solution here pommi.nethuis.nl/collectd-influxdb-grafana-with-downsampling/ I will see where this leads..
@@AndreasSpiess Unfortunately it only works, as the author says, when the end of plotted time is always now. On top of this, it does not take too much query to kill the rpi. it runs out of RAM fast. This kills mosquitto and node red for a while.
Is it possible to use "measurements" from different Influxdb databases in the same "panel" in Grafana? I can't figure out the syntax for the FROM field in the query editor.
Hello Andreas! Thanks for the great video. I have already tried Pete's Script some time ago and am using node red for over 2 years knnow! Worked great, so i'm doing a fresh install now on the latest Raspberry and I am adding Influx DB now for the first time. Iw wanna try to store and read power consumption data that I am transmittting from an ESP8266 in my fuse box. Have you tried anything like that yet?
I have some problem with selecting appropriate choices when running Peter's script through putty ssh. What key do I have to press when I want to select/deselect the available choices?
Andreas. I downloaded your SD Card image and ran on my pi. Using your passwords as given at the end of the video, none of them work for me, in particular, the log-in credentials for the Raspian System are not accepted??. Alan
Hi Andreas, i've tried to use your SD card image on two different SD cards and i cant get any of them to boot, i know this post is two years old, is there an updated version of this i can try? Thank you for your work on the videos they are very interesting and i try to learn from them.
"We like programming, but we love result" Quote of the day. Thank you sir
You are welcome!
Even I can confirm, and I'm a developer.
this :]
Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
I agree 100%. Andreas is amazing!! Thank you so much Sir for sharing knowledge with everyone and great commitment on making amazing videos !!
We are slow here. That is probably why our time runs a little slower... I hope the content still helps you a little for your pollution project!
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
Security is not my specialty, fortunately ;-)
Amazing! I had to stop the video on two occasions I was so overwhelmed by the sheer power of this suite of FREE software! Thanks Andreas!
That is the power of UA-cam. You can use it at your pace ;-)
The timestamps of the example data are from 2019-02-06. Please choose an appropriate date range in Grafana if you want to display them
Extremely usefull video - I am very happy to be one of your Patrion-supporters! Keep it coming, Andreas!
Thank you for your support! It is always good to read that my videos are useful...
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
I transferred historical data exporting to CSV and creating the needed input strings for influxDB with Excel.
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
Maybe you also watch my videos about IOTstack...
@@AndreasSpiess I did Andreas and for the first time doing useful things with this stack. More thanks. My stuff is at rogerfrost.com. Best wishes.
Mr Spiess, my deepest gratitude for your work and for sharing great content in a video like this. I became your Patreon today to support your work!
Thank you for supporting the channel. I appreciate it!
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
I think it is not so easy if you have to do everything from the ground. So we both were happy we were able to stand on the shoulders of others ;-)
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
I mentioned the compression, but not in detail. I do not plan a video on that because it is very special. Maybe once part of another one. Who knows?
Nice to see you're making good use of "the script" Andreass.
I always like to stand on the shoulders of big men ;-) Makes me more productive! Like that, everybody can profit.
@@AndreasSpiess Hah. I do that all the time. My script enables software like Node-Red, written by people cleverer than me!
Thank you both for your wonderful work. I have a beautiful image on my Pi and it’s all working 😁🏴
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
You are welcome. And I agree that retention strategies are not easy to define these days with very cheap disk space...
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
Very fast! Did you try the docker Installation?
no I did not. I'll try to create a surveillance for 24/7 running PCs (if message does not arrive periodically - alert)@@AndreasSpiess
Interesting project!
We created a whole project with lots of useful containers (IOTstack)...
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
Maybe you can save some time now ;-)
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun.
Thanks for another great video and the idea...
There are a few other videos about weather station hacking and Pi and Docker (IOTstack)
I have no words Andreas. Astonishing. Thanks.
You are welcome!
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you.
I have three point about your setup:
- InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination
- This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this)
- All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot.
2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
Glad to read that!
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
I hope you will be successful in transferring the data from your source into InfluxDB.
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
I also like Grafana...
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
Thank You!
I love how well structured your videos are. Suits my taste as german engineer ;)
Thank you! Maybe you also watch the one about Docker. It simplifies the installation of those programs
@@AndreasSpiess Yes i did and everything is working nicely together. :)
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...)
Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) )
Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them.
I like music, but in videos like that, I hate it myself. This is why I do not use it...
Your channel is pure gold!!
Greetings from Chile!
Thank you very much!
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto.
For the installation of Grafana and InfluxDB I found some instructions on the internet.
I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files.
Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
Now you can watch the newer version (dockerized) of this setup.
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
MongoDB seems to use quite a lot of disk space because it is made for other purposes. There is a comparison on the influxDB page...
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
Thank you for your kind words!
Everything new. I had watched twice to understand. I want this setup too for my home sensors. Thank you dear.
You are welcome!
Im now successfully using grafana influxdb with mosquitto (EspMqtt) using node red on my OrangePi Lite.
I'm a programming student and I was this kind of data retention policy saves a lot of work. Glad I don't have to reinvent the wheel.
We live in a good time with all this open source software...
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on UA-cam. Your viewers might find the comparison interesting.
SQL is always a possibility. It is stable and does its job.
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Excellent!
Amzaing effort from you and Pete's , keep the videos coming Andreas 👍
Thank you!
Nice video Andreas!
In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before).
Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting.
I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon.
Point is that any additional knowledge nicely fits in picture. Constant evolution...
Other commenters also mentioned that it is easy with Home assistant. Maybe I have to try it one...
Nice work Andreas! Thank you for putting this together, very informative.
My pleasure!
Excellent demonstration of how little you need to collect and visualise data. I have a spare PINE64 which I'd like to try using these building blocks
You have to check which operating systems the script supports if you want to use it.
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
At least you would have great support for your project. Not as we "simple people" ;-)
this was an eye opener for me. always a good time learning from you
Glad to hear it!
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
So far I always used Node-Red and do not know Home Assistant.
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
Really helpful explanation of the benefits of time series databases for IoT, thank you 👍👍
Glad it was helpful!
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Maybe I have to look at hass.io once...
Awesome guide to getting Grafana up and running, thank you :)
You are welcome!
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info.
GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
... Yes I am looking deep in Hass.io ... but ...
before go in Strong IOT every one should read 'Abusing the Internet of Things'.
For now l prefer to stay with a well lock down reporting tools.
Interesting and useful video!
It seems we are heading back to remote sensing? I like that.
Sensing and acting based on these data is IOT for me ;-)
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
No, but I always have about the newest 10 videos without numbers.
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search.
As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data.
I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)?
Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible.
The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ).
Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure.
If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ...
I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
"use a dedicated field to indicate the type of the value"
My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch.
Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter.
Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it.
The built-in capabilities is the other important point: does Elasticsearch support querying time series data?
Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
Great video, Andreas. Really enjoyed this one.
Thank you!
@Andreas issues are now solved, as you know by latest comments on Pete's blog... i'm MrShark :)
Thanks! The video was already out then. I pin your comment that everyone can read it.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@@squalazzo Influxdb node still requires manual installation - tried just a couple of hours ago
What another great video, thank you Andreas! Your video inspired me for changing my implementation on how to use historical data.
I hope you will be successful!
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana.
And all these thanks to you again :)
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
It was long overdue for me as I wanted to use it for my installation.
I hope you had a good trip home!
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB.
I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
I once did a video about that. For me so far I had no bad experience after more than a year (not Grafana,SQLite)
Thank you , always something to learn every week in an easy way
You are welcome!
Thank you for your great video. These are the projects that I can learn from a lot....
You are welcome!
Hi!
Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
Happy birthday Andreas, hope you have a good one.
Yes, I had a good one. My new bike for the Danube tour arrived right that day :-)
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb.
I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years!
While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
A other viewer commented he is using python scripts instead of Node-red
github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
I did this after the video. It is easier for the selection in the dashboard. But during the take I did not know how to do it :-(
you'd better warn that grafana works on RPI3 only and not on previous raspberry boards.
I did not know that.
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
I am using it on orangepi lite.
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
Very interesting and easy to jump in.
Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance
I never thought about using Jupiter Notebook for something like that. Now I have to look into it. Tanks for the info.
You're awesome, thanks for sharing this knowledge. I've been learning a lot from you.
You are welcome!
Excellent video. Thanks.
Greetings from the galilee
Glad you liked it!
Thank you. I've been struggling for years with mysql and an inflexible php graphing library.
This is maybe a little more flexible...
Fantastic job!
Just what I was needing for my project.
Thank you!
You are welcome!
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
I do not see that this is a topic with lots of interested people. I think you will find some information on the net.
Your situation is perfect as you can play with your data and find out which policy fits your needs
Cool. Great stuff Andreas.
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
Thank you!
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
I am not a specialist in InfluxDB and my data size is small. So I never investigated.
thanks a lot for this research and experimentation. we will definitely use this for our farm.
I hope your application will add some value :-)
Thanks Andreas, did you consider to wrap this config in Docker/Docker Compose instead SD ? It would be easier to upgrade
No, not for the moment. Maybe later...
just putting up xmas lights but as soon as that is done i will be following this video and cant wait to use Mr. Scargills 'script'
Maybe you watch the „ultimate pi server“ video first.
@@AndreasSpiess LOL i was just wondering that !! good job the mrs LOVES your accent so i get away with watching geeky videos
What do you think about using the SD card long term with the amount of writing influx will be doing? Would you use an SSD if you were planning on keeping your setup long term?
No
Did you set your retention policy for the weather station to do hourly averages after one year? If so, how? I am seeing it is ideally done during the database creation but can be done after. Eitherway the syntax isn't so clear to me, and I like the retention policy you mention.
So far I do not use retention policies. The RPI4 is still fast enough.
Thanks for suggesting the script from Peter Scargill. However the script did not install properly and the pi4 seems to be slow now. Is there any way to roll back the script? I rather install the tools one by one.
Maybe you watch my newer videos about IOTstack. Much easier and faster.
Great video thank to this i have a working weather station for a month now, i have some questions about the RETENTION POLICY, is the one you explain the default? do you know any source where i can find more about this? Great work on youtube I´m a long time fan.
I never used retention policies so far. My Pi can handle the data volume up till now.
I love it, now we really talk IoT. One more step Andreas, automated MQTT to influx converter. Let me know if you'd like to have the links. In my porject (google "home smart mesh"), I have such a python converter. It subscribes to a certain MQTT topic pattern, and posts jsons as is into influxdb. Everyone is already very familiar with MQTT directly produced from ESP and others, with an automated converter, you would not need to edit a custom translator for every sensor you add.
Yeah and I forgot something, once MQTT is plugged to influxdb and grafana, you enpower it with something like zigbee2mqtt and have awesome cheap market products on you dashboard !
I would be interested in the links. Did you do a tutorial?
So far I always thought that ZigBee is quite expensive for makers and not well accepted in our community. Maybe I am wrong...
Hello,
I spent a quick 30 min digging into your “home smart mesh” with the objective of understanding just the mqtt to influx converter. I feel close to getting it but honestly got a little lost in trying to peel the mesh application out of it. In my case I have at least a dozen esp and other devices all speaking to mqtt over Wi-Fi to node red then over to RRD databases. I want to explore updating this to mqtt to influx to grafana directly. Examples: /sonoff/1/temperature/ or /base/55/humidity/ or /1wire/“serialnumber”/“englishID”/temperature/
Do I configure all these in JSON or just create rules for each generic example?
And Bravo to Peter and team for that script which I have used several times on Pi and other Linux installs!!
Haven't tried it yet, but there's a Telegraf plugin for MQTT to Influx: github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
@@AndreasSpiess for MQTT to influx with python please find a direct link to the github influx python directory here github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx
I do not have a special tutorial for the MQTT to influx, but to be honest I struggle to make a documentation or tutorials as clear as you do, you are better at that Andreas :)
Nevertheless I spent effort to document the whole "Home Smart Mesh" which has a lot of custom RF in it, and it is not easy to isolate the raspberry pi scripts from the rest.
I tried to document on Hackaday hackaday.io/project/20388-home-smart-mesh
and on Github which is more focused on the nRF52 Hardware github.com/nRFMesh/nRF52_Mesh
When it comes to Zigbee, I can confidently say that it became cheap enough now with all sort of ready sensors ~< 10 € from Ali, and an official nRF52840-Dongle < 10€ from Mouser.
Now I would agree with you if you say nRF52 Thread or Zigbee is not well supported by Arduino, but Andreas, we have to make it happen, by selecting best HW and pushing it to be come easy and popular.
I started wrapping the NRF SDK official from Nordic around a c++ API with classes that make it Arduino like, my latest progress on that here github.com/Roblibs/rover_firmware/blob/master/applications/03_sensored_bldc/main.cpp
but far from being a real arduino core.
The nRF52840 is I believe a real challenger for the ESP32, as the nRF52 has native USB, an Automated Programmable Peripheral Interconnect, and with its custom radio I achieve latency ~ 1 ms.
I'll be glad to share any required further info and details.
@@DavidL3430 please find a direct link to the python script that converts MQTT to influx, it should be possible to take that directory away from all the rest github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx
But I admit, it is not packaged to be used without Python knowledge, as you notice, even if I have the main config in the config.json, a lot of the logic is coded in the def mqtt_on_message, for example the number of words in the topic e.g. mesh/node5/temperature and also I force some types to be float or int in lines 36 to 43.
But on the good side, I even prepared scripts for backup and export, and the note that the config can be different depending on the hstname so that you have many raspberry pi and save all config in the same directory with names config_rasphostname1.json, config_rasphostname2.json, ...
I will gladly provide you any further help if you want.
I suppose all the credentials should be changed to use the SD card image securely? What kind of changes should be done? Paswords for the Raspberry Pi, Node_red, influxDB and Grafana should be changed? It would be nice to have a guide how to do this and how changing the password of one app affects to another app.. in other words, how to keep the system working after passwords are changed.
You can decide what you want to change. Because the server is behind my firewall I leave them as they are.
can you make a video on the topic of retention strategies, it seems to me to be a little more complicated?
Would you use this configuration, shown in the video, for everyday use?
I am not sure if this is stuff which interests a lot of people. But you find infos on the net.
Hi sir. Nice tutorial, I am quite newbie here. Is there a way to export the data from the dashboard trends to excel or CSV?
Yes. Grafana has an export function. Maybe you watch my newer videos about this topic?
@@AndreasSpiess Thanks mate!. Your tutorials are one of the best in this topic
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red?
Thank you - great video!
I usually do not connect sensors to the Raspberry because they have to be at different places. I use ESP32s and connect them with node-red via MQTT.
A verry good video!
Thanks. I all ready got influxdb and node red on my server. But i never used the influxdb node in node red befor. It would be great if your node red flows where published somewhere!!
They are: In the comments, you should find a link.
@@AndreasSpiess thanks! Looked a bit better now and found them.
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
I do not know if it works on the 2W. Two things to consider:
1. The memory of the 2W is very low for such applications. I would go for at least 2G
2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
Thank you for the hundreds of really useful, interesting, up to date tutorials and info, you're pure gold! I have a question regarding the pi node-red server side, i need to have it in a distant location and feed it with 12V batteries (car battery) and let it run node-red influx db and grafana. What would be a good way to convert 12V down to 5V for the pi? Would one of these cheap buck converters do? What do you suggest?
thank you in advance!
A buck converter with at least 3A should do it. There are many projects around if you search with Google
Hello i install successfully nodered, influxdb but i have problem in grafana. Fail to start grafana instance. I am using raspberry Pi zero w(raspberry pi os). Help me if possible...
(I didn't use peter's script, i do it manually)
I never tried this on a Pi zero. Maybe it does not have enough memory. start htop and check the swap file size is zero
@@AndreasSpiess update: i get it. I just uninstall and re install it by dpkg(not apt-get install). And it works well. Thanks for replying.
Thank you for the video, otherwise I have a question is it possible to integrate Grafana in a php page or is it an open source that can be easily integrated into a web page in your site?
Really inspiring ideas, thank you! I'm thinking to move my solar panel yield visualization to an influxdb and grafana stack.
Good idea!
Great work again Andreas!
Thank you!
Great video, as usual.
But you should start to use the ip(8) command and not ifconfig(8), as that is obsolete and are going to be removed. And it is much more flexible and useful, as you don't need the arp(8), route(8) etc as that is all merged into the ip(8) command. And it is shorter to write. :-)
ip -4 address show # can be shorted down to 'ip -4 a s'
ip -6 address show
ip route show # can be shorten down to 'ip r l'
ip neighbour show # can be 'ip n s', also try 'ip -6 n s'
I did not know this command. I will try it. Thanks.
Many Thanks Andreas - Well Done and hope to see more
You are welcome!
Nice for my next step, 3 monts from now, i donate when i download the sd card, Have a Nice Day and Thank,S for so much Help!
You are welcome!
Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff.
The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.
Another great video. By coincidence I was looking at just these products yesterday. Thank you for such a detailed but concise summary.
I am left with one question though, since InfluxDB automatically timestamps data is there any way of moving already existing historical data into an InfluxDB without it all being given the current time?
AFAIK you can insert also timestamp fields. At least this is what I did when I copied data from one RPi to another.
Hi @Andreas, thanks for this video. Definitely got me excited to revise my home setup. Unfortunately, I'm running into a lot of issues. I had two different RPis to run PiHole and Home Assistant. I wanted to move to an SSD to store and analyze data locally (influx, grafana). I had a MQTT broker set up in HA already and was interested to migrate this to a container. Unfortunately I can't get the MQTT addon or integration working in HA. I tried to contact the HA discord support and they were very clear this is not a supported installation method. It seems to mix the supervised/Haasio and core/docker methods. So...I'm concerned about maintainability of this method. Hoping you have some thoughts on viability here. Thanks!
Maybe you have a look at IOTstack. But I am not sure if it makes sense to mix HA with other installations, becasue you can add aeverything inside HA.
@@AndreasSpiess apologies, looks like I commented on the wrong video. Yes, I am using IOTstack. The Home Assistant community was clear this is not a supported way of installing HA. Seems a big downside to not be able to leverage such a well documented and supported community. Hoping it can evolve.
Useful and interesting as usual 😉
:-)
Seems retention policy only tells the data base what old data to delete. Achieving the reduced resolution for older data requires a lot more work and down samples it in separate place. Then it is not possible to have a single graph in graphana. Or am I missing something?
I never tried it so far. Stuff for some future work ;-)
@@AndreasSpiess Indeed, this seems to be complicated, but there is a solution here pommi.nethuis.nl/collectd-influxdb-grafana-with-downsampling/ I will see where this leads..
Thank you for the link!
@@AndreasSpiess Unfortunately it only works, as the author says, when the end of plotted time is always now. On top of this, it does not take too much query to kill the rpi. it runs out of RAM fast. This kills mosquitto and node red for a while.
issue is still open github.com/grafana/grafana/issues/4262
Hi, great project. I want to store DHT sensor data to monogodb locally, how can i do that?
I do not know, I never used Mongodb
Is it possible to use "measurements" from different Influxdb databases in the same "panel" in Grafana? I can't figure out the syntax for the FROM field in the query editor.
I only use one database for one panel. So I do not know if you can use measurements of several databases in one panel.
@@AndreasSpiess I figured it out. grafana.com/docs/grafana/latest/features/datasources/mixed/
Thank you. This is what I needed. I'll get to work right away...
Enjoy!
Hello Andreas! Thanks for the great video. I have already tried Pete's Script some time ago and am using node red for over 2 years knnow! Worked great, so i'm doing a fresh install now on the latest Raspberry and I am adding Influx DB now for the first time. Iw wanna try to store and read power consumption data that I am transmittting from an ESP8266 in my fuse box. Have you tried anything like that yet?
I made a video and an image is there for download. Today I worked on the alarming from Grafana via Telegram to the mobile.
I have some problem with selecting appropriate choices when running Peter's script through putty ssh. What key do I have to press when I want to select/deselect the available choices?
Use the space bar.
Andreas. I downloaded your SD Card image and ran on my pi. Using your passwords as given at the end of the video, none of them work for me, in particular, the log-in credentials for the Raspian System are not accepted??. Alan
Raspbian uses user pi and password raspberry.
Hi Andreas, i've tried to use your SD card image on two different SD cards and i cant get any of them to boot, i know this post is two years old, is there an updated version of this i can try? Thank you for your work on the videos they are very interesting and i try to learn from them.
I will have a look at it.
Great video again, Andreas and superb timing as I too have a project which requires this type of sensor recording/display. Have you used an IOT2020?
No, I usually do not use such "professional" components like the IOT2020