Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you. I have three point about your setup: - InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination - This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this) - All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot. 2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun. Thanks for another great video and the idea...
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto. For the installation of Grafana and InfluxDB I found some instructions on the internet. I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files. Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on UA-cam. Your viewers might find the comparison interesting.
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...) Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) ) Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them. I like music, but in videos like that, I hate it myself. This is why I do not use it...
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
Nice video Andreas! In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before). Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting. I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon. Point is that any additional knowledge nicely fits in picture. Constant evolution...
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info. GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
... Yes I am looking deep in Hass.io ... but ... before go in Strong IOT every one should read 'Abusing the Internet of Things'. For now l prefer to stay with a well lock down reporting tools.
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana. And all these thanks to you again :)
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search. As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data. I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)? Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible. The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ). Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure. If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ... I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
"use a dedicated field to indicate the type of the value" My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch. Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter. Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it. The built-in capabilities is the other important point: does Elasticsearch support querying time series data? Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB. I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
I love it, now we really talk IoT. One more step Andreas, automated MQTT to influx converter. Let me know if you'd like to have the links. In my porject (google "home smart mesh"), I have such a python converter. It subscribes to a certain MQTT topic pattern, and posts jsons as is into influxdb. Everyone is already very familiar with MQTT directly produced from ESP and others, with an automated converter, you would not need to edit a custom translator for every sensor you add. Yeah and I forgot something, once MQTT is plugged to influxdb and grafana, you enpower it with something like zigbee2mqtt and have awesome cheap market products on you dashboard !
I would be interested in the links. Did you do a tutorial? So far I always thought that ZigBee is quite expensive for makers and not well accepted in our community. Maybe I am wrong...
Hello, I spent a quick 30 min digging into your “home smart mesh” with the objective of understanding just the mqtt to influx converter. I feel close to getting it but honestly got a little lost in trying to peel the mesh application out of it. In my case I have at least a dozen esp and other devices all speaking to mqtt over Wi-Fi to node red then over to RRD databases. I want to explore updating this to mqtt to influx to grafana directly. Examples: /sonoff/1/temperature/ or /base/55/humidity/ or /1wire/“serialnumber”/“englishID”/temperature/ Do I configure all these in JSON or just create rules for each generic example? And Bravo to Peter and team for that script which I have used several times on Pi and other Linux installs!!
@@AndreasSpiess for MQTT to influx with python please find a direct link to the github influx python directory here github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx I do not have a special tutorial for the MQTT to influx, but to be honest I struggle to make a documentation or tutorials as clear as you do, you are better at that Andreas :) Nevertheless I spent effort to document the whole "Home Smart Mesh" which has a lot of custom RF in it, and it is not easy to isolate the raspberry pi scripts from the rest. I tried to document on Hackaday hackaday.io/project/20388-home-smart-mesh and on Github which is more focused on the nRF52 Hardware github.com/nRFMesh/nRF52_Mesh When it comes to Zigbee, I can confidently say that it became cheap enough now with all sort of ready sensors ~< 10 € from Ali, and an official nRF52840-Dongle < 10€ from Mouser. Now I would agree with you if you say nRF52 Thread or Zigbee is not well supported by Arduino, but Andreas, we have to make it happen, by selecting best HW and pushing it to be come easy and popular. I started wrapping the NRF SDK official from Nordic around a c++ API with classes that make it Arduino like, my latest progress on that here github.com/Roblibs/rover_firmware/blob/master/applications/03_sensored_bldc/main.cpp but far from being a real arduino core. The nRF52840 is I believe a real challenger for the ESP32, as the nRF52 has native USB, an Automated Programmable Peripheral Interconnect, and with its custom radio I achieve latency ~ 1 ms. I'll be glad to share any required further info and details.
@@DavidL3430 please find a direct link to the python script that converts MQTT to influx, it should be possible to take that directory away from all the rest github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx But I admit, it is not packaged to be used without Python knowledge, as you notice, even if I have the main config in the config.json, a lot of the logic is coded in the def mqtt_on_message, for example the number of words in the topic e.g. mesh/node5/temperature and also I force some types to be float or int in lines 36 to 43. But on the good side, I even prepared scripts for backup and export, and the note that the config can be different depending on the hstname so that you have many raspberry pi and save all config in the same directory with names config_rasphostname1.json, config_rasphostname2.json, ... I will gladly provide you any further help if you want.
Very interesting and easy to jump in. Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance
help please im very new to all of this but willing to learn . Im following video but when i try to run command at 2:38 im getting a message "awaiting response...404 not found " . ive copied the text exactly (several times). like i said im very new and probably should start on something simpler but this does look very interesting. fantastic presentation though ust my lack of understanding holding me back. regards paul
Hi! Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb. I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years! While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
7:30 that function call what, i now collecting data every second and i need to shrink it but i dunno what these function call, and do this function available on phpMySql ?
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
I do not see that this is a topic with lots of interested people. I think you will find some information on the net. Your situation is perfect as you can play with your data and find out which policy fits your needs
Playing with influx and grafana for computer (cpu ram ssd-cache and disk usage) and backup status monitoring. Got it working with docker containers after a few hours tinkering. Did not see the need for node red, though. Using curl for posting, and trying to get raspberry+Screenly working for display.
I like node-red because I know it ;-) there are many other possibilities to bridge from MQTT into influxDB. Did you try docker to use GPIOs or Serial on the RPi?
@@AndreasSpiess I used docker for influx and grafana, not for the data collector /sensor program. You may be able to use gpio and serial inside a container by usingal a privileged container and mounting /sys and /dev/tty*** as volumes on it.
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
Did you set your retention policy for the weather station to do hourly averages after one year? If so, how? I am seeing it is ideally done during the database creation but can be done after. Eitherway the syntax isn't so clear to me, and I like the retention policy you mention.
I have some problem with selecting appropriate choices when running Peter's script through putty ssh. What key do I have to press when I want to select/deselect the available choices?
Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff. The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.
Is it possible to use "measurements" from different Influxdb databases in the same "panel" in Grafana? I can't figure out the syntax for the FROM field in the query editor.
Andreas, can you share in some way settings of nodes that you use in node-red. I try to use image you shared, but my PI is 4b and probably this is why it doesn't boot with image you shared load in my SD card.
can you make a video on the topic of retention strategies, it seems to me to be a little more complicated? Would you use this configuration, shown in the video, for everyday use?
What do you think about using the SD card long term with the amount of writing influx will be doing? Would you use an SSD if you were planning on keeping your setup long term?
Thanks for suggesting the script from Peter Scargill. However the script did not install properly and the pi4 seems to be slow now. Is there any way to roll back the script? I rather install the tools one by one.
Great video, as usual. But you should start to use the ip(8) command and not ifconfig(8), as that is obsolete and are going to be removed. And it is much more flexible and useful, as you don't need the arp(8), route(8) etc as that is all merged into the ip(8) command. And it is shorter to write. :-) ip -4 address show # can be shorted down to 'ip -4 a s' ip -6 address show ip route show # can be shorten down to 'ip r l' ip neighbour show # can be 'ip n s', also try 'ip -6 n s'
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
I do not know if it works on the 2W. Two things to consider: 1. The memory of the 2W is very low for such applications. I would go for at least 2G 2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red? Thank you - great video!
Hi Andreas, i've tried to use your SD card image on two different SD cards and i cant get any of them to boot, i know this post is two years old, is there an updated version of this i can try? Thank you for your work on the videos they are very interesting and i try to learn from them.
Seems retention policy only tells the data base what old data to delete. Achieving the reduced resolution for older data requires a lot more work and down samples it in separate place. Then it is not possible to have a single graph in graphana. Or am I missing something?
@@AndreasSpiess Indeed, this seems to be complicated, but there is a solution here pommi.nethuis.nl/collectd-influxdb-grafana-with-downsampling/ I will see where this leads..
@@AndreasSpiess Unfortunately it only works, as the author says, when the end of plotted time is always now. On top of this, it does not take too much query to kill the rpi. it runs out of RAM fast. This kills mosquitto and node red for a while.
"We like programming, but we love result" Quote of the day. Thank you sir
You are welcome!
Even I can confirm, and I'm a developer.
this :]
Andreas - another perfectly formed video - in every aspect. We appear to run parallel lives as I was working on this for my open air pollution monitoring project. There must be more than 24 hours in the Swiss day as I can't for the life of me see how you manage to produce a) so much high quality material, b) over such a diverse range of subjects c) in such depth in such a short space of time and d) in a Swiss accent!!!).
I agree 100%. Andreas is amazing!! Thank you so much Sir for sharing knowledge with everyone and great commitment on making amazing videos !!
We are slow here. That is probably why our time runs a little slower... I hope the content still helps you a little for your pollution project!
@@AndreasSpiess Yes, it will. The issue is the simplest way of amassing JSON over https from many disparate sources. Not sure how to organise the secure apis.
Security is not my specialty, fortunately ;-)
Amazing! I had to stop the video on two occasions I was so overwhelmed by the sheer power of this suite of FREE software! Thanks Andreas!
That is the power of UA-cam. You can use it at your pace ;-)
Nice to see you're making good use of "the script" Andreass.
I always like to stand on the shoulders of big men ;-) Makes me more productive! Like that, everybody can profit.
@@AndreasSpiess Hah. I do that all the time. My script enables software like Node-Red, written by people cleverer than me!
Thank you both for your wonderful work. I have a beautiful image on my Pi and it’s all working 😁🏴
The timestamps of the example data are from 2019-02-06. Please choose an appropriate date range in Grafana if you want to display them
Well done you. This, #255, is my 'eureka' tutorial. Thank you for explaining why AS WELL AS how to. I've read the rest of the Internet and really should have started here. Sensors have been my professional 'thing' for decades and I can now see a better way to process and examine the data. Thank you.
Maybe you also watch my videos about IOTstack...
@@AndreasSpiess I did Andreas and for the first time doing useful things with this stack. More thanks. My stuff is at rogerfrost.com. Best wishes.
This video just gave me the push to convert my smart energy meter and heating data into Grafana and Influxdb. Until now I was using Highcharts and the Dashboard of Node-RED which is so difficult to setup. Now I can add historical data. Great video, thanks Andreas!
I transferred historical data exporting to CSV and creating the needed input strings for influxDB with Excel.
This set of tools is very flexible, I am using it myself and I was amassed at how easy it was to add a new device and process it's data once the whole system is configured. In my case I didn't have to go through the setup process because I just used a ready made raspberry pi image from the BigClown IoT bundle.
I think it is not so easy if you have to do everything from the ground. So we both were happy we were able to stand on the shoulders of others ;-)
Super useful! Displaying data in a meaningful & useful way is always a great challenge between providing what is of interest to a range of user interests & storing what is needed to create this output in such a way that it is easily accessed & does not need a lot of storage. There were several great ideas within this both as shown with this software or for writing ones own software on different hardware. Thank you for sharing!
You are welcome. And I agree that retention strategies are not easy to define these days with very cheap disk space...
I took the newest RasPi-4 & OS Buster, used your selections @2:53 and the latest version of peter's script. It took 17 minutes for a full installation. Thanks for this video
Very fast! Did you try the docker Installation?
no I did not. I'll try to create a surveillance for 24/7 running PCs (if message does not arrive periodically - alert)@@AndreasSpiess
Interesting project!
We created a whole project with lots of useful containers (IOTstack)...
Just to remind that "Retention Policies" in InfluxDB comes connected with the concept of "Continuous Queries". Maybe in a future video you can explain that. Great video as always. Thanks!
I mentioned the compression, but not in detail. I do not plan a video on that because it is very special. Maybe once part of another one. Who knows?
Extremely usefull video - I am very happy to be one of your Patrion-supporters! Keep it coming, Andreas!
Thank you for your support! It is always good to read that my videos are useful...
Mr Spiess, my deepest gratitude for your work and for sharing great content in a video like this. I became your Patreon today to support your work!
Thank you for supporting the channel. I appreciate it!
I have no words Andreas. Astonishing. Thanks.
You are welcome!
I've been agonizing over my heating system lately (I have been using a homebrew controller for my parents house heating unit for over 10 years now). Now I can just copy your image and dump my logging data in there while using a esp32 to control all the relais and sensors. Thank you so much!
I hope you will be successful in transferring the data from your source into InfluxDB.
I swear, you are a Swiss mind reader or something! I was sitting here, building my Node Red + Grafana + InfluxDB setup and your video got uploaded. O_O
Maybe you can save some time now ;-)
Cool, I use InfluxDB for my home, but never understood the retention policy, thank you.
I have three point about your setup:
- InfluxDB is part of the TICK stack, that has Chronograf as data visualization tool in place of Grafana, a different flavour of the same concept someone might prefer, and has Telegraf agent you can use to collect raspberry's own metrics to track its health, feeding them through the same InfluxDB and Grafana combination
- This raspberry use case tends to put some pressure on the SD because of the big quantity of data written and red, it is important to consider an external HDD and a backup strategy (obviously you already know this)
- All those "installations" and "SD images" are things of the past, you need to switch to the container approach, I am talking about Docker; that will give you ease of installation, configuration, resource balancing, sharing, testing different setup, replication of setup and tenths of other things you will discover
1. I made a video about SD cards and showed how you can enable an HDD. So far I had no problems with my SQLite database. The number of points written is not a lot.
2. As soon as Docker gives full HW access it will be a good solution also for the RPI. I had problems to access a simple USB port on my PC. Otherwise a good concept with a lot of potentials.
You are seriously amazing. Thank you so much for this video - I've spend days trying to get this to work, and with just a couple hours and following your video, I've got everything up and running.
Glad to read that!
Big thanks. I was using the Node Red Dashboard and it was awful, nevermind browser issues if you left it on the dashboard page too long. Pushing my data to InfluxDB and then viewing beautiful charts in Grafana from that DB information is awesome. Thanks,
I also like Grafana...
@@AndreasSpiess I would also add that I'm using Python on a (Pi Zero W) to push MQTT data to a remote server listening for MQTT and is hosting InfluxDB, Node Red, and Grafana. You just need to install Paho MQTT on the Pi (pypi.org/project/paho-mqtt/). Then in the script just import it (import paho.mqtt.client as mqtt), define the remote server (mqttBroker = 'foo.server.com'), set the client with client = mqtt.Client(), connect to the remote server, client.connect(mqttBroker), and then for every variable assign to your data feed, you then just publish it with something like this: client.publish('Temperature', temperature). Where temperature = bmp.temperature, a feed coming from a BMP390 sensor. I any case, following your instructions on Influx DB, Node Red (MQTT->InfluxDB), and Grafana has been a life saver. It sure beats trying to chart something from file data. :-)
I'm a programming student and I was this kind of data retention policy saves a lot of work. Glad I don't have to reinvent the wheel.
We live in a good time with all this open source software...
Very nice Andreas - I've just been playing with InfluxDB and Grafans with InfluxDB fed with SNMP data by telegraf, all in docker containers. This video has given me an idea to move that on and try to connect my weather station rather than just looking at my Internet traffic :) I'll need to try to understand my stations data stream format which should be fun.
Thanks for another great video and the idea...
There are a few other videos about weather station hacking and Pi and Docker (IOTstack)
Excellent video Andreas. Clear instructions and to the point. We have used mqtt (activeMQ) +nodered+mysql and we were planning to use influx dB since its performance is more suited to IoT and time series data. Node red is really amazing and can help you to get to the results without the need of programming. Grafana (even if its free) its excellent visualisation tool. We have everything working under Ubuntu VPS, docker images of MQTT, node red, mysql, grafana etc and the performance is more than enough (more than 25k measurements in one minute transfered from the remote sensor via mqtt to the database without any issues)
it would be great if any of you wants to collaborate for any EU funded research based projects. (as mentioned most of us live parallel lives.. and do similar things)
25k points per minute are quite a lot! I did not know this is possible. Docker is a good concept. But with Peter's script, it is also quite easy to install. And we have full access to the RPI hardware...
Thank you so mucho for this. I succesfully followed this tutorial and now have an energy monitor and a water pressure sensor working and displaying data in grafana.
Excellent!
Thank you - I installed Grafana and InfluxDB on my existing Raspberry Pi 3+ with an already installed Node-Red and Mosquitto.
For the installation of Grafana and InfluxDB I found some instructions on the internet.
I use Node-Red and Mosquitto a lot for my home automation and used until now SQLite or flat files.
Your Video helped me to setup it - and it runs fine. Now I'm going to adapt my home automation to the new data sources.
Now you can watch the newer version (dockerized) of this setup.
This is a new concept for me (and it's very interesting), I have used a more traditional sql approach based on the work by Csongor Varga "Working with time series data in Node Red with SQLite" on UA-cam. Your viewers might find the comparison interesting.
SQL is always a possibility. It is stable and does its job.
Thank You!
I love how well structured your videos are. Suits my taste as german engineer ;)
Thank you! Maybe you also watch the one about Docker. It simplifies the installation of those programs
@@AndreasSpiess Yes i did and everything is working nicely together. :)
Your channel is pure gold!!
Greetings from Chile!
Thank you very much!
Everything new. I had watched twice to understand. I want this setup too for my home sensors. Thank you dear.
You are welcome!
Im now successfully using grafana influxdb with mosquitto (EspMqtt) using node red on my OrangePi Lite.
I started out using node-red and MongoDB, because you used to be able to get free cloud-based MongoDB services, but that time-based data consolidation is really nice, so I'll probably switch to InfluxDB for my projects from now on.
MongoDB seems to use quite a lot of disk space because it is made for other purposes. There is a comparison on the influxDB page...
@@AndreasSpiess yeah, MongoDB seems to store a lot of metadata for every record, which could be useful for some things, but not so much for simple IoT projects.
Amazing video for home use 😇 I have a similar setup running in my new flat, including Gas-Heater control and a combination of all sorts of different mcus with sensors and actuators based on various projects with a centralized mqtt/influx/nodered/grafana server (mostly own-software, tasmota...)
Keep up the amazing work - your channel is still one of the best (mostly because you don't have annoying intros and music or semi-funny jokes ;) )
Greetings from Austria
Thank you! Jokes are not easy in an international environment. Glad to read that you like them.
I like music, but in videos like that, I hate it myself. This is why I do not use it...
Lovely!. I've used InfluxDB + Grafana for almost a year, but for me the confusing part is Node-Red. I've instead used a InfluxDB libary for my ESP8266 codes, there is also for Python, etc.. and a small program "Telegraf" can send data from windows, macs, and linux machines.
I did not know a library exists. Thanks for the tip. In my case, it is more flexible to use Node-Red as an "intermediate". And the rest (telegraph, python) is maybe for other channels ;-)
You can easily send data our from Arduino and the ESP8266 as HTTP, ingest it in a Node-Red inout, then send it out from Node-Red in MQTT on port 1883 or 8883 depending if you need certs for TLS1.2. It can do a lot as an intermediary, as Andreas said.
Really helpful explanation of the benefits of time series databases for IoT, thank you 👍👍
Glad it was helpful!
this was an eye opener for me. always a good time learning from you
Glad to hear it!
Great, finally this seems to solve my IT problems for IoT. HomeAssistant is also good but seems to have an easy start with challenging future. (developing all the plugins etc. for simple additional custom features!)
So far I always used Node-Red and do not know Home Assistant.
@@AndreasSpiess For whom, who might be interested... Home Assistant (HA) has MQTT capability. An MQTT broker can direct messages to HA and HA will store them and display in a nice manner. Only by writing some making configuration files, one can trigger alerts, actions etc... e.g. I could set up a threshold for humidity sensor, after crossing this level, HA triggered a relay, which controls the fan. It is neat but extending it seems to be difficult.
Dear Andreas, great Video! Quite complex things explain so amazing short. Stuff like this can only see very rarely. Love your comments why you choose this tools. Thank You! Greets Thomas
Thank you for your kind words!
Amzaing effort from you and Pete's , keep the videos coming Andreas 👍
Thank you!
Thanks for a great video! I use Influx and Grafana professionally at work to analyze metric data from a mobile application. Thankfully I have an entire team of people to manage maintaining those systems. This kind of inspires me to see if I can hack my solar system on my roof (it uses zigbee apparently) to get data into influx and match it with real time weather so I can analyze better how my solar is performing.
At least you would have great support for your project. Not as we "simple people" ;-)
Nice work Andreas! Thank you for putting this together, very informative.
My pleasure!
Excellent demonstration of how little you need to collect and visualise data. I have a spare PINE64 which I'd like to try using these building blocks
You have to check which operating systems the script supports if you want to use it.
Thank you for your great video. These are the projects that I can learn from a lot....
You are welcome!
Nice video Andreas!
In my case, instead of manual instalation, since I started using Home Assistant a few days ago, I just added InfluxDb and Grafana pluggins in Home Assistant which was like 5 minutes effort (Node Red I already added before).
Anyway your instructions helped a lot for a fast start. And a sugestion - if you didn't already - check out Home Assistant. Nice platform + combined with Mosquitto, Node Red and now InfluxDb and Grafana ... make it very interesting.
I use some manually programmed microcontrolers (bathroom humidity and fan control), Nextion displays from your video over MQTT, and dwo days ago I reprogram own application for LED strip and Sonoff Basic with esphomeyaml. Addionally using Alexa over Node Red, that fits very niceli into Home Assistant. New graphs from bathroom humidity, temperature and fan control is today nice addon.
Point is that any additional knowledge nicely fits in picture. Constant evolution...
Other commenters also mentioned that it is easy with Home assistant. Maybe I have to try it one...
Awesome guide to getting Grafana up and running, thank you :)
You are welcome!
thanks a lot for this research and experimentation. we will definitely use this for our farm.
I hope your application will add some value :-)
Happy birthday Andreas, hope you have a good one.
Yes, I had a good one. My new bike for the Danube tour arrived right that day :-)
Hi Andreas, you might want to consider hass.io which has all as point and click install packages for all you talked about today! Great also to create automation for your tv lighting etc.. all over a web gui so easy it will withstand the wife test.
Maybe I have to look at hass.io once...
Thank you , always something to learn every week in an easy way
You are welcome!
Interesting and useful video!
It seems we are heading back to remote sensing? I like that.
Sensing and acting based on these data is IOT for me ;-)
Excellent video. Thanks.
Greetings from the galilee
Glad you liked it!
Great video, Andreas. Really enjoyed this one.
Thank you!
Hi Andreas, just to share. I created a "home heating system monitor" based on some arduino-nano, reading NTCs/DS1820, and raspberry interconnected by field-bus based on modbus (RS485). Raspberry is running a python script to collect data and store it on influxbd, Grafana is used to visualize data and correlate info.
GOAL: integrate solar PV, heat pump system and Gas heat generator, control floor heating and try to be more green. ... will see...
It sounds like an interesting project. Maybe you have also planned to add something like hass.io? And maybe you also watch the video about IOTstack (if not already done)
... Yes I am looking deep in Hass.io ... but ...
before go in Strong IOT every one should read 'Abusing the Internet of Things'.
For now l prefer to stay with a well lock down reporting tools.
Grüezi Andreas, as always a great video with many explanation that everybody can follow and learn a lot. todays topic is very sexy, since data/process mining is the new lubricant of the world. as already told on TTN con I love your 'Mutterwitz'. cheers Michael (guy form Sulzer)
It was long overdue for me as I wanted to use it for my installation.
I hope you had a good trip home!
You're awesome, thanks for sharing this knowledge. I've been learning a lot from you.
You are welcome!
What another great video, thank you Andreas! Your video inspired me for changing my implementation on how to use historical data.
I hope you will be successful!
@@AndreasSpiess The InfluxDB was really fast and easy! I installed influxDB in a chrooted Linux on an Android TV box where I am using it as a mqtt broker and node-red administrator. Now I am testing the first data recording from my air conditioning energy meter. Next step to install Grafana.
And all these thanks to you again :)
Hi Andreas, relational databases do not necessarily have to be setup with fields like "Temp, Pressure, ...", you can as well do an different approach and use a dedicated field to indicate the type of the value. Something like "Timestamp, Type, Value", that way the only restriction is the type of value you want to store ( integer, float, string, ... ). If you want to get completely rid of that you should have a look at elasticsearch, it's a database system that is highly scalable and very dynamic. You can easily scale it up from one DB node to hundreds in seconds, simply by adding additional nodes and the same way back (in reality there is a little downside in downsizing - let me know if you are interested). Elasticsearch does NOT stick on types, or even database table schemas. It simply stores what you send - if there is no - for example Temperature - it just creates it dynamically for you. Additionally the values are the same flexible - no need to preselect a type like float or a length, it's just dynamically created on the fly. It's even able to store >100k Messages/s depending on the machine it's running on. In reality I had soe time ago a DB running where I was feeding event log data from 9 Active Directory domain controllers and keeping ~1 year of history of that data. Peaks where around 500k messages/s and around 5-7 million entries a day on a 6 year old reused hardware. Basically a perfect match for MQTT where we do not have strict type/name rules as well.
You are right. You can store these primitive sensor values in every database without any problems. AFAIK Elasticsearch is considered to be a "NoSQL" database. They are very good at certain things like you describe: Unstructured data and also full-text search.
As I said, the strength of time-series databases is not the storage side but on the retention as well as the extraction of time-based data.
I do not want to focus on database concepts on this channel because there are people which are much more capable than me in this matter.
It's a search engine, why should you use it as a database? How do you efficiently get the average of a field (or even better over a time period)?
Types and structure shouldn't be abandoned if you don't have to deal with dynamic data.
@@RalfVogler : Elasticsearch is NOT a dumb search engine, it's a NoSQL database ! Beside that you can use several mathematical functions on fields, similar to a relational database. As MQTT as well is a typeless protocol elasticsearch would be a perfect match to store anything that would possibly arrive over MQTT. Along with Kibana it's quite easy to visualize the data, even other solutions ( including Grafana ) are possible.
The answer to your questions is here : www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-avg-aggregation.html, it can be expanded to any other critera the system is aware of ( time, name, field type, ... ).
Given the fact that we talk about sensor data to be stored, mainly through MQTT, it might be worth looking at something like Elasticsearch that could easily handle . However, it's anyway recommended to use the DBMS you prefer an feel safe with ! Additionally it of course depends on the needs and given infrastructure.
If you try to store (any) Windows eventlog data, especially from a whole domain, into a SQL database you'll understand the advantages of Elasticsearch and it's scaleability ...
I'm just bringing it to the table, it's everyone's decission to have a closer look or not. My recommendation is : If you run just a couple of sensors use what you already have. If you want to scale up to hundreds of sensors you might have a closer look to Elasticsearch.
"use a dedicated field to indicate the type of the value"
My point was that if you do that, you can't do things like average (which you are probably interested in for sensor values). Correct me if I'm wrong, I never used Elasticsearch.
Of course you can do anything. Just writing to a file is also fast. I would assume that even for storing data of hundreds of sensors the choice of database doesn't matter.
Where you will see a difference in performance is when you search in non-indexed fields, aggregate data over wrong types or use joins when the database doesn't support it.
The built-in capabilities is the other important point: does Elasticsearch support querying time series data?
Concerning types: either you declare the types for each MQTT topic or you let something like InfluxDB infer them from the first message.
@Andreas issues are now solved, as you know by latest comments on Pete's blog... i'm MrShark :)
Thanks! The video was already out then. I pin your comment that everyone can read it.
@@AndreasSpiess problems were caused by a double install of sqlite node, 1 in Pete's node selection, 1 after that with unsafe-perms, and the sqldbs node which is unmainained from 2 years, we removed it... and now nodes are installed 1 by 1, instead on a single batch, so if 1 fails, it's just that, it does not make the others fail all together... now seems all fine
@@AndreasSpiess minor issues solved this morning: switched to official grafana repository instead of the previous one, as now it exists and it's working properly (it was not so when i added grafana and influxdb to script, a year ago or so...)
@@squalazzo Influxdb node still requires manual installation - tried just a couple of hours ago
Cool. Great stuff Andreas.
Its all the craze doing opensauce. I wonder what made u not choose non elk and or prometheus. I suppose grafana is top for these requirements. Miss lora and esp32 a bit... a lot to be said about long range iot.
Thank you!
Wao!!!! Great idea!!!
:-)
Very nice! The problem with the Pi for DBs is the SD. Even good SDs tend to worn out fast and you lost your valuable DB.
I would recomend to store db in other disk even at the expense of use USB disk. There are other SBC with SATA/PCIe that will be more suited this purpose.
I once did a video about that. For me so far I had no bad experience after more than a year (not Grafana,SQLite)
I love it, now we really talk IoT. One more step Andreas, automated MQTT to influx converter. Let me know if you'd like to have the links. In my porject (google "home smart mesh"), I have such a python converter. It subscribes to a certain MQTT topic pattern, and posts jsons as is into influxdb. Everyone is already very familiar with MQTT directly produced from ESP and others, with an automated converter, you would not need to edit a custom translator for every sensor you add.
Yeah and I forgot something, once MQTT is plugged to influxdb and grafana, you enpower it with something like zigbee2mqtt and have awesome cheap market products on you dashboard !
I would be interested in the links. Did you do a tutorial?
So far I always thought that ZigBee is quite expensive for makers and not well accepted in our community. Maybe I am wrong...
Hello,
I spent a quick 30 min digging into your “home smart mesh” with the objective of understanding just the mqtt to influx converter. I feel close to getting it but honestly got a little lost in trying to peel the mesh application out of it. In my case I have at least a dozen esp and other devices all speaking to mqtt over Wi-Fi to node red then over to RRD databases. I want to explore updating this to mqtt to influx to grafana directly. Examples: /sonoff/1/temperature/ or /base/55/humidity/ or /1wire/“serialnumber”/“englishID”/temperature/
Do I configure all these in JSON or just create rules for each generic example?
And Bravo to Peter and team for that script which I have used several times on Pi and other Linux installs!!
Haven't tried it yet, but there's a Telegraf plugin for MQTT to Influx: github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
@@AndreasSpiess for MQTT to influx with python please find a direct link to the github influx python directory here github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx
I do not have a special tutorial for the MQTT to influx, but to be honest I struggle to make a documentation or tutorials as clear as you do, you are better at that Andreas :)
Nevertheless I spent effort to document the whole "Home Smart Mesh" which has a lot of custom RF in it, and it is not easy to isolate the raspberry pi scripts from the rest.
I tried to document on Hackaday hackaday.io/project/20388-home-smart-mesh
and on Github which is more focused on the nRF52 Hardware github.com/nRFMesh/nRF52_Mesh
When it comes to Zigbee, I can confidently say that it became cheap enough now with all sort of ready sensors ~< 10 € from Ali, and an official nRF52840-Dongle < 10€ from Mouser.
Now I would agree with you if you say nRF52 Thread or Zigbee is not well supported by Arduino, but Andreas, we have to make it happen, by selecting best HW and pushing it to be come easy and popular.
I started wrapping the NRF SDK official from Nordic around a c++ API with classes that make it Arduino like, my latest progress on that here github.com/Roblibs/rover_firmware/blob/master/applications/03_sensored_bldc/main.cpp
but far from being a real arduino core.
The nRF52840 is I believe a real challenger for the ESP32, as the nRF52 has native USB, an Automated Programmable Peripheral Interconnect, and with its custom radio I achieve latency ~ 1 ms.
I'll be glad to share any required further info and details.
@@DavidL3430 please find a direct link to the python script that converts MQTT to influx, it should be possible to take that directory away from all the rest github.com/nRFMesh/nRF52_Mesh/tree/master/raspi/influx
But I admit, it is not packaged to be used without Python knowledge, as you notice, even if I have the main config in the config.json, a lot of the logic is coded in the def mqtt_on_message, for example the number of words in the topic e.g. mesh/node5/temperature and also I force some types to be float or int in lines 36 to 43.
But on the good side, I even prepared scripts for backup and export, and the note that the config can be different depending on the hstname so that you have many raspberry pi and save all config in the same directory with names config_rasphostname1.json, config_rasphostname2.json, ...
I will gladly provide you any further help if you want.
Useful and interesting as usual 😉
:-)
Very interesting and easy to jump in.
Actually I was using a Python Jupiter Notebook for Receiving, parsing, sending payload to the MQTT broker. This solution is great, especially for future maintenance
I never thought about using Jupiter Notebook for something like that. Now I have to look into it. Tanks for the info.
help please im very new to all of this but willing to learn . Im following video but when i try to run command at 2:38 im getting a message "awaiting response...404 not found " . ive copied the text exactly (several times). like i said im very new and probably should start on something simpler but this does look very interesting. fantastic presentation though ust my lack of understanding holding me back. regards paul
UA-cam is only my hobby and unfortunately, I have no time for consulting :-( But I am sure you will find out.
Fantastic job!
Just what I was needing for my project.
Thank you!
You are welcome!
Hi!
Could you please upload the NR-flow? I am really interested in the details in the function-nodes. I am struggling with my own input from NR to InfluxDB. :)
Great video and very timely for me as I am looking to complement my old RRD based graphics site with Grafana and Infuxdb.
I have used Pete’s ( and Mr Shark’s) script along with his node-red big timer for years!
While I do enjoy node red, I am looking to lower my dependence on it. Thus hints of ways to direct mqtt data directly to influxdb are high on my curiosity list!!
A other viewer commented he is using python scripts instead of Node-red
github.com/influxdata/telegraf/tree/master/plugins/inputs/mqtt_consumer
Thank you. I've been struggling for years with mysql and an inflexible php graphing library.
This is maybe a little more flexible...
just putting up xmas lights but as soon as that is done i will be following this video and cant wait to use Mr. Scargills 'script'
Maybe you watch the „ultimate pi server“ video first.
@@AndreasSpiess LOL i was just wondering that !! good job the mrs LOVES your accent so i get away with watching geeky videos
Thank you. This is what I needed. I'll get to work right away...
Enjoy!
Great work again Andreas!
Thank you!
7:30 that function call what, i now collecting data every second and i need to shrink it but i dunno what these function call, and do this function available on phpMySql ?
You can program it also in SQL if you want.
@@AndreasSpiess but i cant found any tutorial about that function for SQL
Many Thanks Andreas - Well Done and hope to see more
You are welcome!
Thanks so much for providing the SD image. Thanks thanks thanks
You are welcome!
Andreas did you stop numbering your videos ( #Number). The last one was #245. In my opinion, it facilitates finding specific videos and contents. I hope you continue using it. Thanks
No, but I always have about the newest 10 videos without numbers.
This is what i'm looking for. Thanks!!
:-)
Fantastic project👍😀
Thanks for sharing😀👍
You are welcome!
I would like to see also a video about continuous queries and retention policies in detail with real examples related to weather (and possibly energy data). I have about three years of data every few seconds for my many weather and energy sensors but find very cumbersome to implement retention policies and continuous queries and to downsample the already existing data.
I do not see that this is a topic with lots of interested people. I think you will find some information on the net.
Your situation is perfect as you can play with your data and find out which policy fits your needs
Playing with influx and grafana for computer (cpu ram ssd-cache and disk usage) and backup status monitoring. Got it working with docker containers after a few hours tinkering. Did not see the need for node red, though. Using curl for posting, and trying to get raspberry+Screenly working for display.
I like node-red because I know it ;-) there are many other possibilities to bridge from MQTT into influxDB. Did you try docker to use GPIOs or Serial on the RPi?
@@AndreasSpiess I used docker for influx and grafana, not for the data collector /sensor program. You may be able to use gpio and serial inside a container by usingal a privileged container and mounting /sys and /dev/tty*** as volumes on it.
@Andreas did you consider writing the name field not as a measurement but as a tag? This is achieved by writing not one object as msg.payload to the InfluxDB node, but writing a list [] of two objects, first one with the measurements, second one with the tags. If you set the name field as a tag, it should also show up automatically as a completion choice in Grafana.
I did this after the video. It is easier for the selection in the dashboard. But during the take I did not know how to do it :-(
Thanks Andreas, did you consider to wrap this config in Docker/Docker Compose instead SD ? It would be easier to upgrade
No, not for the moment. Maybe later...
Really inspiring ideas, thank you! I'm thinking to move my solar panel yield visualization to an influxdb and grafana stack.
Good idea!
Did you set your retention policy for the weather station to do hourly averages after one year? If so, how? I am seeing it is ideally done during the database creation but can be done after. Eitherway the syntax isn't so clear to me, and I like the retention policy you mention.
So far I do not use retention policies. The RPI4 is still fast enough.
Great job!! Very useful video.
Thank you!
I have some problem with selecting appropriate choices when running Peter's script through putty ssh. What key do I have to press when I want to select/deselect the available choices?
Use the space bar.
Nice for my next step, 3 monts from now, i donate when i download the sd card, Have a Nice Day and Thank,S for so much Help!
You are welcome!
Thank You, now i have made a sensor for my garden, Very litle solarpanel, a Mini_Pro that sleeps all Day with a Ds3231 clock, wakeup Once a Day send by WiFi with a ESP8266 with turn On with mosfet and give data to mqtt. The System works and have with 18650 batery power enoff.
The py and MQTT make a Nice http:). So you can do a lot with a ESP8862 in this way. No power problem. I see On my computer How dry the garden is and maby give iT Some action.
Is it possible to use "measurements" from different Influxdb databases in the same "panel" in Grafana? I can't figure out the syntax for the FROM field in the query editor.
I only use one database for one panel. So I do not know if you can use measurements of several databases in one panel.
@@AndreasSpiess I figured it out. grafana.com/docs/grafana/latest/features/datasources/mixed/
Andreas, can you share in some way settings of nodes that you use in node-red. I try to use image you shared, but my PI is 4b and probably this is why it doesn't boot with image you shared load in my SD card.
Maybe you use the newer version using Docker. There is a video about it
Thank you very much sir! Always love your video
You are welcome!
can you make a video on the topic of retention strategies, it seems to me to be a little more complicated?
Would you use this configuration, shown in the video, for everyday use?
I am not sure if this is stuff which interests a lot of people. But you find infos on the net.
I’m glad I was already on the path of least resistance w/ Grafana
Maybe you have a look at the IOTstack project. There you find everything easier...
What do you think about using the SD card long term with the amount of writing influx will be doing? Would you use an SSD if you were planning on keeping your setup long term?
No
7:45 How can we do this? Can You make a video?
So far I have no such plans :-( But you never know.
Thanks for suggesting the script from Peter Scargill. However the script did not install properly and the pi4 seems to be slow now. Is there any way to roll back the script? I rather install the tools one by one.
Maybe you watch my newer videos about IOTstack. Much easier and faster.
Great video, as usual.
But you should start to use the ip(8) command and not ifconfig(8), as that is obsolete and are going to be removed. And it is much more flexible and useful, as you don't need the arp(8), route(8) etc as that is all merged into the ip(8) command. And it is shorter to write. :-)
ip -4 address show # can be shorted down to 'ip -4 a s'
ip -6 address show
ip route show # can be shorten down to 'ip r l'
ip neighbour show # can be 'ip n s', also try 'ip -6 n s'
I did not know this command. I will try it. Thanks.
you'd better warn that grafana works on RPI3 only and not on previous raspberry boards.
I did not know that.
Thanks for warning Giuseppe Suanno. I was about to do this on a zero because i happen to have one available. Even if i suspected it might need a gruntier board like PI3. Why only on Pi3? Can you please elaborate or provide source?
I am using it on orangepi lite.
I've got it running on a RasPi 2, and have also ran it on the original 512MB RasPi Model B. Granted it's slower on the 1st generation Pi, but it runs with no unreasonable difference in speed on a Pi 2 compared to a Pi 3. (which I've also run it on)
Hi Andreas! Love this video, and thanks for putting together the image. I recently ordered a few Rasperry Pi Zero 2 Ws, and I'm trying to load the image you provided to one of them. When I load the flashed SD card and plug in the power, the device doesn't boot. No power light, no indication that it's connected to wifi (after adding wpa_supplicant.conf and SSH to the root directory). Is there any reason the image as provided wouldn't work on a raspi zero 2 W? Thanks again!
I do not know if it works on the 2W. Two things to consider:
1. The memory of the 2W is very low for such applications. I would go for at least 2G
2. Maybe you watch the video about IOTstack. There you find another way of installing all the goodies...
In the measurement "stations". Wouldn't it be the better option (faster for queries) to store the "name" as tag and not as field? Tags are indexed and fields aren't. Still starting with InfluxDB but that's how i understand it.
I am not a specialist in InfluxDB and my data size is small. So I never investigated.
Hello Andreas, does this work with an DHT22 Sensor or an Adafruit Sensirion SHT31-D? Where would I need to set the sensors up, in Node-Red?
Thank you - great video!
I usually do not connect sensors to the Raspberry because they have to be at different places. I use ESP32s and connect them with node-red via MQTT.
Hi, great project. I want to store DHT sensor data to monogodb locally, how can i do that?
I do not know, I never used Mongodb
Hi Andreas, i've tried to use your SD card image on two different SD cards and i cant get any of them to boot, i know this post is two years old, is there an updated version of this i can try? Thank you for your work on the videos they are very interesting and i try to learn from them.
I will have a look at it.
Seems retention policy only tells the data base what old data to delete. Achieving the reduced resolution for older data requires a lot more work and down samples it in separate place. Then it is not possible to have a single graph in graphana. Or am I missing something?
I never tried it so far. Stuff for some future work ;-)
@@AndreasSpiess Indeed, this seems to be complicated, but there is a solution here pommi.nethuis.nl/collectd-influxdb-grafana-with-downsampling/ I will see where this leads..
Thank you for the link!
@@AndreasSpiess Unfortunately it only works, as the author says, when the end of plotted time is always now. On top of this, it does not take too much query to kill the rpi. it runs out of RAM fast. This kills mosquitto and node red for a while.
issue is still open github.com/grafana/grafana/issues/4262