Tech Note 154 - ESP File Server using the Asynchronous Webserver
Вставка
- Опубліковано 15 кві 2021
- This code will enable you to perform a directory of SPIFFS files, upload, download, delete, rename, and stream files in SPIFFS and get a reasonably comprehensive overview of the system and status.
Code: github.com/G6EJD/G6EJD-ESP-Fi... - Наука та технологія
I love your tech notes. Thanks for sharing knowledge
I am learning to much from this channel superb.
Man, I love you for this. Amazing work.
I am learning to much from you sir ,you are the best sir thank you very much .
love the way you program, very neat and easy to read. Thanks
I was taught to keep things neat and aligned as it always uncovers errors during development
Thanks, David, a very nice project for small files.
Very useful video. Thanks for putting it together.
I miss your videos. I learned so much from you.
I’ll try and resurrect my videos I have a number of subjects I can cover, thank you for your support.
@@G6EJD Very cool. Can't wait...
Excellent. Thanks for sharing.
It's amazing software. Thank you very much for sharing it.
perfect, this is exact what i need. Thanks your your help!!!
Man, so nice job !
Been a fan for a while, way to go man!
I’ve always had a question though, any ideas on how to use to send file from one esp32 serving the files to another Esp32? This part I have never been able to get to work. Would be awesome to see how.
Anyway, thank you so much for sharing this with the world. Commendable.
Hi! Did you make it work? Send files from an esp32 to another esp32? I'm interested to know it this is possible
Sir David, for a while, we have not heard from you on your channel. We missed your lessons. I hope that the inhibitor is good and that you are in the best health.
I'm OK thank you, just not been up to making videos at the moment.
@@G6EJD If another video comes our way, you can count on myself and hundreds of others to be here!!! Yours is my favorite channel on youtube
@@theknee5590 thank you for your kinds words
Nice job. Thank you.
your video is awesome
Interesting video. Note I did not get a UA-cam notification.
Hi John, I don’t know why that happened, other than this video was a timed release and I did that a few days ago, maybe there’s a UA-cam issue with doing publishing and notifications.
verry nice!
Thanks you
we are LCD manufacturer for 19 years，we have hundreds kind of LCD display, and our products could connect with Arduino, ESP32,STM32 etc, so I believe there is a strong possibility for us to work with you.
Oh, by the way... great program!
I watched many of your videos and must say they are easy to follow and work without any problems.
As you note file download size is the limiting factor to about 5k6, the user interface on this web server is a lot easier to use to download files from the SD (conversion to it worked without fault) with the populated screen of files on the disk than your earlier non-asynchronous web servers, which coped with the larger files. How easy is it to revert this user interface to a non asyncronous server whilst keeping the easier to use download interface? Presently logs created are around 20-30k in size for an hour block and new files are created every hour, so you can end up with a few files to download. How easy would it be to automate the process and download the entire contents of a directory with one click on the interface?
To go back to to non-event driven server you’d need to change all of the client responses to polling in the main loop. By using the various functions Dir and Download etc you could do everything under program control when required.
Good work. I see that this works with littlefs partitions. Is there a way to get the file server to browse the content of directories?
Well there are no directories in the filing system, just a virtual dir using a prefix so it would need to find all files with a dir prefix, so the dir function could be modified with a parameter that is used as the file prefix.
I have a sensor log file I'm graphing on an esp32 server but the file can get very large - 50MB after a year. I've tried a few different methods of parsing the data so only a small portion (~5MB) is sent. With that method, the ability to zoom in on the data is lost (I record every minute on an micro SD card). Any tips? Thanks.
Unfortunately both the webserver and this asynchwebserver have issues down-loading data from the ESP to say a PC. they both run out of RAM to hold the buffers and I've tried really hard to solve it, but to no avail. I've also tried what's called a chunked transfer so each chunk is an MTU size about 1400-bytes, but that does not work either. From what's I've read no-one else has solved the problem, including the library author. So sorry, no answer. You can do reliable 5K transfers, so read out 5K to a temporary file and then transfer it and repeat many times! If you have a device with lots of PSRAM then the problem goes away, say 8MB of PSRAM plus the ESP32 RAM
@@G6EJD Cheers! It would be cool if some wizard writes a php equivalent for microcontrollers.
How to download and upload to sdcard no spiffs ?
Thanks David for such a good tutorial. Really appreciate.
One problem i m facing it here.. The bigger file sizes are not getting downloaded. I have some sensor readings logs in spiffs that needs to be downloaded using this web browser. The size of the files are 32KB's . Can you suggest what changes need to be done in existing code to make this download happen?
Thanks in advance
Are you using the latest code version, as this was corrected?
@@G6EJD Thank you for the response. Yes i have used ESPAsynch_Server_v1.1.ino code
@@AmritK-pu3rf the problem is with the Asynchronous WebServer, so do you have the latest ESP32 core?
@@G6EJD I think yes... I am using esp32 devkit with chip esp32 wroom 32
Hi, this is fantastic. Could it be used to access SD card over SPI, remotely? Assuming I have wireless connectivity for the esp32?
Eg, I'm in 1 place and the esp32 is in another, not on the same network but has a alternative internet connection. Thank you, sir.
Yes, as long as you have the IP address or you use a domain name server eg a DNS service then you can transfer files remotely. You can change spiffs to SD as all the functions are identical so spiffs usage becomes SD.begin(); and so-on.
@@G6EJD Thank you for the reply! I feel a project coming on 😀
@@_GB84 Hi, Did you do what you said?
Very nice development!!!! did you made it work with esp8266 too? i tried here but dont seem to work on that…
Yes because it’s titled ESP, so you have the latest ESP8266 compiler, or downgrade it back a few versions, then it will work. Unfortunately now it gets little attention.
@@G6EJD Nice to hear! I will certainly try… I was trying this async webserver to substitute my old projects but it caused problems with web update and get or post requests from the server for example to get the ip address from ipify… or retrieving info or sending info to a database in another server…
Did you found a way to do that with this lib? I tried to look in your code but it appear that your server is mostly independent…
@@HIGH_TECH_NEWS you shouldn’t need to use/obtain a remote IP address just the logical name and then let the DNS resolve the IP address, that would greatly simplify your code. GET and PUT requests in the ESP8266 libraries have always been troublesome but by contrast the ESP32 variants work perfectly.
@@G6EJD in my case I need to do some requests to my database and gets some info… it also need to make send some info from the user too… unfortunately I use only the esp8266 for it’s cheaper… the nice thing is I keep a table with the ip addresses of all the devices connected and their ports… so when I connect with my phone it works like a dynamic dns.. I used the tiny lib for this and works well with the async… I was hoping you maybe have had this experience with sending the get and post from the server… any info would help a lot thanks a lot!
ADC2 pins cannot be used when Wi-Fi is used. So, if we are using In ESP 32 Wi-Fi and we are having trouble getting the value from an ADC2 GPIO. How should solve this problem.
Switch ADC1 to another GPIO pin take a measurement and then switch back to its former use.
Hi.. If I download it stops at 5.6kB irrespective of what file (>5kB) that I try to download
Serial prints " Downloading file..., Download handler started.
Am I missing something in setup?
This is because the ESP32 runs out of of RAM, I mention that in the video. It can be solved with devices with PSRAM. I have implemented chunked transfers but it only helps a bit, it’s an Asynchronous Webserver problem.
Sorry... MIssed it
Is it possible to replace SPIFFS to SD card? How big is the file size limit can be download if using SD card?
Yes you can replace SPIFFS with SD just change the object definition so keep SPIFFS but make it SD
Do you know a way to implement - inside your project - a graph that keeps updating every x microseconds as the ESP receives some kind of data?
The graphs use Google Graphs and whilst the concept is achievable, the main challenge is getting data at such a high rate (uSec) storing it and then displaying it because it will take a number of seconds to create the graph and update the screen by which time many readings will be lost. It depends whether your data is in real-time or near real time or can be sampled then displayed. Drawing graphs requires a lot of storage so you’d need to consider the memory requirements needed to get a graph displayed. So yes that’s achievable depending on the incoming data type.
@@G6EJD Yeah the purpose actually is for a real time implementation, such as displaying a sine wave at a certain frequency (i would like to be able to get at least 8kHz). But since it can be kinda hard to display data at such high frequency, do u know if its possible to aquire data through the esp32, save it in a file (a .txt for instance) and then upload it automatically to the server?
Your challenge is describing a sine wave with sufficient resolution to anything meaningful with the samples, so perhaps 256 samples per cycle and then for how many cycles which in turn will set the storage requirements, it could quickly turn into very high values which takes a relatively long time to transfer, by which time many cycles have elapsed so attempting a real time monitoring system is very complex. Plus at an 8kHz analysis rate that needs say a set of samples every 1/8000/256 = 0.49uS and that’s too fast for the ESP32, so you end up having to compromise on the sampling rate and therefore I’m not sure what useful information can be obtained from the sampling. The other challenge is how to start the sampling and at what point in the cycle, usually this would be at the zero crossing point so you get a whole cycle, but then you have to design a zero-crossing detector to trigger the sample collection. Is this a college or university task, if so it is a challenging project that will stretch every aspect of your learning 🙂
Thanks for the video, it was so much help in my project. Could you do a video or atleast upload code with sd card version instead of spiffs?
All you have to do to convert from spiffs to SD is replace all spiffs with SD and add the sd.h library or declare spiffs to be SD then apart from one line no code change.
@@G6EJD Thanks, since you've replied i've been trying to make the changes and been having alot of complications. If possible could you upload a finished sd version? It would be very appreciated not just by me but for others in the future that admire and are motivated to do coding because of you.
@@tamarniharris6852 dont push someone to do your home work, better you trying to make it done and share to everyone :)
@@ardutorial loool i wish it was homework. I actually solved it the next day. And it would be a better option if you uploaded it as you are a arduino tutorial channel?!
@@tamarniharris6852 Then you can share the code that uses SD card instead 🙂
Hi David, It seems that you haven't posted new video after 154th. I hope everything is OK.
Thank you for enquiring, I’ve been a little unwell but getting better again. When I feel up to it I’ll restart.
is posible update firmware (.bin) in option upload file??
Unfortunately not
Hi David, great video!!
Which is the main difference with your previous video? ua-cam.com/video/u6vmUIPU1i4/v-deo.html
About the download speed from the ESP, on the previous version, I was able to reach higher speeds by increasing the buffer size to buf[2048] inside the server.streamfile() function. Anyway, I can obtain a download speed from the ESP of 350 kB/s max.
Do you think this webserver can increase the transfer speed?
Cheers
Enrico
ps: I'm running everything on an SD card, because I have quite big files (more than 10-20 MB)
Can you sahre your code with SD card support? Thank You!
Can you do this on esp-idf?
I can’t see any reasons why not
Hi Dave, the code downloaded from Github “G6EJD-ESP-File-Server” works great, but I haven't been able to combine it with the ease of writing, reading or appending SPIFFS files. I have tried combining it with the SPIFFS and LITLESPIFFS test codes but I always get an error.
I do not know what I'm doing wrong. I have been using your code “Tech Note 087 - ESP32/8266 Download, Upload, Delete, Stream and Directory Services” for more than three years in a project in which I write and read files on a microSD card and it works great; but the possibility of removing the microSD card and simplifying the project keeps me up at night.
Arduino IDE errors point to the FS SPIFFS library. Can you give me a hint on what I should do please?
Hi Paul, what compiler settings are you using for SPIFFS there are numerous options, maybe 1 + 3MB SPIFFS? Also what version of ESP32 compiler are you using - 1.0.6 is the only reliable version IMO.
@@G6EJD Hi Dave, my project uses the MH ET LIVE ESP32MiniKit, the compiler settings are set to “Default with spiffs” and I use compiler version 1.0.6.
Now, by inserting the example code SPIFSStest.ino into your code the IDE points to the line “#define FS SPIFFS:” with the error “variable or field 'listDir' declared void”.
If I comment out the "#define FS SPIFFS:” the IDE points to the line “webpage += "" + ConvBinUnits(FS.totalBytes() - FS.usedBytes(), 1) + "";” with the error “expected primary-expression before '.' token”.
If I comment out that line I get the same error with the next line that contains an "FS" statement
I found a work around replacing all “FS” occurrences with “SPIFFS” and commenting out “#define FS SPIFFS”. Now everything works fine.
Thank you very much for sharing this server code with SPIFFS, it is extremely effective and friendly.
@@paulsalazar4217 this could be due to the transition between SPIFFS and LittleFS which seems to have stalled! What’s wrong with SPIFFS anyway
@@G6EJD Hi Dave, I think there is nothing wrong with SPIFFS, in fact using this library your code compiles and works correctly. But in my project I found that SPFFS affects the operation of the I2C keyboard, making it not responsive.
Replacing all SPIFFS occurrences with LITLESF fixes the problem and the keyboard works fine again, but now the “SPIFFS_ImageReader” library does not read and does not project on the TFT screen the images stored in FLASH memory.
If the two libraries SPIFFS and LITTLESF are included, it does not read or project the images of the FLASH memory either, but also each reboot the FLASH memory is formatted and the stored files are lost.
I'll stick with your File Server with the Asynchronous Webserver code with the LITTLEFS library and wait for something like "LITTLEFS_ImageReader"
Thank you very much for the great contribution.
Hi Dave. The keyboard problem does not seem to be caused by the “SPIFFS” or “LITTLESF” libraries but by “ESPAsyncWebServer”. I have googled and there are some references about problems between I2C connection and ESPAsyncWebServer.
Why I can not download file? Is this work with specific esp Arduino core. ?
All other operation work great.
I tried fsBrowzer example where download work great.
What ESP32 compiler version are you using?
1.06 works
@@G6EJD look like I messed Arduino core file .
May be that's why it not working .
Thanks for helping me.
If problem exists I let you know on GitHub issue 😅.
I think its looks likely may be there way to know esp core version and compatible library need to mentioned in Arduino sketchech.
Wishing there is macro already exists to this task😇.
@@AvinashPudale 1.0.6 is the latest version
Is it possible using ESP8266?
To convert it to the ESP8266 is possible; there are a few commands for WiFi are not supported, but other than those everything else is supported by the ESP8266 compiler. However, it needs lots of memory that the ESP8266 does not have!
This answer
@
David, such a great job, so happy to find your channel! 73 VA7AYG