Remember that incremental (limit) pagination breaks if you have persistence on, queries might trigger from old cached values. Say you query for 10 elements, you might get 2 element from older cache.
The Firestore videos series is awesome. Its nice to have a cool dude make you laugh while you learn about underlying technology. Great content and very helpful. Keep sharing please :D ;)
If I increase the limit from 20 to 40 to 60, do I really fetch also the first 20 documents all the time? I thought that the Firestore client SDKs cache these documents and send me those documents only if it's needed. Or is it just that I get billed for those, but Firestore tells the client SDK that it already has the most recent version of the first 20 documents?
This is something i struggle with as there are just hints in the document of how caching works and when you will benefit from it. I can do tests and sometimes I do to try to figure if a query is expensive, but it would be easier to remember if I could get a lesson on it. Please make this a future topic for an episode or add a section devoted to it in the documentation.
@zztop3000 Yeah I know that Todd says that you are re-requesting the old data, but this confuses me. Why should the documents be fetched again? They are already present in the cache of the app. Or do I just get billed for those documents even though they don't get sent to the client?
@@ToddKerpelmanCorp One follow up question. If with the offset you are basically just querying and charging for every record up to the offset, then what is the point of having that parameter at all? You could just request all the records with limit and that would be the same. I always thought of it like pagination and avoiding requesting and charging money for these records. This is actually a huge thing that should be in the docs.
Some people tend to say awesome for just 'ok' stuff. But this playlist is really... awesome :) The content, the presentation, the animations. Everything is top quality. Keep up the good work!
I think if you want to make this work you need to read the whole collection with attached listener so when anything changes only the change will count as new read
Thx for the simple show of options and associated tradeoffs . PS - where on Earth did you find such a enormous gummy bear? Must have doubled this episode's production costs...
Thanks for this! I would love a video that suggests a schema for commenting on user's posts. It's a little tricky to set it up so it's safe permission wise, but also logical query-wise.
Hey Todd, are you available to do the instructional videos for our app🤣? Seriously, another great video. I haven't got up to thinking about pagination yet but I will soon and this explains pretty much everything I need to know.
Thanks. Would be great to have a new video focusing more in pagination for realtime cases, like chat apps, for instance. It's kind of complicated to find help online for a consistent way to implement pagination with real-time updates with Firestore
So this means that conventional pagination (which has a nav containing 1 - total page count) is impossible out of the box with Firebase - the only option is to infinitely scroll pagination as described. I cant believe this is no way to do this easily with Firebase. Think ill bite the bullet and move over to AWS
@@ToddKerpelmanCorp Gotcha, I think I'll end up using a cloud function to periodically update the random fields. Not a perfect solution, but cheaper than other options that I can think of
great video; 2:31; does firestore support query on one field and order by another field now? just wondering if it now supports this what I love to hear or is it just a makeup example to simplify the video? thanks #askfirebase
@@ToddKerpelmanCorp Thanks for writing back!! You are GOOGLE! There's no such thing for you guys that's tough. If you put your will to it.. I guess you'll get it done. Try using softwares like Video Scribe. They are fast and easy to make yet crisp and useful for educational purposes.
you missed one thing, to paginate without infinite scroll we need to know the amount of records the search has, so we can know the amount of pages, how we can do that?
Firestore is crap, specially the web SDK. Any supposed gain in productivity vanishes when you reach such simple and common use case. In a SQL database, it's terribly simple.
@Todd There is a nice idea for you! What if we can extend range of values to listen to while paginating. For example, I do this query for first 20 docs and listen to it -- [limit: 20]. Then when I reach end while scrolling, I do this another query for next 20 docs [startAfter: , limit: 20]. Now it would be awesome if I could just change the limit of what I was listening to in the 1st batch to 40 or maybe change the "end doc" to listen for changes upto. Not send the request, just change what I am listening to.That way, I will only request for 20 docs (20 reads) in 2nd batch and I would be listening to changes in 1st 40 docs. Problem of realtime updating solved! The problem will also be solved if there is a way to just listen to changes in a range of docs without retrieving (requesting for) the docs in the specified range.
This wouldn't work because as he mentioned, the database won't keep the pagination state, so the range [0,40] for a specific query is not something it can do without re-doing the entire query.
for the 3rd option, why not check for changes before pulling everything. if there are no updates on the old data its should just pull the new one and merge it with the one it has locally on cache
Very nice. But how about jump pagination, imagine i'm on page 3, and want to go to page 7, how do i implement that? there is a function to "jump x docs"?
Thanks, I really like your videos. Very clear. One question though: let’s say i page in 20 documents per page and i decide to get the top 60 documents at my 3rd page request. That means the first 40 documents are probably in my local cach alrery ?righ?. If that is the case, the server would only sent me the last missing 20 and not bill me for the 40 that i already have. Is thatvright? And would the system do that automatically our do i need to configure this local cach somehow? Thank, Rik
How could I implement this idea in a chat according to what it mentions from the minute: 6:40, I am working in a chat and it is ready, but if I limit the number of messages in the first load and through a scroll I want to access data Previous, I miss the realtime, if someone has a very helpful example, thank you.
I have a question. If my documents update index real time, What happen if I separate my collection to 2 collections ? First collection keep data and second keep only index but same key. Then I query first 500 keys from second collection and keep in array call "keys_array". Then I query data from first collection that same key in array 1 by 1 every 20 time when user scroll to the bottom of page. So I can manage pagination by myself and documents will not swap order in user's screen. And if this work, it can make numerical paginations. Because only index that's a number is more low cost than full data of document.
You can cache previous results in your store or what have you. But yeah, if there's a possibility that the user can request say page 20 right off the bat, like say in a forum board app, you'd probably have to work around this. Maybe adding an index to the document so that you can specifically query for document with index > 20 for example.
This would be expensive because the offset query for 20 records (limit: 20 offset: 40) that you would need for true oldschool pagination would cost 60 reads. Wait, let's say you have 5 pages with 10 records each. In your first query, you must retrieve all first and last records of each page. Then you can navigate with 'start before' and 'start after'. Hmm..
after watching this, im just not sure wether to paginate at all.. if you orderBy: date will it still return weird order? can you at least show some best practice code examples?
@@nivaldolemos5280 I cannot believe this isn't a capability. Oh my god. What the h*ll... I might need to migrate to a completely new database platform...
Well, if the reads from database is scalling to the loaded documents+new documents instead of just new documents (already loaded documents not need load again) then is lower than the original recyclerView and RecyclerView FirebaseUI... this is very bad idea (you will lost realtime updates and your reads from database increases exponentially) .-.
Hi! I have approximately 4.000 records on my firestore and I am paginating it by 20.But what if I am at the beginning and I want to search for an item that hasn't been loaded yet?How would I handle this situation?Thank you!
An earlier video in this series explains how security rules cannot be used to limit the amount retrieved. If they are in place and a query is made that would be limited by a security rule, the query will fail.
offset is bad for security. Giving your users the power to read a million documents with a slim query is a leverage point for attacks. And this has nothing to do with Firebase. Limit/skip pagination for arbitrary queries must read all previous documents, and cursor-based ones don't. The only thing is that if you manage your own database this isn't a billing problem, but Firestore is billed per-read.
I tried this with streambuilder and I don't recommend, If you just can load the documents, add in a list and show, It's better, that's how I doing on my app and it's working fine.
so you are charging on every single document read, that's so bad when I just want to have a total number of my documents and you charge me like I am doing 1000 read just because I have 1000 documents
keep a counter that you increment decrement on every document add/delete respectively. Then you will have the count in one read and you can do as many as you want
This tutorial is misleading. There is no pagination. As after wasting half a day, there is still a very hard thing to do to just get the items with index 20 to 40 as they are in a document.
You can build a Cloud Function which cleverly counts the number of documents and saves it somewhere in the database. Keep in mind to deduplicate possible duplicate function executions.
A downloadable Xcode sample project would have been nice.. You spoke on how to use it BUT didn't show the specifics concerning snapshot/listener code implementation... Disappointed.
Awesome but I would prefer either limit star, end or after row_id. That way I can see from code what's going on and save on execution of extra code. May be I am too old fashioned :) Awesome, I had no clue that offset would charge me for all reads I am not actually getting. Another warning that developer should read documentation to get the best results for lowest price :)
This video has opened the way for me, I am doing geospatial queries with geofirex, github.com/codediodeio/geofirex, but it is not possible to page and at the moment it does not have a limit method, I could see in its explanation that the way, although it is expensive would be to increase the limit in each query ... This library does have the ability to limit queries https: //github.com/geofirestore/geofirestore-js ... There is a better alternative for geospatial queries for javascript?
![Lp. Сердце Вселенной #60 РОЖДЕНИЕ ЛОЛОЛОШКИ [Финал] • Майнкрафт](http://i.ytimg.com/vi/YoR0pAV9FVQ/mqdefault.jpg)
Amazing course, good job Todd and Firebase crew! I watched all 7 videos published so far in a single morning and I don't feel lost anymore!
these videos are really really good and instructional, thanks for them
Thank you for the feedback! We aim to provide quality content 😎
Brilliant, and answers real questions with real limitations (eg costs) , please keep creating.
Remember that incremental (limit) pagination breaks if you have persistence on, queries might trigger from old cached values. Say you query for 10 elements, you might get 2 element from older cache.
The Firestore videos series is awesome. Its nice to have a cool dude make you laugh while you learn about underlying technology. Great content and very helpful. Keep sharing please :D ;)
I loved this gummy big bear intro haha
If I increase the limit from 20 to 40 to 60, do I really fetch also the first 20 documents all the time?
I thought that the Firestore client SDKs cache these documents and send me those documents only if it's needed.
Or is it just that I get billed for those, but Firestore tells the client SDK that it already has the most recent version of the first 20 documents?
Great question
This is something i struggle with as there are just hints in the document of how caching works and when you will benefit from it. I can do tests and sometimes I do to try to figure if a query is expensive, but it would be easier to remember if I could get a lesson on it. Please make this a future topic for an episode or add a section devoted to it in the documentation.
@zztop3000 Yeah I know that Todd says that you are re-requesting the old data, but this confuses me.
Why should the documents be fetched again? They are already present in the cache of the app.
Or do I just get billed for those documents even though they don't get sent to the client?
Because offline persistent is not enabled by default everywhere
@@ToddKerpelmanCorp One follow up question. If with the offset you are basically just querying and charging for every record up to the offset, then what is the point of having that parameter at all? You could just request all the records with limit and that would be the same. I always thought of it like pagination and avoiding requesting and charging money for these records. This is actually a huge thing that should be in the docs.
Some people tend to say awesome for just 'ok' stuff. But this playlist is really... awesome :) The content, the presentation, the animations. Everything is top quality. Keep up the good work!
this video is great in explaining the theory behind, but where are the code examples? why is this not in the angularfire docs?
So there is a read charge even clients app cached the same document data
Yeah that seem really confusing, in the third video of the serie they said just the opposite so what's the truth?
I think if you want to make this work you need to read the whole collection with attached listener so when anything changes only the change will count as new read
Another way is order query result by timeStamp descending or ascending, then retrieve document according to document timeStamp
This is a good input. Wonder why Todd didn't add it
Thx for the simple show of options and associated tradeoffs . PS - where on Earth did you find such a enormous gummy bear? Must have doubled this episode's production costs...
Thanks...very informative
Thanks for this! I would love a video that suggests a schema for commenting on user's posts. It's a little tricky to set it up so it's safe permission wise, but also logical query-wise.
Hey Todd, are you available to do the instructional videos for our app🤣? Seriously, another great video. I haven't got up to thinking about pagination yet but I will soon and this explains pretty much everything I need to know.
I don't think Google has ever used Firebase for any of their own project because of how many basic things it seems to lack so casually.
It is great to see a video on pagination best practices. Thank you!
Thanks. Would be great to have a new video focusing more in pagination for realtime cases, like chat apps, for instance. It's kind of complicated to find help online for a consistent way to implement pagination with real-time updates with Firestore
Best teacher end of story
did the api change ? nothing of this works currently
Giant gummy bear analogy: spot on.
So this means that conventional pagination (which has a nav containing 1 - total page count) is impossible out of the box with Firebase - the only option is to infinitely scroll pagination as described. I cant believe this is no way to do this easily with Firebase. Think ill bite the bullet and move over to AWS
Great outro lol. Any way to order by random order while using pagination? Say if I want to give my user 20 random reviews in each batch?
@@ToddKerpelmanCorp Gotcha, I think I'll end up using a cloud function to periodically update the random fields. Not a perfect solution, but cheaper than other options that I can think of
Brilliant as ever
great video; 2:31; does firestore support query on one field and order by another field now? just wondering if it now supports this what I love to hear or is it just a makeup example to simplify the video? thanks #askfirebase
Please make your videos as frequently as possible. It's hard to find good Firestore tutorials or courses online.
@@ToddKerpelmanCorp Thanks for writing back!! You are GOOGLE! There's no such thing for you guys that's tough. If you put your will to it.. I guess you'll get it done. Try using softwares like Video Scribe. They are fast and easy to make yet crisp and useful for educational purposes.
great video - thanks - would be great to see a result count for the query as well, though I understand it's not easy in firestore
This pagination this is really cool, not tedious and very efficient. Nice work Google Firebase Team!
you missed one thing, to paginate without infinite scroll we need to know the amount of records the search has, so we can know the amount of pages, how we can do that?
use a proper database ;)
Firestore is crap, specially the web SDK.
Any supposed gain in productivity vanishes when you reach such simple and common use case.
In a SQL database, it's terribly simple.
Hi Todd, was expecting your video after the GA on the enhancements to Firestore and things to lookout for. Any videos in the making? 🎬
Great explanation! Thanks
@Todd There is a nice idea for you! What if we can extend range of values to listen to while paginating. For example, I do this query for first 20 docs and listen to it -- [limit: 20]. Then when I reach end while scrolling, I do this another query for next 20 docs [startAfter: , limit: 20]. Now it would be awesome if I could just change the limit of what I was listening to in the 1st batch to 40 or maybe change the "end doc" to listen for changes upto. Not send the request, just change what I am listening to.That way, I will only request for 20 docs (20 reads) in 2nd batch and I would be listening to changes in 1st 40 docs. Problem of realtime updating solved!
The problem will also be solved if there is a way to just listen to changes in a range of docs without retrieving (requesting for) the docs in the specified range.
This wouldn't work because as he mentioned, the database won't keep the pagination state, so the range [0,40] for a specific query is not something it can do without re-doing the entire query.
Thank you for sharing this informative video! I hope you are well: Todd & Firebase Team!
I believe the key essence here is, the option you choose is dependent on how real-timeliness you want your app to be!!
How to count documents without being charged
for the 3rd option, why not check for changes before pulling everything. if there are no updates on the old data its should just pull the new one and merge it with the one it has locally on cache
Very nice.
But how about jump pagination, imagine i'm on page 3, and want to go to page 7, how do i implement that? there is a function to "jump x docs"?
Any updates? I am wondering the same thing?
When are we getting that video on cloud functions that was promised in #5...?
Thanks, I really like your videos. Very clear. One question though: let’s say i page in 20 documents per page and i decide to get the top 60 documents at my 3rd page request. That means the first 40 documents are probably in my local cach alrery ?righ?. If that is the case, the server would only sent me the last missing 20 and not bill me for the 40 that i already have. Is thatvright? And would the system do that automatically our do i need to configure this local cach somehow? Thank, Rik
Where doesn't seem to work with pagination. Is there a way to combine a where clause and pagination?
what if i use streams and streambuilder and list.view builder, do i still need to paginate ?
How could I implement this idea in a chat according to what it mentions from the minute: 6:40, I am working in a chat and it is ready, but if I limit the number of messages in the first load and through a scroll I want to access data Previous, I miss the realtime, if someone has a very helpful example, thank you.
Same idea but the query must be for ascending timestampe
Where can I get an edible bear like in the video?
Would U recommend to combine aggregation with limit, or go with one or the other?
I have a question.
If my documents update index real time, What happen if I separate my collection to 2 collections ?
First collection keep data and second keep only index but same key.
Then I query first 500 keys from second collection and keep in array call "keys_array".
Then I query data from first collection that same key in array 1 by 1 every 20 time when user scroll to the bottom of page.
So I can manage pagination by myself and documents will not swap order in user's screen.
And if this work, it can make numerical paginations.
Because only index that's a number is more low cost than full data of document.
Nice video but what about a pagination for webs likes many blogs have with the next and previous pagination?
You can cache previous results in your store or what have you. But yeah, if there's a possibility that the user can request say page 20 right off the bat, like say in a forum board app, you'd probably have to work around this. Maybe adding an index to the document so that you can specifically query for document with index > 20 for example.
This would be expensive because the offset query for 20 records (limit: 20 offset: 40) that you would need for true oldschool pagination would cost 60 reads. Wait, let's say you have 5 pages with 10 records each. In your first query, you must retrieve all first and last records of each page. Then you can navigate with 'start before' and 'start after'. Hmm..
Really nice, super useful!
super cool and useful!
Great video!!! thanks
after watching this, im just not sure wether to paginate at all.. if you orderBy: date will it still return weird order?
can you at least show some best practice code examples?
No! You can go ahead with this: 5:00
This will be a big issue for most of the companies using Firestore, I was NOT expecting this.
how?
Most enterprise apps will need a numeric pagination somewhere, something that Firebase cannot offer. You can only travel to the previous or next page.
@@nivaldolemos5280 I cannot believe this isn't a capability. Oh my god. What the h*ll... I might need to migrate to a completely new database platform...
How do we know how many pages have using this method?
Hi, I have a problem with limit, It always bring 20(for example), when only it is supposed to bring 10 elements. Yes ther another 10 are duplicateds
nice
What about scrolling back up?
Well, if the reads from database is scalling to the loaded documents+new documents instead of just new documents (already loaded documents not need load again) then is lower than the original recyclerView and RecyclerView FirebaseUI... this is very bad idea (you will lost realtime updates and your reads from database increases exponentially) .-.
Could RxFire stream handling paginate?
Wow sorry guys but this is terrible, we need proper paging...
1:40 Infinite Scroll with React Native and Cloud Firestore. How do I this?
Yes
@@fueledbycoffee583 This gets tricker when you want it in realtime
Hi! I have approximately 4.000 records on my firestore and I am paginating it by 20.But what if I am at the beginning and I want to search for an item that hasn't been loaded yet?How would I handle this situation?Thank you!
Query for it !
How to know that it was the last document in the pagination fetch? Lets say I have 100 items and limiting by 20.
@@ToddKerpelmanCorp Thanks!
How can we achieve the pagination in PHP language.
Is there a security rule to limit the amount retrieved?
Yes we can limit it in security rules
An earlier video in this series explains how security rules cannot be used to limit the amount retrieved. If they are in place and a query is made that would be limited by a security rule, the query will fail.
whereEqualTo cannot be used with orderBy!! this video has mistake at [7.57]
Thanks man!
#askfirebase Does the Firestore SDK support something like SELECT * from documents ORDER BY random() LIMIT 1?
it does, but by using bunch of functions provided in the api.
why does everything feel like a one big hack with Cloud Firestore?
Because it is. But, well, "it scales".
hmm, use start instead offset. got it
Bring 'offset' to the front end!!!
offset works, except you, are charged for the docs you skipped. limit(5).offset(10) counts as 15 reads. once again the genius of firebase at work!
offset is bad for security. Giving your users the power to read a million documents with a slim query is a leverage point for attacks. And this has nothing to do with Firebase. Limit/skip pagination for arbitrary queries must read all previous documents, and cursor-based ones don't. The only thing is that if you manage your own database this isn't a billing problem, but Firestore is billed per-read.
Amazing
How can we implement this on stream builder?
I tried this with streambuilder and I don't recommend, If you just can load the documents, add in a list and show, It's better, that's how I doing on my app and it's working fine.
Imagine that gummy bear made out of rum
how do phone otp authentication using react native and firebase please tell me
so you are charging on every single document read, that's so bad when I just want to have a total number of my documents and you charge me like I am doing 1000 read just because I have 1000 documents
keep a counter that you increment decrement on every document add/delete respectively. Then you will have the count in one read and you can do as many as you want
@@noumanhanif1948 that doesn't account for any filters that your user applies in a search/filter bar
LMAO the gummy bear
How to get the last 50 docs ?
Just reverse the sort. Switch ascending with descending or vice-versa
This tutorial is misleading. There is no pagination. As after wasting half a day, there is still a very hard thing to do to just get the items with index 20 to 40 as they are in a document.
#askfirebase How to count without getting all document?
You can build a Cloud Function which cleverly counts the number of documents and saves it somewhere in the database.
Keep in mind to deduplicate possible duplicate function executions.
not possible
Brain: you are supposed to keep watching this tutorial
Me: I WANT THIS GIANT GUMMY BEAR
Same
i just hope that firebase would find a solution for better pagination and dynamic queries, but as of now it's not one of the best for obvious reason.
Brilliant
UA-cam team: What is the budget of UA-cam video?
Google:
UA-cam team: Lets buy a giant gummy bear.
That's how it was done
Awesome
A downloadable Xcode sample project would have been nice.. You spoke on how to use it BUT didn't show the specifics concerning snapshot/listener code implementation... Disappointed.
What is `whereField()` ? :D why not just `where()` ?
I still dont know how to do it , I only learned how not to do it
i want that gummy bear
Awesome but I would prefer either limit star, end or after row_id. That way I can see from code what's going on and save on execution of extra code. May be I am too old fashioned :)
Awesome, I had no clue that offset would charge me for all reads I am not actually getting. Another warning that developer should read documentation to get the best results for lowest price :)
I don't have Money
this is the most confusing function of firestore .....
wow, startAfter instead of offset. Bet loads of people are using offset and wasting $$.
lots of limitations...., poor methods, video is old....
FirebaseUI Pagination not can capturate the Snapshot Document
This video has opened the way for me, I am doing geospatial queries with geofirex, github.com/codediodeio/geofirex, but it is not possible to page and at the moment it does not have a limit method, I could see in its explanation that the way, although it is expensive would be to increase the limit in each query ... This library does have the ability to limit queries https: //github.com/geofirestore/geofirestore-js ... There is a better alternative for geospatial queries for javascript?
broo I spent a lot of time "paginating" as documentation says then I realised that I can use 'offset'... dammit
Maybe you didn't listen till the end... You get billed even for the documents you are supposed to offset by, which defeats the purpose
@@sudomakes ikr, hope that guy read your comment, or his firebase app will give him A BIG CHONKY bill of 500$ one day... xD
Please add getLength() method to collections. Otherwise, pagination feels like walking down stairs blindfolded