Good explanation. Can I use Spark streaming to read from hive table with millions of records. Say my usecase is like I have a source java program which uses Spark stream to read data from hive table and keep on posting data packets to some kafka connector. I am mainly concerned with memory usage. What is your opinion on this ?
Hello, As you mentioned on streaming source-File. Is it possible to stream a file as source and as and when new lines append to a file(csv file), is it possible to do streaming on top of that? if yes.. which streaming source method i need to call...please help
in Dstream in one batch you get only one RDD, check below link stackoverflow.com/questions/35164634/how-many-rdds-does-dstream-generate-for-a-batch-interval
this explanation was very good, thank you sir
Excellent explanation. Keep up the good work and thanks !!
Thanks.. for crystal clear explanation 👍
Good explanation. I didn’t find batch streaming and sliding windows concepts explained like this.
And I am
Excellent job you doing Lime Guru. Thanks for the effort . Expect more on Apache Spark
Good explanation 😍
Good explanation.
Job well done. Nicely explained. This video deserves more views and likes. Kudos.👍👍👍
How to create state again and add it in rocksDB when we restarts our stream job from new checkpoint location?
Thanks too much
Nicely explain
Good explanation. Can I use Spark streaming to read from hive table with millions of records.
Say my usecase is like I have a source java program which uses Spark stream to read data from hive table and keep on posting data packets to some kafka connector.
I am mainly concerned with memory usage. What is your opinion on this ?
Good job
Hello, As you mentioned on streaming source-File. Is it possible to stream a file as source and as and when new lines append to a file(csv file), is it possible to do streaming on top of that? if yes.. which streaming source method i need to call...please help
in Dstream in one batch you get only one RDD, check below link
stackoverflow.com/questions/35164634/how-many-rdds-does-dstream-generate-for-a-batch-interval