Real-Time Streaming Project with Smartphone Sensor Data , Kafka , Rest API & Python
Вставка
- Опубліковано 14 жов 2024
- If you are looking for free real-time or streaming data sources , then you are in right place!
This video explains how to stream data from sensors on your phones using an app, Sensor Logger & ingest them in Kafka Cluster for Processing.
Install Sensor Logger from the below link:
apps.apple.com...
Prerequisite:
-------------------------
Creating a Serverless Apache Kafka(MSK) publisher using AWS Lambda
• Creating a Serverless ...
Code:
--------------
github.com/Sat...
Learn Apache Kafka form scratch
• Apache Kafka for Pytho...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.co...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY UA-cam CHANNEL
Amazing tutorial & demonstration. The key issue with other tutorials is that they focus telling the theory part or at max with dummy data. You have overcome that with this tutorial so that we can see real time data generation and consumption. Thank you very much for all your efforts that you put in !!
Thank you Gaurav Srivastava for your inspiring comment ! Happy Learning
Asombroso
Very informative. Can you make a next video on continuenig this video as how to:
1) Process
2) Query
3) Visualize
This data that we captured. I think you can make a series on it, it is my small suggestion you can add it more.
I like the suggestion. Would appreciate it if this is demonstrated.
@mr.mechanic7170 , Thank you for your suggestion , I have noted in backlog , Stay Tuned for updates 😊
@@gauravsrivastava3884 I have noted in backlog , Stay Tuned for further updates 😊
@mr.mechanic7170 , @gauravsrivastava3884 based on your suggestion , tried to create an application oriented video with Accelerometer Sensor Data -- ua-cam.com/video/QqEgI-Lslqc/v-deo.html Hope you will like this project 🙃
@@KnowledgeAmplifier1 I appreciate my brother. I hope you will provide us such a valuable information in the future as you always do.
Thanks for the great lecture!!!
following your steps but using MSK, I have observed multiple lambda timeout cases during streaming, which means some data failed to arrive to kafka. Added some logs for troubleshooting the lambda function, it shows "data send success" but not receive status code 200 from kafka. In addition, it shows that 'Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms'. Do we need to add some code to close producer in lambda function before it is timed out? otherwise there may be some dangling connections still keep open and hold on kafka resources. Thanks!
Good content!!
But when i tried i am getting below error in the logs:
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'kafka'
Traceback (most recent call last):
i have added the layer as provided by you!!
could you please help here.
Hello Girish Depu, Lambda Layers are specific to the Python version they are configured with. Ensure that your Lambda function is using the same Python version as the one the layer was created for. Like in my case , I created the Lambda Layer for Python 3.8 , so I have chosen 3.8 execution env for the Lambda Function ..
can u make a video about streaming magentic sensor from android
Awesome bro,learned a lot,thank you 💯
I'm glad to hear that you found the tutorial helpful and learned a lot!!
what are the important topics required for aws data engineering?
Hello Avinash, there are several important topics that are required for AWS data engineering. Here are some of the key areas of focus:
AWS Services: Understanding the core AWS services is crucial for data engineering on AWS. This includes services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, and Amazon EMR.
Data Storage: A good understanding of data storage options on AWS is important. This includes understanding of Amazon S3, Amazon EBS, Amazon EFS, and Amazon Glacier.
Data Processing: Being able to process large amounts of data is critical in data engineering. AWS provides a range of data processing tools such as AWS Glue, Amazon EMR, AWS Lambda, and Amazon Kinesis.
Data Warehousing: AWS offers several options for data warehousing, including Amazon Redshift, which is a fully-managed, petabyte-scale data warehouse service.
Data Streaming: AWS provides several services for real-time data processing and streaming, such as Amazon Kinesis and AWS Lambda.
Analytics: Understanding how to extract insights from data is key to data engineering. AWS offers services such as Amazon Athena, Amazon QuickSight, and Amazon EMR to help with analytics.
DevOps: Data engineering on AWS requires a good understanding of DevOps practices, such as continuous integration, continuous delivery, and infrastructure as code.
Security: Ensuring the security of data on AWS is important. AWS provides several security services such as AWS Identity and Access Management (IAM), AWS Key Management Service (KMS), and Amazon CloudWatch.
Machine Learning: AWS provides several machine learning services such as Amazon SageMaker, which is a fully-managed service that provides developers and data scientists with the ability to build, train, and deploy machine learning models.
These are some of the important topics that are required for AWS data engineering. However, the field of data engineering is constantly evolving, so it's important to stay up-to-date with the latest trends and technologies.
Hope this will be helpful.. Happy Learning
Please provide big data engineering end to end project with involving all components of big data with explaination,data and code
Hello tanakam venkata naresh, here are some end-to-end projects already uploaded in this channel --
1)Advertisement Impact Analysis
-------------------------------------------------------
If you are running marketing campaign then one of the major aspect is analyzing how the campaign is impacting for example, what's the message/email open rate , what's the link click rate (if link is attached in message body)
Let's build that Real Time Campaign Analysis Pipeline which can help the Marketing Team to evaluate the performance
ua-cam.com/video/ifw4NGiLOYs/v-deo.html
2)Nearest ATM/McDonald Finder
------------------------------------------------------------
Have you tried any bank website to look for nearest ATM from your current location?
Or may be nearest McDonald Location?😋
If yes , let's explore how that works and help in location based search with AWS Services
ua-cam.com/video/y_R7SljUUT4/v-deo.html
And you can make lot of other projects using the knowledge gained from above 2 ...
Hope this will be helpful .. For more interesting projects stay tuned ... Happy Learning ✌🏻😊
I tried for 2 times and follow exactly same step by step but its not showing the logs into the kafka consumer. What can be the reason???
Hello Abrar Pinjari, please check AWS Lambda Logs once in cloudwatch , add print statement in each line to check where the error is getting started , if still not sorted out , then instead of fire & forget based data write from producer , try with synchronous or asynchronous methods ...
@@KnowledgeAmplifier1 Thanks for your response. But the code that you mentioned in the script is right???
Because it is showing this "{"message":"Not Found"}" after clicking on "API endpoint"
I think it had to show 'Hello from Lambda!'
Is it right or not???
@@abrarpinjari6809 ,if no routes match a request, API Gateway returns {"message":"Not Found"} to the client. Please check once whether along with API Base URL , you have provided the correct resource path or not (you can refer the 19:00 timestamp for details)..
@@KnowledgeAmplifier1 Thanks brother for your kind support, Finally it works.
@@abrarpinjari6809 glad to hear that ! Happy Learning
Thank you sir!!!!!!!!!
You are welcome Manish Pandey! Happy Learning
How can do that in Azure ?