DevOps Real-time Project #2 - How to Stream Logs to Azure Blob Storage

In this video, I will show you how to stream your AKS(Kubernetes) logs to Azure Blob Storage using Vector( A free tool by DataDog).
What is Vector? 🤷‍♂️
Vector is a data pipeline solution that helps collect, transform, and route logs, metrics, and traces to any cloud-native service or third-party application.
📌Below steps were performed in this video:
- Login to your Azure portal and provision an AKS cluster through the Azure portal or Azure CLI.
- Once the AKS cluster is up and running, log in to the cluster using the Azure CLI, create a test namespace, and deploy a sample Nginx deployment.
- Create an Azure blob storage account and container called “akslogs” to store the logs.
- Prepare a Vector configuration file called “vector-config.yaml” by adding the storage account connection string and other relevant fields, allowing Vector to access the storage container and stream the logs.
📌By the end of this video, you will learn -:
◾ What is Vector ( By DataDog)?
◾ How to Provision Azure resources such as resource group, aks cluster, storage account, etc,.
◾ How to create vector configuration to stream logs.
◾ Different sections in the vector configuration file.
◾ How to get your first DevOps job as a fresher?
◾ Test the setup.
VECTOR CONFIGURATION YAML 👀
github.com/piy...
𝗖𝗢𝗠𝗣𝗟𝗘𝗧𝗘 𝗧𝗨𝗧𝗢𝗥𝗜𝗔𝗟 𝗦𝗘𝗥𝗜𝗘𝗦 👌 It’s 🅵🆁🅴🅴 ⛳
Namaste Google Cloud | GCP Tutorial for beginners, a complete playlist for Associate Cloud Engineer Certification Exam:
• Namaste Google Cloud, ...
DevOps Tutorial for beginners
• DevOps Tutorial for be...
AWS tutorial for beginners
• AWS (Amazon Web Servic...
Azure Data fundamentals DP 900 full course
• Microsoft Azure Data F...
Jenkins Tutorial for beginners
• Jenkins Tutorial For B...
DevOps and Cloud Podcasts:
• DevOps and Cloud Podcasts
Subscribe to our channel to get notified about the latest videos.
🔥 / @techtutorialswithpiyush
Connect with me 👋
🌎LINKEDIN: ►
/ piyush-sachdeva
🐦TWITTER: ►
/ techie_piyush
Join our learning communities 👋
/ 1015771332531944
/ 12561913
References 📚:
/ streaming-aks-kubernet...
vector.dev/doc...
vector.dev/doc...
vector.dev/doc...
vector.dev/doc...
#kubernetes #devops #aks #devopsproject

[@pradyumna668]

Hi Piyush,
Thank you for the video can you please let us know how to refer the connection string using secret or how to bring it from keyvault

[@TechTutorialswithPiyush]

I dont think, Vector supports keyvault integration or using secret for connection string
vector.dev/docs/reference/configuration/sinks/azure_blob/#connection_string.
You can use an automated process, using a pipeline that uses the vector config as a template file and replaces the secret from azure key vault.

[@pradyumna668]

@@TechTutorialswithPiyush thank you for the confirmation Piyush ji

[@MatthewKennedyUK]

Can you do another video setting up the vector agent and stateless aggregator separately?

[@TechTutorialswithPiyush]

Will do if more people request it

[@phanindrachitta]

Excellent

[@TechTutorialswithPiyush]

Thank you so much 😀

[@YouTubers-rj9xv]

Bro super teacher

[@TechTutorialswithPiyush]

Thank you so much bro :) I am glad that you found it helpful.

[@user-gs5xl9zl9o]

Thank you for this tutorial, very clear. However, my connection string doesn't work, I don't know why :(
ERROR vector::topology::builder: msg="Healthcheck failed." error=Invalid connection string specified component_kind="sink" component_type="azure_blob" component_id=azure_blob component_name=azure_blob

[@TechTutorialswithPiyush]

Hi Rachel, Thanks for the feedback! Regarding your query, Looks like you have used the incorrect connection string, make sure to use the connection string in your vector config.yaml file from Azure storage and not the access keys. Here's the blog as well that you can refer quickly and see if you missed anything
medium.com/@piyush.sachdeva055/streaming-aks-kubernetes-logs-to-azure-blob-storage-using-vector-a-step-by-step-guide-e3044aa92d57

[@dr.hemantchauhan2613]

Good one

[@TechTutorialswithPiyush]

Thank you 😊🙏

[@user-jb9ml3xg6k]

but issue is that sir ..... after interval time 7 or 8 min new log is generating and it is not append to a exiting on same log file ....pls help on that issue

[@TechTutorialswithPiyush]

Hello, As per the documentation, the default maximum file size is 102400 bytes before it is truncated and create a new file. You can customize it
vector.dev/docs/reference/configuration/sources/file/#max_line_bytes

[@user-jb9ml3xg6k]

@@TechTutorialswithPiyush correct sir but my blob log file size is 2kb only then why is it not append new log data on same blog log file ..

[@user-jb9ml3xg6k]

sources:
my_source_id:
type: kubernetes_logs
auto_partial_merge: true
max_line_bytes: 600000000

[@user-jb9ml3xg6k]

need help sir

[@TechTutorialswithPiyush]

@@user-jb9ml3xg6k It might be the issue with the vector version you are using, can you try using a newer or latest version?

[@AjayKumar-lm4yr]

How to store Grafana Loki logs in Azure Blob Storage

[@TechTutorialswithPiyush]

I dont think Loki/Grafana is one of the supported source by Vector
vector.dev/docs/reference/configuration/sources/