CICD Pipeline to deploy Kubernetes Applications using Terraform, EKS, and Jenkins
Вставка
- Опубліковано 21 сер 2024
- This video walks you through the steps of creating a CI/CD pipeline using Jenkins. Here are the steps we followed:
Jenkins-pipeline-deploy-to-eks
Project Name: End-to-end deployment of Applications to Kubernetes using a Jenkins CI/CD Pipeline
#Steps for the project
1. Create a Keypair that matches your keypair
2. Create a Jenkins Server with all the dependencies, libraries and packages needed.
3. Once completed, access the Jenkins server and set it up
4. Run the Jenkins-pipeline-deploy-to-eks to create Kubernetes Cluster, create deployments and Services
5. Test that the application is running.
6. Destroy infrastructure
Here is the link to access the repositories where the code is found:
Jenkins pipeline repo: github.com/Pri...
Jenkins server infra: github.com/Pri...
#devops #jenkinspipeline #kubernetes #aws #innovation #infrastructure #terraform
Wow, this video is absolutely fantastic! 🌟 The way you explained the concepts was incredibly clear, and I really appreciate the effort you put into providing such valuable content.
Thank you so much for your kind feedback Isabella, and thanks for watching.
Very clear and articulate explanation
Thank you Sir Tone!
Detailly explained. Ama definitely gonna give a try following this tutorial.
Thank you sir!
Thank you so much, Solidman; we appreciate your comments and Kudos as you develop your cicd pipeline.
Perfect man! this is what I was looking for, followed your step and just did it. trying to work around a custom app and deploy different environments like staging, and testing a rough idea would be great. Thanks a lot for saving my week.
You're welcome Noor, we're glad to help. Yes, you can push to the different environments on a merge to each of the environments. we appreciate your comment sir.
thanks pro!
Bravo 👏. This video came right on time. Thanks for your time and for putting up such rich content .
Thank you, Anim, we really appreciate you watching. Thanks for your comment.
Very detailed video, keep it up!!
Thank you Kirtee, appreciate your comment.
Thank you for the excellent tutorial. There is one error of note (in 'jenkins-server-script.sh').
ERROR: "public key for jenkins-2.387.3-1.1.noarch.rpm is not installed"
- This prevents Jenkins to be installed and running. I spent some time troubleshooting this. Here is the fix I found:
__FIX__
Replace:
- sudo yum install jenkins -y
with
yum install jenkins --nogpgcheck
I hope this helps somebody. Cheers... - Joseph P.
Thank you for taking time to fix this Joseph, appreciate it. Amazing!
Absolutely amazing
Thank you sir Pascal.
Insightful and detailed explanations.
You're welcome, Sir Brian.
Thank you so much, Brian; we are glad you loved it.
Much appreciated your efforts
You're welcome kbrajeshwaran .
You are doing great work bro!
Thank you Elvin, appreciate it.
Thanks for putting the work together for everyone to gain some knowledge. I wasn't able to deploy to EKS from my Jenkins Pipeline. It kept throwing an error that kubectl command can't be found in the temporary directory created in the Jenkins workspace, but I have kubectl installed with execution permission. I didn't create the cluster with terraform module.
Thanks for your comment Adedapo, could your reach out at contact@primuslearning.org. We can create sometime to look at your pipeline.
it was a very interesting insightful in this video but it will be very helpful if you could help on below:
1: You created a New VPC and instance via terraform as per video time 14.30, but suppose we have a VPC name main-dev in a AWS account and in the same VPC we install multiple cluster, ec2 etc already. so how we can use existing VPC and attach desired IAM role, Security group etc.
2: How we can install multiple instances in same .tfstate file in same VPC
You can actually get the data from an existing resource and use in terraform. And also pass multiple resources in terraform.
Awesome!👏
Thank you Speeroone.
Bravo 👏
Thank you Mahesh
Excellent tutorial. My recommendation is to use Python or shell script to organize and run terraform code, and just use Jenkin for your CI/CD pipeline. If you could demo a simple application with the CI/CD pipeline is even better, in which you can include web hook, unit test, security scan, smoke test, etc. in the pipeline, and independent of infrastructure code - terraform. Look forward to your next tutorial. Thanks,
Thank you for your suggestion Mikedqin, I will work on a video that include some of the tools mentioned above.
Thank you sir!
You're welcome Mateus.
Ride on. Good Job
Thank you Jonelle's Fun World, really appreciate your comment.
Thank you so much for your video .Could also do a video to show how to create all terraform files you used to deploy the cluster? please
Thank you for watching Yawotentegue, yes I will work on a video demonstrating how the terraform configuration files are created.
Awesome and keep on
Thank you Andrew, I appreciate it.
Thank you for this
You're welcome Abdul, thanks for watching.
Thanks for your tutorial! :)
Just a small question in the Jenkins part:
In your JenkinsFile I see you define the variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, but you never use it to execute the "sh "aws eks update-kubeconfig --name myapp-eks-cluster". Can you telle me pls why or if I am missing something? :)
Because normally we use "withCredentials( )" function to use those credential Variables inside JenkinsPipeline.
Or is it enough just in the ec2 instance add the credentials manually using "aws configure" in the console? :)
Thanks again for your support! :)
Thanks for watching and thanks for your comments; really grateful. What we did with the credentials is we enabled global credentials in Jenkins where we passed the values for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. This was just to allow the pipeline to be able to connect to AWS and have the necessary access.
🤜🤜
Thanks Vishnu, appreciate it.
Quick question, did you have an already created cluster prior to running the create eks cluster pipeline? I am getting No cluster found for name: calcapp-eks-cluster.
Hi Evanson, the Jenkins pipeline that you run will create the EKS cluster, notice the code here: github.com/Primus19/jenkins-pipeline-deploy-to-eks/blob/main/Jenkinsfile and notice the terraform code here: github.com/Primus19/jenkins-pipeline-deploy-to-eks/tree/main/terraform. In the jenkinsfile, there is a stage that creates the cluster.
@Primus after creating nginx application, do we need to use terraform destroy only once in terraform directory? Or do we need to apply once again in k8 directory? If we want to use destroy in multiple directories, what is the order of destroying? Thanks in advance
Hi Ravi, you will need to destroy just once in the terraform directory and everything should be gone. Thanks for watching.
Hey Bro , Awesome work here , really appreciate the content , have a doubt in terraform part that how terraform will come to know about my aws account where it has to deploy ec2 , subnet, etc , also can we use default vpc or vpc is also required ,
You will have to configure your aws account to use access keys and secret access keys. You could use the default vpc for this.
Thanks , will give it a try for sure
Thank very much for this quality teaching but I am stuck in deploying my app. Here is the complain "Unable to connect to the server: dial tcp xx.x.x.xx:443: i/o timeout" please what is the way out?
Hi Prince, thanks for watching. There could be a couple of things - the correct ports not open. Did you grant the correct permissions to the user whose Access and secret access keys you are using? Check that and reconfigure it.
@@primuslearning4235 Please sir, check your mail.
Thanks very much realy helpfull. Also I didn't find your repo in git hub? So that i can get the code for me to practice too.
Thank you for watching Leonard, sorry about the repo, I uploaded just one before someone pointed it to me. I have uploaded both now, please check the video again, you should have access to the second repo.
For some reason my LoadBalancer can't be created, EKS creation and kubernetes deployment return success, but mu LoadBalances is not created. Google says that there is problem with public/private subnets. Do we assign private subnets to our LB ? Or is it publicly open for anyone ?
Error shown on AWS EKS deployment section:
Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB
Did you edit the cidr ranges we used. Please recreate it with the same blocks that we used and see if it solves it.
Thanks for your effort .The jenkins-pipeline-deploy-to-eks is trowing some error and pipeline is failling
Hi, could you include the error here to help with debugging.
What is the role and responsibilities of this project sir coz I want to add in my cv
The roles and responsibilities include Building CICD pipelines using Jenkins, deploying applications to EKS, deploying Terraform code through Jenkins, etc.
Also i'm not able to initialize the backend even after providing credentials for accesskey and secret key.
i did not find anything in video where to and how to do.?
You did not change the bucket names in the backend file, please change them to a bucket in your aws account before initializing.
@@primuslearning4235 ohh yesss I was able to figure out.!
@@primuslearning4235 but now my Jenkin server is not opening when I'm using the public Ip created by ec2 instance. also i tried to ping from command prompt it is not pinging i have disabled firewall but still no response.
what can be the error.?
@primuslearning4235 I tried the tutorial but jenkins wasn't installed on my server. I see that some others had the same issue. Do you have a resolution for this please ?
Hi, I think we will need to look at the error you have, probably wrong with the permissions you have. Everything in the script looks good to be able to install. What permissions are you using for the IAM User?
Hey, lets also do GitOps and Monitoring. I would like to help and work with you on the project if you agree to.
Thank you, Vrash; we will definitely do something on GitOps and monitoring. Will be glad to look for avenues to work with you.
@@primuslearning4235
I am a student trying to learn DevOps / Platform Engineering from free resources that are available online.
I am looking for internship / junior level opportuinities since 1.5 years now and I applied to countless jobs of which my skills exceeded its expectations, but never got any reply from any of them. Although I have learnt Docker, Kubernetes, Jenkins, Terraform, Golang, Linux and what not.
Would you please help me learn as a mentee under your mentorship ?
I just want to learn with you as I see you as a hope to start my DevOps / Platform Eng. career.
I would be really very grateful to you for anything I could learn under your mentorship.
Thank You
I tried all the steps but my pipeline didn't work
Why? In the console output it says my jenkins file is not on my github repo but i am sure it is there
Hi Adebayo, make sure your jenkinsfile is named correctly and is at the root of the project. Also make sure the path is correct in your Jenkins pipeline. If you follow it keenly step by step you should be able to make it work.
I found out that I had the spelling wrong
Hey, how does the terraform in the second repository work without `providers` file
Hi Vrash, I'm sure you missed it, It has a provider defined.
@@primuslearning4235 Oh yeah, I got it, just forgot to delete the question.
Amazing video, thanks a lot
I just connected my server through ssh ans saw that jenkins has not been installed and that's why I am not able to access it on web.
Oups, sorry about that, can you rerun the pipeline, sometimes it occurs, if you follow it correctly, it should get it installed.
@@primuslearning4235 I did a few times, but nothing gets installed.
Hi, I am having the same issue. It looks like everything installs from the script except the command "sudo systemctl enable jenkins". The error that shows is" Failed to execute operation: No such file or directory. I think something is not being created but I cannot figure out what is is just yet. @@primuslearning4235
check to make sure the right java jdk11 is installed , i had a same issue and did some manual installation with java and it finally worked @@kanikagola9613
Thank you for the project, its really useful. When I try and connect to my jenkins servers I had the error message "Hmmm… can't reach this page54.157.207.4 refused to connect." im not sure on how to proceed
Hi, thanks for watching. Are your servers up and running? Ensure that it created correctly and on a running state. You can paste the error here if there is.
Hi Primus, I tried to implement this project but the initialisation doesn't work at all I got this error message (
Initializing the backend...
╷
│ Error: Failed to get existing workspaces: S3 bucket does not exist.
│
│ The referenced S3 bucket must have been previously created. If the S3 bucket│ was created within the last minute, please wait for a minute or two and try
│ again.
│
│ Error: NoSuchBucket: The specified bucket does not exist
│ status code: 404, request id: XKA4ZJKZTVRJKY30, host id: B19gFAQIMXJ5O03C3q/Qtak3SQibwiUdQFl0Ak3wqQc5AZag5qLtiavdRg4ZozbjtVfvb5Ikg+8exh5rDcTQOg== ), and I already create S3
Hi Huguette, it didn't work because you didn't change the bucket name to a bucket in your AWS account in the backend.tf file in these locations:
1. github.com/Primus19/jenkins-pipeline-deploy-to-eks/blob/main/terraform/backend.tf.
2. github.com/Primus19/terraform-code-to-create-jenkins-server/blob/main/backend.tf
Please effect the change and rerun your pipeline.
Thank you Primus for your answer. The bucket was created in my account and I updated the name in the terraform backend file. That was not the issue. To resolve it, I deleted . terraform and initialize again and It works.
Thanks Primus for this really informative end to end demonstration. Very insightful. I was wondering if you could please advise on an error I am getting. I tried to run a Terraform initialization but getting the below error. It doesn't seem to like the S3 bucket.. Could you please advise..?
PS C:\Users\user\Documents\Terraform22\terraform-code-to-create-jenkins-server> terraform init
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Error refreshing state: AllAccessDisabled: All access to this object has been disabled
status code: 403, request id: T7RZMYJ9G16Z5A0H, host id: lnX1pmr53RBi11UOyCaFjF/kGvW4JE/W4EiNbkLknMx/usQ0B4KSYmcr3BtUuEEGuxiGAeQ1GsjYCADqihAIb2OlXJScdxFwYlPBx/kVq1s=
Hi Amuwonge, thanks for watching and following along. I think the user you configured on your cli doesn't have access to write to the S3 bucket you created. Please make sure you give s3 permissions to that user or better still just make it an admin and rerun and see if that fixes it. Also make sure its a bucket in your AWS account, could be that you forgot to change the bucket name and it still has the bucket I used for the demo. One of these things should be the issue. Thanks