I am the chief software architect of a SaaS editor in france, and this is exactly what we do since 9 years+, and works well. Build numbering and tagging is done automatically on monday morning so QA can start week sprint testing on monday. We have a 3 month release cycle (except rare hotfixes). 2 month of feature dev, 2 weeks of feature completion and polishing then 2 weeks of code freeze where we test fully and document known issues. During the last phase we document, take some break and communicate the release changes to the customers. For customer release, we do 10% first wave, then 50% a week later and 100% few days later.
@@biomorphicI have 80+ customers that pay 50K per year to use my software. The time to release of my features is not the major concern. Quality code and control are the only thing that matter. I even reserve 20% of my velocity to code refactoring, I have (almost) no technical debt. Of course when I produce prototypes I will do 1 week sprints and let QA and marketing teams trigger the releases. But that's not enterprise level software development.
And so what? Make yourself a favour, do not try to lecture me. I have been working in IT for 27 years. I have enough experience to know what works and what does not. No need for a noob to tell me what is a SaaS.@@dimitarslezenkovski2116 read and learn.
I put "gold standard" in quotes because trunk based development is just another development philosophy, often promoted by agile consultants. Everything has trade offs, so before you watch this video and think I'm saying every team should be doing this, I'm not. Use whatever approach works for your team and project needs. We use git flow at my workplace project and it works pretty well. We have shipped features to prod for years now using git flow. Research for yourself, ask chatGPT to compare git flow to trunk based development.
trunk-based is definitely for no-fainted-heart people and for mature teams or senior teams. Looking to the perspective of developers, all approaches may be fun as equal if everybody can understand their benefits. However, most of the time developers aren't there to entertain themselves but to comply expectations of people that has no clue about technology and think that the smallest issue in production is a signal of weak team. So, for those reasons, trunk-based needs a huge backup in wide understanding among stakeholders. Non-surprisingly, for companies that are natively technological, as social networks, or big tech in general, that approach or similars may come as natural as it can be. But for a Bank, or another traditional institution, or government, that will not be the best.
Thanks Cody. Your "rambling" is actually quite refreshing and thought-provoking 👏. It provides those of us who are deeply entrenched in these processes with an opportunity to step back and gain a fresh perspective. I particularly want to emphasize one point that you've brought up. The development process becomes inherently more time-consuming when an organization is risk-averse. This gets even more complicated when the product is not only deployed as SaaS but also on-premise for some customers. Project managers find themselves needing release notes and translations that developers may not be able to produce. There are also other stackholders like the security team that may slow the process down.
Yeah I don’t think many people understand that a lot of software is a people problem, and if people do not like prod to go down, often that requires moral formal processes which slow down development speed in place of more environment stability.
This helps a lot! especially for new devs out there, a request, if possible, could you please develop this flow/pipeline/CI-CD process from scratch for a small demo project. Thank you, there are less videos like this out there!
Absolutely would enjoy a video like this! A structure of the “rambling about a topic” style video and then a demo of whatever that was with a small/medium scale to follow it up; man, having those two things on a UA-cam video release schedule would be wicked!
Another element to this process is adding monitors/alarms to each stage. You deploy to a beta or preprod and a workflow bakees for say an hour. If your monitors don’t alert, your pipeline deploys to the next stage, if it does alert you configure it to automatically rollback to the previous revision
Can you go into further detail about the GH actions part of the flow? Like go into an example of building with docker and testing, etc. Please and thanks!
yeah, would love a GH actions tutorial with Dockerfile image building, testing, & deployment together with what are the notable events to listen to in order to kick off an integration/deployment
Working on a similar setup with a team of more than 100 in a monorepo(I mean everyone commits to master in one repo) and it’s has saved us many headaches around gitflow. But as you mentioned it’s simple to explain but the devil is in the detail. If you’ve not done it before it can lol overwhelming as in many areas you might have to build and improve your tooling setup to assist in the entire flow. Tooling really helps
Thank you so much, i have an internship coming up and my recruiter gave me a bunch of courses and he expects me to know all this stuff by tommorow, this is a good summary to what I've learned so far
You know, you might want to make sure the stakeholders get the full context of the application, otherwise they might not sign off on stuff to go into prod, so what you can do is make a separate branch that runs parallel to main, and when it gets pushed, there's an automatic build and deployment to an exclusive environment only the devs can access. And then the devs could branch off of this dev-exclusive branch to work on their features, and make sure everything is bundled together in a single release to pre-prod. And then, you could have a temporary branch for releasing to pre-prod, so you can push any last-minute bug fixes before it gets to the stake holders, and of course that would merge into both dev and main simultaneously to keep everything up to date. But that process is getting a bit cumbersome, so to keep agility high, we can also have branches for hotfixes that branch off of main, and after we fix the bug it gets merged back into main and dev to keep everything up to date. _phew_ we did it boys, we fixed trunk-based development. Wait...
Funniest comment yet. The only solution is tell stake holders that features will get deployed to prod, but they will not be active. Then when you deploy your first couple of bugs on accident, they will say “your approach needs to change”, and now you’re back to git flow.
I must say, it is a joy watching how you've increased in subscribers the last months, just goes to show that consistency and authenticity comes a long way. The approach you have is very refreshing, as you have been told before, it is enjoyable to witness the whole process, including errors/changing stuff mid video cause it didn't work as expected etc. So many videos are shown where everything works by defualt, which is rarely ever the case when you actually are developing something, at least to my experience.
Build a project for an NGO as a volunteer, its actually my first real world project. I recruited a couple of remote devs (volunteers) to help put too, and the project was successfully deployed. After a couple of months learning here and there I feel like improving the project with latest techs and just doing things better. Having these videos is just amazing, I know have an idea of how to do it right, with less to no interference. Also I'm planning to one day to start up a tech start-up, having this insights as early as know is just life saver and will make life easier in the long run and decision making. That's Cody and keep doing what you do pal
Loving these videos man! As I'm starting to look into building my first 'enterprise' system, having these different views on CI/CD, architecture and the pitfalls and benefits of different methods definitely helps! Thanks dude!
I am starting as a Jr. Software Eng next month in an enterprise company. Your large scale software development video and this helped a lot, I understand it might be different from team to team but at least I got to know the overview. Thank you!
I prefer your approach from the last vid. Great to see the different options out there that people are using though. This wouldn't work for us, because our features usually take anywhere from 3 to 10 days to build. We have a main, dev, qa and staging branch, and when a dev works on a new feature/ticket, they checkout from main (which is the most stable branch as it is what is on production). There are times when it is beneficial to checkout from staging, but it is case dependent.
worked in a bank in italy, they basically had 3 separate 'dev' environments where we could tesT/deploy our features, then 1 pre-prod env where users did the UAT and then the prod env. great video. tnx
One thing you could cover is the structure of the projects itself. What i mean - monerepos vs multirepos. How do you split your repositories. Namespaces. What belongs to Infra, what belongs to Dev/Build/Whatever. What belongs to Application. From the perspective of Day 0 bootstrapping. New rack/AWS account. You just plugged servers in the rack/wall socket. How do you structure entire work. Where does goes seed secrets and it's IaC. Etc. One thing is that you already have Infra in place. But another thing is how to get to this point. From the realm of "How do you collaborate if there is no Git server yet? How do you provision Git server if there is no Image factory. How do you harden images if there is no secret store and PKI? How do you test images if there is no servers installed?" But specifically from the PoV of project (Git repo) structure, without going into file structure itself. And another thing is how do you align developers local environment for his own local fast iteration without going trough the whole pipeline.
Your way of explanation with the diagram makes it simple to understand concepts. Please do some videos on how deployment happens with any hosting platform.
wow! These videos are really informative for me. I had just broken into tech as a machine learning engineer from a non technical background. Honestly, these videos explaining the fundamentals of real world software development are life savers for someone like me. Really appreciate the effort!! 🖖
Excellent advice. Here are some things I'd be interested in hearing about, if you understand them: - Continuous delivery vs Continuous deployment(drawbacks of continuous deployment without continuous delivery) - Other version control systems, mainly SVN, from the perspective of developer experience. - Workflow comparison between CI tools(mainly gh actions, gitlab CI and jenkins)
This video is a fucking lifesaver. Been trying to figure out all of this stuff on my own and it's reassuring to know that most of the processes I came up with myself also turned out to be popular processes currently used in the industry.
great video! my friend and i, both new to web dev, launched a newspaper like app to our school last year and this video would have solved much of problems lol the explanation is very straightforward, thanks for sharing your knowledge!
Absolutely loved this and your other video on larger scale development. I'm starting my first SWE position working on a large enterprise product soon, and I really wish this type of information was covered more in my education. Been pretty nervous because this is a huge step forward from the smaller, hackier programming assignments and projects I worked on in school and my free time. Your videos help ease that anxiety and make me excited about figuring all of this out! Thanks man!
As usual always on point about real life software development. The thing that is often not taken into account is that you might introduce features or not corresponding to the client expectation because of not enough details. Which are working features ( so they are not triggering any test flag) it's just that the client is unsatisfied. It happens a lot (especially in agile) which is why i prefer the graph that you detailed on your previous video. Also on some project you just cant accept to have bug in prod. It comes down to what you were saying at the beginning, Every approach have trade off.
Great video! as I said in the other one, I think ppl dont understand working on large scale enterprise cous every company does it somehow different, so its great that you explain things at large scale (like enterprise) cous nobody teaches that and when a dev its just in a small team pushing pretty much to production and finds he's way into a company, they are cluless of how things work
I think on medium to large organizations you at least need some email from the user approving the changes they saw in PrePRod and then you can migrate to prod.
Just recently found the channel and have been really enjoying these diagraming type videos! As a new grad software test engineer who wishes to become a dev. I would love to see some career oriented videos. Or your own career path would be great as well!
Continuous delivery vs automated delivery. The goal is to have one or more deployable commit every day, not necessarily to deploy every day. This allows for manual control over things like... not deploying when you are short-staffed or deploying while key team members are not available.
you should make a video showing how this code is/ and kept secure during these processes. that would be cool. especially for us on the cyber security side of things.
Cody, discussing your experience and thoughts is helpful for a small scale guy like me. Plenty to have in mind as things evolve....best of all you are open minded and just discussing the topic. Appreciate it
*My key takeaway from this video is:* trunk based development might seem easy at the beginning of development, but as you go, the complexity increases...so better use the large scale development process with different branches and environments for testing (like shown in your last video), it will benefit in long run.
01:09 🧩 Trunk-based development promotes continuous integration by merging multiple pull requests to the main branch daily. 03:13 🔄 Trunk-based development encourages knowledge sharing among developers through daily code reviews and understanding of changing features. 08:40 🚀 Trunk-based development aims for low deployment time and rapid feedback for bug fixes and feature requests. 11:58 🚧 Use feature flags to deploy unfinished features to production and involve stakeholders and QA for testing. 14:07 👩💻 Adding additional deployment environments like pre-prod can help mitigate risks and ensure proper verification of features before deploying to production.
One thing a lot of people miss is if your product has multiple live versions or not. If you’re product has multiple live versions, like 1.1, 1.2, 2.0, etc., then trunk based dev may be more challenging. You will need to be able build the product for any version at any time to fix a bug or security issue. Now, if you only have one live version, like most web apps or services, trunk based development is ideal.
I don't know how to say thank you enough, I shared this with my team, we read a business book on blitzscaling and one quote from mark zurkerberg was move fast and break things, we're at that point now 😂, so we're using the trunk based approach right now, but I see us moving slowly moving towards more complex setups cos of the nature of our application
@@WebDevCody any time no rush I know you are busy. I have been working on a plugin that allows you to create text embedding to give your chatgpt context and long term memory with langchain and pinecone. Here is a quick demo. ua-cam.com/video/aJMzRURN6fA/v-deo.html
The team could be doing CI into the dev branch too, but then feature flagging needs to be set up, so as an unfinished feature doesn't cause havoc for the rest of the process.
Main challenge is to know what all the hype words mean. I guess I have spent a decade naming branches feature, but doing trunk based. It seems obvious that 3 weeks work on a branch is likely to lead to merge pains. If you have long running branches it’s your job to rebase regularly
(Edit I was too quick you basically describe a similar workflow as I did here): Yeah basically you set up a CI that does integrations tests (indeed more important than unit tests) on a PR, and only when the PR succeeds a pull request goes through.
in previous gitflow we clients can do testing, but in trunkbased flow can client do testing in any way or just direct testing on prod environment only?
Yes with feature flag. Release a feature behind a flag and only turn it on for a test user. Obviously it requires more orchestration and setup which is why many avoid feature flags
If a commit into main or not introduces a bug locally the solution is super simple. You reset, it takes seconds and after notifying the author you go back to work.
We working like this but without release tags & feature flags. So our pipelines is semi-automatic - build after pushing changes and manual deploy to test & rc env and then similar process after merging request into main branch
As a QA engineer, having essentially experimental code in “main” terrifies the fuck out of me 😂 For me, “main” should be your source of truth. It should be the most complete, stable version of the codebase. There’s just too much risk with the trunk based approach and not all testing can be automated.
I agree not all testing can be automated... and if it can be, it often days of dev effort to write the test that a person can manually verify in 5 minutes.
I like to look at the Linux kernel development as a kind of development. A massive project where the price you pay for a bug is a kernel crash/reboot. This development tends to favor feature branches.
Well done, you basically explained my dev process. What I did though is when I tag say v1.0.0, it kicks off a build and pushes to Docker Hub. Then another action updates an ArgoCD manifest to automatically update the container in a staging Kubernetes cluster and push updates to Slack that it's done. When I'm done testing staging version, then I just tag a release so without the v and it pushes and updates to my production cluster. It's automated enough that I don't even feel like I'm managing anything, I just code.
People saying that you don't need QA if you have good testing obviously never worked on a real, complex project. Would creating perfect tests be preferable? Sure. Is it feasable in the real world? Not really. Every project I've worked on had a "dev-staging-prod" type of situation, because it just works most of the time.
The devil is in the details. LOL.. Awesome! Thank you for your videos. I like how I can see that diagram unfolding on a daily basis. It's great to have this presented in a diagrammatic perspective.
Sometimes QA is required by contract (i.e. government funded projects) and testing a feature might require a ton of functional steps to be performed before you can reach that feature. The setup work needed for testing might be difficult to set up on unit tests or integration tests.
Imo for 80% of companies feature flagging or dark shipping is mostly fine. I think once you get in to SLA’s with required uptimes it becomes more prudent to have a slower time to production cycle
I think this is great except for the environments part, which is a different methodology altogether. It might be considered a deployment strategy when rolling out services and making sure they don’t immediately catch fire but it shouldn’t be part of your release strategy here. All of that testing should be done before the PR is accepted (which also has a quality gate to make sure it’s not behind main. Great video though! (Qualification - principal engineer)
"There are professional QA testers who are really good at finding bugs" - Can't agree more, we have these and occasionally there is a suggestion that these QA engineers simply join the main group of engineers and that we would share out the QA processes between all of the engineers, but this just sounds like a way to let more bugs through. The dedicated QA engineer has environments and configurations in place to robustly test a feature, they spend all of their time building and maintaining these, and become very good at finding bugs as a result.
Now make a video on other developer workflows and where do junior devs fit in like join this whole cycle like on the far left of the canvas?! Or how we might learn more about these cycles!
Thank you. We use regular git flow at our company for building web apps. Usually FE devs build the UI using the API deployed to the dev env so they don't need to run the API locally using docker and they don't need keep the local database data up to date with the latest features. I am just wondering how this might be achieved using the trunk based development.
Trunk based development is a recognition that some companies deploy new changes once a month because they are scared to deploy to prod. We should be able to deploy to prod multiple times daily, that’s the goal, and trunk based development is one approach to help achieve that for projects.
For me, the worst part about this workflow is losing the ability to review a feature in its completeness. If a feature takes e.g. 3 weeks and different people review different PRs at multiple days, the review cannot be as thorough as it would be if done in a bigger PR for the complete feature, hence losing nuance regarding architecture, testing, etc.
It also makes it hard to catch e.g. logical bugs within the feature or evaluate if the feature in its entirety does what the requirements in a ticket say.
I do think the benefit of this approach is that you are evaluating code in smaller chunks which is easier to process so you might be more thorough because you aren't reviewing 1000s lines of code, but of course it's harder to connect those pieces in your head if you have 5 PRs for a single feature. I do also like that shared code can be made available faster to other devs instead of living in a feature branch for a month and waiting the feature to be complete. But on the other hand, I've noticed with this approach is that problems detected during review tend to be pushed "into the next PR", which is rarely then actually done so you get tech debt you don't address and it's basically a "TODO" item that doesn't get addressed. Because with trunk based development the goal is to merge as often as possible, so PRs shouldn't stay open for a longer time which means there is less time to address feedback. Also since idea is to merge often, crappy things can easily slip into the codebase depending on who is reviewing the code. There are many cases where more people don't get a chance to review something, especially in this async world where you might have people from many different timezones working on a single project. Also, feature flagging basically everything can complicate code, is a maintenance burden and requires cleanup afterwards which is easy to ignore "because it's not a priority" unless engineering really pushes for it.
Imo any big feature should be agreed upon upfront by the engineers typically with an RFC (request for comments), this is where you would show your proposed architecture and the idea you have to complete the feature. Once you have approval to do this, then future PRs can reference this open RFC so people wont lose context. Rfcs are a good way to receive feedback and learn a lot. I find them especially helpful when changes cut across many different teams/repos. They can help improve the codebase as well because typically if you receive a big PR you might just go "LGTM" or point out little mistakes or annoyingly have to change the entire thing but when you can see the idea ahead of time it's easier to challenge it and gather multiple perspectives on the problem.
The question coming soon is which method works better for AI feature addition. I think probably Gitflow is more suited for isolating new features generated, tested and documented by AI before merging. We just need good AI business analyst workflow.
I wonder, if you have staging and production environments in this case, does that mean that the staging is updated by Github Actions, and the production is updated later manually?
I don't work on a large team but I like good ol dev-stage-prod. Can go from stage to prod right away if changes aren't huge but every release needs at least some human testing imo. Apps be pushing out WAY too many bugs these days.
I agree. Continuous delivery over continuous deployment. Also not a good idea to deploy when key personnel are on leave, or you are short-staffed (which can easily happen in startups / smaller companies.
I think the idea of agile is good, especially for new products. Ship features, get feedback from users, pivot, ship again. The issue is no one seems to understand the get feedback part but say they are agile.
I am the chief software architect of a SaaS editor in france, and this is exactly what we do since 9 years+, and works well. Build numbering and tagging is done automatically on monday morning so QA can start week sprint testing on monday. We have a 3 month release cycle (except rare hotfixes). 2 month of feature dev, 2 weeks of feature completion and polishing then 2 weeks of code freeze where we test fully and document known issues. During the last phase we document, take some break and communicate the release changes to the customers. For customer release, we do 10% first wave, then 50% a week later and 100% few days later.
That shit wouldn't happen in the states lol Ape mangers always "I need this now"
You have a 3 months release cycle and you do trunk development? You should probably find yourself a job in a factory.
@@biomorphicI have 80+ customers that pay 50K per year to use my software. The time to release of my features is not the major concern. Quality code and control are the only thing that matter. I even reserve 20% of my velocity to code refactoring, I have (almost) no technical debt. Of course when I produce prototypes I will do 1 week sprints and let QA and marketing teams trigger the releases. But that's not enterprise level software development.
@@biomorphic It's a SaaS, dude.
And so what? Make yourself a favour, do not try to lecture me. I have been working in IT for 27 years. I have enough experience to know what works and what does not. No need for a noob to tell me what is a SaaS.@@dimitarslezenkovski2116 read and learn.
I put "gold standard" in quotes because trunk based development is just another development philosophy, often promoted by agile consultants. Everything has trade offs, so before you watch this video and think I'm saying every team should be doing this, I'm not. Use whatever approach works for your team and project needs. We use git flow at my workplace project and it works pretty well. We have shipped features to prod for years now using git flow. Research for yourself, ask chatGPT to compare git flow to trunk based development.
trunk-based is definitely for no-fainted-heart people and for mature teams or senior teams. Looking to the perspective of developers, all approaches may be fun as equal if everybody can understand their benefits. However, most of the time developers aren't there to entertain themselves but to comply expectations of people that has no clue about technology and think that the smallest issue in production is a signal of weak team. So, for those reasons, trunk-based needs a huge backup in wide understanding among stakeholders. Non-surprisingly, for companies that are natively technological, as social networks, or big tech in general, that approach or similars may come as natural as it can be. But for a Bank, or another traditional institution, or government, that will not be the best.
Thanks Cody.
Your "rambling" is actually quite refreshing and thought-provoking 👏. It provides those of us who are deeply entrenched in these processes with an opportunity to step back and gain a fresh perspective.
I particularly want to emphasize one point that you've brought up. The development process becomes inherently more time-consuming when an organization is risk-averse. This gets even more complicated when the product is not only deployed as SaaS but also on-premise for some customers. Project managers find themselves needing release notes and translations that developers may not be able to produce. There are also other stackholders like the security team that may slow the process down.
Yeah I don’t think many people understand that a lot of software is a people problem, and if people do not like prod to go down, often that requires moral formal processes which slow down development speed in place of more environment stability.
This helps a lot! especially for new devs out there, a request, if possible, could you please develop this flow/pipeline/CI-CD process from scratch for a small demo project. Thank you, there are less videos like this out there!
Absolutely would enjoy a video like this!
A structure of the “rambling about a topic” style video and then a demo of whatever that was with a small/medium scale to follow it up; man, having those two things on a UA-cam video release schedule would be wicked!
Yes Please😀😀😀
Another element to this process is adding monitors/alarms to each stage. You deploy to a beta or preprod and a workflow bakees for say an hour. If your monitors don’t alert, your pipeline deploys to the next stage, if it does alert you configure it to automatically rollback to the previous revision
Can you go into further detail about the GH actions part of the flow? Like go into an example of building with docker and testing, etc. Please and thanks!
That would be awesome!
yeah, would love a GH actions tutorial with Dockerfile image building, testing, & deployment together with what are the notable events to listen to in order to kick off an integration/deployment
yeah! this would help to connect dots.
48 seconds in and I'm already learning stuff a 4 year degree didn't teach me. Subscribed & liked. Keep up the great work.
Working on a similar setup with a team of more than 100 in a monorepo(I mean everyone commits to master in one repo) and it’s has saved us many headaches around gitflow. But as you mentioned it’s simple to explain but the devil is in the detail. If you’ve not done it before it can lol overwhelming as in many areas you might have to build and improve your tooling setup to assist in the entire flow. Tooling really helps
Thank you so much, i have an internship coming up and my recruiter gave me a bunch of courses and he expects me to know all this stuff by tommorow, this is a good summary to what I've learned so far
You know, you might want to make sure the stakeholders get the full context of the application, otherwise they might not sign off on stuff to go into prod, so what you can do is make a separate branch that runs parallel to main, and when it gets pushed, there's an automatic build and deployment to an exclusive environment only the devs can access. And then the devs could branch off of this dev-exclusive branch to work on their features, and make sure everything is bundled together in a single release to pre-prod.
And then, you could have a temporary branch for releasing to pre-prod, so you can push any last-minute bug fixes before it gets to the stake holders, and of course that would merge into both dev and main simultaneously to keep everything up to date.
But that process is getting a bit cumbersome, so to keep agility high, we can also have branches for hotfixes that branch off of main, and after we fix the bug it gets merged back into main and dev to keep everything up to date.
_phew_ we did it boys, we fixed trunk-based development.
Wait...
Funniest comment yet. The only solution is tell stake holders that features will get deployed to prod, but they will not be active. Then when you deploy your first couple of bugs on accident, they will say “your approach needs to change”, and now you’re back to git flow.
I must say, it is a joy watching how you've increased in subscribers the last months, just goes to show that consistency and authenticity comes a long way. The approach you have is very refreshing, as you have been told before, it is enjoyable to witness the whole process, including errors/changing stuff mid video cause it didn't work as expected etc. So many videos are shown where everything works by defualt, which is rarely ever the case when you actually are developing something, at least to my experience.
Thanks man, I try to just keep it real
Build a project for an NGO as a volunteer, its actually my first real world project. I recruited a couple of remote devs (volunteers) to help put too, and the project was successfully deployed. After a couple of months learning here and there I feel like improving the project with latest techs and just doing things better. Having these videos is just amazing, I know have an idea of how to do it right, with less to no interference.
Also I'm planning to one day to start up a tech start-up, having this insights as early as know is just life saver and will make life easier in the long run and decision making.
That's Cody and keep doing what you do pal
These are so good - not sure if you're calling this a 'series' but these are great vids.
Loving these videos man! As I'm starting to look into building my first 'enterprise' system, having these different views on CI/CD, architecture and the pitfalls and benefits of different methods definitely helps! Thanks dude!
I really like the brief comparison of options with a "this is what I do".
I am starting as a Jr. Software Eng next month in an enterprise company. Your large scale software development video and this helped a lot, I understand it might be different from team to team but at least I got to know the overview. Thank you!
I prefer your approach from the last vid. Great to see the different options out there that people are using though. This wouldn't work for us, because our features usually take anywhere from 3 to 10 days to build. We have a main, dev, qa and staging branch, and when a dev works on a new feature/ticket, they checkout from main (which is the most stable branch as it is what is on production). There are times when it is beneficial to checkout from staging, but it is case dependent.
worked in a bank in italy, they basically had 3 separate 'dev' environments where we could tesT/deploy our features, then 1 pre-prod env where users did the UAT and then the prod env. great video. tnx
so it's basically like his previous video
dev > test/stage/pre-pod > main/prod
holy shit I love these kind of videos. thank you for giving us a high-level view of the sdlc process
Crazy Cody. I used your icon generator recently for a build I'm working on. Thanks for this video.
One thing you could cover is the structure of the projects itself. What i mean - monerepos vs multirepos. How do you split your repositories. Namespaces. What belongs to Infra, what belongs to Dev/Build/Whatever. What belongs to Application. From the perspective of Day 0 bootstrapping. New rack/AWS account. You just plugged servers in the rack/wall socket. How do you structure entire work. Where does goes seed secrets and it's IaC. Etc.
One thing is that you already have Infra in place. But another thing is how to get to this point. From the realm of "How do you collaborate if there is no Git server yet? How do you provision Git server if there is no Image factory. How do you harden images if there is no secret store and PKI? How do you test images if there is no servers installed?"
But specifically from the PoV of project (Git repo) structure, without going into file structure itself.
And another thing is how do you align developers local environment for his own local fast iteration without going trough the whole pipeline.
Your way of explanation with the diagram makes it simple to understand concepts. Please do some videos on how deployment happens with any hosting platform.
wow! These videos are really informative for me. I had just broken into tech as a machine learning engineer from a non technical background. Honestly, these videos explaining the fundamentals of real world software development are life savers for someone like me. Really appreciate the effort!! 🖖
If you keep making videos like this I will watch every single one of them. This is such a good way of explaining things.
Excellent advice.
Here are some things I'd be interested in hearing about, if you understand them:
- Continuous delivery vs Continuous deployment(drawbacks of continuous deployment without continuous delivery)
- Other version control systems, mainly SVN, from the perspective of developer experience.
- Workflow comparison between CI tools(mainly gh actions, gitlab CI and jenkins)
he definitely understands them L0l
@@safarl45 maybe he was trying to say "if you can explain them"
This video is a fucking lifesaver. Been trying to figure out all of this stuff on my own and it's reassuring to know that most of the processes I came up with myself also turned out to be popular processes currently used in the industry.
im glad i procrastinated on implementing concepts from previous video!
thanks man, this is very helpful
great video! my friend and i, both new to web dev, launched a newspaper like app to our school last year and this video would have solved much of problems lol
the explanation is very straightforward, thanks for sharing your knowledge!
Absolutely loved this and your other video on larger scale development. I'm starting my first SWE position working on a large enterprise product soon, and I really wish this type of information was covered more in my education. Been pretty nervous because this is a huge step forward from the smaller, hackier programming assignments and projects I worked on in school and my free time. Your videos help ease that anxiety and make me excited about figuring all of this out! Thanks man!
Update please? How’s it been
14:48 haha The best part is describing product managers as "... it's basically just people who want to check things off on a checkbox".
Congratulations for your videos my friend! I am enjoying and learning!
I love these high level videos! Great content. Spot on with what we use.
As usual always on point about real life software development.
The thing that is often not taken into account is that you might introduce features or not corresponding to the client expectation because of not enough details. Which are working features ( so they are not triggering any test flag) it's just that the client is unsatisfied. It happens a lot (especially in agile) which is why i prefer the graph that you detailed on your previous video.
Also on some project you just cant accept to have bug in prod.
It comes down to what you were saying at the beginning, Every approach have trade off.
It's actually good to see the full chaos of deployment possibilities. Ramble on! Devs need to see this.
Very valuable information to someone like me who just started learning about these things. Thank you
hell yes another great workflow video 🔥🔥🔥
Great video! as I said in the other one, I think ppl dont understand working on large scale enterprise cous every company does it somehow different, so its great that you explain things at large scale (like enterprise) cous nobody teaches that and when a dev its just in a small team pushing pretty much to production and finds he's way into a company, they are cluless of how things work
I think on medium to large organizations you at least need some email from the user approving the changes they saw in PrePRod and then you can migrate to prod.
Thanks again. keep churning these vids man.
This is really cool. I’ve never had a job on a team. I just do solo projects and it’s really rad to learn how these flows work. 🤙🏼🤙🏼🤙🏼
Just recently found the channel and have been really enjoying these diagraming type videos!
As a new grad software test engineer who wishes to become a dev. I would love to see some career oriented videos. Or your own career path would be great as well!
Not understanding a thing but this whole development idea looks dope
Continuous delivery vs automated delivery. The goal is to have one or more deployable commit every day, not necessarily to deploy every day. This allows for manual control over things like... not deploying when you are short-staffed or deploying while key team members are not available.
Amazing video! Please, keep making them, it’s super educative
you should make a video showing how this code is/ and kept secure during these processes. that would be cool. especially for us on the cyber security side of things.
Please do more of these they're really helpful ❤
Cody, discussing your experience and thoughts is helpful for a small scale guy like me. Plenty to have in mind as things evolve....best of all you are open minded and just discussing the topic. Appreciate it
*My key takeaway from this video is:* trunk based development might seem easy at the beginning of development, but as you go, the complexity increases...so better use the large scale development process with different branches and environments for testing (like shown in your last video), it will benefit in long run.
01:09 🧩 Trunk-based development promotes continuous integration by merging multiple pull requests to the main branch daily.
03:13 🔄 Trunk-based development encourages knowledge sharing among developers through daily code reviews and understanding of changing features.
08:40 🚀 Trunk-based development aims for low deployment time and rapid feedback for bug fixes and feature requests.
11:58 🚧 Use feature flags to deploy unfinished features to production and involve stakeholders and QA for testing.
14:07 👩💻 Adding additional deployment environments like pre-prod can help mitigate risks and ensure proper verification of features before deploying to production.
Thank you! Did you make these or use a service to generate them
@@WebDevCody chrome extension called harpa ai + openai api
One thing a lot of people miss is if your product has multiple live versions or not. If you’re product has multiple live versions, like 1.1, 1.2, 2.0, etc., then trunk based dev may be more challenging. You will need to be able build the product for any version at any time to fix a bug or security issue.
Now, if you only have one live version, like most web apps or services, trunk based development is ideal.
I don't know how to say thank you enough, I shared this with my team, we read a business book on blitzscaling and one quote from mark zurkerberg was move fast and break things, we're at that point now 😂, so we're using the trunk based approach right now, but I see us moving slowly moving towards more complex setups cos of the nature of our application
move fast and break things until people start getting mad you are breaking things 🤣
I have been loving these style videos.
Thanks man! I’ve been meaning to do another live with you.
@@WebDevCody any time no rush I know you are busy.
I have been working on a plugin that allows you to create text embedding to give your chatgpt context and long term memory with langchain and pinecone.
Here is a quick demo. ua-cam.com/video/aJMzRURN6fA/v-deo.html
Well done video that goes beyond the basic principles
00:05 I love hearing this
This is the best channel I found in years
The team could be doing CI into the dev branch too, but then feature flagging needs to be set up, so as an unfinished feature doesn't cause havoc for the rest of the process.
Good job babe!!! You changed post time on me!
Fantastic work! You covered all the points well 🎉 I was wondering if you were going to mention canaries but you did at the end. Superb job
yaep, that last oen was similar to canarian deplyment, but this uses different versions by redirecting users
There's some great information in this video thank you very much for sharing!
Main challenge is to know what all the hype words mean. I guess I have spent a decade naming branches feature, but doing trunk based. It seems obvious that 3 weeks work on a branch is likely to lead to merge pains. If you have long running branches it’s your job to rebase regularly
(Edit I was too quick you basically describe a similar workflow as I did here):
Yeah basically you set up a CI that does integrations tests (indeed more important than unit tests) on a PR, and only when the PR succeeds a pull request goes through.
in previous gitflow we clients can do testing, but in trunkbased flow can client do testing in any way or just direct testing on prod environment only?
Yes with feature flag. Release a feature behind a flag and only turn it on for a test user. Obviously it requires more orchestration and setup which is why many avoid feature flags
I love these cody please keep them up
If a commit into main or not introduces a bug locally the solution is super simple.
You reset, it takes seconds and after notifying the author you go back to work.
Nice work man! Respect
We working like this but without release tags & feature flags. So our pipelines is semi-automatic - build after pushing changes and manual deploy to test & rc env and then similar process after merging request into main branch
Seems like a bunch of risk and added overhead just so people don't have to deal with as many merge conflicts.
As a QA engineer, having essentially experimental code in “main” terrifies the fuck out of me 😂 For me, “main” should be your source of truth. It should be the most complete, stable version of the codebase. There’s just too much risk with the trunk based approach and not all testing can be automated.
I agree not all testing can be automated... and if it can be, it often days of dev effort to write the test that a person can manually verify in 5 minutes.
crazy how useful everything u say, appreciate it 🌀
Glad it helps!
I like to look at the Linux kernel development as a kind of development. A massive project where the price you pay for a bug is a kernel crash/reboot. This development tends to favor feature branches.
Thanks man. Awesome video. Keep it up!
Great overview. Thanks!
Nice videos!! Thank you for sharing your experience!
In a huge projects it could become a merges-hell approach. When there is always a challenge to merge your code. It can take days of developers time.
This is gold, thanks for the content.
Well done, you basically explained my dev process. What I did though is when I tag say v1.0.0, it kicks off a build and pushes to Docker Hub. Then another action updates an ArgoCD manifest to automatically update the container in a staging Kubernetes cluster and push updates to Slack that it's done. When I'm done testing staging version, then I just tag a release so without the v and it pushes and updates to my production cluster. It's automated enough that I don't even feel like I'm managing anything, I just code.
That sounds nice
People saying that you don't need QA if you have good testing obviously never worked on a real, complex project. Would creating perfect tests be preferable? Sure. Is it feasable in the real world? Not really. Every project I've worked on had a "dev-staging-prod" type of situation, because it just works most of the time.
The devil is in the details. LOL.. Awesome! Thank you for your videos. I like how I can see that diagram unfolding on a daily basis. It's great to have this presented in a diagrammatic perspective.
Sometimes QA is required by contract (i.e. government funded projects) and testing a feature might require a ton of functional steps to be performed before you can reach that feature. The setup work needed for testing might be difficult to set up on unit tests or integration tests.
Imo for 80% of companies feature flagging or dark shipping is mostly fine. I think once you get in to SLA’s with required uptimes it becomes more prudent to have a slower time to production cycle
I think this is great except for the environments part, which is a different methodology altogether. It might be considered a deployment strategy when rolling out services and making sure they don’t immediately catch fire but it shouldn’t be part of your release strategy here. All of that testing should be done before the PR is accepted (which also has a quality gate to make sure it’s not behind main. Great video though! (Qualification - principal engineer)
very helpful for starting devs. Its not that easy to find vids on overall dev process
Working in product I needed that!!
Very useful video thank you so much and please keep it going ❤
"There are professional QA testers who are really good at finding bugs" - Can't agree more, we have these and occasionally there is a suggestion that these QA engineers simply join the main group of engineers and that we would share out the QA processes between all of the engineers, but this just sounds like a way to let more bugs through. The dedicated QA engineer has environments and configurations in place to robustly test a feature, they spend all of their time building and maintaining these, and become very good at finding bugs as a result.
Nice. Can you share a little insight about user story and how to make it for us the junior? Really appreciated that 🙌
Now make a video on other developer workflows and where do junior devs fit in like join this whole cycle like on the far left of the canvas?! Or how we might learn more about these cycles!
Thank you. We use regular git flow at our company for building web apps. Usually FE devs build the UI using the API deployed to the dev env so they don't need to run the API locally using docker and they don't need keep the local database data up to date with the latest features. I am just wondering how this might be achieved using the trunk based development.
Brilliantly simplified
Huge help! Thank you so much.
Trunk based development looks like we add a bunch of processes because we're scared of rebasing feature branches
Trunk based development is a recognition that some companies deploy new changes once a month because they are scared to deploy to prod. We should be able to deploy to prod multiple times daily, that’s the goal, and trunk based development is one approach to help achieve that for projects.
For me, the worst part about this workflow is losing the ability to review a feature in its completeness. If a feature takes e.g. 3 weeks and different people review different PRs at multiple days, the review cannot be as thorough as it would be if done in a bigger PR for the complete feature, hence losing nuance regarding architecture, testing, etc.
I could see that being a real issue as well. Instead of reviewing a single PR now it's split up into 10-20 smaller PRs.
It also makes it hard to catch e.g. logical bugs within the feature or evaluate if the feature in its entirety does what the requirements in a ticket say.
I do think the benefit of this approach is that you are evaluating code in smaller chunks which is easier to process so you might be more thorough because you aren't reviewing 1000s lines of code, but of course it's harder to connect those pieces in your head if you have 5 PRs for a single feature. I do also like that shared code can be made available faster to other devs instead of living in a feature branch for a month and waiting the feature to be complete.
But on the other hand, I've noticed with this approach is that problems detected during review tend to be pushed "into the next PR", which is rarely then actually done so you get tech debt you don't address and it's basically a "TODO" item that doesn't get addressed. Because with trunk based development the goal is to merge as often as possible, so PRs shouldn't stay open for a longer time which means there is less time to address feedback. Also since idea is to merge often, crappy things can easily slip into the codebase depending on who is reviewing the code. There are many cases where more people don't get a chance to review something, especially in this async world where you might have people from many different timezones working on a single project.
Also, feature flagging basically everything can complicate code, is a maintenance burden and requires cleanup afterwards which is easy to ignore "because it's not a priority" unless engineering really pushes for it.
Imo any big feature should be agreed upon upfront by the engineers typically with an RFC (request for comments), this is where you would show your proposed architecture and the idea you have to complete the feature. Once you have approval to do this, then future PRs can reference this open RFC so people wont lose context.
Rfcs are a good way to receive feedback and learn a lot. I find them especially helpful when changes cut across many different teams/repos. They can help improve the codebase as well because typically if you receive a big PR you might just go "LGTM" or point out little mistakes or annoyingly have to change the entire thing but when you can see the idea ahead of time it's easier to challenge it and gather multiple perspectives on the problem.
Could you kindly please share the moment when you are building the software than using diagram? I want to see in practical
This is best ❤ Thanks Wdc
The question coming soon is which method works better for AI feature addition. I think probably Gitflow is more suited for isolating new features generated, tested and documented by AI before merging. We just need good AI business analyst workflow.
I can say it was like this when i worked for big company
I wonder, if you have staging and production environments in this case, does that mean that the staging is updated by Github Actions, and the production is updated later manually?
just wonderful!!
Please make more of these types of videos
I don't work on a large team but I like good ol dev-stage-prod. Can go from stage to prod right away if changes aren't huge but every release needs at least some human testing imo. Apps be pushing out WAY too many bugs these days.
I agree. Continuous delivery over continuous deployment. Also not a good idea to deploy when key personnel are on leave, or you are short-staffed (which can easily happen in startups / smaller companies.
Bro we are building huge startup for influencers using AI and your videos are awesome! what is your opinion about agile development/methodology?
I think the idea of agile is good, especially for new products. Ship features, get feedback from users, pivot, ship again. The issue is no one seems to understand the get feedback part but say they are agile.