I’m a contract tech leader and came back to this video after recently joining a new team. I think it’s probably my favourite video of all of the excellent videos on your channel Dave. The kids game analogy is both perfect and hilarious and hence just the tool you need to trigger some self reflection in a team where some members are struggling to get the continuous delivery mindset. I must say thank you for this one. I’ve tried to pay it back as best I can and bought the library of books, many of which are part of my most quoted along with Brooks, Martin, Beck et al. Nice.
This assumes all my changes are always incremental. In practice I often tear down most of what I did after I've cemented the general idea. To introduce my initial ramblings to the team before I even know if it's the direction I want to go, sounds like a whole lot of work for nothing.
Great point, but you can still do CI here. If you plan on throwing away the branch, that doesn't count and is totally fine to take your time on it. Just be aware things could change during that time. Create your "draft" branch, do your thing, then when you are satisfied and ready to write it for real, check out a new fresh branch and do your changes there. Then commit and submit PR. Done.
I've worked for a large firm (close to 100 developers) and we were divided into smaller project teams (10 to 20 people) each responsible for a particular feature set and launch window. "Assumes all changes are incremental" is the problem here. Large changes are not useful to the rest of the team until they are completed, or at least nearly complete (ready for system testing). Working on a branch gives the project team time get their stuff into a usable state, then it gets merged with top of tree, but not before.
Work on your local branch and don't push to origin until it's ready. When complete, do interactive rebase, fixing up all the irrelevant commits into one (or as many as logical), then PR that set of logical commits.
@@JD-mz1rl That only works if much hasn't changed, or if your changes aren't massive. If this is working for you, then either your team is probably already breaking things down small enough (good), your application/team hasn't really scaled yet, or you aren't making many big changes.
Branching is valuable. Many times I use a branch for experimentl changes/refactorings that I’m not yet sure should ever be merged-at least until they prove themselves. If it doesn’t work out, I just delete the branch. Other times I use a branch to make significant refactorings I don’t want to disturb the main code, which is still moving. Its not always possible to think of work as a stream of small incremental changes.
I guess with the more frequent merges team members are more aware of the work of each other. And smaller, daily merges mean quicker reviews. Merge requests are not the only way to review code, anyone at any stage of the development can just read the diffs. Maybe reviewing is also better if continuous.
what do u mean, you don't need to commit changes to create a code review you can use something like Code Collaborator to make a code review based on your local changes
Gerrit accomplishes exactly what you are asking. You push your commit to the main branch but Gerrit does not push it to remote origin until it has been reviewed.
I was about to write a comment on how this was clearly bullshit. I watched the video instead. By the end I got completely what you were saying, and it makes perfect sense. CI isn’t something you can dabble in, and this demonstrates the level of commitment required to do it right. But if the data says it’s the most efficient way of working, then it’s worth seriously considering. Thanks!
We changed our process after this video few months ago and that instantly solved lot of code conflicts problems that we were facing and allowed us to visualize the progress of the day. Thanks a lot for the content
I feel like this would have been better received if it had been about branch lifespan/scope rather than “don’t branch” since branching is the typical means of code review. It seems to me branch/review/merge often is compatible with the CI definition and provides the same benefits you describe. Thanks for the video!
If you have a team that enjoys reviewing code, that'll probably work! Also, if reviews are done in pairs, even better - then you will get higher bandwidth of communication than the async nature of typical "chat based" reviews.
If however either of those are not true (code reviews take 1-3 days to complete or reviews are done through comments on a pull request), you are missing out on something here
My preferred way of dealing with staying updated with feature branches is frequent rebasing, this also allows me to have a linear history instead of a git history looking like a weaved basket. Gerrit is a very interesting Git host, it automates a lot of fiddling with branches to the point development can go on for months without even thinking of creating a feature branch. Because in Gerrit you work with patchsets (internally they are just branches where only the head is used, so it acts like a single commit, with all its history still available) and just "replace" commits, and you can do this as long as you like, rebasing it, running CI against it, until you merge it back to the parent branch.
@@JulianSildenLanglo Actually my prefered way of working is pair-programming with daily pair-rotation, so there are more than 2 authors and for big stories nearly everyone on the team will have worked on it. At which point an extra review is redundant. I work a lot in regulated industries, there is no regulatory framework that I know of, anywhere in the world that I have worked, that won't accept pair-programming as a code-review.
@@ContinuousDelivery Have you always lucky enough to have an even number of developers in a team, or have you had cases where one dev a day had to do some lone working? (I'm coming from an even worse situation, where I'm in a team as a single software dev - nominally there are two of us, but the other person has an astoundingly large project management workload, and hasn't committed code in months)
I just want to express my gratitude for your content. It’s really professional stuff. As a software developer myself, yours is one of the very few channels I can actually still learn from. Bravo!
So if Bob is working on the same file as me, then I'm supposed to pull Bob's changes every hour, understand why he broke my feature then rearrange my feature and break Bob's feature the next hour? Sounds like no fun. Never had a problem with merge conflicts and much prefer to have complete ideas to compare and merge.
Yes, you and Bob should merge your code at least once, but ideally many, times per day to qualify as practicing Continuous Integration. Sure, you may like the way that you work, but the data says that you are more likely to be producing lower-quality software more slowly as a result. That is not me saying that because that is just my guess or preference, that is based on an analysis of 35,000 software projects. My video is an attempt to explain why that fact is the case.
@@BigCarso Ridiculous response for someone whose beliefs got challenged. Nobody requires you to adapt your work style to new information. If you like what you were doing or are too ignorant of realizing the problems you have, that's fine, just keep doing it, no reason to change. However, many companies are struggling with their systems, and in particular the problem you mentioned is one of the main causes: If you two break each others feature all the time, then something is already going seriously wrong, and in such a case, a slower merge (like a 300+ conflicts merge) would be a complete disaster both in terms of time investment as well as in terms of keeping the error rate low. It also seems like you haven't watched the video, because it's pretty damn obvious that a change with fewer changes takes less time to merge than one with more changes. Really not that difficult to understand.
Don’t listen to them, they don’t know what they’re talking about and what he is proposing would just get you laughed at by developers because it’s a stupid approach where he assumes that nobody can communicate. He was late to version control and is confused by it, teams can manage branches and reduce merge conflicts down to 0, especially with microservices (which he is against for stupid reasons)
@@skylark.kraken I fell the same, seems he never worked in a big team, it is so horrible in a real world, he talk about downside of feature branch against continuous delivery but miss to mention the pros
This is much closer to what I was looking for....while youtube is flooded with basic to advanced GIT tutorials but nobody talks about professional implementation and use of CI/CD with lot of confusion on branching.
Especially true as one thing that's missing from this approach is code review. If every developer on a team integrates every day that's 1 code review for 2 or 3 other people every day. Granted they're small changes but reviewers have to understand the big picture behind changes, otherwise all they're doing is reviewing syntax variation and coding style. A big part of review is does the change fit the spec and the desired end result of the bigger change. Granted review against bad coding practices is important, but it's not the only thing.
I am pretty sure most of us have experienced both sides. There are planty of things that can go wrong in either side, one is less efficient but has the upside of hiding merge conflicts until the end and gatekeeping the codebase from bad developers.
Can't wait for all the supporters in here to get in the field and deal with the struggles this methodology lends itself to. Hopefully they won't become cultists over this and one day will be willing to give branching a chance.
@@ContinuousDelivery I highly value your ideas and videos, really. But this no-branch attitude lends itself only to really senior developers. Many other people have mentioned code reviews. The code needs to be reviewed. Why? Because programmers are sometimes too creative with their design, especially the mid-levels who see themselves as seniors. I do agree with a few ideas and it's how I do things: - all branches have to be releasable at any time; developers merge the master branch into their branch every 15 minutes / once a day. This is continuous integration - once the code has been reviewed, it can be merged into master. This can happen once a day, or once every few days - the developer switches back to his branch and continues to work and the cycle restarts This way: - we protect juniors from their own mistakes - we protect the design - we can do pair programming, but doing it all the time is just too much; trust me, I know my juniors - we have design meetings; problem: the junior of the team does not really see the same imaginary code as me when we do the design - and yes, I explain a lot, we do enough pairing, but at the end of the day time efficiency also matters. And releasing often bad design is not time efficient I'm a CTO at a smaller company btw.
I think , this is good for small teams that consist of experts and seniors , but when it comes to large projects with many people and the skills vary, then we need a strategy "called PR" to review juniors code as example. Thank for your video . I enjoy it
Well it scales very well, and I would argue that hiding what people do from one another (branching) may be a defensive strategy for less experienced teams, but is not going to help them improve. My own preference is to combine this with Pair Programming, that way you can mitigate the risk of inexperienced people making a mistake by pairing them with someone more experienced, and they also get the chance to learn more quickly. I describe my thinking on this in this video ua-cam.com/video/aItVJprLYkg/v-deo.html
@@ContinuousDelivery it does not scale well at all, you shouldn’t be teaching something that will lead to more problems. There are no benefits to not branching that can’t be solved with some planning and communication, just because you can’t doesn’t mean that everyone else shouldn’t. People should learn to use branches because that works best, and merge conflicts only mean there was a hole in communication.
@@skylark.kraken Are you speaking from experience of having tried proper CI, or are you just pessimistic because you have no confidence or trust in your team or having been disappointed too many times?
@@skylark.kraken Notwithstanding that every working copy is a branch, every commit a merge... less experienced developers will more often mess up and take longer to complete complex merges... There are secondary benefits to the frequent integration approach - protection against device failure and interruptions, for example. I think the point here is less in the branching itself and more in totality of the approach; small changes, backed by automated tests... Smaller changes means less to go wrong / less to fix if there's a problem...
Yes, this only works for small teams, small projects, or projects that are not mission critical. People who believe in this will be in for a big surprise when they joined a big company or a big project. Just wait and see.
Thank you for sharing this! When the video started I thought, "this is a terrible idea", but the methods you presented for safely dealing with partial features make a lot of sense. I think branches still seem to make sense for open source projects because of the distributed decision making process in the community. But inside a development team I can definitely see this being a lot smoother.
I worked with both branches and incremental changes on master. They are both valid. If there are more than 15 devs in you team it will be very difficult not to spawn new branches.
Another benefit to CI compared to long-lived feature branches - if there is a merge conflict, it is with the code that the developer wrote *in the last 15 minutes!* This code is fresh in the developer's mind, and they will have an easier time fixing the conflicts rather if the conflicted code was written days or weeks ago.
or just merge from the main branch into feature branch every day or even after every other person merges into the main branch. It still keeps your code close to the truth and avoids stepping on other people's toes
well it's merge up from MAIN and run the tests but the problem is when you merge your work to the MAIN, Others should get the changes and run tests on their work and if they are close to release they could get to the state that their long work is no longer relevant and they would have no more time to fix it so they cannot integrate their work to MAIN and the reason is lack of informations about what do the rest of the team doing
@@AloeusCapitalManagem no just too many people working on same system. Usual problem was when someone changed something in the geometric kernel or similar module which everyone used
I've worked with a bloke that had the opportunity to work with you back a decade ago. When I'm watching to your videos I can totally imagine he explaining his values, approaches and techniques. Don't mind the people that didn't get what you're trying to explain them. They will, certainly, learn this one way or another. As more videos comes, I'm sure you'll tone down a bit and reach to wider audience. We need this wisdom. Keep pushing mate. Awesome content.
Interesting topic. I believe that this is the way forward; however, one concern I have is related to code reviews. Im not worried about syntax relates issues; this can be automated using static code analysis. I'm mainly concerned about the architecture/implementation of the feature as a whole. When you make all your feature changes within a single branch, you will have a holistic view of that feature's design and implementation (something which can also be used as a reference). This is something which you will lose when committing directly to master. Just wondering how you have solved this problem. Maybe it's as simple as tagging the commits with a feature id, but review tooling just isn't ready for this scenario.
Yes, I think that needs more care. The best way to achieve that design overview is pair-programming linked with regular pair rotation. I describe that, and some of the reasons why I value it so highly, in this video: ua-cam.com/video/aItVJprLYkg/v-deo.html
@@ContinuousDelivery Nice video :-) I have to say I usually have different opinions, but I really appreciate your work and ideas. They make me think :-) Pair programming is usually advised as a solution for absenting code review. However, I would be interested in your opinions about pair programming in terms of distributed team - not only geographically, but also across multiple timezones. There might very small overlap suitable for pairing. Thanks again for all your work, I really enjoy thoughts it sparks in my mind.
@@milavotradovec1746 Thank you. On remote pair programming, I once worked for a financial trading company, based in Chicago, I worked out of the London office, but the team that I worked closest with was based in Chicago, a 6 hour time difference. We paired during the couple of hours of timezone overlap and found it helped strengthen the bonds in the team. It is different to full-time, local, pairing though I think. I think that it helped the team a lot, but there was also a lot of work that was done not-pairing. The way it worked best for us, was I had some stuff that I was working on and the people on the team in Chicago had some stuff that they were working on. I was the more experienced dev, so that it nearly always ended up that I paired on their stuff. I'd give it a go, and experiment a bit with what kind of things you choose to pair on to find out what works best.
@@ContinuousDelivery Thank you for your reply and sharing your experience. One more question - when you worked on your stuff alone, were you still committing directly to the trunk (main, base, whatever :-))? If so, how were your colleagues in Chicago aware about the changes you did? Thank you very much for sharing this, I find it very inspiring.
Hi. Big fan Continous Delivery (the book) and Accelerate (the other book) here. I've been implementing build processes for more than 10 years for a big variety of projects and I'm a fanatic on software quality. I agree with all the branch by abstraction, feature toggles and dark launch. I have a fundamental disagreement with forcing to work with a single branch. Let me explain my point of view: I want to have the highest software quality. To do so, I automate all the possible quality checks in an automated quality gate, typically: - stupidity checks (duplicate DB migrations, leftover console.log etc ...) - all the necessary linters and static code analysis to spot code smells - unit tests with high code coverage metrics (for code with cyclomatic complexity) - architecture rule enforcement tests (using ArchUnit) - mutation testing - software dependency scanning for vulnerabilities - end-to-end testing for most of the features of the application - UI regression testing with screenshot comparison Well implemented, those checks take 40 minutes wall clock time on a web project after 10 men-years of development (using a standard Spring + Angular stack). I want to deploy to production at least multiple times a day. At any time, tip of the main branch should be production ready. No-one should ever hold his breath before a change getting deployed to production. I've been doing that successfully, with high throughput, over multiple years. How: using short lived branches that have the full quality gate running before being merged. The development cycle is the following: - pick a task to work on - start a branch - as soon as you have some change whose quality is high enough, push, make a self-code review, and get it merged (after pair-review if needed) as soon as the quality checks are ok. - You don't need the task to be complete to merge, you just need the quality to be right, to merge. - You'll typically make quite a few merge requests before completing the task Typical cycle (between branching and merging is less than half a day). If someone wants to keep a branch longer than that, no big deal, he merges the main branch into his development branch as often as necessary (I recommend at least once an hour). He'll have the burden of merging, as he is the one choosing to move away from the "short-lived branch" recommandation. His choice. The higher the threshold of the quality gate, the more frequent the checks are going to be red. If it's red on the main branch, we cannot deploy, so we test before merge. You understand that with such a test battery, considering pipeline 10% of the pushes leading to build failure means the main branch will be red more often than not. This means team blocked hunting for the change failiing the build. Testing on the branches allows as well to: - test the build process update before merging (new compiler chain, different code coverage tool, etc ...) - review app: review UI changes live before they get merged by deploying the branch to a short-lived live environment - code reviews for knowledge sharing (I've read your arguments for pair-programming rather than code review. I'm a big fan of pair-programming, but it's just not always possible, sometimes simply because the work schedule is not uniform among the team members). Conceptually, a developer is always branching. He's working on a local copy that IS kind of a branch. Let's just consider the short-lived branch the equivalent of his local copy. If one is force to push on trunk, he'd better run at least a significant part of the quality checks before pushing. Running well written unit tests for a big project can easily take 5 to 10 minutes (we have many thousand of unit tests). I should run that locally before each push ? Why doing it manually, when tools like Jenkins, Gitlab-CI and many others can do that for you ? Hardware is much cheaper that developers, auto-scalable build farms are just a click away. Short lived branch allow this process to be asynchronous, and let the developer do something more interesting than waiting for the build to be ok on his laptop before pushing. I've now moved on a project that advertise doing trunk-based development (based on your very arguments). Tools have no build-on-branch support (GOCD). Quality gate only contains unit tests and some end-to-end tests. There are just a few deployment to production a month because confidence on the quality is not high enough. There are more days where build gets red at least once than not. I have a strong feeling trunk-based development gets in the way of having a strong quality gate. You can't have both, and without the later, you cannot safely do continuous deployment for mission critical apps. I suggest the following motto: "Long live short-lived branches" (no copyright). I'd be glad to read your feelings about this.
I think that we are 90+% aligned here. A few thoughts... If your "typical cycle (between branching and merging is less than half a day)" then you are doing CI, so I have no argument with that at all. I think that you are doing more typing than me, I work on master locally and then merge all my local commits immediately that I have made them to origin/master. So I don't have to create any branches or merge to them (less typing for same effect that you describe). To be honest, this doesn't matter at all, if you want to type a bit more it doesn't matter. "The higher the threshold of the quality gate, the more frequent the checks are going to be red" Not so! The slower the feedback loop the more often the checks will be red! Adding branches slows the feedback loop. The problem, as you correctly point out, is the efficiency of the feedback loop. One of the more subtle effects of feature-branching, in my experience, is that it gives teams more room to slip into bad habits. Inefficient tests being one of the most important ones. If everyone is working on Trunk, then slow tests are a pain, so the team keeps them fast. I think that most of the rest of what you are describing is about feedback efficiency. Let's try a thought-experiment... If you could get the answer (is it releasable) in 1 minute, would you bother branching? If not, then what we are talking about then is where is the threshold where the efficiency gains of hiding change (branching) outweigh the efficiency gains of exposing change (CI). I think that CI wins hands-down as long as you can get a definitive answer within a working day. Practically, the shorter the feedback cycle the better, but my experience is that under 1 hour is the sweet-spot. That gives you lots of chances to correct any mistake during the same working day. I built one of the world's highest performance financial exchanges and the Point of Sale System for one of the UK's biggest retailers, and we could evaluate the whole system in under 1 hour. So my preference is to spend time on optimising builds and tests, rather than branching.
I like the principle of continuous visibility of changes. However, regardless of feature branch vs CI, there are few people who will 1) actually look at the commit logs of changes pulled, or 2) see the changes made by others when working on their own ticket. The only thing that wakes people up is merge conflicts. Must say, among the merge conflicts I face using feature branching, very few are due to mismatches based on a fundamental approach; it is minor things like imports/formatting. I like feature branching because a reviewer can see things like architecture of a feature within a PR. I fear that CI will make people approve minor changes with less critical thinking about the big picture.
I have the same thoughts about code review on larger features. It seems like it would be impossible to do proper code review. I would love to hear Dave's thoughts on this. The other thing I don't like about committing incomplete features to the master branch is that it makes it difficult to identify zombie code. If you see code that doesn't appear to actually be used, is it zombie code or is it a necessary part of an incomplete feature? You'd have to actually ask the developer who wrote it. What happens to features that were abandoned half-way through? Seems like clean up for that would be a PITA as we can't simply revert the merge commit for a feature branch because the feature was committed via countless tiny commits.
@continuousdelivery I would also greatly appreciate your thoughts on the above situation. A FB PR gives a more holistic view on what the problem being solved is and the architecture decisions made to deliver it. The reviewer can more easily see if there is an issue. However a senior reviewer would likely notice an architecturally poor choice very early on, thus saving the junior dev from continuing down the wrong path.
I have your books and all the Devops , Martin Fowler series. You ability to speak clearly and know all the pitfalls is A+. Steve McConnell is also someone I deeply respect. Maybe you can comment on his work in a video or have a joint discussion. Thx for all your hard work.
If your feature requires changes to shared libraries and database schema, how would you handle potential breaking changes to other areas of the code if you don't isolate them by branching?
You can always integrate your work by syncing/merging the trunk with your feature branch. By keeping your work current with what happens in the trunk you ensure that other features that are introduced work with your changes all the way until merged back into trunk.
Yes, but that only solves the problem as far as what has been committed to trunk, if everyone else is Feature Branching too, then you have no sight of what changes they are making until they merge to Trunk. This is certainly better than not merging frequently from Trunk, but it is not the same thing as CI.
@@ContinuousDelivery that is true, but the only thing I need to consider is what the trunk has, which is "the truth" that is in production. Same for others working in their feature branches. This is however when people work on things without knowing anything what others are doing, which should be rather uncommon unless there is no communication in the team - the occurrences where different team members work on the same bit of code without each other knowing should be rather minimal so "collisions" should not happen unless the code is badly structured as well. By continuously testing my code with the latest changes introduced in trunk I do get CI, I don't see how it isn't. This also helps when my changes are about to be merged into trunk with a pull request, so that other team members get to see if any changes are in fact in line with the current code base and follow the guidelines and principles/best practices the team has decided.
@@strangnet The problem is that if you have a team of people all working like that then except for the last person to commit, trunk isn't really trunk in the sense of the version of code that will go into production. So everyone is testing against a version of the code that wikll never end up in production. In true CI - the definittion of which includes the statement "merge changes into trunk at least once per day" this is not true, at the point of every merge, trunk is definitive. All this is just our opinions though, the data says that if you have feature branches that last for longer than a day, wether you merge from trunk or not, statistically you produce lower quality.
Some features don't pan out or take longer to complete or require several user stories to be done before the feature could be considered complete. If that code is integrated immediately then should it not pan out you have to "unintegrate" it. Sure you can use feature flags but if you have to pull the feature you have have to remove / undo that code. Forward integrating feature branches on a daily basis is the cleanest solution imho.
@@Wouldntyouliketoknow2 a branch should not live longer than a what a sprint is. It can start in the middle of the sprint, and be finished in the middle of the next sprint.
Agreed that they should be very small integrations, integrating at least once a day, but committing and pushing to master allows junior developers to add flaws to master without code review. Merge commits from work branches to main allows associated work (first attempt, write tests, second attempt, add comments) to be grouped as part of a single commit. The commit graph is much more clear about when and why certain changes were made, and a rebase to remove problems can be MUCH simpler.
The premise is that my small daily commits doesn't break the app even it is not in production, so my colleagues can work on their tasks. But some tasks with a lot of refactorings take days to finish and it could make harder for others to work. Also you would get rid off pull requests that should be reviewed. But don't get me wrong I hate long term branches and dealing with conflicts so we try to minimize size of branch to minimum. Also feature flags are good approach we implement very often. What I prefer is to pull from development branch often so I keep up with current state.
I would argue that refactorings are the best use case for trunk-based development. If you are refactoring a large part of the code, you’d better ask other people to stop working on that code or you’ll end up with a disastrous merge hell. If you can organise a refactoring as small batches of work to be integrated frequently, definitely do it.
@@andrealaforgia so you arguing that if you doing a refactoring the 5 other people in your team need to have a pause 🤷🏻♀️ Well I agree that while doing some heavy refactoring, generally, nobody else should work on the same code part. But that doesn’t mean you need to do it in the trunk. Refactoring can sometimes not work well and it’s much easier to verify and to revert changes which were done in a separate (feature) branch.
I really enjoy your videos and content. While I don't feel I'm radically enlightened I see these as a great opportunity to share them with less experienced engineers, as the clarity of your arguments is amazing. Thanks
9:30 (DevOps Handbook) I am not used to this workflow but I can see it’s merit. In my experience, the biggest bottleneck with feature-branch workflows tends to be code reviews (not branch maintenance or testing feature branches). However, code reviews are also the most important step for ensuring code quality and readability. Google/Facebook do NOT bypass this. I think the benefits the DevOps report outlines might be true with trunk-based: - higher throughput - better stability - higher job satisfaction - lower burnout However I don’t think it’s particularly comforting knowing people can commit anything at anytime without any oversight. I’ve seen codebases use this sort of style with just a linter and the commit history was a really long mess (and the code was a mess too). Really was a nightmare having to work off of that. feature-branch workflow compartmentalizes work much better and has a controlled process for integrating. Trunk-based might get you to the end faster but throughput isn’t everything. For example, a project that’s “released” but is being maintained with occasional bug fixes or other minor changes would not benefit at all from continuous integration. I guess it could work on completely green-field projects but most jobs are working on existing money-making products. Just to re-iterate, even Google/Facebook who supposedly use this trunk-based workflow STILL have code reviews. Ultimately the biggest bottleneck is waiting on others.
Thanks for the great videos, but there is a basic flaw with your logic here. That is that the feature branches are somehow completely broken apart from the main branch or trunk. There is nothing wrong with having a long lived feature branch as long as you keep merging the trunk back to it. Your graphic is not quite complete, it only shows the merge of a feature to the trunk. But there are many trunk to feature merges along the way while the feature is being built. This can be done fairly easily and this way your feature is always updated with latest code. The feature itself can be tested completely through CI and if tests pass then you are good to merge your feature to trunk. Adding feature flags, dark features, etc involve huge amounts of extra code/tests which are not necessary. Most of the time this will probably add performance issues and extra unneeded code. In some instances this could be impossible if the new feature demands a DB migration or something similar. Having feature branches or even long lived features branches doesn't mean you are not continuously integrating.
Well, syncing with trunk for a long-lived FB is a good idea, but it doesn't solve the problem of what is going on on everyone else's "long-lived FBs". I am not saying that an FB is "completely broken" I am saying that it is not the truth and you can't tell wether it is completely broken or not. You can reduce the chances, but you can't tell until you merge into the version of changes that represents the "truth".
Edwin, I think you are correct in the observation that dark launches, branch by abstraction, and feature flags sometimes require extra code. But I think the extra code is likely to only lead to a higher quality code base and/or better user experience. With extra code, I mean facilitating code by the way, I think you meant the same? Because there will of course be multiple "feature variants" in the code base, but that is not different with FB, just in branches instead of files. For dark launches, actually no facilitating code is strictly required. For branch by abstraction, the facilitating code (when done right) are mostly interface definitions and IOC mechanisms. Chances are the project will have these anyway, or otherwise are a good addition for other reasons. Lastly, feature flags are a bit more complicated to evaluate. First, they require facilitating code that many projects do not have and would only be a low priority for other reasons. Second, naive implementations impose a configuration burden on the ops team (aka yourself if devops). In my view, feature flags should therefore not be configured, but be exposed as a user preference setting (this can be an internal user of the code, or the end-user). If this is not desired by the user advocate (ux designer, productowner, whoever), then I can probably be convinced to use one of the other strategies entirely. But if the feature flag is desired by the users, then the facilitating code actually improves the user experience, and is not a burden at all. For these reasons I completely agree with Dave that feature flags are more "risky" than the other mentioned strategies. That said, I have in the past dark launched a feature-toggling interface to great effect, and I can't imagine any user advocate not wanting this for themselves (even if they want to keep it hidden from their users indefinitely). When done correctly, the facilitating code should not introduce performance issues, or this indicates other problems. With DB migrations, that should be done by a branch by abstraction on the DB schema itself anyway :-)
pain with long lived branches are not in master, it's in branch. So you can update your branch as many times as you want, but others will not see your changes. And you get merge conflict not because of you pulled changes a week ago, but when you try to merge 2 files that edited same lines. And when you have long lived branch chances that someone else edited that is higher and these people introduced changes in their own branches and then when you want to merge you get a mess, because you have one logic there, another logic here and who is right? If you have highly decoupled work, where you don't work with same code as others, then long lived branches will not create conflicts. But they (long lived branches) will increase business value delivery, regardless.
just merge the current "develop"-branch (source of truth) into your feature-/bug-branch before starting to work on it again, at least every day. It's that simple. Nothing else ever worked with > 10 Devs on one repo. Channels like this show, how far away the theory could be from anything practical :)
@@kajacx It means you were the only one doing the rebase, if all the team does rebase often you will rarely get in that state, I've been using git rebase for several years and very rarely hit points where it was a hell. It is hell if you are the only one doing a rebase.
This approach kinda assumes that no one makes mistakes. Imagine someone is committing every 15 minutes. After 2 hours someone realizes that there is a huge issue with the new code which affects everyone's ability to work on their own tasks. Writing tests won't always help. Things can slip through. Also writing tests that early is often times a waste of time because you don't even know what the final version will look like. Thanks for your video tho!
Tests should be written on the feature-level, the implementation details don't matter. This is not always possible, but I'd say in 99% of cases it is. Not that I'm convinced that branches should be avoided, but I do think that frequent merging would be useful (again, in 99% of cases), but needs to be done right.
@@defeqel6537 Well, there are different kind of tests. Every test kind has its place. I don't see how this is referring to my comment btw. Frequent merging is good, indeed! But you want to merge a fully refined increment not a WIP state to reduce unnecessary friction.
@@Mio2k10 I agree, and disagree. Generally, yeah, fully refined increments should be preferred, but as long as it is flagged out, or similar, I'm starting to see a benefit with WIP state changes being merged too. I do think merging WIP requires care though, otherwise may lead to just frustrating your team members. As for testing, my point was that the tests (usually) shouldn't rely on what the code looks like, so they may be written quite early on, after a bit of planning. As for test types, I prefer unit tests, by which I mean code I control the inputs and outputs for. IMO people should avoid micro-testing, where they try to test each function or class separately (unless those are your deliverables, in which case thoroughly test them).
Thanks for this awesome video. I hear what you are saying, my question is how should you handle code reviews in this case? If I understood correctly, the desired process here is to push to master as often as possible instead of hiding the changes in a branch, but it also means the code needs to be reviewd by someone else(teammate) every 10 or 15 minutes which in reality is very disturbing(in terms of context change for the reviewer) and blocking. What's your point of view on that?
I would suggest using upstreaming for this. In such an approach the whole team is committing into a develop branch regularly - this becomes the trunk. Then separate "merge request" branches are created and relevant code is checked out into them using "git checkout develop -- " and then these are reviewed and merged into master. After that master is merged into develop to bring the two branches in sync.
@@mkschreder Then you're back to multiple branches that aren't always in sync. I would suggest using short-lived branches that are merged within 24 hours. Then, you'd need to make sure your team is reviewing and approving PR's daily.
@@d1morto yeah but that doesn't always work to review daily. The idea here is to commit to develop and then move commits into review branch (or just move changes) I agree that pushing the team to review daily is important but some teams are very slow at this and can take a week to review something that they may not filly understand.
@@mkschreder The devs would sort through the develop branch looking for their individual commits, and then move just those commits into a review branch? Wouldn't that be tedious?
I see a lot of discussion that feature flags solve the parallel development issue, and I believe they do, to an extent, but they also increase the complexity of the code and make it more difficult to reason about. How does flag x impact flag y? Ahhh I hear you say, well that’s all handled in testing and ci. That’s true of feature branches as well. Branches increase process and development infrastructure complexity. Feature flags increase process and code complexity. Both require coordination and communication between team members. I’ve used feature flag extensively for rolling out experiments in the UI/UX space and they work well. They’re isolated and have a finite lifecycle (the length of the experiment). Cyclomatic complexity has long been shown to a have a major impact on system maintainability. Feature flags by their very nature increase cylomatic complexity. PS: Buying Daves book, reviewing considered opinions are always worth your time.
I think that you can implement Feature Flags in different ways, particularly if you use them in combination with one of the other strategies. Feature flags don't have to be in the guts of the code if the design is good. But I agree with the point. I am wary of Feature Flags for a related reason the explosion of potential code paths - what do you test?
First off I agree with a lot of what you are saying. Certain problems (on an otherwise amenable project) tend to require workshopping before release though, and I feel that in cases like that feature branches and a bit more integration work can be worth the headache in order to get things right. Also, maybe I just don't know how, but I can't figure out a great way to do mandatory code review in a "commit anything that is tested as often as possible" codebase without basically mandating pair programming. That wouldn't necessarily be the worst thing ever. I like pair programming, but I also enjoy programming alone, and I think they are both beneficial. In the projects I have worked on we tended to stay somewhere in the middle and the communication in the daily standups generally kept merge conflicts to a minimum, even when some branches lasted for a week or two.
@@paulprescod6150I am not so quick to jump to that conclusion, but I am sceptical of that aspect nonetheless. The biggest thing is in my experience developers are more willing to make improvements to code that isn't in the mainline branch yet than something they already got merged. Analogous to how most developers won't write tests for code that "already works".
Can't upvote this enough. The whole git workflow has made merging and testing a nightmare. I've had developers overwrite code many times due to peer merging as well. That this is what everyone does these days is just bad. Great video!
Although I'm fairly convinced that the way we develop our application (News Site Backend) is fairly optimal for what we do and how we do it, I watched your video with an open mind and tried to see the benefits of it. However, I couldn't find a way to make this work in our environment and being beneficial to stability, increased efficiency during development and deploying realeases. There was always a tradeoff at some point. So my argument is that concepts which were developed maybe one or two decades ago are still valid concepts but with different constraining parametres. Your argument was that a code should be always in a deployable state - which of course is always given for any branch which is deployed on dev, stage, release or however you may call it. On the other hand it doesn't make sense to commit unfinished code with unfinished unit-tests to a dark release or feature switch - even in the dev-stage - when it is simply not finished because it will inevitably break things. Just for the sake of commiting code every 15mins (you don't open a bridge to the public if its only halfway built). So my argument is, that it is probably a good idea to work in that way but adjust the constraints to the environment they apply to. Maybe commit the code not every 15mins but only every 1-2days when a feature is in a state where it actually could be integrated without breaking things. Another parameter (might) be the actual size of the codebase in relation the changed lines in every commit and the number of devs working on it simultaneously. Either way...good content and nowadays it became equally important to actually know how to organize your code compared to know how to write it.
I try to minimize branching, especially long lived branches. I also think a good starting point is to have only few branches at any given point in time. To facilitate this; do pair programming or otherwise divide the work, to work in parallel.
This may work in a "code-only" feature, but in my world, many features require data changes in persistent storage like databases. I'd rather hold on to a feature branch for an extended period of time and (more than) daily pull in changes from main. This way I'm the one solving my own merge conflicts and push to main would be easy.
Since we switched to a model.broadly similar to this, we almost never have merge issues any more. We do not merge branches every 15 min, compared to merging once a day I do not see the added value. Developers usually do not work on the same piece of code on a single day, and having to synchronize your repo with the origin every 15 min seems like an overkill. But as said before, with a more manageable timetable, it's indeed a very good practice. But every video I see on the channel I can recognize myself in most if not all the advices given, good to see someone with some real experience take on youbute!
This is really great and makes things very understandable, thank you. This seems like an optimal strategy. But, isn't frequent merging to main going to generate a pile of merge/pull requests? Who typically reviews and approves these, and how do they manage this volume?
From my understanding, you'd commit directly (without making any pull requests). Ideally, the code you're commiting and the existing code base are covered by tests verifying all behaviours, so it gets a lot harder to introduce bugs without tripping the alarms.
Your topics are very informative, these are hot topics and our teams discuss them in the retro. I personally, enjoy reading the different perspectives in the comments.
Dave, I think the real discussion should be: code review vs continuous deployment. It feels to me that this is where the friction comes from, not the branches themselves, which are used as a means for code review. Maybe it's worth a video?
If you are not using pull requests or mobbing with everyone on the team then what is to stop unreviewed code is getting into the mainline in an uncontrolled way? I know XP is explicit in saying that (at least) pairing is necessary for this to work (partially). What if even pairing is not an option for whatever reason?
Branching shouldn't be responsible for making sure code is reviewed before it's deployed to production. Use jira or whatever to track status of commits and only deploy the application to production when tasks have been code and feature reviewed. CI is supposed to encourage collaboration as everyone is working on the same codebase.
I could see this being important in certain groups/situations, but not all. I think the official Git book says it best when they point out which methodology is used will depend on the group's needs. Issue branches are critical to the way we work since we need to be able to thoroughly test the change, implement it when ready, and be easily able to roll it back if necessary. In general we don't see merge conflicts, and if we do then that's a good indication that the changes need even more testing to ensure a quality product. With all that said, for the most part we only update the training production) load itself once a year unless we have interim critical items that need to be addressed. Stability is more important that rapid deployment for us, which I get is the focus of this channel.
This approach is certainly not just for simple systems, and is increasingly used in safety critical systems, because the data says that this is safer. It's safer because it promotes working in much smaller steps, so each step is less risky. For example, SpaceX practice trunk based development and update their rockets up to 40 minutes before a launch. This includes launches that carry people up to the International Space Station. This also how Tesla work, Amazon, Google and Facebook. Also several banks around the world and increasingly medical device manufacturers.
Great video, as always Dave. It's good to know that some of the strategies to avoid branching have to do with a great effort I've been putting into releasing a more intelligent and manageable Feature Flag platform. Thanks for sharing your thoughts.
For the big teams out there that cant get around branching, Just make sure you cant merge a branch before its been rebased on master (git hub can help you here) then rebase your branch often and don’t keep long running branches. And make sure your software is covered by automatic tests. Ive worked on multiple big teams releasing new features multiple times a day using this rebasing strategy without having to forego the use of branches.
I think that is a work around. It is certainly the best way to work on branches. The problem with merging from trunk to your branch (which I agree is a good idea if you work on branches) is that it still doesn't represent reality, what about all the changes that all your team-mates have on their branches. This approach certainly reduces the chances of a merge conflict, but it doesn't reduce those chances as far as CI does.
@@ContinuousDelivery I see your point, and for most projects, it makes perfect sense to just work directly on master, so long as you keep it in a releasable state all the time. But I think at some point a team can get too big for that to be a viable way of working because the criteria for it being a success can be difficult to achieve (not impossible at all, but difficult).
@@henrikoldenborg86 Google do it with billions of lines of code, so I don't think that scale is a limiting factor. It is a different way of working though, and so that is a challenging switch of approach.
Having implemented ci and cd last century, I can tell you this is a gross oversimplification. Branches are necessary when the need to isolate changes justifies the overhead for having the branch. There are many reasons to branch ( yes, even by feature) and there are many reasons to not branch. The problem lies with knowing when to apply the method and when not to, and unfortunately many developers have such a low understanding of branchinh that they even conflate directories with branches. The factors involved with these decisions can involve everything from the sw architecture, the competance of tge team, the competance of the vc tools merge capabilities for tye files concerned, and yes, even the release management process. Over the decades I have seen so many people try to oversimplify this and have had to clean up the mess afterwards. At the very least your video should provide caveats like " if the merge tool for the code files is not reliably automatable for >95% of merges ( and yes there are tools that exceed this, and git ootb is not one of them) and your development is project based, not product based with product variants and/or your system is not a safety critical system and/or your dev teams are poorly trained on the effective use of branches etc etc etc. To give a real world example of the impact of branch avoidance, I jave seen an entire product line have to be redeveloped as each product variant was a copy pasted directory of the source code (12 product variants). Despite having the same people maintaining the codebases over a number of years, each copy of the code was so completely dissimilar that a core change across all of them worked out to be cheaper to redevelop the entire product line from scratch ( a $12m decision just taking into account the costs for re accreditation of the variants - let alone the cost of redevelopment and opportunity cost to the business.
Naturally, I don't think that it is a "gross over simplification". It is certainly a simplification, because it is a 15 minute video! The approach that I have described is, and has been used, to build large complex systems all over the world - I have worked on several myself. Occasionally we found the need to branch, and it felt like a failure when we did. I don't think that there is much about software development that is easy, but I also think that, as an industry, we are exceptionally good at over-complicating things. Branches being one of those times where we are prone to mis-using them. Any branch is "running with scissors", it exposes you to risks with BIG consequences if you get it wrong (as you point out). Of course, this doesn't mean that every time you create a branch you will fail. It means that your chances of failure are increased. CI has a different set of compromises and fails more gently IMO. My intent with this video was to point out the costs, and benefits, of both, but naturally my advice to any team is to "prefer CI to branching" and "always branch reluctantly and with care". (I probably should have said that in the video - sic!). I agree with a lot of what you say when you describe naive approaches to branching, cloning different versions of products is a horribly common practice - and terrible solution.
@@ContinuousDelivery an excellent clarification. It is vitally important that branching strategies are applied by skilled people who know how to balance the various factors within the constraints provided. Unfortunately, there simply aren aren't enough people who do. CI and CD are good methods when applied appropriately, however, I continue to be concerned about how CI and CD are being pushed as being the default or only way. A recent example of the consequences is actually the recent solarwinds hack. It looks like they had a fully automated CD system with a view to releasing quickly, which on the surface seems like a good idea. That is, until hackers put a backdoor in the IAM modules and let tge CD systems punch it out to prod, and then to 16k of the worlds most crucial orgsnisations. Having a fast means of punching through to production is a good thing to have in general, however for some systems, and for some areas of systems, a more rigorous control process is necessary. The approach of fail fast, fail early, fix quickly is simply not viable for some systems where the failures are safety critical. Also, with monstrously large dev endeavours like I have been involved in ( 1600+ developers concurrently, for example), effective and consistent branching strategies become essential for the efffecttive coordination of change implementation. The result of not branching in these situations leads to high numbers of regressions and an explosion of inconsistent environments.
@@ContinuousDelivery I can't take seriously someone who considers themselves a failure for using a branch, or compares it to running with scissors. That is the sort of holy war language that doesn't belong in a real discussion. Branches work best when they are used like transactions in a database. They should be small and short lived, but they should also be consistent and complete changes (not complete features).
@@tube4Thabor As an embedded software guy I couldn't agree more. There are so many different kinds of software development, creating one nail to solve this problem turns all projects into hammers. And sometimes forcing the project to be a hammer when it's not is way more expensive than choosing the right process in the first place. It's nice to have this tool be available, it's not a solution for all projects.
@@tube4Thabor "Holy War" Yes! It's surprising because most of the developers that get religious about any "development philosophy" including functional, OOP, branch strategy, code coverage, TDD, BDD, Scrum vs Kanban, etc, are *junior*. It should be about applying the best practices in the right amount, but not *all or nothing*.
we tried CI/CD with a small team of relatively green developers and it worked surprisingly well for game development, BUT this required a lot more management than when we did feature branches, and pretty much continuous communication between the devs. I can't imagine how a workflow like this would work with a team of over 4/5 coders or without a team lead that's keeping a lookout for issues 24/7.
I can only say that it does. I have done this many times with teams of 30 - 50 people, and often with teams in the hundreds. Google does a version of this with 25,000 people working in a single repo.
What would happen after you have merged your small changes to master and part of your code is in product but not yet release. You get told the feature is no longer needed or you need to pause work indefinitely. How do you make sure your partly finished function doesn’t pollute the code base.
Well, I’d question the sanity of the product owner who thought this was the most important thing to do a couple of days ago, and now wants to pull it. Then we could either leave the changes there, if we thought that we’d come back to it, or more likely delete them.
@@ContinuousDelivery So it sounds like you're not a fan of rapid prototyping, where you iterate frequently with user input to determine the final shape of the product.
@@jasonzuvela I don't think I am dogmatic, I have tried both approaches, so have an opinion on which works better. Sure, rarely, there are changes that completely invalidate the assumptions that your team has been working to. That is always going to be disruptive. If that happens after a CI approach vs after a FB approach I don't see that either approach helps more than the other to untangle the problem. You are, presumably, going to have to extract code that may or may not have been built on top of. This is not as trivial as merely reverting a branch. So I don't think that CI makes this worse.
I've seen developers committing directly to the main branch and never doing a feature or a test branch. The changes are fine and safe. They don't break anything. And then I see the changes reverting what they committed a week ago with a comment saying, "I've made a mistake, this design turned out to be a bad idea or I found a better way to do it." The next thing you know, so much crappy code are already in there that are non-functional. They're harmless but dead code and it just pollutes the code base.
I don't really have industry experience, so take what I say with a grain of salt, but from what I know of working on personal projects/open source, branching becomes an issue primarily when multiple people are working on big changes and don't push to master often enough. If I remember this video correctly, it discusses how using feature branching leaves the programmers in the dark because the master branch is always behind, and it puts an obstacle between testing features with other parts of the codebase because each feature is worked on in isolation. The issue with committing directly to master is that version control serves 2 purposes: 1. To share work btw teams 2. To serve as a backup if your local machine suddenly explodes Sometimes programmers commit code because they finished a feature, but other times it's the end of the day and they want a proper work in progress in the repo to pick up on next time. Programming is a messy job a lot of the time and committing works in progress that aren't properly tested and cleaned/refactored will cause confusion and unnecessary effort by other programmers trying to work off of buggy/incomplete features. Obviously, these wip commits shouldn't be put into production if they potentially break something. Using feature branches creates a separation of these 2 purposes: If you want to share the code, you push feature branches. If you want to save a WIP, you commit. Features should be atomic in nature: small enough that they cannot be broken up into further features, and they take a small enough time to complete that they are never far away from master. These features represent a handful of commits that you are treating as a single package to be integrated with the master branch. In a way, this can still be thought of as "approaching" continuous integration since your branch is never far off from master, but you are still keeping the master branch clean, while keeping it up to date. tldr: Yes, you should push to master often, but no, it doesn't need to be every commit.
So, I still am pro-branching, but I minimise it to a semi-daily merges that guarantees the work is always very close to the truth. Having unfinished work in production doesn't seem to work for me, as it generates bugs more than good results for the devops process.
Exactly! I mean, just take the argument that branching is "hiding" implementations and therefore cannot be continuous integration... equally, Dark Launching (or Blue/Green) hides the implementation from being in the Production system. The whole "don't branch" thing (and then going on to say that we should minimise branches) just pushes the problem(s) elsewhere in the development pipeline, just so we can continue to use the term "continuous integration." I'm happy to just have "semi-continuous" integration, I don't see why me or my team should be under any pressure to continually put things into Production.
@@redtela there's a mild difference that in dark launch the other developers and users don't run your changes in main line, but already may be executing your unit tests along the stuff they work on, so if they modify something your new changes will plan to use, it may break earlier, than brach-way. That said, I quite agree with your sentiment that branching and dark launch are quite similar, probably depends a lot also on the team, their overall habits, and the SW itself, what it does and how and how the work is split between developers. I can easily imagine situations where is no practical difference between the two whatsoever.
@@ped7g that "difference" (or lack thereof) simply tells me that the advice of "don't branch" is just a synonym of "make it someone else's problem." At some point, we all must take some form of acceptable risk. We follow gitflow (at my insistence), and so, we use feature branches. We also use unit tests, pull requests, integration tests, regression tests, manual QA and UAT. We use Pair Programming, BDD, TDD, and other methods. We have well over 1000 components in the system, each probably has 17 different "versions of the truth" and a hotfix can leave a developer laptop and arrive in Production within 30mins. Could we do it better? Sure. Are we "statistically producing worse code than if we didn't branch" - I doubt it. Could we release faster by not branching? Possibly, possibly not. Do we "hide" information from other developers? No, because a pushed commit can be pulled into any branch, at any time.
I've been professionally coding since 2008 and the best way to organize code I have seen was the one I thought was the worst. It was bank using IIS as the web server, and the site of the bank looked like a single site, it wasn't, it was hundreds of tiny little sites, each of them with its own repository, the manager assigning the tasks was the one controlling who was using each of the modules or tiny sites. And we needed to make a change, we had ownership of that module for as long as it took to make the change.
This was a very fascinating and informative video! I learned a lot from it but I have a question. How do you handle speculative developments where you are trying out ideas to see how feasible they are and how compatible they are with the rest of the system without committing them to the central trunk? I don't see how to do this without a branch. This is because you don't want your speculative work to interfere in any way with the production system until you are satisfied with it because you may end up discarding it entirely if it proves to be a dead end.
Using branching for that specific activity is totally fine and probably the only usage that is acceptable. Branching is not necessarily evil per se. It’s a Continuous Integration anti-pattern, though, so to be avoided when you want to integrate often with other developers in order to deliver changes as fast as possible and avoid merge conflicts (typical of long-lived feature branches). However, bear in mind that the longer your speculative branch staying unaligned with master, the more it will diverge with the rest of the development hence the more likely it is that your speculative work becomes less and less relevant.
For real speculative things, treat them as what they are, experiments, focussed on learning. Aim to always throw away any code you write in these experiments, the value is the learning. This means that you can be more radical in your speculations and aren’t constrained by having to make your experimental code “production quality”. This approach tends to focus you more on what it is you really want to learn too. Try and clearly state what you’d like to find out before you begin.
@@ContinuousDelivery Thank you both for clarifying things for me! This is kind of what I expected the answer might be but I wasn't sure. I really liked the way the method presented in the video prevents huge divergences in the production code. I've been through a few nightmarish project merges and that was back in the SCCS days. I can see where GIT could lead to even worse ones if you don't impose discipline.
I like the focus on true CI/CD workflow here as the ultimate goal. Personally I do like and currently use feature branches day to day. But we also try to keep our changes small and commits frequent so pull requests are small and they do go into prod on merge to main. Maybe this is a middle ground? If a team could ensure pair or group programming most of the time, then committing frequently to main would work nicely, but in reality that can't be maintained just because even the most committed pair programmer wants their own space sometimes. Anyway, my key takeaway from this is that implementing proper CI/CD is the key and how ever that is achieved then happy days! Nice video Dave I'll be sharing at work 👍
Branching doesn't mean you are doing independent work, it's a logical process of adding new features. Branching works in large development projects and is absolutely necessary. If you are doing something like Netflix, then you can proceed as you say, but the real world needs branches.
Not specifically, but I should do something on how roles change for CD. I recommend that you aim eliminate dependence on manual regression testing. That doesn't mean that QA has no place, but the role shifts to exploratory testing of new changes, not going back and checking that old stuff still works. This works most effectively when QA work closely as part of the dev team, evaluating changes while they are bening built, rather than waiting until they are finished. This is easier than it sounds if the team is following the CD principle of "work so that your SW is always releasable". I will think about doing a video that cover some of this in more depth, thanks for the suggestion.
At my previous job we used SVN with a trunk based approach, and we constantly had red test problems or even bugs/crashes that weren't even covered. This resulted in a stop that blocked 200+ people. Later we switched to Git, revamped the package and release system, and we were able to use feature branches as well. This solved many of the issues during development as we had coverage on our branches. OC we frequently synched with master, and we merged back the feature branch as soon as it was done which happened within a sprint. Since we had good code separation among modules the merges were fairly easy. The hardest part was figuring this whole thing out and creating the custom package manager plugins and Jenkins scripts, etc. But because of the previous experiences nobody wanted to go back to trunk based development although we could have solved the problem. With that said I think I understand your point, but we always have to evaluate our situation. Currently, I work on an enormous 15-year-old monolith software with a lot of legacy and new parts on top of it with more than 1000 devs. A code change in one place can break a lot of stuff at other parts and it's very hard to identify what went wrong. Not to mention that we need to wait until the next day to get test results. In this situation I wouldn't concentrate on trunk based development. At least not at this state. :D
There are certainly times when adding FBs can reduce immediate pain, but I don't think they make much sense as a long-term fix, for the reasons in the video. I have seen the opposite too. I saw a company that did what you did, divide up a large code-base, each team began working on FBs, but they had lots of problems because of unpleasant dependencies in their code-base. So they made the branches last longer, to ease the pain. It did ease the pain, locally, within each team. But when I saw them, they had not been able to build their system as a whole for over 18 months. The need to integrate is real, the longer we defer it, the more difficult it will become.
Also it's not like: #1 teams don't talk what the end plan is, #2 we can't look at other branches, #3 create our feature branch on top of another unmerged feature branch.. #4 rebase our work on top of any current branch.
Exactly, avoiding branches just causes a massive amount of problems. The only “problem” with branches is merge conflicts but they arise from a lack of communication and planning and it highlights what needs fixing for it to not happen again, handling the merge conflict itself isn’t even that bad and it’s less inconvenient than everything breaking suddenly because someone pushed in a rush and the bug wasn’t contained in their branch.
@@skylark.kraken But if the team writes tests, this "pushed bug"will not be committed until fixed, right? I think that will happen if the project does not have the test coverage
@@nero1375 Nope. You changed the Client history functionality and updated the tests. It's all working for you. I'm creating functionalities A, B and C that uses that history you've changed. Your code works and is tested, the moment I download it it can break my code, because I'm not expecting to receive your changes.
I just wanted to reach out and say that I recently have gotten in trouble with Branching... The issue is that I was working on a new feature, so I made a new feature branch, so far no issue. However this feature branch eventually started to explode way beyond the scope that it was originally supposed to encompass. The branch quickly went from Integrate with the new API and make a framework for it, to create a control to work with said API and test some stuff, to make some controls and use the API... and then it starts to get it to many many other areas of the program. Lesson learned. SMALL INCREMENTAL CHANGES!!!!
I think u making the assumption that there is one way of implementing CICD and that there is only one place where all feature branches converges which is PROD. I think with a good branching strategy (feature/(123) -> develop -> release/(1234) -> production ) with each branch or branch prefix mapping to a environment, life of that dev team would be much better. This allows for less conflict, two convergence points (merge) for all features (in develop and release) before prod and 3 test env (where 2 env: develop and release have the integrated changes) . Pretty sure there are other ways but a development team of atleast 15 members-without branching it would be a nightmare.
@@Stylpe Well, it changes code review to pair programming, according to the video and comments. If the best developers/teams merge many times per day, the code reviews have to go away. Even if you have the automated testing to cover your code confidently, you're still then missing out on reviews of a feature as a whole, and instead getting reviews or pairing on small changes without the context of the entire feature. Oddly, even Martin Fowler himself says that for actual features, not just small changes or bug fixes, the best practice is short-lived feature branches.
@@rhakka I'd argue those are issues with better solutions than feature branches, and which this style of CI will force you to learn, for the better. Better separation of concerns, dark/dormant features and feature flags, realising that you don't have to, nay, shouldn't, be producing new code 100% of the time but also review and explore other recent changes both for your own learning and as a kind of code review, having the whole team paying attention during sprint demos, understanding the difference between deploying and shipping.
I'm working on a project that, for political/organizational/financial/historical reasons, i.e. reasons "out of my team's, site's and organization's control" do require environments that we can't just virtualize or mock completely. So we're usually creating throwaway test environments that don't rely on some of its surroundings, but for some features we must create these throwaway-test-environments in the context of some long running services that are "production-like" and there we are relying in huge parts on manual tests. These are contended resources. So if I started pushing changes every few minutes to the main branch, QA could never keep up with manual and explorative tests and If a colleague was doing the same commit frequency in the same vicinity of the code, there would be a permanent conflict too. Taking that to the extreme it would be like having the whole team develop in Etherpad, scheduling a build+test on every save and just keeping those builds that pass the tests. Which sounds like a frustrating edit-war. The "Race to change first" is the same in the CI case, isn't it? Whether I commit a branch that contains my changes, or whether I commit directly on the main branch, we don't get rid of the situation that "the dog always bites the last one". If I branched on behavior and tested on behavior, what I'd get is a combinatory explosion on all behaviors available at that point in time. If we discovered a bad behavior in one setting that the tests haven't caught, and opted to revert to the old behavior, I might now have a new combination of another behavior that was built upon the newer behavior and might not cope with the old one, even if I thought the new behavior was equal to the old one, but just missed a small crucial difference. So doesn't behavioral branching just shift the burden to another place? From the branch to the system under test, i.e. from the dev phase to the test phase (that might even be different teams, based on the company structure)? As I understand the problem, it's not so much about the time a branch lives, but rather generally the lifetime of _any_ overlap in different behaviors, be it on a branch, setting or any dark or productive environment and shifting that overlap elsewhere doesn't really solve the problem. of course we cannot eliminate this overlap without blocking people from progressing. IMHO that overlap-time really boils down to the one crucial thing: The time it needs to verify a change works, i.e. to run all tests required. The faster the tests, the quicker the increments can come in and likewise because short increments mean less things to test, small increments are generally beneficial. However, while it's true that smaller increments are good for testing, we must not forget that there is a minimum effort for every test, no matter how small the increment. And that minimum test overhead is really what really limits the pace. If that's too high, branches/settings/dark environments will need to live longer to get things tested. In my experience shared, contended environments are always a point of pain and there are two things that make them miserable: If you have a highly contented branch, changing too quickly, you're never programming against something reliable, changing too slowly makes big bang changes that also create big conflicts. If you've got a highly contented test system, you never get to test something stable, if changes come too quick and you'll need to test too much, if changes become too big. If you never get a stable state, you're never going to be comfortable releasing, likewise, too large changes make you react too slow if something goes wrong. Every company has to find the best balance for them, review, adjust, try and repeat.
I'm a dev with 20 years professional experience. I disagree with this. Code should not land in master if it's not ready. It doesn't have to be perfect and bug free, but it shouldn't be horribly broken. Where do you keep this code while you are getting it ready. But I think there are some important rules for when working in branches: * Merge/rebase master into your branch often. * As soon as you have some parts of your code ready, get those commits merged in to master asap. A git rebase -i is useful to move the commits that are ready to earlier in the commit graph so that these can be merge without bringing in the commits that aren't ready yet. * Do pay attention to what the other devs are doing. If two devs are working on the same bit of code, then they should maybe merging each others branches.
He didn't state the assumption here, but I think Dave lives in Blue Sky World where there's good unit test coverage and you can see in 10 seconds or less whether you've broken something in main, so you fix it before pushing back in. Lots of high functioning teams operate that way, but it's not always easy to end up on such a team without orienting your career specifically around that goal. In any event you can introduce change incrementally, and I'd say adequate (and fast!) test coverage is prerequisite to even a hint of "pure" CI as described here.
@@vectorhacker-r2 , CI is just a means to an end. The objective is to have a quality code in the main branch. You can thoroughly test the main branch, which you should but you also need to do due diligence in making sure you don't continuously pollute it.
I'm pretty sure that continuous delivery is simply what we all used to do before branching became de riguer. Code mindfully and you don't break the code base.
This is terrible, just because you were late to source control and can’t handle branches doesn’t mean that your idea is good in the slightest. Stick to branches, it makes the development cycle easier and cleaner. Communication is needed and it solves all of the problems. Also if the code base wasn’t monolithic, oh wait you’re teaching against microservices and want monolithic code that can spawn all of the merge conflicts. I’m really disappointed that new programmers are looking up to you, they don’t know how bad what you are teaching is. Please actually teach what new programmers should be doing and not what you are wrongly thinking it should be done, times have changed. Feature branches are useful in that not all code should be in the main branch, you want to be able to isolate areas and work on them and have them reviewed before merging. By allowing everyone to push to main you are removing the review phase, and you only have everyone to push to the main branch if you can’t be bothered to communicate so it’s going to be left to everyone to review their own code. You want a dystopian developer experience and it’s horrible, just because you want to see what everyone else is working on easier and want half baked features now it doesn’t warrant telling everyone to destroy their code bases for it. Feature branches work extremely well, they require a little bit of communication and a little bit of planning to resolve all of the “problems” (which just flag deficiencies in how everyone is working), you shouldn’t be teaching everyone a backwards dated methodology. I’m just really glad that actual developers aren’t going to be reverting back to the Stone Age and ruining their codebases, it is here to stay unless we get something even better (pushing to main branch is not better, it’s a lot lot worse). Everyone who has thought they had learnt something here are just going to find out that they’re a massive fool and have to learn about feature branches on the job, it’ll be embarrassing for them though. You did describe yourself as “old school CI” but times have changed.
I wonder which of us has tried both? I have to confess that "late to source control" makes me smile, I wrote my own in the early 1990's 'cos we didn't have one for our project. The data is on my side "teams that don't merge to trunk at least once per day have poorer scores in Throughput and Stability" - State of DevOps Report. "If you merge your changes every day you don't have merge conflicts" - Linus Torvalds (Inventor of Git).
@@ContinuousDelivery with all due respect, could you be misinterpreting Torvald’s quote? I always thought, and practiced, merging frequently, but from master into my own branch. That way, when the day comes to merge my own branch into master, everything goes smoothly. What you propose may work on a very small team of very disciplined and senior devs. Outside of that, it becomes a crap fest of people pushing half finished features that break other half finished features from other people. As for building the complex systems you quote, could you be misleading people by not making a clear distinction between deployment and pushing into master? I bet that all those commits frequently pushed into master were not going live with the same frequency, not even daily. Financial and retail systems have strict policies about what goes live, when, and for what reasons. So you may had a swarm of developers pushing half finished features into master, but that did not translate into users, even a subset of them via flags, exercising these features until much later in the cycle. Then net result from a productivity perspective would be just the same as if these features were not merged into master until they were mature enough to be usable, but not before. So nothing gained but noise, friction, and arguments by having everyone pushing half finished code.
Commiting unfinished work to production is kinda scary. I also feel that the “behavior hiding” can introduce unwanted bugs and will need more efforts of removing these hidden behaviors in case the requirement didn’t push through or the requirements have been changed totally in the course of development (dev will have to look for each and every commit just to remove these tiny pieces of hidden behaviors). Also, does this approach considers the need for QA to test the changes? Does this approach requires QA to test the production version whenever there is a new change?
Wow that was some interesting insights. I have always seen master-branch development as lazy, sloppy and dangerous. I usually do it like this: i have a feature branch, but rebase onto master every day. When the Feature is finished we merge into master. Its safe and due to daily rebasing there are usually no big merge conflicts. The way of continuously working on master all together as a team, as suggested here does have some undeniable advantages. I will discuss this with my team.
Personally, when the scenario you describes remains as you describe, it is generally good enough. It's when discipline wanes and the features that aren't yet done start sticking around in multiple branches that live for 3, 4, 5 days, and then weeks, and then months.... that's where it really goes terribly wrong.
@@michaelrstover i have discussed this with my team (I'm the youngest member of that team) and they told me that corporate has denied them master branch development. But we do not work on branches by feature, we have branches by story/task. Which means the branches mostly only life a few days before being merged into master. As I have seen what some people dare to push (untested code full of dublication that doesn't even compile) I'd say we sadly aren't ready to get rid of our branches.
@@jimihenrik11 Branch-by-Jira is also what we do, but it does still lead to quite a few branches becoming quite long lived, and the delays between creating the PR and it getting reviewed and merged are super frustrating to me, and create a ton of inefficient context-switching. But, alas, I also failed to convince my management of the advantage of trunk-based development (and I'm the oldest member of the team!).
I think he is totally wrong. A branch is only a pointer to a commit, nothing more. Feature branches is a developing method for mainly creating a clean commit history. If you have conflicts in your team because too many developers are on the same code it's better to try to find a different WoW e.g. swarm on that task or don't work on the same code at the same time. So don't stop using feature branches it's the only thing that keeps the commit history clean which is really important thing for finding and understating how bugs were created
@@ContinuousDelivery That is dependent on what type of software is produced and how it is structured. As long as the features are truly independent it is not problem to let the branches live a bit longer.
@@karsh001 Isn't branching used to prevent getting in each other's way, where one might change a code breaking someone else's code? What is branching used for in your case, where things are completely independent? Just curious.
@@nicat6153 Its a good reason to use branching, but not the only one. Controlled delivery to production and simplified dev testing is the reason I use it in my projects. Due to the way the problem space work it simplifies some stake holder management and approval processes and we can quite easily drop features if they are rejected somewhere along the line. Dropping features mid implementation is very common, and the ability to just drop the branch becomes a value in itself. Once implemented, the features live a long time, usually many years with none or very small changes. Its all dependent on how the solution is structured and I wouldn't recommend long branches for a more standard SW project where you risk breaking each others code. For my projects, though it works great.
I have to agree. We manage a legacy system that is very broad in scope. This means Multiple features can be worked on at the same time without overlap, so feature branching works well, and turns out to be the simplest system to manage in this case. Having a sql server database at its core combined with being a legacy system full of bad design means feature flags tend to be difficult to implement confidently. And we’re rarely working on new elements that would allow dark features. We also really battled against a dev/master branch method, there was just so much drift between the two, after a month or so we had no confidence that dev was anything like what was in master. So I’m totally on board with the frequent merging of continuous integration, it’s just turned out to be a easier method to use feature branches for us, as conflicts are rare
Here I have multiple feature branches that stay open for MONTHS, and some of them are even from the same developer (myself). The code in testing is a long living branch containing a huge merge of all the currently open features. A few days before delivering I merge and close all the branches and curse in horror, some of them remain not merged, because they are not planned for release. Our customer-side managers are uneducated in everything software-related and Devops team is there just to add bureocratic weight to my job. Also the other company working on our system systematically forgets to close their branches, but that's another story.
I feel like once a competent person goes into devops or management, they start trying to make the development process more efficient. The problem is that software development is a creative labor. is complex and messy at times. There's a lot involved and there's no way to implement source code control without affecting the development process. Eventually, there will be so many constraints around developers that it will become impossible for them to do their job. I understand where all of this management/organizational techniques come from. They don't want to depend on a few talented individuals, so they want to hire a bunch of code monkeys and enhance their abilities. Basically, streamline software development. But the best you can do is to try to get the really good developers and hope that they are smart and have good communication skills. If you hire of bad developers, no amount of organizational techniques is going to solve your problem.
I’m a contract tech leader and came back to this video after recently joining a new team. I think it’s probably my favourite video of all of the excellent videos on your channel Dave. The kids game analogy is both perfect and hilarious and hence just the tool you need to trigger some self reflection in a team where some members are struggling to get the continuous delivery mindset. I must say thank you for this one. I’ve tried to pay it back as best I can and bought the library of books, many of which are part of my most quoted along with Brooks, Martin, Beck et al. Nice.
This assumes all my changes are always incremental. In practice I often tear down most of what I did after I've cemented the general idea. To introduce my initial ramblings to the team before I even know if it's the direction I want to go, sounds like a whole lot of work for nothing.
Great point, but you can still do CI here. If you plan on throwing away the branch, that doesn't count and is totally fine to take your time on it. Just be aware things could change during that time. Create your "draft" branch, do your thing, then when you are satisfied and ready to write it for real, check out a new fresh branch and do your changes there. Then commit and submit PR. Done.
I've worked for a large firm (close to 100 developers) and we were divided into smaller project teams (10 to 20 people) each responsible for a particular feature set and launch window. "Assumes all changes are incremental" is the problem here. Large changes are not useful to the rest of the team until they are completed, or at least nearly complete (ready for system testing). Working on a branch gives the project team time get their stuff into a usable state, then it gets merged with top of tree, but not before.
@@MidnighterClub You can break work into smaller pieces. It doesn't have to be useful to merge, it just can't cause regressions.
Work on your local branch and don't push to origin until it's ready. When complete, do interactive rebase, fixing up all the irrelevant commits into one (or as many as logical), then PR that set of logical commits.
@@JD-mz1rl That only works if much hasn't changed, or if your changes aren't massive. If this is working for you, then either your team is probably already breaking things down small enough (good), your application/team hasn't really scaled yet, or you aren't making many big changes.
Branching is valuable. Many times I use a branch for experimentl changes/refactorings that I’m not yet sure should ever be merged-at least until they prove themselves. If it doesn’t work out, I just delete the branch. Other times I use a branch to make significant refactorings I don’t want to disturb the main code, which is still moving. Its not always possible to think of work as a stream of small incremental changes.
How do you do code review before a commit to the main branch if you are not branching and submitting merge requests ?
Having a dev branch from which you can create one single, big PR to be merged into master? Not sure tbh
I guess with the more frequent merges team members are more aware of the work of each other. And smaller, daily merges mean quicker reviews. Merge requests are not the only way to review code, anyone at any stage of the development can just read the diffs. Maybe reviewing is also better if continuous.
what do u mean, you don't need to commit changes to create a code review you can use something like Code Collaborator to make a code review based on your local changes
Code review does seem rather more complicated in the CI context.
Gerrit accomplishes exactly what you are asking. You push your commit to the main branch but Gerrit does not push it to remote origin until it has been reviewed.
I was about to write a comment on how this was clearly bullshit. I watched the video instead. By the end I got completely what you were saying, and it makes perfect sense. CI isn’t something you can dabble in, and this demonstrates the level of commitment required to do it right. But if the data says it’s the most efficient way of working, then it’s worth seriously considering. Thanks!
We changed our process after this video few months ago and that instantly solved lot of code conflicts problems that we were facing and allowed us to visualize the progress of the day. Thanks a lot for the content
Thanks for the feedback, I am pleased that you found this helpful.
I feel like this would have been better received if it had been about branch lifespan/scope rather than “don’t branch” since branching is the typical means of code review. It seems to me branch/review/merge often is compatible with the CI definition and provides the same benefits you describe. Thanks for the video!
If you have a team that enjoys reviewing code, that'll probably work! Also, if reviews are done in pairs, even better - then you will get higher bandwidth of communication than the async nature of typical "chat based" reviews.
If however either of those are not true (code reviews take 1-3 days to complete or reviews are done through comments on a pull request), you are missing out on something here
Can you get your devs to do code reviews DAILY though ?
@@lvleminckxthat's why pairing or mobbing is my preferred way of working. Code reviews just create lagging or queues at a system level = low flow
My preferred way of dealing with staying updated with feature branches is frequent rebasing, this also allows me to have a linear history instead of a git history looking like a weaved basket.
Gerrit is a very interesting Git host, it automates a lot of fiddling with branches to the point development can go on for months without even thinking of creating a feature branch. Because in Gerrit you work with patchsets (internally they are just branches where only the head is used, so it acts like a single commit, with all its history still available) and just "replace" commits, and you can do this as long as you like, rebasing it, running CI against it, until you merge it back to the parent branch.
rebasing is cheating
Im not even a programmer but, I'm loving this channel. Found it through Cyberpunk criticism. Very fascinating!
cool, don't stop learning
Me too...
Me three! :D
Glad you enjoy it!
Sounds like your destined to become a programmer...
At what point do you perform code review if you commit to trunk every 15 minutes?
All the time, with pair programming is my preference. I talk about that in this video: ua-cam.com/video/aItVJprLYkg/v-deo.html
@@ContinuousDelivery But with pair programming you end up with two authors of the code so now you need a third person to review it.
@@JulianSildenLanglo Actually my prefered way of working is pair-programming with daily pair-rotation, so there are more than 2 authors and for big stories nearly everyone on the team will have worked on it. At which point an extra review is redundant. I work a lot in regulated industries, there is no regulatory framework that I know of, anywhere in the world that I have worked, that won't accept pair-programming as a code-review.
@@ContinuousDelivery Have you always lucky enough to have an even number of developers in a team, or have you had cases where one dev a day had to do some lone working?
(I'm coming from an even worse situation, where I'm in a team as a single software dev - nominally there are two of us, but the other person has an astoundingly large project management workload, and hasn't committed code in months)
Best programming content on UA-cam, can’t get this level of experience and amount of useful info anywhere else
I just want to express my gratitude for your content. It’s really professional stuff. As a software developer myself, yours is one of the very few channels I can actually still learn from. Bravo!
Thanks, I am pleased that you find my stuff interesting.
@@mmick66 I completely agree!!!
So if Bob is working on the same file as me, then I'm supposed to pull Bob's changes every hour, understand why he broke my feature then rearrange my feature and break Bob's feature the next hour? Sounds like no fun. Never had a problem with merge conflicts and much prefer to have complete ideas to compare and merge.
Yes, you and Bob should merge your code at least once, but ideally many, times per day to qualify as practicing Continuous Integration.
Sure, you may like the way that you work, but the data says that you are more likely to be producing lower-quality software more slowly as a result.
That is not me saying that because that is just my guess or preference, that is based on an analysis of 35,000 software projects.
My video is an attempt to explain why that fact is the case.
@@ContinuousDelivery Did you provide a link to that data?
The data doesn't say that at all.
@@ContinuousDelivery I’m sorry, you have not analysed 35,000 software projects and come to that conclusion - that is utter utter rubbish.
@@BigCarso Ridiculous response for someone whose beliefs got challenged. Nobody requires you to adapt your work style to new information. If you like what you were doing or are too ignorant of realizing the problems you have, that's fine, just keep doing it, no reason to change. However, many companies are struggling with their systems, and in particular the problem you mentioned is one of the main causes: If you two break each others feature all the time, then something is already going seriously wrong, and in such a case, a slower merge (like a 300+ conflicts merge) would be a complete disaster both in terms of time investment as well as in terms of keeping the error rate low. It also seems like you haven't watched the video, because it's pretty damn obvious that a change with fewer changes takes less time to merge than one with more changes. Really not that difficult to understand.
You are the programming mentor i was always looking for but never found. Well now you are here. Thank you for your videos.
Thank you, welcome to the channel.
Don’t listen to them, they don’t know what they’re talking about and what he is proposing would just get you laughed at by developers because it’s a stupid approach where he assumes that nobody can communicate.
He was late to version control and is confused by it, teams can manage branches and reduce merge conflicts down to 0, especially with microservices (which he is against for stupid reasons)
@@skylark.kraken I fell the same, seems he never worked in a big team, it is so horrible in a real world, he talk about downside of feature branch against continuous delivery but miss to mention the pros
The definition of continuous integration sounds like 10 developers coding live on the same computer but with different keyboards.
It's like working together on a Google Doc
This is much closer to what I was looking for....while youtube is flooded with basic to advanced GIT tutorials but nobody talks about professional implementation and use of CI/CD with lot of confusion on branching.
As someone who has experienced both sides, I can say that there's no perfect solution, it's basically a "pick your poison" situation.
So true.
Especially true as one thing that's missing from this approach is code review. If every developer on a team integrates every day that's 1 code review for 2 or 3 other people every day. Granted they're small changes but reviewers have to understand the big picture behind changes, otherwise all they're doing is reviewing syntax variation and coding style. A big part of review is does the change fit the spec and the desired end result of the bigger change. Granted review against bad coding practices is important, but it's not the only thing.
I am pretty sure most of us have experienced both sides. There are planty of things that can go wrong in either side, one is less efficient but has the upside of hiding merge conflicts until the end and gatekeeping the codebase from bad developers.
Can't wait for all the supporters in here to get in the field and deal with the struggles this methodology lends itself to.
Hopefully they won't become cultists over this and one day will be willing to give branching a chance.
Maybe you should try it? I have tried both and know which I prefer.
@@ContinuousDelivery I highly value your ideas and videos, really. But this no-branch attitude lends itself only to really senior developers.
Many other people have mentioned code reviews. The code needs to be reviewed. Why? Because programmers are sometimes too creative with their design, especially the mid-levels who see themselves as seniors.
I do agree with a few ideas and it's how I do things:
- all branches have to be releasable at any time; developers merge the master branch into their branch every 15 minutes / once a day. This is continuous integration
- once the code has been reviewed, it can be merged into master. This can happen once a day, or once every few days
- the developer switches back to his branch and continues to work and the cycle restarts
This way:
- we protect juniors from their own mistakes
- we protect the design
- we can do pair programming, but doing it all the time is just too much; trust me, I know my juniors
- we have design meetings; problem: the junior of the team does not really see the same imaginary code as me when we do the design
- and yes, I explain a lot, we do enough pairing, but at the end of the day time efficiency also matters. And releasing often bad design is not time efficient
I'm a CTO at a smaller company btw.
I think , this is good for small teams that consist of experts and seniors , but when it comes to large projects with many people and the skills vary, then we need a strategy "called PR" to review juniors code as example. Thank for your video . I enjoy it
Well it scales very well, and I would argue that hiding what people do from one another (branching) may be a defensive strategy for less experienced teams, but is not going to help them improve. My own preference is to combine this with Pair Programming, that way you can mitigate the risk of inexperienced people making a mistake by pairing them with someone more experienced, and they also get the chance to learn more quickly. I describe my thinking on this in this video ua-cam.com/video/aItVJprLYkg/v-deo.html
@@ContinuousDelivery it does not scale well at all, you shouldn’t be teaching something that will lead to more problems. There are no benefits to not branching that can’t be solved with some planning and communication, just because you can’t doesn’t mean that everyone else shouldn’t.
People should learn to use branches because that works best, and merge conflicts only mean there was a hole in communication.
@@skylark.kraken Are you speaking from experience of having tried proper CI, or are you just pessimistic because you have no confidence or trust in your team or having been disappointed too many times?
@@skylark.kraken Notwithstanding that every working copy is a branch, every commit a merge... less experienced developers will more often mess up and take longer to complete complex merges...
There are secondary benefits to the frequent integration approach - protection against device failure and interruptions, for example.
I think the point here is less in the branching itself and more in totality of the approach; small changes, backed by automated tests... Smaller changes means less to go wrong / less to fix if there's a problem...
Yes, this only works for small teams, small projects, or projects that are not mission critical. People who believe in this will be in for a big surprise when they joined a big company or a big project. Just wait and see.
Thank you for sharing this! When the video started I thought, "this is a terrible idea", but the methods you presented for safely dealing with partial features make a lot of sense. I think branches still seem to make sense for open source projects because of the distributed decision making process in the community. But inside a development team I can definitely see this being a lot smoother.
I worked with both branches and incremental changes on master. They are both valid. If there are more than 15 devs in you team it will be very difficult not to spawn new branches.
Another benefit to CI compared to long-lived feature branches - if there is a merge conflict, it is with the code that the developer wrote *in the last 15 minutes!* This code is fresh in the developer's mind, and they will have an easier time fixing the conflicts rather if the conflicted code was written days or weeks ago.
or just merge from the main branch into feature branch every day or even after every other person merges into the main branch. It still keeps your code close to the truth and avoids stepping on other people's toes
well it's merge up from MAIN and run the tests but the problem is when you merge your work to the MAIN, Others should get the changes and run tests on their work and if they are close to release they could get to the state that their long work is no longer relevant and they would have no more time to fix it so they cannot integrate their work to MAIN and the reason is lack of informations about what do the rest of the team doing
@@75yado uh that shouldn't happen that sounds like dependency hell
@@AloeusCapitalManagem no just too many people working on same system. Usual problem was when someone changed something in the geometric kernel or similar module which everyone used
that does not work
That only solves half the problem. You need others to take your changes as well, that only works if you push to Main.
I've worked with a bloke that had the opportunity to work with you back a decade ago. When I'm watching to your videos I can totally imagine he explaining his values, approaches and techniques.
Don't mind the people that didn't get what you're trying to explain them. They will, certainly, learn this one way or another. As more videos comes, I'm sure you'll tone down a bit and reach to wider audience. We need this wisdom.
Keep pushing mate. Awesome content.
Thanks, who are you working with that I know?
Interesting topic. I believe that this is the way forward; however, one concern I have is related to code reviews. Im not worried about syntax relates issues; this can be automated using static code analysis. I'm mainly concerned about the architecture/implementation of the feature as a whole.
When you make all your feature changes within a single branch, you will have a holistic view of that feature's design and implementation (something which can also be used as a reference). This is something which you will lose when committing directly to master.
Just wondering how you have solved this problem. Maybe it's as simple as tagging the commits with a feature id, but review tooling just isn't ready for this scenario.
Yes, I think that needs more care. The best way to achieve that design overview is pair-programming linked with regular pair rotation.
I describe that, and some of the reasons why I value it so highly, in this video: ua-cam.com/video/aItVJprLYkg/v-deo.html
I was going to ask that exact same question.
@@ContinuousDelivery Nice video :-) I have to say I usually have different opinions, but I really appreciate your work and ideas. They make me think :-) Pair programming is usually advised as a solution for absenting code review. However, I would be interested in your opinions about pair programming in terms of distributed team - not only geographically, but also across multiple timezones. There might very small overlap suitable for pairing.
Thanks again for all your work, I really enjoy thoughts it sparks in my mind.
@@milavotradovec1746 Thank you. On remote pair programming, I once worked for a financial trading company, based in Chicago, I worked out of the London office, but the team that I worked closest with was based in Chicago, a 6 hour time difference.
We paired during the couple of hours of timezone overlap and found it helped strengthen the bonds in the team. It is different to full-time, local, pairing though I think. I think that it helped the team a lot, but there was also a lot of work that was done not-pairing.
The way it worked best for us, was I had some stuff that I was working on and the people on the team in Chicago had some stuff that they were working on. I was the more experienced dev, so that it nearly always ended up that I paired on their stuff.
I'd give it a go, and experiment a bit with what kind of things you choose to pair on to find out what works best.
@@ContinuousDelivery Thank you for your reply and sharing your experience. One more question - when you worked on your stuff alone, were you still committing directly to the trunk (main, base, whatever :-))? If so, how were your colleagues in Chicago aware about the changes you did? Thank you very much for sharing this, I find it very inspiring.
Hi. Big fan Continous Delivery (the book) and Accelerate (the other book) here.
I've been implementing build processes for more than 10 years for a big variety of projects and I'm a fanatic on software quality.
I agree with all the branch by abstraction, feature toggles and dark launch.
I have a fundamental disagreement with forcing to work with a single branch.
Let me explain my point of view:
I want to have the highest software quality. To do so, I automate all the possible quality checks in an automated quality gate, typically:
- stupidity checks (duplicate DB migrations, leftover console.log etc ...)
- all the necessary linters and static code analysis to spot code smells
- unit tests with high code coverage metrics (for code with cyclomatic complexity)
- architecture rule enforcement tests (using ArchUnit)
- mutation testing
- software dependency scanning for vulnerabilities
- end-to-end testing for most of the features of the application
- UI regression testing with screenshot comparison
Well implemented, those checks take 40 minutes wall clock time on a web project after 10 men-years of development (using a standard Spring + Angular stack).
I want to deploy to production at least multiple times a day.
At any time, tip of the main branch should be production ready.
No-one should ever hold his breath before a change getting deployed to production.
I've been doing that successfully, with high throughput, over multiple years.
How: using short lived branches that have the full quality gate running before being merged.
The development cycle is the following:
- pick a task to work on
- start a branch
- as soon as you have some change whose quality is high enough, push, make a self-code review, and get it merged (after pair-review if needed) as soon as the quality checks are ok.
- You don't need the task to be complete to merge, you just need the quality to be right, to merge.
- You'll typically make quite a few merge requests before completing the task
Typical cycle (between branching and merging is less than half a day).
If someone wants to keep a branch longer than that, no big deal, he merges the main branch into his development branch as often as necessary (I recommend at least once an hour). He'll have the burden of merging, as he is the one choosing to move away from the "short-lived branch" recommandation. His choice.
The higher the threshold of the quality gate, the more frequent the checks are going to be red. If it's red on the main branch, we cannot deploy, so we test before merge.
You understand that with such a test battery, considering pipeline 10% of the pushes leading to build failure means the main branch will be red more often than not. This means team blocked hunting for the change failiing the build.
Testing on the branches allows as well to:
- test the build process update before merging (new compiler chain, different code coverage tool, etc ...)
- review app: review UI changes live before they get merged by deploying the branch to a short-lived live environment
- code reviews for knowledge sharing (I've read your arguments for pair-programming rather than code review. I'm a big fan of pair-programming, but it's just not always possible, sometimes simply because the work schedule is not uniform among the team members).
Conceptually, a developer is always branching. He's working on a local copy that IS kind of a branch.
Let's just consider the short-lived branch the equivalent of his local copy.
If one is force to push on trunk, he'd better run at least a significant part of the quality checks before pushing.
Running well written unit tests for a big project can easily take 5 to 10 minutes (we have many thousand of unit tests). I should run that locally before each push ?
Why doing it manually, when tools like Jenkins, Gitlab-CI and many others can do that for you ?
Hardware is much cheaper that developers, auto-scalable build farms are just a click away.
Short lived branch allow this process to be asynchronous, and let the developer do something more interesting than waiting for the build to be ok on his laptop before pushing.
I've now moved on a project that advertise doing trunk-based development (based on your very arguments).
Tools have no build-on-branch support (GOCD).
Quality gate only contains unit tests and some end-to-end tests.
There are just a few deployment to production a month because confidence on the quality is not high enough.
There are more days where build gets red at least once than not.
I have a strong feeling trunk-based development gets in the way of having a strong quality gate. You can't have both, and without the later, you cannot safely do continuous deployment for mission critical apps.
I suggest the following motto: "Long live short-lived branches" (no copyright).
I'd be glad to read your feelings about this.
I think that we are 90+% aligned here.
A few thoughts...
If your "typical cycle (between branching and merging is less than half a day)" then you are doing CI, so I have no argument with that at all.
I think that you are doing more typing than me, I work on master locally and then merge all my local commits immediately that I have made them to origin/master. So I don't have to create any branches or merge to them (less typing for same effect that you describe). To be honest, this doesn't matter at all, if you want to type a bit more it doesn't matter.
"The higher the threshold of the quality gate, the more frequent the checks are going to be red" Not so! The slower the feedback loop the more often the checks will be red! Adding branches slows the feedback loop. The problem, as you correctly point out, is the efficiency of the feedback loop.
One of the more subtle effects of feature-branching, in my experience, is that it gives teams more room to slip into bad habits. Inefficient tests being one of the most important ones. If everyone is working on Trunk, then slow tests are a pain, so the team keeps them fast.
I think that most of the rest of what you are describing is about feedback efficiency. Let's try a thought-experiment... If you could get the answer (is it releasable) in 1 minute, would you bother branching?
If not, then what we are talking about then is where is the threshold where the efficiency gains of hiding change (branching) outweigh the efficiency gains of exposing change (CI). I think that CI wins hands-down as long as you can get a definitive answer within a working day. Practically, the shorter the feedback cycle the better, but my experience is that under 1 hour is the sweet-spot. That gives you lots of chances to correct any mistake during the same working day. I built one of the world's highest performance financial exchanges and the Point of Sale System for one of the UK's biggest retailers, and we could evaluate the whole system in under 1 hour.
So my preference is to spend time on optimising builds and tests, rather than branching.
@@ContinuousDelivery Thanks for the feedback!
I like the principle of continuous visibility of changes. However, regardless of feature branch vs CI, there are few people who will 1) actually look at the commit logs of changes pulled, or 2) see the changes made by others when working on their own ticket. The only thing that wakes people up is merge conflicts. Must say, among the merge conflicts I face using feature branching, very few are due to mismatches based on a fundamental approach; it is minor things like imports/formatting.
I like feature branching because a reviewer can see things like architecture of a feature within a PR. I fear that CI will make people approve minor changes with less critical thinking about the big picture.
I have the same thoughts about code review on larger features. It seems like it would be impossible to do proper code review. I would love to hear Dave's thoughts on this.
The other thing I don't like about committing incomplete features to the master branch is that it makes it difficult to identify zombie code. If you see code that doesn't appear to actually be used, is it zombie code or is it a necessary part of an incomplete feature? You'd have to actually ask the developer who wrote it.
What happens to features that were abandoned half-way through? Seems like clean up for that would be a PITA as we can't simply revert the merge commit for a feature branch because the feature was committed via countless tiny commits.
this 100%
@continuousdelivery I would also greatly appreciate your thoughts on the above situation. A FB PR gives a more holistic view on what the problem being solved is and the architecture decisions made to deliver it. The reviewer can more easily see if there is an issue.
However a senior reviewer would likely notice an architecturally poor choice very early on, thus saving the junior dev from continuing down the wrong path.
I have your books and all the Devops , Martin Fowler series. You ability to speak clearly and know all the pitfalls is A+. Steve McConnell is also someone I deeply respect. Maybe you can comment on his work in a video or have a joint discussion. Thx for all your hard work.
If your feature requires changes to shared libraries and database schema, how would you handle potential breaking changes to other areas of the code if you don't isolate them by branching?
You can always integrate your work by syncing/merging the trunk with your feature branch. By keeping your work current with what happens in the trunk you ensure that other features that are introduced work with your changes all the way until merged back into trunk.
Yes, but that only solves the problem as far as what has been committed to trunk, if everyone else is Feature Branching too, then you have no sight of what changes they are making until they merge to Trunk. This is certainly better than not merging frequently from Trunk, but it is not the same thing as CI.
@@ContinuousDelivery that is true, but the only thing I need to consider is what the trunk has, which is "the truth" that is in production. Same for others working in their feature branches. This is however when people work on things without knowing anything what others are doing, which should be rather uncommon unless there is no communication in the team - the occurrences where different team members work on the same bit of code without each other knowing should be rather minimal so "collisions" should not happen unless the code is badly structured as well.
By continuously testing my code with the latest changes introduced in trunk I do get CI, I don't see how it isn't. This also helps when my changes are about to be merged into trunk with a pull request, so that other team members get to see if any changes are in fact in line with the current code base and follow the guidelines and principles/best practices the team has decided.
@@strangnet The problem is that if you have a team of people all working like that then except for the last person to commit, trunk isn't really trunk in the sense of the version of code that will go into production. So everyone is testing against a version of the code that wikll never end up in production. In true CI - the definittion of which includes the statement "merge changes into trunk at least once per day" this is not true, at the point of every merge, trunk is definitive.
All this is just our opinions though, the data says that if you have feature branches that last for longer than a day, wether you merge from trunk or not, statistically you produce lower quality.
Some features don't pan out or take longer to complete or require several user stories to be done before the feature could be considered complete. If that code is integrated immediately then should it not pan out you have to "unintegrate" it. Sure you can use feature flags but if you have to pull the feature you have have to remove / undo that code. Forward integrating feature branches on a daily basis is the cleanest solution imho.
@@Wouldntyouliketoknow2 a branch should not live longer than a what a sprint is. It can start in the middle of the sprint, and be finished in the middle of the next sprint.
Agreed that they should be very small integrations, integrating at least once a day, but committing and pushing to master allows junior developers to add flaws to master without code review. Merge commits from work branches to main allows associated work (first attempt, write tests, second attempt, add comments) to be grouped as part of a single commit. The commit graph is much more clear about when and why certain changes were made, and a rebase to remove problems can be MUCH simpler.
Or do pair programming, even simpler, and the junior devs get to learn faster.
WTF. why this channel doesn't have 1 million subscribers yet?
The premise is that my small daily commits doesn't break the app even it is not in production, so my colleagues can work on their tasks. But some tasks with a lot of refactorings take days to finish and it could make harder for others to work. Also you would get rid off pull requests that should be reviewed. But don't get me wrong I hate long term branches and dealing with conflicts so we try to minimize size of branch to minimum. Also feature flags are good approach we implement very often. What I prefer is to pull from development branch often so I keep up with current state.
I would argue that refactorings are the best use case for trunk-based development. If you are refactoring a large part of the code, you’d better ask other people to stop working on that code or you’ll end up with a disastrous merge hell. If you can organise a refactoring as small batches of work to be integrated frequently, definitely do it.
Yes, I agree with that approach. TBD is a MUCH better way to support refactoring.
@@andrealaforgia so you arguing that if you doing a refactoring the 5 other people in your team need to have a pause 🤷🏻♀️ Well I agree that while doing some heavy refactoring, generally, nobody else should work on the same code part. But that doesn’t mean you need to do it in the trunk. Refactoring can sometimes not work well and it’s much easier to verify and to revert changes which were done in a separate (feature) branch.
I really enjoy your videos and content.
While I don't feel I'm radically enlightened I see these as a great opportunity to share them with less experienced engineers, as the clarity of your arguments is amazing.
Thanks
Thank you!
9:30 (DevOps Handbook)
I am not used to this workflow but I can see it’s merit.
In my experience, the biggest bottleneck with feature-branch workflows tends to be code reviews (not branch maintenance or testing feature branches). However, code reviews are also the most important step for ensuring code quality and readability. Google/Facebook do NOT bypass this.
I think the benefits the DevOps report outlines might be true with trunk-based:
- higher throughput
- better stability
- higher job satisfaction
- lower burnout
However I don’t think it’s particularly comforting knowing people can commit anything at anytime without any oversight. I’ve seen codebases use this sort of style with just a linter and the commit history was a really long mess (and the code was a mess too). Really was a nightmare having to work off of that.
feature-branch workflow compartmentalizes work much better and has a controlled process for integrating. Trunk-based might get you to the end faster but throughput isn’t everything.
For example, a project that’s “released” but is being maintained with occasional bug fixes or other minor changes would not benefit at all from continuous integration. I guess it could work on completely green-field projects but most jobs are working on existing money-making products.
Just to re-iterate, even Google/Facebook who supposedly use this trunk-based workflow STILL have code reviews. Ultimately the biggest bottleneck is waiting on others.
You can do code reviews without needing to branch and can be done on each commit.
Thanks for the great videos, but there is a basic flaw with your logic here. That is that the feature branches are somehow completely broken apart from the main branch or trunk. There is nothing wrong with having a long lived feature branch as long as you keep merging the trunk back to it. Your graphic is not quite complete, it only shows the merge of a feature to the trunk. But there are many trunk to feature merges along the way while the feature is being built. This can be done fairly easily and this way your feature is always updated with latest code. The feature itself can be tested completely through CI and if tests pass then you are good to merge your feature to trunk. Adding feature flags, dark features, etc involve huge amounts of extra code/tests which are not necessary. Most of the time this will probably add performance issues and extra unneeded code. In some instances this could be impossible if the new feature demands a DB migration or something similar. Having feature branches or even long lived features branches doesn't mean you are not continuously integrating.
Well, syncing with trunk for a long-lived FB is a good idea, but it doesn't solve the problem of what is going on on everyone else's "long-lived FBs".
I am not saying that an FB is "completely broken" I am saying that it is not the truth and you can't tell wether it is completely broken or not.
You can reduce the chances, but you can't tell until you merge into the version of changes that represents the "truth".
Edwin, I think you are correct in the observation that dark launches, branch by abstraction, and feature flags sometimes require extra code. But I think the extra code is likely to only lead to a higher quality code base and/or better user experience.
With extra code, I mean facilitating code by the way, I think you meant the same? Because there will of course be multiple "feature variants" in the code base, but that is not different with FB, just in branches instead of files. For dark launches, actually no facilitating code is strictly required. For branch by abstraction, the facilitating code (when done right) are mostly interface definitions and IOC mechanisms. Chances are the project will have these anyway, or otherwise are a good addition for other reasons.
Lastly, feature flags are a bit more complicated to evaluate. First, they require facilitating code that many projects do not have and would only be a low priority for other reasons. Second, naive implementations impose a configuration burden on the ops team (aka yourself if devops). In my view, feature flags should therefore not be configured, but be exposed as a user preference setting (this can be an internal user of the code, or the end-user). If this is not desired by the user advocate (ux designer, productowner, whoever), then I can probably be convinced to use one of the other strategies entirely. But if the feature flag is desired by the users, then the facilitating code actually improves the user experience, and is not a burden at all.
For these reasons I completely agree with Dave that feature flags are more "risky" than the other mentioned strategies. That said, I have in the past dark launched a feature-toggling interface to great effect, and I can't imagine any user advocate not wanting this for themselves (even if they want to keep it hidden from their users indefinitely).
When done correctly, the facilitating code should not introduce performance issues, or this indicates other problems. With DB migrations, that should be done by a branch by abstraction on the DB schema itself anyway :-)
pain with long lived branches are not in master, it's in branch. So you can update your branch as many times as you want, but others will not see your changes. And you get merge conflict not because of you pulled changes a week ago, but when you try to merge 2 files that edited same lines. And when you have long lived branch chances that someone else edited that is higher and these people introduced changes in their own branches and then when you want to merge you get a mess, because you have one logic there, another logic here and who is right? If you have highly decoupled work, where you don't work with same code as others, then long lived branches will not create conflicts. But they (long lived branches) will increase business value delivery, regardless.
just merge the current "develop"-branch (source of truth) into your feature-/bug-branch before starting to work on it again, at least every day. It's that simple. Nothing else ever worked with > 10 Devs on one repo. Channels like this show, how far away the theory could be from anything practical :)
Create branches and rebase often. Happy life :)
Fail othen and fast
@@kajacx It means you were the only one doing the rebase, if all the team does rebase often you will rarely get in that state, I've been using git rebase for several years and very rarely hit points where it was a hell. It is hell if you are the only one doing a rebase.
This approach kinda assumes that no one makes mistakes. Imagine someone is committing every 15 minutes. After 2 hours someone realizes that there is a huge issue with the new code which affects everyone's ability to work on their own tasks. Writing tests won't always help. Things can slip through. Also writing tests that early is often times a waste of time because you don't even know what the final version will look like. Thanks for your video tho!
Tests should be written on the feature-level, the implementation details don't matter. This is not always possible, but I'd say in 99% of cases it is.
Not that I'm convinced that branches should be avoided, but I do think that frequent merging would be useful (again, in 99% of cases), but needs to be done right.
@@defeqel6537 Well, there are different kind of tests. Every test kind has its place. I don't see how this is referring to my comment btw. Frequent merging is good, indeed! But you want to merge a fully refined increment not a WIP state to reduce unnecessary friction.
@@Mio2k10 I agree, and disagree. Generally, yeah, fully refined increments should be preferred, but as long as it is flagged out, or similar, I'm starting to see a benefit with WIP state changes being merged too. I do think merging WIP requires care though, otherwise may lead to just frustrating your team members.
As for testing, my point was that the tests (usually) shouldn't rely on what the code looks like, so they may be written quite early on, after a bit of planning.
As for test types, I prefer unit tests, by which I mean code I control the inputs and outputs for. IMO people should avoid micro-testing, where they try to test each function or class separately (unless those are your deliverables, in which case thoroughly test them).
Thanks for this awesome video. I hear what you are saying, my question is how should you handle code reviews in this case? If I understood correctly, the desired process here is to push to master as often as possible instead of hiding the changes in a branch, but it also means the code needs to be reviewd by someone else(teammate) every 10 or 15 minutes which in reality is very disturbing(in terms of context change for the reviewer) and blocking. What's your point of view on that?
I would suggest using upstreaming for this. In such an approach the whole team is committing into a develop branch regularly - this becomes the trunk. Then separate "merge request" branches are created and relevant code is checked out into them using "git checkout develop -- " and then these are reviewed and merged into master. After that master is merged into develop to bring the two branches in sync.
@@mkschreder Then you're back to multiple branches that aren't always in sync.
I would suggest using short-lived branches that are merged within 24 hours. Then, you'd need to make sure your team is reviewing and approving PR's daily.
@@d1morto yeah but that doesn't always work to review daily. The idea here is to commit to develop and then move commits into review branch (or just move changes) I agree that pushing the team to review daily is important but some teams are very slow at this and can take a week to review something that they may not filly understand.
@@mkschreder The devs would sort through the develop branch looking for their individual commits, and then move just those commits into a review branch? Wouldn't that be tedious?
This channel deserves more subscriptions and more views in the videos
Thanks
I see a lot of discussion that feature flags solve the parallel development issue, and I believe they do, to an extent, but they also increase the complexity of the code and make it more difficult to reason about. How does flag x impact flag y? Ahhh I hear you say, well that’s all handled in testing and ci. That’s true of feature branches as well. Branches increase process and development infrastructure complexity. Feature flags increase process and code complexity. Both require coordination and communication between team members. I’ve used feature flag extensively for rolling out experiments in the UI/UX space and they work well. They’re isolated and have a finite lifecycle (the length of the experiment). Cyclomatic complexity has long been shown to a have a major impact on system maintainability. Feature flags by their very nature increase cylomatic complexity.
PS: Buying Daves book, reviewing considered opinions are always worth your time.
I think that you can implement Feature Flags in different ways, particularly if you use them in combination with one of the other strategies. Feature flags don't have to be in the guts of the code if the design is good. But I agree with the point. I am wary of Feature Flags for a related reason the explosion of potential code paths - what do you test?
First off I agree with a lot of what you are saying. Certain problems (on an otherwise amenable project) tend to require workshopping before release though, and I feel that in cases like that feature branches and a bit more integration work can be worth the headache in order to get things right.
Also, maybe I just don't know how, but I can't figure out a great way to do mandatory code review in a "commit anything that is tested as often as possible" codebase without basically mandating pair programming. That wouldn't necessarily be the worst thing ever. I like pair programming, but I also enjoy programming alone, and I think they are both beneficial. In the projects I have worked on we tended to stay somewhere in the middle and the communication in the daily standups generally kept merge conflicts to a minimum, even when some branches lasted for a week or two.
Yes, he didn't mention code reviews at all and I infer that they don't have mandatory code reviews.
Yes, and on GitHub for example code reviews are intended to be done via pull requests
@@paulprescod6150I am not so quick to jump to that conclusion, but I am sceptical of that aspect nonetheless.
The biggest thing is in my experience developers are more willing to make improvements to code that isn't in the mainline branch yet than something they already got merged. Analogous to how most developers won't write tests for code that "already works".
I really love this idea. But, we have developed an alternative way. Can I make a response video and than we talk about the trade offs?
Why ask for permission? Just do it.
Can't upvote this enough. The whole git workflow has made merging and testing a nightmare. I've had developers overwrite code many times due to peer merging as well. That this is what everyone does these days is just bad. Great video!
Although I'm fairly convinced that the way we develop our application (News Site Backend) is fairly optimal for what we do and how we do it, I watched your video with an open mind and tried to see the benefits of it. However, I couldn't find a way to make this work in our environment and being beneficial to stability, increased efficiency during development and deploying realeases. There was always a tradeoff at some point. So my argument is that concepts which were developed maybe one or two decades ago are still valid concepts but with different constraining parametres. Your argument was that a code should be always in a deployable state - which of course is always given for any branch which is deployed on dev, stage, release or however you may call it. On the other hand it doesn't make sense to commit unfinished code with unfinished unit-tests to a dark release or feature switch - even in the dev-stage - when it is simply not finished because it will inevitably break things. Just for the sake of commiting code every 15mins (you don't open a bridge to the public if its only halfway built). So my argument is, that it is probably a good idea to work in that way but adjust the constraints to the environment they apply to. Maybe commit the code not every 15mins but only every 1-2days when a feature is in a state where it actually could be integrated without breaking things. Another parameter (might) be the actual size of the codebase in relation the changed lines in every commit and the number of devs working on it simultaneously.
Either way...good content and nowadays it became equally important to actually know how to organize your code compared to know how to write it.
This guy is making things complicated!
Nope, things are already complicated, but not everyone notices that they are😏
I try to minimize branching, especially long lived branches.
I also think a good starting point is to have only few branches at any given point in time. To facilitate this; do pair programming or otherwise divide the work, to work in parallel.
Sounds too ideal to me. Never worked on a team that is even near to have what it takes to do true CI. Thanks for the vid.
I know nothing of programming, but this is just so relaxing to listen to and so interesting :)
Glad you enjoy it!
Wow! I'm happy to have found this channel. Great videos!
This may work in a "code-only" feature, but in my world, many features require data changes in persistent storage like databases. I'd rather hold on to a feature branch for an extended period of time and (more than) daily pull in changes from main. This way I'm the one solving my own merge conflicts and push to main would be easy.
Since we switched to a model.broadly similar to this, we almost never have merge issues any more.
We do not merge branches every 15 min, compared to merging once a day I do not see the added value. Developers usually do not work on the same piece of code on a single day, and having to synchronize your repo with the origin every 15 min seems like an overkill.
But as said before, with a more manageable timetable, it's indeed a very good practice.
But every video I see on the channel I can recognize myself in most if not all the advices given, good to see someone with some real experience take on youbute!
This is really great and makes things very understandable, thank you. This seems like an optimal strategy. But, isn't frequent merging to main going to generate a pile of merge/pull requests? Who typically reviews and approves these, and how do they manage this volume?
From my understanding, you'd commit directly (without making any pull requests). Ideally, the code you're commiting and the existing code base are covered by tests verifying all behaviours, so it gets a lot harder to introduce bugs without tripping the alarms.
Very insightful!
Love Dave's channel.
Thanks 😎
Wow, your video is amazing! I wish there was a way for me to give you a thousand likes for it. Keep up the fantastic work!
Very thought-provoking. Thank you.
Your topics are very informative, these are hot topics and our teams discuss them in the retro. I personally, enjoy reading the different perspectives in the comments.
How do you implement code reviews if you commit every 15 minutes?!
Compulsory review on commit.
Dave, I think the real discussion should be: code review vs continuous deployment.
It feels to me that this is where the friction comes from, not the branches themselves, which are used as a means for code review.
Maybe it's worth a video?
One of my favorite channels to watch, I enjoy the style of the videos :)
Thank you.
in a perfect world "Continuous Integration" is possible, but our world is anything but perfect. :-)
The nice thing about software, is that it is easier to change than the rest of the world, maybe not perfect, but certainly "good enough for CI" 😁
If you are not using pull requests or mobbing with everyone on the team then what is to stop unreviewed code is getting into the mainline in an uncontrolled way? I know XP is explicit in saying that (at least) pairing is necessary for this to work (partially). What if even pairing is not an option for whatever reason?
Branching shouldn't be responsible for making sure code is reviewed before it's deployed to production.
Use jira or whatever to track status of commits and only deploy the application to production when tasks have been code and feature reviewed.
CI is supposed to encourage collaboration as everyone is working on the same codebase.
I could see this being important in certain groups/situations, but not all. I think the official Git book says it best when they point out which methodology is used will depend on the group's needs.
Issue branches are critical to the way we work since we need to be able to thoroughly test the change, implement it when ready, and be easily able to roll it back if necessary. In general we don't see merge conflicts, and if we do then that's a good indication that the changes need even more testing to ensure a quality product. With all that said, for the most part we only update the training production) load itself once a year unless we have interim critical items that need to be addressed. Stability is more important that rapid deployment for us, which I get is the focus of this channel.
This approach is certainly not just for simple systems, and is increasingly used in safety critical systems, because the data says that this is safer. It's safer because it promotes working in much smaller steps, so each step is less risky. For example, SpaceX practice trunk based development and update their rockets up to 40 minutes before a launch.
This includes launches that carry people up to the International Space Station.
This also how Tesla work, Amazon, Google and Facebook. Also several banks around the world and increasingly medical device manufacturers.
Great video, as always Dave.
It's good to know that some of the strategies to avoid branching have to do with a great effort I've been putting into releasing a more intelligent and manageable Feature Flag platform. Thanks for sharing your thoughts.
Thanks
For the big teams out there that cant get around branching, Just make sure you cant merge a branch before its been rebased on master (git hub can help you here) then rebase your branch often and don’t keep long running branches. And make sure your software is covered by automatic tests. Ive worked on multiple big teams releasing new features multiple times a day using this rebasing strategy without having to forego the use of branches.
I think that is a work around. It is certainly the best way to work on branches. The problem with merging from trunk to your branch (which I agree is a good idea if you work on branches) is that it still doesn't represent reality, what about all the changes that all your team-mates have on their branches. This approach certainly reduces the chances of a merge conflict, but it doesn't reduce those chances as far as CI does.
@@ContinuousDelivery I see your point, and for most projects, it makes perfect sense to just work directly on master, so long as you keep it in a releasable state all the time. But I think at some point a team can get too big for that to be a viable way of working because the criteria for it being a success can be difficult to achieve (not impossible at all, but difficult).
@@henrikoldenborg86 Google do it with billions of lines of code, so I don't think that scale is a limiting factor. It is a different way of working though, and so that is a challenging switch of approach.
@@ContinuousDelivery thats actually very interesting, do you know of a resource were I can learn more about their specific approach?
@@henrikoldenborg86 There is some stuff, including more links to follow here paulhammant.com/2013/05/06/googles-scaled-trunk-based-development/
Having implemented ci and cd last century, I can tell you this is a gross oversimplification. Branches are necessary when the need to isolate changes justifies the overhead for having the branch. There are many reasons to branch ( yes, even by feature) and there are many reasons to not branch. The problem lies with knowing when to apply the method and when not to, and unfortunately many developers have such a low understanding of branchinh that they even conflate directories with branches. The factors involved with these decisions can involve everything from the sw architecture, the competance of tge team, the competance of the vc tools merge capabilities for tye files concerned, and yes, even the release management process. Over the decades I have seen so many people try to oversimplify this and have had to clean up the mess afterwards. At the very least your video should provide caveats like " if the merge tool for the code files is not reliably automatable for >95% of merges ( and yes there are tools that exceed this, and git ootb is not one of them) and your development is project based, not product based with product variants and/or your system is not a safety critical system and/or your dev teams are poorly trained on the effective use of branches etc etc etc. To give a real world example of the impact of branch avoidance, I jave seen an entire product line have to be redeveloped as each product variant was a copy pasted directory of the source code (12 product variants). Despite having the same people maintaining the codebases over a number of years, each copy of the code was so completely dissimilar that a core change across all of them worked out to be cheaper to redevelop the entire product line from scratch ( a $12m decision just taking into account the costs for re accreditation of the variants - let alone the cost of redevelopment and opportunity cost to the business.
Naturally, I don't think that it is a "gross over simplification". It is certainly a simplification, because it is a 15 minute video!
The approach that I have described is, and has been used, to build large complex systems all over the world - I have worked on several myself. Occasionally we found the need to branch, and it felt like a failure when we did.
I don't think that there is much about software development that is easy, but I also think that, as an industry, we are exceptionally good at over-complicating things. Branches being one of those times where we are prone to mis-using them.
Any branch is "running with scissors", it exposes you to risks with BIG consequences if you get it wrong (as you point out). Of course, this doesn't mean that every time you create a branch you will fail. It means that your chances of failure are increased.
CI has a different set of compromises and fails more gently IMO. My intent with this video was to point out the costs, and benefits, of both, but naturally my advice to any team is to "prefer CI to branching" and "always branch reluctantly and with care". (I probably should have said that in the video - sic!).
I agree with a lot of what you say when you describe naive approaches to branching, cloning different versions of products is a horribly common practice - and terrible solution.
@@ContinuousDelivery an excellent clarification. It is vitally important that branching strategies are applied by skilled people who know how to balance the various factors within the constraints provided. Unfortunately, there simply aren aren't enough people who do. CI and CD are good methods when applied appropriately, however, I continue to be concerned about how CI and CD are being pushed as being the default or only way. A recent example of the consequences is actually the recent solarwinds hack. It looks like they had a fully automated CD system with a view to releasing quickly, which on the surface seems like a good idea. That is, until hackers put a backdoor in the IAM modules and let tge CD systems punch it out to prod, and then to 16k of the worlds most crucial orgsnisations. Having a fast means of punching through to production is a good thing to have in general, however for some systems, and for some areas of systems, a more rigorous control process is necessary. The approach of fail fast, fail early, fix quickly is simply not viable for some systems where the failures are safety critical. Also, with monstrously large dev endeavours like I have been involved in ( 1600+ developers concurrently, for example), effective and consistent branching strategies become essential for the efffecttive coordination of change implementation. The result of not branching in these situations leads to high numbers of regressions and an explosion of inconsistent environments.
@@ContinuousDelivery I can't take seriously someone who considers themselves a failure for using a branch, or compares it to running with scissors. That is the sort of holy war language that doesn't belong in a real discussion.
Branches work best when they are used like transactions in a database. They should be small and short lived, but they should also be consistent and complete changes (not complete features).
@@tube4Thabor As an embedded software guy I couldn't agree more. There are so many different kinds of software development, creating one nail to solve this problem turns all projects into hammers. And sometimes forcing the project to be a hammer when it's not is way more expensive than choosing the right process in the first place. It's nice to have this tool be available, it's not a solution for all projects.
@@tube4Thabor "Holy War" Yes! It's surprising because most of the developers that get religious about any "development philosophy" including functional, OOP, branch strategy, code coverage, TDD, BDD, Scrum vs Kanban, etc, are *junior*. It should be about applying the best practices in the right amount, but not *all or nothing*.
we tried CI/CD with a small team of relatively green developers and it worked surprisingly well for game development, BUT this required a lot more management than when we did feature branches, and pretty much continuous communication between the devs. I can't imagine how a workflow like this would work with a team of over 4/5 coders or without a team lead that's keeping a lookout for issues 24/7.
I can only say that it does. I have done this many times with teams of 30 - 50 people, and often with teams in the hundreds. Google does a version of this with 25,000 people working in a single repo.
I am reading this as a team of people were in learning mode all at the same time. This is expected.
What would happen after you have merged your small changes to master and part of your code is in product but not yet release. You get told the feature is no longer needed or you need to pause work indefinitely. How do you make sure your partly finished function doesn’t pollute the code base.
Well, I’d question the sanity of the product owner who thought this was the most important thing to do a couple of days ago, and now wants to pull it. Then we could either leave the changes there, if we thought that we’d come back to it, or more likely delete them.
@@ContinuousDelivery So it sounds like you're not a fan of rapid prototyping, where you iterate frequently with user input to determine the final shape of the product.
Maybe they cancelled the contract or the client folded, or funding was lost. Your initial response shows the tunnel vision of a dogmatic approach.
@@jasonzuvela I don't think I am dogmatic, I have tried both approaches, so have an opinion on which works better. Sure, rarely, there are changes that completely invalidate the assumptions that your team has been working to. That is always going to be disruptive. If that happens after a CI approach vs after a FB approach I don't see that either approach helps more than the other to untangle the problem. You are, presumably, going to have to extract code that may or may not have been built on top of. This is not as trivial as merely reverting a branch. So I don't think that CI makes this worse.
I've seen developers committing directly to the main branch and never doing a feature or a test branch. The changes are fine and safe. They don't break anything. And then I see the changes reverting what they committed a week ago with a comment saying, "I've made a mistake, this design turned out to be a bad idea or I found a better way to do it." The next thing you know, so much crappy code are already in there that are non-functional. They're harmless but dead code and it just pollutes the code base.
I don't really have industry experience, so take what I say with a grain of salt, but from what I know of working on personal projects/open source, branching becomes an issue primarily when multiple people are working on big changes and don't push to master often enough. If I remember this video correctly, it discusses how using feature branching leaves the programmers in the dark because the master branch is always behind, and it puts an obstacle between testing features with other parts of the codebase because each feature is worked on in isolation.
The issue with committing directly to master is that version control serves 2 purposes:
1. To share work btw teams
2. To serve as a backup if your local machine suddenly explodes
Sometimes programmers commit code because they finished a feature, but other times it's the end of the day and they want a proper work in progress in the repo to pick up on next time. Programming is a messy job a lot of the time and committing works in progress that aren't properly tested and cleaned/refactored will cause confusion and unnecessary effort by other programmers trying to work off of buggy/incomplete features. Obviously, these wip commits shouldn't be put into production if they potentially break something.
Using feature branches creates a separation of these 2 purposes: If you want to share the code, you push feature branches. If you want to save a WIP, you commit. Features should be atomic in nature: small enough that they cannot be broken up into further features, and they take a small enough time to complete that they are never far away from master. These features represent a handful of commits that you are treating as a single package to be integrated with the master branch. In a way, this can still be thought of as "approaching" continuous integration since your branch is never far off from master, but you are still keeping the master branch clean, while keeping it up to date.
tldr: Yes, you should push to master often, but no, it doesn't need to be every commit.
So, I still am pro-branching, but I minimise it to a semi-daily merges that guarantees the work is always very close to the truth. Having unfinished work in production doesn't seem to work for me, as it generates bugs more than good results for the devops process.
Exactly! I mean, just take the argument that branching is "hiding" implementations and therefore cannot be continuous integration... equally, Dark Launching (or Blue/Green) hides the implementation from being in the Production system.
The whole "don't branch" thing (and then going on to say that we should minimise branches) just pushes the problem(s) elsewhere in the development pipeline, just so we can continue to use the term "continuous integration."
I'm happy to just have "semi-continuous" integration, I don't see why me or my team should be under any pressure to continually put things into Production.
@@redtela there's a mild difference that in dark launch the other developers and users don't run your changes in main line, but already may be executing your unit tests along the stuff they work on, so if they modify something your new changes will plan to use, it may break earlier, than brach-way.
That said, I quite agree with your sentiment that branching and dark launch are quite similar, probably depends a lot also on the team, their overall habits, and the SW itself, what it does and how and how the work is split between developers. I can easily imagine situations where is no practical difference between the two whatsoever.
@@ped7g that "difference" (or lack thereof) simply tells me that the advice of "don't branch" is just a synonym of "make it someone else's problem."
At some point, we all must take some form of acceptable risk.
We follow gitflow (at my insistence), and so, we use feature branches. We also use unit tests, pull requests, integration tests, regression tests, manual QA and UAT. We use Pair Programming, BDD, TDD, and other methods. We have well over 1000 components in the system, each probably has 17 different "versions of the truth" and a hotfix can leave a developer laptop and arrive in Production within 30mins.
Could we do it better? Sure. Are we "statistically producing worse code than if we didn't branch" - I doubt it. Could we release faster by not branching? Possibly, possibly not. Do we "hide" information from other developers? No, because a pushed commit can be pulled into any branch, at any time.
@@redtela you are doing it wrong
Branch in behavior, not in code. That’s a great one. Eye opener.
Nicely explained! Thanks
I've been professionally coding since 2008 and the best way to organize code I have seen was the one I thought was the worst. It was bank using IIS as the web server, and the site of the bank looked like a single site, it wasn't, it was hundreds of tiny little sites, each of them with its own repository, the manager assigning the tasks was the one controlling who was using each of the modules or tiny sites. And we needed to make a change, we had ownership of that module for as long as it took to make the change.
This was a very fascinating and informative video! I learned a lot from it but I have a question.
How do you handle speculative developments where you are trying out ideas to see how feasible they are and how compatible they are with the rest of the system without committing them to the central trunk? I don't see how to do this without a branch. This is because you don't want your speculative work to interfere in any way with the production system until you are satisfied with it because you may end up discarding it entirely if it proves to be a dead end.
Using branching for that specific activity is totally fine and probably the only usage that is acceptable. Branching is not necessarily evil per se. It’s a Continuous Integration anti-pattern, though, so to be avoided when you want to integrate often with other developers in order to deliver changes as fast as possible and avoid merge conflicts (typical of long-lived feature branches). However, bear in mind that the longer your speculative branch staying unaligned with master, the more it will diverge with the rest of the development hence the more likely it is that your speculative work becomes less and less relevant.
For real speculative things, treat them as what they are, experiments, focussed on learning. Aim to always throw away any code you write in these experiments, the value is the learning. This means that you can be more radical in your speculations and aren’t constrained by having to make your experimental code “production quality”. This approach tends to focus you more on what it is you really want to learn too. Try and clearly state what you’d like to find out before you begin.
@@ContinuousDelivery Thank you both for clarifying things for me! This is kind of what I expected the answer might be but I wasn't sure.
I really liked the way the method presented in the video prevents huge divergences in the production code. I've been through a few nightmarish project merges and that was back in the SCCS days. I can see where GIT could lead to even worse ones if you don't impose discipline.
I like the focus on true CI/CD workflow here as the ultimate goal.
Personally I do like and currently use feature branches day to day. But we also try to keep our changes small and commits frequent so pull requests are small and they do go into prod on merge to main.
Maybe this is a middle ground?
If a team could ensure pair or group programming most of the time, then committing frequently to main would work nicely, but in reality that can't be maintained just because even the most committed pair programmer wants their own space sometimes.
Anyway, my key takeaway from this is that implementing proper CI/CD is the key and how ever that is achieved then happy days!
Nice video Dave I'll be sharing at work 👍
Branching doesn't mean you are doing independent work, it's a logical process of adding new features. Branching works in large development projects and is absolutely necessary. If you are doing something like Netflix, then you can proceed as you say, but the real world needs branches.
Feature flags.
Espectacular Dave! I loved this
Do you have any videos that explain where QA processes such as regression and feature testing would fit within CI and CD?
Not specifically, but I should do something on how roles change for CD. I recommend that you aim eliminate dependence on manual regression testing. That doesn't mean that QA has no place, but the role shifts to exploratory testing of new changes, not going back and checking that old stuff still works.
This works most effectively when QA work closely as part of the dev team, evaluating changes while they are bening built, rather than waiting until they are finished. This is easier than it sounds if the team is following the CD principle of "work so that your SW is always releasable".
I will think about doing a video that cover some of this in more depth, thanks for the suggestion.
At my previous job we used SVN with a trunk based approach, and we constantly had red test problems or even bugs/crashes that weren't even covered. This resulted in a stop that blocked 200+ people. Later we switched to Git, revamped the package and release system, and we were able to use feature branches as well. This solved many of the issues during development as we had coverage on our branches. OC we frequently synched with master, and we merged back the feature branch as soon as it was done which happened within a sprint. Since we had good code separation among modules the merges were fairly easy.
The hardest part was figuring this whole thing out and creating the custom package manager plugins and Jenkins scripts, etc. But because of the previous experiences nobody wanted to go back to trunk based development although we could have solved the problem.
With that said I think I understand your point, but we always have to evaluate our situation. Currently, I work on an enormous 15-year-old monolith software with a lot of legacy and new parts on top of it with more than 1000 devs. A code change in one place can break a lot of stuff at other parts and it's very hard to identify what went wrong. Not to mention that we need to wait until the next day to get test results.
In this situation I wouldn't concentrate on trunk based development. At least not at this state. :D
There are certainly times when adding FBs can reduce immediate pain, but I don't think they make much sense as a long-term fix, for the reasons in the video. I have seen the opposite too. I saw a company that did what you did, divide up a large code-base, each team began working on FBs, but they had lots of problems because of unpleasant dependencies in their code-base. So they made the branches last longer, to ease the pain. It did ease the pain, locally, within each team. But when I saw them, they had not been able to build their system as a whole for over 18 months.
The need to integrate is real, the longer we defer it, the more difficult it will become.
@@ContinuousDelivery I completely agree. If we hadn't been able to merge back frequently we would have had the same problem.
Yeah... I love it when I get coworkers changes right away and it breaks the code for whole team....
Also it's not like: #1 teams don't talk what the end plan is, #2 we can't look at other branches, #3 create our feature branch on top of another unmerged feature branch.. #4 rebase our work on top of any current branch.
Write some tests and gate the commits!
Exactly, avoiding branches just causes a massive amount of problems. The only “problem” with branches is merge conflicts but they arise from a lack of communication and planning and it highlights what needs fixing for it to not happen again, handling the merge conflict itself isn’t even that bad and it’s less inconvenient than everything breaking suddenly because someone pushed in a rush and the bug wasn’t contained in their branch.
@@skylark.kraken But if the team writes tests, this "pushed bug"will not be committed until fixed, right? I think that will happen if the project does not have the test coverage
@@nero1375 Nope. You changed the Client history functionality and updated the tests. It's all working for you. I'm creating functionalities A, B and C that uses that history you've changed. Your code works and is tested, the moment I download it it can break my code, because I'm not expecting to receive your changes.
I just wanted to reach out and say that I recently have gotten in trouble with Branching... The issue is that I was working on a new feature, so I made a new feature branch, so far no issue. However this feature branch eventually started to explode way beyond the scope that it was originally supposed to encompass. The branch quickly went from Integrate with the new API and make a framework for it, to create a control to work with said API and test some stuff, to make some controls and use the API... and then it starts to get it to many many other areas of the program. Lesson learned. SMALL INCREMENTAL CHANGES!!!!
"Hey boss, can I talk to marketing? I think my feature branch can be a new product."
I think u making the assumption that there is one way of implementing CICD and that there is only one place where all feature branches converges which is PROD. I think with a good branching strategy (feature/(123) -> develop -> release/(1234) -> production ) with each branch or branch prefix mapping to a environment, life of that dev team would be much better. This allows for less conflict, two convergence points (merge) for all features (in develop and release) before prod and 3 test env (where 2 env: develop and release have the integrated changes) . Pretty sure there are other ways but a development team of atleast 15 members-without branching it would be a nightmare.
Very well spoken, constructive and informative. Thank you!
This is a big problem. Unsafe code ending up in production - waiting to get exploited. I think the demand for instant relief has gone to far.
Nope, this is how some of the biggest, and some of the most reliable, systems have been built.
@muteza You write that as if you think it replaces all other common practices like code review and static analysis and so many more tools
@@Stylpe Well, it changes code review to pair programming, according to the video and comments. If the best developers/teams merge many times per day, the code reviews have to go away. Even if you have the automated testing to cover your code confidently, you're still then missing out on reviews of a feature as a whole, and instead getting reviews or pairing on small changes without the context of the entire feature. Oddly, even Martin Fowler himself says that for actual features, not just small changes or bug fixes, the best practice is short-lived feature branches.
@@rhakka I'd argue those are issues with better solutions than feature branches, and which this style of CI will force you to learn, for the better. Better separation of concerns, dark/dormant features and feature flags, realising that you don't have to, nay, shouldn't, be producing new code 100% of the time but also review and explore other recent changes both for your own learning and as a kind of code review, having the whole team paying attention during sprint demos, understanding the difference between deploying and shipping.
I'm working on a project that, for political/organizational/financial/historical reasons, i.e. reasons "out of my team's, site's and organization's control" do require environments that we can't just virtualize or mock completely. So we're usually creating throwaway test environments that don't rely on some of its surroundings, but for some features we must create these throwaway-test-environments in the context of some long running services that are "production-like" and there we are relying in huge parts on manual tests. These are contended resources.
So if I started pushing changes every few minutes to the main branch, QA could never keep up with manual and explorative tests and If a colleague was doing the same commit frequency in the same vicinity of the code, there would be a permanent conflict too.
Taking that to the extreme it would be like having the whole team develop in Etherpad, scheduling a build+test on every save and just keeping those builds that pass the tests. Which sounds like a frustrating edit-war.
The "Race to change first" is the same in the CI case, isn't it? Whether I commit a branch that contains my changes, or whether I commit directly on the main branch, we don't get rid of the situation that "the dog always bites the last one".
If I branched on behavior and tested on behavior, what I'd get is a combinatory explosion on all behaviors available at that point in time.
If we discovered a bad behavior in one setting that the tests haven't caught, and opted to revert to the old behavior, I might now have a new combination of another behavior that was built upon the newer behavior and might not cope with the old one, even if I thought the new behavior was equal to the old one, but just missed a small crucial difference.
So doesn't behavioral branching just shift the burden to another place? From the branch to the system under test, i.e. from the dev phase to the test phase (that might even be different teams, based on the company structure)?
As I understand the problem, it's not so much about the time a branch lives, but rather generally the lifetime of _any_ overlap in different behaviors, be it on a branch, setting or any dark or productive environment and shifting that overlap elsewhere doesn't really solve the problem. of course we cannot eliminate this overlap without blocking people from progressing.
IMHO that overlap-time really boils down to the one crucial thing: The time it needs to verify a change works, i.e. to run all tests required.
The faster the tests, the quicker the increments can come in and likewise because short increments mean less things to test, small increments are generally beneficial.
However, while it's true that smaller increments are good for testing, we must not forget that there is a minimum effort for every test, no matter how small the increment.
And that minimum test overhead is really what really limits the pace.
If that's too high, branches/settings/dark environments will need to live longer to get things tested.
In my experience shared, contended environments are always a point of pain and there are two things that make them miserable:
If you have a highly contented branch, changing too quickly, you're never programming against something reliable, changing too slowly makes big bang changes that also create big conflicts.
If you've got a highly contented test system, you never get to test something stable, if changes come too quick and you'll need to test too much, if changes become too big.
If you never get a stable state, you're never going to be comfortable releasing, likewise, too large changes make you react too slow if something goes wrong.
Every company has to find the best balance for them, review, adjust, try and repeat.
I'm a dev with 20 years professional experience. I disagree with this. Code should not land in master if it's not ready. It doesn't have to be perfect and bug free, but it shouldn't be horribly broken. Where do you keep this code while you are getting it ready. But I think there are some important rules for when working in branches:
* Merge/rebase master into your branch often.
* As soon as you have some parts of your code ready, get those commits merged in to master asap. A git rebase -i is useful to move the commits that are ready to earlier in the commit graph so that these can be merge without bringing in the commits that aren't ready yet.
* Do pay attention to what the other devs are doing. If two devs are working on the same bit of code, then they should maybe merging each others branches.
Where did he say it would be horribly broken. This is who’s point. Master is stable, is up to date.
He didn't state the assumption here, but I think Dave lives in Blue Sky World where there's good unit test coverage and you can see in 10 seconds or less whether you've broken something in main, so you fix it before pushing back in.
Lots of high functioning teams operate that way, but it's not always easy to end up on such a team without orienting your career specifically around that goal. In any event you can introduce change incrementally, and I'd say adequate (and fast!) test coverage is prerequisite to even a hint of "pure" CI as described here.
Then you’re not doing continuous integration.
@@vectorhacker-r2 , CI is just a means to an end. The objective is to have a quality code in the main branch. You can thoroughly test the main branch, which you should but you also need to do due diligence in making sure you don't continuously pollute it.
I'm pretty sure that continuous delivery is simply what we all used to do before branching became de riguer. Code mindfully and you don't break the code base.
This is terrible, just because you were late to source control and can’t handle branches doesn’t mean that your idea is good in the slightest.
Stick to branches, it makes the development cycle easier and cleaner. Communication is needed and it solves all of the problems. Also if the code base wasn’t monolithic, oh wait you’re teaching against microservices and want monolithic code that can spawn all of the merge conflicts.
I’m really disappointed that new programmers are looking up to you, they don’t know how bad what you are teaching is. Please actually teach what new programmers should be doing and not what you are wrongly thinking it should be done, times have changed.
Feature branches are useful in that not all code should be in the main branch, you want to be able to isolate areas and work on them and have them reviewed before merging. By allowing everyone to push to main you are removing the review phase, and you only have everyone to push to the main branch if you can’t be bothered to communicate so it’s going to be left to everyone to review their own code. You want a dystopian developer experience and it’s horrible, just because you want to see what everyone else is working on easier and want half baked features now it doesn’t warrant telling everyone to destroy their code bases for it.
Feature branches work extremely well, they require a little bit of communication and a little bit of planning to resolve all of the “problems” (which just flag deficiencies in how everyone is working), you shouldn’t be teaching everyone a backwards dated methodology.
I’m just really glad that actual developers aren’t going to be reverting back to the Stone Age and ruining their codebases, it is here to stay unless we get something even better (pushing to main branch is not better, it’s a lot lot worse).
Everyone who has thought they had learnt something here are just going to find out that they’re a massive fool and have to learn about feature branches on the job, it’ll be embarrassing for them though.
You did describe yourself as “old school CI” but times have changed.
I wonder which of us has tried both? I have to confess that "late to source control" makes me smile, I wrote my own in the early 1990's 'cos we didn't have one for our project.
The data is on my side "teams that don't merge to trunk at least once per day have poorer scores in Throughput and Stability" - State of DevOps Report.
"If you merge your changes every day you don't have merge conflicts" - Linus Torvalds (Inventor of Git).
@@ContinuousDelivery with all due respect, could you be misinterpreting Torvald’s quote? I always thought, and practiced, merging frequently, but from master into my own branch. That way, when the day comes to merge my own branch into master, everything goes smoothly.
What you propose may work on a very small team of very disciplined and senior devs. Outside of that, it becomes a crap fest of people pushing half finished features that break other half finished features from other people.
As for building the complex systems you quote, could you be misleading people by not making a clear distinction between deployment and pushing into master? I bet that all those commits frequently pushed into master were not going live with the same frequency, not even daily. Financial and retail systems have strict policies about what goes live, when, and for what reasons. So you may had a swarm of developers pushing half finished features into master, but that did not translate into users, even a subset of them via flags, exercising these features until much later in the cycle. Then net result from a productivity perspective would be just the same as if these features were not merged into master until they were mature enough to be usable, but not before. So nothing gained but noise, friction, and arguments by having everyone pushing half finished code.
Commiting unfinished work to production is kinda scary. I also feel that the “behavior hiding” can introduce unwanted bugs and will need more efforts of removing these hidden behaviors in case the requirement didn’t push through or the requirements have been changed totally in the course of development (dev will have to look for each and every commit just to remove these tiny pieces of hidden behaviors). Also, does this approach considers the need for QA to test the changes? Does this approach requires QA to test the production version whenever there is a new change?
Ok so I'm 8 minutes in and it seems you're suggesting to minimize the size of the branches, not to be rid of them completely.
Wow that was some interesting insights. I have always seen master-branch development as lazy, sloppy and dangerous. I usually do it like this: i have a feature branch, but rebase onto master every day. When the Feature is finished we merge into master. Its safe and due to daily rebasing there are usually no big merge conflicts. The way of continuously working on master all together as a team, as suggested here does have some undeniable advantages. I will discuss this with my team.
Frequent rebase is better than not, but you miss what others are doing on their branches.
Personally, when the scenario you describes remains as you describe, it is generally good enough. It's when discipline wanes and the features that aren't yet done start sticking around in multiple branches that live for 3, 4, 5 days, and then weeks, and then months.... that's where it really goes terribly wrong.
@@michaelrstover i have discussed this with my team (I'm the youngest member of that team) and they told me that corporate has denied them master branch development. But we do not work on branches by feature, we have branches by story/task. Which means the branches mostly only life a few days before being merged into master. As I have seen what some people dare to push (untested code full of dublication that doesn't even compile) I'd say we sadly aren't ready to get rid of our branches.
@@jimihenrik11 Branch-by-Jira is also what we do, but it does still lead to quite a few branches becoming quite long lived, and the delays between creating the PR and it getting reviewed and merged are super frustrating to me, and create a ton of inefficient context-switching. But, alas, I also failed to convince my management of the advantage of trunk-based development (and I'm the oldest member of the team!).
@@michaelrstover usually we do pair programming, So we don't need context switching or wait time for someone to approve the PR.
I think he is totally wrong. A branch is only a pointer to a commit, nothing more. Feature branches is a developing method for mainly creating a clean commit history. If you have conflicts in your team because too many developers are on the same code it's better to try to find a different WoW e.g. swarm on that task or don't work on the same code at the same time. So don't stop using feature branches it's the only thing that keeps the commit history clean which is really important thing for finding and understating how bugs were created
Excellent, in-depth and easy to understand explanations by a real expert! Keep up the good work! And maybe we can call the "master" branch "main" :).
Master master master
Your videos are great and inspiring! Thank you!
Thank you
Thanks David for all your explanations! Very helpful
I like branching, but then I work in a field where features are truly independent on each other. Usually the branches in my projects live 2-5 days.
You can like it, but the data says that you are likely to be producing lower-quality software more slowly.
@@ContinuousDelivery That is dependent on what type of software is produced and how it is structured. As long as the features are truly independent it is not problem to let the branches live a bit longer.
@@karsh001 Isn't branching used to prevent getting in each other's way, where one might change a code breaking someone else's code? What is branching used for in your case, where things are completely independent? Just curious.
@@nicat6153 Its a good reason to use branching, but not the only one. Controlled delivery to production and simplified dev testing is the reason I use it in my projects.
Due to the way the problem space work it simplifies some stake holder management and approval processes and we can quite easily drop features if they are rejected somewhere along the line. Dropping features mid implementation is very common, and the ability to just drop the branch becomes a value in itself. Once implemented, the features live a long time, usually many years with none or very small changes.
Its all dependent on how the solution is structured and I wouldn't recommend long branches for a more standard SW project where you risk breaking each others code.
For my projects, though it works great.
I have to agree. We manage a legacy system that is very broad in scope.
This means Multiple features can be worked on at the same time without overlap, so feature branching works well, and turns out to be the simplest system to manage in this case.
Having a sql server database at its core combined with being a legacy system full of bad design means feature flags tend to be difficult to implement confidently. And we’re rarely working on new elements that would allow dark features.
We also really battled against a dev/master branch method, there was just so much drift between the two, after a month or so we had no confidence that dev was anything like what was in master. So I’m totally on board with the frequent merging of continuous integration, it’s just turned out to be a easier method to use feature branches for us, as conflicts are rare
Here I have multiple feature branches that stay open for MONTHS, and some of them are even from the same developer (myself). The code in testing is a long living branch containing a huge merge of all the currently open features. A few days before delivering I merge and close all the branches and curse in horror, some of them remain not merged, because they are not planned for release.
Our customer-side managers are uneducated in everything software-related and Devops team is there just to add bureocratic weight to my job.
Also the other company working on our system systematically forgets to close their branches, but that's another story.
I feeel your pain
I feel like once a competent person goes into devops or management, they start trying to make the development process more efficient. The problem is that software development is a creative labor. is complex and messy at times. There's a lot involved and there's no way to implement source code control without affecting the development process. Eventually, there will be so many constraints around developers that it will become impossible for them to do their job.
I understand where all of this management/organizational techniques come from. They don't want to depend on a few talented individuals, so they want to hire a bunch of code monkeys and enhance their abilities. Basically, streamline software development. But the best you can do is to try to get the really good developers and hope that they are smart and have good communication skills. If you hire of bad developers, no amount of organizational techniques is going to solve your problem.
Just be a solo developer and do it all your way 😁