If you don't think a single developer working alone can benefit from feature-branching, I don't think you and I are talking about the same thing when we say the word "branch", which would certainly explain the discrepancy. a "branch" collects units of work (commits) together. Each commit is the smallest change which works on its own. Each branch is the smallest change which justifies its own existence. Within a branch, prior to integration, commits are not a commitment - they are rewritten / re-ordered regularly, as the discovery process continues. Integrating them doesn't make sense, because they aren't final yet. Forcing integration prior to the change being done makes the history unreadable because each change becomes separated from its justification. And of course, the one and only reason we're even using version control (instead of just sending each-other diffs against the last timestamped snapshot) is to keep track of *why* changes happen. I think we both agree that integration needs to happen very regularly, but disagree strongly about the acceptable scope of both feature branches and commits. I don't want to be "more continuous" than the smallest justifiable unit. I also disagree that "a branch which can't be integrated" is the same as "lost work", and I would have thought (based on your other video) that you would be a strong advocate against that sort of position. Sometimes code needs to be rewritten based on new information, and when that happens, it's always faster than writing the original code. It really sounds like you're talking about traditional, long-lived branches, rather than feature branches. As with so many discussions encountered on the Internet, I'm pretty sure I agree with everything you say, except for the specific terminology.
@@OggerFN their individual justification, yes, but not their overall justification. Not the context of which commits relate to it, in order to define the goal. If your commit is fully justified without any related commits, go ahead and merge that. If whoever is reviewing could ever ask "why?", then it should probably be grouped into a branch. Meanwhile, if whoever is reviewing could never ask "why?" for any of your individual commits, then your commits are too large.
@@________w You should write the task the commit is supposed to accomplish AND the reason for that in a commit message. For example a habit I have been using is naming the ticket a commit is relating to in it's message in order to "group" commits.
Once more insightful thinking of how sw can better made. Thanks Dave Farley. Since this is a so difficult practice to get the teams and/or Devs to move to, if possible I would love a future video of how to start with it. Something like what needs to be done from the beginning, and a possible adherence to other techniques, etc. Thanks in advance.
You must still pass the integration pipeline, though.^^
3 роки тому+19
Love the sarcasm! And also, I agree! It's perfectly acceptable to use many different strategies and CI and Feature branching aren't mutually exclusive. You can do feature branching in one step of your process and CI in a later stage. No worries. Honestly, this video is too dogmatic.
As someone pointed out...only if you get through the pipeline. And that should have human beings in it after the automated checks. :-) I mean..i heard about QA Somewhere......
@@timothyblazer1749 I don't think that this pipeline as described in the video includes human beings. (EDIT: You wouldn't be able to integrate that often if QA would look at every integration.)
I recently retired after 47 years in the field, slowly trending from coding to writing tools to version control to configuration management to as close as I could get to continuous delivery, which wasn't very close because I could not convince my teams of some key points. I am hugely enjoying your videos and your explanations, and thought I'd share a few observations from my time in the field. 1) I haven't heard you say this actual phrase, but some major luminary very early in the software world's development stated "Nothing of any significance has ever been accomplished by a team of more than 7". I've heard you express much the same idea, though, quite a few times. More people than that exponentially increases the chances of miscommunication about the design of the system. 2) I used to sort of half-heartedly suggest feature branches for extensive, disruptive, dangerous changes, but my teams always resisted it hard for the reasons that you clarify in this and the other videos about "don't feature branch"; essentially, merge hell will invariably result. 3) I worked in about 9 different shops that all claimed to be doing Agile Development. In one of them, they were actually doing Agile Development. In the others, they were indulging in rituals designed to fire off the buzz words managers wanted to hear, but these rituals usually were bumbled so badly they didn't do much good and in some cases were actually harmful, like when the boss was the scrum master without training and did most of the talking.
I'm interested in what your opinion is on Code Reviewing practices, in light with your strong preference for CI workflow. Some companies I've worked at, have a pre-merge code-review process like Github's PR system that needs a review before being merged into trunk. A pre-merge code-review process or mindset would then only work in an environment that considers a pair-programmed changeset as properly reviewed? Would you consider a pre-merge code-review policy to be an impediment towards incorporating CI? In companies that have a post-merge code-review process this problem doesn't seem to exist, but one would probably require an adequate level of (non)functional coverage in their automated tests to catch mistakes quickly, such that code-review is only needing to (ideally) focus on design. Are you therefore of the oppinion that a post-merge code-review process (or pairing as mention before) are two key processes to have in place for enabling CI?
This is a topic I see come up frequently, so I hope he does a video about it. If I remember the topic correctly from the Accelerate-book, as much as possible from the pull requests should be automated, leaving stuff like exploratory testing and such to the humans. And for the pull requests that remain, the team should make code reviews a high priority to ensure flow
Same question as I wanted to pose as from my experience the code review part is the one that causes your branch to stay non-merged longer than the design and implementation parts put together and even multiplied. For post-merge code reviews I don't see how they would work as they require far more experienced developers to go through (specially with continuously pushing unfinished work that makes the same mess as merge instead of rebase does in git... nightmare for reviewing so you just rubber stamp it and be done with it). With partially complete pushes you also require far more experienced developers as they must understand that something is still work in progress and somehow still envision the final design. And in my experience experienced and high quality developers are hard to come by so I still can't imagine getting higher quality code from CI compared to shorter tasks, rebasing and serious reviews of the design. Short term faster... Yes. Long term more maintainable... Can't imagine how.
@@defeqel6537 I've seen that video and it might be just me but I don't see pair programming as replacement for code reviews (at least not those that smell a bit of formal code reviews) - to be honest I can't even imagine somebody constantly messing up my train of thoughts in that manner...
One of the issues I have around CI vs Feature branching is more related to maturity. As far as I can tell, it works great when you have a stream aligned team working with a code base where most of the work happens in parallel. When you have a monolith, the problem with CI (for me) is that integration events become unpredictable. One team may be testing a set of changes, when another team commits a change that is automatically deployed. All of the tests cases that had been performed at have been invalidated as a result of the change. That's primarily why we've used feature branches in the past, to isolate changes just long enough to test them.
@@michaelslattery3050 Does it not get somewhat tricky to understand the subjective concept of 'short-lived' here? How would you define short-lived? How would you enforce it? I have seen teams that aspire to that and somehow these short-lived branches suddenly become very long-lived.
If the conflict is only coming up in a meeting, you don't have good communications or process. Those should be being resolved individually by the actors involved, for the most part.
Just wait until a "certified master" that doesn't know anything about development starts hassling you about why "can't you just merge it in".... and to add insult to injury said person usually doesn't even posess enough of an understanding of the concepts involved to be able to understand a straightwordward explanation... whilst simultaneously they're looking down on you for being a "boffin".
@@dafyddrees2287 what you describe is some toxic work environment. In a healthy functional team this is not a concern at all. I wouldn't work in a team that would act the way you describe.
@@vyli1 "healthy functional team this is not a concern at all." That's a very subjective thing to say. All the TBD I've worked on has been awesome and all the branch-tastic stuff has been awful. It corroborates what Dave has been saying in his books and videos.
- "When the information is modified in each place, then the long the time, the more those copies will diverge from each other". Well that is the description of entropy, it is so fascinating to understand that basic properties in universe affect everything even how we commit ...
Dave, I agree with what you say on so many levels. I am one of the people commenting on the other video. I would like to also refine my view. Please keep in mind, what I don't say explicitly is because I think we see things the same way. Let's recap what was the main issue in the other video's comments: the beginners of the team. Beginners are a healthy factor in a team for other reasons than the quality of the code. Technically, we tackle beginners by reviewing their code. Another reason to review the code is compliance. Either way, beginners will make mistakes that cannot be caught automatically, no matter how good your tests are. Let's add one more to the list of things to keep in mind: we fully agree that things should be integrated as often as possible. What we don't agree on is doing it at every commit. This is literally insane, for the reasons I hinted above. What we do instead is redefine the notion of a "feature". It's not an user story, but a subtask. Let's take an example. The junior on my team has been working for almost two months on a user story which ended up having about 30 subtasks. So on average, he finished (and integrated) one feature every 1-2 days. The reality is, some features (subtasks) were easy even for him, so some features were half a day, while a few took him as long as a week. I am fine with this, because we reduced the risk of breaking master, while also reducing the waste of effort (it was the beginner tinkering, so 1 week does not cost that much). Keep in mind, most of the time we were integrating every 1-2 days. We can also integrate all combinations of feature branches automatically and test them because since they're short-lived, there's less of them! We remove the branches after integration. So roughly we have only as many branches as the number of people in the team. Integrating at every commit would work only in a team of only seniors.
Not so. I have operated this way with junior devs, even raw beginners, many times, but you do it in combination with pair-programming. It is the fastest way to get juniors up to speed and productive, that I have seen.
@@ContinuousDelivery pair programming for 2 months continuously is not productive. They need to chew over problems themselves, sleep over them, then come back the next day and solve them. Yes, we paired up, and in addition to that he asks for help every 2-3 hours. I found this works best, in addition to pair programming. Also, the other big issue raised by me and others is code review, which you have not addressed so far. What is it then, reviewing each commit?
Interesting. We used feature branches at first - it is hard. Now we embraced, trunk based development and release branches. Release branches allow us to keep CI intact and enables patching. For new features, finished or not - we use feature toggles. Feature toggles allow several things: review partially implemented UX, understand complexity of database migration requirements. We also starting to embrace side-by-side approach vCurrent-by-vNext. You may say it is a lot of maintenance, on the other hand you are doing it anyways. Now we have much less feature branches, always up CI/CD, easier database CI/CD. Love the videos. Would like to see you touch code reviews topic. Often it is seen as one person offers ideas and knowledge sharing on optimizations or better code structure. But I think that it is more important to review how people handle exceptions, how people ensure that APIs are complete. Not all devs are experienced. I mean code reviews are not about beauty, it is about cleaning sh** left by others. This is more realistic and gives more value. Code reviews is about getting more experienced, not just about learning more patterns or algorithms.
I think big take away we need to remember is: “Continuos integration works by limiting the time period and thus limiting the fan out of the cone of changes, this reduces the risk of a clash”. That makes me wonder, if we use feature branches but could somehow guarantee the time period of unchecked code is the same or less as not using feature branches and forbid branching of feature branches to avoid the cone, are feature branches still bad?
That's precisely what you should do. In fact, this approach has many added benefits, such as you can increase your confidence, that no bad code will be pushed to trunk. With trunk based development, nobody can prevent a developer to accidentally push a commit that fails to build or breaks the project completely.
Depends on how would you guarantee that. If you have only 1 "big" feature branch and everything else goes to master, then you can actually keep the system integrated just fine (or systems to be precise - one on master, and one in the feature branch) just by merging master to that feature branch regularly. This has the downside of someone making a change in master that breaks you feature branch, but since that change is small, you can easily fix it in your feature branch after you pull that change from master. But what if you have 2 feature branches that take long to finish? If you merge them between each other (as well as pulling from master from both), then you have still all systems integrated AND keep the advantage of neighter branch "polluting" master until it is finished. But now you cannot release one branch without the other. One solution is to only merge those 2 branches "one way" (say, only merge A to B), but then B cannot be released without also releasing A. Also this strategy doesn't work for multiple unrelated branches. So you are left with 3 options: 1) don't work on the same code: when a big branch is ongoing, do not change the same codebase / features as this branch is changing. That way, you will avoid conflicts when merging it. 2) bite the bullet: just give up and prepare to do a massive merge once in a while 3) make smaller changes: by integrated smaller changes, you reduce improve in both above points - you are less likely to "hit" the same code as another change if those changes are small, AND when you do, the conflict is going to be small too. Of course, making smaller changes isn't as easy when you are not used to it and requires a complete paradigm shift in how you look at programming, especially if you want to go "i will work on this feature for 3 weeks and rebase my commit as I want and it only has to work at the end" to "I make a change and push it as working commit every 10 minutes", it will not happen over night.
3:20 I often feature branch when I'm alone. That gives me the benefit of switching features when I get stuck and I can come back later. You may argue that I should comment the changes out or implement a software switch when rewriting a function. That can indeed be a solution and I admit that I haven't tried it yet.
When working alone, I prefer to push every change immediately to Production, potentially toggled if it's still work in progress that users shouldn't see. The only way I can know whether my feature works as intended is to get real usage on it. Of course people's philosophies differ.
And that is why I enjoy the whole microservice hype in web development. It allows you to work in a small team. This way, you can have feature branches without worrying too much, given the tasks are also correctly planned in a way where the conflicts chance is very low. For such teams I find it very easy and enjoyable to have even bigger PRs, as they contain the full feature and you can observe all of the changes at once, giving you the opportunity to find more optimization routes.
Hmm, I'm still not 100% sold... Sometimes CI sounds like a really great idea, but the problem also is what do I do when I approach larger refactorings (like updating dependencies to a new major with breaking changes) that take time and will leave the project in an unworkable state until completely resolved or changes that we are not sure if we should carry them through... How do really revert a series of commits in a CI world, where they are mixed in with other people's stuff.
That's a really good point. I think CI as described is good for actual user-facing feature development, but you need a more nuanced approach when doing things like what you're describing. The platform upgrade is the obvious exception, but there are other types of changes that require a careful decision on a case-by-case basis. "Changes that we are not sure if we should carry them through" is an interesting one. Say for example you want to unify how your application does caching, and you need to audit every single network call it makes to update it to the new system. If you take the time to update each one before sending it out for review, and then (hopefully) merging, what's the likelihood of running into some catastrophic merge conflict? Seems pretty high. Maybe a smaller proof of concept would be more helpful then. But then say your team does dozens of such experiments, starting on them, but never actually following through to implementing them project-wide? Each experiment you run creates ambiguity in the codebase, where it's no longer clear just from looking at the code what the right thing to do is. In this case, maybe it really would be better to make a long-term branch, knowing full well that lots of code will have to be essentially rewritten multiple times over the course of the project. The new pattern isn't the deliverable in that case, but rather the consistency. I guess I'm not seeing easy answers either way.
If you wait for your code to rot into a state where a "large all-encompassing refactoring" is needed, you've already failed. Clean up your code before committing it and you won't have this problem.
@@froobly agree with Ross. even with legacy code, the first step is to write tests - both at a "unit" level and black box level (won't go into details). Make the constraints of the behavior of the code explicit, refactoring can definitely be done step by step in incremental and relatively risky way. if you are dealing with huge migration like breaking framework changes, then I would agree CI doesn't really apply, because it is a different problem. but you still cannot get away with automated tests.
3 роки тому+4
Day one: Checkout a new feature branch from develop. Day two: Merge said feature branch into develop. Rebase first if needed (if there are any conflicts). Day three: Merge develop into prod or staging or whatever deployment pipeline you have set up. Day four: Repeat. Feature branching and CI aren't mutually exclusive. You can do both. And I think you should. Keeping features small is just a sane thing to do and has nothing to do with CI or feature branching. If you can't keep the feature branch small, you should rebase it continuously (against the develop branch) until it's ready to merge. So, do feature branches against the develop-branch and CI/CD between develop and master with whatever steps in between you want. Ofc there's also other ways of doing it. Don't be dogmatic. Be flexible and adapt to needs and the preferences of your team.
I like this approach, but I would say you can also automate "merge dev into master": Day 1: Checkout a new feature branch from dev. Day 2.0: Merge said feature branch into dev. [you don't really need rebase, you can fix merging conflicts without it] Day 2.1: Automatically run tests using the lastest dev branch. (optional) Day 2.2: If all tests pass, automatically merge onto master (CI) (or merge manually) (after merging to master) Day 2.3: Run automated tests using master branch And of course, 2.2 could be done manually, allowing lets say a PR request or a code review to be made before it is merged.
As far as I understand, this is essentially the same thing as trunk based development. You could skip the "check out a new feature branch" step and get the same result.
My favorite story concerning this topic was having a dev lead who had no concept of how to merge using git. The dev lead was responsible for insuring the integration branch was kept in order. Additionally the dev lead had to coordinate this for multiple pods. The majority of our pod was able to commit and merge to the integration branch on our own. The dev lead insisted on doing the merge for the more junior devs. I noticed that one feature branch worked on by a junior dev was spanning multiple sprints. I made sure to let the dev lead know that having that branch that long was going to be trouble. I was ignored and you can all guess what happened...The dev lead overwrote a lot of changes from the others. To end the story, that dev lead no longer works at the company.
Nope - that's not continuously integrated is it? The risk is in the merges - and not understanding what other people are changing. If you wait until they've finished "their" feature (big problem already there with code ownership) then you're merging great big lumps of changes - which is asking for trouble.
@@joek4563 Great - now you have a broken build with tons of shit to track down. It will be much more difficult to sort out because you will be reconciling your finished work against the latest. Merges of huge amounts of changes suck absolute balls to sort out. It's much easier if you have to reconcile your tiny change against somebody else's tiny change.
@@joek4563 I've been doing this for almost twenty years now and the difference between trunk-based dev and this gitflow bullshit is massive. You just can't do big system-level refactorings at all if you're going to have lots of branches. You'd either break everybody else or never be able to work fast enough to deal with all the conflicts you'd get all the time. The only option is to work WITH other people - not to temporarily hide away from them.
How to use trunk-based development in a legacy system where there is huge technical debt and most of the code doesn't have good tests we can trust? How to ensure quality where each commit to the trunk can potentially break the system. Of course, we try to write more tests and do refactoring doing some new feature, but writing tests is not always possible when everything is coupled together.
Unfortunately you're in a difficult situation so you have to make moves to reduce the number of branches... Of course rewriting is one way of doing that, as nuclear as that may seem. There is no theoretical limit to how hard to maintain a system can become, but at some practical level it can become unmaintainable. All the best
With open source software, you do three merge/pull requests. If and when they are successful, these qualify you to be a developer. As a member of the development team, continuous integration is the way to go. Note: The size of the project does not matter. Whether we are talking about that upcoming JavaScript framework or the Linux kernel, three contributions are enough, roughly.
Continuous Integration is a practice that requires a high trust environment-where you can trust the people on your team. In low-trust environments, like Open Source development, Pull Requests shine. GitHub was largely founded on Open Source, so it makes sense that Pull Requests were largely invented by GitHub. CI would only be appropriate for a small core team in Open Source.
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period. No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on. Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others. Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around. The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models. The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
I think feature branching within a single repository can work, as they are visible to the team, but you need to treat each branch sort of like it's a trunk and regularly merge branches into other branches, not just mainline trunk. The advantage of this is if you do get some nasty merge conflicts you can put them off, let those branches diverge a little while you consult with your team about how to resolve them and prevent them in the future, all while continuing to make forward progress. Some of that progress may end up being discarded, but some of it won't, so overall you aren't completely halted by having to deal with the divergence.
My expierienced rule of thumb: smaller skilled good teamworking may start & come quit far with a strong disciplined checkin strategy directly on *the main* trunk/master branch, but those conditions tends to be not realiable with much more growing/larger projects teams especially including unskilled/newbies. Fitting VCS checkin & branch strategies depend of multiple aspects: a) project/task change-sizes & risks/fail costs (e.g.: not every deep re-design/re-strucure checked in as lots of tiny checkins but project-wide atmoic commit/submit), b) developers team's size, skills and discipline/motivation, c) surrounding IT & Development Infrastructure including build tool chain support, dedicated DevOps, Q&A and a good project (not directory) scaled merge/integration tool support (bare git is not enough)!
I think the context of what kind of team and what kind of product you are working with will affect whether this is effective. When you have many disparate repositories of code, each with few changes over time, it makes more sense to use pull requests, since things move so lethargically. But if you spend most of your time in one repository, CI seems like a much better option. For example, you won't have to re-synchronize your feature branches to ensure you're on latest if you are always working off trunk.
In general I'd agree, but in some cases it still seems useful to branch to me. Especially when doing some more experimental stuff, where you don't know, if you even want to add this change. But if you already know, you're going to merge this branch/feature sooner or later, it's always better to merge it now. It's much worse, when others already depend on a feature, which often changes, than when you are not able to merge this feature anymore. And you will always be able to merge your feature in most cases.
It seems to me that you need to watch the first video and use one of the techniques for deploying without releasing. What I hear when you say "feature branching is useful" is "I'm not good at deploy-without-release". I think you should get better at that and see if the tools, techniques, and paradigms you develop change your mind about feature branches over time. If you always fall back to feature branches, you'll never get good.
I am working in a big software company. 6 years ago we had big problems with pushing features from the different teams back to master because of the changes by other teams as you explained. Now nothing changed except two things: 1) We must push any change as soon as possible back to master. Even if the feature is not completed if a pice of code, e.g. on function, is completed so it can be pushed to master without coder breaking it should be pushed. 2) Daily build We do not have any integration problems anymore. Because the discipline of the developers increased because they do not want to break a daily build code quality and efficiency increased. And the "old" developer who knows the "old times" saying that work is now easier.
So. How do you get the branch through QA? Oh...that happens after you merge? What about merge conflicts? How long to resolve those? We know that feature branching is convenient for developers. But that's precisely why it is bad for development. CI is far better an approach overall.
@@timothyblazer1749 In most VCS merges are more complicated than they need to be. Git for example always needs a merge commit if you merge something which is not just a fast forward. In Pijul, which is still in development, you only need a merge commit if two changes change the same line of the same file. And merge conflicts should appear less often anyway.
At the company I work for, we implement CI/CD, but instead of master, we work on the UAT branch. Then once a week, we do a pull request to merge UAT into master. That way our production environment only gets clean and tested code, and hasn't crashed due to broken code in over a year now.
This is pretty much close to how I always worked. I usually work on the master branch and push/commit regularly. Pretty much any change that brings a benefit while not breaking anything. I use feature branches for bigger changes that take longer, but I can't check in the intermediate work because it might break the existing state. It would not compile or break features. Usually a bigger refactoring. Where I want the intermediate states in the VCS but not break the master. As soon as possible I'd merge the feature branch into Master and continue working on it there. In some cases I combined it with the Dark Launch or Feature Toggles. Minimizing the time I work on a separate code base. Funnily enough, I rarely have merge conflicts as I usually merge Master back into my feature. My colleagues kept longer running branches and had to spend much more time on the merges. Mostly because they rarely pulled Master.
Added costs of feature toggles is not always worth it. It takes a lot of upfront work to design and plan work on features so that they wont clash with each other. If not managed carefully, it can end up in really messed up spaghetti code. Especially in early stages of projects, when no one knows what they are really doing.
Not only that, there are data model changes that simply cannot live together with the legacy model. What kind of toggle allows you to have a table column value unique in a place of the code and allows multiple values in other? Answer: none that realistically enforces the constraint. I’like to hear how his version of CI deals with that.
What about changes that can't be completed in one commit and also can't really use tactics like Dark launching. In one of my pervious projects we had to make significant changes to multiple connected db-tables. I don't see how I can get around using a branch here. Yes the other developers will need to merke, but else we have broken code in the Main branch.
There may be strategies to work around that. For example, you could maintain more than one database table (or column) to represent the same data in different ways, and drop the older version when nobody uses it anymore.
@@cristianpallares7565 I guess that would be a good solution. I just fear the amount of copied code and the work to make sure everything gets deleted in the end. Well... those big changes should not happen to often anyway.
@@christophlutticke7150 I think this is only one way, you need to have that copy, cause I think yo need to have no downtime for you product during the deployment.
I totally agree with fact that the longer you wait, the more clashes and conflicts may occur. The problem I faced with my team is that some time the work on a feature is incomplete before a deadline and we wanted to deliver some other complete features. Without feature branches we had to do all sorts of things, like turn off flags, hiding controls and even rolling back changes.
Teams that embrace CD change how they organise their work. CD is working so that your SW is always releasable, so after every successful commit, it could go into production. You can do that by waiting till each feature is complete, but that slows feedback and results in lower quality, or you can work so that it’s ok to release, even if a feature isn’t complete. This sounds weird, but is the higher-quality approach, according to the State of DevOps Report.
I think the concept of CI has a lot of merit. We had to change our concept of a 'feature' to help work towards this. On the basis that 'no one should commit code to master that breaks the build', we couldn't get an entire 'feature' developed quick enough to commit to trunk/master, so we used feature branches. But then we just found that if we broke down what we meant by 'feature' into smaller bits, we could do commit to trunk/master without 'breaking' anything. This did lead to some issues about testing these 'mini-features', but nothing insurmountable. And breaking a classical idea of a 'feature' down into 'mini-features' gave us a lot of insight into just what and why the 'feature' was in the first place. This worked well with a general 'Agile' thinking in that there were a few times our initial 'mini-feature' break down showed us where we had gone astray.
I think we can have the best of both worlds: consider the following workflow: The is a "trunk" branch called "dev", and each person creates a feature branch from it and then merge it back to dev after the feature is complete, with a pull request and ab associated code review. After an arbitrary number of features are merged, a new release os born, and it gets merged into "master", the production branch. If we create a "dev-ci" branch, and have the team muse feature branch, but at the same time rebasing/pulling from dev-ci and merging their changes into dev-ci frequently, we then achieve the ci goal, and the feature branch with commits filtered by author is still good for a traditional code review
We basically do what you describe in the first paragraph. I don't think I understand how the process you describe in the second paragraph differs, though.
@@ShawnThuris sorry for the typos. Instead of commiting and pulling directly on the trunk branch, you merge and pull/rebase from your feature branch keeping it synced with the trunk branch. It may sound as extra work to accomplish the same result, but it would still allow us to link work items/issues on the management platform (GitHub, azure devops, etc.)
That is biased, in my view. What you call fundamental is not working in 100% real life scenarios. There are cases when it's useful to guarantee changes isolation for the sake of risks mitigation. Think of it as full transaction isolation in DB, like when evential consistency needs to be avoided.
Since you seemed curious about why people have an emotional response to this subject, I thought I'd chime in. What emotionally turns me off about ci is sharing my code before it's finished. Imposter syndrome is rampant in our profession and merging code into a shared branch before the feature is "done" make me anxious. Rationally I agree with you. Shorter cycles lead to better code and seeing everyone's else's "incomplete" might even reduce feeling of inferiority and serve as a learning aid for junior developers. Nevertheless the anxiety remains. Love the videos
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period. No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on. Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others. Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around. The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models. The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
you cant just pick the worst examples of your opposing viewpoint and the best examples of your own and then say yours is better... sure if you are messy you can lose 12 months of work using feature branches. Another thing: Your method introduces loads of dead code into the production branch. If I look into the production branch, I want to only see code that is active, or about to become active. Not half written code for a feature that will be released in five months. For me personally, a clean codebase is way more important than occasionally having a merge conflict or two.
I'm thinking here that even in a team that works using "feature branches" and has periodic long delays in "pull requests" getting reviewed it would still make perfect sense to split the work into a sequence of small focused changes for the reviewer to review simply because it's gonna be easier to go deal with and to give feedback on compared to dealing with a big ball of mud of changes that inevitably turns into merge hell. To me this seems like it would be a clear step forward by practicing the parts of CI that one can in a broader environment that doesn't do CI... maybe this could be used to smuggle more CI thinking in there in the long term anyway if folks don't believe in its benefits until they themselves have seem them working for someone else in a part of what they do. This is seems like something that is perfectly doable with just about any regular toolkit. This whole thing reminding me of how in an environment that isn't agile the easiest way to make things easier for oneself is to have one's own surroundings self-organize (even if it's just you) to the extent one has the ability to do so and to just work like an agile organization within a wider organization because it's still better for one team to be agile inside bigger organization as that at least still makes working more efficient and pleasant for you as a developer.
I tend to rebase on top of whatever the main branch is regularly so as to act as if I started my work after the current code base state. This isn't technically that different, but I find it easier to reason about. Essentially I agree with your video that a continuous straight line commit graph is preferable to me. At the same time, small merges are pretty inconsequential and easy to reason about. People do get quite heated about this so I leave it to them to deal with the worst merge nightmares while I commit on top of the latest when I'm done.
So the team I'm working with now has three branches: master, validation, and integration. If I'm introducing a 'feature': I make a personal branch, complete the 'feature', and submit a PR to merge to integration branch. I'm putting 'feature' in quotes because I'm referring to any changes in the code base. We generally limit our changes to take no more than one week of work. Twice a day, integration is merged to a new validation branch (ie. validation-2021-05-27), which runs an enormous library of validation tests. If any of those tests fail, the culprit PR will be found and either fixed or reverted before merging to master. These fixes / reverts will then be merged back into integration. If your PR was reverted, you simply pull integration to a personal branch, revert the revert-commit, fix the issue and restart the process. Now my question is: is this CI or feature branching? There are separate branches for features, but each branch only consists of a single person doing a maximum of one week of work. Does feature branching turn into CI when the amount of people on a branch, and the lifespan of a branch, are limited? For instance what if we implemented the same strategy, but 'features' were split into daily deliverable chunks, as opposed to a maximum of one week, does that count as CI? Edit: Just found a video where you actually explain the difference explicitly, guess I'll watch that later.
I have the same problem. I couldn't find any good solution to do code review after code is merged. Comparing selected commits seems to be very challenging (or even impossible). That is the only blocker which prevents my team to commit directly into master branch. At the moment we trying to creating branches which lives 1 or max 2 days. But I would prefer to merge directly into master.
@@rafacypcer1408 Why don't you use tags or a specified commit message? You can review the "sum" of these specific changes iteration by iteration. We used JIRA and Bitbucket, and we had commit message conventions where we put the feature id and the task id. This enabled us to review the task alone and also the feature ("FeatureId/TaskId message"). Each commit was visible under both the feature and task in JIRA. We used feature branches but with this you still can observe what is happening with your feature, and you can always create more tasks under your feature as it is only closed when the team is done. The only downside I see is that you can't reject feature merges, just the merges of your little tasks. But with trunk based development it's not really a factor as you want to apply changes rapidly instead of creating a big/coherent feature changeset.
Don't use code reviews as a gating mechanism for changes. Code Review should be something that gets done to look at how and why things were done the way they were and how it can be improved. If you use code reviews for enforcing methodologies and finding defects use automation for that. Yes it's 'after the fact' and the incorrect code is already in master, but using CI it can be quickly fixed. The flip side is if you and the other developers are constantly pushing changes and pulling changes code is being indirectly reviewed all the time.
@@Rekettyelovag Thanks. I also use similar pattern for commit message. I'm also able to see in my managment tool (in my case Azure Pull Requests) connected commits. But the problem is that I can see the commits, but not files with diff's (which is obvious, because it most probably git limitation in my case). I don't know any git feature which gathers selected commits and show diff's in files comparing the last commit. I san see changes by selecting two commits - not more. I understand that with CI/CD we need to change methodology. CI/CD is used mostly for fast development - not for code quality. Code quality should be resolved somehow different - like changing team culture, improving automated tests, applying linters/code rules etc. For approach with code review before commiting to master we had step where someone could verify someone code (which blocked development - because code 'frozen' during code review shouldn't be temporary used, by someone else). Merging directly to master forces every developer to review changes everytime when he/she needs to pull new changes. But it happens constantly. With feature branches reviewer was analysing the whole feature context. With branching directly to master this context is lost, because of continues delivery.
It would be great to have a clear explanation of the hypothesis. It's not clear what feature branching is. I believe that CI is not alien to use your own branch to work on your changes and then allow peers review. I understand that short time feature branches could be used. It's not clear to me if those are not even accepted in CI.
I think there is more to it than thinking ourselves as heroic loaners 15:01, one of the ways we breakdown complex problems and also with the code we write is the idea of breaking down a complex problem into specific concerns where we can think solely about each part in as much isolation as possible be it steps in a process or the responsibilities of an object. Isolate a single part of the greater problem at a time. Therefore, merge conflicts with someone else's code means we need to context switch to understand a potential different problem context in order to resolve the merge issue. Now in my experience its the guy who commits later that is responsible for resolving the issue (is this the unresolved special source that's nagging at the back of my head), and hence has to do this extra work of understanding an additional problem domain/context. Resolving who has to context switch stop the breakdown of their own feature and understand how their own problem context interacts with someone else's current work in the same problem domain ... should this be left to who commits second is seen as responsible for the merge ... the associated work is delegated on nothing more than who can smash that commit button before the other? It also means this is the point at which people's different coding styles interact, has the code been refactored or is still in a state of just putting together something to pass unit tests. I'm not saying leaving people in isolation till their own feature is complete before attempting merge with another complete feature, meaning there is a potential for a large divergence of code. I just think there is a social/phycological dimension to ... THE MERGE!!! I've found myself merging code ... before commit, "oh wait ... it's changed again" ... ok spend the time understanding the merge ... "ok right think that's my changes merged ... better do a pull form the remote" ... "oh jeez again some more small changes made before I was able to commit". This is doing my head in if only I had just been able to commit before I had to pull my hair out trying to understand this stuff written by someone else. Maybe it creates a feeling of winners and losers, and it's the speed at which you commit that is being rewarded not the quality of the code. Maybe I'm just crap and slow at merging peoples code, maybe needs ideas on "Pair Merging?", but does anyone else have experience of he who commits last is the loser??
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs. There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done". As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions. My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE. Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
@@ddanielsandberg I find it a bit scary when normal human phycology gets referred to as “A BUG”, I think everyone has different personality traits, strengths and weaknesses, and in different situations weaknesses become strengths, in some situations being an introvert that isolates themselves to push through a single focused idea is what’s needed, but then there’s other situations where it’s collaboration that’s going to be the enabler. That doesn’t mean I haven’t been pissed off and got people kicked off a team … but I find it most stimulating and sometimes most frustrating to work in a diverse team, argh and though it pains me to say so, good management of people counts.
@@derekcarter4157 I didn't mean to imply that people are bugs. :( I meant it more like the system is buggy if it enables and incentivises certain negative behaviours and discourages other positive behaviours. The software industry has spent the past 40 years hiring and attracting certain personality types and I think it has had a detrimental effect. I know how it feels some days, I'm thinking of leaving the software industry and become a carpenter or painter or something because this has given me CPPS. “A bad system will beat a good person every time.”
@@ddanielsandberg aye this covid lockdown from last year seems to have been terrible for my eyes just being in the house never looking more than 7 feet away (or maybe I’m just getting old). But yes it’s a software dev is a creative endeavour, even if all our best ideas are stolen! Processes and techniques are to help people create and communicate (and I’m the case of SAFe and Scrum make a quick buck) teeheehee.
Literally a day ago was talking with a friend of mine who is a QA lead at a big IT. He complained that it's a nightmare to deal with 10 teams pushing changes into the develop branch. He said that it is unmanageable because it's almost impossible to blame what feature, what change, and who introduced a bug. They spent hours just to identify this info. They do CI, but everyone understands and complains for that kind of workload branching model would most likely be much more manageable for QA.
Not to be confused by not branching based on user stories. It is still helpfull to create isolated environment, a.k.a. branch, for user stories which takes couple of days only to implement and integrate. It is the smallest agile/scrum unit of work that validates a branch. It is more easy to discard a branch than to go back in history in case something goes wrong and developer has to start implement the functionality from scratch. Issue is, as pointed out in a video, a feature branch which takes several weeks until it gets tested and merged. Just my 2cents of course.
I've really enjoyed your channel for a while now. I have picked up far more insightful ideas than I ever thought I would when I began watching. So my first serious comment on your channel is this... I absolutely love the shirt!
I've only rarely run into major conflicts when merging. Even when we had several people working on major additions to the same product. Most of the time branches are used to make changes or experiment without breaking the main branch builds. And since there are many things that we are unable to build and run locally code has to be committed to put it through the build, test and deploy pipeline. We could certainly investigate making it possible for each person to do that locally, but since branching is actually working it would be hard to justify the time and expense. Personally I'm not a fan of pair programming. We certainly use it on occasion, but I'd burn out in months if I had to work with someone looking over my shoulder all day. I've turned down lucrative job offers because the job would have involved working like that all day every day.
Excellent illustration of why the ingrained scatter-gather pattern of working is inherently flawed. We're experimenting with mob programming where all these concerns melt away, since no one is sent to their cubicle to work in parallel and in isolation.
I have a shell script, git-sync, that replaces git pull --all + git rebase + git commit + git push, which mitigates some of these issues. Dave criticizes this approach at 7:21, but my script goes farther. It also checks for git conflicts across all other active features branches (we delete branches after merge), and list other branches. Finally, we have very small tickets, and we break them down even smaller to tasks which are safely atomically merged to trunk. We aim for "tasks" to last no more than 24 hours. In stand-up, we view branches older than 24 hours. Of course, none of that's full integration, but it gets us very close. Close enough to be better than having tons of feature flags, which is often required for trunk-based.
Thanks for the vid! It's very refreshing to see motivations for "why we do things like we do" other than: * that's how google does it * that's how good developers do it I appreciate you adding a scientific-like point of view. That being said, the context where CI really matters is almost clear, but not quite, so I'd like to understand: * Does CI matter more for companies where really big teams (or lots of teams) are working on the same code-base? * Does a microservice architecture make CI more relevant or slightly less important?
While all I can say is "I have an experience-based opinion", here's my take on the two questions: 1) It matters for different reasons. In smaller teams, you will have fewer versions of truth. At the same time, you should also have fewer reasons to diverge. In bigger teams, the it's the exponential growth of failure points over time that makes CI pretty much a necessity. 2) There's something interesting about microservices, and that's the decoupling of implementation from API. As long as you're not changing the API, there's no need to withhold any tested change. Indeed, if you can push out the change the minute it's made, you're maximizing the value of your work by minimizing lead time. And one of the core advantages of microservices is that they let you do this, so if you don't - you miss out on a large portion of their value. BUT - when you're continuously changing the API, you need a strategy. Many companies don't think about that, and hence get into a terrible mess when CI'ing microservices.
We have builds and test (functional and unit) runs on every commit of our branch. We do rebasing instead of merging of main develop branch into ours. We strive to make multiple smaller MR/PRs but they always have to be a finished working bit. Since codebase requires quite a lot of refactoring so that C++ api guarantees code correctness far more than we are able to achieve in Java version of the product even though Java version is younger... Which is really important for multi threading purposes (when using shared data). But the most important part are code reviews (no tests are not replacement for code reviews - and I'm talking about to some extent almost formal code reviews - they are additional help and documentation). Based on this and the fact that branch is developed for a week and then gets stuck in code review for about 14 days (including changes to the code) I would really like to see how CI and constant pushing of code would produce better code. Faster written for short term gains? Yes. Better... Please explain how as I can't see it work for anything more than service development where partial uptime is more important than stability of the whole.
I'm curious about when/if Product testing fits in CI. Say that a developer (alone or working as part of a pair) is working on a feature, which was specified by a Product owner of sorts. There's always a chance of some level of misunderstanding between what was asked and what's being delivered. So it's now about whether the code works or not, but more about whether it does the right thing. In that case, being able to deploy a version of the code to a test environment that can be exercised by a Product person is quite useful. However, if we're always committing to trunk and the code is doing the wrong thing, it might already be too late. I can imagine one solution for it being feature flags for every single customer facing feature. While I can see that working, it can clearly add a ton of incidental complexity to the act of adding features and changing a product. What's your recommendation for that kind of scenario? Thanks!
Thank you for the video. How would you go about prototyping different solutions whilst working on the same branch? It seems to me that rollback might be a really complicated in this case.
The more often you commit, and therefore the smaller your commits, the simpler the rollback! However, for experimental things treat them as something different. Do them on a branch, but then throw the branch away (or at least ignore the code in it) and re-write the code from scratch to production quality once you have learned the lesson. This approach is sometimes called "creating a spike". A spike is an exploration of some idea, the result is the learning, not the code. That gives you the freedom to write crap code if it helps you learn faster.
I do get the overall idea but of course there exist several cases where feature branching is actually more productive and perhaps even mandatory. A few examples that come to mind: 1. perform a refactoring in order to replace an external library 2. Any form of "experiments" that are not supposed to be deployed in prod 3. large changes that are difficult to release "darkly" (or that can be released darkly, but with a high extra cost in coding the "darkly" (which is actually a feature itself). Again: I get the general advice but in my opinion "continuous" must be taken with a grain of salt, as with most practices.
I'd say it the other way around. CI works for nearly all cases. Occasionally branching, not necessarily feature branching, is useful in unusual situations. So "take branching with a grain of salt". The other take on this is that CI applies a pressure on you to think a little more about the design of your system. So, for example, I think it extremely unusual to need to use branching for your first example. Sure, I have done it, but these days I don't need to. Because I'd probably have a facade or adapter between my code and the third party library to make my code testable. So switching out the library would be easy. That is only not true when your code is tightly-coupled to the library, which is in some, very unusual, circumstances I may allow in my code, but very rarely.
thanks for your comment@@ContinuousDelivery. As stated in my first comment, I agree that the worst evil is multiple long lived branching, to the point of every single developer in the team creating a personal branch which quickly becomes impossible to merge (I personally witnessed a few of these edge cases). Yet sometimes branching is good, but you (that is: not the single developer but the team maintaining the codebase) must know why and what you are doing. Other aspect, this being more a technical detail: what do you think about the "short lived branching" technique, compared to the "commit to trunk"? In my opinion the short lived branching (branching but merging the branch very often) preserves the goal of continuous integration but is more solid in terms of protecting the main branch from breaking commits and it's easier to implement with modern CI tools.
Generally agree with this precept to integrate small bits of code frequently. The only exception I can think of has to do with cybersecurity or life-safety system requirements that disallow dead code (i.e., can’t get there except in an unplanned error state) under all circumstances due to the risk of accidentally entering those code segments. In less critical codebases, I can easily agree that feature toggles (and similar control measures) are adequate and should be used instead of feature branches. If you follow a stratified development approach (e.g., Strata from Vitech for systems engineering), then you can, for example, stub the feature and integrate the stubs with the trunk, then add functionality as the feature is developed.
Working in isolation in a feature branch keeps your unfinished code from being merged into the main branch until its completely done. If you check in code constantly to main, then you will have unfinished code or one big push with all your local commits. Keeping commits local without pushing is a problem because they can get lost if your machine crashes. You should always keep your feature branch in sync with the other developers by locally merging the main branch into your feature branch, atleast once a day. Though these features branches should also be very small. I'd would say that they should only live for maximum a day before it's being merged with main. Otherwise your features are too big and you need to break it down further.
I'll only disagree that a feature that takes more than one day is too big. I know from experience that some things just don't break down that small and the effort to force everything into smaller pieces invariably takes longer than just implementing the bigger feature would have.
Great video Dave. I heard you first on the Changelog podcast, then looked you up here. You've spoken to my years of managing feature branched projects and suffering the inevitable conflicts - even in the very small teams I worked with. It's spot on. I'm running the rebuild of a Drupal project, which will initially be built without any automated tests and all in feature branches. In the very long run I'd like to move towards CI and TDD. One of many questions I have though: Our development so far has gone like this: A developer works in branch, I review the PR and test functionality and pass it over to another non-dev stakeholder who also reviews it against the spec and tests it on a staging set up. In CI, do non-dev stakeholders have to do their testing and reviewing directly on production? In our experience this non-dev stakeholder often catches mistakes where the spec was misunderstood, and we feel strongly that these mistakes should never make it into production. Sorry if that's been addressed already! Thanks for the videos.
My preference is to try an in-line this kind of overview, so that it isn't a "gate" in the development/release process, but is part of the development. PRs and reviews tend to slow the process, so the idea is to do these things continually alongside the development. Instead of performing a code review at the time of a PR, pair-programming gives you a continuous review as well as lots of other benefits. Instead of non-dev stakeholders looking at the code catching mistakes after the code is thought to be finished, build automated testing into the development process so that catches most bugs. Where a review is meant to help sanity check that the change looks as expected, rather than acting as a regression test, then do this early in the dev process as soon as you can. Overall the aim is to build quality into the code from the outset, rather than try to "inspect quality in" to the code later. Thanks for the question, I think I may do a video on QA and it's role in CD teams that may explain some of this stuff.
@@ContinuousDelivery Thanks for the reply! Yes, I could review the code over the dev's shoulder prior to commit I guess, but the functionality that was added has to be tested/reviewed within some non-production environment. So yeah, that is a gate in our process. It seems to me that in CI you have no need of a staging environment. You appear to just have local development environments, and production. So if that's true, then our non-dev stakeholder can only review what is built when it's on production, because they can't use the dev's local environment. For us, the key thing that this person brings to quality is ensuring that what is built matches what was intended by the spec. Automated tests written by a dev that has misunderstood the spec won't catch that sort of mistake, it can only be caught by the person who knows the requirements. Love to hear your thoughts on that!
@@dansearle5994 Not really, the deployment pipeline is a controlled environment, outside of local development environment of an individual dev. It is a shared space where changes are integrated and evaluated. Usually in production-like test environments that mimic prod config and scenarios. These are close to a traditional staging env, but not quite the same. Ideally the person that "knows the requirements" is part of the team too, so close enough to the dev to see the soliution evolve and so catch these kinds of misunderstandings before they progress to the (too late) point of be deemed "finished" by the dev team.
@@ContinuousDelivery ok I see. We're used to reviewing the produced work in our own time, on our own machines (accessing the staging environment), so it's a big shift to go from that to review system where we're with the dev more often, looking at what they're working on on their machine (either in person or via screenshare I imagine) in little bits. It's food for thought, I'll see what I can do, over time!
Interesting, I certainly agree with the principal, but think that using feature branch and PRs is useful. IMO I like to isolate CI from source control as much as possible, so there is one mechanism to trigger a build. Tags. A modular code base helps avoid conflicts more naturally also. Frequent conflicts between independent features is a sign that concerns are not well encapsulated or decoupled.
Branching does allow for an individual developer to create ownership over the feature they are working on. I'm not sure if it's always the "I'm the hero" motivation described in the video. Having pride over the work you do does create motivation to complete it which is important. It's also important to note that individual contribution does have a tendency to create tangents of thought that could become unpredictably useful or not useful. That variation can come in handy. That said, I do find it hard to argue against the idea of more closely collaborating to create better awareness of the application state. Maybe the experimentation and offshooting should just be left to a branch that's not necessarily meant to integrate for experimentation purposes; left to individuals who want to propose changes in structure and such.
I don't think your wrong, it's just that the software industry case been convinced that a feature is the smallest change worth integrating. This allows for all unit tests, functional tests, etc to be done over several commits that then mark this feature complete. We more use feature branches as short term backups then branches. I think where you would make this all one commit and the push it to the trunk, is really equivalent to a squash commit on the feature branch before merging. Also GitHub better supports the feature branch model with PRs, and other expectations of that system. Where would the review go in your CI model?
If you are dealing with a live system, with live customers who may report bugs that need to be fixed ASAP, maintaining an active "golden master" branch is the easiest, safest and fastest way to deal with it - and that means NOT having a bunch of barely tested or reviewed code go into the mainline. And even for an individual developer, I often use feature branches on my own projects, where I'm the only developer, if I want to experiment with different approaches that I may end up discarding later. I really don't get the hate for feature branches. I work with some trainee developers that need more review than others. I don't WANT their code to be continuously integrated - that would actually cause problems. And as for long-lived feature branches creating a growing tech debt/merge risk - just keep rebasing your side branch on whatever's been accepted into master, and you'll never be that far from a merge. Seems pretty obvious to me. To avoid branches and to just have everyone work in the master branch seems like chaos to me, and putting any customers of a live system at risk of not being able to provide them with timely bugfixes.
People hate feature branches because they don't know how to merge and are not competent devs imho and want to pretend they are some code ninja when they are just code hackers. 100% as you say keep rebasing from master on a regular basis. It is not rocket science to pull to keep up to date.
I agree. It's sad many people after gaining experience will look back on this video and say they got ill advice about not branching. What made Git popular is that rebase and merge are easy because branching is a must.
The problems mentioned in this comment seem to be the product of not enough test automation and not utilizing pair programming. Having junior developers work on features by themself in the dark is the reason why you don't want to integrate their code. Have the seniors pair with them instead of waiting for a PR to review.
Thanks, you correctly answered my arguments of pulling every day. I agree CI is the best way and committing to the trunk every day is the best idea. That graphics was useful. Only point is we must have test suite there that covers well.
A) We are pretty much stuck with feature branches on an organizational level - we are trying to keep our branches small and frequent though (still not perfect, but we are improving), in the end, communication is key here whether on the individual feature-branch level or when talking about the "complete" feature and the plans for it B) I fear CI is too straightforward (for a lack of a better word) - you either need really good pairs programming together to avoid silly and/ or hard to find bugs creeping in, especially as less capable developers are just as capable of breaking tests as any other part of the code - you miss that part of separation that a pull request and review gives you; this is personal, but I can better focus on reviewing changes and trying to figure out what they affect when I go into code review for a (small[ish]) PR. This is also true of my own PRs. This, I think, is especially true when your code deals with multiple concepts that you need to mentally context switch from. I'm fully willing to accept that this might be my prejudice talking, since I haven't really been involved in a real CI project thus far. edit: I do adore your videos though, including this one
This is an argument against distributed version control, not just branching. With DVCS, even if no one branches, separately developed code isn’t integrated until it’s all pushed to a shared repository. Merely committing won’t combine anything.
There may be an issue with semantics; TLDR; If we branch off of the trunk, but the branches are so small that several of them are PR'd in a day per person, does this still count as feature branching, or is it CI/CD as you describe? I run a small team of 3, sometimes 4 people, and we never work directly off of the trunk (which is actually our staging branch, but that is another story), as we want it protected, and instead, we make what we call 'feature branches', which is often small break downs of larger features into tasks (so maybe we should call it task branches), small enough to complete at least one, but usually several of in a day. We want the trunk protected as we want to review all the code via PRs before it goes into the trunk. We do this so frequently, that merge conflicts are quite rare, and our CD pipeline is often backlogged near the end of the day. In addition to this, we have small 10 to 30 minute (depending on what needs to be discussed) meetings throughout the day to discuss what we are working on and the direction we are taking, which helps avoid having to re-do a lot of work at PR. This strategy involves, more frequent, but much smaller PR that requires very little comment. Seems to be working great for us, except as the lead I need to work on getting my code reviewed by the other members more often and not just self approve. So basically, does this workflow still count as CI/CD as you describe, because it's still frequent, or is it disqualified because we are not working directly off of trunk. And if it's the latter, is it possible to protect the trunk in any way in this workflow?
This is exactly my situation with my team. We create small task/feature branches that get made into a PR within the day. On a good day I can make as many as 4-5 small branches/PRs. The hard thing though is our code review culture. Because reviews require context-switching and there’s so many PRs that get generated, there’s too much pressure put on the two reviewers that each PR requires (and we don’t practice pair programming). What ends up happening is people only review 1 or 2 PRs early in the day before tuning out of reviews to do their own work until the end of the day and by that time there’s a backlog of more PRs. Curious about the answer to your question and the code review process too.
@@DanielSanchez_dsanc89 The short meetings I mentioned later in the day is where we do our code reviews, which helps contains the context switching to specific, narrow parts of the day, and since we do it verbally and with screen sharing, it goes faster as well, and we do them in batches. And the other thing that I mentioned that helps reduce PRs is showing the code to each other (and hence your reviewer) in a checkup meeting earlier in the day, before any PRs, to pre-empt mistakes before they are made and need to be corrected in a PR. So our meetings are an early morning standup (7:30), where we plan out what we are going to do (but we sometimes skip it if we discussed it at the end of the previous day), then we have an 11 AM code checkup meeting where there are no PRs expected (but sometimes quick tasks get PRs by then), then a 2:30 PM code review session where we are supposed to do all the PRs (but in practice half the time it's another code checkup), then a 4:30 PM staging review session, where we are supposed to see the finished work running in a staging environment (or production with feature flagging if we would not have a staging environment) (but in practice half the time it's code review and seeing the code running on our local dev machines). The goal is to get your code running as close to production as possible on the same day that you are making it. Our client wants a staging environment, so the closest we can do is a staging server, but we do have feature flagging so we can deploy but disable incomplete features to master, so it's seldom far behind.
@@DanielSanchez_dsanc89 Sounds like our situation: the review tasks piled up. We switched from a gated review process to pair programming. We are not pairing for every tasks, but for important ones (with a more or less accurate guess)
@@DanielSanchez_dsanc89 I've found that talking about your goals and ideas before-hand can help reduce the burden when reviewing PRs. The reviews go from "what is being attempted here" to actually evaluating whether the code accomplishes the intended changes (without side effects).
I was struggling to articulate similar thoughts. Short-lived branches for PRs of small, well-focused changes; and being disciplined to always keep master in not just buildable but in a state where it is deployable to production seems like it achieves the desired goal.
One thing confuses me with the “no branch” concept: in order to have a code review prior to merging, we *do need* a branch other than master (or whatever trunk) to be able to create a pull request, right? Or how else a code review would work if everyone is committing to the same branch?
While you could do "trunk based development" (commit and push directly), CI does not necessarily mean no branches. You still might want code review and other QA gates in place.
I have also seen "short-lived" branches basically for code-review and such. So you do all your TBD in this "short lived" branch and merge after code-review/checks
Well, that's the thing. Often, to do trunk-based development as effectively as possible, you don't do a pull request based code review workflow - instead favouring continuous review via pairing or mobbing. That's the ideal. However: there's a difference between a pull request branch and a feature branch, and it's mostly a question of scale. If said branch only lasts for hours, then sure, that's continuous integration. Directly pushing to trunk is just the taking that approach to the max.
I don't do branches. Everything on main. Git tells me when I haven't updated when I try to push. I don't see the point in PRs either. Whenever I see a PR I lack the context around the decisions, I let the pipeline feedback in the form of testing and static scans etc to ensure quality is brought in up front.
Yeah, I don't see how it makes sense at the end of the day without some branching. My understanding about what he's pushing for is just more frequent pull requests
What if one day your product owner decided not to release and undo that one complex feature (i.e. shared library change, db schema update) that everyone else had already integrated deep into the trunk?
This is the reason you need loose coupling in software. Ideally, you'd have some level of abstraction between your code and dependencies, such that you can swap out one dependency for another, one DBMS for another, with the core code blissfully unaware of the change. It's why the Repository layer exists in Android applications.
In solutions of tigthly coupled code, more monolithic, where boundaries are not clear, branching can cause problems and is expensive to manage. Simply because it is more difficult to understand the interactions within the code. However the alternative is much worse. Many arbitrary check-ins mean you may never achieve a stable build. The team just stops working until it is resolved - huge limits to team scaling. In practice you fall back to your local disk as your stable branch! The team don't pull everyone becomes terrified to push. The scope of changes becomes very very small until the code is refactored to allow multiple contributions.
Contrast that to software with boundaries interfaces, services, single responsibilities then it becomes more clear which areas of the code a project, team or individual may touch. You know which component are likely to change for a given project or team due to the nature of the project. Then you get a better understanding of the changes across the various teams or projects and likely clashes. You may get a tendency for teams to focus or own specific components. In this situation, a more modern approach to design, branching makes much more sense. The answer to clashes may again be to refactor code. In this case the branching is easier. At this highly componentised end of the code base spectrum software is typically packaged. Adding tags to NPM or NUGET is like branching. So we also support a sort of branching of packages but in this environment it's designed to be short lived - pre-release labels. Sort of a clash with branches here btw. In all cases a stable version of the code, that a individual can build, is always required. If not you have no development. So a more abstract 'branch' is always present. So the statement "don't branch" is not a practical position at all (techi click bait and you got me). Where are the branches? What are the minimum branches needed? Are more sensible questions. My feeling is the answer to brancing nightmares is usually code refactoring such that branching in whatever form works better.
That's why team members talk with each other. You can't say people don't know what the other is doing. If you work like that then you don't have team, but instead group of individuals or problems with project management. Pushing to master everyday means fixing merge conflicts everyday. Is it best way to deal with this problem? Even better, you can't decide feature one is ready to deploy, but feature two need some changes. How do you do this? Do you remove feature two from master and add it after release? Finally you chose the most corner case you could. Multiple big changes in the same code base at the same time. I don't know on what projects do you work but in my carier I didn't stumble upon that situation too many times. Because as I said at beginning, my team is there on plannings and rise concerns if someone tries to plan features like that. If it's needed you have more possibilities. You can create feature branch for 2 or more people and they work on the same code base but not on main branch. Still I needed this technic like few times in my carier.
I would really like to hear about how folks do code review in trunk based development. Years ago I worked for a place that did trunk based dev without code review. Integration worked well and the test suite was solid but the design of the code sucked because there wasn't a system for reviewing code design and architecture. I am wondering if it is possible to change workflow to accommodate this without having to have every change reviewed? I wonder if docs around feature architecture committed to source control coupled with well planned conversations would make sense? Anyone tried something like that?
In my experience, long lived branches are partially the result of poor planning and lack of design, and a proclamation that integration is difficult because the "complexity" of the feature doesn't lend itself to being worked in pieces. A lot of teams and their developers just want to jump into development of that next feature, rather than first reasoning about how the feature will be integrated, and the design considerations to incrementally build the code.
The issue can be "there is no design" - this is the case in large companies where development is off shored to vendors but responsibility is maintained internally. Management simply want metrics to justify their budgets (which can be huge given the number of people involved).
But then the problem is long-lived branches rather than branching itself. If someone is out on a branch for months, then I can see the problem. If they split off for a couple of weeks because they want to try some radical changes I don't see any problem.
I agree with the principal that maintaining multiple branches can go wrong quickly but branches are not always evil. Here is my scenario. I have a production system pulling from master. I need to make large breaking changes to the API and UI. If i were to do this in the master branch I would lock myself out of maintaining the production system since I can't push or pull anything into production due to all the breaking changes in my code base. This has caused me not to be able to pull any changes into production for months until the dev changes were complete or having to directly modify the code on the production server. Not a spot you want to find yourself in. Having a single dev branch gives me the ability to switch to master, make a hotfix, commit the changes to master, pull to production and then switch back to my dev branch, pull from master to integrate the change and continue development on the features. Not branching in a scenario like this causes a seriously hard to maintain system since you end up having to patch all code changes directly on the server since you have now locked yourself out of your version control system due to all the breaking changes on master. Don't abuse branches but do not avoid them when you need them.
Not looking for an argument here, just wanting to point some things out. CI and TBD is designed to "solve" the problems you describe by exposing them instead of trying to work around them with band aids, processes, phases, hand-offs, etc. I would go so far as to say that the whole point/secret to (agile) software development has been forgotten in-place of process-itis and navel gazing. Just my opinion. Here is a great quote about agile "Agile's biggest strength lies not in solving problems, per se, but rather in exposing your buried problems and making them visible so you can deal with them. For a team switching to agile it can feel like things are getting worse, not better, but that’s often because they’re being made aware of pre-existing problems they didn’t know they had. It may not seem like it but this is actually a good thing." - Someone over 10 years ago. Or in the words of Jez Humble (co-author of continuous delivery) "If it hurts, do it more frequently, and bring the pain forward." Instead of getting bogged down in the thinking that "we can't" or "we must", what if you asked yourself a different question? "What would need to change for master/main to no longer be blocked/deployable for new features, fixes, etc; to be able to do any of these things as needed, when needed, and never be more than a few minutes from being able to deploy to production?" Problem-solving is more about asking the right questions than providing solutions. Adding complicated processes to hide the systemic issues is seldom the answer.
Actually "if it hurts, bring the pain forward" is a quote from our book, so both of us said that, not just Jez 😁😎 Otherwise I agree entirely with you comments. Agile is not so much a recipe for producing great software, as it is a recipe to allow us to spot the problems that prevent us from creating great software so that we can fix them, and so create great software. "Inspect & Adapt"!
So this topic makes a lot of sense in a lot of ways but one issue I see is with how my studio works. We have new programmers and interns from a local college working on projects often. This means that any one of them could break things on a moments notice with CI just due to discipline and skill being not there yet. On the flip side it would force them to pay attention and work more effectively though at a cost. This isn't a finished thought really and context matters but I felt like voicing it can't hurt, all while I continue to think about it.
I think that is a fair point. My prefered way to introduce people to a team is through pair-programming, that instils the discipline and gives the newby a guide to the codebase, working practices and so on.
Branching with small FF trunk based development just works. Especially if you want to do different types of testing that is not similar to your main app (i.e. different frontend with a ephemeral env). This is something that doesn't have to even be defended. It's not either or, we can take parts of what your saying and branch as well. Branching is not guess work though and having everything in 1 branch stops dev testing and exploration. Testing one small change to code is testing more than just that code itself. You have to also test how it interacts with the env you have set up and I don't see how that would work with only 1 branch.
You mention the cost of CI, and IMO that's exactly what it's about: "cost". For many teams, industries, features, and contextual situations, being able to make continuous changes that are "ready to deploy" but not complete is pretty hard. There is little experience or insight on the strategies like feature toggles; branch by abstraction; dark launch; etc. But even if there is experience, every project and team always deals with something new, the team needs to know how to apply those techniques to their current situation, and that is not easy. For those teams, continuous integration is more costly than the pain of merge conflicts and the window of time where devs are "in the dark". This cost is understandable and at least easy to identify, so teams choose it and forego CI For continuous integration to get a bigger going, IMO that cost needs to be reduced, which I think is actually pretty large. In general, there are a lot of features; projects; etc; that don't fall neatly into a continuous integration approach, and they would require a lot of thinking and good design to be able to adapt to CI (with the strategies you mention). Basically, to be able to perform continuous integration you need all of your team to be extremely good at their job, you basically need all of them to be seniors, and good ones at that (at least in those cases). If we can reduce this cost, then teams would naturally adopt CI much more often IMO, otherwise they would ditch it the moment they encounter a big difficulty.
I'm a fan of CI, but I'm having an issue with the idea that a feature branch necessarily means longer times between commits. If I were to run a development team, I would require the same interval between commits regardless ... basically when a small feature is ready. The gain (as I see it) from feature branching is the ability to merge from trunk and see any conflicts or failures before merging to common code.
I had some time to think about your views. This is going to be longer comment, but I really hope I can get an answer from you, because I believe I have some strong arguments. I think you're loading into the idea of 'feature branches' something more, than I do. In particular, I agree completely, that the problem is in how OFTEN developers in the team integrate their work with the trunk. And that the less time they do, the better. I think nobody reasonable challenges this. Where the problem arises is that in fact NOT having feature branches does not prevent developers from diverging their work for too long and vice versa, using feature branches does not automatically mean long time to integrate (which is what I imagine is, what you believe the term 'feature branch' means). With trunk based approach, a developer could conceivably work on one commit for few days and create a humongous commit. In fact, sometimes there's just no way to avoid such commits. They are very rare, but they do come up, for example, if one needs to do large scale refactorings, that affect the whole code base. On the other side of the spectrum, if a developer uses short lived feature branches (say a branch contains maybe 2-3 commits and is merged with day or two after it was started), it poses about the same risk, as trunk based development. This approach of creating short lived branches, that are ideally small (2 or 3 commits), and are merged shortly after they're created (day or two after they're created) has many advantages over trunk based developments, with the primary disadvantage being, that it slightly increases the window for integration, but really only slightly (nothing in the realms of the horror story mentioned in the video). The primary advantage of this type of feature branches is, that there are plenty of tools (Github, Gitlab, Bitbucket, etc), that enable performing code review on a feature branch and this way code that is not optimal is not pushed to the trunk. There's one killer argument, that makes trunk based development completely unviable, unless there are special tools (that are not industry standard such as Github, Gitlab etc); With trunk based development a member of the team could conceivably push a commit, that fails to build. Now every developer that will checkout trunk will be unable to build their local branches. This is a disaster! And no, pair programming will not prevent this from happening. How does feature branch based development prevent it? By running automated build on the feature branch. Not sure about Gitlab and others, but Github can be set up in such a way, that the feature branch must be up to date with trunk changes prior to merge and automated builds have to pass in order to enable merging. This guarantees at least builds that will pass. And hopefully more people can review changes and if runtime errors would break the project at least more people have a chance to figure it out before it actually gets merged to trunk. I think if feature based development is done right (in the manner described above), it must be superior to trunk based development. I have not discussed code review, I know you advocate for pair programming, which honestly still should NOT absolve code from being code reviewed by other members of the team, so I think pair programming and code review complement each other, it's not one or the other. I'm really curious what you have to say about this and in particular how would you avoid problems with accidental push to trunk of a commit that breaks the project in trunk based development.
I don't like the term "Trunk based development" much, I prefer the original term "Continuous Integration" it is more accurate. In CI, it is not possible, by definition, for a developer to work on one commit for a few days. The definition of CI includes "merge to mainline (trunk) at least once per day". If you merge FBs at least once per day, then it can be classed as CI, if not, by definition "not CI". "If FB is done right it must be better than TBD" - The evidence is against you, take a look at the State of DevOps report, "Teams that have FBs that last longer than a day have worse scores in Stability & Throughput". Stability measures the quality of code based on defects per change and mean time to recover from a defect. Throughput measures effciency based on time from commit to release and frequency of release.
@@ContinuousDelivery still, accidents can happen, it is possible to break trunk. I have worked this way before and even with best efforts, sometimes it just happens and in less experienced teams it's going to happen more often. I don't know about the research that you mentioned, I would need to see the methodology first to be able to conclude it's a valid conclusion. I can imagine already many problems with trying to make a valid research that would prove such conclusion.
@@vyli1 I have a question. What does it matter that trunk is broken just once-in-a-while (meaning seldom)? If it's broken so that unit tests, functional tests or acceptance tests fails it will be discovered (and fixed) quickly. If it's broken in such a way that it's only found during manual testing, you would probably have that happen anyway, and sometimes it's broken without us knowing - we call that a production bug. There is this misunderstanding that "always releasable" means "mainline never broken, always pristine, represents production, etc". "Always" is not black and white! While we should strive to minimize the time mainline is broken (because it blocks everyone else) the goal is not to eradicate all mistakes (impossible). The purpose of CI, CD and TBD is fast feedback *IF* we broke something. It's okay if it was due to an honest mistake, if it was due to sloppiness then the team have some difficult learning to do. In both cases we have learned something we can act on. To para-quote Jez Humble: _"Every build is a release candidate. The goal of the CI/CD pipeline is not to prove that it works, it is to prove that it does _*_NOT_*_ work. If you can't prove that it doesn't work you should be able to deploy that version (if you choose to)."_ . As I have written elsewhere, you don't release a branch, you don't deploy a branch, you deploy a good known *commit/version* of your code. If the mainline is broken the same hour that we need to deploy to production - it doesn't matter - because we *always* use/deploy the latest *known* working commit/version. Version control is not a file server and a branch is not a directory, it is a versioned graph of snapshots in time so we can use whichever one we'd like.
@@ddanielsandberg yes, you're right that I'm probably making it much bigger deal than it is. A commit can be reverted. But still it's something that causes inconvenience to the whole team and sometimes it's not necessarily obvious WHICH commit broke the build etc. It is some time lost, but yes, in most cases it's probably not such a big problem. I'd say it depends on the culture of the team on how big of an issue this is. I have other issues with committing directly to trunk, I don't consider pair programming enough of a code review, I want multiple people, in fact ideally all team members, to review the code. That's not only to get feedback, but also so that other devs are aware of what's going on in the project, have the opportunity to learn something, etc. In a team with excellent culture, it is possible to review the code after it was pushed to the trunk, but it's not as ideal, if subpar code gets pushed to the trunk as opposed to that code never reaching the code base at all and a proper version instead being pushed to trunk in the first attempt (which branches allow to do). Or it is possible to share a patch or use some other non-standard tools to request review before pushing to trunk. Those problems are solvable, but they require extra work and really good team culture. I'm sure such teams exist, but by far they'd be in minority. In my team, it's me (team lead) and 5 other junior to mid level devs. Pair programming would not help, because any pair not including me would end up pushing code with problems (I end up catching bugs in the code review quite often, silly mistakes in structure of the code, etc.) I simply don't have trust in my team members that they'd push proper code to trunk. Not because I'm distrustful, but because I know their work, know the limitations of their work and it would end up adding more work for me, if they could just push whatever to trunk before me having any chance to review it.
CD, i'm new to programming and I just wonder, you often say you expect these CI branches to last rought around 15-20 minutes, or a day a most. So my question is: What if the feature I'm refactoring or working on is incomplete? like say we're making a video game, and I decide that the jump mechanic needs work - just doesn't feel good to jump in this game. Well 20 minutes into doing it over, i realize I need to spend some hours researching the subject and how other games did it. Do I commit, or possibly spend a day or two being thorough in my current approach to the jump mechanic? meanwhile, other devs are building level design choices based on the jump mechanic that depends on it being what they thought it was.
I've asked in one of your previous videos about the process of reviewing changes to a code base with continuous integration. I'm a bit confused about what the workflow would look like. I work with a small team, so cashes with merges are far and few between, but the most important part of our workflow is the review process we get when we ask for merge request. What does it looks like? How do you get that feedback from your teammates? Specially since that feedback happens when you got hands around the changes and it's all fresh in your mind, not 3 weeks from now when I moved on and a teammate happens to look at that part of the code.
Yeah, I am in a similar situation. I am working on a small team with currently 2 other developers. We do use feature branching and when we are done, we issue a pr requesting a review from the others. Even, or perhaps especially, in a bigger team I think reviewing code is important. It also helps you seeing what impact changes from others may have to your own branch and knowing what you will need to adapt. With CI, I feel like you might run into situations where you see a conflict and don't know what the other's changes are for and how to adapt them, whereas a more complete view of there changes may help with that. That being said, it is generally obviously easier to merge tiny commits. I should also mention, while we usually have one, and exactly one, bigger feature in development at all times, the other branches are usually pretty small and never last more than a couple days or maybe a week in some cases, so there are never big discrepancies and when working on the bigger feature, it is feasible to pull from master. We work in feature branches on user stories, not epics, and I feel like this video might talk about the idea of developing epics in a feature branch, which I would agree with is not a good idea and would make the review process quite cumbersome as well.
I also have this question. If we want to be continuously ready to deploy, we need to have some limitation on developers just pushing to master. Even if we do pair programming (which is another big change to the development workflow) just those two developers may not be enough to confirm a change is correct. At my work we are trying shorter lived branches. So still basically feature branches, but putting it up for review/feedback/merge before the whole feature is done. Aiming to have branches live a few days at more. It does take more time for reviews, but I think it is worth it for the more frequent feedback and avoiding conflicts.
@@edwardweir5784 Well, the techniques all work together. If you're doing TDD, you should have some confidence that the code is correct. If you're doing pair programming, the code gets reviewed before it goes in. And if you're doing CI, the code goes into the main branch frequently and you don't get surprised by sudden big changes.
You get feedback by communicating with each other. A feature branch in itself won't review your code for you, and CI won't either. If you want to set up a code review process, you can literally do it however you want with CI. Use the method you want and one that works for your team. It's that simple.
@@markjreed even if you do TDD, that still can mean, that a developer doesn't write a test together with the new feature, so the new code could be incorrect, or the test itself could be incorrect.
Linus created git to protect his kernel from (insert your preferred adjective here) jrs and Dave says to accept them daily. I think you two need to get a coffee together.
I wish people would stop comparing every project with the Linux kernel. It's vanishingly unlikely most people are going to do anything like that - and yet the entire industry is out there gawping at Linus and copying everything he does as though it's gospel. 1) What he does is not the only way. 2) Linus is not a team player. He is a successful asshole that gets away with being an bully by having a successful, widely-used project. Most people need a team - with team players on it not assholes. 3) Linus can actually be wrong about things. 4) There's a big difference between working on a well-integrated team of trusted people and being a Linux kernel maestro assisted by a team of trusted lieutenants taking patches from thousands of the great unwashed of the world. There is actually a cost in terms of time and attention in serving as an open source project - that isn't necessarily appropriate even for most things.
But I do have a question. If I have to create features, that can take more than one day, I must create a new branch, and create a Merge Request, right? Or even if my feature takes more than 2 days, should I merge it into master without review from other developers? I agree with CI, but I have some doubts about large features and hotfixes.
The client I'm working for they have a controlled feature release, so constant merging seems impossible for this case. We only merge if the feature is business approve else we don't. Is CI can be implemented with this case?
I think I can see the point you're making, but I don't think not branching is the way to go if you'd like to be able to always deploy the head of your main branch (master of development). In mature projects there can be huge CI pipelines, which are not desirable to run locally. If this pipeline determines I introduced a performance problem in the code for example, I would not want to have this in my main branch. Instead I'd want to fix it in my own little sandbox (my feature branch) before merging. In companies where I worked, the burden of proof was always with the person trying to merge new code. Thus, you merge the main branch into your feature branch regularly and you fix any problems that are introduced (in your feature branch, not in the main branch) before merging your branch. Besides that, I'd recon this is why we have all those scrum rituals to inspect and adapt.
What about things like ROS where you have different distributions that behave differently? Typically, some packages will have different versions on different branches depending on your distro.
Where I work we use feature branches.. one thing that happens from time to time is to remove a particular feature branch from a release candidate... good luck doing that if you constantly committing to master.
Well it is easy enough if the design is nice and modular. Version control doesn't fix this problem, because if you need to merge your changes with other peoples, you still have to disentangle them to revert. This is much more about good design than it is about VCS strategy.
@@ContinuousDelivery Appreciate the reply! Find your videos challenging but I am trying to keep an open mind. I'd guess what I am doing according to TDD is an antipattern. I rebase all of my commits in my feature branch into a single or fewer commits instead of filling git history with 100 commits for a single feature. I feel like I'm dong part of TDD as far as starting with the test cases first.. I guess I'm trying to wrap my mind around how to implement TDD fully and thinking of the use cases that I do go thru during the release process. For example merge hell for feature branches that are really old. .that is one major drawback. Thanks for the videos!
For example with feature branches.. if my boss decides for whatever reason to remove a feature from the release.. we can create a RC branch without merging that feature branch.. if everything is always committed to master.. that is going to be a lot of work. If things go really bad.. I can just destroy the RC branch and try again (leaving out any feature branch) from that RC branch. I guess that is my point.. thanks again!
What you left out, is that for me doing CI (as you describe it) offers a "normal" team, to become a much better team. Yes many developers think they are the best in the world, yet not everyone can be "the best in the world" and by doing CI with "not the best developers in the world" you get results that out perform much better developers that are branching. And not just once, all the time (because you don't have to spend time on the branching part)
make sense to me, but I have this question, I have worked in real big companies with hundres of developers around the world, 24/7 coding and adding stuff, using the branching approach, some devs (no me :P) sometimes pushes their broken branch to master, blocking the rest of the team, with CI that will happen too, and even more often is my suposition, Thanks for the video.
in providers like Github and Bitbucket there exists settings and controls to ensure that tests pass after merging. there's also a practice for the team to fix a broken build immediately if it fails for example see GitHub.com -> Administering a repository -> Defining the mergeability of pull requests -> About protected branches
What about the argument of atomic commits? Should every push to trunk be guarded by tests? If so, wouldn’t the scope of tests dictate the frequency of integration? Is it therefore better to base integration on a set time frame, or on set scopes? How would you determine scope?
Yes, every commit should be covered by tests. Yes that means that a very effective approach is to allow tests to dictate the commit frequency. If you practice TDD, as I do, that means that my commit frequency is usually roughly once every 10 or 15 minutes. That is pretty close to "continuous" in a practical sense.
I am often a single developer. I feature branch even when alone because in git there is often little to no overhead in doing so, the tooling for it is excellent. And most importantly, when I feature branch I feel more confident making larger changes. You want to be able to commit often so you can reasonably bisect. But you don't necessarily want to commit often on master because of the hassle of cleaning it up. As an IT professional, I am not sure how you live without PRs.
Ok, fair enough, feature branches are cheap in Git. But they cost your effort to create them and to delete them after they are completed. And the benefit is IMHO none, the same can be achieved with normal series of local Git commits, and an eventual rebase (with cleaning up the commit history) before the push. You are isolated on the same level during these series of local commits as you would be in a feature branch.
@@miletacekovic I have no idea what you are talking about. It takes literally 5 seconds to create them, and they are deleted automatically by default when you merge them in most modern environments, like Azure DevOps, Github, etc. It seems like you do not work in an environment where PRs are mandatory and for a good reason. You can also decide to squash by default, but that makes it harder to bisect. "Cleaning up commit history" makes to harder to find bugs. Also, integrating main/master/develop is a single click in most modern environments.
Single developer why do you even need to create feature branch? Doesn't make any sense. Why most people create feature branch? Because they're working with other people and the need to review others code before allowing them to merge. In short, there're lack of trust and this creates more work, process or things to do before a work can be done = delay, less efficient. In the past, before version control or for companies that didn't uses version control, they had to integrate the code manually, and devs continuously find ways to make the integration easier by doing frequent code merge, tries to make the merge easier (e.g. new functions at the bottom), refactoring existing code to communicate constantly to ensure the team get the latest changes immediately and avoid conflict. In the past, it's also very common for multiple developers to work on a single file with thousands lines of code and there are no version control yet they can manage the integration efficiently. It's all because they had the same goal, to reduce conflict and make their code always up to date and they do this by constantly communicating and integrating. Somehow, with branching, are we actually working better? Or it created more delays, more conflicts, more failures (since it's my branch, I feel safe to make mistakes or break the build in my branch), delaying review because I need time to review a huge commit and I need to make sure the other developer are doing it correctly before I would approve the PR.
@@Gaming214-y3g It takes literally 15 seconds to make a feature branch. If I am going to make large changes to a project where I want to commit during the process, I obviously make a feature-branch even if I am just one person. Integrating "main" or "master" or "develop" or whatever you are considering merging to is extremely simple in most cases if you do it often enough. Are you trying to say that if you were to, say, move a game from Box 2D physics engine to the Unity internal physics engine, you would do that on the main branch? What if, in the process, you found out that there is an issue with the internal engine? If you have not committed to main because it either doesn't compile yet or it doesn't meet your expectations, you are basically coding without version control. If you have committed constantly to the main branch, you have a messed up main branch, will need to hunt in tags/commit history to find something that builds and you will get the ugliest and least sensible commit-history possible. Branches just fix all of this and that is what they are there for. I will probably never understand why (some) people are against them.
@@larsthomasdenstad9082 what you said, doesn't make any sense? By having branch, you're saying it's ok to make mistakes? Do you mean, you won't merge to master and later found out there's a problem? This is not the purpose feature branching was created for, but having feature branching , it created lots of bad habits and quality that were not desirable for developers, e.g. not ensuring your code is working before pushing, delaying integration, keeping bully large commits, having the safety aka 'insurance' that is your own branch so that you don't break master. Question: how many single developers actually use feature branching? When feature branching is needed? When you have open source and someone would like to contribute a change, they will create a PR. You can't just let anyone merge to master. In a company, where you work in team, you'll eventually find out that the feature branch is just some safety net to prevent someone to merge because... Of trust, confidence in others... But what if they make a mistake? Or so what? Fix it. The idea is to become a more mature team, learn from mistakes and become a better programmer. For single developer, there's no reason you need branching. By having branching, it creates bad habits, the problem you mention, it's not because of branching, to be honest. It's hard to accept this, I've work with most developers who argues the need for branching but the truth is, the problem is the way they work, the habits and it's hard to them to acknowledge them because everyone wants to be right and entitled to their own opinion. We're all adults, we should think for ourselves, ask ourselves questions, look at the problem and how to improve and get better.
You are correct. I was hired to manage and fix this exact situation. Clashes between developers was causing awful customer experience. Since a feature can take 10 minutes or 4 days, this WILL happen and more often than we would like. Thanks for the great video. A. Amburn, PhD
Hello, I get the idea and I don't disagree. I think we'd need some detailed examples though, about how to do it. The rationale seems as correct as it is high level. The practical level is, I believe, what keeps many doubtful.
Not merging for weeks is not called feature branching that's just working on a different project.
not really.
Some things just take very long and even as a CI enthusiast I have to admit that long living branches can have a purpose
What about merging other branches into yours once on a while to test the diversion and revert the merge after that?
@@OggerFN Then your features are too big and needs to be broken down into smaller tasks. A feature branch should only live for a day.
@@PelFox
Some things can't be broken down so you either implement it unfinished and feature toggle it or branch until it's finished.
@@PelFox a day is a bit on the short side but it definitely shouldn't last longer than a sprint
branching is like time travel, it's sometimes helpful but if you stay away for a long time the consequences are unpredictable
🤣🤣🤣👌👌👌👌
And Dave Farley said not to branch and there was no branching. And Dave Farley saw no branches, and it was good.
If you don't think a single developer working alone can benefit from feature-branching, I don't think you and I are talking about the same thing when we say the word "branch", which would certainly explain the discrepancy.
a "branch" collects units of work (commits) together. Each commit is the smallest change which works on its own. Each branch is the smallest change which justifies its own existence.
Within a branch, prior to integration, commits are not a commitment - they are rewritten / re-ordered regularly, as the discovery process continues. Integrating them doesn't make sense, because they aren't final yet. Forcing integration prior to the change being done makes the history unreadable because each change becomes separated from its justification. And of course, the one and only reason we're even using version control (instead of just sending each-other diffs against the last timestamped snapshot) is to keep track of *why* changes happen.
I think we both agree that integration needs to happen very regularly, but disagree strongly about the acceptable scope of both feature branches and commits. I don't want to be "more continuous" than the smallest justifiable unit.
I also disagree that "a branch which can't be integrated" is the same as "lost work", and I would have thought (based on your other video) that you would be a strong advocate against that sort of position. Sometimes code needs to be rewritten based on new information, and when that happens, it's always faster than writing the original code.
It really sounds like you're talking about traditional, long-lived branches, rather than feature branches.
As with so many discussions encountered on the Internet, I'm pretty sure I agree with everything you say, except for the specific terminology.
the changes aren't separated from their justification if you know how to write commit messages.
@@OggerFN their individual justification, yes, but not their overall justification. Not the context of which commits relate to it, in order to define the goal. If your commit is fully justified without any related commits, go ahead and merge that. If whoever is reviewing could ever ask "why?", then it should probably be grouped into a branch.
Meanwhile, if whoever is reviewing could never ask "why?" for any of your individual commits, then your commits are too large.
@@________w
You should write the task the commit is supposed to accomplish AND the reason for that in a commit message.
For example a habit I have been using is naming the ticket a commit is relating to in it's message in order to "group" commits.
@@OggerFN You can also just squash everything into a single commit once you have some sort of logical unit related to your feature.
@@mAcCoLo666
Yes if you care enough to do that
Once more insightful thinking of how sw can better made.
Thanks Dave Farley.
Since this is a so difficult practice to get the teams and/or Devs to move to, if possible I would love a future video of how to start with it.
Something like what needs to be done from the beginning, and a possible adherence to other techniques, etc.
Thanks in advance.
Yay! Finally I got an excuse for committing directly into master: it's called CI
You must still pass the integration pipeline, though.^^
Love the sarcasm! And also, I agree!
It's perfectly acceptable to use many different strategies and CI and Feature branching aren't mutually exclusive. You can do feature branching in one step of your process and CI in a later stage. No worries.
Honestly, this video is too dogmatic.
As someone pointed out...only if you get through the pipeline. And that should have human beings in it after the automated checks. :-) I mean..i heard about QA Somewhere......
@@timothyblazer1749 I don't think that this pipeline as described in the video includes human beings. (EDIT: You wouldn't be able to integrate that often if QA would look at every integration.)
@@timothyblazer1749 so the branch is still needed for a code review and QA? Wait, that's a feature branch!
I recently retired after 47 years in the field, slowly trending from coding to writing tools to version control to configuration management to as close as I could get to continuous delivery, which wasn't very close because I could not convince my teams of some key points. I am hugely enjoying your videos and your explanations, and thought I'd share a few observations from my time in the field.
1) I haven't heard you say this actual phrase, but some major luminary very early in the software world's development stated "Nothing of any significance has ever been accomplished by a team of more than 7". I've heard you express much the same idea, though, quite a few times. More people than that exponentially increases the chances of miscommunication about the design of the system.
2) I used to sort of half-heartedly suggest feature branches for extensive, disruptive, dangerous changes, but my teams always resisted it hard for the reasons that you clarify in this and the other videos about "don't feature branch"; essentially, merge hell will invariably result.
3) I worked in about 9 different shops that all claimed to be doing Agile Development. In one of them, they were actually doing Agile Development. In the others, they were indulging in rituals designed to fire off the buzz words managers wanted to hear, but these rituals usually were bumbled so badly they didn't do much good and in some cases were actually harmful, like when the boss was the scrum master without training and did most of the talking.
I'm interested in what your opinion is on Code Reviewing practices, in light with your strong preference for CI workflow. Some companies I've worked at, have a pre-merge code-review process like Github's PR system that needs a review before being merged into trunk. A pre-merge code-review process or mindset would then only work in an environment that considers a pair-programmed changeset as properly reviewed? Would you consider a pre-merge code-review policy to be an impediment towards incorporating CI?
In companies that have a post-merge code-review process this problem doesn't seem to exist, but one would probably require an adequate level of (non)functional coverage in their automated tests to catch mistakes quickly, such that code-review is only needing to (ideally) focus on design. Are you therefore of the oppinion that a post-merge code-review process (or pairing as mention before) are two key processes to have in place for enabling CI?
This is a topic I see come up frequently, so I hope he does a video about it. If I remember the topic correctly from the Accelerate-book, as much as possible from the pull requests should be automated, leaving stuff like exploratory testing and such to the humans. And for the pull requests that remain, the team should make code reviews a high priority to ensure flow
Same question as I wanted to pose as from my experience the code review part is the one that causes your branch to stay non-merged longer than the design and implementation parts put together and even multiplied.
For post-merge code reviews I don't see how they would work as they require far more experienced developers to go through (specially with continuously pushing unfinished work that makes the same mess as merge instead of rebase does in git... nightmare for reviewing so you just rubber stamp it and be done with it).
With partially complete pushes you also require far more experienced developers as they must understand that something is still work in progress and somehow still envision the final design.
And in my experience experienced and high quality developers are hard to come by so I still can't imagine getting higher quality code from CI compared to shorter tasks, rebasing and serious reviews of the design. Short term faster... Yes. Long term more maintainable... Can't imagine how.
He has made a video in the past about preferring pair programming, and seeing it as valid continuous code review.
@@defeqel6537 I've seen that video and it might be just me but I don't see pair programming as replacement for code reviews (at least not those that smell a bit of formal code reviews) - to be honest I can't even imagine somebody constantly messing up my train of thoughts in that manner...
@@DerDoMeN Yeah, I'm not sure I agree with him on that, but lacking the experience, I couldn't really say.
One of the issues I have around CI vs Feature branching is more related to maturity. As far as I can tell, it works great when you have a stream aligned team working with a code base where most of the work happens in parallel. When you have a monolith, the problem with CI (for me) is that integration events become unpredictable. One team may be testing a set of changes, when another team commits a change that is automatically deployed. All of the tests cases that had been performed at have been invalidated as a result of the change. That's primarily why we've used feature branches in the past, to isolate changes just long enough to test them.
Feature flags partially solve most of those issues. That said, I still prefer short-lived feature branches as I don't like tons of feature flags.
@@michaelslattery3050 Does it not get somewhat tricky to understand the subjective concept of 'short-lived' here? How would you define short-lived? How would you enforce it? I have seen teams that aspire to that and somehow these short-lived branches suddenly become very long-lived.
This 🙌
Hey who doesn't like the smell of a merge conflict in the morning standup meeting? 😜🤔
Clearly not all the *--ours* fans out there. Excellent coworkers, the lot of them.
If the conflict is only coming up in a meeting, you don't have good communications or process. Those should be being resolved individually by the actors involved, for the most part.
Just wait until a "certified master" that doesn't know anything about development starts hassling you about why "can't you just merge it in".... and to add insult to injury said person usually doesn't even posess enough of an understanding of the concepts involved to be able to understand a straightwordward explanation... whilst simultaneously they're looking down on you for being a "boffin".
@@dafyddrees2287 what you describe is some toxic work environment. In a healthy functional team this is not a concern at all. I wouldn't work in a team that would act the way you describe.
@@vyli1 "healthy functional team this is not a concern at all." That's a very subjective thing to say. All the TBD I've worked on has been awesome and all the branch-tastic stuff has been awful. It corroborates what Dave has been saying in his books and videos.
- "When the information is modified in each place, then the long the time, the more those copies will diverge from each other".
Well that is the description of entropy, it is so fascinating to understand that basic properties in universe affect everything even how we commit ...
Dave, I agree with what you say on so many levels. I am one of the people commenting on the other video. I would like to also refine my view. Please keep in mind, what I don't say explicitly is because I think we see things the same way.
Let's recap what was the main issue in the other video's comments: the beginners of the team. Beginners are a healthy factor in a team for other reasons than the quality of the code.
Technically, we tackle beginners by reviewing their code. Another reason to review the code is compliance. Either way, beginners will make mistakes that cannot be caught automatically, no matter how good your tests are.
Let's add one more to the list of things to keep in mind: we fully agree that things should be integrated as often as possible.
What we don't agree on is doing it at every commit.
This is literally insane, for the reasons I hinted above.
What we do instead is redefine the notion of a "feature". It's not an user story, but a subtask.
Let's take an example. The junior on my team has been working for almost two months on a user story which ended up having about 30 subtasks. So on average, he finished (and integrated) one feature every 1-2 days. The reality is, some features (subtasks) were easy even for him, so some features were half a day, while a few took him as long as a week.
I am fine with this, because we reduced the risk of breaking master, while also reducing the waste of effort (it was the beginner tinkering, so 1 week does not cost that much).
Keep in mind, most of the time we were integrating every 1-2 days.
We can also integrate all combinations of feature branches automatically and test them because since they're short-lived, there's less of them! We remove the branches after integration.
So roughly we have only as many branches as the number of people in the team.
Integrating at every commit would work only in a team of only seniors.
Not so. I have operated this way with junior devs, even raw beginners, many times, but you do it in combination with pair-programming. It is the fastest way to get juniors up to speed and productive, that I have seen.
@@ContinuousDelivery pair programming for 2 months continuously is not productive. They need to chew over problems themselves, sleep over them, then come back the next day and solve them.
Yes, we paired up, and in addition to that he asks for help every 2-3 hours. I found this works best, in addition to pair programming.
Also, the other big issue raised by me and others is code review, which you have not addressed so far.
What is it then, reviewing each commit?
Interesting. We used feature branches at first - it is hard. Now we embraced, trunk based development and release branches. Release branches allow us to keep CI intact and enables patching. For new features, finished or not - we use feature toggles. Feature toggles allow several things: review partially implemented UX, understand complexity of database migration requirements. We also starting to embrace side-by-side approach vCurrent-by-vNext. You may say it is a lot of maintenance, on the other hand you are doing it anyways. Now we have much less feature branches, always up CI/CD, easier database CI/CD.
Love the videos.
Would like to see you touch code reviews topic. Often it is seen as one person offers ideas and knowledge sharing on optimizations or better code structure. But I think that it is more important to review how people handle exceptions, how people ensure that APIs are complete. Not all devs are experienced. I mean code reviews are not about beauty, it is about cleaning sh** left by others. This is more realistic and gives more value. Code reviews is about getting more experienced, not just about learning more patterns or algorithms.
Did you completely get rid of branches and started to push directly to trunk?
I think big take away we need to remember is: “Continuos integration works by limiting the time period and thus limiting the fan out of the cone of changes, this reduces the risk of a clash”. That makes me wonder, if we use feature branches but could somehow guarantee the time period of unchecked code is the same or less as not using feature branches and forbid branching of feature branches to avoid the cone, are feature branches still bad?
That's precisely what you should do. In fact, this approach has many added benefits, such as you can increase your confidence, that no bad code will be pushed to trunk. With trunk based development, nobody can prevent a developer to accidentally push a commit that fails to build or breaks the project completely.
Depends on how would you guarantee that. If you have only 1 "big" feature branch and everything else goes to master, then you can actually keep the system integrated just fine (or systems to be precise - one on master, and one in the feature branch) just by merging master to that feature branch regularly.
This has the downside of someone making a change in master that breaks you feature branch, but since that change is small, you can easily fix it in your feature branch after you pull that change from master.
But what if you have 2 feature branches that take long to finish? If you merge them between each other (as well as pulling from master from both), then you have still all systems integrated AND keep the advantage of neighter branch "polluting" master until it is finished. But now you cannot release one branch without the other.
One solution is to only merge those 2 branches "one way" (say, only merge A to B), but then B cannot be released without also releasing A. Also this strategy doesn't work for multiple unrelated branches.
So you are left with 3 options:
1) don't work on the same code: when a big branch is ongoing, do not change the same codebase / features as this branch is changing. That way, you will avoid conflicts when merging it.
2) bite the bullet: just give up and prepare to do a massive merge once in a while
3) make smaller changes: by integrated smaller changes, you reduce improve in both above points - you are less likely to "hit" the same code as another change if those changes are small, AND when you do, the conflict is going to be small too.
Of course, making smaller changes isn't as easy when you are not used to it and requires a complete paradigm shift in how you look at programming, especially if you want to go "i will work on this feature for 3 weeks and rebase my commit as I want and it only has to work at the end" to "I make a change and push it as working commit every 10 minutes", it will not happen over night.
3:20 I often feature branch when I'm alone. That gives me the benefit of switching features when I get stuck and I can come back later. You may argue that I should comment the changes out or implement a software switch when rewriting a function. That can indeed be a solution and I admit that I haven't tried it yet.
When working alone, I prefer to push every change immediately to Production, potentially toggled if it's still work in progress that users shouldn't see. The only way I can know whether my feature works as intended is to get real usage on it.
Of course people's philosophies differ.
And that is why I enjoy the whole microservice hype in web development. It allows you to work in a small team. This way, you can have feature branches without worrying too much, given the tasks are also correctly planned in a way where the conflicts chance is very low. For such teams I find it very easy and enjoyable to have even bigger PRs, as they contain the full feature and you can observe all of the changes at once, giving you the opportunity to find more optimization routes.
Hmm, I'm still not 100% sold... Sometimes CI sounds like a really great idea, but the problem also is what do I do when I approach larger refactorings (like updating dependencies to a new major with breaking changes) that take time and will leave the project in an unworkable state until completely resolved or changes that we are not sure if we should carry them through... How do really revert a series of commits in a CI world, where they are mixed in with other people's stuff.
That's a really good point. I think CI as described is good for actual user-facing feature development, but you need a more nuanced approach when doing things like what you're describing. The platform upgrade is the obvious exception, but there are other types of changes that require a careful decision on a case-by-case basis. "Changes that we are not sure if we should carry them through" is an interesting one.
Say for example you want to unify how your application does caching, and you need to audit every single network call it makes to update it to the new system. If you take the time to update each one before sending it out for review, and then (hopefully) merging, what's the likelihood of running into some catastrophic merge conflict? Seems pretty high. Maybe a smaller proof of concept would be more helpful then.
But then say your team does dozens of such experiments, starting on them, but never actually following through to implementing them project-wide? Each experiment you run creates ambiguity in the codebase, where it's no longer clear just from looking at the code what the right thing to do is. In this case, maybe it really would be better to make a long-term branch, knowing full well that lots of code will have to be essentially rewritten multiple times over the course of the project. The new pattern isn't the deliverable in that case, but rather the consistency.
I guess I'm not seeing easy answers either way.
I actually think continuous integration solves this far better than feature branches.
If you wait for your code to rot into a state where a "large all-encompassing refactoring" is needed, you've already failed. Clean up your code before committing it and you won't have this problem.
@@rosstempo3765 great if you have that choice. If you're brought in on a legacy code base, you need other tools at your disposal besides time travel.
@@froobly agree with Ross. even with legacy code, the first step is to write tests - both at a "unit" level and black box level (won't go into details). Make the constraints of the behavior of the code explicit, refactoring can definitely be done step by step in incremental and relatively risky way. if you are dealing with huge migration like breaking framework changes, then I would agree CI doesn't really apply, because it is a different problem. but you still cannot get away with automated tests.
Day one: Checkout a new feature branch from develop.
Day two: Merge said feature branch into develop. Rebase first if needed (if there are any conflicts).
Day three: Merge develop into prod or staging or whatever deployment pipeline you have set up.
Day four: Repeat.
Feature branching and CI aren't mutually exclusive. You can do both. And I think you should.
Keeping features small is just a sane thing to do and has nothing to do with CI or feature branching.
If you can't keep the feature branch small, you should rebase it continuously (against the develop branch) until it's ready to merge.
So, do feature branches against the develop-branch and CI/CD between develop and master with whatever steps in between you want.
Ofc there's also other ways of doing it. Don't be dogmatic. Be flexible and adapt to needs and the preferences of your team.
I like this approach, but I would say you can also automate "merge dev into master":
Day 1: Checkout a new feature branch from dev.
Day 2.0: Merge said feature branch into dev. [you don't really need rebase, you can fix merging conflicts without it]
Day 2.1: Automatically run tests using the lastest dev branch.
(optional) Day 2.2: If all tests pass, automatically merge onto master (CI) (or merge manually)
(after merging to master) Day 2.3: Run automated tests using master branch
And of course, 2.2 could be done manually, allowing lets say a PR request or a code review to be made before it is merged.
As far as I understand, this is essentially the same thing as trunk based development. You could skip the "check out a new feature branch" step and get the same result.
I get a lot of pleasure from watching these videos!
The reactive system one has blown my mind and really got me thinking.
My favorite story concerning this topic was having a dev lead who had no concept of how to merge using git. The dev lead was responsible for insuring the integration branch was kept in order. Additionally the dev lead had to coordinate this for multiple pods. The majority of our pod was able to commit and merge to the integration branch on our own. The dev lead insisted on doing the merge for the more junior devs. I noticed that one feature branch worked on by a junior dev was spanning multiple sprints. I made sure to let the dev lead know that having that branch that long was going to be trouble. I was ignored and you can all guess what happened...The dev lead overwrote a lot of changes from the others. To end the story, that dev lead no longer works at the company.
What about code reviews before closing an issue/feature?
Nope - that's not continuously integrated is it? The risk is in the merges - and not understanding what other people are changing. If you wait until they've finished "their" feature (big problem already there with code ownership) then you're merging great big lumps of changes - which is asking for trouble.
@@dafyddrees2287 that’s why you write automated tests
@@joek4563 Great - now you have a broken build with tons of shit to track down. It will be much more difficult to sort out because you will be reconciling your finished work against the latest. Merges of huge amounts of changes suck absolute balls to sort out. It's much easier if you have to reconcile your tiny change against somebody else's tiny change.
@@joek4563 I've been doing this for almost twenty years now and the difference between trunk-based dev and this gitflow bullshit is massive. You just can't do big system-level refactorings at all if you're going to have lots of branches. You'd either break everybody else or never be able to work fast enough to deal with all the conflicts you'd get all the time. The only option is to work WITH other people - not to temporarily hide away from them.
@@dafyddrees2287 So code reviews are incompatible with CI?
How to use trunk-based development in a legacy system where there is huge technical debt and most of the code doesn't have good tests we can trust? How to ensure quality where each commit to the trunk can potentially break the system.
Of course, we try to write more tests and do refactoring doing some new feature, but writing tests is not always possible when everything is coupled together.
Unfortunately you're in a difficult situation so you have to make moves to reduce the number of branches... Of course rewriting is one way of doing that, as nuclear as that may seem. There is no theoretical limit to how hard to maintain a system can become, but at some practical level it can become unmaintainable. All the best
Can you please further discuss the application of this in the context of open source software. Thanks
Yes, like with pull requests (aka "feature branches") and code review in general
With open source software, you do three merge/pull requests. If and when they are successful, these qualify you to be a developer. As a member of the development team, continuous integration is the way to go.
Note: The size of the project does not matter. Whether we are talking about that upcoming JavaScript framework or the Linux kernel, three contributions are enough, roughly.
Continuous Integration is a practice that requires a high trust environment-where you can trust the people on your team. In low-trust environments, like Open Source development, Pull Requests shine. GitHub was largely founded on Open Source, so it makes sense that Pull Requests were largely invented by GitHub. CI would only be appropriate for a small core team in Open Source.
@@djason_music CI can, and should be used in both cases.
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period.
No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on.
Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others.
Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around.
The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models.
The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
I’m trying to promote the term “reluctant integration” (RI) for those techniques that involve maintaining lots of branches...
I like to call it
'severe heap integration technique'
as a mnemonic acronym. :)
I've been using "continuous disintegration"
wishful thinking integration procrastination - trying to reduce risk by delaying integration rather than meeting them head on
I think feature branching within a single repository can work, as they are visible to the team, but you need to treat each branch sort of like it's a trunk and regularly merge branches into other branches, not just mainline trunk. The advantage of this is if you do get some nasty merge conflicts you can put them off, let those branches diverge a little while you consult with your team about how to resolve them and prevent them in the future, all while continuing to make forward progress. Some of that progress may end up being discarded, but some of it won't, so overall you aren't completely halted by having to deal with the divergence.
My expierienced rule of thumb: smaller skilled good teamworking may start & come quit far with a strong disciplined checkin strategy directly on *the main* trunk/master branch, but those conditions tends to be not realiable with much more growing/larger projects teams especially including unskilled/newbies.
Fitting VCS checkin & branch strategies depend of multiple aspects:
a) project/task change-sizes & risks/fail costs (e.g.: not every deep re-design/re-strucure checked in as lots of tiny checkins but project-wide atmoic commit/submit),
b) developers team's size, skills and discipline/motivation,
c) surrounding IT & Development Infrastructure including build tool chain support, dedicated DevOps, Q&A and a good project (not directory) scaled merge/integration tool support (bare git is not enough)!
Wow, 368 comments in a few days. You channel sure has grown, that's great to see.
I think the context of what kind of team and what kind of product you are working with will affect whether this is effective.
When you have many disparate repositories of code, each with few changes over time, it makes more sense to use pull requests, since things move so lethargically.
But if you spend most of your time in one repository, CI seems like a much better option. For example, you won't have to re-synchronize your feature branches to ensure you're on latest if you are always working off trunk.
In general I'd agree, but in some cases it still seems useful to branch to me. Especially when doing some more experimental stuff, where you don't know, if you even want to add this change.
But if you already know, you're going to merge this branch/feature sooner or later, it's always better to merge it now.
It's much worse, when others already depend on a feature, which often changes, than when you are not able to merge this feature anymore.
And you will always be able to merge your feature in most cases.
It seems to me that you need to watch the first video and use one of the techniques for deploying without releasing. What I hear when you say "feature branching is useful" is "I'm not good at deploy-without-release". I think you should get better at that and see if the tools, techniques, and paradigms you develop change your mind about feature branches over time. If you always fall back to feature branches, you'll never get good.
I am working in a big software company. 6 years ago we had big problems with pushing features from the different teams back to master because of the changes by other teams as you explained.
Now nothing changed except two things:
1) We must push any change as soon as possible back to master. Even if the feature is not completed if a pice of code, e.g. on function, is completed so it can be pushed to master without coder breaking it should be pushed.
2) Daily build
We do not have any integration problems anymore. Because the discipline of the developers increased because they do not want to break a daily build code quality and efficiency increased. And the "old" developer who knows the "old times" saying that work is now easier.
Also when working alone, branching is sometimes useful. I've already done it a few times.
Yes, I actually do it regularly. I often have between two and 4 branches in my personal projects.
So. How do you get the branch through QA?
Oh...that happens after you merge? What about merge conflicts? How long to resolve those?
We know that feature branching is convenient for developers. But that's precisely why it is bad for development. CI is far better an approach overall.
When working alone, branching is only a way to mark where you're going to come back to if your experiment is a flop.
@@timothyblazer1749 In most VCS merges are more complicated than they need to be.
Git for example always needs a merge commit if you merge something which is not just a fast forward.
In Pijul, which is still in development, you only need a merge commit if two changes change the same line of the same file.
And merge conflicts should appear less often anyway.
But that's more of a problem when doing CI. When everyone is pushing on the same branch, up to every second commit is a merge.
At the company I work for, we implement CI/CD, but instead of master, we work on the UAT branch. Then once a week, we do a pull request to merge UAT into master. That way our production environment only gets clean and tested code, and hasn't crashed due to broken code in over a year now.
This is pretty much close to how I always worked. I usually work on the master branch and push/commit regularly. Pretty much any change that brings a benefit while not breaking anything.
I use feature branches for bigger changes that take longer, but I can't check in the intermediate work because it might break the existing state. It would not compile or break features. Usually a bigger refactoring. Where I want the intermediate states in the VCS but not break the master. As soon as possible I'd merge the feature branch into Master and continue working on it there.
In some cases I combined it with the Dark Launch or Feature Toggles. Minimizing the time I work on a separate code base.
Funnily enough, I rarely have merge conflicts as I usually merge Master back into my feature. My colleagues kept longer running branches and had to spend much more time on the merges. Mostly because they rarely pulled Master.
Thank you for making the CD concepts presented clear and concise, that aspect pulled me in from busy, busy work. And much to my benefit :)
Thanks, glad to be of help 😎
Added costs of feature toggles is not always worth it. It takes a lot of upfront work to design and plan work on features so that they wont clash with each other. If not managed carefully, it can end up in really messed up spaghetti code. Especially in early stages of projects, when no one knows what they are really doing.
Not only that, there are data model changes that simply cannot live together with the legacy model. What kind of toggle allows you to have a table column value unique in a place of the code and allows multiple values in other? Answer: none that realistically enforces the constraint. I’like to hear how his version of CI deals with that.
What about changes that can't be completed in one commit and also can't really use tactics like Dark launching.
In one of my pervious projects we had to make significant changes to multiple connected db-tables. I don't see how I can get around using a branch here. Yes the other developers will need to merke, but else we have broken code in the Main branch.
There may be strategies to work around that. For example, you could maintain more than one database table (or column) to represent the same data in different ways, and drop the older version when nobody uses it anymore.
@@cristianpallares7565 I guess that would be a good solution. I just fear the amount of copied code and the work to make sure everything gets deleted in the end. Well... those big changes should not happen to often anyway.
@@christophlutticke7150 I think this is only one way, you need to have that copy, cause I think yo need to have no downtime for you product during the deployment.
I totally agree with fact that the longer you wait, the more clashes and conflicts may occur. The problem I faced with my team is that some time the work on a feature is incomplete before a deadline and we wanted to deliver some other complete features. Without feature branches we had to do all sorts of things, like turn off flags, hiding controls and even rolling back changes.
Teams that embrace CD change how they organise their work. CD is working so that your SW is always releasable, so after every successful commit, it could go into production. You can do that by waiting till each feature is complete, but that slows feedback and results in lower quality, or you can work so that it’s ok to release, even if a feature isn’t complete. This sounds weird, but is the higher-quality approach, according to the State of DevOps Report.
@@ContinuousDelivery Thank you so much for clarification.
I think the concept of CI has a lot of merit. We had to change our concept of a 'feature' to help work towards this. On the basis that 'no one should commit code to master that breaks the build', we couldn't get an entire 'feature' developed quick enough to commit to trunk/master, so we used feature branches. But then we just found that if we broke down what we meant by 'feature' into smaller bits, we could do commit to trunk/master without 'breaking' anything. This did lead to some issues about testing these 'mini-features', but nothing insurmountable. And breaking a classical idea of a 'feature' down into 'mini-features' gave us a lot of insight into just what and why the 'feature' was in the first place. This worked well with a general 'Agile' thinking in that there were a few times our initial 'mini-feature' break down showed us where we had gone astray.
I think we can have the best of both worlds: consider the following workflow:
The is a "trunk" branch called "dev", and each person creates a feature branch from it and then merge it back to dev after the feature is complete, with a pull request and ab associated code review. After an arbitrary number of features are merged, a new release os born, and it gets merged into "master", the production branch.
If we create a "dev-ci" branch, and have the team muse feature branch, but at the same time rebasing/pulling from dev-ci and merging their changes into dev-ci frequently, we then achieve the ci goal, and the feature branch with commits filtered by author is still good for a traditional code review
We basically do what you describe in the first paragraph. I don't think I understand how the process you describe in the second paragraph differs, though.
@@ShawnThuris sorry for the typos. Instead of commiting and pulling directly on the trunk branch, you merge and pull/rebase from your feature branch keeping it synced with the trunk branch.
It may sound as extra work to accomplish the same result, but it would still allow us to link work items/issues on the management platform (GitHub, azure devops, etc.)
You are basically describing what he did in the video though.
@@defeqel6537 yes, but using both feature branch and trunk ci
That is biased, in my view. What you call fundamental is not working in 100% real life scenarios. There are cases when it's useful to guarantee changes isolation for the sake of risks mitigation. Think of it as full transaction isolation in DB, like when evential consistency needs to be avoided.
Since you seemed curious about why people have an emotional response to this subject, I thought I'd chime in.
What emotionally turns me off about ci is sharing my code before it's finished. Imposter syndrome is rampant in our profession and merging code into a shared branch before the feature is "done" make me anxious.
Rationally I agree with you. Shorter cycles lead to better code and seeing everyone's else's "incomplete" might even reduce feeling of inferiority and serve as a learning aid for junior developers. Nevertheless the anxiety remains.
Love the videos
Look at the Apache Maven project and how they manage all of their stuff, the same way I do, a single branch. Period.
No need for a release branch either, to cut a release, tag it, build a release from the tag, and move on.
Mainline development is what I recommend for all projects, as I’m sick of cleaning up left over branches from others.
Git and Jira feature branches just makes things worse, as it tends to make the processes we follow fit the tools we use. The tools should suit the way we want to work, not the other way around.
The number of times I’ve had to say “Son, just because you can do a thing, doesn’t meant you should do a thing”. Especially around Git branching models.
The only time I use a release branch is when I have really long life releases (and I’m thinking 1-2 years here) and I have non backwards compatible breaking changes in happening. A good example of this would be a change of underlying middleware or java version for instance. Then each release branch effectively becomes its own master/trunk, and master fades into obscurity. I probably just triggered some with that last bit. But from first principles, it’s the easiest way to work - especially when the devs are inexperienced.
you cant just pick the worst examples of your opposing viewpoint and the best examples of your own and then say yours is better... sure if you are messy you can lose 12 months of work using feature branches.
Another thing: Your method introduces loads of dead code into the production branch. If I look into the production branch, I want to only see code that is active, or about to become active. Not half written code for a feature that will be released in five months. For me personally, a clean codebase is way more important than occasionally having a merge conflict or two.
I'm thinking here that even in a team that works using "feature branches" and has periodic long delays in "pull requests" getting reviewed it would still make perfect sense to split the work into a sequence of small focused changes for the reviewer to review simply because it's gonna be easier to go deal with and to give feedback on compared to dealing with a big ball of mud of changes that inevitably turns into merge hell.
To me this seems like it would be a clear step forward by practicing the parts of CI that one can in a broader environment that doesn't do CI... maybe this could be used to smuggle more CI thinking in there in the long term anyway if folks don't believe in its benefits until they themselves have seem them working for someone else in a part of what they do. This is seems like something that is perfectly doable with just about any regular toolkit.
This whole thing reminding me of how in an environment that isn't agile the easiest way to make things easier for oneself is to have one's own surroundings self-organize (even if it's just you) to the extent one has the ability to do so and to just work like an agile organization within a wider organization because it's still better for one team to be agile inside bigger organization as that at least still makes working more efficient and pleasant for you as a developer.
I tend to rebase on top of whatever the main branch is regularly so as to act as if I started my work after the current code base state. This isn't technically that different, but I find it easier to reason about. Essentially I agree with your video that a continuous straight line commit graph is preferable to me. At the same time, small merges are pretty inconsequential and easy to reason about.
People do get quite heated about this so I leave it to them to deal with the worst merge nightmares while I commit on top of the latest when I'm done.
So the team I'm working with now has three branches: master, validation, and integration. If I'm introducing a 'feature': I make a personal branch, complete the 'feature', and submit a PR to merge to integration branch. I'm putting 'feature' in quotes because I'm referring to any changes in the code base. We generally limit our changes to take no more than one week of work.
Twice a day, integration is merged to a new validation branch (ie. validation-2021-05-27), which runs an enormous library of validation tests. If any of those tests fail, the culprit PR will be found and either fixed or reverted before merging to master. These fixes / reverts will then be merged back into integration. If your PR was reverted, you simply pull integration to a personal branch, revert the revert-commit, fix the issue and restart the process.
Now my question is: is this CI or feature branching? There are separate branches for features, but each branch only consists of a single person doing a maximum of one week of work. Does feature branching turn into CI when the amount of people on a branch, and the lifespan of a branch, are limited? For instance what if we implemented the same strategy, but 'features' were split into daily deliverable chunks, as opposed to a maximum of one week, does that count as CI?
Edit: Just found a video where you actually explain the difference explicitly, guess I'll watch that later.
What to do with code review? I can't imagine that there is no place for it in Continuous Integration ?
I have the same problem. I couldn't find any good solution to do code review after code is merged. Comparing selected commits seems to be very challenging (or even impossible). That is the only blocker which prevents my team to commit directly into master branch. At the moment we trying to creating branches which lives 1 or max 2 days. But I would prefer to merge directly into master.
@@rafacypcer1408 Why don't you use tags or a specified commit message? You can review the "sum" of these specific changes iteration by iteration.
We used JIRA and Bitbucket, and we had commit message conventions where we put the feature id and the task id. This enabled us to review the task alone and also the feature ("FeatureId/TaskId message"). Each commit was visible under both the feature and task in JIRA.
We used feature branches but with this you still can observe what is happening with your feature, and you can always create more tasks under your feature as it is only closed when the team is done. The only downside I see is that you can't reject feature merges, just the merges of your little tasks. But with trunk based development it's not really a factor as you want to apply changes rapidly instead of creating a big/coherent feature changeset.
In Daves philosophy you ensure code quality through TDD and a significant amount of pair programming. No code review
Don't use code reviews as a gating mechanism for changes. Code Review should be something that gets done to look at how and why things were done the way they were and how it can be improved. If you use code reviews for enforcing methodologies and finding defects use automation for that. Yes it's 'after the fact' and the incorrect code is already in master, but using CI it can be quickly fixed.
The flip side is if you and the other developers are constantly pushing changes and pulling changes code is being indirectly reviewed all the time.
@@Rekettyelovag Thanks. I also use similar pattern for commit message. I'm also able to see in my managment tool (in my case Azure Pull Requests) connected commits. But the problem is that I can see the commits, but not files with diff's (which is obvious, because it most probably git limitation in my case). I don't know any git feature which gathers selected commits and show diff's in files comparing the last commit. I san see changes by selecting two commits - not more.
I understand that with CI/CD we need to change methodology. CI/CD is used mostly for fast development - not for code quality. Code quality should be resolved somehow different - like changing team culture, improving automated tests, applying linters/code rules etc.
For approach with code review before commiting to master we had step where someone could verify someone code (which blocked development - because code 'frozen' during code review shouldn't be temporary used, by someone else). Merging directly to master forces every developer to review changes everytime when he/she needs to pull new changes. But it happens constantly. With feature branches reviewer was analysing the whole feature context. With branching directly to master this context is lost, because of continues delivery.
It would be great to have a clear explanation of the hypothesis. It's not clear what feature branching is. I believe that CI is not alien to use your own branch to work on your changes and then allow peers review. I understand that short time feature branches could be used. It's not clear to me if those are not even accepted in CI.
I think there is more to it than thinking ourselves as heroic loaners 15:01, one of the ways we breakdown complex problems and also with the code we write is the idea of breaking down a complex problem into specific concerns where we can think solely about each part in as much isolation as possible be it steps in a process or the responsibilities of an object. Isolate a single part of the greater problem at a time. Therefore, merge conflicts with someone else's code means we need to context switch to understand a potential different problem context in order to resolve the merge issue. Now in my experience its the guy who commits later that is responsible for resolving the issue (is this the unresolved special source that's nagging at the back of my head), and hence has to do this extra work of understanding an additional problem domain/context. Resolving who has to context switch stop the breakdown of their own feature and understand how their own problem context interacts with someone else's current work in the same problem domain ... should this be left to who commits second is seen as responsible for the merge ... the associated work is delegated on nothing more than who can smash that commit button before the other? It also means this is the point at which people's different coding styles interact, has the code been refactored or is still in a state of just putting together something to pass unit tests. I'm not saying leaving people in isolation till their own feature is complete before attempting merge with another complete feature, meaning there is a potential for a large divergence of code. I just think there is a social/phycological dimension to ... THE MERGE!!!
I've found myself merging code ... before commit, "oh wait ... it's changed again" ... ok spend the time understanding the merge ... "ok right think that's my changes merged ... better do a pull form the remote" ... "oh jeez again some more small changes made before I was able to commit". This is doing my head in if only I had just been able to commit before I had to pull my hair out trying to understand this stuff written by someone else.
Maybe it creates a feeling of winners and losers, and it's the speed at which you commit that is being rewarded not the quality of the code. Maybe I'm just crap and slow at merging peoples code, maybe needs ideas on "Pair Merging?", but does anyone else have experience of he who commits last is the loser??
Never really had that problem myself. But then I have always been vocal when I see conflicts and hero-culture and try to break out of it as soon as I see the signs.
There is a saying in the DevOps world - "a bug, is a bug, is a bug". Meaning that bad code, bad process, bad security, bad compliance, low trust, bad logging, metrics and audit are all bugs. But we tend to pretend they are someone else's problem because "I'm just trying to get my task done".
As long as everyone has the mindset that it's a contest, that developers shall sit alone in their corners with their headphones on, typing code all day. Instead of working as a team (across roles), have good practices, a good modular design (so that there is less chance of stepping on each other), working in pairs and sometimes entire groups (see mob programming) we tend to default to defensive behaviour and then there will be friction, frustration and no joy. And then we try to smooth over the cultural issues with processes, Gitflow and feature branches and silos and handoffs with 27 new roles. A team should be measured based on result and outcomes - not individual contributions.
My point? Any mindset or culture that focuses on individual "performance" (stack ranking, ego, loudest-guy-opinion, "winners-and-losers") and cop-outs like "but I'm an introvert" is A BUG! The issue with seeing ourselves as heroic loners is indeed the biggest issue in SWE.
Since in your scenario both of you apparently were working in related parts of the codebase - I think you should have taken a step back, sat down with each other, talk, plan, pair-program and collaborate on both of the changes instead if sitting there frustrated that "the other guy screwed up my changes again and now I'm getting behind and losing the scrum story-point race". Maybe I'm taking this over the top, but only because I've seen it happen to other teams so many times. Learned helplessness is the worst place to be, because it absolutely drains all energy and joy of going to work and life in general.
@@ddanielsandberg I find it a bit scary when normal human phycology gets referred to as “A BUG”, I think everyone has different personality traits, strengths and weaknesses, and in different situations weaknesses become strengths, in some situations being an introvert that isolates themselves to push through a single focused idea is what’s needed, but then there’s other situations where it’s collaboration that’s going to be the enabler. That doesn’t mean I haven’t been pissed off and got people kicked off a team … but I find it most stimulating and sometimes most frustrating to work in a diverse team, argh and though it pains me to say so, good management of people counts.
@@derekcarter4157 I didn't mean to imply that people are bugs. :(
I meant it more like the system is buggy if it enables and incentivises certain negative behaviours and discourages other positive behaviours.
The software industry has spent the past 40 years hiring and attracting certain personality types and I think it has had a detrimental effect.
I know how it feels some days, I'm thinking of leaving the software industry and become a carpenter or painter or something because this has given me CPPS.
“A bad system will beat a good person every time.”
@@ddanielsandberg aye this covid lockdown from last year seems to have been terrible for my eyes just being in the house never looking more than 7 feet away (or maybe I’m just getting old). But yes it’s a software dev is a creative endeavour, even if all our best ideas are stolen! Processes and techniques are to help people create and communicate (and I’m the case of SAFe and Scrum make a quick buck) teeheehee.
Literally a day ago was talking with a friend of mine who is a QA lead at a big IT. He complained that it's a nightmare to deal with 10 teams pushing changes into the develop branch. He said that it is unmanageable because it's almost impossible to blame what feature, what change, and who introduced a bug. They spent hours just to identify this info. They do CI, but everyone understands and complains for that kind of workload branching model would most likely be much more manageable for QA.
I hate to break it to you, but if you have a develop branch and each team is branching using something like git-flow, you're not doing CI.
Not to be confused by not branching based on user stories. It is still helpfull to create isolated environment, a.k.a. branch, for user stories which takes couple of days only to implement and integrate. It is the smallest agile/scrum unit of work that validates a branch. It is more easy to discard a branch than to go back in history in case something goes wrong and developer has to start implement the functionality from scratch.
Issue is, as pointed out in a video, a feature branch which takes several weeks until it gets tested and merged. Just my 2cents of course.
I've really enjoyed your channel for a while now. I have picked up far more insightful ideas than I ever thought I would when I began watching.
So my first serious comment on your channel is this...
I absolutely love the shirt!
I've only rarely run into major conflicts when merging. Even when we had several people working on major additions to the same product. Most of the time branches are used to make changes or experiment without breaking the main branch builds. And since there are many things that we are unable to build and run locally code has to be committed to put it through the build, test and deploy pipeline. We could certainly investigate making it possible for each person to do that locally, but since branching is actually working it would be hard to justify the time and expense.
Personally I'm not a fan of pair programming. We certainly use it on occasion, but I'd burn out in months if I had to work with someone looking over my shoulder all day. I've turned down lucrative job offers because the job would have involved working like that all day every day.
Excellent illustration of why the ingrained scatter-gather pattern of working is inherently flawed. We're experimenting with mob programming where all these concerns melt away, since no one is sent to their cubicle to work in parallel and in isolation.
I have a shell script, git-sync, that replaces git pull --all + git rebase + git commit + git push, which mitigates some of these issues. Dave criticizes this approach at 7:21, but my script goes farther. It also checks for git conflicts across all other active features branches (we delete branches after merge), and list other branches. Finally, we have very small tickets, and we break them down even smaller to tasks which are safely atomically merged to trunk. We aim for "tasks" to last no more than 24 hours. In stand-up, we view branches older than 24 hours. Of course, none of that's full integration, but it gets us very close. Close enough to be better than having tons of feature flags, which is often required for trunk-based.
Thanks for the vid! It's very refreshing to see motivations for "why we do things like we do" other than:
* that's how google does it
* that's how good developers do it
I appreciate you adding a scientific-like point of view.
That being said, the context where CI really matters is almost clear, but not quite, so I'd like to understand:
* Does CI matter more for companies where really big teams (or lots of teams) are working on the same code-base?
* Does a microservice architecture make CI more relevant or slightly less important?
While all I can say is "I have an experience-based opinion", here's my take on the two questions:
1) It matters for different reasons.
In smaller teams, you will have fewer versions of truth. At the same time, you should also have fewer reasons to diverge.
In bigger teams, the it's the exponential growth of failure points over time that makes CI pretty much a necessity.
2) There's something interesting about microservices, and that's the decoupling of implementation from API.
As long as you're not changing the API, there's no need to withhold any tested change. Indeed, if you can push out the change the minute it's made, you're maximizing the value of your work by minimizing lead time. And one of the core advantages of microservices is that they let you do this, so if you don't - you miss out on a large portion of their value.
BUT - when you're continuously changing the API, you need a strategy. Many companies don't think about that, and hence get into a terrible mess when CI'ing microservices.
We have builds and test (functional and unit) runs on every commit of our branch. We do rebasing instead of merging of main develop branch into ours. We strive to make multiple smaller MR/PRs but they always have to be a finished working bit. Since codebase requires quite a lot of refactoring so that C++ api guarantees code correctness far more than we are able to achieve in Java version of the product even though Java version is younger... Which is really important for multi threading purposes (when using shared data). But the most important part are code reviews (no tests are not replacement for code reviews - and I'm talking about to some extent almost formal code reviews - they are additional help and documentation). Based on this and the fact that branch is developed for a week and then gets stuck in code review for about 14 days (including changes to the code) I would really like to see how CI and constant pushing of code would produce better code. Faster written for short term gains? Yes. Better... Please explain how as I can't see it work for anything more than service development where partial uptime is more important than stability of the whole.
I have come to love this channel. Thank you for the videos!
I'm curious about when/if Product testing fits in CI. Say that a developer (alone or working as part of a pair) is working on a feature, which was specified by a Product owner of sorts. There's always a chance of some level of misunderstanding between what was asked and what's being delivered. So it's now about whether the code works or not, but more about whether it does the right thing. In that case, being able to deploy a version of the code to a test environment that can be exercised by a Product person is quite useful. However, if we're always committing to trunk and the code is doing the wrong thing, it might already be too late.
I can imagine one solution for it being feature flags for every single customer facing feature. While I can see that working, it can clearly add a ton of incidental complexity to the act of adding features and changing a product.
What's your recommendation for that kind of scenario?
Thanks!
Thank you for the video. How would you go about prototyping different solutions whilst working on the same branch? It seems to me that rollback might be a really complicated in this case.
The more often you commit, and therefore the smaller your commits, the simpler the rollback! However, for experimental things treat them as something different. Do them on a branch, but then throw the branch away (or at least ignore the code in it) and re-write the code from scratch to production quality once you have learned the lesson. This approach is sometimes called "creating a spike". A spike is an exploration of some idea, the result is the learning, not the code. That gives you the freedom to write crap code if it helps you learn faster.
I do get the overall idea but of course there exist several cases where feature branching is actually more productive and perhaps even mandatory. A few examples that come to mind: 1. perform a refactoring in order to replace an external library 2. Any form of "experiments" that are not supposed to be deployed in prod 3. large changes that are difficult to release "darkly" (or that can be released darkly, but with a high extra cost in coding the "darkly" (which is actually a feature itself). Again: I get the general advice but in my opinion "continuous" must be taken with a grain of salt, as with most practices.
I'd say it the other way around. CI works for nearly all cases. Occasionally branching, not necessarily feature branching, is useful in unusual situations. So "take branching with a grain of salt".
The other take on this is that CI applies a pressure on you to think a little more about the design of your system. So, for example, I think it extremely unusual to need to use branching for your first example. Sure, I have done it, but these days I don't need to. Because I'd probably have a facade or adapter between my code and the third party library to make my code testable. So switching out the library would be easy. That is only not true when your code is tightly-coupled to the library, which is in some, very unusual, circumstances I may allow in my code, but very rarely.
thanks for your comment@@ContinuousDelivery. As stated in my first comment, I agree that the worst evil is multiple long lived branching, to the point of every single developer in the team creating a personal branch which quickly becomes impossible to merge (I personally witnessed a few of these edge cases). Yet sometimes branching is good, but you (that is: not the single developer but the team maintaining the codebase) must know why and what you are doing. Other aspect, this being more a technical detail: what do you think about the "short lived branching" technique, compared to the "commit to trunk"? In my opinion the short lived branching (branching but merging the branch very often) preserves the goal of continuous integration but is more solid in terms of protecting the main branch from breaking commits and it's easier to implement with modern CI tools.
Generally agree with this precept to integrate small bits of code frequently. The only exception I can think of has to do with cybersecurity or life-safety system requirements that disallow dead code (i.e., can’t get there except in an unplanned error state) under all circumstances due to the risk of accidentally entering those code segments. In less critical codebases, I can easily agree that feature toggles (and similar control measures) are adequate and should be used instead of feature branches. If you follow a stratified development approach (e.g., Strata from Vitech for systems engineering), then you can, for example, stub the feature and integrate the stubs with the trunk, then add functionality as the feature is developed.
Working in isolation in a feature branch keeps your unfinished code from being merged into the main branch until its completely done.
If you check in code constantly to main, then you will have unfinished code or one big push with all your local commits. Keeping commits local without pushing is a problem because they can get lost if your machine crashes.
You should always keep your feature branch in sync with the other developers by locally merging the main branch into your feature branch, atleast once a day.
Though these features branches should also be very small. I'd would say that they should only live for maximum a day before it's being merged with main. Otherwise your features are too big and you need to break it down further.
I'll only disagree that a feature that takes more than one day is too big. I know from experience that some things just don't break down that small and the effort to force everything into smaller pieces invariably takes longer than just implementing the bigger feature would have.
Great video Dave. I heard you first on the Changelog podcast, then looked you up here. You've spoken to my years of managing feature branched projects and suffering the inevitable conflicts - even in the very small teams I worked with. It's spot on. I'm running the rebuild of a Drupal project, which will initially be built without any automated tests and all in feature branches. In the very long run I'd like to move towards CI and TDD.
One of many questions I have though: Our development so far has gone like this: A developer works in branch, I review the PR and test functionality and pass it over to another non-dev stakeholder who also reviews it against the spec and tests it on a staging set up. In CI, do non-dev stakeholders have to do their testing and reviewing directly on production? In our experience this non-dev stakeholder often catches mistakes where the spec was misunderstood, and we feel strongly that these mistakes should never make it into production. Sorry if that's been addressed already! Thanks for the videos.
My preference is to try an in-line this kind of overview, so that it isn't a "gate" in the development/release process, but is part of the development. PRs and reviews tend to slow the process, so the idea is to do these things continually alongside the development.
Instead of performing a code review at the time of a PR, pair-programming gives you a continuous review as well as lots of other benefits.
Instead of non-dev stakeholders looking at the code catching mistakes after the code is thought to be finished, build automated testing into the development process so that catches most bugs. Where a review is meant to help sanity check that the change looks as expected, rather than acting as a regression test, then do this early in the dev process as soon as you can.
Overall the aim is to build quality into the code from the outset, rather than try to "inspect quality in" to the code later.
Thanks for the question, I think I may do a video on QA and it's role in CD teams that may explain some of this stuff.
@@ContinuousDelivery Thanks for the reply! Yes, I could review the code over the dev's shoulder prior to commit I guess, but the functionality that was added has to be tested/reviewed within some non-production environment. So yeah, that is a gate in our process.
It seems to me that in CI you have no need of a staging environment. You appear to just have local development environments, and production. So if that's true, then our non-dev stakeholder can only review what is built when it's on production, because they can't use the dev's local environment. For us, the key thing that this person brings to quality is ensuring that what is built matches what was intended by the spec. Automated tests written by a dev that has misunderstood the spec won't catch that sort of mistake, it can only be caught by the person who knows the requirements. Love to hear your thoughts on that!
@@dansearle5994 Not really, the deployment pipeline is a controlled environment, outside of local development environment of an individual dev. It is a shared space where changes are integrated and evaluated. Usually in production-like test environments that mimic prod config and scenarios. These are close to a traditional staging env, but not quite the same. Ideally the person that "knows the requirements" is part of the team too, so close enough to the dev to see the soliution evolve and so catch these kinds of misunderstandings before they progress to the (too late) point of be deemed "finished" by the dev team.
@@ContinuousDelivery ok I see. We're used to reviewing the produced work in our own time, on our own machines (accessing the staging environment), so it's a big shift to go from that to review system where we're with the dev more often, looking at what they're working on on their machine (either in person or via screenshare I imagine) in little bits. It's food for thought, I'll see what I can do, over time!
Interesting, I certainly agree with the principal, but think that using feature branch and PRs is useful. IMO I like to isolate CI from source control as much as possible, so there is one mechanism to trigger a build. Tags. A modular code base helps avoid conflicts more naturally also. Frequent conflicts between independent features is a sign that concerns are not well encapsulated or decoupled.
I don't agree with a lot of stuff he is saying but the comment section is great to read. There are some really interesting points of views.
Could you be more specific in what you disagree with?
Branching does allow for an individual developer to create ownership over the feature they are working on. I'm not sure if it's always the "I'm the hero" motivation described in the video. Having pride over the work you do does create motivation to complete it which is important. It's also important to note that individual contribution does have a tendency to create tangents of thought that could become unpredictably useful or not useful. That variation can come in handy. That said, I do find it hard to argue against the idea of more closely collaborating to create better awareness of the application state. Maybe the experimentation and offshooting should just be left to a branch that's not necessarily meant to integrate for experimentation purposes; left to individuals who want to propose changes in structure and such.
I don't think your wrong, it's just that the software industry case been convinced that a feature is the smallest change worth integrating. This allows for all unit tests, functional tests, etc to be done over several commits that then mark this feature complete. We more use feature branches as short term backups then branches. I think where you would make this all one commit and the push it to the trunk, is really equivalent to a squash commit on the feature branch before merging. Also GitHub better supports the feature branch model with PRs, and other expectations of that system. Where would the review go in your CI model?
If you are dealing with a live system, with live customers who may report bugs that need to be fixed ASAP, maintaining an active "golden master" branch is the easiest, safest and fastest way to deal with it - and that means NOT having a bunch of barely tested or reviewed code go into the mainline. And even for an individual developer, I often use feature branches on my own projects, where I'm the only developer, if I want to experiment with different approaches that I may end up discarding later.
I really don't get the hate for feature branches. I work with some trainee developers that need more review than others. I don't WANT their code to be continuously integrated - that would actually cause problems. And as for long-lived feature branches creating a growing tech debt/merge risk - just keep rebasing your side branch on whatever's been accepted into master, and you'll never be that far from a merge. Seems pretty obvious to me.
To avoid branches and to just have everyone work in the master branch seems like chaos to me, and putting any customers of a live system at risk of not being able to provide them with timely bugfixes.
People hate feature branches because they don't know how to merge and are not competent devs imho and want to pretend they are some code ninja when they are just code hackers. 100% as you say keep rebasing from master on a regular basis. It is not rocket science to pull to keep up to date.
I agree. It's sad many people after gaining experience will look back on this video and say they got ill advice about not branching. What made Git popular is that rebase and merge are easy because branching is a must.
The problems mentioned in this comment seem to be the product of not enough test automation and not utilizing pair programming. Having junior developers work on features by themself in the dark is the reason why you don't want to integrate their code. Have the seniors pair with them instead of waiting for a PR to review.
Thanks, you correctly answered my arguments of pulling every day. I agree CI is the best way and committing to the trunk every day is the best idea. That graphics was useful.
Only point is we must have test suite there that covers well.
A) We are pretty much stuck with feature branches on an organizational level
- we are trying to keep our branches small and frequent though (still not perfect, but we are improving), in the end, communication is key here whether on the individual feature-branch level or when talking about the "complete" feature and the plans for it
B) I fear CI is too straightforward (for a lack of a better word)
- you either need really good pairs programming together to avoid silly and/ or hard to find bugs creeping in, especially as less capable developers are just as capable of breaking tests as any other part of the code
- you miss that part of separation that a pull request and review gives you; this is personal, but I can better focus on reviewing changes and trying to figure out what they affect when I go into code review for a (small[ish]) PR. This is also true of my own PRs. This, I think, is especially true when your code deals with multiple concepts that you need to mentally context switch from. I'm fully willing to accept that this might be my prejudice talking, since I haven't really been involved in a real CI project thus far.
edit: I do adore your videos though, including this one
This is an argument against distributed version control, not just branching. With DVCS, even if no one branches, separately developed code isn’t integrated until it’s all pushed to a shared repository. Merely committing won’t combine anything.
There may be an issue with semantics;
TLDR; If we branch off of the trunk, but the branches are so small that several of them are PR'd in a day per person, does this still count as feature branching, or is it CI/CD as you describe?
I run a small team of 3, sometimes 4 people, and we never work directly off of the trunk (which is actually our staging branch, but that is another story), as we want it protected, and instead, we make what we call 'feature branches', which is often small break downs of larger features into tasks (so maybe we should call it task branches), small enough to complete at least one, but usually several of in a day.
We want the trunk protected as we want to review all the code via PRs before it goes into the trunk.
We do this so frequently, that merge conflicts are quite rare, and our CD pipeline is often backlogged near the end of the day.
In addition to this, we have small 10 to 30 minute (depending on what needs to be discussed) meetings throughout the day to discuss what we are working on and the direction we are taking, which helps avoid having to re-do a lot of work at PR.
This strategy involves, more frequent, but much smaller PR that requires very little comment.
Seems to be working great for us, except as the lead I need to work on getting my code reviewed by the other members more often and not just self approve.
So basically, does this workflow still count as CI/CD as you describe, because it's still frequent, or is it disqualified because we are not working directly off of trunk. And if it's the latter, is it possible to protect the trunk in any way in this workflow?
This is exactly my situation with my team. We create small task/feature branches that get made into a PR within the day. On a good day I can make as many as 4-5 small branches/PRs.
The hard thing though is our code review culture. Because reviews require context-switching and there’s so many PRs that get generated, there’s too much pressure put on the two reviewers that each PR requires (and we don’t practice pair programming). What ends up happening is people only review 1 or 2 PRs early in the day before tuning out of reviews to do their own work until the end of the day and by that time there’s a backlog of more PRs.
Curious about the answer to your question and the code review process too.
@@DanielSanchez_dsanc89 The short meetings I mentioned later in the day is where we do our code reviews, which helps contains the context switching to specific, narrow parts of the day, and since we do it verbally and with screen sharing, it goes faster as well, and we do them in batches.
And the other thing that I mentioned that helps reduce PRs is showing the code to each other (and hence your reviewer) in a checkup meeting earlier in the day, before any PRs, to pre-empt mistakes before they are made and need to be corrected in a PR.
So our meetings are an early morning standup (7:30), where we plan out what we are going to do (but we sometimes skip it if we discussed it at the end of the previous day), then we have an 11 AM code checkup meeting where there are no PRs expected (but sometimes quick tasks get PRs by then), then a 2:30 PM code review session where we are supposed to do all the PRs (but in practice half the time it's another code checkup), then a 4:30 PM staging review session, where we are supposed to see the finished work running in a staging environment (or production with feature flagging if we would not have a staging environment) (but in practice half the time it's code review and seeing the code running on our local dev machines).
The goal is to get your code running as close to production as possible on the same day that you are making it. Our client wants a staging environment, so the closest we can do is a staging server, but we do have feature flagging so we can deploy but disable incomplete features to master, so it's seldom far behind.
@@DanielSanchez_dsanc89 Sounds like our situation: the review tasks piled up. We switched from a gated review process to pair programming. We are not pairing for every tasks, but for important ones (with a more or less accurate guess)
@@DanielSanchez_dsanc89 I've found that talking about your goals and ideas before-hand can help reduce the burden when reviewing PRs. The reviews go from "what is being attempted here" to actually evaluating whether the code accomplishes the intended changes (without side effects).
I was struggling to articulate similar thoughts. Short-lived branches for PRs of small, well-focused changes; and being disciplined to always keep master in not just buildable but in a state where it is deployable to production seems like it achieves the desired goal.
One thing confuses me with the “no branch” concept: in order to have a code review prior to merging, we *do need* a branch other than master (or whatever trunk) to be able to create a pull request, right? Or how else a code review would work if everyone is committing to the same branch?
While you could do "trunk based development" (commit and push directly), CI does not necessarily mean no branches. You still might want code review and other QA gates in place.
I have also seen "short-lived" branches basically for code-review and such. So you do all your TBD in this "short lived" branch and merge after code-review/checks
Well, that's the thing. Often, to do trunk-based development as effectively as possible, you don't do a pull request based code review workflow - instead favouring continuous review via pairing or mobbing. That's the ideal.
However: there's a difference between a pull request branch and a feature branch, and it's mostly a question of scale. If said branch only lasts for hours, then sure, that's continuous integration. Directly pushing to trunk is just the taking that approach to the max.
I don't do branches. Everything on main. Git tells me when I haven't updated when I try to push.
I don't see the point in PRs either.
Whenever I see a PR I lack the context around the decisions, I let the pipeline feedback in the form of testing and static scans etc to ensure quality is brought in up front.
Yeah, I don't see how it makes sense at the end of the day without some branching. My understanding about what he's pushing for is just more frequent pull requests
What if one day your product owner decided not to release and undo that one complex feature (i.e. shared library change, db schema update) that everyone else had already integrated deep into the trunk?
This is the reason you need loose coupling in software. Ideally, you'd have some level of abstraction between your code and dependencies, such that you can swap out one dependency for another, one DBMS for another, with the core code blissfully unaware of the change. It's why the Repository layer exists in Android applications.
In solutions of tigthly coupled code, more monolithic, where boundaries are not clear, branching can cause problems and is expensive to manage. Simply because it is more difficult to understand the interactions within the code. However the alternative is much worse. Many arbitrary check-ins mean you may never achieve a stable build. The team just stops working until it is resolved - huge limits to team scaling. In practice you fall back to your local disk as your stable branch! The team don't pull everyone becomes terrified to push. The scope of changes becomes very very small until the code is refactored to allow multiple contributions.
Contrast that to software with boundaries interfaces, services, single responsibilities then it becomes more clear which areas of the code a project, team or individual may touch. You know which component are likely to change for a given project or team due to the nature of the project. Then you get a better understanding of the changes across the various teams or projects and likely clashes. You may get a tendency for teams to focus or own specific components. In this situation, a more modern approach to design, branching makes much more sense. The answer to clashes may again be to refactor code. In this case the branching is easier. At this highly componentised end of the code base spectrum software is typically packaged. Adding tags to NPM or NUGET is like branching. So we also support a sort of branching of packages but in this environment it's designed to be short lived - pre-release labels. Sort of a clash with branches here btw.
In all cases a stable version of the code, that a individual can build, is always required. If not you have no development. So a more abstract 'branch' is always present. So the statement "don't branch" is not a practical position at all (techi click bait and you got me). Where are the branches? What are the minimum branches needed? Are more sensible questions. My feeling is the answer to brancing nightmares is usually code refactoring such that branching in whatever form works better.
That's why team members talk with each other. You can't say people don't know what the other is doing. If you work like that then you don't have team, but instead group of individuals or problems with project management.
Pushing to master everyday means fixing merge conflicts everyday. Is it best way to deal with this problem? Even better, you can't decide feature one is ready to deploy, but feature two need some changes. How do you do this? Do you remove feature two from master and add it after release?
Finally you chose the most corner case you could. Multiple big changes in the same code base at the same time. I don't know on what projects do you work but in my carier I didn't stumble upon that situation too many times. Because as I said at beginning, my team is there on plannings and rise concerns if someone tries to plan features like that. If it's needed you have more possibilities.
You can create feature branch for 2 or more people and they work on the same code base but not on main branch. Still I needed this technic like few times in my carier.
I would really like to hear about how folks do code review in trunk based development. Years ago I worked for a place that did trunk based dev without code review. Integration worked well and the test suite was solid but the design of the code sucked because there wasn't a system for reviewing code design and architecture. I am wondering if it is possible to change workflow to accommodate this without having to have every change reviewed? I wonder if docs around feature architecture committed to source control coupled with well planned conversations would make sense? Anyone tried something like that?
Just realised I am one of many asking the same thing here
In my experience, long lived branches are partially the result of poor planning and lack of design, and a proclamation that integration is difficult because the "complexity" of the feature doesn't lend itself to being worked in pieces.
A lot of teams and their developers just want to jump into development of that next feature, rather than first reasoning about how the feature will be integrated, and the design considerations to incrementally build the code.
The issue can be "there is no design" - this is the case in large companies where development is off shored to vendors but responsibility is maintained internally. Management simply want metrics to justify their budgets (which can be huge given the number of people involved).
But then the problem is long-lived branches rather than branching itself. If someone is out on a branch for months, then I can see the problem. If they split off for a couple of weeks because they want to try some radical changes I don't see any problem.
I agree with the principal that maintaining multiple branches can go wrong quickly but branches are not always evil. Here is my scenario. I have a production system pulling from master. I need to make large breaking changes to the API and UI. If i were to do this in the master branch I would lock myself out of maintaining the production system since I can't push or pull anything into production due to all the breaking changes in my code base. This has caused me not to be able to pull any changes into production for months until the dev changes were complete or having to directly modify the code on the production server. Not a spot you want to find yourself in. Having a single dev branch gives me the ability to switch to master, make a hotfix, commit the changes to master, pull to production and then switch back to my dev branch, pull from master to integrate the change and continue development on the features. Not branching in a scenario like this causes a seriously hard to maintain system since you end up having to patch all code changes directly on the server since you have now locked yourself out of your version control system due to all the breaking changes on master. Don't abuse branches but do not avoid them when you need them.
Not looking for an argument here, just wanting to point some things out.
CI and TBD is designed to "solve" the problems you describe by exposing them instead of trying to work around them with band aids, processes, phases, hand-offs, etc. I would go so far as to say that the whole point/secret to (agile) software development has been forgotten in-place of process-itis and navel gazing. Just my opinion.
Here is a great quote about agile "Agile's biggest strength lies not in solving problems, per se, but rather in exposing your buried problems and making them visible so you can deal with them. For a team switching to agile it can feel like things are getting worse, not better, but that’s often because they’re being made aware of pre-existing problems they didn’t know they had. It may not seem like it but this is actually a good thing." - Someone over 10 years ago.
Or in the words of Jez Humble (co-author of continuous delivery) "If it hurts, do it more frequently, and bring the pain forward."
Instead of getting bogged down in the thinking that "we can't" or "we must", what if you asked yourself a different question? "What would need to change for master/main to no longer be blocked/deployable for new features, fixes, etc; to be able to do any of these things as needed, when needed, and never be more than a few minutes from being able to deploy to production?"
Problem-solving is more about asking the right questions than providing solutions. Adding complicated processes to hide the systemic issues is seldom the answer.
Actually "if it hurts, bring the pain forward" is a quote from our book, so both of us said that, not just Jez 😁😎
Otherwise I agree entirely with you comments. Agile is not so much a recipe for producing great software, as it is a recipe to allow us to spot the problems that prevent us from creating great software so that we can fix them, and so create great software. "Inspect & Adapt"!
So this topic makes a lot of sense in a lot of ways but one issue I see is with how my studio works. We have new programmers and interns from a local college working on projects often. This means that any one of them could break things on a moments notice with CI just due to discipline and skill being not there yet. On the flip side it would force them to pay attention and work more effectively though at a cost.
This isn't a finished thought really and context matters but I felt like voicing it can't hurt, all while I continue to think about it.
I think that is a fair point. My prefered way to introduce people to a team is through pair-programming, that instils the discipline and gives the newby a guide to the codebase, working practices and so on.
Branching with small FF trunk based development just works. Especially if you want to do different types of testing that is not similar to your main app (i.e. different frontend with a ephemeral env). This is something that doesn't have to even be defended. It's not either or, we can take parts of what your saying and branch as well. Branching is not guess work though and having everything in 1 branch stops dev testing and exploration. Testing one small change to code is testing more than just that code itself. You have to also test how it interacts with the env you have set up and I don't see how that would work with only 1 branch.
You mention the cost of CI, and IMO that's exactly what it's about: "cost". For many teams, industries, features, and contextual situations, being able to make continuous changes that are "ready to deploy" but not complete is pretty hard. There is little experience or insight on the strategies like feature toggles; branch by abstraction; dark launch; etc. But even if there is experience, every project and team always deals with something new, the team needs to know how to apply those techniques to their current situation, and that is not easy.
For those teams, continuous integration is more costly than the pain of merge conflicts and the window of time where devs are "in the dark". This cost is understandable and at least easy to identify, so teams choose it and forego CI
For continuous integration to get a bigger going, IMO that cost needs to be reduced, which I think is actually pretty large. In general, there are a lot of features; projects; etc; that don't fall neatly into a continuous integration approach, and they would require a lot of thinking and good design to be able to adapt to CI (with the strategies you mention). Basically, to be able to perform continuous integration you need all of your team to be extremely good at their job, you basically need all of them to be seniors, and good ones at that (at least in those cases). If we can reduce this cost, then teams would naturally adopt CI much more often IMO, otherwise they would ditch it the moment they encounter a big difficulty.
Great content and videos btw!
I'm a fan of CI, but I'm having an issue with the idea that a feature branch necessarily means longer times between commits. If I were to run a development team, I would require the same interval between commits regardless ... basically when a small feature is ready. The gain (as I see it) from feature branching is the ability to merge from trunk and see any conflicts or failures before merging to common code.
I had some time to think about your views. This is going to be longer comment, but I really hope I can get an answer from you, because I believe I have some strong arguments. I think you're loading into the idea of 'feature branches' something more, than I do.
In particular, I agree completely, that the problem is in how OFTEN developers in the team integrate their work with the trunk. And that the less time they do, the better. I think nobody reasonable challenges this.
Where the problem arises is that in fact NOT having feature branches does not prevent developers from diverging their work for too long and vice versa, using feature branches does not automatically mean long time to integrate (which is what I imagine is, what you believe the term 'feature branch' means).
With trunk based approach, a developer could conceivably work on one commit for few days and create a humongous commit. In fact, sometimes there's just no way to avoid such commits. They are very rare, but they do come up, for example, if one needs to do large scale refactorings, that affect the whole code base.
On the other side of the spectrum, if a developer uses short lived feature branches (say a branch contains maybe 2-3 commits and is merged with day or two after it was started), it poses about the same risk, as trunk based development.
This approach of creating short lived branches, that are ideally small (2 or 3 commits), and are merged shortly after they're created (day or two after they're created) has many advantages over trunk based developments, with the primary disadvantage being, that it slightly increases the window for integration, but really only slightly (nothing in the realms of the horror story mentioned in the video).
The primary advantage of this type of feature branches is, that there are plenty of tools (Github, Gitlab, Bitbucket, etc), that enable performing code review on a feature branch and this way code that is not optimal is not pushed to the trunk. There's one killer argument, that makes trunk based development completely unviable, unless there are special tools (that are not industry standard such as Github, Gitlab etc);
With trunk based development a member of the team could conceivably push a commit, that fails to build. Now every developer that will checkout trunk will be unable to build their local branches. This is a disaster! And no, pair programming will not prevent this from happening.
How does feature branch based development prevent it?
By running automated build on the feature branch. Not sure about Gitlab and others, but Github can be set up in such a way, that the feature branch must be up to date with trunk changes prior to merge and automated builds have to pass in order to enable merging. This guarantees at least builds that will pass. And hopefully more people can review changes and if runtime errors would break the project at least more people have a chance to figure it out before it actually gets merged to trunk.
I think if feature based development is done right (in the manner described above), it must be superior to trunk based development.
I have not discussed code review, I know you advocate for pair programming, which honestly still should NOT absolve code from being code reviewed by other members of the team, so I think pair programming and code review complement each other, it's not one or the other.
I'm really curious what you have to say about this and in particular how would you avoid problems with accidental push to trunk of a commit that breaks the project in trunk based development.
I don't like the term "Trunk based development" much, I prefer the original term "Continuous Integration" it is more accurate. In CI, it is not possible, by definition, for a developer to work on one commit for a few days. The definition of CI includes "merge to mainline (trunk) at least once per day".
If you merge FBs at least once per day, then it can be classed as CI, if not, by definition "not CI".
"If FB is done right it must be better than TBD" - The evidence is against you, take a look at the State of DevOps report, "Teams that have FBs that last longer than a day have worse scores in Stability & Throughput". Stability measures the quality of code based on defects per change and mean time to recover from a defect. Throughput measures effciency based on time from commit to release and frequency of release.
Oh, and you avoid breakages to trunk by doing a decent job of testing. 😉
@@ContinuousDelivery still, accidents can happen, it is possible to break trunk. I have worked this way before and even with best efforts, sometimes it just happens and in less experienced teams it's going to happen more often.
I don't know about the research that you mentioned, I would need to see the methodology first to be able to conclude it's a valid conclusion. I can imagine already many problems with trying to make a valid research that would prove such conclusion.
@@vyli1 I have a question. What does it matter that trunk is broken just once-in-a-while (meaning seldom)? If it's broken so that unit tests, functional tests or acceptance tests fails it will be discovered (and fixed) quickly. If it's broken in such a way that it's only found during manual testing, you would probably have that happen anyway, and sometimes it's broken without us knowing - we call that a production bug.
There is this misunderstanding that "always releasable" means "mainline never broken, always pristine, represents production, etc". "Always" is not black and white! While we should strive to minimize the time mainline is broken (because it blocks everyone else) the goal is not to eradicate all mistakes (impossible). The purpose of CI, CD and TBD is fast feedback *IF* we broke something. It's okay if it was due to an honest mistake, if it was due to sloppiness then the team have some difficult learning to do. In both cases we have learned something we can act on. To para-quote Jez Humble: _"Every build is a release candidate. The goal of the CI/CD pipeline is not to prove that it works, it is to prove that it does _*_NOT_*_ work. If you can't prove that it doesn't work you should be able to deploy that version (if you choose to)."_ .
As I have written elsewhere, you don't release a branch, you don't deploy a branch, you deploy a good known *commit/version* of your code. If the mainline is broken the same hour that we need to deploy to production - it doesn't matter - because we *always* use/deploy the latest *known* working commit/version. Version control is not a file server and a branch is not a directory, it is a versioned graph of snapshots in time so we can use whichever one we'd like.
@@ddanielsandberg yes, you're right that I'm probably making it much bigger deal than it is. A commit can be reverted. But still it's something that causes inconvenience to the whole team and sometimes it's not necessarily obvious WHICH commit broke the build etc. It is some time lost, but yes, in most cases it's probably not such a big problem. I'd say it depends on the culture of the team on how big of an issue this is.
I have other issues with committing directly to trunk, I don't consider pair programming enough of a code review, I want multiple people, in fact ideally all team members, to review the code. That's not only to get feedback, but also so that other devs are aware of what's going on in the project, have the opportunity to learn something, etc.
In a team with excellent culture, it is possible to review the code after it was pushed to the trunk, but it's not as ideal, if subpar code gets pushed to the trunk as opposed to that code never reaching the code base at all and a proper version instead being pushed to trunk in the first attempt (which branches allow to do).
Or it is possible to share a patch or use some other non-standard tools to request review before pushing to trunk.
Those problems are solvable, but they require extra work and really good team culture.
I'm sure such teams exist, but by far they'd be in minority.
In my team, it's me (team lead) and 5 other junior to mid level devs. Pair programming would not help, because any pair not including me would end up pushing code with problems (I end up catching bugs in the code review quite often, silly mistakes in structure of the code, etc.)
I simply don't have trust in my team members that they'd push proper code to trunk. Not because I'm distrustful, but because I know their work, know the limitations of their work and it would end up adding more work for me, if they could just push whatever to trunk before me having any chance to review it.
CD, i'm new to programming and I just wonder, you often say you expect these CI branches to last rought around 15-20 minutes, or a day a most.
So my question is: What if the feature I'm refactoring or working on is incomplete? like say we're making a video game, and I decide that the jump mechanic needs work - just doesn't feel good to jump in this game. Well 20 minutes into doing it over, i realize I need to spend some hours researching the subject and how other games did it. Do I commit, or possibly spend a day or two being thorough in my current approach to the jump mechanic? meanwhile, other devs are building level design choices based on the jump mechanic that depends on it being what they thought it was.
I've asked in one of your previous videos about the process of reviewing changes to a code base with continuous integration. I'm a bit confused about what the workflow would look like. I work with a small team, so cashes with merges are far and few between, but the most important part of our workflow is the review process we get when we ask for merge request. What does it looks like? How do you get that feedback from your teammates? Specially since that feedback happens when you got hands around the changes and it's all fresh in your mind, not 3 weeks from now when I moved on and a teammate happens to look at that part of the code.
Yeah, I am in a similar situation. I am working on a small team with currently 2 other developers. We do use feature branching and when we are done, we issue a pr requesting a review from the others. Even, or perhaps especially, in a bigger team I think reviewing code is important.
It also helps you seeing what impact changes from others may have to your own branch and knowing what you will need to adapt. With CI, I feel like you might run into situations where you see a conflict and don't know what the other's changes are for and how to adapt them, whereas a more complete view of there changes may help with that. That being said, it is generally obviously easier to merge tiny commits.
I should also mention, while we usually have one, and exactly one, bigger feature in development at all times, the other branches are usually pretty small and never last more than a couple days or maybe a week in some cases, so there are never big discrepancies and when working on the bigger feature, it is feasible to pull from master.
We work in feature branches on user stories, not epics, and I feel like this video might talk about the idea of developing epics in a feature branch, which I would agree with is not a good idea and would make the review process quite cumbersome as well.
I also have this question. If we want to be continuously ready to deploy, we need to have some limitation on developers just pushing to master. Even if we do pair programming (which is another big change to the development workflow) just those two developers may not be enough to confirm a change is correct.
At my work we are trying shorter lived branches. So still basically feature branches, but putting it up for review/feedback/merge before the whole feature is done. Aiming to have branches live a few days at more. It does take more time for reviews, but I think it is worth it for the more frequent feedback and avoiding conflicts.
@@edwardweir5784 Well, the techniques all work together. If you're doing TDD, you should have some confidence that the code is correct. If you're doing pair programming, the code gets reviewed before it goes in. And if you're doing CI, the code goes into the main branch frequently and you don't get surprised by sudden big changes.
You get feedback by communicating with each other. A feature branch in itself won't review your code for you, and CI won't either.
If you want to set up a code review process, you can literally do it however you want with CI. Use the method you want and one that works for your team. It's that simple.
@@markjreed even if you do TDD, that still can mean, that a developer doesn't write a test together with the new feature, so the new code could be incorrect, or the test itself could be incorrect.
Linus created git to protect his kernel from (insert your preferred adjective here) jrs and Dave says to accept them daily. I think you two need to get a coffee together.
Dave believes in a CI pipeline, which would serve the same purpose, but better. There are people in a proper pipeline. :-)
I wish people would stop comparing every project with the Linux kernel. It's vanishingly unlikely most people are going to do anything like that - and yet the entire industry is out there gawping at Linus and copying everything he does as though it's gospel. 1) What he does is not the only way. 2) Linus is not a team player. He is a successful asshole that gets away with being an bully by having a successful, widely-used project. Most people need a team - with team players on it not assholes. 3) Linus can actually be wrong about things. 4) There's a big difference between working on a well-integrated team of trusted people and being a Linux kernel maestro assisted by a team of trusted lieutenants taking patches from thousands of the great unwashed of the world. There is actually a cost in terms of time and attention in serving as an open source project - that isn't necessarily appropriate even for most things.
But I do have a question. If I have to create features, that can take more than one day, I must create a new branch, and create a Merge Request, right?
Or even if my feature takes more than 2 days, should I merge it into master without review from other developers?
I agree with CI, but I have some doubts about large features and hotfixes.
The client I'm working for they have a controlled feature release, so constant merging seems impossible for this case. We only merge if the feature is business approve else we don't. Is CI can be implemented with this case?
I think I can see the point you're making, but I don't think not branching is the way to go if you'd like to be able to always deploy the head of your main branch (master of development). In mature projects there can be huge CI pipelines, which are not desirable to run locally. If this pipeline determines I introduced a performance problem in the code for example, I would not want to have this in my main branch. Instead I'd want to fix it in my own little sandbox (my feature branch) before merging.
In companies where I worked, the burden of proof was always with the person trying to merge new code. Thus, you merge the main branch into your feature branch regularly and you fix any problems that are introduced (in your feature branch, not in the main branch) before merging your branch.
Besides that, I'd recon this is why we have all those scrum rituals to inspect and adapt.
What about things like ROS where you have different distributions that behave differently? Typically, some packages will have different versions on different branches depending on your distro.
Where I work we use feature branches.. one thing that happens from time to time is to remove a particular feature branch from a release candidate... good luck doing that if you constantly committing to master.
Well it is easy enough if the design is nice and modular. Version control doesn't fix this problem, because if you need to merge your changes with other peoples, you still have to disentangle them to revert. This is much more about good design than it is about VCS strategy.
@@ContinuousDelivery Appreciate the reply! Find your videos challenging but I am trying to keep an open mind. I'd guess what I am doing according to TDD is an antipattern. I rebase all of my commits in my feature branch into a single or fewer commits instead of filling git history with 100 commits for a single feature. I feel like I'm dong part of TDD as far as starting with the test cases first.. I guess I'm trying to wrap my mind around how to implement TDD fully and thinking of the use cases that I do go thru during the release process. For example merge hell for feature branches that are really old. .that is one major drawback. Thanks for the videos!
For example with feature branches.. if my boss decides for whatever reason to remove a feature from the release.. we can create a RC branch without merging that feature branch.. if everything is always committed to master.. that is going to be a lot of work. If things go really bad.. I can just destroy the RC branch and try again (leaving out any feature branch) from that RC branch. I guess that is my point.. thanks again!
What you left out, is that for me doing CI (as you describe it) offers a "normal" team, to become a much better team. Yes many developers think they are the best in the world, yet not everyone can be "the best in the world" and by doing CI with "not the best developers in the world" you get results that out perform much better developers that are branching. And not just once, all the time (because you don't have to spend time on the branching part)
make sense to me, but I have this question, I have worked in real big companies with hundres of developers around the world, 24/7 coding and adding stuff, using the branching approach, some devs (no me :P) sometimes pushes their broken branch to master, blocking the rest of the team, with CI that will happen too, and even more often is my suposition, Thanks for the video.
Rigorous automated testing is imperative when it comes to CI. Devs need to pass all the automated tests before they push to master.
in providers like Github and Bitbucket there exists settings and controls to ensure that tests pass after merging. there's also a practice for the team to fix a broken build immediately if it fails
for example see
GitHub.com -> Administering a repository -> Defining the mergeability of pull requests -> About protected branches
What about the argument of atomic commits? Should every push to trunk be guarded by tests? If so, wouldn’t the scope of tests dictate the frequency of integration? Is it therefore better to base integration on a set time frame, or on set scopes? How would you determine scope?
Yes, every commit should be covered by tests. Yes that means that a very effective approach is to allow tests to dictate the commit frequency. If you practice TDD, as I do, that means that my commit frequency is usually roughly once every 10 or 15 minutes. That is pretty close to "continuous" in a practical sense.
I am often a single developer. I feature branch even when alone because in git there is often little to no overhead in doing so, the tooling for it is excellent. And most importantly, when I feature branch I feel more confident making larger changes. You want to be able to commit often so you can reasonably bisect. But you don't necessarily want to commit often on master because of the hassle of cleaning it up. As an IT professional, I am not sure how you live without PRs.
Ok, fair enough, feature branches are cheap in Git. But they cost your effort to create them and to delete them after they are completed. And the benefit is IMHO none, the same can be achieved with normal series of local Git commits, and an eventual rebase (with cleaning up the commit history) before the push. You are isolated on the same level during these series of local commits as you would be in a feature branch.
@@miletacekovic I have no idea what you are talking about. It takes literally 5 seconds to create them, and they are deleted automatically by default when you merge them in most modern environments, like Azure DevOps, Github, etc. It seems like you do not work in an environment where PRs are mandatory and for a good reason. You can also decide to squash by default, but that makes it harder to bisect. "Cleaning up commit history" makes to harder to find bugs. Also, integrating main/master/develop is a single click in most modern environments.
Single developer why do you even need to create feature branch? Doesn't make any sense.
Why most people create feature branch? Because they're working with other people and the need to review others code before allowing them to merge. In short, there're lack of trust and this creates more work, process or things to do before a work can be done = delay, less efficient.
In the past, before version control or for companies that didn't uses version control, they had to integrate the code manually, and devs continuously find ways to make the integration easier by doing frequent code merge, tries to make the merge easier (e.g. new functions at the bottom), refactoring existing code to communicate constantly to ensure the team get the latest changes immediately and avoid conflict. In the past, it's also very common for multiple developers to work on a single file with thousands lines of code and there are no version control yet they can manage the integration efficiently. It's all because they had the same goal, to reduce conflict and make their code always up to date and they do this by constantly communicating and integrating. Somehow, with branching, are we actually working better? Or it created more delays, more conflicts, more failures (since it's my branch, I feel safe to make mistakes or break the build in my branch), delaying review because I need time to review a huge commit and I need to make sure the other developer are doing it correctly before I would approve the PR.
@@Gaming214-y3g It takes literally 15 seconds to make a feature branch. If I am going to make large changes to a project where I want to commit during the process, I obviously make a feature-branch even if I am just one person. Integrating "main" or "master" or "develop" or whatever you are considering merging to is extremely simple in most cases if you do it often enough. Are you trying to say that if you were to, say, move a game from Box 2D physics engine to the Unity internal physics engine, you would do that on the main branch? What if, in the process, you found out that there is an issue with the internal engine? If you have not committed to main because it either doesn't compile yet or it doesn't meet your expectations, you are basically coding without version control. If you have committed constantly to the main branch, you have a messed up main branch, will need to hunt in tags/commit history to find something that builds and you will get the ugliest and least sensible commit-history possible. Branches just fix all of this and that is what they are there for. I will probably never understand why (some) people are against them.
@@larsthomasdenstad9082 what you said, doesn't make any sense? By having branch, you're saying it's ok to make mistakes? Do you mean, you won't merge to master and later found out there's a problem?
This is not the purpose feature branching was created for, but having feature branching , it created lots of bad habits and quality that were not desirable for developers, e.g. not ensuring your code is working before pushing, delaying integration, keeping bully large commits, having the safety aka 'insurance' that is your own branch so that you don't break master.
Question: how many single developers actually use feature branching?
When feature branching is needed? When you have open source and someone would like to contribute a change, they will create a PR. You can't just let anyone merge to master.
In a company, where you work in team, you'll eventually find out that the feature branch is just some safety net to prevent someone to merge because... Of trust, confidence in others... But what if they make a mistake? Or so what? Fix it. The idea is to become a more mature team, learn from mistakes and become a better programmer.
For single developer, there's no reason you need branching. By having branching, it creates bad habits, the problem you mention, it's not because of branching, to be honest. It's hard to accept this, I've work with most developers who argues the need for branching but the truth is, the problem is the way they work, the habits and it's hard to them to acknowledge them because everyone wants to be right and entitled to their own opinion. We're all adults, we should think for ourselves, ask ourselves questions, look at the problem and how to improve and get better.
You are correct. I was hired to manage and fix this exact situation. Clashes between developers was causing awful customer experience. Since a feature can take 10 minutes or 4 days, this WILL happen and more often than we would like. Thanks for the great video. A. Amburn, PhD
Great videos! Thank you.
Thanks :)
Hello, I get the idea and I don't disagree. I think we'd need some detailed examples though, about how to do it. The rationale seems as correct as it is high level. The practical level is, I believe, what keeps many doubtful.