Keys to Success for Continous Delivery with Distributed Teams
Continous delivery is complicated, and adding the elment of working with an offshore team certainly presents an extra layer of complexity. Like any new concept, initial adoption can be challenging, but continuous delivery’s rewards are well worth the risks; the return is monumental. With that thought in mind, here are a few things you need to think about as you embark on implementing continuous delivery with distributed teams.
Remember that many of the most successful organizations in the world have embraced continuous delivery to provide them with a competitive advantage. Amazon, for instance, deploys new software features every 11.6 seconds [ref: https://news.ycombinator.com/item?id=2971521] . You might not desire or need that “warp speed,” but knowing that it is possible offers some level of comfort in pursuing continous delivery goals.
The number one concern of those who are thinking about using continuous delivery with distributed teams is communication. They simply can’t imagine how the added level of communication needed for successful continuous delivery will work if team members aren’t in the same place—or even the same time zone.
We live in an age where we have access to many highly advanced communications and task management tools. It’s critical to support collaboration by making people feel more connected and more accountable as well. E-mail is truly just the tip of the iceberg in terms of communications tools—use Skype and videoconferencing extensively. The addition of something as simple as video has a big impact on helping distributed teams feel and function like one team.
Rely on task management and task collaboration tools like Jira, TFS, and Basecamp to ensure everyone is on the same page. At a high level, these tools show who’s accountable for each task, which allows others who integrate or depend on that task to have a reference to return to in addition to the attached hand-o notes. Developers and testers must understand what’s being worked on, as well as the goals, tasks, and any information related to those tasks. TFS is great for this, and it also provides easy and fast status updates.
It may appear counterproductive to involve every team member in most meetings, but sharing information helps everyone understand the objectives, make good decisions at every stage of the project, and jump in to help one another as needed. Additionally, encourage the culture of informal meetings among team members whenever needed. In this case, more meetings and sharing of information aids, not hinders productivity.
Increasing the time overlap between teams in different time zones is an important way to support not just communication, but teambuilding and collaboration as well. Plan for about four hours of overlap per day between distributed teams to ensure no roadblocks occur during the continuous delivery process.
METHODOLOGY + PROCESS
It is hard to imagine implementing continuous delivery without Agile. Choose from its different flavors and employ those that are the best fit for your team and most appropriate for the specific product being developed. Agile makes continuous delivery practical, and past roadblocks to production make it necessary. As a bonus, you will reduce your productivity tax. But continuous delivery with distributed teams? For this, just as for non-distributed teams, you need to choose the methodology that works best for you.
Adopting a new development methodology like Scrum, Kanban, Lean, or a custom flavor when employing continuous delivery with distributed teams is an evolutionary process that gets rectified as you work and progress. There is no clear “right” methodology.
XP, in particular, is an Agile development methodology that can be modified for distributed teams. The following are the activities for XP that you might employ: pair programming, test-driven development, continuous integration, and collective code ownership.
Pair programming adds a bit of “spice” to the development process. Based on the premise that two heads are better than one, pair programming involves two developers working on one task at the same time— two heads, one screen. But how to achieve the “two heads—one screen” objective in a distributed team environment? Ask two engineers in two different locations to team up and share tasks. Similar to traditional pair programming, this modified approach improves code quality, promotes knowledge sharing, and with “round the clock” development there is a big uptick in speed.
One of the advantages of Agile that applies to XP, Scrum, Kanban, and virtually every Agile process, is a prioritized backlog. Adopting continuous delivery emphasizes swarming on tasks in order, with the priority being set by the product owner. It’s a top-down process that results in the first task being completed before the second is started, and so on. So the team as a whole has to consolidate their power and collective efforts to get user stories done top down. (In some Agile flavors, such as Scrum, developers have the freedom to complete tasks in any order they choose, a methodology that seems at odds with continuous delivery when mission-critical or high-complexity features are in play.) Consider layering swarming on top of pair programming. For example, assign four engineers to a user story. They pair up to run tasks in parallel. It’s a bit of a mind shift in the beginning, but when developers see the benefits they are big proponents of the approach.
Because continuous delivery results in such a dynamic environment, it’s critical to have a strong process for version control. There are many good options for tools to implement consistent version control such as using TFS or Github. Both tools have web- based versions that work great for distributed teams. This allows your team to remain organized and gives the product owner a bird’s eye view of the status of each task.
Branching strategy is another vital process component. Creating a branch for each feature and then moving completed features into the product branch is one way to go, but consider another option that works better with continuous delivery: including the features on the main branch with an on/off switch. This enables your team to have more flexibility when it comes to the order in which features are addressed—and that can be critical to being able to meet faster continuous delivery schedules.
It is worth mentioning the problems that branching strategy is tackling: developers keep adding features but not all features are relevant to the business or suitable for the market at the current moment. Allowing the developers to work independently while having means to include/exclude features from the live environment is the main goal for whatever branching strategy is adopted.
What about regression testing? On top of Agile, this must be done upon shipment, and since shipments occur daily, and even twice daily, with continuous delivery, it is mandatory to have automation testing in place to facilitate this crucial process. You’ve heard the quote, perhaps: “Automated tests transform fear into boredom.” Tools like Selenium and BDD (Behavior Driven Development) with SpecFlow are invaluable, as they can automatically test more than 95% of your team’s work, but the balance must still be manually tested.
In the fast-paced world of continuous delivery, the more automation the better, because it’s important that quality is not sacrificed in the desire to ship features at breakneck speed. In addition to saving time, having automation testing in place ensures that testing one module isn’t going to screw up others and even break them.
Most developers are comfortable with using automation to check code, and if the logic is changed, it can be verified through code. UI testing can be more of a challenge to automate, as the tools available to do it aren’t as reliable as those designed for regression testing.
There will always be a small portion of code that needs to be manually tested, because it’s either not testable via automation or it is too expensive to go that route. To streamline your continuous delivery process, it’s important to have procedures in place to address instances where manual testing is necessary.
Given the pace at which continuous delivery occurs, when you add in a distributed team environment having the right processes in place is an absolute necessity to support your team’s success. (Additional process challenges to be aware of include code quality, technical debt, automated deployments, and good health monitoring of your systems.)
Not to be overlooked as you work to implement continuous delivery with a distributed team are the cultural aspects of this decision. It’s likely you will get some resistance from your developers when you initially explain the concept because they likely have never been asked to deploy products every day. It is very typical to hear concerns from your team about how continuous delivery will affect product quality – especially if they are working in a distributed team environment.
The best way to overcome team members’ fears is to cite some of the many organizations that have successfully implemented continuous delivery—with no adverse affects. To the contrary, you can show that those organizations are capitalizing on the value of continuous delivery to provide them with a competitive advantage. Google, Facebook, and Amazon are examples of these organizations.
From there, you should emphasize the tools and communication methods being employed to ensure that the distributed team environment will not hinder continuous delivery implementation efforts. It’s not unusual for people to resist change of all types, but in this case, you must facilitate your team members’ courage to move forward and try this new way of shipping features in small bites rather than as a complete product. Focus on the fact that they will certainly be learning something as they embark on embracing continuous delivery—and if for some reason it’s not found appropriate
for your particular needs, you can always change back to the “tried and true.”
Related to our earlier example of modified pair programming, you may face the challenge of convincing developers to release an incomplete task to another person—and to someone whom they have never met! You can imagine the reaction to this suggestion. People worry about whether the developer they are paired with will understand their approach, finish the task with the same quality they produce; they are generally uncomfortable with the idea of shipping off unfinished code. Ask people to keep an open mind and test the concept. Once they see the benefits, they begin to share the idea with colleagues on other teams.
In our experience, once developers get the hang of continuous delivery, they understand its significant value and they are quite satisfied working in this fashion. It might be helpful to compare the adoption of continuous delivery to beginning a running program; the first time you attempt a five-mile run will be hard, but after you do it over and over, it actually becomes enjoyable and even fun.
Continuous delivery gives the developers and other team members a real feel of ownership–their task is their baby and it’s going to be online once they finish their work.
Along those same lines, due to the speed with which continuous delivery occurs, there is no room for blame; instead, team members must own all aspects of their work. Things may be moving so fast that it’s a bit scary, but everything that’s pushed out must be of high quality. (There is a philosophy in continuous delivery environments that an issue in production doesn’t mean that somebody screwed up–it means that the automation tests coverage needs to expand to include that scenario).
For this reason, it is important that the outsourced team feels as accountable and integrated as the internal team. Your distributed team must be up for the accountability and commitment necessary for continuous delivery to work.
Intelligent, thoughtful planning is at the core of successful continuous delivery. It also requires a slightly different mentality from the typical “whatever green light goes.” In a standard development process, developers code and testers provide QA, but the lines are blurred under continuous delivery. As a result, it is necessary to ensure that test cases cover functionality and use scripts to make sure functionality is covered by code.
It is also an environment that will enable the whole team to shine and on the other hand it exposes bad behaviors. Every single person’s work is exposed. It is not hidden behind and within other people’s work.
Working under the umbrella of continuous delivery, more than one person takes ownership of each feature, and to ensure the highest level of success, members of a distributed team must have an equal sense of ownership with your internal team members.
It is quite common to treat offshore teams like second-string players, but that simply cannot be if you want to be successful using continuous delivery. You must invest in engaging your offshore team so you expect the same level of performance from it that you receive from your in-house team. In continuous delivery with distributed teams, there can be no “B” team; everyone must be working at an extremely high level.
Your offshore team should be treated the same way you treat your internal team. You’ll find that when you do that, in addition to motivating and driving these professionals to be the best they can be, the results of continuous delivery will be amazing. You may wonder why you ever hesitated in using continuous delivery with your distributed team.
Many product owners have jumped on the continuous delivery bandwagon with distributed teams, perhaps despite having some concerns about how it would work. Through best practice and real-life experiences success can be realized.
You know that continuous delivery is no longer something merely nice to have, but a business requirement for development teams that want to get and stay ahead of the pack. You should not be wary of implementing it when working with an offshore team; certainly understand the challenges you may face, but have the technical courage to look past those to see the rewards that await successful implementation.