Workflows that work

Write the documentation today that will help you work more efficiently tomorrow. Learn how to create your own documentation and walk through the process of creating and deploying a simple software product. Read Chapter 4 from Git for Teams here.

By Emma Hogbin Westby

March 15, 2016

Kings Cross (source: Pixabay)

I love working with teams of people to hash out a plan of action—the more sticky notes and whiteboards the better. Throughout the process, there may be a bit of arguing, and some compromises made, but eventually you get to a point where people can agree on a basic process. Everyone gets back to their desks, clear about the direction they need to go in and suddenly, one by one, people start asking, “But how do I start?” The more cues you can give your team to get working, the more they can focus on the hard bits. Version control should never be the hard part.

By the end of this chapter, you will be able to create step-by-step documentation covering:

Learn faster. Dig deeper. See farther.

Join the O'Reilly online learning platform. Get a free trial today and find answers on the fly, or master something new and useful.

Learn more

Basic workflow
Integration branches
Release schedules
Post-launch hotfixes

This chapter is essentially a set of abstracted case studies on how I have effectively used Git while working in teams. You will notice my strong preference for Agile methodologies, in particular Scrum, in this chapter. This process for collaboration works well with the popular workflow model, GitFlow. If you are already very familiar with GitFlow, you should still read the first section in this chapter on establishing and documenting your team’s procedures.

Evolving Workflows

In not available, you learned about governance models, and in not available, you learned about branching strategies. The way we work together through Git can get quite complicated very quickly, and the greater the complexity, the harder it is to remember how it all works. Establishing conventions with your team will help to maintain consistency, which will help you to quickly decipher the history of your code.

In this section you will discover:

Basic tools to document your team’s process
Where documentation should be placed
What types of things need to be documented
Sample states for your ticketing system

It is never too late to talk to your team about how they want to work together, and it’s never too late to improve on the processes you have in place. If you are using Agile methodologies, you may already have dedicated time for retrospective meetings, or Kaizens, to review your development process.

Documenting Your Process

Git, as an inanimate piece of software, doesn’t actually care how you set things up. Rest easy, because Git won’t suddenly reach out from your computer and wag its finger at you crossly if you use the wrong branch name or use merge when you should have rebased (although sometimes I think it would be nice if it did). It’s up to you to decide how you want to use Git.

The easiest way to be consistent is to follow a set of rules, or a checklist. Each time you begin working on a new site you should document the workflow. By starting from a template (Example 1-1), you will ensure “obvious” details are still obvious when you onboard new people, or even better, in a moment of crisis.

Example 1-1. Template workflow

Product Manager: Name
Dev site: URL
Branch deployed on dev site: name of branch
Live site: URL
Branch deployed on live site: name of branch
When starting a dev ticket, branch from: name of branch
When starting a hotfix ticket, branch from: name of branch
When updating your work, use: git command
When merging your work post review, use: git command

The more details you include in your documentation, the more consistency you will have among your teammates, and the easier it will be to unpack the historical record of your repository.

If you are collocated, sit down and sketch out the diagram of where the permission divisions should be made in your code. If you’re a distributed team, that doesn’t mean you can’t still sketch things out. And you don’t need to be an illustrator. There are lots of decent diagram programs out there to help you sketch out your ideas. I’m a fan of Balsamiq for very basic diagrams. Others have also recommended Pencil, OmniGraffle, Dia, and Inkscape. The diagrams from not available will be a useful starting point for many teams. All of the diagrams from this book are also available as both SVG files and Balsamiq files. You can download them from the Git for Teams Diagrams repository.

Documenting Encoded Decisions

Throughout this book, I will talk about working on tickets, or issues. The rigor of open source software projects has enforced more than a few good work habits, one of which is the use of a bug tracker to capture all requirements. For open source projects, I’ve used product-specific trackers, such as the Drupal Project module (affectionately referred to as The Issue Queue); and generic solutions, such as GitHub. For internal projects, I’ve also used Pivotal Tracker, JIRA, Redmine, and Unfuddle, among others.

Each of these systems has positive and negative aspects. I don’t have any one favorite product. At their core, these systems allow you to document and track the discussion of the work to be done, the tasks that need to be completed, and a summary of any follow-up issues that may have been discovered during quality assurance testing. I cannot imagine working with a team where there wasn’t a centralized ticket tracker capturing the information about the work being done.

Collocated teams may choose to use a whiteboard and sticky notes to show what is currently being worked on. Some teams also use very simple spreadsheets to track who is currently working on what task. Perhaps the conversations and related assets (e.g., diagrams, design assets, wireframes) are stored in a wiki so that whiteboards can be wiped down and used for the next conversation. No matter which system you use, I encourage you to track at least the rationale for the decisions that are made about why features are being built in an easy-to-read and searchable system. If you don’t capture this information in writing somewhere, you may have to resort to guessing about why decisions were made in the past.

Using ticketing systems, however, can make teams dependent on sticking with that particular system if the decisions aren’t also captured in the commit messages for each change to the repository. Your team may choose to think of the conversation as ephemeral, tracking conclusions in commit messages and allowing themselves to move on from the conversation itself.

It’s a balance. The trick is to anticipate future conversations and ensure your tracking system has a way to easily answer questions. Perhaps you want to prevent a future developer from forcing you to rehash a conversation after a decision is made. In this case, you’ll want a ticketing system that shows the progression of arguments from both sides (as comments) as well as the final conclusion, and a link to the commit where the decision was solidified as code. Perhaps you are creating software that is subject to industry regulations and you are required to prove that software has been through a specific review process. In this case, it may be sufficient for your software repository to have signed commits from individual quality assurance testers.

I don’t think there is any one system that is better at tracking software development. Many have strengths, and they all have their limitations. If you are using a specific process management philosophy that advocates a specific task workflow, you may find it easier to use software products that have been optimized for this process. For example, a Kanban board is a very specific way of dealing with tasks.

Most of the Git hosting platforms also have a basic ticket tracker to help you coordinate the development of your project. not available covers three of these systems (Bitbucket, GitHub, and GitLab) in greater detail.

Ticket Progression

Even if you are working on an internal project without fixed deadlines, I recommend finding a small unit of time to iterate through. My personal preference is for one-week sprints. For internal projects, these sprints can act as arbitrary deadlines to keep the team motivated and moving forward. At the end of each sprint, I recommend hosting an internal demo so that the team can show off their work. This public display of work keeps developers accountable. If your team is distributed, you can host these demos over Google Hangout, or GoToMeeting for larger teams.

Project methodologies that track the work of people will all have some variation of these basic ideas:

Not Now: In Scrum terms, this would be referred to as the product backlog, Essentially, though, it’s anything that has not been deemed relevant for this work effort (or sprint). Developers should not pick from this list of tickets. The backlog should be prioritized to give hints to the team on what should be worked on in the next work sprint. Recently, a team that I worked with referred to this as the “super very important for later” pile.
Ready for Work: Prioritized tickets for this work iteration. These tickets might be blockers for tickets in the backlog, or simply be the next piece the team has chosen to work on. Your team may want to subdivide this stage into separate subcategories, such as: Ready for Development, Ready for Code Review, Ready for Testing, Ready for Client Approval, and Ready for Deployment.
In Progress: A developer is currently working on this ticket, or a quality assurance review is being done. With larger teams, you may want to break this category down further as well. For example: In Definition, In Development, and In Testing.
Completed: The work has been finished, or has been canceled. Perhaps there were follow-up tickets, but only very rarely should a ticket be reopened after it has passed a code review, quality assurance review, and a client review.

Warning

Do Not Allow Your Project Managers to Overcategorize!

Allow your team to grow into states as needed. I have worked on too many projects where a team of project managers had decided on a range of categories that described every possible state. The system was always cumbersome to use. (And I am a category loving manager!) The developers never liked trying to remember to micro-shift their tickets, and, more often than not, the tickets weren’t in the right state unless a project manager was the one moving the tickets through the progression of states. Have compassion for developers who want to develop, not spend their day updating timesheets and micromanaging ticket updates. Start simple. Make as few categories as possible. As the team of developers asks for new states, add them.

As an example of a variation, the team I worked with in the fall of 2014 had nine people working in the ticket tracker on the tickets throughout the project (a relatively small project, but a typical team size for Agile projects). The ticket tracker had summary columns for the following statuses:

On Deck: This ticket is ready to be worked on, and should be completed during this week’s work.
In Progress: This ticket is actively being worked on.
Pull Request: The code has been written, and is ready to be reviewed and merged into the main branch.
Needs Testing: The code has been reviewed, and rolled into the development branch. It is ready to be reviewed on the quality assurance server by a team member.
Done: The ticket is completed. This state is also used for tickets that are closed without being completed (duplicate task, feature no longer needed).

The backlog was simply a collection of tickets without a status assigned.

If a developer was ever blocked, he or she would reassign the ticket to the person most likely to “unstick” the issue. Getting into the habit of trading tickets to communicate with others is a cultural piece that won’t work for all teams—but it does seem to work well for distributed teams where you can’t just tap someone on the shoulder to get your questions answered.

I love a categorization system more than the average developer; however, adding complexity has consequences. Complexity increases the time it takes people to decide which variation their ticket currently belongs to (“is this Needs Testing or Pull Request?”). It also increases the number of times developers have to open the ticketing system, instead of their code editor. This has the potential of both improving communication with other developers and slowing down the actual doing of the work. You’ll need to monitor this closely to see where you can make refinements to improve your own process.

Tip

Pick Your Own Battles

Teams I’ve worked on have responded well to developers being able to self-assign at least a few of their own tickets. Sure, there may be some tickets that require the specialized knowledge of one person, but it’s amazing how much of a difference it can make when it’s that one person who identifies he or she needs to work on that ticket instead of being told what to do.

It is near impossible to over-communicate with your team members. I don’t mean filling your time with unstructured meetings; I mean truly communicating what you are working on, and what is preventing you from getting your tasks completed. The ticket status helps you to standardize the communication—so make it easy to keep up to date, and ensure everyone on the team gets into the habit of confirming their ticket status once a day.

A Basic Workflow

This basic workflow is appropriate for small teams of one or two trusted developers. As was mentioned in the introduction, it is a stripped down version of GitFlow; but without the extra levels of complexity, it also resembles a branch-per-feature workflow. As such, you may find it also works well for teams of developers with a testing infrastructure, who are aiming for rapid deployment of code.

Key characteristics include:

Governance model: contributors with shared maintenance
Integration merge: performed by original developer
Integration branch: develop

My personal preference for this workflow is closer to a Kanban-style system, which allows tickets to flow through a work board; however, I find it much easier to communicate plans to outside stakeholders by using the Scrum approach to time-boxed sprints. In Scrum, a specific set of tickets is loaded into a sprint and the goal is to get the number of outstanding tickets down to zero by the deadline. For internal projects, Scrum-style sprinting can act as arbitrary deadlines to keep teams motivated and moving forward.

At the end of each sprint, I recommend hosting an internal demo so that the team can show off their work and ask for help from the wider group if they are stuck on a specific piece.

The workflow is as follows:

As you begin a ticket, update the status in the ticket tracker to say the ticket is In Progress. This will notify your team about what you are currently working on, and will give you the number for the branch you will create to work on your ticket.
From the branch develop, create a new branch whose name includes the ticket ID and a terse description of the work. If you are working on tickets that have subtasks, ensure the branch name uses the most relevant ticket number. For a bigger feature, this ticket might be referred to in your ticketing system as a Meta ticket or Epic ticket. If you are working on only part of the larger feature, you should use the smallest relevant ticket number. Your ticket system might refer to this as a user story, an issue, or bug ticket.
Work on your ticket, ensuring you keep the ticket branch up to date with any changes that might have been incorporated into the branch master since you started your work. Begin each commit message with the ticket number enclosed in square brackets: [#1234].
Run relevant tests for your code to ensure typos and basic errors are caught. This may include a spellcheck, and a language syntax check (linting). If you are working in a test driven environment you will definitely have additional tests to run.
When you have completed your work (or think you have!), make a final commit with the keyword “Resolves” and then the ticket number: Resolves #1234.
Optionally, push your ticket branch to the code hosting repository. With the keyword in place in your commit message, this will move your ticket tracker forward to the next step.
In your ticket tracker add a comment to the ticket to include a note about the rationale for the approach you took and some kind of proof that the work was been completed. For example, a screenshot of how the ticket changes the display on your local development environment. This acts as a sanity check later if, suddenly, things stop working.
Ensure the ticket branch is up-to-date and then merge your work in the branch develop, and, assuming there are no merge conflicts, push the updated branch to the central repository.
Assuming there were no new problems introduced by the new work (regressions), the ticket can be closed.
Finally, delete your local ticket branch and the remote copy of the ticket branch.

Tip

In some ticketing systems, adding a pound sign (#) will automatically link the commit message to the ticket number. Adding square brackets around the ticket number will ensure that commit messages aren’t omitted if you choose to rebase your work because lines beginning with a # are ignored when commit messages are automatically composed for the new commit objects. In many systems, including the keyword “resolves” will automatically move a ticket from In Progress to the next state (for example, Needs Testing or Closed); this will vary depending on the ticketing system you’re using. Check the documentation for whatever system you are using.

This pattern works extremely well for small teams with no peer review requirements. As your team starts to grow, or if you have a specific quality assurance process you need to undergo, you may find this pattern is not rigorous enough for your needs.

Trusted Developers with Peer Review

This expands the basic team workflow by adding a peer review process. Now, every ticket is reviewed by someone on the team from a code perspective. The rationale for peer review testing, and not just automated testing (or test-driven development), is covered in not available.

Key characteristics include:

Governance model: contributors with shared maintenance
Integration merge: performed by the reviewer
Integration branch: develop

The workflow is as follows:

As you begin a ticket, update the status in the ticket tracker to say the ticket is In Progress.
From your local copy of the branch develop, create a new branch.
Work on your ticket, ensuring your branch is kept up to date with rebasing. Begin each commit message with the ticket number enclosed in square brackets ([#1234]), or with the keyword “Resolves” and then the ticket number: Resolves #1234.
Run relevant tests for your code to ensure typos and basic errors are caught. This may include a spellcheck, and a language syntax check (linting). If you are working in a test driven environment you will definitely have additional tests to run.
Push your branch to the code hosting repository. This acts as your backup, so don’t skimp on this step!
When you’ve finished working on your ticket, ensure the branch is up to date with develop, and uploaded to the code hosting system. Mark your ticket as Needs Testing in the ticket tracker.

Assuming a manual review is necessary, and there isn’t a series of automated tests, the reviewer will finish off the remaining steps:

Perform a review of the work according to the original ticket description. It is the coder’s responsibility to ensure his or her work is clear, and that the steps to test the work are coherent. If necessary, send the ticket back to the developer with any necessary changes, or to bring the branch up to date if it has gotten out of sync with develop.
Merge the ticket branch into the branch develop, and, assuming there are no merge conflicts, push the updated branch to the central repository.
Assuming there were no regressions, the reviewer will now close the ticket and notify the developer that his or her work has been merged into the main branch. Both the developer and the reviewer can now delete their local copies of the ticket branch. Because they are currently in cleanup mode, the reviewer should delete the remote copy of the ticket branch; the developer might have to break focus in the current task to do the cleanup. Wherever possible, we should protect the focus, and flow, of our teammates.

Once your team is large enough to have a review process, it makes sense to also have a shared development server from which the team can conduct regular demos of their work. This development server can also double as a quality assurance machine during the development process. To reduce the overhead for team members needing to check out the latest version of the develop branch and build the software, you may choose to set up a Jenkins instance to automate the process.

Untrusted Developers with QA Gatekeepers

This process is a minor variation on the previous section Trusted Developers with Peer Review. This time the process assumes an untrusted developer, who is not allowed to merge anyone’s work into the main branch. Instead, a trusted quality assurance (QA) team performs the final merge.

Key characteristics include:

Governance model: contributors with collocated repositories
Integration merge: performed by the reviewer
Integration branch: develop

Developers begin by creating a fork of the project on the code hosting system, and then creating a local clone from this forked copy of the repository. This step is only performed once.

The workflow is as follows:

To begin a ticket, update the status in the ticket tracker to say the ticket is In Progress.
From your local copy of the branch develop, create a new branch.
Work on your ticket, ensuring your branch is kept up to date with rebasing. Push your ticket branch to your fork of the project as a backup of your work in progress.
Run relevant tests for your code to ensure typos and basic errors are caught. This may include a spellcheck, and a language syntax check (linting). If you are working in a test driven environment you will definitely have additional tests to run.
When you’ve finished working on your ticket, ensure the branch is up to date with develop, and push your work to your forked repository. Open a pull request (in some ticketing systems, this might be called a merge request) for your work.

The reviewer will finish off the remaining steps:

Perform a review of the work according to the original ticket description.
On the main copy of the repository, accept the pull request. Depending on the ticketing system, this might be done via a web UI, or in a local clone of the repository.
Assuming there were no regressions, close the ticket. Because the work was completed in a fork of the main project, there is no additional cleanup in the main repository.

This approach also works well if your team is mostly trusted developers, but you have a few contractors as well. You might want to have your contractors working in a fork of the repository, instead of giving them write access to the main project. For some types of software, this split might even be a requirement for your own staff. For example, if you were working on firmware for a medical device, you might have very strict government regulations you need to follow on who is allowed to check in work, and how that work must be reviewed before it is added to a repository.

Releasing Software According to Schedule

The vast majority of the projects I have worked on have used a release schedule to expose new versions of the software to its users. The process described in this section is based almost entirely on the very popular workflow, GitFlow. If you are deploying continuously, and do not collate multiple tickets worth of work into a specific release, this section will not be relevant to you.

Publishing a Stable Release

Up to this point, all of the examples have been working from the branch, develop. Eventually, though, you’ll want to release the product you’ve been working on. When you’re ready to do this, you will need to split your repository into a public-facing stable version of the product, and a developer-facing “no guarantees” version of the product.

When the first version of your software is launched, a development manager will prepare the repository for a code release. Generally this work is done locally, and then pushed up to the main copy of the repository.

The workflow is as follows:

From an agreed-upon commit, create a new branch named master.
Tag the agreed-upon commit with a version number with an easy-to-remember naming convention. For example, v1.0.
Push the updated repository to the central code hosting system. If an automated build process is not being used, update the relevant servers with the new code.

Once the first release has been published, you will now split your work into stable, public-facing work and ongoing development.

Ongoing Development

Once an official release of a product has happened, your team will effectively be forced to think in two separate spaces at once: monitoring the health of the live code, and continuing the development process to add new features or improve those that already exist.

My preference, again, is for short work sprints. Developers are motivated to see their work in action. The longer the sprint, the longer people have to wait to see others engaging with their work.

The one-week release schedule I commonly use has the following routine. The days vary a little from team to team, but the generalized procedure is a good starting point for many teams.

Setup (Mondays):

All work in the develop branch is merged into the testing branch, qa. Any work that isn’t completed and peer reviewed by Monday simply remains in its ticket branches.
The testing server is updated with the latest version of the qa branch.
A QA checklist should be created for each of the user stories completed in the last week of work. A standardized ticket format will make this list easy to compile.

You may want to compile your QA checklist in a separate document, such as a Google Doc, or an internal wiki. I’ve also used saved queries in JIRA to look for tickets resolved in the last week, or which have been tagged for a specific release. This will depend entirely on how you choose to track progress in your ticketing system.

Testing (Mondays and Tuesdays):

Automated tests are run to ensure no new business-critical interactions have suffered regressions (site visitors and members can still use all expected functionality).
Team members responsible for testing complete the checklist and update the tickets according to a PASS or FAIL grade.
Any bugs that are found have new tickets opened and are addressed before launch day by either a new fix, or by removing the relevant commits from the qa branch.

Launch Day (Wednesday):

The qa branch is merged into the master branch and tagged with the release version.
The live site is brought up to date by checking out the commit on the master branch, which has been designated as the newest release for the project. Using an explicit tag ensures you can easily roll back to a previous known state.

Announcements about the latest features and fixes are posted to the development blog. Many teams choose to wait a day or two after Launch Day before publishing the blog post. This allows the team to ensure the release is stable and does not need to be reversed.

Post-Launch Hotfix

Sometimes deployed code has mistakes in it. If a bug needs to be fixed quickly before the next batch of software is ready, an out-of-cycle fix might need to be made. These deployments are referred to as a hotfix.

In a hotfix, the work begins not from the develop branch, but from the master branch. This ensures the changes only introduce a fix that addresses the problem identified in the deployed code.

The workflow is as follows:

To create a hotfix branch, start by checking out the master branch, not the develop branch. This will ensure that no additional features accidentally sneak into the fix.
Generate a list of tag names, and locate the latest tagged release.
From the latest tagged release on the master branch, create a new branch, using the branch name hotfix- <ticket_number>–<description>. For example, hotfix-1234-fixing_three_seat_issue.
Complete the same review steps as you would for a development ticket.
Merge the tested hotfix branch into the master branch.
Tag the new commit on the master branch with the latest release version. For example, v1.0.1.
Merge the tested hotfix branch into the development branch so that the changes are not lost in the next official release of the software.

Collaborating on Nonsoftware Projects

Git isn’t just for software developers! As a technical author, I’ve used Git a lot to track changes to files that weren’t software—for example, configuration files, articles I’m writing, and even this book! Some people even use Git to maintain a personal journal. To illustrate the importance of matching the Git commands to the team’s process, let me explain how I structured the repository for this book.

While writing this book, I worked with the O’Reilly automated build tool, Atlas. This system also has a web-based GUI, which allows editors to work on book files directly, and saved files are immediately committed to the master branch. Due to the GUI, there is no peer review process because anyone on my team is able to make edits directly to a file. My preference, however, is to work locally, and not through a web GUI. Initially, I had been keeping the branch overhead low locally and had just been working in master as well. It only took me one merge conflict to alter the way I was working locally.

When I wanted to update my work, I would use the command fetch to see if any changes had been made by my editors. With the fetch completed, I would compare my copy of the master branch with their copy of the master branch (origin/master). Assuming I agreed with all the editors’ changes, I would merge in the editors’ copy of the branch; if I disagreed, I would merge in their branch with the strategy --strategy-option=ours, effectively throwing out their changes but letting Git think that the two branches were merged.

This can be done on a per-commit basis, or if there is a merge conflict, it can be done on a very granular line-by-line basis with a merge tool. (It feels a bit passive-aggressive to be throwing stuff out, but really it’s just the limitation of a single branch system where you don’t have the ability to talk about the proposed changes in a separate branch.) Depending on the granularity of the commits, I might also choose to cherry-pick some commits (and keep them), but discard other commits.

Then I started getting reviews as marked-up PDFs and realized, once again, I had another way that I wanted to separate work. I wanted to be able to write a chapter and keep those commits nice and tidy, but sometimes I was mid-chapter when an edit came in that I wanted to address immediately. Instead of intermingling these commits, I set up the following structure for my branches: master, drafts, and one branch per chapter.

The branch drafts gave me a place to integrate all of the work that I’d been doing. It was kept up to date by merging in chapters as they were completed, or rebasing the master branch if changes had been made by one of my editors. When I was first writing chapters on my own, without others contributing, multiple branches would have been a lot of overhead to maintain, but as more contributors started offering different kinds of contributions, more granularity in branches allowed me to pick and choose how I wanted the manuscript to progress.

As you can see, my process differed wildly from the workflows used for software projects, but it’s still Git that I’m working with! Your work may have its own idiosyncrasies that justify nonstandard branches. Don’t be afraid to experiment, but when you do, document your process well so that others can understand what is expected of them.

Summary

The workflows described in this chapter have been successfully implemented in teams I have worked with. Your own team might want to make adjustments, but starting from something will be a lot easier than starting from nothing.

The workflow you use may change before and after the launch of your product.
Before launch, you will likely have fewer integration branches, because the concept of a hotfix is unlikely to be an issue.
By using your documentation to complete your work, you ensure your documentation is always up to date, which makes it faster to onboard others if you need help.

In not available, you will learn the commands necessary to implement the processes described in this part.

Post topics: Open Source