Developer Workflow

Original Author: Sumon C, May 27, 2021

Introduction

Below is the software development workflow adopted by REDCap@Yale, written and explained from the perspective of the individual developer contributing to a software project being developed by the team. You should have some understanding of version control using Git before proceeding. The following was designed to implement a distributed version control scheme while avoiding unnecessary administrative overhead.

Overview

REDCap@Yale follows a distributed development workflow where there exists:

1. A central "official" remote repository, from which releases and deployments to production REDCap instances are created,
2. An individual developer's remote repository, from which pull requests to the official remote repository are created, and
3. An individual developer's local repository, where the developer commits their changes to the codebase.

This is either referred to as the Integration-Manager Workflow or the Forking Workflow, depending on the resources consulted.

Definitions

The following definitions are used in this guide to clarify our process.

• Developer: a person contributing to the software development of the REDCap@Yale external module or other digital tools through feature development, bug fixes, and testing. A person in this role can contribute to a project, but the integration manager must review the submitted code before the changes can be incorporated.
• Integration manager: a person maintaining an external module or digital tool development project by code-reviewing changes submitted by contributing team developers and incorporating these changes into the project. Generally, the integration manager is the owner of the project, but does not have to be.
• Central repository: the central "official" remote repository which resides in the REDCap@Yale organization GitHub and or Yale Enterprise GitHub account. This repository contains all of the work that has been vetted and tested. Production releases and deployments to our REDCap servers will be done from the codebase that resides in this repository.
• Remote repository: the developer's remote repository which resides in the developer's personal and/or Yale Enterprise GitHub account. This repository contains, ideally, all of the work that exists in the central repository, as well as the modifications that the developer has made to the project.
• Local repository: the local repository that exists in the filesystem of the developer's local machine. This repository contains, ideally, all of the work that exists in the central repository, as well as both tested and untested modifications that the developer is making and testing on their machine.
• Fork: the process where the central repository is copied over to a developer's personal GitHub account, creating the remote repository.
• Clone: the process where the remote repository is copied over to the filesystem of the developer's local machine, creating the local repository.
• Push: the process where code in the local repository is uploaded, through Git, to the developer's remote repository.
• Pull: the process where code in either the central or remote repository is downloaded, through Git, to the developer's local repository.
• Pull Request: a request filed by the developer to have the developer's changes, housed in the remote repository, added to the codebase in the central repository, through GitHub. Once a request is filed, the integration manager reviews the request, performs more tests if necessary, and pulls the changes from the remote repository to the central repository.

Process Summary

The summarized process is:

1. The integration manager creates and/or pushes code to the central repository.
2. A contributing developer forks the central repository, creating their remote repository.
3. The developer clones the remote repository to their local machine, creating their local repository.
4. The developer makes changes within the local repository.
5. The developer pushes these changes to the remote repository.
6. The developer submits a pull request in the GitHub web interface, notifying the integration manager.
7. The integration manager either:
   • merges the changes directly on the GitHub web interface, or
   • adds the developer’s remote repository as a remote and merges locally.
8. The developer then pulls from the central repository to the local repository, ensuring their copy is up to date.

The figure above illustrates the general workflow.

One of the main advantages of this approach is that the developer can continue to work, and the integration manager of the central repository can pull in the developer's changes at any time. Developers do not have to wait for the project to incorporate their changes — each party can work at their own pace.

Detailed Developer Workflow Process

Described below in detail is the REDCap@Yale distributed software development workflow that you, as the developer, should be following when developing REDCap external modules and other digital tools used by the REDCap@Yale team. In the screenshots and code snippets below, test-repo or test-repo.git should be replaced with the actual project repository name or address, and github.com should be replaced with git.yale.edu if using the Yale Enterprise GitHub.

In this workflow, we are treating the master branch of the central repository as the "bleeding edge" version of the software, where all bug fixes, new features, and enhancements will be centrally incorporated. Various releases and versions will be created by branching off of this master branch.

1. The integration manager creates the central repository

Anyone who has access to the REDCap@Yale organization GitHub account can create the central repository. In most projects, the integration manager will create the central repository in the GitHub web interface. For further details, refer to the workflow guide for the integration manager.

2. Fork the central repository

Once created, the central repository can be forked by each individual developer using their personal GitHub accounts. You must be added as a member of REDCap@Yale.

Fork the repository:

3. Clone the remote repository to the local machine

Once forked, the remote repository can be cloned to your local machine.

First, make sure you are in the forked remote repository. Then, obtain the HTTPS address of the remote repository:

Then clone the remote repository in your local machine's command line:

git clone https://github.com/<your-GitHub-account>/test-repo.git

If you will be cloning a private git repository -- for example from your private Yale repository -- you will need to provide authentication information. The syntax in this case will be:

git clone https://<your-netid>:<your-netid-password>@git.yale.edu/<your-netid>/<repository-name>

4. Make changes within the local repository

At this point, you should have a local repository on your machine where you can make all of your changes during development.

Make changes to your working directory, stage these changes, and commit them to your local repository.

For more details about branching, git commit messages, and other development conventions, refer to the development standards and conventions guide.

5. Push local changes to the remote repository

Whenever you want the changes in the local repository to be reflected within the remote repository, push these changes:

git push origin master

master should be changed to the name of the branch that you are currently working on.

6. Submit a pull request for the central repository to integrate the changes

Once satisfied with the changes you have made and thoroughly tested, you will want to push your changes from the remote repository to the central repository.

In the GitHub interface, you can see that your remote repository is now ahead of the central repository. Select the "Pull request" button to create a pull request:

Make sure that the changes are being pulled from your remote repository to the central repository, and then click the "Create pull request" button:

In this page, you can also check to see what the difference between the two repositories would be:

Type in a descriptive name and, if needed, a short explanation of the changes in the pull request. Then, create the pull request:

If needed, this page can also be used to see the difference between the two repositories, which can be used to create the name or description of the pull request:

Once you see a similar screen as below, you've successfully created a pull request:

Make sure to add a label to the pull request so that the integration manager can immediately decipher the nature of the changes:

7. The integration manager reviews and merges the pull request

Once you've reached that screen, you've done everything that you should as a developer. In most cases, you will not see the "Merge pull request" button, as you will not have the permissions to merge your branch into the central repository. That will be the integration manager's job.

Once a pull request is submitted, the integration manager will be notified about the submission through email. Even so, it may be good practice to be in contact (whether through email, chat, or in person) with the integration manager regarding your recent pull request.

The integration manager will now either:

• merge the changes directly on the GitHub web interface, or
• add the developer’s remote repository as a remote and merge locally.

For further details, refer to the workflow guide for the integration manager.

In this step, the integration manager may ask you to make further changes before they approve your pull request. In this case, simply checkout the branch in which you were making your changes, make the requested changes, and push to your remote repository. In the example below, change master to the branch you were currently working on:

git checkout master
...
git add .
git commit -m "Adds further requested changes"
git push origin master

Once pushed, GitHub will show the new commits within the same request. To see these changes, simply scroll down further in the pull request.

8. Keep the local repository up to date

Once your pull request is reviewed and merged, update your local repository to reflect the changes made in the central repository. This is also good practice, as the central repository may contain multiple changes made by others if a team of developers are working on the same project.

Check to see if your local repository has an upstream source:

git remote -v

If an upstream source does not exist, add the central repository as source:

git remote add upstream https://github.com/yale-redcap/test-repo.git

Once the central repository is added, fetch the central repository's branches and latest commits to bring them into your local repository:

git fetch upstream

Check to see that you have fetched all of the branches:

git branch -va

The resulting output should look similar to the following, with both origin and upstream branches present:

* master                           612175d Adds another line to README
  remotes/origin/HEAD              -> origin/master
  remotes/origin/master            612175d Adds another line to README
  remotes/upstream/master          7551580 Adds updates that the developer will have to fetch

Now, checkout your master branch:

git checkout master

Then, merge all the changes present in the central repository's masterbranch:

git merge upstream/master

In most cases, Git will simply perform a fast-forward merge. If the merge results in a conflict, make sure to keep all of the changes that exist in the upstream central repository.

Push these changes to the remote repository, and you've successfully kept your local and remote repositories up to date:

git push origin master

Conclusion

The three existing repositories (central, remote, and local) provide ample version control and back up capabilities, while also enabling thorough vetting of codebase modifications before deployment and production. Additionally, this workflow lets the developers continue to work while waiting for the project to incorporate their changes, and the integration manager can pull in enhancements and fixes at any time, allowing each party to work at their own pace.

At this point, you should have a good enough understanding of the workflow process to immediately contribute to a REDCap@Yale software project. Steps 4 through 8 can be repeated to continue the development and maintenance of the project.