Hotfix Deployment Process

Purpose

An efficient and well-defined process for deploying hotfixes is essential to delivering critical updates on time while maintaining system stability and long-term maintainability. Under the current process, deploying hotfixes is a challenge especially when working with a single stage environment where only certain commits may need to be extracted and deployed to production. The hotfix workflow management outlined below offers a structured approach for identifying, implementing, testing, and deploying hotfixes, ensuring minimal disruption and upholding both code quality and operational efficiency.

Benefits

• Cleaner Commit History
  • No redundant revert commits in stage
  • No merge commits from hotfix integration into stage, keeping history simple and linear
• Isolated Hotfix Development
  • Hotfixes are developed on a dedicated branch based off main, reducing the risk of conflicting changes from stage
  • No need for cherry-picking from stage to main or vice versa, eliminating the hassle of partial commits
• Easier Syncing Between Branches
  • Simplifies syncing: Once hotfix is merged into main, it will naturally propagate to stage when rebased
• Clearer Testing & Reverts
  • Hotfixes can be safely tested on stage without worrying about unnecessary commits being merged back into main
• Better Control Over Hotfix Merges
  • Avoids complicated merge scenarios since the hotfix comes from main
• No Impact on Other Engineers
  • While the hotfix is being developed, other engineers can continue working on stage as long as their changes don't conflict with the hotfix files
  • Ensures minimal disruption to other team members working on unrelated tasks
• Reduced Risk of Errors
  • Dropping the revert hotfix commit(s) prevents any accidental duplication or conflicts when syncing stage and main
  • Ensures fewer chances of merging unnecessary or broken code from the hotfix branch

Hotfix Step-by-Step Workflow

1 - Create a Hotfix Branch Based on main

This ensures the hotfix is independent of stage commits

Add all necessary commit fixes into this hotfix branch

git checkout main
git pull origin main
git checkout -b hotfix/<prev-release-version> (e.g. hotfix/v2.9.0)

2 - Test Hotfix in its Own Hotfix Branch using ?unitylibs=<hotfix-branch> in the URL

This is necessary for initial validations and ensures some functional verification before merging into stage

3 - Merge Hotfix into stage Temporarily for Signed-in User Testing (when applicable)

This allows QE to test signed-in scenarios without bringing in unrelated stage commits

Create a separate feature branch off of stage and cherry-pick the hotfix commits into it

Raise a PR of the feature branch to stage and the Release Manager will squash and merge the PR once approved

git checkout stage
git pull origin stage
git checkout -b <hotfix-feature-branch>
git cherry-pick <hotfix-commit-hash>
git fetch origin stage
git rebase -i stage
git push -u origin stage

4 - Revert the Hotfix from stage After Testing

Once QE has completed testing, revert the hotfix to restore stage

Create a revert PR while resolving any conflicts if any and the Release Manager will squash and merge into stage once approved

5 - Create and Merge a PR for the Hotfix into main

Open a PR from hotfix/<prev-release-version> to main

Once approved, the Release Manager will merge the PR into main

At this point, main contains the hotfix and stage has reverted it

6 - Sync stage with main Without Losing Other Commits

In the interactive rebase editor, drop all the hotfix commits and their associated revert commits (since it's already in main)

Resolve any conflicts and then force push the rebased stage branch

Now stage and main are in sync, while keeping any unreleased stage commits intact

git checkout stage
git rebase -i main
git rebase --continue (if needed after resolving conflicts)
git push origin -f

Final Outcome

• Hotfix is developed and tested separately before merging
• Signed-in user testing happens on stage without disrupting history
• No unnecessary commits from stage go into main
• stage and main branches remain clean and synchronized

Workflow Considerations

Should other engineers halt merging to stage during a hotfix?

No strict halt is required, but careful coordination is necessary.

If feature commits don't affect the hotfix:

• Other engineers can still merge to stage since their changes won't interfere
• Would need to ensure that these commit changes are independent of the hotfix files

If feature commits touch the same files as the hotfix:

• Pause merging these changes to stage until the hotfix is resolved
• Otherwise, we risk conflicts when rebasing or reverting the hotfix from stage

For urgent business-critical hotfixes, we will:

• Enforce a temporary freeze on stage to prevent unexpected issues
• Communicate to engineers that stage is temporarily locked
• Have a dedicated hotfix QE reviewer to expedite testing