On this page

The DataHub Development Operations Process

This page details the practices used by the Data 8 tech team in order to maintain and upgrade their JupyterHub, called the DataHub. The DataHub is a JupyterHub deployment using Kubernetes that provides the environment in which all Data 8 students work.

The DataHub deployment repository

The current deployment configuration for the DataHub is hosted as a GitHub repository. Any changes made to this repository should immediately be reflected in the live DataHub.

There are two branches for the DataHub repository - one for staging, and another for prod (short for “production”). Each branch controls a different JupyterHub deployment.

  • staging lives at staging.datahub.berkeley.edu. It is used by the Data 8 development team to test modifications before they’re deployed to the production environment. Students do not use this JupyterHub.

  • prod lives at datahub.berkeley.edu. It is the JupyterHub that students use, and as such has higher standards for stability.

The DataHub uses Travis CI (short for “continuous integration”) to automatically run tests and code every time a change is made to the DataHub master repository. This is also used to automatically deploy these changes to the staging and prod DataHub deployments when new changes are made.

Modifying the DataHub

The following steps are followed by the Data 8 tech team any time a modification must be made to the DataHub.

  1. Create a fork of the DataHub repository and work from this fork.

  2. Pull the latest changes from the DataHub repository into your fork.

  3. Create a new branch for your changes.

  4. Make the modifications necessary to this branch (this is usually something like modifying the DataHub helm chart or editing the DataHub Docker Image).

  5. Commit your changes and create a pull request. The pull request should be against staging, not against prod.

  6. The pull request will trigger a Travis CI process, and potentially a rebuild of docker images depending on what modifications have been proposed.

  7. Once the CI process is complete and if there are no problems, the committer requests that someone review the PR before merging.

  8. If all looks well, merge the PR into staging. This will trigger another Travis step which automatically upgrades the DataHub deployment at staging.datahub.berkeley.edu.

  9. After this upgrade finishes (per the logs in Travis CI), the team tests the changes by performing actions on staging.datahub.berkeley.edu (e.g., trying to import a package that was added)

  10. If staging tests fail, or if there are any problems during the live user testing, never update prod. Revert the change via the GitHub UI and debug before submitting another PR to staging.

  11. If the change passes tests and looks successful on staging, create a new PR to merge staging into prod. This will trigger a similar Travis process to deploy to prod at datahub.berkeley.edu. Test your change on production for good measure.

  12. Celebrate!

Administering JupyterHub The Data 8 Tech Stack @ Berkeley