Migration from GitLab to GitHub

This document explains how to migrate from GitLab to Github including git history, issues, milestones, labels and merge requests as well as github pages and using GitHub workflow actions to be used instead of Jenkins or GitLab runners. Here are the steps that need to be followed respectively.

  • 1. Migrate git history, branches, tags, merges
  • 2. Migrate issues, milestones, labels, pull requests
  • 3. Migrate from Gitlab pages to GitHub pages (optional)
  • 4. Integrate CI with GitHub workflow actions (optional)

Before starting

To keep everything in one place and to be able to follow the instructions of this document easily, please create a directory called github-migration

mkdir github-migration
cd github-migration

1. Migrate git history, branches, tags, merges

Clone mirror of GitLab repository. Please make sure that you changed your directory to github-migration and run following command. The command below will create a mirror repo of gitlab in gitlab-repo directory. This clone will contain entire git history.

git clone --mirror <gitlab-repo-url-ssh-or-https> gitlab-repo

1.1 Create a new repository in GitHub

Please go to your GitHub account or the account of your organization, and create your empty repository. Do not add README.md by default. The repository should be completely empty.

1.2 Push gitlab mirror to GitHub

Please change your directory to gitlab-repo and push the Gitlab clone to GitHub repo

cd gitlab-repo
git push --no-verify --mirror <github-repository-url-ssh>

Afterwards, github repository will be mirror of gitlab.

Set push origin as github

git remote set-url --push origin <github-repository-url-ssh>

From now on, every push operation from your local repo will go into github repository, and every pull will be from GitLab (it is to synchronize with GitLab repo if there are recent changes during the migration.)

1.3 Syncing GitHub repository with Gitlab

If there are recent changes in the gitlab repository (during the migration) that are not pushed to GitHub, you can update Github repository:

git fetch -p origin
git push --no-verify --mirror

1.4 Add branch protection rules and set default branch

  • Please go Settings > General and set your default branch
  • Go Settings > Branches > Add new rule to add branch protection rule for important branches such as master or develop. If you type the branch name and keep other options as unselected the basic protection rules will be applied like the block for deleting the branch.

2. Transfer issues, milestones, labels etc.

After git history succeeds you can start migrating the issues. Issues and other written content can be transferred using third party library called node-gitlab-2-github. It is the node library that requires nodejs installed on your system and is compatible with nodejs version between 12 and 14.

Please make sure that you are in github-migration directory and clone node-gitlab-2-github library. This documentation has been tested on master branch where the head is 322ab012d22f3846945141cae7f7e6ee3a5c4e50.

git clone https://github.com/piceaTech/node-gitlab-2-github.git gitlab-to-github
cd gitlab-to-github

To install nodejs without breaking any version in your local, please use conda environment.

conda create -n github-migrate
conda activate github-migrate
conda install nodejs==14.8.0
npm i

IMPORTANT: Nodejs version should be between 12.x.x and 14. (It may change the version of node-gitlab-2-github that is used)

IMPORTANT: Please make sure that there is no issue created by hand on github repository before migration.

2.1 Configuration

Go to the root directory of gitlab-to-github, copy sample_settings.ts and modify it respecting your GitLab and GitHub configuration.

Please see an example settings file below. However, there might be extra fields or some changes that are added in newer versions.

import Settings from './src/settings';

export default {
  gitlab: {
    url:  '<main gitlab url>', // for example: 'https://gitlab.inria.fr'
    token: '<tken-gitlab>', // only read permission
    projectId: '<id of Fed-BioMed repo in Gitlab>', // e.g 44815,
    listArchivedProjects: false,
    sessionCookie: null,
  },
  github: {
    baseUrl: 'https://github.com',
    apiUrl: 'https://api.github.com',
    owner: '<owner of repository>', // organization name if it is an organization
    ownerIsOrg: true, // true if owner is an organization
    token: '<toke_github>', // normal repo permission,
    token_owner: '<user name of the owner of the token>',
    repo: '<repo-name>', //e.g. fedbiomed
    recreateRepo: false, // repo is already created
  },
  usermap: {
    '<user-name-gitlab>': '<user-name-in-github>',
    '<user-name-gitlab-2>': '<user-name-in-github-2>',
  },
  projectmap: {},
  conversion: {
    useLowerCaseLabels: true,
  },
  transfer: {
    description: true,
    milestones: true,
    labels: true,
    issues: true,
    mergeRequests: true,
    releases: true,
  },

  debug: true,
  useIssueImportAPI: true,
  usePlaceholderMilestonesForMissingMilestones: true,
  usePlaceholderIssuesForMissingIssues: true,
  useReplacementIssuesForCreationFails: true,
  useIssuesForAllMergeRequests: false,
  filterByLabel: null,
  trimOversizedLabelDescriptions: true,
  skipMergeRequestStates: [], // Skip creating issue for merged and closed merge requests example ['merged', 'closed']
  skipMatchingComments: [],
  mergeRequests: {
    logFile: './merge-requests.json',
    log: false,
  },
} as Settings;

You have to provide GitHub and GitLab token to access issues and write those in GitHub
Please update Github and Gitlab tokens. Please see 2.2 and 2.3 for details.

  gitlab: {
    url: 'https://gitlab.inria.fr',
    token: <token-gitlab> only read permission but as developer
    projectId: <id of Fed-BioMed repo in Gitlab>, // 44815,
    ...
  },
  github: {
    baseUrl: 'https://github.com',
    apiUrl: 'https://api.github.com',
    owner: 'fedbiomed',
    ownerIsOrg: true,
    token: <toke_github> normal repo permission,
    token_owner: '<user name of the owner of the token>',
    repo: '<repo-name e.g. fedbiomed>',
    recreateRepo: false, # It will use existing repository
  },

2.2 GitHub PAT (Personal Access Token)

Profile > Settings > Personal Access Token

Personal access token should the classic one that has only repo access as read/write.

2.3 GitLab Access Token

Settings > Access Token

  • Select read-api
  • Select role as developer ( the role should be developer otherwise non-public entries won’t be readable such as Milestones)

2.4 Run migration

In settings.ts debug is true by default. This will allow to test migration by reading from Gitlab but not pushing to Github. It will also allow to test installation of the module.

Please run following command while debug: True

npm run start

Please note that it is normal to encounter warning or error messages during the testing process, particularly when the API writes milestones. These messages are typically not indicative of system errors or module-related issues. They arise due to dependencies between entites that should have been writen to GitHub. As testing mode does not write any entries to GitHub, the API cannot retrieve accurate information, resulting in the display of error messages.

After the testing is completed change debug:false and execute following command.

npm run start

2.5 Limitations

GitHub PAT allows 5000 request per day. Therefore, if the repository contains too many issues the process might be stopped in the middle. To avoid such error, migration can be done in multiple steps.

Nevertheless, migrating issues, milestones, labels must be done in a fixed order

  1. Labels are migrated
  2. Milestones are migrated
  3. Issues are migrated
  4. Pull request are migrated
  5. Releases are migrated.

By respecting this order, migration can be done in multiple days or using multiple PAT of different users. For example, let’s say that there are two users as A, B and C who has admin privilege for the repository.

  • 1. User A migrates labels and milestones:
    • In the settings.ts other than labels and milestones should be set as false
  • 2. After labels and milestones migrated. user B migrates issues:
    • In the settings.ts other than issues should be set as false
  • 3. After issues are migrated, user B migrates merge requests:
    • In the settings.ts other than issues should be set as false
2.5.1 PAT and Creation of issues and milestones

Since the issues are migrated using PAT, they are created in github by the user whose PAT token is used. However, the issues desc and comments contain the information about when and who actually created this issue in Gitlab.

3. Migrating Gitlabpages

In GitLab, pages are published using .gitlab-ci.yml that tunes the process that builds documentation and publishes it. The logic in GitHub is quite the same. However, there are several options to deploy documentation.

1- Using a dedicated branch where source of documentation (HTML, css etc.) served
2- Using github action and github environment to deploy github pages.

3.1 Using dedicated branch

  • Please create a branch called gh-pages
  • Go to Settings > Pages and select the branch to serve documentation.

You can build your documentation in local and push changes to gh-pages branch or you can write a Github workflow action to build and publish/push changes to gh-pages.

3.2 Using GitHub action

Using github actions works as gitlab-ci.yml. In this workflow, you have to build your documentation, upload your artifacts and deploy pages. This approach does not require to have dedicated branch for serving web content.

Here is an example pages deployment using github actions. This workflow action will be triggered each time a new version tag is released.

Before adding a similar workflow action, you may need to configure your Pages settings to use Github Actions instead of dedicated branch. Please go Settings > Pages > Build and Deployment > Select “Github Actions” > click “create your own”. This option will generate github-pages environment for deployment.

name: Publish new version in documentation!
on:
  push:
    # Pattern matched against refs/tags
    tags:        
      - '*' 
  release:
    types: [published]

jobs:
  build-doc:
    name: Publish version in documentation
    environment: github-pages
    runs-on: ubuntu-latest
    steps:
      - name: Checkout to repository
        uses: actions/checkout@v3

      - name: Setup python
        uses: actions/setup-python@v4
        with:
            python-version: '3.11' 

      - name: Install Dependencies
        run: pip install -r envs/development/docs-requirements.txt

      - name: Build documentation
        run: mkdocs build

      # Upload built documentation source code
      - name: Upload Artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: 'build'

  deploy:
    name: Deploy Documentation
    # Add a dependency to the build job
    needs: build-doc

    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
    permissions:
      pages: write      # to deploy to Pages
      id-token: write   # to verify the deployment originates from an appropriate source

    # Deploy to the github-pages environment
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}

    # Specify runner + deployment step
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v2 # or the latest "vX.X.X" version tag for this action

3.3 Using organization or user domain for documentation.

The pages url that are deployed under a repository will be your-organization.github.io/your-repository-name. To make your repository documentation page visible on the main domain (your-organization.github.io/) you have to create a repository that is named as your-organization.github.io and build and publish your documentation in that repository.

However, you can also build your documentation in one repository and push the build content in another repository to serve it (e .g your-organization.github.io). It is doable using Github Actions. A workflow file can build the documentation build the library repository and push the built files to the repository your-organization.github.io. You can trigger build process using triggers for actions such as each time a new tag is pushed.

Please see workflow files in Fed-BioMed repository as an example implementation of what is described above.

4. CI Integration

Github free version’s CI usage limits on slaves are quite low. So many projects need to use external slaves, such as those provided by Inria’s CI (CloudStack). This is the case documented here.

As in GitLab, it is possible to integrate Jenkins using Webhooks. However, webhooks are not as advanced as GitLab. For example, blocking a merge request when the pipeline does not succeed requires many configurations on Jenkins side that includes configuring PAT token and calling API end points for GitHub pull request checks. Therefore, the best way to manage CI pipelines is through GitHub workflow actions. There are several ways that CI can be integrated using GitHub actions.

4.1 Calling Jenkins Webhook from an action

Instead of using default webhook options of GitHub to call Jenkins pipeline endpoint, you can call Jenkins in GitHub actions.

For example, here is an example action file

name: Run Jenkins jo 
on: 
  pull_request:
    branches: 
      - 'master'
      - 'develop'

jobs: 
  build-test:
    name: Jenkins API call
    runs-on: ubuntu-latest
    environment: jenkins
    steps: 
      - name: Checkout
        uses: actions/checkout@v3

      - name: Call api 
        run: <type the command makes API call> + you need to use secret token {secrets.jenkins_token}

      - name: Validate the results
        run: <write the command to validate result fail or success>

The action file above is executed each time a pull request is open for develop or master. Since the action files are public, you have to create an environment where you can set some secret variable and access it from action.yml file. To do that, please go to Settings > Actions > Environments > New Environment.

4.2 migrating from Jenkins to GitHub action

The best way to integrate CI in github is to use Github Actions natively which means running CI commands/jobs directly from the actions file. However, if you are using a free GitHub account/organization there are some limitations regarding CPU power, RAM and disk size but you can always add self hosted runner. (e.g. VM/slaves that are located in ci.inria.fr)

GitHub actions is quite similar to GitLab runners.

Please see this useful article about migrating from Jenkins to Github Actions: https://docs.github.com/en/actions/migrating-to-github-actions/manual-migrations/migrating-from-jenkins-to-github-actions.

Please also see the article about adding self hosted runners for github actions.
https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners

4.2.1 Displaying code coverage of unit tests

It is always useful to see the result of the unit tests that are run through CI. You can always display GitHub actions output from the Actions section of the repository but it will be a console output that is quite difficult to interpret. Therefore, you can integrate thirdparty tool that takes the results of the tests and display in a fancy way that is easy to interpret.

One of the tools that is quite nice and easy to integrate is Codecov. The only thing that you have to is to register Codecov with your Github account, add Codecov plugin into your account or your organization (from: Settings page of organization > Apps) and retrieve the token to call Codecov actions. This token should be set as secret variable in a GitHub environment that it can be accessed form an action file.

After the installation/integration of the plugin is completed, you can add extra step to your action.yml file to send test result to Codecov profile of your repository. Here is an example steps block

...
  steps: 
     - name: Run test
       run: <here you have tests and obtain result as xml>

     - name: Report to codecov
       uses: codecov/codecov-action@v3
       with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: tests/coverage.xml
          flags: unittests # optional
          name: codecov-umbrella # optional
          fail_ci_if_error: true # optional (default = false)
          verbose: true # optional (default = false)

If the workflow action is configured to be triggered each time there is a pull request, it automatically adds comment to pull request that shows the overall code coverage. Above to merge button you are able to see the checks as well as the codecov details. You can also visit the Checks section of the pull request to display all the checks that has been ran for the pull request including codecov/patch.

4.2.2 Blocking pull request if the checks does not succeed

One of the nice advantages of using GitHub workflow actions is to be able to block a pull request if one of the checks (workflow actions) is not passed. To enable this feature:

  • please go Settings > Branches > Add new rules
  • Type name of the branch that you want to protect (this is the branch that pull request will be created for)
  • Activate Require status checks to pass before merging
  • and search for the workflow – job name to add as required check.

Note: Job’s names may not be visible until they are ran at least once. Checks can not be the entire workflow, it should be only the job or jobs in a workflow.

Leave a Reply

Your email address will not be published.