GitLab support: extend the extractor beyond GitHub

[caviri]

Motivation

The Swiss research community publishes a large fraction of its code on self-hosted GitLab instances, not GitHub. Today GME only speaks GitHub, which leaves the bulk of the EPFL / ETHZ / Renku v1 corpus invisible to the open-pulse graph.

Public-only project counts (via GET /api/v4/projects?visibility=public, x-total header):

Instance	URL	Public	Active (non-archived)
gitlab.renkulab.io (Renku v1)	/explore	9,874	—
gitlab.ethz.ch	/explore	2,908	2,791
gitlab.epfl.ch	/explore/projects/active	1,206	1,177
Total		~14,000

Adding GitLab support roughly 10× the addressable corpus for free, with no architectural rewrite (GitLab's REST API is well-documented and similar in shape to GitHub's).

Scope

In scope:

• Self-hosted GitLab CE/EE instances (gitlab.ethz.ch, gitlab.epfl.ch, gitlab.renkulab.io, gitlab.com).
• Extraction of the same fields GME currently pulls for GitHub repos: identity, metadata, contributors, languages, topics, license, README, dependents-where-feasible.
• Two-token auth: GitHub token stays, plus a per-instance GitLab PRIVATE-TOKEN env var.

Out of scope (separate follow-ups):

• Bitbucket / Gitea / Forgejo (architecturally easier once the provider abstraction lands, but defer).
• Renku 2.0 ingestion — tracked in Ingest Renku 2.0 projects + data connectors into the RDF graph #53.
• CI/CD pipeline metadata (we currently don't extract GH Actions either; parity first).

Provider abstraction

The cleanest path is a thin provider layer so existing GitHub code doesn't get touched on the hot path:

extractor/
  providers/
    __init__.py            # registry: resolve(url) → provider
    base.py                # Protocol: identity, fetch_metadata, fetch_contributors, ...
    github.py              # current code, refactored behind the Protocol
    gitlab.py              # new

Provider selection driven by URL parsing, not config:

def resolve_provider(url: str) -> Provider:
    host = urlparse(url).hostname
    if host == 'github.com':           return GitHubProvider()
    if host and 'gitlab' in host:      return GitLabProvider(host)
    raise UnknownProviderError(url)

URL shape differences

GitHub is strictly two-level (owner/repo). GitLab supports arbitrary group nesting:

Pattern	Example
github.com/{owner}/{name}	github.com/sdsc-ordes/open-pulse
gitlab.example.com/{namespace*}/{name}	gitlab.epfl.ch/lasa/research/control-systems/foo

Practical consequences:

• The extractor's notion of owner becomes a namespace path (slash-separated) for GitLab. Existing GitHub callers must keep seeing the single-segment owner.
• Group-vs-user ownership is exposed via namespace.kind (group | user) in the API; map both to the same schema:author Object range, distinguished by rdf:type.
• API resolution requires URL-encoding the full project path: GET /api/v4/projects/{URL_ENCODED_PATH}.

Field mapping

GitLab project payload → existing RDF shape (no new predicates needed for core fields):

GitLab field	RDF predicate	Notes
web_url	@id (subject IRI)	strip trailing /
name	schema:name
path_with_namespace	schema:identifier
description	schema:description
created_at	schema:dateCreated
last_activity_at	schema:dateModified
default_branch	pulse:defaultBranch	already exists
topics[]	schema:keywords
star_count	pulse:stargazers
forks_count	pulse:forks
visibility	pulse:visibility	public / internal / private
archived	schema:archivedAt (when true)
license.key	schema:license	SPDX lookup
namespace.full_path	schema:author (group/user IRI)	nested group joins
forked_from_project.web_url	pulse:forkOf
http_url_to_repo	pulse:cloneUrl
readme_url	dereference → schema:text

Endpoints used per repo:

• GET /api/v4/projects/{enc(path)} — core metadata.
• GET /api/v4/projects/{id}/repository/contributors — contributor list.
• GET /api/v4/projects/{id}/languages — language breakdown.
• GET /api/v4/projects/{id}/repository/files/README.md/raw?ref=HEAD — README content (404 fallback to README, README.rst, etc.).

Auth strategy

providers:
  github:
    token_env: GITHUB_TOKEN
  gitlab:
    instances:
      - host: gitlab.epfl.ch
        token_env: GITLAB_EPFL_TOKEN          # null → anonymous, public projects only
      - host: gitlab.ethz.ch
        token_env: GITLAB_ETHZ_TOKEN
      - host: gitlab.renkulab.io
        token_env: GITLAB_RENKU_TOKEN
      - host: gitlab.com
        token_env: GITLAB_TOKEN

Token is sent as PRIVATE-TOKEN: <token> (GitLab's native header) — distinct from GitHub's Authorization: Bearer. Anonymous works for public; private/internal need the appropriate per-host token.

Pagination + rate limiting

GitLab paginates with x-total, x-total-pages, x-page, x-per-page headers and a next Link header (same shape as GitHub). Re-use existing pagination helper, parameterised on header names.

Rate limits are per-instance and configurable by admins — observe the RateLimit-* headers (lowercase, hyphenated; not the GitHub form) and back off on 429.

URL discovery / seeding

The crawler today seeds from GitHub orgs and users. For GitLab we want analogous seeds:

seeds:
  gitlab:
    - host: gitlab.epfl.ch
      include: all-public           # iterate /api/v4/projects?visibility=public
    - host: gitlab.epfl.ch
      include: group               # iterate one group recursively
      group: lasa
    - host: gitlab.renkulab.io
      include: topic
      topic: renku                 # filter by GitLab topic

Three discovery modes: whole-instance, single-group (recursive), topic-filtered. Mirrors the GitHub org / user / search seed types.

Acceptance criteria

• providers/ directory with base.py Protocol, github.py (existing code refactored), gitlab.py (new).
• URL parser auto-resolves the provider; no config flag needed at call sites.
• GitLabProvider supports gitlab.com + self-hosted (host passed at construction time).
• All current schema:SoftwareSourceCode predicates emitted for GitLab repos with the same shape as GitHub, so downstream open-pulse SPARQL queries don't need a single change.
• Per-host PRIVATE-TOKEN env vars supported; anonymous fallback works for public projects.
• Pagination + 429 back-off honour GitLab's RateLimit-* headers.
• CLI + API both accept GitLab URLs interchangeably with GitHub URLs.
• Test corpus: at least one repo from each of gitlab.epfl.ch, gitlab.ethz.ch, gitlab.renkulab.io, gitlab.com, mocked + recorded.
• README + config docs updated with the new providers.gitlab.instances block.
• Backwards-compatible: existing GitHub-only invocations and configs continue to work unchanged.

Companion work

• Open-pulse seed config will gain gitlab.* seed types in a follow-up PR there.
• Renku 2.0 ingest (Ingest Renku 2.0 projects + data connectors into the RDF graph #53) is independent: it consumes Renku's JSON API, not GitLab's, even though Renku v1 projects live on gitlab.renkulab.io — once GitLab support lands, v1 projects come in via the GitLab path automatically.

Open questions (non-blocking)

1. Owner abstraction: keep a single owner string field on Repo (storing the full namespace path for GitLab) vs introduce owner + namespace_path? Proposal: single field with semantic-by-provider; cleaner for downstream.
2. Group/user RDF type: emit schema:Organization for GitLab groups regardless of internal kind, or thread kind through? Proposal: schema:Organization for group, schema:Person for user.
3. Default branch handling: GitLab can have null default branch (empty repo). Skip extraction or emit minimal record? Proposal: emit minimal record with pulse:isEmpty true.

[caviri]

Discovered Swiss GitLab instances (broader landscape)

Probed https://{host}/api/v4/projects?visibility=public and read the x-total header. Numbers are anonymous-discoverable public projects only — internal / private aren't counted.

Confirmed instances

Institution	Host	Public projects
Renku v1 (SDSC)	gitlab.renkulab.io	9,874
ETH Zurich	gitlab.ethz.ch	2,908
FHNW (NW Switzerland UAS)	gitlab.fhnw.ch	1,551
EPFL	gitlab.epfl.ch	1,206
Univ Geneva	gitlab.unige.ch	1,114
OST (Eastern Switzerland UAS)	gitlab.ost.ch	824
Univ Zurich	gitlab.uzh.ch	605
SWITCH (.ch network)	gitlab.switch.ch	454
Idiap (Martigny)	gitlab.idiap.ch	428
UniBAS sciCORE	git.scicore.unibas.ch	116
SIB (bioinformatics)	gitlab.sib.swiss	112
SDSC	gitlab.datascience.ch	107
Eawag (water)	gitlab.eawag.ch	81
ETH Informatics dept	gitlab.inf.ethz.ch	76
HES-SO Valais	gitlab.hevs.ch	8
Swiss TPH	gitlab.swisstph.ch	5
EMPA (materials)	gitlab.empa.ch	0 (instance up, no public)
TOTAL public, discoverable		~19,500

Special case: CERN

gitlab.cern.ch — Geneva-based, Swiss-hosted but intergovernmental. Returns HTTP 200 on /api/v4/projects but disables the x-total / x-total-pages headers for unauthenticated callers (a GitLab admin option introduced for performance on very large instances). The link header still exposes next pagination, so the count is reachable by walking pages — just not in a single round trip. Anecdotally large (tens of thousands of LHC-related projects).

Implications for the extractor:

• Don't assume x-total is always present. Fall back to walking until the next link disappears.
• Configurable cap per host (max_projects_per_instance) so CERN doesn't dominate a crawl run.

Not found / no public GitLab DNS

DNS doesn't resolve under the obvious gitlab.<host> pattern (no self-hosted instance, behind SSO without public API, or running a non-GitLab forge): PSI, WSL, CSCS, UniBE, UniBAS (main), UniL, USI, UniFR, UniNE, ZHAW, HSLU, BFH, HES-SO central, HEIG-VD, HEIA-FR, SUPSI, PHZH. Many of these publish via GitHub orgs instead (UniBE, UniBAS, USI, ZHAW, HSLU, BFH have visible GitHub presence) — those are already covered by the existing GitHub path.

Suggested seed config (post-implementation)

providers:
  gitlab:
    instances:
      - { host: gitlab.renkulab.io,        token_env: GITLAB_RENKU_TOKEN }
      - { host: gitlab.ethz.ch,            token_env: GITLAB_ETHZ_TOKEN  }
      - { host: gitlab.fhnw.ch,            token_env: GITLAB_FHNW_TOKEN  }
      - { host: gitlab.epfl.ch,            token_env: GITLAB_EPFL_TOKEN  }
      - { host: gitlab.unige.ch,           token_env: GITLAB_UNIGE_TOKEN }
      - { host: gitlab.ost.ch,             token_env: GITLAB_OST_TOKEN   }
      - { host: gitlab.uzh.ch,             token_env: GITLAB_UZH_TOKEN   }
      - { host: gitlab.switch.ch,          token_env: GITLAB_SWITCH_TOKEN }
      - { host: gitlab.idiap.ch,           token_env: GITLAB_IDIAP_TOKEN }
      - { host: git.scicore.unibas.ch,     token_env: GITLAB_SCICORE_TOKEN }
      - { host: gitlab.sib.swiss,          token_env: GITLAB_SIB_TOKEN   }
      - { host: gitlab.datascience.ch,     token_env: GITLAB_SDSC_TOKEN  }
      - { host: gitlab.eawag.ch,           token_env: GITLAB_EAWAG_TOKEN }
      - { host: gitlab.inf.ethz.ch,        token_env: GITLAB_ETH_INF_TOKEN }
      # Optional, large + slow:
      - { host: gitlab.cern.ch,            token_env: GITLAB_CERN_TOKEN,
          totals_disabled: true, max_projects: 5000 }

Updated acceptance criterion

Add to the list in the issue body:

• Tolerate GitLab instances that disable x-total (CERN-style): paginator must drive off the link: rel="next" header alone when totals are missing.