Creating your own catalog

A public catalog can be browsed in both modes. A private catalog needs server mode, which keeps your access token — see Deployment modes. Installing an entry needs server mode in every case.

This page is for whoever sets up the catalog: it involves GitLab, its tokens and its CI. To simply browse an existing catalog or install an entry, see Community catalog.

How a catalog works

A catalog does not hold the entities themselves, only pointers. Each entry — a small file in the entries/ folder — names a git repository where an entity lives: a project, a schema, a concept mapping, an ETL pipeline… Linkr downloads the index, displays the entries, and clones an entry’s repository when you install it.

Entries arrive in two ways, freely combined:

• By scanning — the sources.json file lists GitLab groups (or GitHub accounts). A scheduled job walks them, subgroups included, and creates an entry for every repository holding a Linkr entity — recognisable by the entity.json file Linkr writes at the root when you push an entity. This is the normal route: publishing an entity in a watched group is enough to make it appear.
• By hand — for a repository outside the watched groups, you write its entry yourself. A scan never deletes an entry; it only reports the ones no group covers.

From the entries, the CI generates two files Linkr reads: catalog.json, every entry in full, and catalog-index.json, a mere fingerprint of each. Linkr checks the latter to know whether anything changed, and only downloads the catalog again when it did — hence the Updates available banner.

Public or private

The catalog itself must be on a GitLab instance — gitlab.com, framagit.org, or your institution’s GitLab: Linkr reads it through GitLab’s API. The repositories it lists can be on GitLab or GitHub.

Step 1 — start from the template

The catalog template contains everything needed: the scripts that scan and generate the index, the entry schema and the CI. Nothing to install.

Create an empty project on your GitLab (no README), then push the template into it:

git clone https://framagit.org/interhop/linkr/linkr-catalog-template.git my-catalog
cd my-catalog
git remote set-url origin git@gitlab.com:my-team/my-catalog.git
git push -u origin main

GitLab’s New project → Import project → Repository by URL does the same without a command line. Pick the project’s visibility at that point: Public or Private.

Step 2 — say what to scan

The sources.json file, at the root, lists the groups to watch:

{
  "sources": [
    { "url": "https://gitlab.com/my-team/linkr-content", "note": "Our Linkr entities" }
  ]
}

Each group is walked with its subgroups. Repositories without a Linkr entity — the catalog itself, a website, a code repository — are ignored.

Step 3 — give the CI a token

The template’s CI has three tasks — jobs, in GitLab’s vocabulary:

build and rescan write to the repository: they need a token, provided by a CI variable named CATALOG_BOT_TOKEN. rescan also uses it to read the groups in sources.json, and only ever sends it to the GitLab instance hosting the catalog.

Which token

The fine-grained token is created from your profile: Edit profile → Access → Personal access tokens → Generate token → Fine-grained token. Two settings matter:

• Group and project access — the groups (or subgroups) to scan, plus the one holding the catalog, so that it can push there.
• Add resource permissions, Group and project tab — three resources to select in the left panel, each with its permission in the right one:
  • Project → Read — list a group’s projects;
  • Repository → Read — read their content, entity.json included;
  • Code → Download and Push — clone, and push the index to main.

Leave the User and Global tabs empty. If GitLab answers “Add at least one resource with permissions”, these permissions were not ticked: the groups chosen above are not enough.

Where to put it

In the catalog project: Settings → CI/CD → Variables → Add variable.

Write down the token’s expiry date — the template’s MAINTAINING.md file has a line for it. On that day, build and rescan fail with an authentication message that says nothing about expiry.

What about GitHub?

The catalog itself stays on GitLab: Linkr reads it through GitLab’s API, and the template’s CI is a GitLab CI. sources.json, however, can list GitHub accounts or organizations — https://github.com/my-organization — next to GitLab groups.

• Only the public repositories of a GitHub account are scanned, even with a token. A private GitHub repository is added through an entry written by hand; to install it, each user will need, in Linkr, a github.com token that gives access to it.
• No token is required, but GitHub limits anonymous calls to 60 per hour, quickly reached on a large organization. Add a second CI variable then, GITHUB_TOKEN, created like the first one (masked, not protected). On GitHub: Settings → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token, with Repository access set to Public repositories — read-only, it is enough. This token is only ever sent to GitHub.

Step 4 — schedule the rescan

In the catalog project: Build → Pipeline schedules → New schedule.

The row’s ▶ button runs the rescan right away, without waiting for Monday. Its log lists every entry added (A) or updated (M); it commits them to main itself, there is nothing to merge. When everything is already up to date, it ends green with Nothing to do.

The rescan can also be run on demand, from Build → Pipelines → Run pipeline on main.

Step 5 — add it to Linkr

On the Catalog page, the gear icon — Manage catalogs — opens the Catalogs window. Add a catalog asks for a Name, the Catalog repository URL and its Branch; the catalog loads as soon as it is added. This window is described in detail in Using an internal catalog.

For a private catalog, turn on Private repository and paste your Access token. This token is yours, not the CI’s:

• it is saved to your account for that host — gitlab.com, for instance — and reused for all its repositories: reading the catalog, installing its entries, syncing with git. Other users never see it;
• it must be able to read the catalog and the repositories it lists: a legacy token with read_repository is enough — add write_repository if you also push from Linkr;
• every user of the server enters their own.

If your account already holds a token for that host, the field says so; leave it blank to keep it.

Adding an entry by hand

For a repository outside the watched groups, create a file in entries/, named after its identifier:

{
  "id": "my-icu-queries",
  "type": "sql-collection",
  "git": { "url": "https://gitlab.com/team/my-icu-queries", "branch": "main" },
  "name": { "en": "ICU queries", "fr": "Requêtes de réanimation" },
  "description": { "en": "Reusable ICU queries.", "fr": "Requêtes réutilisables en réanimation." },
  "badges": ["icu", "omop"]
}

Required are id — identical to the file name —, type, git, name and description, with at least English (en) in the last two. type is one of project, mapping-project, sql-collection, etl-pipeline, data-catalog, dq-rule-set, schema-preset, database or workspace. The full reference is the template’s schema/entry.schema.json file.

Then run node scripts/build.mjs to regenerate the index, commit and push — or open a merge request: the validate job checks the entry and says precisely what is wrong.

When a job fails

The template’s MAINTAINING.md file repeats this table, with what to do.

Going further

• Community catalog — browsing a catalog and installing its entries.
• Publishing content — preparing an entity before a catalog points at it.
• Git versioning — pushing an entity to its repository, which creates its entity.json.