In short
An ETL pipeline fills a target database from a source one, using SQL scripts. The scripts never name the databases directly: they write source. and target., which Linkr substitutes at execution time. That is what lets you take another site’s OMOP conversion pipeline and run it against your own databases. The scripts are written by hand or generated from the two databases’ schemas and from a mapping project.
The only place that writes
Everywhere else, Linkr reads. Databases are attached read-only, and neither widgets nor analysis scripts can modify anything.
An ETL pipeline is the exception, and it is a narrow one: only the target database is opened for writing, and only while the pipeline runs. The source and the vocabulary stay read-only in the same connection, so a script can read one and write the other — INSERT INTO target.person SELECT … FROM source.patients — without ever being able to modify what it reads.
Not to be confused with a project's pipeline
An ETL pipeline lives in the warehouse and goes from one database to another: this is where a source model is converted to OMOP.
A project’s pipeline starts from the warehouse and produces datasets for analysis — long format to wide. The first builds the warehouse, the second exploits it.
The three roles
A pipeline knows its databases by their role, never by their name.
source. — where the data comes from
Your operational database: the patient-record export, an ICU software’s tables. Read-only.
target. — where it goes
Typically an empty OMOP database, created from a schema with New from schema. The only one opened for writing.
vocab. — the reference terminology
The ATHENA vocabulary of a mapping project, to translate your local codes into standard concepts. Optional, read-only.
INSERT INTO target.person (person_id, gender_concept_id, year_of_birth)
SELECT p.id, v.target_concept_id, EXTRACT(year FROM p.birth_date)
FROM source.patients p
LEFT JOIN vocab.concept_relationship v ON v.source_code = p.sex_codeNever write a database's real name in an ETL script
This is the most important rule on this page. A database’s technical name contains an identifier specific to your instance: a script that hardcodes it stops working the moment it is exported elsewhere, where that identifier does not exist.
Roles survive the journey. Whoever imports your pipeline re-selects their two databases in the dropdowns, and every script works — without a single line of SQL being changed.
The substitution happens when running, not when saving: what goes into git stays portable. It is also careful — a source. inside a string, a comment or a longer identifier is left alone.
In this guide
A pipeline is created from the warehouse’s ETL pipelines page, then set up tab by tab. The first page follows the path from end to end: the tabs, writing the scripts, running and the quality check. The second details the two generators that write most of a conversion to OMOP:
Building and running a pipeline
The tabs, writing the scripts, the step-by-step run — pause, stop, errors — and the quality check of what was loaded.
Generating the scripts
The OMOP load scripts worked out from the schema mappings, and the vocabulary scripts drawn from a mapping project.
Sharing a pipeline
A pipeline exports, versions and publishes like every other entity — and this is where the roles earn their keep.
What travels is the scripts, never the data. A site that has converted its local model to OMOP can publish that work; another installs it, rebinds its two databases, and runs. It is probably the most directly reusable thing in all of Linkr: an OMOP conversion represents months of work, and it transfers in a single export.
Data files are excluded from versioning by default
A pipeline sometimes handles auxiliary files — lookup tables, reference lists. They are gitignored by default and re-included one at a time, explicitly. The default rule is that data does not leave; including it is a conscious decision, taken file by file.
Going further
- Building and running a pipeline — the tabs, the scripts, the run and the quality check.
- Generating the scripts — the OMOP load and vocabulary scripts, written by Linkr.
- Databases — creating the empty target database from a schema.
- Concept mapping — producing the correspondences the vocabulary consumes.
- Data quality — checking the target database once filled.
- Project pipeline — the other pipeline, the one producing datasets.