Skip to content

Maintenance

We have created a python library to enable easy access to our large data catalog. It also assists our work in ETL, as it contains various methods and objects essential to the data wrangling procceses.

Currently, this library lives in the etl repository ( find it here).

Installation

Simply install it from PyPI:

pip install owid-catalog

Update release

After working on your changes in the library, publishing to PyPI is automated:

  1. Bump the version in lib/catalog/pyproject.toml
  2. Update the changelog in lib/catalog/README.md
  3. Refresh both lockfiles — the version is recorded in lib/catalog/uv.lock and in the repo-root uv.lock; run make .venv in lib/catalog/ and at the repo root
  4. Commit and push to master - the package will be automatically published to PyPI via GitHub Actions

The workflow triggers automatically when lib/catalog/pyproject.toml changes on the master branch. It includes a safety check to ensure the version was actually bumped before publishing.

Manual trigger: You can still manually trigger the workflow by clicking Run Workflow in GitHub Actions if needed.

Nothing bumps the version for you: a PR that only changes library source files publishes nothing, and those changes ship with whoever bumps next. The full maintainer guide — release checklist, versioning practice, dev environment, and which checks actually cover lib/catalog — is in lib/catalog/DEVELOPMENT.md.

Generate llms.txt

The library ships an llms.txt file (at docs/libraries/catalog/llms.txt) that is auto-generated from module docstrings and documentation markdown files. To regenerate it after changing docstrings or docs:

make docs.llms

This runs docs/ignore/others/bake_llms_txt.py, which inspects the public API surface and doc files so the output stays in sync with the codebase.