Development Setup

This python based project includes a lot of best pratices that have been established within CLS. To get familiar with the most basic ones please see have a look at the pyproject.toml file. If you are not familiar with uv please go ahead and read the following sections. It is highly recommended to use uv to properly handle python dependencies within this project.

To provide a stable model an essential part is clean code. To support this, several things have been preconfigured here.

For example,

• python code linting
• code formatting
• unit tests
• credential handling
• additional checks via pre-commit hooks

Most of the above mentioned examples is discussed in more detail below.

Development installation

In case you want to develop mother further, do the following steps:

• install uv following the steps outlined here (https://docs.astral.sh/uv/getting-started/installation/)
• clone the repository
• create the uv venv with base dependencies: uv sync
• optionally install extras: uv sync --extra report --extra torch --extra tabpfn
• run unittests uv run poe test-unit
• run acceptance tests uv run poe test-acceptance

Understanding Extras vs Dependency Groups

Mother uses two different mechanisms for optional dependencies:

Extras (User-Facing)

Extras are pip-installable optional dependencies that end users can install:

Install multiple extras:

uv sync --extra report --extra torch --extra tabpfn

Dependency Groups (Development Only)

Dependency groups are development dependencies that are NOT published with the package:

Install multiple groups:

uv sync --group examples --group docs --group test_duration

Complete developer setup:

# Install base dependencies + all extras + all dev groups
uv sync --all-extras --all-groups

Developing a new feature

• create a new feature branch git checkout -b feature
• use pre-commit hooks
• make changes to mother and add tests accordingly
• run unittests uv run poe test-unit
• run acceptance tests uv run poe test-acceptance
• git push changes to a separate branch describing the feature
• Do not forget to use semantic versioning commands in your commit message (version is bumped by semantic-versioning)
• create a pull request on GitHub

Publishing a new package version

A new package will be pushed via the mother CI/CD pipeline. No manual steps required.

Codespaces

Skip local install and run mother from codespaces; all compute runs on github. Automate install using docker. This is a work in progress. User docs come after this is tested.

Run Tests

Confirm mother is working by running some tests.

• Go to "Testing" tab, wait until loaded and select tests to run
• There are three types of test to run. Each github commit triggers running all these tests. See github actions for more info.
• Acceptance are long running tests. Run these remotely
• Unit test we can run locally, as they are quick to run.

Using a virtual environment

Key for propper project setup is a virtual environment. The recommended way is to use uv, which manages virtual environments automatically:

uv sync

Alternatively, you can manually create a virtual environment:

• Create the virtual environment: python -m virtualenv .venv
• Initiate the virtual environment: source .venv/bin/activate
• Then install the base requirements: pip install -r requirements.dev.txt

Using uv

If you want to use uv as your dependency manager use the provided pyproject.toml file. uv automatically manages virtual environments and lockfiles for you. Consistency of the requirements files is ensured if the pre-commit hooks are enabled.

Development

To properly develop please start by enabling the pre-commit hooks if not already done using pre-commit install.

Pull Requests

Merging your changes to the main branch is just allowed via PRs. For that please provide clean commit messages for your changes. When merging the PR please use Squash and merge and ensure linear history. The changelog is automatically created via github actions and takes the changelog from the PR message. See here.

Testing

pytest is the prefered testing package on the one included in the requirements.dev.txt. A github actions workflow runs tests before merges, but you should run your own tests before pushing to avoid the github actions failing.

Estimating Test Durations

Some of our unit tests are pretty slow. But which one, we do not exactly now. There is the possibility to use --durations to get durations for each test. However, this does not lead to consistent timings.

I installed the modules uv add --group test_duration pytest-xdist[psutil] pytest-html to see how this would improve timings (according to ChatGPT).

pytest test --durations=0 --html=pytest_report.html --self-contained-html -n auto

import json
from pathlib import Path

durations_path = Path("durations.json")
if durations_path.exists():
    with durations_path.open("r", encoding="utf-8") as json_file:
        data = json.load(json_file)
else:
    # Example fallback so the snippet can run as-is.
    data = [
        {"nodeid": "test/unit/test_core.py::test_fit", "duration": 0.12},
        {"nodeid": "test/unit/test_ml.py::test_predict", "duration": 0.08},
    ]

test_durations = {item["nodeid"]: item["duration"] for item in data}
for test_name, duration in test_durations.items():
    print(f"Test {test_name} took {duration:.2f} seconds")

Credential Handling

You should NEVER add your credentials into any script. Please use the .envfile.

Linting

pylint is the prefered testing package on the one included in the requirements.dev.txt. The github actions workflow is set to fail if the score is below 8, so to make sure your action will not fail run pylint before you push.

Formatting

Formating is done for you by ruff with github actions automatically and as a pre-commit hook.

SonarQube

SonarQube is a great way to analyze your code and make sure its set up properly. To add your project you can simply link it to your github repository here. Coverage of your project is automatically generated. To learn how python test coverage is handled in sonar qube have a look at the docs sonarqube coverage.

Pre-Commit Hooks

pre-commit hooks can be enabled via pre-commit install. Please ensure to properly activate the python environment. If you use uv use uv run poe install-hook. These hooks are simple scripts that run before a commit is performed to ensure consistency of the added code and checks for credentials as well. See .pre-commit-config.yaml for provided hooks or add your own if desired.

Python Semantic Release

This project uses Python Semantic Release (PSR) to automate version management and changelog generation. PSR analyzes commit messages to automatically determine the next version number and generate release notes.

Conventional Commits

To work effectively with semantic release, all commit messages must follow the Conventional Commits specification:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Commit Types

The project recognizes the following commit types:

• feat: A new feature (triggers a minor version bump)
• fix: A bug fix (triggers a patch version bump)
• perf: A code change that improves performance (triggers a patch version bump)
• docs: Documentation only changes (triggers a patch version bump)
• style: Changes that do not affect the meaning of the code (triggers a patch version bump)
• refactor: A code change that neither fixes a bug nor adds a feature (triggers a patch version bump)
• test: Adding missing tests or correcting existing tests (triggers a patch version bump)
• build: Changes that affect the build system or external dependencies (triggers a patch version bump)
• ci: Changes to CI configuration files and scripts (triggers a patch version bump)
• chore: Other changes that don't modify src or test files (triggers a patch version bump)

Breaking Changes

To trigger a major version bump, add BREAKING CHANGE: in the commit body or append ! to the type:

feat!: remove deprecated API endpoint

BREAKING CHANGE: The /api/v1/old-endpoint has been removed. Use /api/v2/new-endpoint instead.

Examples

# Minor version bump
feat: add new clustering algorithm support

# Patch version bump
fix: resolve memory leak in preprocessing pipeline
docs: update installation instructions
refactor: optimize feature selection performance

# Major version bump
feat!: redesign configuration system

BREAKING CHANGE: Configuration file format has changed from YAML to JSON.

Configuration

The semantic release configuration is defined in pyproject.toml:

[tool.semantic_release]
version_toml = ["pyproject.toml:project.version"]
tag_format = "v{version}"
commit_message = "chore(release): {version} [skip ci]\n\nAutomatically generated by python-semantic-release"
build_command = "uv lock && git add uv.lock"

[tool.semantic_release.changelog]
mode = "update"
insertion_flag = "<!-- version list -->"

[tool.semantic_release.changelog.default_templates]
changelog_file = "docs/Changelog.md"
output_format = "md"

Keep <!-- version list --> above the release entries in docs/Changelog.md. In update mode, PSR leaves a non-empty changelog unchanged if this marker is missing. Restoring it enables new entries but does not backfill missed releases.

build_command regenerates and stages uv.lock after PSR updates the version, so the lockfile is included in the release commit. Package building happens in a separate uv build step. Do not add the lockfile to PSR's assets: those files are also uploaded as GitHub release attachments by semantic-release version. The top-level upload_to_vcs_release setting is not valid in PSR 10, and publish.upload_to_vcs_release does not control these version-command uploads.

The [skip ci] in commit_message stops the release commit from re-triggering CI. The release job also excludes commits containing PSR's generated-commit message as a second guard.

Automated Release Process

Releases run on pushes to main after the required checks pass. All steps are defined in .github/workflows/workflow.yml; the separate release.yml workflow has been removed. See the CI/CD overview for the full job dependency order.

1. Checks: style runs first, followed by release-preflight, the Python 3.11-3.14 test matrix, and test-slow. Preflight uploads a report and fails on non-conventional commit subjects before the tests start.
2. Release setup: the release job uses the release environment, checks out the full Git history without persisting checkout credentials, and sets up uv. PAT_RELEASE_PIPELINE is passed as GH_TOKEN for PSR's repository writes.
3. Version and changelog: uv run semantic-release version runs on the host, analyzes commits, computes the next version, updates docs/Changelog.md and pyproject.toml, then runs uv lock && git add uv.lock.
4. GitHub release: PSR pushes the release commit and tag, then creates the GitHub release and release notes. The lockfile is committed, not attached.
5. Distributions: the workflow compares the tag before and after PSR. Only when it changes does the job run uv build, validate dist/* with uv tool run twine check --strict dist/*, and upload the Packages Actions artifact. Distributions are not attached to the GitHub release by this job.
6. PyPI publication: the separate publish-pypi job downloads Packages and publishes with pypa/gh-action-pypi-publish using OIDC Trusted Publishing in the pypi environment. It requires a successful new release and the canonical Bayer-Group/MotherML repository; no PyPI API token is used.

build-docs runs independently. deploy-docs depends only on that build and runs on non-PR executions on main, not on the release or PyPI jobs.

The preflight report is designed to complement the update-docs skill. It does not replace the skill's issue and milestone reconciliation, but it ensures contributors get an automatic reminder and an auditable report on every PR and on main.

PR Preflight Comment

On same-repository pull requests, the separate comment-preflight job posts (and updates) a sticky bot comment with the release-preflight-report.md output. It runs even when preflight fails, uses an isolated write-scoped token, and never checks out or executes PR code. Fork PRs still run preflight but do not receive this comment.

Expected behavior:

• The comment is updated on each workflow run instead of creating duplicates.
• The report artifact is uploaded even if strict checks fail.
• The job fails at the end if non-conventional commits are detected.

Typical failure causes:

• Commit subject does not follow Conventional Commits (for example feat: ..., fix: ..., docs: ...).
• Unsupported commit type in the subject.

How to resolve failures:

1. Rewrite or squash commit subjects to Conventional Commit format.
2. Re-run CI by pushing the updated branch.
3. Verify the updated PR comment no longer lists non-conventional commits.

Manual CI Runs and Release Preview

There is no separate manual release workflow. Using Run workflow on CI triggers workflow_dispatch, which runs checks but skips release and publish-pypi: the release job requires a push event on main. A manual CI run on main can still deploy documentation.

To preview the next version locally without changing files:

uv run semantic-release version --print

This command does not publish a release. For a normal release, merge a PR with an appropriate Conventional Commit message and let the push-to-main pipeline complete. Do not bump the version or create a release tag by hand.

Best Practices

1. Consistent Commit Messages: Always follow conventional commit format
2. Atomic Commits: Make each commit focused on a single change
3. Descriptive Messages: Write clear, concise commit descriptions
4. Scope Usage: Use scopes to indicate which part of the codebase is affected:
   Bash

   feat(preprocessing): add new normalization method
   fix(ml): correct cross-validation scoring
   docs(api): update docstring examples
   

5. Merge Strategy: Use "Squash and merge" for PRs to maintain clean commit history
6. Release Notes: The changelog is automatically generated from commit messages, so write them for your users

Troubleshooting

No Release Generated

• Ensure commits follow conventional commit format
• Check that commit types are recognized (see configuration above)
• Verify this is a push to main, not a PR or manual dispatch, and that the required jobs passed
• Check whether PSR found a new version; without a new tag, package building and publication are skipped
• Changelog exclusions control which entries appear in release notes, not which commit types trigger a version bump

Version Not Updated

• Check that version_toml path is correct in configuration
• Ensure uv is properly configured
• Verify GitHub Actions has necessary permissions

Failed Release

• Check GitHub Actions logs for detailed error messages
• Verify PAT_RELEASE_PIPELINE, the release environment, and repository write permissions
• For PyPI failures, verify the Trusted Publisher matches this repository, workflow.yml, and the pypi environment
• If lockfile regeneration fails, check that uv lock succeeds locally
• If an upload fails for uv.lock, ensure it is staged by the build command rather than listed in PSR's assets

A failure after PSR pushes can leave the release commit, tag, and GitHub release in place even though distribution building or PyPI publication never completed. Inspect the remote tag, GitHub release, Packages artifact, and PyPI version before retrying. Rerunning a failed PyPI job can reuse an existing artifact, but rerunning PSR is not a general recovery mechanism: an existing release may produce no new tag, so the downstream build and publish steps are skipped. If publication never happened and no usable artifact exists, merge the correction with an appropriate Conventional Commit to produce the next release through CI; do not delete or move a published tag as a routine retry.

For more details, consult the Python Semantic Release documentation.