Skip to content

Run code from your own package

The first tutorial called Python's built-in sum function. This tutorial installs a small package into the managed environment and calls one of its functions.

You will also pass a NumPy array to the worker and receive a new array as the result.

Prerequisites

Complete Run your first task first. Install Wetlands with host-side NumPy support:

pip install "wetlands[shared-memory]"

This tutorial runs the example package included in the Wetlands repository. Download it and enter its root directory:

git clone https://github.com/arthursw/wetlands.git
cd wetlands

Example package

The repository's examples directory is an installable package with this relevant structure:

examples/
├── pyproject.toml
├── example_module.py
└── getting_started.py

example_module.py contains an ordinary Python function:

def threshold(image, value):
    return image > value

The worker package does not need to import Wetlands for ordinary calls.

Its pyproject.toml gives the package a name and tells Python how to install the example_module module:

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "wetlands-example-workers"
version = "0.0.0"

[tool.setuptools]
py-modules = ["example_module"]

Your own worker code can use a regular package layout instead.

Complete application

from __future__ import annotations

from pathlib import Path

import numpy as np

from wetlands import EnvironmentManager, EnvironmentSpec, LocalPackage


def main(root: Path = Path("wetlands")) -> None:
    example_directory = Path(__file__).parent
    spec = EnvironmentSpec(
        python="3.12.*",
        conda=("numpy>=2", "pip"),
        local=(LocalPackage(example_directory),),
    )

    with EnvironmentManager(root=root) as manager:
        environment = manager.provision("numpy-example", spec).wait_for()
        image = np.arange(16, dtype=np.float32).reshape(4, 4)

        with environment.start() as workers:
            mask = workers.execute_import(
                "example_module:threshold",
                kwargs={"image": image, "value": 7.5},
            )

    print(mask)


if __name__ == "__main__":
    main()

Run the example from the repository root:

$ python examples/getting_started.py
[[False False False False]
 [False False False False]
 [ True  True  True  True]
 [ True  True  True  True]]

Understand the boundary

The application process imports wetlands and the host copy of NumPy. It does not import example_module.

LocalPackage(example_directory) tells Wetlands to install the example package inside the managed environment. The environment also has its own NumPy installation because the specification includes numpy>=2.

example_module:threshold means “import example_module inside the worker, then call its threshold attribute.”

When the application calls threshold, Wetlands transfers the NumPy data through operating-system shared memory automatically:

  1. the application copies image into a temporary host-owned shared-memory segment;
  2. the worker copies that data into a private, writable array;
  3. the worker places the returned mask in a temporary worker-owned shared-memory segment;
  4. the application copies the result into a normal array that it owns.

Wetlands cleans up both temporary segments. This is shared-memory transport, not a shared mutable array or zero-copy API, so changes made by the worker cannot modify the application's original array.

The worker and manager stop when their context managers exit. The ready numpy-example environment remains below the Wetlands root so a later run can reuse it.

Use a published package

Production applications normally install a versioned package from PyPI instead of a local directory:

spec = EnvironmentSpec(
    python="3.12.*",
    conda=("numpy>=2",),
    pypi=("my-worker-package==1.4.0",),
)

Call it with the same package.module:function syntax.

Next steps