Explicitly set Python requirements?

[NiklasRosenstein]

This is actually a question but the only discussion category is "Show and tell" and the only Issue categories are "Bug report" and "Feature request".

The Windmill documentation states:

# You can import any package from PyPI, even if the assistant complains

This seems to rely on a direct mapping from the import name to the Python package name, but that is not always the case. Is there a way to specify the Python requirements of a Windmill workflow explicitly?

I'm just testing a few imports and I get errors like these. All are rather surprising, because these imports work properly if installed into a normal Python virtual environment.

# from kraken.core import Project  (PyPI package: kraken-core)
{
    "error": {
        "name": "ExecutionErr",
        "message": "error during execution of the script:
pip compile failed: Lock file generation failed: ExecutionErr(\"process terminated by signal: Some(\
    9,\
), stopped_signal: None, core_dumped: false\")"
    }
}

# import terracomp_typer  (PyPI package: terracomp-typer)
{
    "error": {
        "name": "ExecutionErr",
        "message": "File \"<frozen runpy>\", line 198, in _run_module_as_main
  File \"<frozen runpy>\", line 88, in _run_code
  File \"/tmp/windmill/dt-worker-WG82L-EUGZm/01873a0d-29bc-ba2b-2f1b-73328ec568bb/wrapper.py\", line 8, in <module>
    from u.niklas import accommodative_script as inner_script
  File \"/tmp/windmill/dt-worker-WG82L-EUGZm/01873a0d-29bc-ba2b-2f1b-73328ec568bb/u/niklas/accommodative_script.py\", line 3, in <module>
    import terracomp_typer
  File \"/tmp/windmill/cache/pip/terracomp-typer==0.0.6/terracomp_typer/__init__.py\", line 3, in <module>
    from ._build import build_app_from_module
  File \"/tmp/windmill/cache/pip/terracomp-typer==0.0.6/terracomp_typer/_build.py\", line 14, in <module>
    from ._injector import DependencyInjector
  File \"/tmp/windmill/cache/pip/terracomp-typer==0.0.6/terracomp_typer/_injector.py\", line 12, in <module>
    from nr.stream import Supplier
ModuleNotFoundError: No module named 'nr'"
    }
}

[rubenfiszel]

Thanks for the great question,

You can indeed: https://docs.windmill.dev/docs/reference#dependency-management

example, at the top of your script:

#requirements:
#dependency
#version_pinned_dependency==0.4

Note that you will need to list all the dependencies.

Also if you tell us what dependencies doesn't work with the current heuristic, we can add it to the list of mapping

[NiklasRosenstein]

Thanks for the quick reply! That's what I've been looking for.

If I understand you correctly, you mean to say that I need to explicitly specify all transitive dependencies. Is there any particular reason for this? That seems rather inconvenient.

[rubenfiszel]

You do not need to specify all the transitive dependencies, but you will have to declare all the direct dependencies needed whereas one could have believed this was an additional set of imports

[rubenfiszel]

I've added a new 'Questions' category and we will think of ways of making this answer easier to find directly in the product.

[rubenfiszel]

@NiklasRosenstein We now have a dedicate page for this on the docs: https://docs.windmill.dev/docs/advanced/imports#python and are linking to it in the default python script

[pat-mw]

Hey, is it possible to link private dependencies from a github repo?
In the same way you would install the repo using 'setup.py' and 'pip install -e .'

I need access to my own private python packages within Windmill, and the documentation doesn't outline how one might do this via the repo

[mhuang74]

@rubenfiszel I have the same problem. Need to install a patched version of Experta via github. Is that possible in Windmill?

I have tried adding this line to requirements.txt and run wmill script generate-metadata, but this command isn't working.

https://github.com/openmotics/om-experta.git

error

Generating metadata for f/m1_campaign_anomaly_report/cause_analyzer.py
error: Uncaught (in promise) Error: Failed to generate lockfile. Status was: OK, {"error": {"name": "ExecutionErr", "message": "error during execution of the script:\nError locking file: Lock file generation failed: ExitStatus(1)"}}, Error: Failed to generate lockfile. Full response was: {"error":{"name":"ExecutionErr","message":"error during execution of the script:\nError locking file: Lock file generation failed: ExitStatus(1)"}}
    throw new Error(
          ^
    at updateScriptLock (https://deno.land/x/[email protected]/metadata.ts:388:11)
    at eventLoopTick (ext:core/01_core.js:168:7)
    at async generateScriptMetadataInternal (https://deno.land/x/[email protected]/metadata.ts:256:7)
    at async Command.generateMetadata [as actionHandler] (https://deno.land/x/[email protected]/script.ts:868:5)
    at async Command.execute (https://jsr.io/@windmill-labs/cliffy-command/1.0.0-rc.5/command.ts:1940:7)
    at async Command.parseCommand (https://jsr.io/@windmill-labs/cliffy-command/1.0.0-rc.5/command.ts:1772:14)
    at async main (https://deno.land/x/[email protected]/main.ts:197:5)

version

wmill 1.402.1

[rubenfiszel]

@mhuang74 try this:

# import wmill
#requirements:
#om-experta@git+https://github.com/openmotics/om-experta.git

import experta
def main(x: str):
    print(experta)
    return x

[mhuang74]

@rubenfiszel Thank you very much. It worked. And appreciate the super quick response on a weekend.

May I suggest the following changes to the Python dependency example. Having the same dependency keyword on both lines threw me off. Thought they were the same library.

Here's the Pull Request.

Current

#requirements:
#dependency
#version_pinned_dependency==0.4

Suggested. It may be clearer this way. Also include example to install from repo.

#requirements:
#dependency1[optional_module]
#dependency2>=0.40
#dependency3@git+https://github.com/myrepo/dependency3.git

import dependency1
import dependency2
import dependency3