Compiler Configuration Reference¶

project.yml¶

Inside any project the compiler expects a project.yml file that defines metadata about the project, the location to store modules, repositories where to find modules and possibly specific versions of modules.

For basic usage information, see Project creation guide.

The project.yml file defines the following settings:

class inmanta.module.ProjectMetadata(*, requires: list[str] = [], name: str, description: str | None = None, freeze_recursive: bool = False, freeze_operator: str = '~=', author: str | None = None, author_email: NameEmail | None = None, license: str | None = None, copyright: str | None = None, modulepath: list[str] = [], repo: list[ModuleRepoInfo] = [], downloadpath: str | None = None, install_mode: InstallMode = InstallMode.release, relation_precedence_policy: list[str] = [], strict_deps_check: bool = True, agent_install_dependency_modules: bool = True, pip: ProjectPipConfig = ProjectPipConfig(index_url=None, extra_index_url=[], pre=None, use_system_config=False))[source]¶

Parameters:

name – The name of the project.
description – (Optional) An optional description of the project
author – (Optional) The author of the project
author_email – (Optional) The contact email address of author
license – (Optional) License the project is released under
copyright – (Optional) Copyright holder name and date.
modulepath – (Optional) This value is a list of paths where Inmanta should search for modules.
downloadpath – (Optional) This value determines the path where Inmanta should download modules from repositories. This path is not automatically included in the modulepath!
install_mode – (Optional) [DEPRECATED] This key was used to determine what version of a module should be selected when a module is downloaded. For more information see InstallMode. This should now be set via the pre option of the pip section.
repo –
(Optional) A list (a yaml list) of repositories where Inmanta can find modules. Inmanta tries each repository in the order they appear in the list. Each element of this list requires a type and a url field. The type field can have the following values:
- git: When the type is set to git, the url field should contain a template of the Git repo URL. Inmanta creates the git repo url by formatting {} or {0} with the name of the module. If no formatter is present it appends the name of the module to the URL.
- package: [DEPRECATED] Setting up pip indexes should be done via the index_urls option of the pip section. Refer to the migration guide for more information.
The old syntax, which only defines a Git URL per list entry is maintained for backward compatibility.
requires – (Optional) This key can contain a list (a yaml list) of version constraints for modules used in this project. Similar to the module, version constraints are defined using PEP440 syntax.
freeze_recursive – (Optional) This key determines if the freeze command will behave recursively or not. If freeze_recursive is set to false or not set, the current version of all modules imported directly in the main.cf file will be set in project.yml. If it is set to true, the versions of all modules used in this project will be set in project.yml.
freeze_operator – (Optional) This key determines the comparison operator used by the freeze command. Valid values are [==, ~=, >=]. Default is ‘~=’
relation_precedence_policy – [EXPERIMENTAL FEATURE] A list of rules that indicate the order in which the compiler should freeze lists. The following syntax should be used to specify a rule <first-type>.<relation-name> before <then-type>.<relation-name>. With this rule in place, the compiler will first freeze first-type.relation-name and only then then-type.relation-name.
strict_deps_check – Determines whether the compiler or inmanta tools that install/update module dependencies, should check the virtual environment for version conflicts in a strict way or not. A strict check means that all transitive dependencies will be checked for version conflicts and that any violation will result in an error. When a non-strict check is done, only version conflicts in a direct dependency will result in an error. All other violations will only result in a warning message.
agent_install_dependency_modules – [DEPRECATED] If true, when a module declares Python dependencies on other (v2) modules, the agent will install these dependency modules with pip. This option should only be enabled if the agent is configured with the appropriate pip related environment variables. The option allows to an extent for inter-module dependencies within handler code, even if the dependency module doesn’t have any handlers that would otherwise be considered relevant for this agent. Care should still be taken when you use inter-module imports. The current code loading mechanism does not explicitly order reloads. A general guideline is to use qualified imports where you can (import the module rather than objects from the module). When this is not feasible, you should be aware of Python’s reload semantics and take this into account when making changes to handler code. Another caveat is that if the dependency module does contain code that is relevant for the agent, it will be loaded like any other handler code and it will be this code that is imported by any dependent modules (though depending on the load order the very first import may use the version installed by pip). If at some point this dependency module’s handlers cease to be relevant for this agent, its code will remain stale. Therefore this feature should not be depended on in transient scenarios like this.
pip – A configuration section that holds information about the pip configuration that should be taken into account when installing Python packages (See: inmanta.module.ProjectPipConfig for more details).

class inmanta.module.ProjectPipConfig(*, index_url: str | None = None, extra_index_url: Sequence[str] = [], pre: bool | None = None, use_system_config: bool = False)[source]¶

Parameters:

index_url – one pip index url for this project.
extra_index_url – additional pip index urls for this project. This is generally only recommended if all configured indexes are under full control of the end user to protect against dependency confusion attacks. See the pip install documentation and PEP 708 (draft) for more information.
pre – allow pre-releases when installing Python packages, i.e. pip –pre. If null, behaves like false unless pip.use-system-config=true, in which case system config is respected.
use_system_config –
defaults to false. When true, sets the pip’s index url, extra index urls and pre according to the respective settings outlined above but otherwise respect any pip environment variables and/or config in the pip config file, including any extra-index-urls.

If no indexes are configured in pip.index-url/pip.extra-index-url, this option falls back to pip’s default behavior, meaning it uses the pip index url from the environment, the config file, or PyPi, in that order.

For development, it is recommended to set this option to false, both for portability (and related compatibility with tools like pytest-inmanta-lsm) and for security (dependency confusion attacks could affect users that aren’t aware that inmanta installs Python packages).

See the section about setting up pip index for more information.

The code snippet below provides an example of a complete project.yml file:

name: quickstart
description: A quickstart project that installs a drupal website.
author: Inmanta
author_email: code@inmanta.com
license: Apache 2.0
copyright: Inmanta (2021)
modulepath: libs
downloadpath: libs
install_mode: release
repo:
  - url: https://github.com/inmanta/
    type: git
requires:
  - apache ~= 0.5.2
  - drupal ~= 0.7.3
  - exec ~= 1.1.4
  - ip ~= 1.2.1
  - mysql ~= 0.6.2
  - net ~= 1.0.5
  - php ~= 0.3.1
  - redhat ~= 0.9.2
  - std ~= 3.0.2
  - web ~= 0.3.3
  - yum ~= 0.6.2
freeze_recursive: true
freeze_operator: ~=
pip:
    index-url: https://pypi.org/simple/
    extra-index-url: []
    pre: false
    use-system-config: false

Configure pip index¶

This section explains how to configure a project-wide pip index. This index will be used to download v2 modules and v1 modules’ dependencies. By default, a project created using the Project creation guide is configured to install packages from https://pypi.org/simple/. The ProjectPipConfig section of the project.yml file offers options to configure this behaviour. Some of these options are detailed below:

pip.use-system-config¶

This option determines the isolation level of the project’s pip config. When false (the default), any pip config set on the system through pip config files is ignored, the PIP_INDEX_URL, PIP_EXTRA_INDEX_URL and PIP_PRE environment variables are ignored, and pip will only look for packages in the index(es) defined in the project.yml. When true, the orchestrator will use the system’s pip configuration for the pip-related settings except when explicitly overriden in the project.yml (See below for more details).

Setting this to false is generally recommended, especially during development, both for portability (achieving consistent behavior regardless of the system it runs on, which is important for reproductive testing on developer machines, easy compatibility with Inmanta pytest extensions, and consistency between compiler and agents) and for security (the isolation reduces the risk of dependency confusion attacks).

Setting this to true will have the following consequences:

If no index is set in the project.yml file i.e. both index-url and extra-index-url are unset, then Pip’s default search behaviour will be used: environment variables, pip config files and then PyPi (in that order).
If index-url is set, this value will be used over any index defined in the system’s environment variables or pip config files.
If extra-index-url is set, these indexes will be used in addition to any extra index defined in the system’s environment variables or pip config files, and passed to pip as extra indexes.
If pre is set, it will supersede pip’s pre option set by the PIP_PRE environment variable or in pip

config file. When true, pre-release versions are allowed when installing v2 modules or v1 modules’ dependencies.

Auto-started agents live on the same host as the server, and so they will share the pip config at the system level.

Warning

use-system-config = true should only be used if the pip configuration is fully managed at the system level and secure for each component of the orchestrator.

Example scenario¶

During development

Using a single pip index isolated from any system config is the recommended approach. The pre=true option allows pip to use pre-release versions, e.g. when testing dev versions of modules published to the dev index. Here is an example of a dev config:

pip:
    index-url: https://my_repo.secure.example.com/repository/dev
    extra-index-url: []
    pre: true
    use-system-config: false

In production

Using a single pip index is still the recommended approach, and the use of pre-release versions should be disabled.

For a portable project (recommended), disable use-system-config and set index-url to the secure internal repo e.g.:

pip:
    index-url: https://my_repo.secure.example.com/repository/inmanta-production
    pre: false
    use-system-config: false

If you prefer to manage the pip configuration at the system level, use use-system-config: true e.g.:

pip:
    pre: false
    use-system-config: true

Note

Any pip config set explicitly in the project config will always take precedence over the system config. For more details see pip.use-system-config.

Pip-related settings that are not supported by the project config are not overridden.

To use a setting from the system’s pip configuration without overriding it, leave the corresponding option unset in the project.yml file.

Note

Set up authentication towards the index using netrc. See this section for more information.

Migrate to project-wide pip config¶

This section is a migration guide for upgrading to inmanta-service-orchestrator 7.0.0 or inmanta 2024.0. inmanta-core 11.0.0 introduced new options to configure pip settings for the whole project in a centralized way. For detailed information, see here. The following code sample can be used as a baseline in the project.yml file:

pip:
    index-url: https://my_repo.secure.example.com/repository/inmanta-production
    pre: false
    use-system-config: false

Alternatively, if you prefer to manage the pip config at the system level, refer to this section.

All the v2 module sources currently set in a repo section of the project_yml with type package should also be duplicated in the pip.index-url (and pip.extra-index-url if more than one index is being used).

If you want to allow pre-releases for v2 modules and other Python packages, set pip.pre = true in the project config file. This used to be controlled by the InstallMode set at the project level or at a module level.

Make sure the agents have access to the index(es) configured at the project level.

Run a full compile after upgrading in order to export the project pip config to the server, so that it is available for agents. This will ensure that the agents follow the pip config defined in the project. For reference, prior to inmanta-core 11.0.0, the agents were always using their respective system’s pip config.

Breaking changes:¶

Indexes defined through the repo option with type package will be ignored.

Dependencies for v1 modules will now be installed according to the pip config in the project configuration file, while they previously always used the system’s pip config.

The agent will follow the pip configuration defined in the project.yml.

PIP_PRE is now ignored unless use-system-config is set.

Allowing the installation of pre-release versions for v2 modules through the InstallMode is no longer supported. Use the project.yml pip.pre section instead.

Changes relative to `inmanta-2023.4` (OSS):¶

pip.use_config_file is refactored into pip.use-system-config.

An error is now raised if pip.use-system-config is false and no “primary” index is set through pip.index-url.

Pip environment variables are no longer ignored when pip.use-system-config is true and the corresponding option from the project_yml is unset.

Module metadata files¶

The metadata of a V1 module is present in the module.yml file. V2 modules keep their metadata in the setup.cfg file. Below sections describe each of these metadata files.

module.yml¶

Inside any V1 module the compiler expects a module.yml file that defines metadata about the module.

The module.yml file defines the following settings:

class inmanta.module.ModuleMetadata(*, name: str, description: str | None = None, freeze_recursive: bool = False, freeze_operator: str = '~=', version: str, license: str, deprecated: bool | None = None)[source]¶

The code snippet below provides an example of a complete module.yml file:

name: openstack
description: A module to manage networks, routers, virtual machine, etc. on an Openstack cluster.
version: 3.7.1
license: Apache 2.0
compiler_version: 2020.2
requires:
  - ip
  - net
  - platform
  - ssh
  - std
freeze_recursive: false
freeze_operator: ~=

setup.cfg¶

Inside any V2 module the compiler expects a setup.cfg file that defines metadata about the module.

The code snippet below provides an example of a complete setup.cfg file:

[metadata]
name = inmanta-module-openstack
description = A module to manage networks, routers, virtual machine, etc. on an Openstack cluster.
version = 3.7.1
license = Apache 2.0
compiler_version = 2020.2
freeze_recursive = false
freeze_operator = ~=

[options]
install_requires =
  inmanta-modules-ip
  inmanta-modules-net
  inmanta-modules-platform
  inmanta-modules-ssh
  inmanta-modules-std

Compiler Configuration Reference¶

project.yml¶

Configure pip index¶

pip.use-system-config¶

Example scenario¶

Migrate to project-wide pip config¶

Breaking changes:¶

Changes relative to inmanta-2023.4 (OSS):¶

Module metadata files¶

module.yml¶

setup.cfg¶

Changes relative to `inmanta-2023.4` (OSS):¶