Logging

This page describes the different logs files produced by the Inmanta server and its agents and explains what can be configured regarding to logging.

Overview different log files

By default log files are collected in the directory /var/log/inmanta/. The following files are expected:

  1. server.log is the main log file of the server. It shows general information about actions performed by the Inmanta server (renewing parameters, purging resource action logs, etc.), and the access log of the API.

  2. resource-actions-<environment-id>.log contains all actions performed by all resources. Each environment has one resource action log file. This file mirrors the actions logged on resources in the database.

  3. agent-<environment-id>.log is the main log of the scheduler and executors. It contains information about when any executor started a deployment, which trigger caused that deployment, whether heartbeat messages are received from the server, etc.

  4. agent-<environment-id>.out: This log file contains all the messages written to the standard output stream of the scheduler and executors. Expected to be empty.

  5. agent-<environment-id>.err: This log file contains all the messages written to the standard error stream of the scheduler and executors. Expected to be empty.

For reasons of backward compatibility, the scheduler files are called ‘agent’ and not ‘scheduler’

Configure logging

Logging can be configured in two main ways:

  • coarse grained configuration using configuration and command line options. This is sufficient in most cases.

  • fine grained configuration using a config file. Here the logging config is fully user controlled.

Coarse grained configuration

The following log-related options can be set in an Inmanta config file:

  • config.log-dir: determines the folder that will contain the log files

As command line options, the following are available:

  • --timed-logs: Add timestamps to logs

  • --log-file: Path to the logfile, enables logging to file, disables logging to console

  • --log-file-level: Log level for messages going to the logfile, options ERROR, WARNING, INFO, DEBUG and TRACE

  • -v: Log level for messages going to the console. Default is warnings only. -v warning, -vv info, -vvv debug and -vvvv trace.

  • -X: When exiting with an error, show full stack trace.

  • --keep-logger-names: When using the compiler, don’t shorten logger names.

To update the server startup config when using the RPM based install, copy the inmanta-server service file at /usr/lib/systemd/system/inmanta-server.service to /etc/systemd/system/inmanta-server.service and edit it.

[Unit]
Description=The server of the Inmanta platform
After=network.target

[Service]
Type=simple
User=inmanta
Group=inmanta
ExecStart=/usr/bin/inmanta --log-file /var/log/inmanta/server.log --log-file-level INFO --timed-logs server
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl restart inmanta-server

Fine grained configuration

For fine grained configuration, a standard python dict config file can be passed in via the config file for each component individually:

[logging]
server = server_log.yml
scheduler = scheduler_log.tmpl
compiler = compiler.yml

or via a cli option:

inmanta --logging-config server_log.yml server

The log config has to be either a yaml file, containing a python dict config or a template of a yaml file. In this case, the file name has to end with tmpl.

The following log-related options can be set in an Inmanta config file:

  • logging.compiler: determines the log config for the compiler. This affects the output of all compiles performed by the server and can be used to fine tune the output displayed in the compile details page of the Inmanta Web Console.

  • logging.server: determines the log config for the server

  • logging.scheduler: determines the log config for the scheduler. This is always a template.

Each of the above-mentioned configurations can also be provided directly into an environment variable in the following way:

  • config.logging-config: INMANTA_CONFIG_LOGGING_CONFIG_CONTENT or INMANTA_CONFIG_LOGGING_CONFIG_TMPL

  • logging.compiler: INMANTA_LOGGING_COMPILER_CONTENT or INMANTA_LOGGING_COMPILER_TMPL

  • logging.server: INMANTA_LOGGING_SERVER_CONTENT or INMANTA_LOGGING_SERVER_TMPL

  • logging.scheduler: INMANTA_LOGGING_SCHEDULER_CONTENT or INMANTA_LOGGING_SCHEDULER_TMPL

The _TMPL suffix indicates that the provided configuration is a template. The _CONTENT suffix indicates a non-template configuration.

As command line options, the following are available:

  • -v: When used in combination with a log file, it will force a CLI logger to be loaded on top of the provided configuration

  • --logging-config: Log configuration file for this command, overrides the config option

  • -X: When exiting with an error, show full stack trace.

  • If a config file is loaded, all other coarse grained configuration options are ignored!

For templated config files, we use python f-string syntax. For the scheduler, one variable is available in the template: {environment}. This is used to customize the scheduler log files to the environment the scheduler is working for.

Converting to fine grained configuration

A tool is provided to convert the existing coarse grained configuration into a config file.

For example, to convert the config for a component, use the output-default-logging-config command.

inmanta -c /etc/inmanta/inmanta.cfg \
    --log-file /var/log/inmanta/server.log \
    --log-file-level INFO \
    --timed-logs \
    output-default-logging-config \
    --component server \
    config.yml

Default configurations

Default configuration for the server, including LSM:

server_log.yml
 1# Generated using: inmanta -c /etc/inmanta/inmanta.cfg --log-file /var/log/inmanta/server.log --log-file-level 2 --timed-logs output-default-logging-config --component server server_lsm_log.yml
 2formatters:
 3  core_console_formatter:
 4    # Console formatter with coloring.
 5    # This formatter also does indentation of multi-line log message, so all lines are put nicely underneath each other.
 6    (): inmanta.logging.MultiLineFormatter
 7    fmt: '%(asctime)s %(log_color)s%(name)-25s%(levelname)-8s%(reset)s%(blue)s%(message)s'
 8    # Don't shorten logger names by default
 9    keep_logger_names: true
10    log_colors:
11      CRITICAL: red
12      DEBUG: cyan
13      ERROR: red
14      INFO: green
15      WARNING: yellow
16  # Normal formatter
17  core_log_formatter:
18    format: '%(asctime)s %(levelname)-8s %(name)-10s %(message)s'
19  # Lsm callback
20  lsm_callback_formatter:
21    format: '%(asctime)s %(name)-25s%(levelname)-8s%(message)s'
22handlers:
23  server_handler:
24    # Main handler for the server
25    class: logging.handlers.WatchedFileHandler
26    filename: /var/log/inmanta/server.log
27    formatter: core_log_formatter
28    level: INFO
29    mode: a+
30  lsm_callback_handler:
31    # LSM callback log
32    class: logging.FileHandler
33    filename: /var/log/inmanta/callback.log
34    formatter: lsm_callback_formatter
35  core_tornado_debug_log_handler:
36    # Special handler that captures tornado max_clients limit reached messages and turns them into warnings
37    # As these may cause instability
38    class: inmanta.logging.TornadoDebugLogHandler
39    level: DEBUG
40loggers:
41  inmanta_lsm.callback:
42    handlers:
43    - lsm_callback_handler
44    level: INFO
45    propagate: true
46  tornado.general:
47    handlers:
48    - core_tornado_debug_log_handler
49    level: DEBUG
50    propagate: true
51root:
52  handlers:
53  - server_handler
54  level: INFO
55version: 1
56disable_existing_loggers: false

Default configuration for the scheduler:

scheduler_log.yml.tmpl
 1# Generated using: inmanta -c /etc/inmanta/inmanta.cfg --log-file-level DEBUG --timed-logs output-default-logging-config --component scheduler scheduler_log.yml.tmpl
 2formatters:
 3  core_console_formatter:
 4    # Console formatter with coloring
 5    (): inmanta.logging.MultiLineFormatter
 6    fmt: '%(log_color)s%(name)-25s%(levelname)-8s%(reset)s%(blue)s%(message)s'
 7    # Don't shorten logger names by default
 8    keep_logger_names: true
 9    log_colors:
10      CRITICAL: red
11      DEBUG: cyan
12      ERROR: red
13      INFO: green
14      WARNING: yellow
15    no_color: false
16    reset: true
17  # Normal formatter
18  core_log_formatter:
19    format: '%(asctime)s %(levelname)-8s %(name)-10s %(message)s'
20handlers:
21  scheduler_handler:
22    # Root handler for the scheduler
23    class: logging.handlers.WatchedFileHandler
24    filename: /var/log/inmanta/agent-{environment}.log
25    formatter: core_log_formatter
26    level: DEBUG
27    mode: a+
28  scheduler_resource_action_handler:
29    # log only the resource actions as reported by the executors, live
30    class: logging.handlers.WatchedFileHandler
31    filename: /var/log/inmanta/resource-actions-{environment}.log
32    formatter: core_log_formatter
33    level: DEBUG
34  core_tornado_debug_log_handler:
35    # Special handler that captures tornado max_clients limit reached messages and turns them into warnings
36    # As these may cause instability
37    class: inmanta.logging.TornadoDebugLogHandler
38    level: DEBUG
39loggers:
40  inmanta.resource_action:
41    # parent for all resource action logs
42    # name is inmanta.resource_action.[agentname] for all agents
43    handlers:
44    - scheduler_resource_action_handler
45    level: DEBUG
46    propagate: true
47  tornado.general:
48    handlers:
49    - core_tornado_debug_log_handler
50    level: DEBUG
51    propagate: true
52root:
53  handlers:
54  - scheduler_handler
55  level: DEBUG
56version: 1
57disable_existing_loggers: false