Logging

This page describes the different logs files produced by the Inmanta server and its agents and explains what can be configured regarding to logging.

Overview different log files

By default log files are collected in the directory /var/log/inmanta/. The following files are expected:

  1. server.log is the main log file of the server. It shows general information about actions performed by the Inmanta server (renewing parameters, purging resource action logs, etc.), and the access log of the API.

  2. resource-actions-<environment-id>.log contains all actions performed by all resources. Each environment has one resource action log file. This file mirrors the actions logged on resources in the database.

  3. agent-<environment-id>.log is the main log of the scheduler and executors. It contains information about when any executor started a deployment, which trigger caused that deployment, whether heartbeat messages are received from the server, etc.

  4. agent-<environment-id>.out: This log file contains all the messages written to the standard output stream of the scheduler and executors. Expected to be empty.

  5. agent-<environment-id>.err: This log file contains all the messages written to the standard error stream of the scheduler and executors. Expected to be empty.

For reasons of backward compatibility, the scheduler files are called ‘agent’ and not ‘scheduler’

Configure logging

Logging can be configured in two main ways:

  • coarse grained configuration using configuration and command line options. This is sufficient in most cases.

  • fine grained configuration using a config file. Here the logging config is fully user controlled.

Coarse grained configuration

The following log-related options can be set in an Inmanta config file:

  • config.log-dir: determines the folder that will contain the log files

As command line options, the following are available:

  • --timed-logs: Add timestamps to logs

  • --log-file: Path to the logfile, enables logging to file, disables logging to console

  • --log-file-level: Log level for messages going to the logfile, options ERROR, WARNING, INFO, DEBUG and TRACE

  • -v: Log level for messages going to the console. Default is warnings only. -v warning, -vv info, -vvv debug and -vvvv trace.

  • -X: When exiting with an error, show full stack trace.

  • --keep-logger-names: When using the compiler, don’t shorten logger names.

To update the server startup config when using the RPM based install, copy the inmanta-server service file at /usr/lib/systemd/system/inmanta-server.service to /etc/systemd/system/inmanta-server.service and edit it.

[Unit]
Description=The server of the Inmanta platform
After=network.target

[Service]
Type=simple
User=inmanta
Group=inmanta
ExecStart=/usr/bin/inmanta --log-file /var/log/inmanta/server.log --log-file-level INFO --timed-logs server
Restart=on-failure

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl restart inmanta-server

Fine grained configuration

For fine grained configuration, a standard python dict config file can be passed in via the config file for each component individually:

[logging]
server = server_log.yml
scheduler = scheduler.log.tmpl
compiler = compiler.yml

or via a cli option:

inmanta --logging-config server_log.yml server

The log config has to be either a yaml file, containing a python dict config or a template of a yaml file. In this case, the file name has to end with tmpl.

The following log-related options can be set in an Inmanta config file:

As command line options, the following are available:

  • -v: When used in combination with a log file, it will force a CLI logger to be loaded on top of the provided configuration

  • --logging-config: Log configuration file for this command, overrides the config option

  • -X: When exiting with an error, show full stack trace.

  • If a config file is loaded, all other coarse grained configuration options are ignored!

For templated config files, we use python f-string syntax. For the scheduler, one variable is available in the template: {environment}. This is used to customize the scheduler log files to the environment the scheduler is working for.

Converting to fine grained configuration

A tool is provided to convert the existing coarse grained configuration into a config file.

For example, to convert the config for a component, take the command you use to start it, then put print-default-logging-config before the server, compiler or scheduler:

inmanta -c /etc/inmanta/inmanta.cfg \
    --log-file /var/log/inmanta/server.log \
    --log-file-level INFO \
    --timed-logs \
    print-default-logging-config server

Default configurations

Default configuration for the server:

server_log.yml
 1# Generated using inmanta -c /etc/inmanta/inmanta.cfg --log-file /var/log/inmanta/server.log --log-file-level 2 --timed-logs print_default_logging_config server
 2formatters:
 3  core_console_formatter:
 4    # Console formatter with coloring
 5    # This formatter also does indentation of multi-line log message, so all lines are put nicely underneath each other.
 6    (): inmanta.logging.MultiLineFormatter
 7    fmt: '%(asctime)s %(log_color)s%(name)-25s%(levelname)-8s%(reset)s%(blue)s%(message)s'
 8    # Don't shorten logger names by default
 9    keep_logger_names: true
10    log_colors:
11      CRITICAL: red
12      DEBUG: cyan
13      ERROR: red
14      INFO: green
15      WARNING: yellow
16    no_color: false
17    reset: true
18  # Normal formatter
19  core_log_formatter:
20    format: '%(asctime)s %(levelname)-8s %(name)-10s %(message)s'
21handlers:
22  server_handler:
23    # Main handler for the server
24    class: logging.handlers.WatchedFileHandler
25    filename: /var/log/inmanta/server.log
26    formatter: core_log_formatter
27    level: INFO
28    mode: a+
29  core_tornado_debug_log_handler:
30    # Special handler that captures tornado max_clients limit reached messages and turns them into warnings
31    # As these may cause instability
32    class: inmanta.logging.TornadoDebugLogHandler
33    level: DEBUG
34loggers:
35  tornado.general:
36    handlers:
37    - core_tornado_debug_log_handler
38    level: DEBUG
39    propagate: true
40root:
41  handlers:
42  - server_handler
43  level: INFO
44
45version: 1
46disable_existing_loggers: false

Default configuration for the scheduler:

scheduler_log.yml.tmpl
 1# Generated using inmanta -c /etc/inmanta/inmanta.cfg --log-file-level DEBUG --timed-logs print_default_logging_config scheduler
 2formatters:
 3  core_console_formatter:
 4    # Console formatter with coloring
 5    (): inmanta.logging.MultiLineFormatter
 6    fmt: '%(log_color)s%(name)-25s%(levelname)-8s%(reset)s%(blue)s%(message)s'
 7    # Don't shorten logger names by default
 8    keep_logger_names: true
 9    log_colors:
10      CRITICAL: red
11      DEBUG: cyan
12      ERROR: red
13      INFO: green
14      WARNING: yellow
15    no_color: false
16    reset: true
17  # Normal formatter
18  core_log_formatter:
19    format: '%(asctime)s %(levelname)-8s %(name)-10s %(message)s'
20handlers:
21  scheduler_handler:
22    # Root handler for the scheduler
23    class: logging.handlers.WatchedFileHandler
24    filename: /var/log/inmanta/agent-{environment}.log
25    formatter: core_log_formatter
26    level: DEBUG
27    mode: a+
28  scheduler_resource_action_handler:
29    # log only the resource actions as reported by the executors, live
30    class: logging.handlers.WatchedFileHandler
31    filename: /var/log/inmanta/resource-actions-{environment}.log
32    formatter: core_log_formatter
33    level: DEBUG
34  core_tornado_debug_log_handler:
35    # Special handler that captures tornado max_clients limit reached messages and turns them into warnings
36    # As these may cause instability
37    class: inmanta.logging.TornadoDebugLogHandler
38    level: DEBUG
39loggers:
40  inmanta.resource_action:
41    # parent for all resource action logs
42    # name is inmanta.resource_action.[agentname] for all agents
43    handlers:
44    - scheduler_resource_action_handler
45    level: DEBUG
46    propagate: true
47  tornado.general:
48    handlers:
49    - core_tornado_debug_log_handler
50    level: DEBUG
51    propagate: true
52root:
53  handlers:
54  - scheduler_handler
55  level: DEBUG
56version: 1
57disable_existing_loggers: false