Troubleshooting

This page describes typical failure scenario’s and provides a guideline on how to troubleshoot them.

A resources is stuck in the state available

When a resource is stuck in the available state, it usually means that the agent, which should deploy the resource, is currently down or paused. Click on the version of the configuration model, shown in the versions tab of the Inmanta dashboard, to get an overview of the different resources in the model. This overview shows the state of each resource and the name of its agent. Filter on resources in the available state and check which resource are ready to be deployed (i.e. a resource without dependencies or a resource for which all dependencies were deployed successfully). The agent of that resource, is the agent that causes the problem. In the figure below, the epel-release package should be ready to deploy on agent vm2

_images/resources_overview_stuck_in_available_state.png

Next, go to the agents tab of the dashboard to verify the state of that agent.

_images/agent_is_paused.png

An agent can be in one of the following states:

  • Down

  • Paused

  • Up

Each of the following subsections describes what should be done when the agent is in each of the different states.

The agent is down

The Section Agent doesn’t come up provides information on how to troubleshoot the scenario where an agent that shouldn’t be down is down.

The agent is paused

Unpause the agent by clicking the Unpause agent button in the agents tab of the dashboard.

_images/agent_is_paused.png

The agent is up

When the agent is in the up state, it should be ready to deploy resources. Read the agent log to verify it doesn’t contain error or warning messages that would explain why the agent is not deploying any resources. For auto-started agents, three different log files exist. The log files are present in <config.log-dir>/agent-<environment-id>.[log|out|err]. The environment ID can be found in the URL of the dashboard. More information about the different log files can be found here. For manually started agents the log file is present in /var/log/inmanta/agent.log. If the log file doesn’t provide any more information, trigger the agents to execute a deployment by clicking on the Force Repair button in the versions tab of the dashboard, as shown in the figure below:

_images/force_repair_button.png

When the agent receives the notification from the server, it writes the following log message in its log:

INFO     inmanta.agent.agent Agent <agent-name> got a trigger to update in environment <environment ID>

If the notification from the server doesn’t appear in the log file of the agent after clicking the Force Repair button, the problem is situated on the server side. Check if the server log contains any error messages or warning that could explain the reason why the agent didn’t get a notification from the server. The server log file is situated at <config.log-dir>/server.log.

The deployment of a resource fails

When a resource cannot be deployed, it ends up in one of the following deployment states:

  • failed: A resource ends up in the failed state when the handler of that resource raises an uncaught exception. Check the log of the resource to get more details about the issue.

  • unavailable: A resource ends up in the unavailable state when no handler could be found to deploy that resource. Check the log of the resource to get more details about the issue.

  • undefined: A resource ends up in the undefined state when a fact, required by that resource didn’t yet resolve to a value. Read Section Check which facts are not yet resolved to find out which fact is still unknown.

  • skipped: When a resource is in the skipped state, it can mean two different things. Either the resource cannot be deployed because one of its dependencies ended up the failed state or the handler itself raised a SkipResource exception to indicate that the resource in not yet ready to be deployed. The latter case can occur when a VM is still booting for example. Check the log of the resource to get more information about actual root cause.

  • skipped_for_undefined: The skipped_for_undefined state indicates that the resource cannot be deployed because one of its dependencies cannot be deployed. Check the log of the resource to get information about the actual dependency that cannot be deployed.

Read the logs of a resource

This section describes how to obtain the logs for a specific resource. In the versions tab of the dashboard, click on the version of the configuration model being deployed to get a list of all the resource in that configuration model. Next, click on the magnifier in front of a resource, as shown in the figure below, to get the logs for that specific resource. The log messages for the different stages of the deployment are grouped together.

_images/get_logs_failed_resource.png

The magnifier in front of each log message can be used to get a more structured output for that specific log message.

_images/action_log.png

In the figure below, the traceback of the exception is shown.

_images/action_log_specific_message.png

Check which facts are not yet resolved

To find out which fact of a certain resource is not yet resolved, click on the magnifier in front of the resource in the undefined state, as shown in the figure below.

_images/resources_in_the_undefined_state.png

The list of attributes of that resource, will contain one attribute which is marked as undefined (See figure below). This is the attribute that wasn’t resolved yet. Track the source of this attribute down within the configuration model to find out why this attribute is undefined.

_images/undefined_attribute.png

Agent doesn’t come up

This section explains how to troubleshoot the problem where an agent is in the down state while it should be up. In the figure shown below, the agent vm1 is down.

_images/agent_in_down_state.png

Agents can be started in two different ways, either automatically by the inmanta server (auto-started agents) or manually (manually-started) agents. More information about the configuration of both types of agent can be found on this page. The Section Auto-started agents describes how to troubleshoot this issue for agents started by the Inmanta server. The Section Manually-started agents describes how to troubleshoot this issue for agents that were started manually.

Auto-started agents

An auto-started agent is only started when that agent is present in the autostart_agent_map environment setting. Verify that requirement via the settings tab of the inmanta dashboard as shown in the figure below.

_images/environment_settings_autostart_agent_map.png

When the autostart_agent_map is configured correctly, but the agent is still not up, read the logs of the auto-started agent . These logs can be found at the following location: <config.log-dir>/agent-<environment-id>.[log|out|err]. The environment ID is present in the URL of the dashboard. More information about the different log files can be found here. When reading those log files, pay specific attention to error messages and warnings that could explain why the agent is marked as down. Also, ensure that the name of the agent under consideration is added as an endpoint to the agent process. The log file should contain the following message when a certain agent is added as an endpoint to the process:

inmanta.agent.agent Adding endpoint <agent-name>

When the agent is not added as an endpoint, log an issue on https://github.com/inmanta/inmanta/issues.

An autostarted-agent connects to the Inmanta server via the address configured in the server.server-address config option. If this option is set incorrectly, the agent will not be able to connect to the server.

Manually started agents

When a manually-started agent doesn’t come up, verify whether the agent process is still running via the following command:

$ systemctl status inmanta-agent

If the agent process is down, start and enable it via the following command:

$ systemctl enable --now inmanta-agent

Also check the log file of the manually-started agent. This log file is located at /var/log/inmanta/agent.log. The standard output and the standard error streams produced by the agent, can be obtained via journalctl:

$ journalctl -u inmanta-agent

Potential reasons why an agent doesn’t start

This section provides a list of potential reasons why an agent wouldn’t start:

  • bind-address set incorrectly: The Inmanta server listens on all the interfaces configured via the server.bind-address option. If the server doesn’t listen on an interface used by a remote agent, the agent will not be able to connect to the server.

  • Authentication issue: If the Inmanta server has been setup with authentication, a misconfiguration may deny an agent access to the Inmanta API. For example, not configuring a token provider (issuer) with sign=true in the auth_jwt_<ID> section of the Inmanta configuration file. Documentation on how to configure authentication correctly can be found here.

  • SSL problems: If the Inmanta server is configured to use SSL, the Agent should be configured to use SSL as well (See the SSL-related configuration options in the server and agent_rest_transport section of the Inmanta configuration reference)

  • Network issue: Many network-related issue may exist which don’t allow the agent to establish a connection with the Inmanta server. A firewall may blocks traffic between the Inmanta agent and the server, no network route may exist towards the Inmanta server, etc.

No version appears after recompile trigger

After clicking the Recompile button of the dashboard, a new version of the configuration model should appear in the list of versions. If this doesn’t happen, the compilation has failed. Click on the Compile Reports button, as shown in the figure below, to get the compile report of the latest compilation. This report will give more information about the exact problem.

_images/compile_report_button.png

Each step of the compile process is shown, together with the output produced by that step and the return code. Verify that the timestamp of the compile report corresponds to the time the compilation was triggered in the dashboard. If no compile report was generated or the compile report doesn’t show any errors, check the server logs as well. By default the server log is present in <config.log-dir>/server.log.

_images/compile_report.png

Logs show “empty model” after export

This log message indicates that something went wrong during the compilation or the export of the model to the server. To get more information about the problem, rerun the command with the -vvv and the -X options. The -vvv option increases the log level of the command to the DEBUG level and the -X option shows stack traces and errors.

$ inmanta -vvv export -X