This page describes typical failure scenario’s and provides a guideline on how to troubleshoot them.
A resources is stuck in the state available¶
When a resource is stuck in the available state, it usually means that the agent, which should deploy the resource, is currently down or paused. Click on the version of the configuration model, shown in the versions tab of the Inmanta dashboard, to get an overview of the different resources in the model. This overview shows the state of each resource and the name of its agent. Filter on resources in the available state and check which resource are ready to be deployed (i.e. a resource without dependencies or a resource for which all dependencies were deployed successfully). The agent of that resource, is the agent that causes the problem. In the figure below, the epel-release package should be ready to deploy on agent vm2
Next, go to the agents tab of the dashboard to verify the state of that agent.
An agent can be in one of the following states:
Each of the following subsections describes what should be done when the agent is in each of the different states.
The agent is down¶
The Section Agent doesn’t come up provides information on how to troubleshoot the scenario where an agent that shouldn’t be down is down.
The agent is paused¶
Unpause the agent by clicking the
Unpause agent button in the agents tab of the dashboard.
The agent is up¶
When the agent is in the up state, it should be ready to deploy resources. Read the agent log to verify it doesn’t contain
error or warning messages that would explain why the agent is not deploying any resources. For auto-started agents, three
different log files exist. The log files are present in
environment ID can be found in the URL of the dashboard. More information about the different log files can be found
here. For manually started agents the log file is present in
/var/log/inmanta/agent.log. If the log file doesn’t provide any more information, trigger the agents to execute a
deployment by clicking on the
Force Repair button in the versions tab of the dashboard, as shown in the figure below:
When the agent receives the notification from the server, it writes the following log message in its log:
INFO inmanta.agent.agent Agent <agent-name> got a trigger to update in environment <environment ID>
If the notification from the server doesn’t appear in the log file of the agent after clicking the
Force Repair button, the
problem is situated on the server side. Check if the server log contains any error messages or warning that could explain
the reason why the agent didn’t get a notification from the server. The server log file is situated at
The deployment of a resource fails¶
When a resource cannot be deployed, it ends up in one of the following deployment states:
failed: A resource ends up in the
failedstate when the handler of that resource raises an uncaught exception. Check the log of the resource to get more details about the issue.
unavailable: A resource ends up in the
unavailablestate when no handler could be found to deploy that resource. Check the log of the resource to get more details about the issue.
undefined: A resource ends up in the
undefinedstate when a fact, required by that resource didn’t yet resolve to a value. Read Section Check which facts are not yet resolved to find out which fact is still unknown.
skipped: When a resource is in the
skippedstate, it can mean two different things. Either the resource cannot be deployed because one of its dependencies ended up the failed state or the handler itself raised a SkipResource exception to indicate that the resource in not yet ready to be deployed. The latter case can occur when a VM is still booting for example. Check the log of the resource to get more information about actual root cause.
skipped_for_undefinedstate indicates that the resource cannot be deployed because one of its dependencies cannot be deployed. Check the log of the resource to get information about the actual dependency that cannot be deployed.
Read the logs of a resource¶
This section describes how to obtain the logs for a specific resource. In the versions tab of the dashboard, click on the version of the configuration model being deployed to get a list of all the resource in that configuration model. Next, click on the magnifier in front of a resource, as shown in the figure below, to get the logs for that specific resource. The log messages for the different stages of the deployment are grouped together.
The magnifier in front of each log message can be used to get a more structured output for that specific log message.
In the figure below, the traceback of the exception is shown.
Check which facts are not yet resolved¶
To find out which fact of a certain resource is not yet resolved, click on the magnifier in front of the resource in the
undefined state, as shown in the figure below.
The list of attributes of that resource, will contain one attribute which is marked as undefined (See figure below). This is the attribute that wasn’t resolved yet. Track the source of this attribute down within the configuration model to find out why this attribute is undefined.
Agent doesn’t come up¶
This section explains how to troubleshoot the problem where an agent is in the down state while it should be up. In the figure shown below, the agent vm1 is down.
Agents can be started in two different ways, either automatically by the inmanta server (auto-started agents) or manually (manually-started) agents. More information about the configuration of both types of agent can be found on this page. The Section Auto-started agents describes how to troubleshoot this issue for agents started by the Inmanta server. The Section Manually-started agents describes how to troubleshoot this issue for agents that were started manually.
An auto-started agent is only started when that agent is present in the
autostart_agent_map environment setting. Verify that
requirement via the settings tab of the inmanta dashboard as shown in the figure below.
autostart_agent_map is configured correctly, but the agent is still not up, read the logs of the auto-started agent
. These logs can be found at the following location:
environment ID is present in the URL of the dashboard. More information about the different log files can be found
here. When reading those log files, pay specific attention to error
messages and warnings that could explain why the agent is marked as down. Also, ensure that the name of the agent under
consideration is added as an endpoint to the agent process. The log file should contain the following message when a certain
agent is added as an endpoint to the process:
inmanta.agent.agent Adding endpoint <agent-name>
When the agent is not added as an endpoint, log an issue on https://github.com/inmanta/inmanta-core/issues.
An autostarted-agent connects to the Inmanta server via the address configured in the
server.server-address config option. If this option is set incorrectly, the agent will not be able to
connect to the server.
Manually started agents¶
When a manually-started agent doesn’t come up, verify whether the agent process is still running via the following command:
$ systemctl status inmanta-agent
If the agent process is down, start and enable it via the following command:
$ systemctl enable --now inmanta-agent
Also check the log file of the manually-started agent. This log file is located at
/var/log/inmanta/agent.log. The standard
output and the standard error streams produced by the agent, can be obtained via journalctl:
$ journalctl -u inmanta-agent
Potential reasons why an agent doesn’t start¶
This section provides a list of potential reasons why an agent wouldn’t start:
bind-address set incorrectly: The Inmanta server listens on all the interfaces configured via the
server.bind-addressoption. If the server doesn’t listen on an interface used by a remote agent, the agent will not be able to connect to the server.
Authentication issue: If the Inmanta server has been setup with authentication, a misconfiguration may deny an agent access to the Inmanta API. For example, not configuring a token provider (issuer) with
auth_jwt_<ID>section of the Inmanta configuration file. Documentation on how to configure authentication correctly can be found here.
SSL problems: If the Inmanta server is configured to use SSL, the Agent should be configured to use SSL as well (See the SSL-related configuration options in the
agent_rest_transportsection of the Inmanta configuration reference)
Network issue: Many network-related issue may exist which don’t allow the agent to establish a connection with the Inmanta server. A firewall may blocks traffic between the Inmanta agent and the server, no network route may exist towards the Inmanta server, etc.
No version appears after recompile trigger¶
After clicking the
Recompile button of the dashboard, a new version of the configuration model should appear in the list of
versions. If this doesn’t happen, the compilation has failed. Click on the
Compile Reports button, as shown in the
figure below, to get the compile report of the latest compilation. This report will give more information about the exact
Each step of the compile process is shown, together with the output produced by that step and the return code. Verify that the
timestamp of the compile report corresponds to the time the compilation was triggered in the dashboard. If no compile report was
generated or the compile report doesn’t show any errors, check the server logs as well. By default the server log is present in
Logs show “empty model” after export¶
This log message indicates that something went wrong during the compilation or the export of the model to the server. To get
more information about the problem, rerun the command with the
-vvv and the
-X options. The
-vvv option increases
the log level of the command to the DEBUG level and the
-X option shows stack traces and errors.
$ inmanta -vvv export -X
Debugging the server is possible in case the rpdb package is installed.
rpdb package to the virtual environment used by Inmanta by default can be done the following way:
$ /opt/inmanta/bin/python3 -m pip install rpdb
Rpdb can be triggered by sending a TRAP signal to the inmanta server process.
$ kill -5 <PID>
After receiving the signal, the process hangs, and it’s possible to attach a
by connecting to 127.0.0.1, on port 4444 (for example using telnet).