Allocation V2

Allocation V2 is a new framework, similar to allocation (v1). It happens in the same lifecycle stage and serves the same purpose: filling up read-only values of a service instance.

It comes to fill some gaps in the functionalities of allocation (v1) and takes advantages of the experience and learnings that using allocation (v1) taught us. It is a more complete, functional, and elegant framework.

Example

The example below show you the use case where a single allocator is used the same way on both the service instance and an embedded entity.

main.cf
 1import lsm
 2import lsm::fsm
 3
 4
 5entity ValueService extends lsm::ServiceEntity:
 6    string                      name
 7    lsm::attribute_modifier     name__modifier="rw"
 8
 9    int?                        first_value
10    lsm::attribute_modifier     first_value__modifier="r"
11end
12ValueService.embedded_values [0:] -- EmbeddedValue
13
14entity EmbeddedValue extends lsm::EmbeddedEntity:
15    string                      id
16    lsm::attribute_modifier     id__modifier="rw"
17
18    int?                        third_value
19    lsm::attribute_modifier     third_value__modifier="r"
20
21    string[]? __lsm_key_attributes = ["id"]
22end
23
24index EmbeddedValue(id)
25
26implement ValueService using parents
27implement EmbeddedValue using std::none
28
29binding = lsm::ServiceEntityBinding(
30    service_entity="__config__::ValueService",
31    lifecycle=lsm::fsm::simple,
32    service_entity_name="value-service",
33    allocation_spec="value_allocation",
34    strict_modifier_enforcement=true,
35)
36
37for assignment in lsm::all(binding):
38    attributes = assignment["attributes"]
39
40    service = ValueService(
41        instance_id=assignment["id"],
42        entity_binding=binding,
43        name=attributes["name"],
44        first_value=attributes["first_value"],
45    )
46
47    for embedded_value in attributes["embedded_values"]:
48        service.embedded_values += EmbeddedValue(
49            **embedded_value
50        )
51    end
52end
plugins/__init__.py
 1"""
 2    Inmanta LSM
 3
 4    :copyright: 2022 Inmanta
 5    :contact: code@inmanta.com
 6    :license: Inmanta EULA
 7"""
 8
 9from inmanta.util import dict_path
10from inmanta_plugins.lsm.allocation import AllocationSpecV2
11from inmanta_plugins.lsm.allocation_v2.framework import AllocatorV2, ContextV2, ForEach
12
13
14class IntegerAllocator(AllocatorV2):
15    def __init__(self, value: int, attribute: str) -> None:
16        self.value = value
17        self.attribute = dict_path.to_path(attribute)
18
19    def needs_allocation(self, context: ContextV2) -> bool:
20        try:
21            if not context.get_instance().get(self.attribute):
22                # Attribute not present
23                return True
24        except IndexError:
25            return True
26        return False
27
28    def allocate(self, context: ContextV2) -> None:
29        context.set_value(self.attribute, self.value)
30
31
32AllocationSpecV2(
33    "value_allocation",
34    IntegerAllocator(value=1, attribute="first_value"),
35    ForEach(
36        item="item",
37        in_list="embedded_values",
38        identified_by="id",
39        apply=[
40            IntegerAllocator(
41                value=3,
42                attribute="third_value",
43            ),
44        ],
45    ),
46)

Allocation V2 features

The two main additions to allocation v2 when compared to v1 are:

  • The new ContextV2 object (replacement for AllocationContext object), which goes in pair with AllocatorV2 and AllocationSpecV2

  • The support for allocating attributes in embedded entities.

Setting a read-only attribute on an embedded entity, like done in the above-mentioned example, is only possible when strict_modifier_enforcement is enabled. On legacy services, where strict_modifier_enforcement is not enabled, read-only attributes can be set on embedded entities using the workaround mentioned the Section Legacy: Set attributes on embedded entities.

Warning

To use allocation safely, allocators should not keep any state between invocations, but pass all state via the ContextV2 object.

ContextV2

A context object that will be passed to each allocator and that should be used to set values. This context always shows the attributes the allocator should have access to, based on its level in the allocators tree. This means a top level allocator will see all the attributes, but an allocator used on embedded entities will only see the attributes of such embedded entity (as if it was a standalone entity). The context object can also be used to store values at each “level of allocation”, reachable by all allocators at the same level.

class inmanta_plugins.lsm.allocation_v2.framework.ContextV2(instance: ServiceInstance, plugin_context: Context)[source]

All interactions between allocators happen via this context.

The context is responsible for

  1. exposing the current instance to the allocator

  2. allow the allocator to set values

  3. allow allocators to be used in the same way on embedded instances as on top-level instances

  4. carry shared state between allocators (such as connections to the inventory)

Each Context exposes a ServiceInstance via the get_instance() method. It exposes this instance as if it is the service itself. I.e. embedded entities are presented in the same way as top level services.

To allow the context to carry additional state (such as a connection to an inventory), make a subclass of this class.

Variables:

  • plugin_context – The plugin context attached to this context

  • instance – The instance this context is attached to

  • flush_set – A dict containing a trace of all the set_value call that where made on this context object. The key is the :param path: that was passed to the method The value is the :param value: that was passed to the method

build_child(path: DictPath, wild_path: WildDictPath) → ChildContextV2[source]

Produce a new child instance of this context

(this method acts as a polymorphic constructor)

Parameters:

  • path – the path for the new context, relative to the parent

  • wild_path – the path for the new context, or any context at the same level, relative to the parent

classmethod clone(instance: ServiceInstance, plugin_context: Context) → TC[source]

Produce a new instance of this context

(this method acts as a polymorphic constructor)

Parameters:

instance – the instance this new context should be attached to

property environment: str

Return the environment id

get_child(relation: str) → ChildContextV2[source]

Not intended for external usage

Idempotent method

Get a context for the child that matches the path relation

get_full_path() → DictPath | None[source]

Get the full path up to the instance in this context. The root context should return None as there is no path required to reach it.

get_instance() → ServiceInstance[source]
Returns:

Get the service instance bound to this context

get_list_children(relation: str, key_attribute: str) → List[ChildContextV2][source]

Not intended for external usage

Idempotent method

Get a context for every child that matches the path relation[key_attribute=*] in the attributes set of the instance

get_wild_full_path() → WildDictPath | None[source]

Get the full path up to the instance in this context, replacing all KeyedList by WildKeyedList. The root context should return None as there is no path required to reach it.

set_value(path: DictPath, value: object) → None[source]

Set the value at the given path and make sure this allocation is written back to the LSM database.

This call will update the ServiceInstance and the LSM database.

The flushing to database can either be when allocation succeeds, when it fails or immediately. This behavior is determined by the context.

Parameters:

  • path – the path at which to set the value

  • value – the value to set at the given path

In the example at the beginning of this page, the same allocator can be used to set a value on the service entity and an embedded entity. In needs_allocation, when calling context.get_instance(), we receive as dict the full service entity when allocating first_value and the embedded entity when allocating third_value.

AllocatorV2

A base class for all v2 allocators, they are provided with a ContextV2 object for those two methods: needs_allocation and allocate. The main difference with v1, is that the allocate method doesn’t return any value to allocate, it sets them using the context object: context.set_value(name, value).

class inmanta_plugins.lsm.allocation_v2.framework.AllocatorV2[source]

Base class for all V2 Allocators

abstract allocate(context: ContextV2) → None[source]

Perform the actual allocation

Allocated values will be written back to the context using ContextV2.set_value()

abstract needs_allocation(context: ContextV2) → bool[source]

Determine if this allocator has any work to do or if all values have already been allocated correctly.

post_allocate() → None[source]

Called after allocation, can be used e.g., to teardown a connection to the external inventory

pre_allocate() → None[source]

Called before allocation, can be used e.g., to establish a connection to the external inventory

AllocationSpecV2

The collector for all AllocatorV2.

class inmanta_plugins.lsm.allocation.AllocationSpecV2(name: str, *allocators: AllocatorV2)[source]
__init__(name: str, *allocators: AllocatorV2) → None[source]

Create the allocation spec and register it in the global collector

Parameters:

  • name – The name of the allocator, must be unique

  • allocators – The allocators contained in this aggregator

Legacy: Set attributes on embedded entities

The server doesn’t have support to set read-only attributes on embedded entities when strict_modifier_enforcement is disabled. Thanks to the allocator ContextV2Wrapper and the plugin lsm::context_v2_unwrapper a workaround exists to do allocation on an embedded entity’s attributes with strict_modifier_enforcement disabled. This workaround saves all the allocated values in a dict, in an attribute of the service instance (added to the instance for this single purpose). That way, the server accepts the update.

class inmanta_plugins.lsm.allocation_v2.framework.ContextV2Wrapper(fallback_attribute: str, *allocators: AllocatorV2, track_deletes: bool = False)[source]

This wrapper allows to use allocators for attributes of embedded entities. It performs the allocation for all the provided allocators, and save/allocate all of their allocated values in a single key.

This wrapper can be used in pair with the context_v2_unwrapper plugin, which allows to place all those allocated values back where they are supposed to go.

__init__(fallback_attribute: str, *allocators: AllocatorV2, track_deletes: bool = False) → None[source]

This wrapper allows to use allocators for attributes of embedded entities. It will condense all the changes in a dict: - whose keys are dict_path to the attribute of the embedded entities - whose values are the one to allocate

This dict is itself saved at the root of the changed attributes dict, saved in the fallback attributes specified in the constructor.

Parameters:

  • fallback_attributes – The attribute in which all the allocated values should be saved

  • allocators – The allocators whose values should be saved

  • track_deletes – when this is set, when an embedded entity is deleted during an update, it will also be dropped from the fallback_attribute. This also implies that embedded entities which are created by allocation alone (all attributes of type ‘r’) are not supported.

e.g:

lsm.AllocationSpecV2(
    "service_allocator_spec",
    allocation.ContextV2Wrapper(
        "allocated",
        allocation.ForEach(
            item="subservice",
            in_list="subservices",
            identified_by="id",
            apply=[
                ValueAllocator(
                    into="to_allocate",
                ),
            ],
        ),
    ),
)

This would generate as update attribute dict something like

{
    "allocated": {
        "subservices[id=1].to_allocate": "a",
        "subservices[id=2].to_allocate": "b",
    }
}
allocate(context: ContextV2) → None[source]

Run allocation for all the children, then aggregate all the allocated values into a single value, in the parent context.

needs_allocation(context: ContextV2) → bool[source]

Needs allocation if any of the children allocators need allocation.

inmanta_plugins.lsm.context_v2_unwrapper(assignments: dict[], fallback_attribute: string, track_deletes: bool = False) → dict[][source]

This plugin can be used to wrap the instances coming out of lsm::all and place all allocated values in :param fallback_attribute: where they should go. The returned value is what has been given as input, except for the allocated values being set where they should. :param track_deletes: drop deleted embedded entities, even if they still exist in the fallback set.

This should be used together with ContextV2Wrapper allocator.

Each assignment is an attribute dict containing a fallback attribute assigned with allocated values as produced by the ContextV2 (one-level deep, keys are string representation of the paths, values are allocated values) and update the dict placing values where their key-path would reach them.

Parameters:

  • assignments – The list of service instance dictionaries as returned by lsm::all

  • fallback_attributes – The attribute name at the root of the instance attributes that contains all the allocated values

e.g.:

context_v2_unwrapper(
    [
        {
            "environment": "8f7bf3a5-d655-4bcb-bbd4-6222407be999",
            "id": "f93acfad-7894-4a12-9770-b27cbdd85c74",
            "service_entity": "carrierEthernetEvc",
            "version": 4,
            "config": {},
            "state": "allocating",
            "attributes": {
                "allocated": {
                    "evcEndPoints[identifier=my-evc-ep-1].uni": {
                        "uniref": "inmanta:456-852-789",
                        "test_value": "test_value",
                    },
                    "evcEndPoints[identifier=my-evc-ep-2].uni": {
                        "uniref": "inmanta:123-852-456",
                        "test_value": "test_value",
                    },
                },
                "another_key": "any value",
            },
            "candidate_attributes": {
                "allocated": {
                    "evcEndPoints[identifier=my-evc-ep-1].uni": {
                        "uniref": "inmanta:456-852-789",
                        "test_value": "test_value",
                    },
                    "evcEndPoints[identifier=my-evc-ep-2].uni": {
                        "uniref": "inmanta:123-852-456",
                        "test_value": "test_value",
                    },
                },
                "another_key": "any value",
            },
            "active_attributes": {},
            "rollback_attributes": {},
        }
    ],
    "allocated",
)

will return:

[
    {
        "environment": "8f7bf3a5-d655-4bcb-bbd4-6222407be999",
        "id": "f93acfad-7894-4a12-9770-b27cbdd85c74",
        "service_entity": "carrierEthernetEvc",
        "version": 4,
        "config": {},
        "state": "allocating",
        "attributes": {
            "allocated": {
                "evcEndPoints[identifier=my-evc-ep-1].uni": {
                    "uniref": "inmanta:456-852-789",
                    "test_value": "test_value",
                },
                "evcEndPoints[identifier=my-evc-ep-2].uni": {
                    "uniref": "inmanta:123-852-456",
                    "test_value": "test_value",
                },
            },
            "evcEndPoints": [
                {
                    "identifier": "my-evc-ep-1",
                    "uni": {
                        "uniref": "inmanta:456-852-789",
                        "test_value": "test_value",
                    },
                },
                {
                    "identifier": "my-evc-ep-2",
                    "uni": {
                        "uniref": "inmanta:123-852-456",
                        "test_value": "test_value",
                    },
                },
            ],
            "another_key": "any value",
        },
        "candidate_attributes": {
            "allocated": {
                "evcEndPoints[identifier=my-evc-ep-1].uni": {
                    "uniref": "inmanta:456-852-789",
                    "test_value": "test_value",
                },
                "evcEndPoints[identifier=my-evc-ep-2].uni": {
                    "uniref": "inmanta:123-852-456",
                    "test_value": "test_value",
                },
            },
            "evcEndPoints": [
                {
                    "identifier": "my-evc-ep-1",
                    "uni": {
                        "uniref": "inmanta:456-852-789",
                        "test_value": "test_value",
                    },
                },
                {
                    "identifier": "my-evc-ep-2",
                    "uni": {
                        "uniref": "inmanta:123-852-456",
                        "test_value": "test_value",
                    },
                },
            ],
            "another_key": "any value",
        },
        "active_attributes": {},
        "rollback_attributes": {},
    }

]

The ContextV2Wrapper, which has to be used at the root of the allocation tree, will collect and save all the allocated value in a single dict. And when getting all the service instances in your model, with lsm::all, you can simply wrap the call to lsm::all with a call to lsm::context_v2_unwrapper, which will place all the allocated values saved in the dict, directly where they belong, in the embedded entities.

When using the ContextV2Wrapper and the lsm::context_v2_unwrapper plugin, you will have to specify in which attributes all the allocated values should be saved.

main.cf
 1import lsm
 2import lsm::fsm
 3
 4
 5entity ValueService extends lsm::ServiceEntity:
 6    string                      name
 7    lsm::attribute_modifier     name__modifier="rw"
 8
 9    int?                        first_value
10    lsm::attribute_modifier     first_value__modifier="r"
11
12    dict?                       allocated
13    lsm::attribute_modifier     allocated__modifier="r"
14end
15ValueService.embedded_values [0:] -- EmbeddedValue
16
17entity EmbeddedValue extends lsm::EmbeddedEntity:
18    string                      id
19    lsm::attribute_modifier     id__modifier="rw"
20
21    int?                        third_value
22    lsm::attribute_modifier     third_value__modifier="r"
23end
24
25implement ValueService using parents
26implement EmbeddedValue using std::none
27
28binding = lsm::ServiceEntityBinding(
29    service_entity="__config__::ValueService",
30    lifecycle=lsm::fsm::simple,
31    service_entity_name="value-service",
32    allocation_spec="value_allocation",
33)
34
35for assignment in lsm::context_v2_unwrapper(
36    assignments=lsm::all(binding),
37    fallback_attribute="allocated",
38):
39    attributes = assignment["attributes"]
40
41    service = ValueService(
42        instance_id=assignment["id"],
43        entity_binding=binding,
44        name=attributes["name"],
45        first_value=attributes["first_value"],
46        allocated=attributes["allocated"],
47    )
48
49    for embedded_value in attributes["embedded_values"]:
50        service.embedded_values += EmbeddedValue(
51            **embedded_value
52        )
53    end
54end
plugins/__init__.py
 1"""
 2    Inmanta LSM
 3
 4    :copyright: 2022 Inmanta
 5    :contact: code@inmanta.com
 6    :license: Inmanta EULA
 7"""
 8
 9from inmanta.util import dict_path
10from inmanta_plugins.lsm.allocation import AllocationSpecV2
11from inmanta_plugins.lsm.allocation_v2.framework import (
12    AllocatorV2,
13    ContextV2,
14    ContextV2Wrapper,
15    ForEach,
16)
17
18
19class IntegerAllocator(AllocatorV2):
20    def __init__(self, value: int, attribute: str) -> None:
21        self.value = value
22        self.attribute = dict_path.to_path(attribute)
23
24    def needs_allocation(self, context: ContextV2) -> bool:
25        try:
26            if not context.get_instance().get(self.attribute):
27                # Attribute not present
28                return True
29        except IndexError:
30            return True
31        return False
32
33    def allocate(self, context: ContextV2) -> None:
34        context.set_value(self.attribute, self.value)
35
36
37AllocationSpecV2(
38    "value_allocation",
39    IntegerAllocator(value=1, attribute="first_value"),
40    ContextV2Wrapper(
41        "allocated",
42        ForEach(
43            item="item",
44            in_list="embedded_values",
45            identified_by="id",
46            apply=[
47                IntegerAllocator(
48                    value=3,
49                    attribute="third_value",
50                ),
51            ],
52        ),
53    ),
54)

To facilitate allocation on embedded entities, the ForEach allocator can be used.

class inmanta_plugins.lsm.allocation_v2.framework.ForEach(item: str, in_list: str, identified_by: str, apply: List[AllocatorV2])[source]

Allocator to apply a given set of allocators to each instance in a list of instances.

The instances are uniquely identified by an attribute

__init__(item: str, in_list: str, identified_by: str, apply: List[AllocatorV2]) → None[source]
Parameters:

  • item – name of the loop variable (not used, to improve readability and error reporting)

  • in_list – the relation from which to get the items

  • identified_by – the identifying attribute of each item

  • apply – the allocators to apply to each item

e.g:

entity Service extends lsm::ServiceInstance:
end

entity SubService extends lsm::EmbeddedEntity:
    string id
    string to_allocate
    lsm::attribute_modifier to_allocate__modifier="r"
end

Service.subservices [0:] -- SubService

lsm.AllocationSpecV2(
    "service_allocator_spec",
    allocation.ContextV2Wrapper(
        "allocated",
        allocation.ForEach(
            item="subservice",
            in_list="subservices",
            identified_by="id",
            apply=[
                ValueAllocator(
                    into="to_allocate",
                ),
            ],
        ),
    ),
)
allocate(context: ContextV2) → None[source]

Perform the actual allocation

Allocated values will be written back to the context using ContextV2.set_value()

needs_allocation(context: ContextV2) → bool[source]

Determine if this allocator has any work to do or if all values have already been allocated correctly.

Deleting of Embedded entities

When you want to support deletion of embedded entities during updates, a slightly different configuration is needed. Because all allocated values are stored in a single attribute, the deleted entities will be recreated when unwrapping.

To prevent this, use track_deletes=true on both the the allocator ContextV2Wrapper and the plugin lsm::context_v2_unwrapper

Additionally, to re-trigger allocation when an item is deleted, use SetSensitiveForEach instead of ForEach.

main.cf
 1import lsm
 2import lsm::fsm
 3
 4
 5entity ValueService extends lsm::ServiceEntity:
 6    string                      name
 7    lsm::attribute_modifier     name__modifier="rw"
 8
 9    int?                        first_value
10    lsm::attribute_modifier     first_value__modifier="r"
11
12    dict?                       allocated
13    lsm::attribute_modifier     allocated__modifier="r"
14end
15ValueService.embedded_values [0:] lsm::__rwplus__ EmbeddedValue
16
17entity EmbeddedValue extends lsm::EmbeddedEntity:
18    string                      id
19    lsm::attribute_modifier     id__modifier="rw"
20
21    int?                        third_value
22    lsm::attribute_modifier     third_value__modifier="r"
23
24    string                      other_value
25    lsm::attribute_modifier     other_value__modifier="rw"
26end
27
28index EmbeddedValue(id)
29
30implement ValueService using parents
31implement EmbeddedValue using std::none
32
33binding = lsm::ServiceEntityBindingV2(
34    service_entity="__config__::ValueService",
35    lifecycle=lsm::fsm::simple,
36    service_entity_name="value-service",
37    allocation_spec="value_allocation",
38)
39
40for assignment in lsm::context_v2_unwrapper(
41    assignments=lsm::all(binding),
42    fallback_attribute="allocated",
43    track_deletes=true,
44):
45    attributes = assignment["attributes"]
46
47    service = ValueService(
48        instance_id=assignment["id"],
49        entity_binding=binding,
50        name=attributes["name"],
51        first_value=attributes["first_value"],
52        allocated=attributes["allocated"],
53    )
54
55    for embedded_value in attributes["embedded_values"]:
56        service.embedded_values += EmbeddedValue(
57            **embedded_value
58        )
59    end
60end
plugins/__init__.py
 1"""
 2    Inmanta LSM
 3
 4    :copyright: 2020 Inmanta
 5    :contact: code@inmanta.com
 6    :license: Inmanta EULA
 7"""
 8
 9from inmanta.util import dict_path
10from inmanta_plugins.lsm.allocation import AllocationSpecV2
11from inmanta_plugins.lsm.allocation_v2.framework import (
12    AllocatorV2,
13    ContextV2,
14    ContextV2Wrapper,
15    SetSensitiveForEach,
16)
17
18
19class IntegerAllocator(AllocatorV2):
20    def __init__(self, value: int, attribute: str) -> None:
21        self.value = value
22        self.attribute = dict_path.to_path(attribute)
23
24    def needs_allocation(self, context: ContextV2) -> bool:
25        try:
26            if not context.get_instance().get(self.attribute):
27                # Attribute not present
28                return True
29        except IndexError:
30            return True
31        return False
32
33    def allocate(self, context: ContextV2) -> None:
34        if self.needs_allocation(context):
35            context.set_value(self.attribute, self.value)
36
37
38AllocationSpecV2(
39    "value_allocation",
40    IntegerAllocator(value=1, attribute="first_value"),
41    ContextV2Wrapper(
42        "allocated",
43        SetSensitiveForEach(
44            item="item",
45            in_list="embedded_values",
46            identified_by="id",
47            apply=[
48                IntegerAllocator(
49                    value=3,
50                    attribute="third_value",
51                ),
52            ],
53        ),
54        track_deletes=True,
55    ),
56)