Data Type "Dictionary" to allow for nested variables

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/12093

Created by lebvanih on 2016-07-04 11:05:38 +00:00

Assignee: (none)
Status: New
Target Version: (none)
Last Update: 2016-08-10 19:27:13 +00:00 (in Redmine)

---

Hello,

We are currently trying to import our running configuration in Icinga Director (1.1.0). Our configuration rely on dictionaries and "nested variables" in our hosts. These are used in our "apply for" rules.
The issue is, that as from what we saw (I checked on this tracker and in the example), there is at the moment no options to reflect this kind of configuration in Director. The example provided in the documentation of Director only refer to Array.

Our configuration is mostly based on the following example from the official documentation (see [[http://docs.icinga.org/icinga2/latest/doc/module/icinga2/toc\#!/icinga2/latest/doc/module/icinga2/chapter/monitoring-basics#using-apply-for-custom-attribute-override\]\]):

vars.interfaces["GigabitEthernet0/2"] = {
     iftraffic_units = "g"
     iftraffic_community = IftrafficSnmpCommunity
     iftraffic_bandwidth = 1
     vlan = "internal"
     qos = "disabled"
  }

We also use this kind of structure:

vars.an_ncpa = {
    token = "secret"
    load = { warning=10, critical=20 }
    cpuload = { warning=80, critical=85 }
    disk["/"] = { disk_warning="15%", disk_critical="10%" }
    disk["/home"] = { disk_warning="15%", disk_critical="10%" }
  }

The structure above was used for three reasons in our configuration:

1. Easy to ready (for us at least)
2. No need to prefix all variable
3. Valid with Apply For.

I tried to workaround with a data field named like this: "an_ncpa.token", but it get renamed to "an_npcatoken" as soon as the configuration is rendered.
Is there any plan to allow dictionary creation in Icinga Director and same question for the "nested variable" (dictionary inside a dictionary for example) ?

If this kind of data structure is not "valid" could you please guide us on what we should use instead?

Attachments

• filter_form_playground.png tgelf - 2016-08-09 14:57:19 +00:00

---

Relations:

• blocks #11849

[icinga-migration]

Updated by itbess on 2016-08-03 05:08:59 +00:00

+1

[icinga-migration]

Updated by msmol on 2016-08-03 19:50:09 +00:00

+1
My team (jprivard & cardeois) and I also need this functionality and are ready to develop it. We've discussed several possible solutions and have decided the best approach would be to create a new object similar to the DataList, called DataObject. The difference being DataList contains only "key", and "value", whereas DataObject will contain "key", "value", and "type".

In the aforementioned example :

vars.an_ncpa = {
    token = "secret" 
    load = { warning=10, critical=20 }
    cpuload = { warning=80, critical=85 }
    disk["/"] = { disk_warning="15%", disk_critical="10%" }
    disk["/home"] = { disk_warning="15%", disk_critical="10%" }
  }

The "an_ncpa" form definition could be represented by a DataObject such as:

[{
    "key": "token", 
    "value": "Token",
    "type": String
},
{
    "key": "load", 
    "value": "Load",
    "type": DataObject['notification_threshold']
},
...]

Where "notification_threshold" is another DataObject that can be used generically for both 'load' and 'cpuload' etc.

That being said, the "disk" structure above would be difficult to implement using this approach. A possible workaround would be to use a structure like this instead:

vars.an_ncpa = {
    ...
    disk =  [
        {disk_warning="15%", disk_critical="10%", path="/"}
    ]
    ...
}

This would be generated by a DataObject like so:

{
    "key": "disk", 
    "value": "Disks",
    "type": Array(DataObject['disk'])
}

Would this be an acceptable solution for you? Open to suggestions / alternatives / reasons why this will fail miserably.
Seems like a lot of work but we are willing to get started if there's interest & no objections to our proposed solution.

Cheers,
Mitch

[icinga-migration]

Updated by tgelf on 2016-08-03 22:05:23 +00:00

Hi Mitch,

thanks a lot for your input, this absolutely makes sense to me. What about calling it "Dictionary" instead of "Data Object" to follow Icinga 2 naming conventions? Should the value really be part of the type? I'd consider leaving it away. Right now, for all data fields default values are specified when assigning a field to a real object. Then, a DataType currently consists of a class name stored in an entry in the datafield table. So why not basing the dictionary on this, creating a list of datafield_ids and related arity?

Let me show you the DB tables that could reflect this:

CREATE TABLE director_dictionary (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  dictionary_name varchar(255) NOT NULL,
  PRIMARY KEY (id),
  UNIQUE KEY dictionary_name (dictionary_name)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE director_dictionary_field (
  dictionary_id int(10) unsigned NOT NULL,
  datafield_id INT(10) USIGNED NOT NULL,
  is_required ENUM('y','n') NOT NULL,
  allow_multiple ENUM('y','n') NOT NULL,
  PRIMARY KEY (dictionary_id,datafield_id),
  CONSTRAINT director_dictionary_field_dictionary
    FOREIGN KEY (dictionary_id)
    REFERENCES director_dictionary (id)
    ON DELETE CASCADE
    ON UPDATE CASCADE,
  CONSTRAINT director_dictionary_field_datafield
    FOREIGN KEY (datafield_id)
    REFERENCES director_datafield (id)
    ON DELETE RESTRICT
    ON UPDATE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

We could combine is_required and allow_multiple into a single "arity" property or use min/max-count, but as is_required is already used in the *_field table it seemed to fit better. Please note that this schema would also allow one to create two different types for the same key, something that could also be desired. Your initial disk example should perfectly be possible with this, your workaround (Array of disks) isn't. But that's what we should then add as a new DataType "Array of SomeType" while renaming the current Array to "Array of Strings".

Cheers,
Thomas

[icinga-migration]

Updated by jprivard on 2016-08-04 14:54:02 +00:00

cardeois, msmol and I just reviewed your solution and we all agree to it. We'll start working on it right now.

[icinga-migration]

Updated by tgelf on 2016-08-04 15:02:40 +00:00

jprivard wrote:

cardeois, msmol and I just reviewed your solution and we all agree to it. We'll start working on it right now.

Fantastic! Can't wait to see the result ;-) This is gonna be a pretty cool feature I guess.

[icinga-migration]

Updated by msmol on 2016-08-08 19:57:47 +00:00

Hi Thomas,

We started working on this feature and have a a few things done, and a few things left to do.

So far, we've added 2 new tabs in datalist section, for creating Dictionaries and DictionaryFields.

What we're missing is the ability to set a custom attribute with type Dictionary, as we're not really sure how to show this kind of nested data in a reasonably nice way. cardeois tells me that he thinks you're already working on something like this. Can you confirm whether or not you are, and if so if you've made much progress?

You can see our internal PR inside our fork here: internap#1

Cheers,
Mitch

[icinga-migration]

Updated by tgelf on 2016-08-09 15:20:08 +00:00

• File added filter_form_playground.png

Hi Mitch!

msmol wrote:

So far, we've added 2 new tabs in datalist section, for creating Dictionaries and DictionaryFields.

Perfect. I had a quick look at it, looks good to me. What was the reason for the "Rename dictionary_field to dictionaryfield" commit? All the other implemented fields use "_field" as a postfix, did you run in any problems with this?

What we're missing is the ability to set a custom attribute with type Dictionary, as we're not really sure how to show this kind of nested data in a reasonably nice way. cardeois tells me that he thinks you're already working on something like this. Can you confirm whether or not you are, and if so if you've made much progress?

One missing part is the data type itself, please have a look into library/Director/DataType for examples and register your new type in configuration.php. Registration might seem useless, but it is pretty cool as it allows other Icinga Web 2 modules to provide their very own data types in an upgrade-safe way.

The DataType must provide a form field, and this is where it starts to become tricky. Such a form field is basically a ZF1 form element. You'll need to create one for Dictionaries in library/Director/Web/Form/Element. That element can specify a custom view helper, you'll also need one, should be placed into application/views/helpers/.

The web form is the trickiest part of your feature request and the main reason why I avoided to start working on it so far ;-) You could have a look at the ExtensibleSet for an example, but that one isn't isolated very good.

What I'm currently working on (I think that's what cardeois was referring to) is a form element for "Filters", allowing one to create deep nested filter definitions while allowing the developer to handle it like a single-element form field. That component got larger than expected, as I want to use it for nested apply rules and added support for data types. Please have a look at the attached screenshot:

Look & feel might be improved. However, it is pretty cool. Every icon is a form button, you can nest it as far as your screen allows. And still, in your code you just write

$this->addElement('dataFilter', 'element_name', ...);

...and that's is. $element->getValue() gives you a full Filter object that can be serialized and stored to DB. I wanted to have this feature in v1.1.1, even if I haven't been forced to do so. By the end, it became the main reason why I didn't release v1.1.1 yet :p I'm currently on vacation, but I'll continue to work on this these days. Still a little bit unhappy with how I have to deal with data types, but I'll find a solution.

As you can see, the problems I have to tackle for this feature are very similar to yours. So it could absolutely make sense for you to wait for this, but you could also start with this in parallel. Regardless of this I'd suggest to go on with the provided DataType itself. Then make your first steps with a rough custom FormElement. Do not spend to much time there, eventually cheat by using a simple text field asking for JSON in the meantime. Try to figure out whether you can use CustomVariableDictionary for validation/serialization, or what other components might be missing.

I hope this helps, please do not hesitate letting me know when I just confused you instead ;-)

Cheers,
Thomas

[icinga-migration]

Updated by msmol on 2016-08-10 19:27:13 +00:00

What was the reason for the "Rename dictionary_field to dictionaryfield" commit? All the other implemented fields use "_field" as a postfix, did you run in any problems with this?

Yes, I don't remember exactly what the issue was, but between whatever the issue was (been a few days now), and seeing how "DirectorDatafield" uses "Datafield" and not "DataField", it seemed like a simple solution that solved the issue at the time.

One missing part is the data type itself, please have a look into library/Director/DataType for examples and register your new type in configuration.php. Registration might seem useless, but it is pretty cool as it allows other Icinga Web 2 modules to provide their very own data types in an upgrade-safe way.

Yes, already started work on that, but isn't committed yet. Basically for the reasons you listed, i.e. wasn't sure what to do for the web form, and seeing as cardeois mentioned you were maybe already working on something for this, wrote the comment above :-)

I think I'll take your advice and cheat for now with a simple text field.

Thanks for the help, and enjoy your vacation

Mitch

[icinga-migration]

Updated by tgelf on 2016-10-25 12:02:39 +00:00

• Blocks set to 11849

[cbnorman]

when is this feature likely to be made available?

[cschneemann]

Still working on this? It would be great to have dictionary-support in director

[tobiias123]

I'am currently writing my bachelor thesis about the Icinga Director. One point of this is to analyze how to migrate an existing configuration to the Director. There I ran into the issue with the dictionary-support. So are there any news about this? Is there a workaround or something?

[likg]

Propose to add an ability to insert user-defined functions also. Icinga2 core supports complex data types since 2015 (new-features-in-icinga-2-3) and the following vars definition is perfectly valid in Icinga2 world:

vars.http_vhosts["icinga.org"] = {
    http_address = "icinga.org"
    interval = 1m
  }

vars.dummy_text = {{ return "Dummy text here" }}
vars.dummy_state = {{ return 0 }}

Director object inspection already represents functions as:

vars: {
        dummy_state: { type: "Function" },
        dummy_text: { type: "Function" }
      }

Therefore comment sound reasonable to me.