Issue 144 remove named objects (#422)

* Issue #144 - Add removal of named objects to the model

* Issue #144 - Add removal of named objects to create; improve logging

* Issue #144 - Use WLST name for delete

* Issue #144 - Add delete capabilities to JNDI properties and group parameters; updated documentation

* Refined logic for get_delete_item_name()

Author: rakillen

README.md

@@ -15,6 +15,7 @@ Many organizations are using WebLogic Server, with or without other Oracle Fusio
     - [Simple Example](#simple-example)
     - [Model Names](#model-names)
     - [Model Semantics](#model-semantics)
+    - [Declaring Named MBeans to Delete](#declaring-named-mbeans-to-delete)
     - [Administration Server Configuration](site/admin_server.md)
     - [Model Security](site/security.md)
       - [Modeling Security Providers](site/security_providers.md)
@@ -52,7 +53,7 @@ As new use cases are discovered, new tools will likely be added to cover those o
 
 ## The Metadata Model
 
-The metadata model (or model, for short) is a version-independent description of a WebLogic Server domain configuration.  The tools are designed to support a sparse model so that the model need only describe what is required for the specific operation without describing other artifacts.  For example, to deploy an application that depends on a JDBC data source into an existing domain that may contain other applications or data sources, the model needs to describe only the application and the data source in question.  If the datasource was previously created, the `deployApps` tool will not try to recreate it but may update part of that data source's configuration if the model description is different than the existing values.  If the application was previously deployed, the `deployApps` tool will compare the binaries to determine if the application needs to be redeployed or not.  In short, the `deployApps` tool supports an iterative deployment model so there is no need to change the model to remove pieces that were created in a previous deployment.
+The metadata model (or model, for short) is a version-independent description of a WebLogic Server domain configuration.  The tools are designed to support a sparse model so that the model need only describe what is required for the specific operation without describing other artifacts.  For example, to deploy an application that depends on a JDBC data source into an existing domain that may contain other applications or data sources, the model needs to describe only the application and the data source in question.  If the data source was previously created, the `deployApps` tool will not try to recreate it but may update part of that data source's configuration if the model description is different than the existing values.  If the application was previously deployed, the `deployApps` tool will compare the binaries to determine if the application needs to be redeployed or not.
 
 The model structure, and its folder and attribute names, are based on the WLST 12.2.1.3 offline structure and names with redundant folders removed to keep the model simple.  For example, the WLST path to the URL for a JDBC data source is `/JDBCSystemResource/<data-source-name>/JdbcResource/<data-source-name>/JDBCDriverParams/NO_NAME_0/URL`.  In the model, it is `resources:/JDBCSystemResource/<data-source-name>/JdbcResource/JDBCDriverParams/URL` (where `resources` is the top-level model section where all WebLogic Server resources/services are described).
 
@@ -220,6 +221,36 @@ In the example above, the `Target` attribute is specified three different ways,
 
 One of the primary goals of the WebLogic Deploy Tooling is to support a sparse model where the user can specify just the configuration needed for a particular situation.  What this implies varies somewhat between the tools but, in general, this implies that the tools are using an additive model.  That is, the tools add to what is already there in the existing domain or domain templates (when creating a new domain) rather than making the domain conform exactly to the specified model.  Where it makes sense, a similar, additive approach is taken when setting the value of multi-valued attributes.  For example, if the model specified the cluster `mycluster` as the target for an artifact, the tooling will add `mycluster` to any existing list of targets for the artifact.  While the development team has tried to mark attributes that do not make sense to merge accordingly in our knowledge base, this behavior can be disabled on an attribute-by-attribute basis, by adding an additional annotation in the knowledge base data files.  The development team is already thinking about how to handle situations that require a non-additive, converge-to-the-model approach, and how that might be supported, but this still remains a wish list item.  Users with these requirements should raise an issue for this support.
 
+### Declaring Named MBeans to Delete
+
+With WebLogic Deploy Tooling release 1.3.0, you can specify named items in the model to be deleted using the Create Domain, Update Domain, and Deploy Applications Tools.  Named items are those that have multiple instances that are distinguished by user-provided names, such as managed servers, data sources, and security realms.  Items to be deleted are prepended with an exclamation point (!) in the model.
+
+In this example, the managed server ```obsoleteServer``` will be deleted, and ```newServer``` will be created:
+
+```yaml
+    Server:
+        !obsoleteServer:
+        newServer:
+            ListenAddress: 127.0.0.1
+            ListenPort: 9005
+```
+
+This feature can also remove items that were created by WebLogic Server templates. For example, the base template creates a default security realm called ```myrealm```.  If a user chooses to declare a custom realm, ```myrealm``` is no longer needed.  In this example, ```myrealm``` will be deleted, and the custom realm ```newrealm``` will be created, and declared as the default realm:
+
+```yaml
+    SecurityConfiguration:
+        DefaultRealm: newrealm
+        Realm:
+            !myrealm:
+            newrealm:
+                AuthenticationProvider:
+                ...
+```
+
+This feature does not apply to named security providers within a realm. These items follow a special set of rules that are required to maintain their ordering. See [Modeling Security Providers](site/security_providers.md) for detailed information.
+
+This feature cannot be use to un-deploy applications or remove libraries.
+
 ### Using Multiple Models
 
 The Create Domain, Update Domain, Deploy Applications, and Validate Model Tools allow the specification of multiple models on the command line. For example:

core/src/main/antlr4/oracle/weblogic/deploy/yaml/Yaml.g4

@@ -275,6 +275,7 @@ fragment ID_START
     : [_]
     | [A-Z]
     | [a-z]
+    | '!'
     ;
 
 fragment ID_CONTINUE

core/src/main/python/wlsdeploy/tool/create/creator.py

@@ -71,6 +71,9 @@ def _create_named_mbeans(self, type_name, model_nodes, base_location, log_create
         existing_folder_names = self._get_existing_folders(list_path)
         for model_name in model_nodes:
             name = self.wlst_helper.get_quoted_name_for_wlst(model_name)
+            if deployer_utils.is_delete_name(name):
+                deployer_utils.delete_named_element(location, name, existing_folder_names, self.alias_helper)
+                continue
 
             if token_name is not None:
                 location.add_name_token(token_name, name)

core/src/main/python/wlsdeploy/tool/deploy/deployer.py

@@ -80,6 +80,10 @@ def _add_named_elements(self, type_name, model_nodes, location):
 
         token = self.alias_helper.get_name_token(location)
         for name in model_nodes:
+            if deployer_utils.is_delete_name(name):
+                deployer_utils.delete_named_element(location, name, existing_names, self.alias_helper)
+                continue
+
             is_add = name not in existing_names
             log_helper.log_updating_named_folder(type_name, name, parent_type, parent_name, is_add, self._class_name,
                                                  _method_name)

core/src/main/python/wlsdeploy/tool/deploy/deployer_utils.py

@@ -177,6 +177,54 @@ def merge_lists(existing_list, model_list, separator=',', return_as_string=True)
     return result
 
 
+def is_delete_name(name):
+    """
+    Determines if the specified name is flagged for deletion with the "!" prefix.
+    :param name: the name to be checked
+    :return: True if the name is prefixed, false otherwise
+    """
+    return name.startswith("!")
+
+
+def get_delete_item_name(name):
+    """
+    Returns the WLST name of the item to be deleted.
+    Removes the "!" prefix from the name. An exception is thrown if the name is not prefixed.
+    :param name: the prefixed model name of the item to be deleted
+    :return: the model name of the item to be deleted
+    """
+    _method_name = 'get_delete_item_name'
+
+    if is_delete_name(name):
+        return name[1:]
+
+    ex = exception_helper.create_deploy_exception('WLSDPLY-09111', name)
+    _logger.throwing(ex, class_name=_class_name, method_name=_method_name)
+    raise ex
+
+
+def delete_named_element(location, delete_name, existing_names, alias_helper):
+    """
+    Delete the specified named element if present. If the name is not present, log a warning and return.
+    :param location: the location of the element to be deleted
+    :param delete_name: the name of the element to be deleted, assumed to include the "!" prefix
+    :param existing_names: a list of names to check against
+    :param alias_helper: alias helper for lookups
+    """
+    _method_name = 'delete_named_element'
+
+    name = get_delete_item_name(delete_name)
+    type_name = alias_helper.get_wlst_mbean_type(location)
+
+    if name not in existing_names:
+        _logger.warning('WLSDPLY-09109', type_name, name, class_name=_class_name, method_name=_method_name)
+    else:
+        _logger.info('WLSDPLY-09110', type_name, name, class_name=_class_name, method_name=_method_name)
+        type_path = alias_helper.get_wlst_create_path(location)
+        _wlst_helper.cd(type_path)
+        _wlst_helper.delete(name, type_name)
+
+
 def ensure_no_uncommitted_changes_or_edit_sessions():
     """
     Ensure that the domain does not contain any uncommitted changes and there is no existing edit session.

core/src/main/python/wlsdeploy/tool/deploy/jms_resources_deployer.py

@@ -138,6 +138,9 @@ def _add_jndi_properties(self, property_name_nodes, location):
         if len(property_name_nodes) == 0:
             return
 
+        # use a copy of the dictionary to remove items as they are deleted
+        remaining_name_nodes = property_name_nodes.copy()
+
         parent_type, parent_name = self.get_location_type_and_name(location)
         is_online = self.wlst_mode == WlstModes.ONLINE
         if is_online and deployer_utils.is_in_resource_group_or_template(location):
@@ -151,10 +154,16 @@ def _add_jndi_properties(self, property_name_nodes, location):
         name_attribute = self.alias_helper.get_wlst_attribute_name(properties_location, KEY)
         mbean_type = self.alias_helper.get_wlst_mbean_type(properties_location)
 
+        for property_name in property_name_nodes:
+            if deployer_utils.is_delete_name(property_name):
+                name = deployer_utils.get_delete_item_name(property_name)
+                self._delete_mapped_mbean(properties_location, properties_token, mbean_type, name_attribute, name)
+                del remaining_name_nodes[property_name]
+
         # loop once to create and name any missing folders.
         folder_map = self._build_folder_map(properties_location, properties_token, name_attribute)
 
-        for property_name in property_name_nodes:
+        for property_name in remaining_name_nodes:
             folder_name = dictionary_utils.get_element(folder_map, property_name)
             if folder_name is None:
                 self.wlst_helper.cd(foreign_server_path)
@@ -164,7 +173,7 @@ def _add_jndi_properties(self, property_name_nodes, location):
         # loop a second time to set attributes
         new_folder_map = self._build_folder_map(properties_location, properties_token, name_attribute)
 
-        for property_name in property_name_nodes:
+        for property_name in remaining_name_nodes:
             is_add = property_name not in folder_map
             log_helper.log_updating_named_folder(JNDI_PROPERTY, property_name, parent_type, parent_name, is_add,
                                                  self._class_name, _method_name)
@@ -173,7 +182,7 @@ def _add_jndi_properties(self, property_name_nodes, location):
             properties_location.add_name_token(properties_token, folder_name)
             self.wlst_helper.cd(self.alias_helper.get_wlst_attributes_path(properties_location))
 
-            property_nodes = property_name_nodes[property_name]
+            property_nodes = remaining_name_nodes[property_name]
             self.set_attributes(properties_location, property_nodes)
 
     def _add_group_params(self, group_name_nodes, location):
@@ -188,6 +197,9 @@ def _add_group_params(self, group_name_nodes, location):
         if len(group_name_nodes) == 0:
             return
 
+        # use a copy of the dictionary to remove items as they are deleted
+        remaining_name_nodes = group_name_nodes.copy()
+
         parent_type, parent_name = self.get_location_type_and_name(location)
         template_path = self.alias_helper.get_wlst_subfolders_path(location)
         groups_location = LocationContext(location)
@@ -196,11 +208,20 @@ def _add_group_params(self, group_name_nodes, location):
         name_attribute = self.alias_helper.get_wlst_attribute_name(groups_location, SUB_DEPLOYMENT_NAME)
         mbean_type = self.alias_helper.get_wlst_mbean_type(groups_location)
 
+        for group_name in group_name_nodes:
+            if deployer_utils.is_delete_name(group_name):
+                group_nodes = group_name_nodes[group_name]
+                name = deployer_utils.get_delete_item_name(group_name)
+                sub_name = self._get_subdeployment_name(group_nodes, name)
+                self._delete_mapped_mbean(groups_location, groups_token, mbean_type, name_attribute, sub_name)
+                del remaining_name_nodes[group_name]
+
         # loop once to create and name any missing folders.
         folder_map = self._build_folder_map(groups_location, groups_token, name_attribute)
 
-        for group_name in group_name_nodes:
-            sub_deployment_name = self._get_subdeployment_name(group_name_nodes, group_name)
+        for group_name in remaining_name_nodes:
+            group_nodes = remaining_name_nodes[group_name]
+            sub_deployment_name = self._get_subdeployment_name(group_nodes, group_name)
             folder_name = dictionary_utils.get_element(folder_map, sub_deployment_name)
             if folder_name is None:
                 self.wlst_helper.cd(template_path)
@@ -210,19 +231,52 @@ def _add_group_params(self, group_name_nodes, location):
         # loop a second time to set attributes
         new_folder_map = self._build_folder_map(groups_location, groups_token, name_attribute)
 
-        for group_name in group_name_nodes:
-            sub_deployment_name = self._get_subdeployment_name(group_name_nodes, group_name)
+        for group_name in remaining_name_nodes:
+            group_nodes = remaining_name_nodes[group_name]
+            sub_deployment_name = self._get_subdeployment_name(group_nodes, group_name)
             is_add = sub_deployment_name not in folder_map
             log_helper.log_updating_named_folder(GROUP_PARAMS, group_name, parent_type, parent_name, is_add,
                                                  self._class_name, _method_name)
 
             folder_name = dictionary_utils.get_element(new_folder_map, sub_deployment_name)
             groups_location.add_name_token(groups_token, folder_name)
             self.wlst_helper.cd(self.alias_helper.get_wlst_attributes_path(groups_location))
-
-            group_nodes = group_name_nodes[group_name]
             self.set_attributes(groups_location, group_nodes)
 
+    def _delete_mapped_mbean(self, folder_location, folder_token, mbean_type, name_attribute, name):
+        """
+        Delete the child MBean with an attribute value that matches the specified name.
+        This requires looping through the child MBeans for each deletion, since the MBean names change on delete.
+        :param folder_location: the WLST location for the folder with named sub-elements
+        :param folder_token: the folder token used to iterate through the MBean names
+        :param mbean_type: the MBean type of the sub-elements
+        :param name_attribute: the attribute for the name in each sub-element
+        :param name: the name of the sub-element to be deleted
+        """
+        _method_name = '_delete_mapped_mbean'
+
+        original_path = self.wlst_helper.get_pwd()
+        mapped_folder_name = None
+
+        folder_names = deployer_utils.get_existing_object_list(folder_location, self.alias_helper)
+        for folder_name in folder_names:
+            folder_location.add_name_token(folder_token, folder_name)
+            self.wlst_helper.cd(self.alias_helper.get_wlst_attributes_path(folder_location))
+            attribute_value = self.wlst_helper.get(name_attribute)
+            if attribute_value == name:
+                mapped_folder_name = folder_name
+                break
+
+        self.wlst_helper.cd(original_path)
+
+        if mapped_folder_name is None:
+            self.logger.warning('WLSDPLY-09109', mbean_type, name,
+                                class_name=self._class_name, method_name=_method_name)
+        else:
+            self.logger.info('WLSDPLY-09110', mbean_type, name,
+                             class_name=self._class_name, method_name=_method_name)
+            self.wlst_helper.delete(mapped_folder_name, mbean_type)
+
     def _build_folder_map(self, folder_location, folder_token, name_attribute):
         """
         Build a map of existing sub-element names to folders.
@@ -245,15 +299,14 @@ def _build_folder_map(self, folder_location, folder_token, name_attribute):
         self.wlst_helper.cd(original_path)
         return folder_map
 
-    def _get_subdeployment_name(self, group_name_nodes, group_name):
+    def _get_subdeployment_name(self, group_nodes, group_name):
         """
         Returns the derived sub-deployment name for the specified group name.
         The sub-deployment name attribute is returned if specified, otherwise the group name is returned.
-        :param group_name_nodes: the group name nodes from the model
+        :param group_nodes: the group nodes from the model
         :param group_name: the group name being examined
         :return: the derived sub-deployment name
         """
-        group_nodes = group_name_nodes[group_name]
         sub_deployment_name = dictionary_utils.get_element(group_nodes, SUB_DEPLOYMENT_NAME)
         if sub_deployment_name is not None:
             return sub_deployment_name

core/src/main/python/wlsdeploy/tool/util/topology_helper.py

@@ -113,6 +113,10 @@ def create_placeholder_named_elements(self, location, model_type, model_nodes):
 
             name_nodes = dictionary_utils.get_dictionary_element(model_nodes, model_type)
             for name in name_nodes:
+                if deployer_utils.is_delete_name(name):
+                    # don't create placeholder for delete names
+                    continue
+
                 if name not in existing_names:
                     self.logger.info('WLSDPLY-19403', model_type, name, class_name=self.__class_name,
                                      method_name=_method_name)

core/src/main/resources/oracle/weblogic/deploy/messages/wlsdeploy_rb.properties

@@ -936,6 +936,9 @@ WLSDPLY-09106=Shared library name {0} contained {1} @ signs when only zero or on
 WLSDPLY-09107=Shared library name {0} contained {1} # signs when only zero or one are allowed
 WLSDPLY-09108=Model attribute {0} at model location {1} with value {2} references a location inside \
   the archive file but the archive file was not provided
+WLSDPLY-09109=Unable to delete {0} {1}, name does not exist
+WLSDPLY-09110=Deleting {0} {1}
+WLSDPLY-09111=Unable to get delete item name for {0}
 
 # wlsdeploy/tool/deploy/deployer.py
 WLSDPLY-09200=Error setting attribute {0} for {1} {2}: {3}