Server fails to process agent's inventory report due to 'UnsupportedOperationException' when resource type is missing on the server

Solution Unverified - Updated 5 Aug 2024

Environment

Red Hat JBoss Operations Network (ON) 3.3
JBoss ON agent is sending its inventory report to the server

Issue

Server and other resources are not discovered

Run Autodiscovery platform operation see the following WARN message in the agent logs when sending [runtime] inventory report to server:

  WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Exception caught while executing runtime discovery scan rooted at [platform].
  [Warning] javax.ejb.EJBException: java.lang.UnsupportedOperationException

Server mergeInventoryReport fails during merging of inventory report from agent:

  ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:7080-4) JBAS014134: EJB Invocation failed on component DiscoveryBossBean for method public abstract org.rhq.core.domain.discovery.MergeInventoryReportResults org.rhq.enterprise.server.discovery.DiscoveryBossLocal.mergeInventoryReport(org.rhq.core.clientapi.server.discovery.InventoryReport) throws org.rhq.core.clientapi.server.discovery.InvalidInventoryReportException: javax.ejb.EJBException: java.lang.UnsupportedOperationException
  ...
  ERROR [org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl] (http-/0.0.0.0:7080-4) Fatal error occurred during merging of inventory report from agent [Agent[id=0,name=jboss.example.com,address=null,port=0,remote-endpoint=null,last-availability-ping=null,last-availability-report=null]].: javax.ejb.EJBException: java.lang.UnsupportedOperationException

Resolution

You will need to restore the missing resource type by re-installing the agent plug-in that defines the type.

WARNING: Do not delete the plug-in. Doing so will result in the loss of all resources and history related to the plug-in and its types.

Locate the offending agent plug-in and make a copy. Agent plug-ins are installed on the server in the directory <RHQ_SERVER_HOME>/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-plugins. For example:
```
 cp /opt/jboss/on/jon-server/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-plugins/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar "$HOME"
```
This would copy the JBoss EAP 6 agent plug-in into the user's home directory.
Now, create a second copy of the agent plug-in file using a different file name. For example:
```
 cp "$HOME"/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar "$HOME"/rhq-jboss-as-7-plugin-MODIFIED.jar
```
This would create a copy of the JBoss EAP 6 agent plug-in located in the user's home directory to the same location but with its version replace with MODIFIED. It does not matter what you name it as long as it ends with .jar.
Create an empty file in a location that can be accessed from where you have stored the second copy of the agent plug-in JAR. For example:
```
 touch emptyfile
```
You can use any file really, its contents and name is irrelevant.
Add the empty file to the second copy of the agent plug-in JAR file using a ZIP utility. For example:
```
 zip "$HOME"/rhq-jboss-as-7-plugin-MODIFIED.jar emptyfile
```
You could also use the jar utility:
```
 jar uf "$HOME"/rhq-jboss-as-7-plugin-MODIFIED.jar emptyfile
```
From the JBoss ON user-interface (UI) install the modified agent plug-in from the agent plug-ins configuration administration page. Based on the previous example, that would be the JAR file named rhq-jboss-as-7-plugin-MODIFIED.jar.
Wait until the plug-in is successfully updated.
Update the file modification date and time for the first copy of the agent plug-in. For example:
```
 touch "$HOME"/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar
```
This can also be done by creating a new file from the contents of the existing file. For example:
```
 cat "$HOME"/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar > "$HOME"/NEW-rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar
```
The file modification date and time must be newer then that of the rhq-jboss-as-7-plugin-MODIFIED.jar file installed in the previous step.

Please note that this file is the original plug-in that was copied from the JBoss ON server's plug-ins directory. The only difference is that it now has a newer modification time then the modified version we created earlier.
Using the agent plug-ins configuration administration page, install the plug-in JAR that has the updated file modification time from the previous step. From the example, that is rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar or NEW-rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar.
Wait until the plug-in is successfully updated.

Root Cause

An agent has sent an inventory report which includes a resource that is using a resource type that is not available in the JBoss ON server. When the resource using the unknown resource type is processed from the inventory report an attempt is made to remove the resource from the report as it is considered invalid by the server. The method used to remove the resource fails due to the Collection's Iterator not supporting the remove method. This results in a java.lang.UnsupportedOperationException being thrown and returned to the agent.

Under normal operating conditions, an agent should not have resource types that the server does not know about. However, this can happen in one of the following circumstances:

An agent plug-in installed on the JBoss ON server has recently been deleted but has not been fully purged from the system.

In this situation, one or more resource types defined by the plug-in have been marked for deletion but have not been purged because their associated resources and data are still being removed from the JBoss ON system. Because the plug-in has not yet been purged, the agent continues to use the plug-in and discover resources using the resource type that has been marked for deletion.
An agent plug-in installed on the JBoss ON server has recently been installed or updated but a failure occurred during the installation of its resource types.

In this situation, the server has failed to install all the available resource types provided by the newly installed or updated plug-in. However, when the agent downloads the updated plug-in file from the server, it is able to load all the defined types and therefore discover resources using types the server does not know about.

This issue has been captured in This content is not included.Red Hat Bugzilla 1187645 and will be addressed in an upcoming release of JBoss ON.

Diagnostic Steps

Review the agent log for a warning caused by UnsupportedOperationException at the time it is sending its inventory report to the server:

  INFO  [InventoryManager.discovery-1] (rhq.core.pc.inventory.InventoryManager)- Sending [runtime] inventory report to Server...
  WARN  [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Exception caught while executing runtime discovery scan rooted at [platform].
  [Warning] javax.ejb.EJBException: java.lang.UnsupportedOperationException
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:215)
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:261)
      ...
      at org.rhq.enterprise.server.discovery.DiscoveryBossLocal$$$view100.mergeInventoryReport(Unknown Source)
      at org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl.mergeInventoryReport(DiscoveryServerServiceImpl.java:85)
      ...
  Caused by: java.lang.UnsupportedOperationException
      at java.util.concurrent.CopyOnWriteArrayList$COWIterator.remove(CopyOnWriteArrayList.java:1004)
      at org.rhq.enterprise.server.discovery.DiscoveryBossBean.initResourceTypes(DiscoveryBossBean.java:1335)
      at org.rhq.enterprise.server.discovery.DiscoveryBossBean.mergeInventoryReport(DiscoveryBossBean.java:223)
      ...

Review the server log for an error caused by UnsupportedOperationException at the time it was processing and merging the impacted agent's inventory report:

  ERROR [org.jboss.as.ejb3.invocation] (http-/0.0.0.0:7080-7) JBAS014134: EJB Invocation failed on component DiscoveryBossBean for method public abstract org.rhq.core.domain.discovery.MergeInventoryReportResults org.rhq.enterprise.server.discovery.DiscoveryBossLocal.mergeInventoryReport(org.rhq.core.clientapi.server.discovery.InventoryReport) throws org.rhq.core.clientapi.server.discovery.InvalidInventoryReportException: javax.ejb.EJBException: java.lang.UnsupportedOperationException
  ...
  ERROR [org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl] (http-/0.0.0.0:7080-7) Fatal error occurred during merging of inventory report from agent [Agent[id=0,name=jboss.example.com,address=null,port=0,remote-endpoint=null,last-availability-ping=null,last-availability-report=null]].: javax.ejb.EJBException: java.lang.UnsupportedOperationException
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInNoTx(CMTTxInterceptor.java:215) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:261) [jboss-as-ejb3-7.4.0.Final-redhat-4.jar:7.4.0.Final-redhat-4]
      ...
      at org.rhq.enterprise.server.discovery.DiscoveryBossLocal$$$view100.mergeInventoryReport(Unknown Source) [rhq-server.jar:4.13.0]
      at org.rhq.enterprise.server.discovery.DiscoveryServerServiceImpl.mergeInventoryReport(DiscoveryServerServiceImpl.java:85) [rhq-server.jar:4.13.0]
      ...
  Caused by: java.lang.UnsupportedOperationException
      at java.util.concurrent.CopyOnWriteArrayList$COWIterator.remove(CopyOnWriteArrayList.java:1040) [rt.jar:1.7.0_55]
      at org.rhq.enterprise.server.discovery.DiscoveryBossBean.initResourceTypes(DiscoveryBossBean.java:1335) [rhq-server.jar:4.13.0]
      at org.rhq.enterprise.server.discovery.DiscoveryBossBean.mergeInventoryReport(DiscoveryBossBean.java:223) [rhq-server.jar:4.13.0]
      ...

Have any agent plug-ins recently been installed or updated? For example, upgrading from one version to another?

Review the JBoss ON server log covering the time period in where the updated plug-ins were installed. Pay special attention to messages that indicate that plug-in or resource type processing failed. For example:

  ERROR [org.rhq.enterprise.server.core.plugin.ProductPluginDeployer] (EJB default - 5) Failed to register RHQ plugin file [file:/opt/jboss/on/jon-server/modules/org/rhq/server-startup/main/deployments/rhq.ear/rhq-downloads/rhq-plugins/rhq-jboss-as-7-plugin-4.12.0.JON330GA.jar]: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
  ...
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [Windows] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [Linux] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [Solaris] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [HP-UX] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [AIX] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [FreeBSD] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.
  WARN  [org.rhq.core.clientapi.agent.metadata.PluginMetadataParser] (pool-6-thread-1) There is no resource type named [Mac OS X] from a plugin named [Platforms]. This is probably because that plugin is missing. Resource Type [{JBossAS7}Wildfly/JBoss EAP Patch Handler] will not have that missing type as a possible parent.

Review the resource types and resources in the agent's detail inventory report:
```
  inventory --xml --export agent-inventory.xml
```
Identify all resource types included in the agent's inventory report.
Do any of the resource types included in the agent's inventory report have a deleted status reported in the JBoss ON server? A database query can be executed against the JBoss ON database to display all resource types in a deleted state.
- For Oracle Database:
```
  SELECT * FROM rhq_resource_type WHERE deleted = 1;
```
- For PostgreSQL:
```
  SELECT * FROM rhq_resource_type WHERE deleted = true;
```
Are any of the resource types included in the agent's inventory report missing from JBoss ON server? A database query can be executed against the JBoss ON database to display all resource types:
```
      SELECT * FROM rhq_resource_type;
```

Use the following Content from byteman.jboss.org is not included.JBoss Byteman rule to capture the resources and their children that are being passed to the initResourceTypes method. This will allow you to see what resource and child resources, if any, were passed to the initResourceTypes method right before java.lang.UnsupportedOperationException is thrown. From this data you can easily identify the offending resource type.

      ########################################################################
      # To the extent possible under law, Red Hat, Inc. has dedicated all 
      # copyright to this software to the public domain worldwide, pursuant 
      # to the CC0 Public Domain Dedication. This software is distributed 
      # without any warranty.  
      # 
      # See <http://creativecommons.org/publicdomain/zero/1.0/>.
      #
      #
      # JBM-BZ1187645 Rule File
      # 
      # This JBoss Byteman rule set may be helpful in identifying the resource
      # that is causing the UnsupportedOperationException during the 
      # mergeInventoryReport process due to the underlying initResourceTypes 
      # method execution failing as identified in Red Hat Bugzilla 1187645 <https://bugzilla.redhat.com/show_bug.cgi?id=1187645>.
      #
      # For details, see Server fails to process agent's inventory report due 
      # to 'UnsupportedOperationException' <https://access.redhat.com/solutions/1355133>
      #

      RULE Log resource information on invocation of initResourceTypes
      CLASS ^org.rhq.enterprise.server.discovery.DiscoveryBossBean
      METHOD boolean initResourceTypes(org.rhq.core.domain.resource.Resource, java.util.Map)
      AT ENTRY
      BIND resourceId:int             = $1.getId();
           resourceName:String        = $1.getName();
           resourceAncestry:String    = $1.getAncestry();
           pluginName:String          = $1.getResourceType().getPlugin();
           resourceTypeName:String    = $1.getResourceType().getName();
      IF TRUE
      DO System.out.println("JBM-BZ1187645: initResourceTypes executed with resource " + resourceName + "[id=" + resourceId + ",ancestry=" + resourceAncestry + ",plugin=" + pluginName + ",type=" + resourceTypeName + "] with child resources of " + $1.getChildResources());
      ENDRULE

SBR

Product(s)

Red Hat JBoss Operations Network

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.