JBoss ON server log continuously reports failed to purge resource types if resource type was a target of a group operation in a compatible or auto resource group

Solution Unverified - Updated 5 Aug 2024

Environment

Red Hat JBoss Operations Network (ON) 3.3
An agent plug-in has recently been deleted
A resource operation had previously been invoked on a group of resources with the same resource type from the agent plug-in which is to be deleted
- This could be a compatible resource group or a resource type auto grouping

Issue

server.log continues to report integrity constraint violated every few minutes:

  WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-5) SQL Error: 2292, SQLState: 23000
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-5) ORA-02292: integrity constraint (RHQADMIN.SYS_C007902) violated - child record found
  ERROR [org.hibernate.engine.jdbc.batch.internal.BatchingBatch] (RHQScheduler_Worker-5) HHH000315: Exception executing batch [could not execute batch]
  ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] (RHQScheduler_Worker-5) Failed to execute job [PurgeResourceTypesJob]: javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types

PurgeResourceTypesJob fails due to database integrity or foreign key constraint violation when attempting to remove a resource type's operation definition

  WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_OPERATION_DEF where ID=10582 was aborted.  Call getNextException to see the cause.
  WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_operation_def" violates foreign key constraint "rhq_operation_history_operation_def_id_fkey" on table "rhq_operation_history"
   (id)=(10582) is still referenced from table "rhq_operation_history".

Resolution

This issue will be addressed in a future release of JBoss ON.

For manually created or dynamic resource groups, you can workaround the issue by deleting the empty groups that once contained resources from the plug-in that is being deleted.

Root Cause

A resource group's operation history is referencing an operation definition defined by the plug-in that is being deleted. When an attempt is made to remove the resource type a database constraint violation occurs because its corresponding operation definition cannot be deleted due to the resource group's operation history continuing to reference the resource type's operation definition.

This issue has been captured in This content is not included.Red Hat Bugzilla 1211407 and will be addressed in a future release.

Diagnostic Steps

Review server.log for error messages similar to:

  ERROR [org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob] (RHQScheduler_Worker-5) Failed to execute job [PurgeResourceTypesJob]: javax.ejb.EJBException: java.lang.RuntimeException: Failed to purge resource types
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:189) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:274) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
      ...
      at org.rhq.enterprise.server.inventory.InventoryManagerLocal$$$view34.purgeDeletedResourceType(Unknown Source) [rhq-server.jar:4.12.0.JON330GA]
      at org.rhq.enterprise.server.scheduler.jobs.PurgeResourceTypesJob.executeJobCode(PurgeResourceTypesJob.java:59) [rhq-server.jar:4.12.0.JON330GA]
      at org.rhq.enterprise.server.scheduler.jobs.AbstractStatefulJob.execute(AbstractStatefulJob.java:48) [rhq-server.jar:4.12.0.JON330GA]
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-1.6.5.jar:1.6.5]
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) [quartz-1.6.5.jar:1.6.5]
  Caused by: java.lang.RuntimeException: Failed to purge resource types
      at org.rhq.enterprise.server.inventory.InventoryManagerBean.purgeDeletedResourceType(InventoryManagerBean.java:122) [rhq-server.jar:4.12.0.JON330GA]
      at sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source) [:1.7.0_75]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_75]
      ...
  Caused by: javax.ejb.EJBTransactionRolledbackException: org.hibernate.exception.ConstraintViolationException: could not execute batch
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleInCallerTx(CMTTxInterceptor.java:162) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
      at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInCallerTx(CMTTxInterceptor.java:252) [jboss-as-ejb3-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
      ...
      at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.2.Final-redhat-1.jar:1.1.2.Final-redhat-1]
      at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73) [jboss-as-ee-7.4.0.Final-redhat-19.jar:7.4.0.Final-redhat-19]
      at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerLocal$$$view82.completeRemoveResourceType(Unknown Source) [rhq-server.jar:4.12.0.JON330GA]
      at org.rhq.enterprise.server.inventory.InventoryManagerBean.purgeDeletedResourceType(InventoryManagerBean.java:120) [rhq-server.jar:4.12.0.JON330GA]
      ... 70 more
  Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute batch
      at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) [hibernate-entitymanager-4.2.14.SP1-redhat-1.jar:4.2.14.SP1-redhat-1]
      at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) [hibernate-entitymanager-4.2.14.SP1-redhat-1.jar:4.2.14.SP1-redhat-1]
      ...
      at org.rhq.enterprise.server.resource.metadata.ResourceMetadataManagerBean.completeRemoveResourceType(ResourceMetadataManagerBean.java:294) [rhq-server.jar:4.12.0.JON330GA]
      at sun.reflect.GeneratedMethodAccessor303.invoke(Unknown Source) [:1.7.0_75]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_75]
      ...
  Caused by: org.hibernate.exception.ConstraintViolationException: could not execute batch
      at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129) [hibernate-core-4.2.14.SP1-redhat-1.jar:4.2.14.SP1-redhat-1]
      at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49) [hibernate-core-4.2.14.SP1-redhat-1.jar:4.2.14.SP1-redhat-1]
      ...
  Caused by: java.sql.BatchUpdateException: ORA-02292: integrity constraint (RHQADMIN.SYS_C007902) violated - child record found
      at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:11190) [ojdbc6-12.1.0.1.0.jar:12.1.0.1.0]
      at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:244) [ojdbc6-12.1.0.1.0.jar:12.1.0.1.0]
      ...

In the above excerpt, the constraint is identified as RHQADMIN.SYS_C007902 and is from an Oracle Database.

For the same error from a PostgreSQL database, the message will be very similar but will provide more details about the constraint:

  WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) Batch entry 0 delete from RHQ_OPERATION_DEF where ID=10582 was aborted.  Call getNextException to see the cause.
  WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) SQL Error: 0, SQLState: 23503
  ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (RHQScheduler_Worker-2) ERROR: update or delete on table "rhq_operation_def" violates foreign key constraint "rhq_operation_history_operation_def_id_fkey" on table "rhq_operation_history"
   (id)=(10582) is still referenced from table "rhq_operation_history".

Are there any operation history entries associated with an operation definition which is defined by a resource type in a deleted plug-in:

  SELECT rhq_operation_history.group_id AS op_his_grp_id
  FROM   rhq_plugin
         LEFT JOIN rhq_resource_type
                ON rhq_resource_type.plugin = rhq_plugin.NAME
         LEFT JOIN rhq_operation_def
                ON rhq_operation_def.resource_type_id = rhq_resource_type.id
         RIGHT JOIN rhq_operation_history
                 ON rhq_operation_history.operation_def_id = rhq_operation_def.id AND rhq_operation_history.group_id IS NOT NULL
  WHERE  rhq_plugin.status = 'DELETED';

If any results are returned, this issue applies. The value returned will represent the group id for the offending group. This can then be used to identify the resource group using the JBoss ON user-interface (UI). You can manually build a URL that includes the group id:

Content from jonserver.example.com is not included.http://jonserver.example.com:7080/coregui/#ResourceGroup/19274
The group should be empty. If an error is returned indicating the group does not exist, then this is most likely an auto-group. This issue still applies but you will not be able to see or modify the offending group.

If the JBoss ON system is using Oracle Database, It may be necessary to identify the table for the defined constraint. This can be done using the constraint name listed in the error message and the following SQL query:
```
  SELECT TABLE_NAME FROM all_constraints WHERE owner = 'RHQADMIN' AND constraint_name = 'SYS_C007902';
```
If the TABLE_NAME value is RHQ_OPERATION_HISTORY, this issue applies.

SBR

Product(s)

Red Hat JBoss Operations Network

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.