OUD 11.1.2.2 failed to start with Checksum invalid error.

OUD uses embedded Berkley DB to store the data. All these data is saved in file system.
There are many chances in which this DB can corrupt. To avoid this data corruption you can plan the periodic data backup using OUD backup utility or ldif backup.

If in case your OUD instance got corrupted becasue of any reason then OUD startup script will fail to start the OUD instance with the sample error shown below.

category=CORE severity=NOTICE msgID=458891 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.Dire
ctoryServerShutdown, alert ID 458893): The Directory Server has started the shutdown process. The shutdown was initiated by an instance of class org.opends.server.core.DirectoryServer and the reason provided for the sh
utdown was An error occurred while trying to start the Directory Server: An error occurred while trying to initialize a workflow element from class org.opends.server.workflowelement.localbackend.DBLocalBackendWorkflowEl
ement with the information in configuration entry cn=userRoot,cn=Workflow Elements,cn=config: org.opends.server.types.InitializationException: An error occurred while trying to initialize a backend loaded from class org
.opends.server.backends.jeb.BackendImpl with the information in configuration entry cn=userRoot,cn=Workflow Elements,cn=config: The database environment could not be opened: (JE 5.0.86) /u01/app/oracle/admin/OUD_Instanc
es/OUD1/OUD/db/userRoot com.sleepycat.je.log.ChecksumException: Incomplete log entry header, size=0 lsn=0xb4/0x1c290aa LOG_CHECKSUM: Checksum invalid on read, log is likely invalid. Environment is invalid and must be clo
sed. fetchTarget of 0xb4/0x1c290aa parent IN=3 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0xb5/0x42ea62 lastLoggedVersion=0xb5/0xb69382 parent.getDirty()=true state=0 (BackendImpl.java:1959 BackendImpl.java:366 L
ocalBackendWorkflowElement.java:500 LocalBackendWorkflowElement.java:396 DBLocalBackendWorkflowElement.java:61 NativeMethodAccessorImpl.java:-2 NativeMethodAccessorImpl.java:57 DelegatingMethodAccessorImpl.java:43 Method
.java:606 WorkflowElementConfigManager.java:420 WorkflowElementConfigManager.java:362 WorkflowElementConfigManager.java:144 DirectoryServer.java:10297 WorkflowElementConfigManager.java:117 DirectoryServer.java:2629 Direc
toryServer.java:1600 DirectoryServer.java:9930). This backend will be disabled. This workflow element will be disabled.

To resolve this issue you need the working backup of the OUD instance which you can use for restore. But if you dont have the backup then you can take the below approach to resolve the startup issue. In this approach you will lose some of your data.

1. Navigate to OUD instance folder and take the full backup of db and changelogDb folders.
2. Navigate to the original db/userRoot folder.
3. Delete the latest .jdb file.
4. Now navigate to changelogDb folder and delete the latest .jdb file.
5. Now run the start-ds command and this time your OUD instance should start with no error.

Stuck Threads in OIM (11.1.2.2) server

In OIM-OAM-OUD (or any directory) integrated environment if you see stuck threads in OIM servers and in threads dump you see the below error, Then apply the oracle patch 19939451 on oracle_common folder of OIM middleware.

[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f2de401e000 nid=0x7de4 waiting on condition [0x00007f2de0986000]
   java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  <0x00000005ae41a948> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
 at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
 at oracle.ods.virtualization.engine.backend.jndi.JNDIConnectionPool.getLdapContext(JNDIConnectionPool.java:272)
 at oracle.ods.virtualization.engine.backend.jndi.JNDIConnectionPool.checkOutContext(JNDIConnectionPool.java:226)
 at oracle.ods.virtualization.engine.backend.jndi.BackendJNDI.getLDAPContext(BackendJNDI.java:1068)
 at oracle.ods.virtualization.engine.backend.jndi.BackendJNDI.getConnection(BackendJNDI.java:969)
 at oracle.ods.virtualization.engine.backend.jndi.ConnectionHandle.getHolder(ConnectionHandle.java:445)
 at oracle.ods.virtualization.engine.backend.jndi.ConnectionHandle.search(ConnectionHandle.java:268)
 at oracle.ods.virtualization.engine.backend.jndi.JNDIEntrySet.initialize(JNDIEntrySet.java:221)
 at oracle.ods.virtualization.engine.backend.jndi.BackendJNDI.get(BackendJNDI.java:765)
 at oracle.ods.virtualization.engine.chain.Chain.nextGet(Chain.java:303)