The object identified by: 31 could not be found

The Problem Abstract 

I encountered a strange exception while restarting my admin WebLogic server. Everything was working fine a day ago. But a day later its was giving me strange issue of "object 31" error.
After struggling for a while I realized that some startup option was creating the issue. The startup options was containing JVM arguments "-Dweblogic.management.server=t3://10.188.103.55:7002". The admin server should not contain this, as it makes the server to run in managed server mode. So the issue was, admin server was starting as managed server and trying to contact itself, creating a deadlock kind scenario.

Usually ${ADMIN_URL} sets up the variable "weblogic.management.server". I tried to find out that who set this variable for admin mode startup, as setDomain.sh script enforces this variable is null for admin server startup sequence. On little debugging on environment variables at operating system level, it was clear ${JAVA_OPTS} was exported in the UNIX session, as one can also see in the startup sequence that the variables are repeated. So the sequence of events which landed me in this soup was:
  1. I started one managed server which exported JAVA_OPTS variable in the system session.
  2. When I restarted admin server, it appended its options in already exported JAVA_OPTS as the variable was not cleared after managed server start/ restart.

Solution 

  1. Either manually purge the startup variables from UNIX, Linux session if you are stubborn and don't want to close your session.
  2. Kill the UNIX Linux session. Start a new session, it will automatically purge the exported variables. Now start your admin server. 

Example Log excerpt with startup options  

-Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/apps2/BEA/WebLogic1036/wlserver_10.3/server/lib/cacerts -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/apps2/BEA/WebLogic1036/wlserver_10.3/server/lib/cacerts -Dweblogic.ProductionModeEnabled=true -Dweblogic.ProductionModeEnabled=true -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=http://10.188.103.55:7002 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -da -Dplatform.home=/apps2/BEA/WebLogic1036/wlserver_10.3 -Dwls.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.home=/apps2/BEA/WebLogic1036/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/apps2/BEA/WebLogic1036/patch_wls1036/profiles/default/sysext_manifest_classpath:/apps2/BEA/WebLogic1036/patch_ocp371/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=weblogic -Dweblogic.management.password=weblogic2

<<WLS Kernel>> <> <> <1389043279519> <BEA-400000> <Initializing message interception service>
####<Jan 6, 2014 1:21:19 PM PST> <Error> <Configuration Management> <dh6802c1> <OWSAdmin> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1389043279801> <BEA-150000> <An error occurred while establishing a connection back to the adminstration server t3://141.204.67.218:7002 during startup. Since bootstrap succeeded check that t3://141.204.67.218:7002 uniquely identifies the administration server.
javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:173)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:146)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:455)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:173)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:146)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
>
####<Jan 6, 2014 1:21:19 PM PST> <Critical> <WebLogicServer> <dh6802c1> <OWSAdmin> <main> <<WLS Kernel>> <> <> <1389043279816> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

javax.naming.ServiceUnavailableException [Root exception is java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.]
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:70)
        at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:32)
        at weblogic.jndi.WLInitialContextFactoryDelegate.toNamingException(WLInitialContextFactoryDelegate.java:792)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:431)
        at weblogic.jndi.Environment.getInitialReference(Environment.java:245)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:173)
        at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:146)
        at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:88)
        at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:174)
        at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:154)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.rmi.NoSuchObjectException: The object identified by: '31' could not be found.  Either it was has not been exported or it has been collected by the distributed garbage collector.
        at weblogic.rmi.internal.OIDManager.getServerReference(OIDManager.java:251)
        at weblogic.rmi.extensions.server.ServerHelper.getLocalInitialReference(ServerHelper.java:455)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:131)
        at weblogic.rmi.extensions.StubFactory.getStub(StubFactory.java:124)
        at weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:427)
        ... 9 more

Comments

Post a Comment

Popular posts from this blog

Caused by: java.sql.SQLTimeoutException: ORA-01013: user requested cancel of current operation

Many times we encounter this error and think that there might be some JPA issue or network issue. However, the error is pretty explicit in its nature, but still not clear to ring the bell very first time. As the error says there is a timeout, and since Oracle is saying, we shall accept it. Another clue is "user requested cancel of current operation". This means that the connection was closed by the application server not the database server, hence it can be safely concluded that application has configured timeouts on transactions at its end. However, if you feel that this transaction should have been concluded within the application server limits, your doubt might be true. This problem can arise in case there is a lock on the record being updated by JPA or lock on the table in which JPA is attempting to insert. The lock is most probably being held by some other application, and since application server configuration is on aggressive SLA, the transaction is waiting its
Read more

HashiCorp Vault Integration with Ansible Etower using approle

Image
HashiCorp Vault is  a secrets management tool specifically designed to control access to sensitive credentials in a low-trust environment . It can be used to store sensitive values and at the same time dynamically generate access for specific services/applications on lease.  Integrating the vault with Ansible Etower provides robust and secure automation. Following is the step-by-step guide for the integration. Enable key-value secret engine in Hashi Vault (also known as "kv" engine). Let's call the engine, the "kv" engine. Create a secret inside "kv". A secret can be a collection of key-value pairs or a JSON for nested structure.            Lets assume that secrets are stored as JSON  in the format   { "my_app":{ "service_account_name": "some_service", "service_account_password": "some_password" } } Create a secret policy defining what can be done with the above-defined secret. Create an
Read more

utility to extract date from text with java

Image
A date pattern recognition algorithm to not only identify date pattern but also fetches probable date in Java date format. This algorithm is very fast and lightweight. The processing time is linear and all dates are identified in a single pass. Algorithm resolves date using tree traverse mechanism. Tree data structures are custom created to build supported date, time and month patterns. Following Trees are used (Note: ^ sign denotes complete pattern) DATE PATTERN TREE Month Pattern Tree (Jan and January both are valid, identified using ^ sign) Time pattern (Identified as suffix to Date, $ sign indicates SPACE character) Tree structures are used in the following algorithm.  Date Part Identification algorithm flowchart Time part identification algorithm flowchart These flow charts are self-explanatory in most part. --> sign is used to denote next match in respective tree structures. The algorithm also acknowledges multipl
Read more