OEM 13C Useful EMCTL Agent Commands

Here’s a list of some useful emctl commands to manage your OEM 13c agent.
Get the status of the agent:
$ emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Last Reload            : (none)
Last successful upload                       : 2018-05-15 09:04:32
Last attempted upload                        : 2018-05-18 13:12:29
Total Megabytes of XML files uploaded so far : 13.6
Number of XML files pending upload           : 1,956
Size of XML files pending upload(MB)         : 3.64
Available disk space on upload filesystem    : 81.90%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable [not running]
Last attempted heartbeat to OMS              : 2018-05-18 13:12:02
Last successful heartbeat to OMS             : 2018-05-15 09:10:00
Next scheduled heartbeat to OMS              : 2018-05-18 13:12:44
—————————————————————
Agent is Running and Ready
Get the list of targets currently monitored by the agent:
$ emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
[localhost.localdomain.com, host]
[localhost.localdomain.com:3872, oracle_emd]
[Management Services and Repository, oracle_emrep]
[/EMGC_GCDomain/GCDomain/EMGC_OMS1, weblogic_j2eeserver]
[NodeManager_localhost.localdomain.com_1, weblogic_nodemanager]
[oms13c1_1_localhost.localdomain.com_4754, oracle_home]
Get the status of a particular target:
$ emctl status agent target /EMGC_GCDomain/GCDomain/EMGC_OMS1,weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Target Name : /EMGC_GCDomain/GCDomain/EMGC_OMS1
Target Type : weblogic_j2eeserver
Current severity state
———————-
Metric        Column name             Key             State           Timestamp
——————————————————————————–
Response      Status                  n/a             CRITICAL        Fri May 18 13:13:25 EDT 2018
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:07:21 2017/254 CRITICAL        Fri Jun 23 16:07:58 EDT 2017
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:08:10 2017/263 CRITICAL        Fri Jun 23 16:12:58 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jul 10 06:06:34 2017/290 CRITICAL        Mon Jul 10 06:08:52 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jun 26 13:04:51 2017/272 CRITICAL        Mon Jun 26 13:08:48 EDT 2017
alertLogAdrIncident adr_problemKey          Tue Jun 27 17:50:54 2017/281 CRITICAL        Tue Jun 27 17:53:48 EDT 2017
jvm           heapUsedPercentage.value n/a             CLEAR           Tue May 15 09:07:27 EDT 2018
jvm_threads   deadlockedThreadCount.value n/a             CLEAR           Tue May 15 09:06:15 EDT 2018
—————————————————————
Agent is Running and Ready
Clear the current status of a target’s metrics:
$ emctl clearstate agent /EMGC_GCDomain/GCDomain/EMGC_OMS1,weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Clear target severity state
Now they show as undefined until the next collection happens:
$ emctl status agent target /EMGC_GCDomain/GCDomain/EMGC_OMS1,weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Target Name : /EMGC_GCDomain/GCDomain/EMGC_OMS1
Target Type : weblogic_j2eeserver
Current severity state
———————-
Metric        Column name             Key             State           Timestamp
——————————————————————————–
Response      Status                  n/a             UNDEFINED       Fri May 18 13:14:25 EDT 2018
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:07:21 2017/254 UNDEFINED       Fri Jun 23 16:07:58 EDT 2017
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:08:10 2017/263 UNDEFINED       Fri Jun 23 16:12:58 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jul 10 06:06:34 2017/290 UNDEFINED       Mon Jul 10 06:08:52 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jun 26 13:04:51 2017/272 UNDEFINED       Mon Jun 26 13:08:48 EDT 2017
alertLogAdrIncident adr_problemKey          Tue Jun 27 17:50:54 2017/281 UNDEFINED       Tue Jun 27 17:53:48 EDT 2017
jvm           heapUsedPercentage.value n/a             UNDEFINED       Tue May 15 09:07:27 EDT 2018
jvm_threads   deadlockedThreadCount.value n/a             UNDEFINED       Tue May 15 09:06:15 EDT 2018
—————————————————————
Agent is Running and Ready
Force the agent to collect dynamic properties:
$ emctl reload agent dynamicproperties /EMGC_GCDomain/GCDomain/EMGC_OMS1:weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
EMD recompute dynprops completed successfully
Thanks,
Alfredo

OMS responded illegally [ERROR- Failed to Update Target Type Metadata] in 13c R2

I want to discuss about this OMS error coming from the agent’s heartbeat status. The agent shows with an unreachable status but is actually up and running.
$: emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 13.2.0.0.0
OMS Version            : (unknown)
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 2,210
Size of XML files pending upload(MB)         : 2.89
Available disk space on upload filesystem    : 56.39%
Collection Status                            : Collections enabled
Heartbeat Status       : OMS responded illegally [ERROR- Failed to Update Target Type Metadata]
Last attempted heartbeat to OMS              : 2017-01-24 21:09:21
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2017-01-24 21:09:51
—————————————————————
Agent is Running and Ready
In this case the agent is not able to upload the metadata because is different in the OMS.
This is mainly due to a patch in the agent side that is missing in the OMS. By mistake I applied the latest bundle patch to the agent before applying it to the OMS.
Just another proof that any patches have to be installed to the OMS first then the agents.
Thanks,

Alfredo

OEM Agent Plug-in Testing

Today I want to write about Oracle Enterprise Manager (OEM) 12c plug-ins and how to test them. 

OEM is divided into two logical parts, the EM Platform and the EM Plug-ins. The EM Platform basically consists of the UI console, job system, metrics, EM agent, etc. The EM Plug-ins are modules that you can attach to you EM Platform to extend the monitoring functionality. As a result, we have plug-ins for Oracle Databases, Fusion Middleware, even hardware or 3rd-party vendors like MS SQL Server.

The plug-ins have a set of Perl scripts that the agent uses to compute status and metrics for the targets. You can find these scripts under “/plugins//scripts”.
Let us see an example:

Fusion Middleware plug-in version 12.1.0.7:
$ cd  /plugins/oracle.sysman.emas.agent.plugin_12.1.0.7.0/scripts

In there you can see tons of Perl scripts. These scripts are being used by the agent to compute response and metrics for the targets registered in the agent.

Now let’s say you have a Web Cache target that shows an incorrect status in OEM and you want to know how the agent computes the response status of this target.

 $ ls webcache*.pl
webcacheesm.pl  webcacheIsStandalone.pl  webcacheresource.pl  webcacheresponse.pl

You can see that there’s a response script available. 
Now the question is, how to test it?

Based on the MOS id 1534087.1, you can create an env.sh script to load the required variables to your shell. I did create the env.sh file and source it:

$ . env.sh

Now I’m able to test the webcacheresponse.pl:

$ perl webcacheresponse.pl
em_result=1

Looks like having 1 as the result means that the Web Cache is up per the script’s logic. This result is passed to the agent and finally to the OMS.

Hope this is helpful for you while trying to investigate status related issues with your targets.
Thanks,

Alfredo

How to remove OEM agent in the release 3 of the Oracle Enterprise Manager.

OEM 12c R3 has a very useful Perl script which allows removing an installed OEM agent in the host. The process is quite straightforward just by invoking the AgentDeinstall.pl script located under:

$AGENT_HOME/sysman/install/AgentDeinstall.pl

[oracle@ol6-112 12.1.0.3.0]$ /u01/app/oracle/agent12cr3/core/12.1.0.3.0/perl/bin/perl /u01/app/oracle/agent12cr3/core/12.1.0.3.0/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME

Agent Oracle Home: /u01/app/oracle/agent12cr3/core/12.1.0.3.0

NOTE: The agent base directory: /u01/app/oracle/agent12cr3 will be removed after successful deinstallation of agent home.

DetachHome Command executed:/u01/app/oracle/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/u01/app/oracle/agent12cr3/core/12.1.0.3.0 -waitForCompletion -invPtrLoc /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8015 MB    Passed
The inventory pointer is located at /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8015 MB    Passed
The inventory pointer is located at /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
The Oracle home '/u01/app/oracle/agent12cr3/sbin' could not be updated as it does not exist.

Deinstall Command executed:/u01/app/oracle/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/u01/app/oracle/agent12cr3/core/12.1.0.3.0}" -waitForCompletion -removeAllFiles -invPtrLoc /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8015 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-11-11_02-22-46PM. Please wait ...Oracle Universal Installer, Version 11.1.0.11.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.

Starting deinstall

Deinstall in progress (Monday, November 11, 2013 2:23:00 PM CST)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Monday, November 11, 2013 2:23:37 PM CST)
End of deinstallations
Please check '/u01/app/oraInventory/logs/silentInstall2013-11-11_02-22-46PM.log' for more details.

Do you want us to delete the old oracle home [yes/no] :y

Hope this help you when need to remove an OEM agent installation.

Thanks,
Alfredo