OEM 13c – AMP EBS Discovery Error Due To Lack Of Privileges

We recently had to enable monitoring for an EBS 11i system using the Application Management Pack. Even though the EM_MONITOR user was already provisioned by the EBS patch, the discovery was failing due to lack of privileges on several FND tables.
My Oracle Support shows several bugs related to missing privileges like the one below, but I wasn’t able to find one for 11i.

Patch 21951154: 1OFF:12.2.0: READ ACCESS NOT PRESENT FROM EM_MONITOR USER FOR FOLLOWING TABLES

I decided to manually track the missing privileges on these tables and here’s the list I found in order to make the discovery work.
GRANT SELECT ON “APPLSYS”.”AD_APPL_TOPS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”AD_FIXED_ISSUES” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”AD_PATCH_DRIVER_LANGS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”AD_PATCH_RUNS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”AD_PATCH_RUN_BUGS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”FND_LOG_EXCEPTIONS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”FND_LOG_UNIQUE_EXCEPTIONS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”FND_NODES” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”FND_PROFILE_OPTIONS_TL” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPLSYS”.”FND_USER” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “APPS”.”HR_OPERATING_UNITS” TO “EM_OAM_MONITOR_ROLE”;
GRANT EXECUTE ON “APPS”.”JTF_DIAG_DEPENDENCIES” TO “EM_OAM_MONITOR_ROLE”;
GRANT EXECUTE ON “APPS”.”JTF_DIAG_INPUTS” TO “EM_OAM_MONITOR_ROLE”;
GRANT EXECUTE ON “APPS”.”JTF_DIAG_INPUTTBL” TO “EM_OAM_MONITOR_ROLE”;
GRANT EXECUTE ON “APPS”.”JTF_DIAG_VERSION” TO “EM_OAM_MONITOR_ROLE”;
GRANT EXECUTE ON “APPS”.”JTF_DIAG_VERSION_NT” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”ALL_IND_COLUMNS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”ALL_OBJECTS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”ALL_QUEUES” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”ALL_TRIGGERS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_DATA_FILES” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_OBJECTS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_PROCEDURES” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_SOURCE” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_TAB_PRIVS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_TEMP_FILES” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_USERS” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”DBA_USERS_WITH_DEFPWD” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”GV_$INSTANCE” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”GV_$LOGFILE” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”GV_$SESSION” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”V_$INSTANCE” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”V_$THREAD” TO “EM_OAM_MONITOR_ROLE”;
GRANT SELECT ON “SYS”.”V_$VERSION” TO “EM_OAM_MONITOR_ROLE”;
Hope this helps to your discovery.
Thanks,
Alfredo

OEM 13c Reporting Incorrect Tablespace Space Usage (%) After July 2018 PSU

The topic for today is Tablespace monitoring. OEM 13c started reporting inaccurate values for the Space Usage (%) metric after applying either the April 2018 or the July 2018 Database PSU’s. The way this was detected, is that OEM 13c Corrective Actions started firing here and there after the patches.
Oracle confirmed this is tracked on BUG 26198757.
This patch is available on top of both the April 2018 and the October 2018 PSU’s but is not available for October 2018 PSU. The issue could also manifest itself as huge wait times on “Sync ASM rebalance” event.
Looks like the issue is corrected after applying the patch on top of July 2018 PSU. Below are some MOS realted to this.
Bug 26198757 – dba_tablespace_usage_metrics.used_percent is incorrect after applying 25397136 (Doc ID 26198757.8)
EM13c Space Monitoring Query (sqlid=69p6my4hpdm3j) On dba_tablespace_usage_metrics Timed Out after Db Upgraded To 12.2 (Doc ID 2375714.1)
EM 13c, 12.1.0.5: Enterprise Manager Cloud Control Tablespace Space Used (%) Metric Incorrectly Triggering (Doc ID 2313520.1)
DBA_TABLESPACE_USAGE_METRICS Returns Incorrect Information After applying 12.1.0.2.170418 (25397136) Bundle patch (Doc ID 2289448.1)
Thanks and happy patching,
Alfredo

Exalogic Agents being automatically deleted from OEM 13c

Oracle Enterprise Manager (OEM) 13c is capable of monitoring virtual guest hosts running on Exalogic (virtual configuration). OEM will connect (read-only) to the Oracle VM Manager and get the configuration and status of the VM’s running there. Then you can deploy an agent to those VM hosts and you have a complete monitoring solution for Exalogic.
The problem comes when for whatever reason you have to drop those VM’s from Exalogic but you want to keep the OEM configuration in place. For example, you want to drop the VM from Exalogic and then restore it from backup.
In this case you want to keep your monitoring configuration (metrics, credentials, etc.), but you realize that the agent was removed (along with all targets) as soon as you drop the VM from Exalogic.
This setup is being driven by metric like configuration. This synchronization is happening every minute by default.  This is covered on below Oracle’s document:
The solution was to set this synchronization schedule from every 1 minute to every 1 day.
Thanks,
Alfredo

OEM 13C Useful EMCTL Agent Commands

Here’s a list of some useful emctl commands to manage your OEM 13c agent.
Get the status of the agent:
$ emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Last Reload            : (none)
Last successful upload                       : 2018-05-15 09:04:32
Last attempted upload                        : 2018-05-18 13:12:29
Total Megabytes of XML files uploaded so far : 13.6
Number of XML files pending upload           : 1,956
Size of XML files pending upload(MB)         : 3.64
Available disk space on upload filesystem    : 81.90%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable [not running]
Last attempted heartbeat to OMS              : 2018-05-18 13:12:02
Last successful heartbeat to OMS             : 2018-05-15 09:10:00
Next scheduled heartbeat to OMS              : 2018-05-18 13:12:44
—————————————————————
Agent is Running and Ready
Get the list of targets currently monitored by the agent:
$ emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
[localhost.localdomain.com, host]
[localhost.localdomain.com:3872, oracle_emd]
[Management Services and Repository, oracle_emrep]
[/EMGC_GCDomain/GCDomain/EMGC_OMS1, weblogic_j2eeserver]
[NodeManager_localhost.localdomain.com_1, weblogic_nodemanager]
[oms13c1_1_localhost.localdomain.com_4754, oracle_home]
Get the status of a particular target:
$ emctl status agent target /EMGC_GCDomain/GCDomain/EMGC_OMS1,weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Target Name : /EMGC_GCDomain/GCDomain/EMGC_OMS1
Target Type : weblogic_j2eeserver
Current severity state
———————-
Metric        Column name             Key             State           Timestamp
——————————————————————————–
Response      Status                  n/a             CRITICAL        Fri May 18 13:13:25 EDT 2018
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:07:21 2017/254 CRITICAL        Fri Jun 23 16:07:58 EDT 2017
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:08:10 2017/263 CRITICAL        Fri Jun 23 16:12:58 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jul 10 06:06:34 2017/290 CRITICAL        Mon Jul 10 06:08:52 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jun 26 13:04:51 2017/272 CRITICAL        Mon Jun 26 13:08:48 EDT 2017
alertLogAdrIncident adr_problemKey          Tue Jun 27 17:50:54 2017/281 CRITICAL        Tue Jun 27 17:53:48 EDT 2017
jvm           heapUsedPercentage.value n/a             CLEAR           Tue May 15 09:07:27 EDT 2018
jvm_threads   deadlockedThreadCount.value n/a             CLEAR           Tue May 15 09:06:15 EDT 2018
—————————————————————
Agent is Running and Ready
Clear the current status of a target’s metrics:
$ emctl clearstate agent /EMGC_GCDomain/GCDomain/EMGC_OMS1,weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Clear target severity state
Now they show as undefined until the next collection happens:
$ emctl status agent target /EMGC_GCDomain/GCDomain/EMGC_OMS1,weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
Target Name : /EMGC_GCDomain/GCDomain/EMGC_OMS1
Target Type : weblogic_j2eeserver
Current severity state
———————-
Metric        Column name             Key             State           Timestamp
——————————————————————————–
Response      Status                  n/a             UNDEFINED       Fri May 18 13:14:25 EDT 2018
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:07:21 2017/254 UNDEFINED       Fri Jun 23 16:07:58 EDT 2017
alertLogAdrIncident adr_problemKey          Fri Jun 23 16:08:10 2017/263 UNDEFINED       Fri Jun 23 16:12:58 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jul 10 06:06:34 2017/290 UNDEFINED       Mon Jul 10 06:08:52 EDT 2017
alertLogAdrIncident adr_problemKey          Mon Jun 26 13:04:51 2017/272 UNDEFINED       Mon Jun 26 13:08:48 EDT 2017
alertLogAdrIncident adr_problemKey          Tue Jun 27 17:50:54 2017/281 UNDEFINED       Tue Jun 27 17:53:48 EDT 2017
jvm           heapUsedPercentage.value n/a             UNDEFINED       Tue May 15 09:07:27 EDT 2018
jvm_threads   deadlockedThreadCount.value n/a             UNDEFINED       Tue May 15 09:06:15 EDT 2018
—————————————————————
Agent is Running and Ready
Force the agent to collect dynamic properties:
$ emctl reload agent dynamicproperties /EMGC_GCDomain/GCDomain/EMGC_OMS1:weblogic_j2eeserver
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
—————————————————————
EMD recompute dynprops completed successfully
Thanks,
Alfredo

Upload the OPatch that is of version “13.8.0.0.0” and platform “226” to the Software Library

I was having an issue trying to deploy a patch to an agent due to the version of OPatch available in the software library.
The error I got was “Upload the OPatch that is of version “13.8.0.0.0” and platform “226” to the Software Library”

The solution was to run the “OPatch Update” job available from the job console in OEM.

Choose “OPatch Update” as job type:

Type a name for the job:

And click submit:

After this the agent patching was successful.
Thanks,
Alfredo

“Failed to find the OPatch patch directory” while applying patch 25105555 ( 13.2.0.0.161231) from OEM console

While applying patch 25105555 ( 13.2.0.0.161231) to an agent using the “provisioning and Patching” console from OEM 13.2 and choosing the “Upgrade OPatch” option, got this error:
———- Error Message ———-
Error: 010
Failed to find the OPatch patch directory “/tmp/p6880880_600000000057471_2000_0/oracle/OPatch”.
If you chose “No” for the “StagePatches” option, ensure that you manually place the patch at the required location and retry the operation.
———– End Message ———–
Tue Aug 27 15:41:48 2017 – Patching aborted.
You have to follow the solution section from below MOS note:
EM 13c: Applying EM-AGENT Bundle Patch 25105555 ( 13.2.0.0.161231) from Enterprise
Manager 13.2 Cloud Control Fails with Message: Error: 010 Failed to find the OPatch patch directory (Doc ID 2229452.1)
It depends on whether you have an online or offline OMS configuration.
After doing this I was able to apply patch 25105555 to the agent.
Hope this helps.

Alfredo

OEM 13c New Features – Corrective Actions

Today’s post is about another new feature of Oracle Enterprise Manager (OEM) 13c. Well, not a new feature itself but some awesome enhancements to Corrective Actions (CA).
In this new release 13c several new CA types were added, including “Add Space To Tablespace”, “Chef Recipe” for those who like automation and now can be added to Service Level Agreement (SLA) alerts and job status events.
I’m a big enthusiast of OEM jobs and now having the option to set a CA for them is just more than cool!
I wrote an article a year ago on IOUG’s Select journal about Corrective Actions and didn’t want to finish my day without setting a CA for job. But wait a minute, I know I can set CA for metrics using the “Metric and Collection Settings” page, but there’s no such thing for jobs.
Here’s how we setup a corrective action for jobs. First we need to enable events for jobs, this means that we as administrators decide what kind of job status will create an event.
In order to do this, you need to navigate to Setup -> Incidents -> Job Events
Step 1. Here you decide what job status and job severity generates an event.


Step 2. You decide if you want to generate events for jobs without targets.
Step 3. Choose the target you want to generate job events for.
Once you have the proper events setup, then the next step is to catch this event and do something with it. In this case fire a Corrective Action (CA). The only place we can catch these events and do something with them is an Incident Rule.
Navigate to Setup-> Incident -> Incident Rules
Then create or modify an existing Incident Rule Set. Inside this Rule Set create a new Rule to catch these job events. Inside the Rule Actions there’s a new section to Submit a Corrective Action.
Click on the “Select corrective action” and choose the CA from your CA Library.



Cool doesn’t it! One of the uses I can think of is with Database backup jobs. Sometimes the RMAN backup fails when is not able to find an archivelog and the solution is to run a crosscheck command and retry the backup job. This CA can do that for us and just notify us once it completes.
It still needs further testing but is another tool to save us time.
If you want information about the “Add Space To Tablespace” CA you better check Kellyn’s post about it.
Thanks,

Alfredo

OEM 13c New Features – System Broadcast

I can’t finish the year without a post about OEM 13c. I finally had some time to install it and from first hand go through its new features.  I find this “System Broadcast” feature so useful, especially when you have tons of users using OEM.
You can try to email users, give them a call but they will always forget about the next maintenance window and call you right away when OEM is not available. This feature helps you to send a notification to all the users or a particular list of users.
System Broadcast messages (up to 200 characters) can only be sent using EMCLI, there’s no graphical option yet.
Here’s an example on how to send a message to a user named OEMADMIN:

$ emcli send_system_broadcast -toOption=”SPECIFIC” -to=”OEMADMIN” -message=”OEM will be down for maintenance Friday December 30th”
Successfully requested to send System Broadcast to users.

This is what you see once you login to OEM console:



Send the message to all the users:


$ emcli send_system_broadcast -toOption=”ALL” -message=”OEM will be down for maintenance Friday December 30th”
Successfully requested to send System Broadcast to users.
You need to be logged in EMCLI to be able to send these messages, if you are not you’ll get this error:

$ emcli send_system_broadcast -toOption=”ALL” -message=”OEM will be down for maintenance Friday December 30th”
Status:Unauthorized 401
As I said, you need to be logged in EMCLI:

$emcli login -username=sysman
Enter password:

Login successful

$ emcli send_system_broadcast -toOption=”ALL” -message=”OEM will be down for maintenance Friday December 30th”
Successfully requested to send System Broadcast to users.
Here’s the verb syntax:
emcli send_system_broadcast
      -toOption=”ALL|SPECIFIC”
      [-to=”comma separated user names”]
      [-messageType=”INFO|CONF|WARN|ERROR|FATAL” (default is INFO)]
      -message=”message details”
[ ]  indicates that the parameter is optional.
Thank you and happy new year!
Alfredo