OEM - alfredokriegdba.com

OEM 12c very slow after upgrade to 12.1.0.4

I noticed that OEM 12c console was very slow a few hours after the upgrade to 12.1.0.4 version.

Looking at the repository DB, found several OMS sessions consuming significant CPU resources.

Bug 19199023 explains that some SQL queries executed against the repository consume high CPU on the servers. This bug affects the DB plug-in 12.1.0.6 and the patch 19176910 should be applied to the plug-ins.

More information available on MOS note, 12.1.0.4 OEM: High CPU utilization on Repository Database due to SYSMAN query WITH TARGETGUID AS (SELECT target_guid, host_name FROM mgmt_targets (Doc ID 1912172.1)

Thanks,

Alfredo

Upgrade Oracle Enterprise Manager to 12.1.0.4

Enterprise Manager 12.1.0.4 was just released and now is time to test and document the process. The official documentation can be obtained from here (http://docs.oracle.com/cd/E24628_01/upgrade.121/e22625/upgrading_12101_PS1_gtgstrtd.htm) and this document cover the steps I followed and may not work on yours (I’m not responsible of any issues/problems caused to your system).

This document covers Oracle Enterprise Manager upgrade from 12.1.0.3 to 12.1.0.4 version in a Single-OMS configuration using one system upgrade method and is divided in 3 sections describing prerequisites, upgrade and post upgrade tasks.

Prerequisites

Download the software from (http://www.oracle.com/technetwork/oem/enterprise-manager/downloads/index.html) on your server and extract it to a stage directory.

mkdir /stage

unzip em12104_linux64_disk1.zip -d /stage

unzip em12104_linux64_disk2.zip -d /stage

unzip em12104_linux64_disk3.zip -d /stage

Ensure that you have a valid OMS (middleware home & inventory), Repository and Software Library backup.
If you have BI Publisher installed, then stop BIP server through WebLogic Admin Console.
Drop any snapshot created against Repository tables.

SQL> select master , log_table from all_mview_logs where log_owner=’SYSMAN’

no rows selected

Check and compile invalid objects in the Repository schema.

SQL> select ‘alter ‘||object_type||’ ‘||owner||’.’||object_name||’ compile;’ from all_objects

2 where status=’INVALID’;

no rows selected

Execute privileges on DBMS_RANDOM are required during the upgrade, run the below commands to set them.

SQL> grant execute on DBMS_RANDOM to DBSNMP;

SQL> grant execute on DBMS_RANDOM to SYSMAN;

SQL> revoke execute on DBMS_RANDOM from PUBLIC;

Copy the emkey from OMS to the Management Repository.

$ emctl config emkey -copy_to_repos

Oracle Enterprise Manager Cloud Control 12c Release 3

Enter Enterprise Manager Root (SYSMAN) Password :

The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.

After the required operation has been completed, secure the EMKey by running “emctl config emkey -remove_from_repos”.

This is a critical mandatory step. You must apply the following patches to your Repository database in order to avoid an upgrade failure (full list can be found http://docs.oracle.com/cd/E24628_01/upgrade.121/e22625/upgrading_12101_PS1_gtgstrtd.htm#BABGEHCI). In my case need to apply patch 12419378 for 11.2.0.1 database version.

Stop OMS, extended instances and both OMS and Repository agents.

$ emctl extended oms jvmd stop -all

Oracle Enterprise Manager Cloud Control 12c Release 3

Please enter the SYSMAN password:

Stopping all Engines

{}

No engines found for this operation

Emctl Extended verb failed. Please check /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for more details

$ emctl extended oms adp stop -all

Oracle Enterprise Manager Cloud Control 12c Release 3

Please enter the SYSMAN password:

Stopping all Engines

{}

No engines found for this operation

Emctl Extended verb failed. Please check /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for more details

$ emctl stop oms -all

Oracle Enterprise Manager Cloud Control 12c Release 3

Stopping WebTier…

WebTier Successfully Stopped

Stopping Oracle Management Server…

Oracle Management Server Successfully Stopped

AdminServer Successfully Stopped

Oracle Management Server is Down

$ ./emctl stop agent

Oracle Enterprise Manager Cloud Control 12c Release 3

Stopping agent ….. stopped.

Upgrade

Run the runInstaller executable from the folder where the software was extracted.

$ cd /stage

$ ./runInstaller

Type My Oracle Support account details.

Choose if you want to search for any software updates.

Review and fix any prerequisite failure.

Choose the upgrade method, in my case One system Upgrade then select the Oracle Home.

Type the “NEW” Middleware Home location.

Type the required Repository credentials.

Some prerequisites failed, click Yes to let the installer fix them automatically.

Review the list of plug-ins that are going to be automatically upgraded.

Select any additional plug-ins to be deployed after the upgrade.

Type the required WebLogic Domain credentials.

Review the all the information provided and click Install.

Review the progress.

Run the required allroot.sh script as root.

Review the final information about the upgrade process.

Login to Cloud Control and verify the version. Note that some targets are marked as down, we are going to fix them in the post upgrade tasks.

Upgrade Oracle Management Agents (at least OMS and Repository). Go to Setup -> Manage Cloud Control -> Upgrade Agents.

Type a Job Name (or use the default name) and click Add to select the agents you want to upgrade to 12.1.0.4.

Select the agents from the list and click OK.

Choose credentials and click Submit. There’s the option to schedule the upgrade for later under Additional Inputs.

You must run the root.sh script after the upgrade, click OK.

Review the agent upgrade process.

Agent upgrade was successful.

Post Upgrade

Deleting the Old OMS Home is an optional task, however I have decided to perform it in order to save space and have a clean Inventory.

Invoke the installer from the old OMS home

$ export ORACLE_HOME=/u01/app/oracle/oms12cr3/oms

$ /oui/bin/runInstaller -deinstall

Select only the OMS-related plug-ins homes (from the Old Home only!!) and click Remove.

Select the Java Development Kit JDK home and click Remove.

Select the Oracle WebTier home and click Remove. After finishing the deinstall click Close.

Finally select the OMS home & the Oracle Common directories and click Remove.

All targets marked up.

Thanks,

Alfredo

How to remove OEM agent in the release 3 of the Oracle Enterprise Manager.

OEM 12c R3 has a very useful Perl script which allows removing an installed OEM agent in the host. The process is quite straightforward just by invoking the AgentDeinstall.pl script located under:

$AGENT_HOME/sysman/install/AgentDeinstall.pl

[oracle@ol6-112 12.1.0.3.0]$ /u01/app/oracle/agent12cr3/core/12.1.0.3.0/perl/bin/perl /u01/app/oracle/agent12cr3/core/12.1.0.3.0/sysman/install/AgentDeinstall.pl -agentHome $AGENT_HOME

Agent Oracle Home: /u01/app/oracle/agent12cr3/core/12.1.0.3.0

NOTE: The agent base directory: /u01/app/oracle/agent12cr3 will be removed after successful deinstallation of agent home.

DetachHome Command executed:/u01/app/oracle/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -detachHome -force -depHomesOnly -silent ORACLE_HOME=/u01/app/oracle/agent12cr3/core/12.1.0.3.0 -waitForCompletion -invPtrLoc /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8015 MB    Passed
The inventory pointer is located at /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
'DetachHome' was successful.
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8015 MB    Passed
The inventory pointer is located at /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
The Oracle home '/u01/app/oracle/agent12cr3/sbin' could not be updated as it does not exist.

Deinstall Command executed:/u01/app/oracle/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -deinstall -silent "REMOVE_HOMES={/u01/app/oracle/agent12cr3/core/12.1.0.3.0}" -waitForCompletion -removeAllFiles -invPtrLoc /u01/app/oracle/agent12cr3/core/12.1.0.3.0/oraInst.loc
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 8015 MB    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-11-11_02-22-46PM. Please wait ...Oracle Universal Installer, Version 11.1.0.11.0 Production
Copyright (C) 1999, 2013, Oracle. All rights reserved.

Starting deinstall

Deinstall in progress (Monday, November 11, 2013 2:23:00 PM CST)
Configuration assistant "Agent Deinstall Assistant" succeeded
............................................................... 100% Done.

Deinstall successful

End of install phases.(Monday, November 11, 2013 2:23:37 PM CST)
End of deinstallations
Please check '/u01/app/oraInventory/logs/silentInstall2013-11-11_02-22-46PM.log' for more details.

Do you want us to delete the old oracle home [yes/no] :y

Hope this help you when need to remove an OEM agent installation.

Thanks,
Alfredo

Oracle Enterprise Manager 12c Release 3 New Features

Today I will talk about some new features in the release 3 of the Oracle Enterprise Manager.

All Metrics Chart Enhancements

This new feature show more data in the metrics page, like minimum and maximum values, it will highlight periods when the agent was unreachable or target was down, it will also show the period when the metric was in warning or critical severity.

This really helps DBAs to better understand the behavior of the target including the metric data.

Metrics Schedule Enhancements

Metrics now support additional collection schedule like weekly on specified days of the week and monthly on specified days of the month.

Service Target Dashboard

Provides a quickly out-of-box overview of the health of services, this is really useful when looking for a quick view of what service is having problem in the environment.

Performance Diagnostics Enhancements

Even when this is not a new feature of the OEM 12c R3, ASH Analytics is really cool. It allows DBAs to diagnose performance exactly when DBA desire; it has customized selection of the desired point in time when issues appeared.

Well, these are some new and cool features of the Oracle Enterprise Manager 12c Release 3. Hope you can enjoy this features as much as me.

Thanks,

Alfredo