ORA-21780: Maximum number of object durations exceeded.

I noticed that the Oracle Enterprise Manager (OEM) repository database was throwing ORA-21780 errors every 10 seconds. This is what I found in the alertlog file:
Mon May 11 17:51:56 2010
Errors in file /oracle_binaries/app/diag/rdbms/oem/OEM1/trace/OEM1_smon_12465.trc:
ORA-21780: Maximum number of object durations exceeded.
Errors in file /oracle_binaries/app/diag/rdbms/oem/OEM1/trace/OEM1_smon_12465.trc:
ORA-21780: Maximum number of object durations exceeded.
Now looking at the SMON trace file:
*** 2015-05-11 17:56:57.999
SMON: following errors trapped and ignored:
ORA-21780: Maximum number of object durations exceeded.
*** 2015-05-11 17:57:08.155
         Drop transient type:   SYSTPE9CqEeDOZcXgUPAKvfBsfQ==
*** 2015-05-11 17:57:08.155
SMON: following errors trapped and ignored:
ORA-21780: Maximum number of object durations exceeded.
*** 2015-05-11 17:57:18.245
         Drop transient type:   SYSTPE9CqEeDOZcXgUPAKvfBsfQ==
*** 2015-05-11 17:57:18.245
SMON: following errors trapped and ignored:
ORA-21780: Maximum number of object durations exceeded.
As per MOS note 1081950.1, this is a non-reproducible issue and the cause is not identified.
It also states that this only happens in RAC environments.  The workaround provided is to manually drop those types, so let’s drop them.
First, we need to identify the problematic types and the owner.
SQL> select distinct(o.OWNER#) from obj$ o, type$ t
where o.oid$ = t.tvoid and
bitand(t.properties,8388608) = 8388608 and (sysdate-o.ctime) > 0.0007;
NAME                         
——————————
115
Then we need to identify the username of the previous query:
SQL> select username from dba_users where user_id=115;
USERNAME
——————————
SYSMAN
And finally, identify and drop the types:
SQL> alter session set nls_date_format=’DD-MON-YYYY HH24:MI:SS’;
set pagesize 1000
select ‘DROP TYPE SYSMAN.”‘||o.name||'” FORCE;’ from obj$ o, type$ t
where o.oid$ = t.tvoid and
bitand(t.properties,8388608) = 8388608 and (sysdate-o.ctime) > 0.0007;
Session altered.
SQL> SQL>   2    3
‘DROPTYPESYSMAN.”‘||O.NAME||'”FORCE;’
——————————————————–
DROP TYPE SYSMAN.”SYSTPE/jlp3fkcwzgUPAKvfAxHg==” FORCE;
DROP TYPE SYSMAN.”SYSTPFHGYs/8tGgLgUPAKvfAKqg==” FORCE;
DROP TYPE SYSMAN.”SYSTPFOpLxs22zYbgUPAKvfAnig==” FORCE;
DROP TYPE SYSMAN.”SYSTPFe/P5cSmQTDgUPAKvfA/HQ==” FORCE;
DROP TYPE SYSMAN.”SYSTPFWL+wF51Fc/gUPAKvfByCA==” FORCE;
DROP TYPE SYSMAN.”SYSTPFiwpaJHEvQTgUPAKvfAwiw==” FORCE;
DROP TYPE SYSMAN.”SYSTPFkBG80DWv1ngUPAKvfBD9Q==” FORCE;
DROP TYPE SYSMAN.”SYSTPFkEdeM4CTOngUPAKvfBrTQ==” FORCE;
8 rows selected.
After those types were dropped, no more error messages appeared in the alertlog file.
Thanks,
Alfredo

Free Database Seminar in Timisoara, Romania

Today I want to talk about a free Database Seminar that ACI is sponsoring in Timisoara, Romania. A friend of mine and former co-worker asked me to speak in the seminar about Oracle Enterprise Manager Administration. 

Because of the Time zone’s differences, I’m going to make a recording of the presentation and upload it to the blog, so he can show it there.

If you are nearby Timisoara, don’t loose the opportunity to assist and learn little bit more about databases.

The event is going to take place April, 8th at 5 pm @ City Business Centre, building D, Timisoara, RO.

Did I forget to mention that the event is for free?

For more information go to:






Thanks,

Alfredo

RMAN jobs not working after OEM upgrade to 12.1.0.4

If you are planning to upgrade your OEM to 12.1.0.4 and you have RMAN jobs scheduled in Cloud Control, you should consider applying patch 19519190 to the OMS. I noticed that most of the RMAN jobs were having issues and even worst, some steps were empty!!! 

Obviously, the jobs were succeeding as the step is empty. In other words, the jobs were doing nothing.

Looks like this patch is not part of any PSU, yet! But having a problem with hundreds of jobs and especially with RMAN jobs is very risky.

Take a look at EM 12c: RMAN Step Commands are Being Removed from Multi-step RMAN Script Jobs in Enterprise Manager 12.1.0.4 Cloud Control (Doc ID 1914916.1).

Thanks,

Alfredo

Using OMS DEBUG mode to troubleshoot OEM 12c problems

This time, I want to show you how to troubleshoot OEM problems by enabling DEBUG mode in the OMS. The virtual machine (VM) running my sandbox installation of OEM 12c 12.1.0.4 crashed during the night. After restarting the VM and all the OEM components, I wasn’t able to login using the SYSMAN account. The error from the console was not very explicit, just, “Authentication failed. If problem persists, contact your system administrator.”

In order to get more details about the error, I decided to enable DEBUG mode for the OMS and reproduce the error. This is what I did to enable DEBUG mode.

$ cd /u01/app/oracle/oms/oms/bin
$ ./emctl set property -name log4j.rootCategory -value “DEBUG, emlogAppender, emtrcAppender” -module logging
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
SYSMAN password:
Property log4j.rootCategory has been set to value DEBUG, emlogAppender, emtrcAppender for all Management Servers
OMS restart is not required to reflect the new property value
After enabling DEBUG mode, I reproduced the error several times using the console. I also wrote down the approximate time of the error, just to easy the search in the log file. Searching in the emoms.trc file located under /em/EMGC_OMS1/sysman/log/, found an ORA-14400 error. The MOS note 1493151.1, explains how to fix the issue by adding a new audit partition.

$ cd /u01/app/oracle/gc_inst/em/EMGC_OMS1/sysman/log/
$ view emoms.trc
java.sql.SQLException: ORA-14400: inserted partition key does not map to any partition
The final step is to disable the DEBUG mode for your OMS, otherwise the log files can grow real big and the performance could be affected.

$ ./emctl set property -name log4j.rootCategory -module LOGGING -value “WARN, emlogAppender, emtrcAppender”
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation.  All rights reserved.
SYSMAN password:
Property log4j.rootCategory has been set to value WARN, emlogAppender, emtrcAppender for all Management Servers
OMS restart is not required to reflect the new property value
I hope this information is useful to you next time you are troubleshooting an OEM 12c issue.
Thanks,

Alfredo

OEM 12c very slow after upgrade to 12.1.0.4

I noticed that OEM 12c console was very slow a few hours after the upgrade to 12.1.0.4 version. 

Looking at the repository DB, found several OMS sessions consuming significant CPU resources.
Bug 19199023 explains that some SQL queries executed against the repository consume high CPU on the servers. This bug affects the DB plug-in 12.1.0.6 and the patch 19176910 should be applied to the plug-ins.

More information available on MOS note, 12.1.0.4 OEM: High CPU utilization on Repository Database due to SYSMAN query WITH TARGETGUID AS (SELECT target_guid, host_name FROM mgmt_targets (Doc ID 1912172.1)
Thanks,

Alfredo

Cannot start ASM – ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”

Today, I faced and issue with an ASM instance. After bouncing the server, CRS went up along with the ASM instance, but the diskgroups were offline.
$ crsctl status resource -t
——————————————————————————–
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
               ONLINE  OFFLINE      hosta
ora.FRA.dg
               ONLINE  OFFLINE      hosta
ora.LISTENER.lsnr
               ONLINE  ONLINE       hosta
ora.LISTENER_1.lsnr
               ONLINE  ONLINE       hosta
ora.asm
               ONLINE  ONLINE       hosta                 Started
ora.ons
               OFFLINE OFFLINE      hosta
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.cssd
      1        ONLINE  ONLINE       hosta
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       hosta
ora.database.db
      1        ONLINE  OFFLINE                               Instance Shutdown
ora.database1.db
      1        OFFLINE OFFLINE                               Instance Shutdown
I tried to start ora.DATA.dg resource, but it failed.

$ crsctl start resource ora.DATA.dg
CRS-2672: Attempting to start ‘ora.DATA.dg’ on ‘hosta’
CRS-5017: The resource action “ora.DATA.dg start” encountered the following error:
ORA-15032: not all alterations performed
ORA-15017: diskgroup “DATA” cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”
. For details refer to “(:CLSN00107:)” in “/u01/oracle/grid/log/hosta/agent/ohasd/oraagent_oracle/oraagent_oracle.log”.
CRS-2674: Start of ‘ora.DATA.dg’ on ‘hosta’ failed
CRS-2679: Attempting to clean ‘ora.DATA.dg’ on ‘hosta’
CRS-2681: Clean of ‘ora.DATA.dg’ on ‘hosta’ succeeded
CRS-4000: Command Start failed, or completed with errors.
After checking the RAW devices on the host, everything appeared to be properly configured. Then I checked the configuration of the ASM instance, finding the ASM_DISKSTRING empty.
$ crsctl stat resource ora.asm -f
NAME=ora.asm
TYPE=ora.asm.type
STATE=OFFLINE
TARGET=OFFLINE
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r–
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=
ASM_DISKSTRING=
AUTO_START=restore
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CREATION_SEED=11
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
ID=ora.asm
LOAD=1
LOGGING_LEVEL=1
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
SPFILE=+DATA/asm/asmparameterfile/registry.123.785123625
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(ora.cssd)
STOP_TIMEOUT=600
TYPE_VERSION=1.2
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.3.0
I updated ASM_DISKSTRING with the discovery path of the disks and then bounced ASM instance.
$ srvctl modify asm -d ‘/dev/sd*’
$ srvctl stop asm
$ srvctl start asm
After this the ASM instance came up cleanly and the diskgroups were mounted.

$ crsctl status resource -t
——————————————————————————–
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
               ONLINE  ONLINE      hosta
ora.FRA.dg
               ONLINE  ONLINE      hosta
ora.LISTENER.lsnr
               ONLINE  ONLINE       hosta
ora.LISTENER_1.lsnr
               ONLINE  ONLINE       hosta
ora.asm
               ONLINE  ONLINE       hosta                 Started
ora.ons
               OFFLINE OFFLINE      hosta
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.cssd
      1        ONLINE  ONLINE       hosta
ora.diskmon
      1        OFFLINE OFFLINE
ora.evmd
      1        ONLINE  ONLINE       hosta
ora.database.db
      1        ONLINE  ONLINE                               Open
ora.database1.db
      1        OFFLINE ONLINE                               Open
Hope this help you to troubleshoot and fix the issue on your ASM, when is not able to find the disks.

Thanks,

Alfredo

Understanding Oracle SQL Plan Management SPM – Part 3

This is the third post of SQL Plan Management. In the previous post, Understanding SQL Plan Management– Part 2, I described the three main components of SPM, this time I want to show you how to manually capture plans from the cursor cache. In other words, manually capture plans that reside in memory.

In order to manually capture plans, I’m considering that automatic capture is disabled or OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES is set to FALSE. Oracle provides a function named DBMS_SPM.load_plans_from_cursor_cache to manual capture plans from cursor cache.

DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE (
   sql_id            IN  VARCHAR2,
   plan_hash_value   IN  NUMBER   := NULL,
   sql_text          IN  CLOB,
   fixed             IN  VARCHAR2 := ‘NO’,
   enabled           IN  VARCHAR2 := ‘YES’)
 RETURN PLS_INTEGER;
The common use for this function, is to specify the sql_id of the statement we want to capture. We also can specify the plan_hash_value or the sql_text. Two more attributes can be set within this function, fixed is ‘NO’ by default and enabledis ‘YES’ by default. If you want to load plans, but you don’t want the CBO to make use if them; set the attribute enabled to ‘NO’.

In the below example I’m capturing all the SQL plans for sql_id4c372tsuhtunm and printing the number of plans loaded:
SQL> SET SERVEROUTPUT ON
DECLARE
  l_plans_loaded  PLS_INTEGER;
BEGIN
  l_plans_loaded := DBMS_SPM.load_plans_from_cursor_cache(
    sql_id => ‘4c372tsuhtunm’);
  DBMS_OUTPUT.put_line(‘Plans Loaded: ‘ || l_plans_loaded);
END;
/SQL>   2    3    4    5    6    7    8    9
Plans Loaded: 8
PL/SQL procedure successfully completed.
To verify the SQL baselines created for this sql_id, I just need to query DBA_SQL_PLAN_BASELINES view:
SQL> select SIGNATURE,SQL_HANDLE,PLAN_NAME,ORIGIN,ENABLED,ACCEPTED,FIXED,REPRODUCED from dba_sql_plan_baselines;
            SIGNATURE SQL_HANDLE            PLAN_NAME                      ORIGIN         ENABLED ACCEPTED FIXED REPRODUCED
——————— ——————— —————————— ————– ——- ——– —– ———-
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5ye8c8b02d MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5yc568a49d MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5ybff74238 MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5y0310173f MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5y8e42f3cc MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5y25375ef9 MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5y9de69d5d MANUAL-LOAD    YES     YES      NO    YES
  5535156277878016190 SQL_4cd0d34ee73148be  SQL_PLAN_4tn6m9vmm2k5yb76f0084 MANUAL-LOAD    YES     YES      NO    YES
8 rows selected.
Look at the data for the ORIGIN column, it shows from where the plans were loaded, either from manual load or auto capture.

Another method is to manual load plans from SQL Tuning Sets (STS). The STS may contain plans that are not present in memory, like plans in the AWR repository. This method is very useful when you want to create baselines of plans that were created by the CBO few days ago and are not in the cursor cache at this time. Keep in mind that STS requires a special license in order to be used. Oracle provides the function DBMS_SPM.load_plans_from_sqlset to accomplish this task.
DBMS_SPM.LOAD_PLANS_FROM_SQLSET (
   sqlset_name      IN  VARCHAR2,
   sqlset_owner     IN  VARCHAR2 := NULL,
   basic_filter     IN  VARCHAR2 := NULL,
   fixed            IN  VARCHAR2 := ‘NO’,
   enabled          IN  VARCHAR2 := ‘YES’
   commit_rows      IN  NUMBER   := 1000)
RETURN PLS_INTEGER;
You just need to specify the sqlset_name and the sqlset_owner in order to load the plans contained in the STS. It also has the default parameters fixed, set to ‘NO’ and enabled,set to ‘YES’. Two interesting attributes can be set within this function. The basic_filter attribute allows you to select only the plans that meet this filter criteria, in other words, is like a ‘where’ in a query.
basic_filter => ‘sql_text like ”select /*LOAD_STS*/%”’ orbasic_filter => ‘sql_id=”4c372tsuhtunm“‘
The commit_rows attribute, allows you to commit after the value specified, which is 1000 by default. Let’s assume that inside this STS we have 30000 plans and we want to load all of them, SPM will commit every 1000 plans to help to reduce the undo log.

Keep in mind that once you manually load plans for a specific SQL (signature), the CBO will continue capturing plans for it, even if the automatic capture is disabled. Those automatic captured plans will not be used by the CBO until you verify and evolve them.

In the next post, I’m going to show you how to evolve plans in 11g and 12c versions.

Thanks,

Alfredo