Extract AWR data to build Response Time graphs (awrrtsys.sql)

This SQL script extract AWR data required to perform Response Time analysis for Oracle Databases. All information about it will be presented in the IOUG Collaborate 2014 session “553: Oracle Database Performance: Are Database Users Telling Me The Truth?”
–#********************************************************************
–#– Filename           : awrrtsys.sql
–#– Author             : Alfredo Krieg
–#– Original           : 17-Oct-12
–#– Last Update        : 21-Ago-13
–#– Description        : awrrtsys.sql- System Response time per snapshot in the last x days
–#                        per desired unit of work –
–#– Usage              : start awrrtsys.sql
–#– This script use AWR data (licensing is required)
–#********************************************************************
set termout on
set feedback off
set heading on
set linesize 150
set pagesize 100
set feedback off verify off
col snap_id           format  999999     heading “Snapshot Id”
col instance_number   format  99         heading “Instance Number”
col snap_time         format  a25        heading “Snap Begin Time”
col non_idle_wait     format  9999999.99 heading “Qt (s)”
col cpu_time          format  9999999.99 heading “St (s)”
col lio               format  9999999.99 heading “LIO”
col pio               format  9999999.99 heading “PIO”
col uw_variable       format  9999999.99 heading “UC”
col rt_ms_per_lio     format  9999999.99 heading “RT (ms/lio)”
col rt_ms_per_pio     format  9999999.99 heading “RT (ms/pio)”
col rt_ms_per_uw      format  9999999.99 heading “RT (ms/uc)”
SELECT a.SNAP_ID,
         b.instance_number,
         TO_CHAR (END_INTERVAL_TIME, ‘mon/dd/yyyy HH24:mi’) AS snap_time,
         non_idle_wait,
         background_cpu+
         db_cpu as cpu_time,
         lio,
         pio,
         uwvariable AS UW_VARIABLE,
         ROUND (
            ( ( (background_cpu + db_cpu) / lio) + (non_idle_wait / lio))
            * 1000,
            4)
            AS RT_ms_per_lio,
         ROUND (
            ( ( (background_cpu + db_cpu) / pio) + (non_idle_wait / pio))
            * 1000,
            4)
            AS RT_ms_per_pio,
         ROUND (
            ( ( (background_cpu + db_cpu) / (uwvariable))
             + (non_idle_wait / (uwvariable)))
            * 1000,
            4)
            AS RT_ms_per_uw
    FROM ( 
    SELECT SNAP_ID,
                   SUM (non_idle_wait) AS non_idle_wait,
                   ROUND (SUM (background_cpu), 0) AS background_cpu,
                   ROUND (SUM (db_cpu), 0) AS db_cpu,
                   SUM (lio) AS lio,
                   SUM (pio) AS pio,
                   SUM (uwvariable) AS uwvariable
              FROM (
              SELECT SNAP_ID,
                           ROUND (time_secs, 0) non_idle_wait,
                           0 AS background_cpu,
                           0 AS db_cpu,
                           0 AS lio,
                           0 AS pio,
                           0 AS uwvariable
                      FROM (  SELECT snap_id,
                                     ‘WAIT’ AS wait_class,
                                     ‘Non Idle’ AS name,
                                     (SUM (VALUE) – SUM (value1)) / 1000000
                                        AS time_secs
                                FROM (  SELECT snap_id,
                                               0 AS snap2,
                                               SUM (time_waited_micro) AS VALUE,
                                               0 AS value1
                                          FROM dba_hist_system_event
                                         WHERE wait_class ‘Idle’
                                      GROUP BY snap_id
                                      UNION
                                        SELECT snap_id + 1,
                                               snap_id,
                                               0,
                                               SUM (time_waited_micro) AS value1
                                          FROM dba_hist_system_event
                                         WHERE wait_class ‘Idle’
                                      GROUP BY snap_id)
                            GROUP BY snap_id)
                    UNION
                      SELECT snap_id,
                             0,
                             (SUM (VALUE) – SUM (value1)) / 1000000
                                AS background_cpu,
                             0,
                             0,
                             0,
                             0                       
                        FROM (SELECT snap_id,
                                     0 AS snap2,
                                     stat_name,
                                     VALUE,
                                     0 AS value1
                                FROM dba_hist_sys_time_model
                               WHERE stat_name IN (‘background cpu time’)
                              UNION
                              SELECT snap_id + 1,
                                     snap_id,
                                     stat_name,
                                     0,
                                     VALUE AS value1
                                FROM dba_hist_sys_time_model
                               WHERE stat_name IN (‘background cpu time’))
                    GROUP BY snap_id, stat_name
                    UNION
                      SELECT snap_id,
                             0,
                             0,
                             (SUM (VALUE) – SUM (value1)) / 1000000 AS db_cpu,
                             0,
                             0,
                             0                          
                        FROM (SELECT snap_id,
                                     0 AS snap2,
                                     stat_name,
                                     VALUE,
                                     0 AS value1
                                FROM dba_hist_sys_time_model
                               WHERE stat_name IN (‘DB CPU’)
                              UNION
                              SELECT snap_id + 1,
                                     snap_id,
                                     stat_name,
                                     0,
                                     VALUE AS value1
                                FROM dba_hist_sys_time_model
                               WHERE stat_name IN (‘DB CPU’))
                    GROUP BY snap_id, stat_name
                    UNION
                      SELECT snap_id,
                             0,
                             0,
                             0,
                             SUM (VALUE) – SUM (value1) AS lio,
                             0,
                             0                           
                        FROM (SELECT snap_id,
                                     0 AS snap2,
                                     stat_name,
                                     VALUE,
                                     0 AS value1
                                FROM DBA_HIST_SYSSTAT
                               WHERE stat_name IN (‘session logical reads’)
                              UNION
                              SELECT snap_id + 1,
                                     snap_id,
                                     stat_name,
                                     0,
                                     VALUE AS value1
                                FROM DBA_HIST_SYSSTAT
                               WHERE stat_name IN (‘session logical reads’))
                    GROUP BY snap_id, stat_name
                    UNION
                      SELECT snap_id,
                             0,
                             0,
                             0,
                             0,
                             SUM (VALUE) – SUM (value1) AS pio,
                             0                           
                        FROM (SELECT snap_id,
                                     0 AS snap2,
                                     stat_name,
                                     VALUE,
                                     0 AS value1
                                FROM DBA_HIST_SYSSTAT
                               WHERE stat_name IN (‘physical reads’)
                              UNION
                              SELECT snap_id + 1,
                                     snap_id,
                                     stat_name,
                                     0,
                                     VALUE AS value1
                                FROM DBA_HIST_SYSSTAT
                               WHERE stat_name IN (‘physical reads’))
                    GROUP BY snap_id, stat_name
                    UNION
                      SELECT snap_id,
                             0,
                             0,
                             0,
                             0,
                             0,
                             SUM (VALUE) – SUM (value1) AS uwvariable                          
                        FROM (SELECT snap_id,
                                     0 AS snap2,
                                     stat_name,
                                     VALUE,
                                     0 AS value1
                                FROM DBA_HIST_SYSSTAT
                               WHERE stat_name IN (‘&1’)
                              UNION
                              SELECT snap_id + 1,
                                     snap_id,
                                     stat_name,
                                     0,
                                     VALUE AS value1
                                FROM DBA_HIST_SYSSTAT
                               WHERE stat_name IN (‘&1’))
                    GROUP BY snap_id, stat_name
                    )
          GROUP BY snap_id
            HAVING SUM (background_cpu) >= 0
          ORDER BY snap_id DESC) a, DBA_HIST_SNAPSHOT B
   WHERE a.SNAP_ID = b.SNAP_ID AND BEGIN_INTERVAL_TIME > SYSDATE – &2

ORDER BY 1;

Oracle 12c ASM new features


Today I will talk about some new features for ASM on 12c version.
Oracle Flex ASM
Oracle Flex ASM is a set of new features that provide critical capabilities required for cloud computing. Oracle Flex ASM redefines the traditional ASM cluster architecture of having one ASM instance on every node in the cluster; this means you can have less ASM instances than nodes in your cluster. In this configuration the number of ASM instances running is called ASM cardinality, by default the ASM cardinality is 3 and can be changed with a Clusterware command.
What are the benefits?
On the traditional architecture when an ASM instance fails; all DB instances connected to that instance in the node will also fail. With Oracle Flex ASM if the ASM instance fail the Clusterware will relocate that ASM instance to a different node and the DB instances will remotely (through private network) connect to the relocated ASM instance, all without disruption to the DB client.
Dedicated ASM Network
This new 12c feature provides the option to dedicate a private network for ASM network traffic only. There’s also the option to use the Oracle Clusterware interconnect private network.
Remote Access
As per the new Oracle Flex ASM feature, the DB instance can remotely connect to ASM instance hence the need of a password file in order to authenticate remote DB instances to ASM. Oracle Flex ASM has the ability of storing password files in a Disk Group and is extended to DB clients; this is really useful to avoid synchronizing multiple password files within the cluster.
Oracle Flex ASM is one of the most important enhancements to ASM in 12c version, however in the below whitepaper describes the rest of new features present.
Bertrand Drouvot has already tested this Oracle Flex ASM feature and uploaded the contents into his own blog (http://bdrouvot.wordpress.com/2013/06/29/build-your-own-flex-asm-12c-lab-using-virtual-box/).
Another interesting feature is the Extent Reading Selection Enhancement which evenly distributes the selection of which copy of a block is read in ASM Disk Groups with normal or high redundancy. This feature is enabled by default in 12c and states that users on I/O bound systems should notice a performance improvement while reading blocks from disks.
Thanks,
Alfredo

Why my new Disk Group is not getting mounted automatically when ASM instance starts?

Today I faced the issue that a new ASM Disk Group created in instance +ASM1 is not getting automatically mounted in instance +ASM2.

The first step was to check DB dependencies:

[oracle@ol5-112-rac2 trace]$ srvctl config database -d RAC
Database unique name: RAC
Database name: RAC
Oracle home: /u01/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/RAC/spfileRAC.ora
Domain: localdomain
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: RAC
Database instances: RAC1,RAC2
Disk Groups: DATA
Services: reports
Database is administrator managed
You can see that only dependent Disk Groups are ‘DATA’.

Let us add ‘FRA’ also as dependent Disk Group:

[oracle@ol5-112-rac2 trace]$ srvctl modify database -d RAC -a ‘DATA,FRA’
[oracle@ol5-112-rac2 trace]$ srvctl config database -d RAC
Database unique name: RAC
Database name: RAC
Oracle home: /u01/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/RAC/spfileRAC.ora
Domain: localdomain
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: RAC
Database instances: RAC1,RAC2
Disk Groups: DATA,FRA
Services: reports
Database is administrator managed
The next step is to add the Disk Groups to ASM_DISKGROUPS parameter:

SQL> ALTER SYSTEM SET ASM_DISKGROUPS=’DATA’,’FRA’ scope=both sid=’+ASM1′;
System altered.
SQL> ALTER SYSTEM SET ASM_DISKGROUPS=’DATA’,’FRA’ scope=both sid=’+ASM2′;
System altered.
After restarting +ASM2 instance we can see that FRA Disk Group is mounted automatically:

[oracle@ol5-112-rac2 trace]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 12 15:44:44 2013
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set lines 1000
SQL> select * from V$ASM_DISKGROUP;
GROUP_NUMBER NAME                           SECTOR_SIZE BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE       TYPE     TOTAL_MB    FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS COMPATIBILITY                                                DATABASE_COMPATIBILITY                                       V
———— —————————— ———– ———- ——————– ———– —— ———- ———- ———– ———— ———————– ————– ————- ———————————————————— ———————————————————— –
           1 DATA                                   512       4096              1048576 MOUNTED     EXTERN      15342      12616           0         2726                       0          12616     0 11.2.0.0.0                                                   10.1.0.0.0                                                   N
           2 FRA                                    512       4096              1048576 MOUNTED     EXTERN       6133       5962           0          171                       0           5962     0 11.2.0.0.0                                                   10.1.0.0.0                                                   N
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – Production
With the Real Application Clusters and Automatic Storage Management options
Thanks,

Alfredo

ORA-15097: cannot SHUTDOWN ASM instance with connected client

In 11g R2 Clusterware configurations the OCR is located in ASM disk, that’s why ASM can’t be shutdown while the cluster is running.  You have to shutdown your cluster in order to shutdown ASM instance.

Oracle support note 984663.1

Here’s a good blog about this:
Example, how to shutdown ASM instance on node ol5-112-rac1:
[oracle@ol5-112-rac1 ~]$ crsctl status resource -t
——————————————————————————–
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
——————————————————————————–
Local Resources
——————————————————————————–
ora.DATA.dg
               ONLINE  ONLINE       ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2
ora.FRA.dg
               OFFLINE OFFLINE      ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2
ora.LISTENER.lsnr
               ONLINE  ONLINE       ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2
ora.asm
               ONLINE  ONLINE       ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2             Started
ora.eons
               ONLINE  ONLINE       ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2
ora.gsd
               OFFLINE OFFLINE      ol5-112-rac1
               OFFLINE OFFLINE      ol5-112-rac2
ora.net1.network
               ONLINE  ONLINE       ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2
ora.ons
               ONLINE  ONLINE       ol5-112-rac1
               ONLINE  ONLINE       ol5-112-rac2
——————————————————————————–
Cluster Resources
——————————————————————————–
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       ol5-112-rac1
ora.oc4j
      1        OFFLINE OFFLINE
ora.ol5-112-rac1.vip
      1        ONLINE  ONLINE       ol5-112-rac1
ora.ol5-112-rac2.vip
      1        ONLINE  ONLINE       ol5-112-rac2
ora.rac.db
      1        ONLINE  ONLINE       ol5-112-rac2             Open
      2        ONLINE  ONLINE       ol5-112-rac1
ora.rac.reports.svc
      1        ONLINE  ONLINE       ol5-112-rac1
ora.scan1.vip
      1        ONLINE  ONLINE       ol5-112-rac1
[oracle@ol5-112-rac1 ~]$
Shutdown RAC1 instance first:

[oracle@ol5-112-rac1 ~]$ srvctl stop instance –d RAC –I RAC1 –o immediate
Then proceed to shutdown cluster (root required):

[oracle@ol5-112-rac1 ~]$ su –
[root@ol5-112-rac1 bin]#
[root@ol5-112-rac1 bin]# ./crsctl stop cluster
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘ol5-112-rac1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘ol5-112-rac1’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘ol5-112-rac1’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac1’ succeeded
CRS-2672: Attempting to start ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac2’
CRS-2676: Start of ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac2’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.asm’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘ol5-112-rac1’
CRS-2673: Attempting to stop ‘ora.eons’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.ons’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.net1.network’ on ‘ol5-112-rac1’ succeeded
CRS-2677: Stop of ‘ora.eons’ on ‘ol5-112-rac1’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘ol5-112-rac1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssdmonitor’ on ‘ol5-112-rac1’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘ol5-112-rac1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘ol5-112-rac1’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.cssdmonitor’ on ‘ol5-112-rac1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘ol5-112-rac1’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘ol5-112-rac1’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.cssd’ on ‘ol5-112-rac1’ succeeded
CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘ol5-112-rac1’
CRS-2677: Stop of ‘ora.diskmon’ on ‘ol5-112-rac1’ succeeded
[root@ol5-112-rac1 bin]# ps -ef|grep pmon
root      7072  5722  0 14:45 pts/2    00:00:00 grep pmon
[root@ol5-112-rac1 bin]# ps -ef|grep +ASM1
root      7074  5722  0 14:46 pts/2    00:00:00 grep +ASM1
[root@ol5-112-rac1 bin]#
At this point ASM & Cluster Ready Services managed resources are down.

In order to start them again just need to issue:

[root@ol5-112-rac1 bin]# ./crsctl start cluster
[root@ol5-112-rac1 bin]# exit
[oracle@ol5-112-rac1 ~]$ srvctl start instance –d RAC –I RAC1

Thanks,
Alfredo