Blog Feed
ORA-15097: cannot SHUTDOWN ASM instance with connected client
Hands on practice ASM
Today I want to share with you an ASM exercise which includes how to add an additional disk to our VM RAC nodes, prepare the disk, add the disk to ASM configuration and create a new diskgroup using this disk.
The very first step is to have our VM machines turned off. Then go into your VM host server’s terminal and get into ASM shared disks location.
alfredo@alfredo-N56VM:~$ cd /u04
alfredo@alfredo-N56VM:/u04$ ls
VirtualBox
alfredo@alfredo-N56VM:/u04$ cd VirtualBox/
alfredo@alfredo-N56VM:/u04/VirtualBox$ ls
ol5-112-rac
alfredo@alfredo-N56VM:/u04/VirtualBox$ cd ol5-112-rac/
alfredo@alfredo-N56VM:/u04/VirtualBox/ol5-112-rac$ ls
asm1.vdi asm2.vdi asm3.vdi asm4.vdi
Then create the new VM shared disk and assign it to your VM machines.
alfredo@alfredo-N56VM:/u04/VirtualBox/ol5-112-rac$ VBoxManage createhd --filename asm5.vdi --size 1024 --format VDI --variant Fixed
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: f317a4fb-d132-49fa-8572-76ca7a509e04
alfredo@alfredo-N56VM:/u04/VirtualBox/ol5-112-rac$ VBoxManage storageattach ol5-112-rac1 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable
alfredo@alfredo-N56VM:/u04/VirtualBox/ol5-112-rac$ VBoxManage modifyhd asm5.vdi --type shareable
alfredo@alfredo-N56VM:/u04/VirtualBox/ol5-112-rac$ VBoxManage storageattach ol5-112-rac2 --storagectl "SATA" --port 5 --device 0 --type hdd --medium asm5.vdi --mtype shareable
alfredo@alfredo-N56VM:/u04/VirtualBox/ol5-112-rac$ VBoxManage modifyhd asm5.vdi --type shareable
Now turn your on your node 1, open a terminal as root and format the disk.
[oracle@ol5-112-rac1 ~]$ su -
Password:
[root@ol5-112-rac1 ~]# cd /dev
[root@ol5-112-rac1 dev]# ls sd*
sda sda1 sda2 sdb sdb1 sdc sdc1 sdd sdd1 sde sde1 sdf sdf1 sdg sdg1 sdh
[root@ol5-112-rac1 dev]# fdisk /dev/sdh
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-130, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-130, default 130):
Using default value 130
Command (m for help): p
Disk /dev/sdh: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdh1 1 130 1044193+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Turn on the second VM node.
The next step is to add the newly partitioned disk to ASM.
[root@ol5-112-rac1 dev]# /usr/sbin/oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
[root@ol5-112-rac1 dev]# /usr/sbin/oracleasm createdisk DISK5 /dev/sdh1
Writing disk header: done
Instantiating disk: done
[root@ol5-112-rac1 dev]# /usr/sbin/oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5
[root@ol5-112-rac1 dev]# exit
At this point the new disk is already added to ASM configuration; let’s login to ASM instance and confirm that new disk is present.
[oracle@ol5-112-rac2 ~]$ . oraenv
ORACLE_SID = [RAC2] ? +ASM2
The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/oracle
[oracle@ol5-112-rac2 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ol5-112-rac1
ONLINE ONLINE ol5-112-rac2
ora.LISTENER.lsnr
ONLINE ONLINE ol5-112-rac1
ONLINE ONLINE ol5-112-rac2
ora.asm
ONLINE ONLINE ol5-112-rac1
ONLINE ONLINE ol5-112-rac2 Started
ora.eons
ONLINE ONLINE ol5-112-rac1
ONLINE ONLINE ol5-112-rac2
ora.gsd
OFFLINE OFFLINE ol5-112-rac1
OFFLINE OFFLINE ol5-112-rac2
ora.net1.network
ONLINE ONLINE ol5-112-rac1
ONLINE ONLINE ol5-112-rac2
ora.ons
ONLINE ONLINE ol5-112-rac1
ONLINE ONLINE ol5-112-rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE ol5-112-rac1
ora.oc4j
1 OFFLINE OFFLINE
ora.ol5-112-rac1.vip
1 ONLINE ONLINE ol5-112-rac1
ora.ol5-112-rac2.vip
1 ONLINE ONLINE ol5-112-rac2
ora.rac.db
1 ONLINE ONLINE ol5-112-rac2 Open
2 ONLINE ONLINE ol5-112-rac1
ora.rac.reports.svc
1 ONLINE ONLINE ol5-112-rac1
ora.scan1.vip
1 ONLINE ONLINE ol5-112-rac1
[oracle@ol5-112-rac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Sun Jun 9 16:06:27 2013
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production
With the Real Application Clusters and Automatic Storage Management options
SQL> set lines 1000
SQL> column path format a50
SQL> SELECT PATH,NAME,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,REDUNDANCY,OS_MB,TOTAL_MB,FREE_MB FROM V$ASM_DISK;
PATH NAME GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE REDUNDA OS_MB TOTAL_MB FREE_MB
-------------------------------------------------- ------------------------------ ------------ ----------- ------- ------------ ------- -------- ------- ---------- ---------- ----------
/dev/oracleasm/disks/DISK5 0 0 CLOSED PROVISIONED ONLINE NORMAL UNKNOWN 1019 0 0
/dev/oracleasm/disks/DISK4 DATA_0003 1 3 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4439
/dev/oracleasm/disks/DISK3 DATA_0002 1 2 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4439
/dev/oracleasm/disks/DISK2 DATA_0001 1 1 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4444
/dev/oracleasm/disks/DISK1 DATA_0000 1 0 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4406
You can see the DISK5 present in the ASM system view, the HEADER_STATUS is ‘PROVISIONED’. The PROVISIONED
status implies that an additional platform-specific action has been taken by an administrator to make the disk available for ASM.
Now, let’s drop DISK4 from DATA diskgroup, this will free DISK4 for our new configuration.
SQL> select * from V$ASM_DISKGROUP;
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ------------------------------ ----------- ---------- -------------------- ----------- ------ ---------- ---------- ----------- ------------ ----------------------- -------------- ------------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 DATA 512 4096 1048576 MOUNTED EXTERN 20456 17728 0 2728 0 17728 0 11.2.0.0.0 10.1.0.0.0 N
SQL> ALTER DISKGROUP DATA DROP DISK DATA_0003;
Diskgroup altered.
SQL> SELECT PATH,NAME,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,REDUNDANCY,OS_MB,TOTAL_MB,FREE_MB FROM V$ASM_DISK;
PATH NAME GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE REDUNDA OS_MB TOTAL_MB FREE_MB
-------------------------------------------------- ------------------------------ ------------ ----------- ------- ------------ ------- -------- ------- ---------- ---------- ----------
/dev/oracleasm/disks/DISK5 0 0 CLOSED PROVISIONED ONLINE NORMAL UNKNOWN 1019 0 0
/dev/oracleasm/disks/DISK4 DATA_0003 1 3 CACHED MEMBER ONLINE DROPPING UNKNOWN 5114 5114 4546
/dev/oracleasm/disks/DISK3 DATA_0002 1 2 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4404
/dev/oracleasm/disks/DISK2 DATA_0001 1 1 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4409
/dev/oracleasm/disks/DISK1 DATA_0000 1 0 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4369
You can see the STATE of DISK4 is DROPPING, in this stage ASM will rebalance all the data from DISK4 into the disk of diskgroup DATA. In order to speed this operation we are going to modify the rebalance power of the diskgroup.
SQL> ALTER DISKGROUP DATA REBALANCE POWER 5;
Diskgroup altered.
SQL> SELECT PATH,NAME,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,REDUNDANCY,OS_MB,TOTAL_MB,FREE_MB FROM V$ASM_DISK;
PATH NAME GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE REDUNDA OS_MB TOTAL_MB FREE_MB
-------------------------------------------------- ------------------------------ ------------ ----------- ------- ------------ ------- -------- ------- ---------- ---------- ----------
/dev/oracleasm/disks/DISK5 0 0 CLOSED PROVISIONED ONLINE NORMAL UNKNOWN 1019 0 0
/dev/oracleasm/disks/DISK4 0 1 CLOSED FORMER ONLINE NORMAL UNKNOWN 5114 0 0
/dev/oracleasm/disks/DISK3 DATA_0002 1 2 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4213
/dev/oracleasm/disks/DISK2 DATA_0001 1 1 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4218
/dev/oracleasm/disks/DISK1 DATA_0000 1 0 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4185
SQL> show parameter power
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
asm_power_limit integer 1
SQL> ALTER DISKGROUP DATA REBALANCE POWER 1;
Diskgroup altered.
After modifying the rebalance power to 5 ASM will provide more resources to the rebalance process, this results in faster rebalance times. You can see that DISK4’s header status is now ‘FORMER’ this means a disk was once part of a disk group but has been dropped cleanly from the group. It may be added to a new disk group with the ALTER DISKGROUP
statement.
Be sure to modify back the power of the disk group DATA to 1.
Now let’s create a new disk group named FRA using DISK4 & DISK5.
SQL> CREATE DISKGROUP FRA EXTERNAL REDUNDANCY DISK '/dev/oracleasm/disks/DISK4' NAME FRA_0000, '/dev/oracleasm/disks/DISK5' NAME FRA_0001;
Diskgroup created.
SQL> SELECT PATH,NAME,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,REDUNDANCY,OS_MB,TOTAL_MB,FREE_MB FROM V$ASM_DISK;
PATH NAME GROUP_NUMBER DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE REDUNDA OS_MB TOTAL_MB FREE_MB
-------------------------------------------------- ------------------------------ ------------ ----------- ------- ------------ ------- -------- ------- ---------- ---------- ----------
/dev/oracleasm/disks/DISK4 FRA_0000 2 0 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 5072
/dev/oracleasm/disks/DISK3 DATA_0002 1 2 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4215
/dev/oracleasm/disks/DISK2 DATA_0001 1 1 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4218
/dev/oracleasm/disks/DISK1 DATA_0000 1 0 CACHED MEMBER ONLINE NORMAL UNKNOWN 5114 5114 4183
/dev/oracleasm/disks/DISK5 FRA_0001 2 1 CACHED MEMBER ONLINE NORMAL UNKNOWN 1019 1019 1009
SQL> select * from V$ASM_DISKGROUP;
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ------------------------------ ----------- ---------- -------------------- ----------- ------ ---------- ---------- ----------- ------------ ----------------------- -------------- ------------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 DATA 512 4096 1048576 MOUNTED EXTERN 15342 12616 0 2726 0 12616 0 11.2.0.0.0 10.1.0.0.0 N
2 FRA 512 4096 1048576 MOUNTED EXTERN 6133 6081 0 52 0 6081 0 10.1.0.0.0 10.1.0.0.0 N
SQL>
ALTER DISKGROUP FRA SET ATTRIBUTE 'compatible.asm'='11.2.0.0.0';
SQL> ALTER DISKGROUP FRA SET ATTRIBUTE 'compatible.asm'='11.2.0.0.0';
Diskgroup altered.
SQL> select * from V$ASM_DISKGROUP;
GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS COMPATIBILITY DATABASE_COMPATIBILITY V
------------ ------------------------------ ----------- ---------- -------------------- ----------- ------ ---------- ---------- ----------- ------------ ----------------------- -------------- ------------- ------------------------------------------------------------ ------------------------------------------------------------ -
1 DATA 512 4096 1048576 MOUNTED EXTERN 15342 12616 0 2726 0 12616 0 11.2.0.0.0 10.1.0.0.0 N
2 FRA 512 4096 1048576 MOUNTED EXTERN 6133 6079 0 54 0 6079 0 11.2.0.0.0 10.1.0.0.0 N
SQL>
FRA disk group is ready to host our flash recovery area for archivelogs and backups.
This is a small but useful example of the ASM functionality.
Thanks,
Alfredo
GoldenGate 11G Install on Linux x86
Why the execution plan changed? Case study
CRSCTL command examples
$ crs_stat -t
Name Type Target State Host
————————————————————
ora.DATA.dg ora….up.type ONLINE ONLINE ol5-…rac1
ora….ER.lsnr ora….er.type ONLINE ONLINE ol5-…rac1
ora….N1.lsnr ora….er.type ONLINE ONLINE ol5-…rac1
ora.asm ora.asm.type ONLINE ONLINE ol5-…rac1
ora.eons ora.eons.type ONLINE ONLINE ol5-…rac1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora….network ora….rk.type ONLINE ONLINE ol5-…rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora….SM1.asm application ONLINE ONLINE ol5-…rac1
ora….C1.lsnr application ONLINE ONLINE ol5-…rac1
ora….ac1.gsd application OFFLINE OFFLINE
ora….ac1.ons application ONLINE ONLINE ol5-…rac1
ora….ac1.vip ora….t1.type ONLINE ONLINE ol5-…rac1
ora….SM2.asm application ONLINE ONLINE ol5-…rac2
ora….C2.lsnr application ONLINE ONLINE ol5-…rac2
ora….ac2.gsd application OFFLINE OFFLINE
ora….ac2.ons application ONLINE ONLINE ol5-…rac2
ora….ac2.vip ora….t1.type ONLINE ONLINE ol5-…rac2
ora.ons ora.ons.type ONLINE ONLINE ol5-…rac1
ora.rac.db ora….se.type ONLINE ONLINE ol5-…rac2
ora….rts.svc ora….ce.type ONLINE ONLINE ol5-…rac1
ora.scan1.vip ora….ip.type ONLINE ONLINE ol5-…rac1
Basic Clusterware Administration Commands
In the previous blog we talk about Oracle Clusterware basics and definitions. Now let’s talk about basic administration commands.
- olsnodes
The OLSNODE
S
command provides the list of nodes and other information for all nodes participating in the cluster. The syntax for the OLSNODES
command is:
OLSNODES
command without any command parameters, the command returns a listing of the nodes in the cluster:# olsnodes
node1
node2
node3
node4
OLSNODES
command to obtain additional cluster-related information.Option | Description |
---|---|
-g |
Logs cluster verification information with more details.
|
-i |
Lists all nodes participating in the cluster and includes the Virtual Internet Protocol (VIP) address assigned to each node.
|
-l |
Displays the local node name.
|
-n |
Lists al nodes participating in the cluster and includes the assigned node numbers.
|
-p |
Lists all nodes participating in the cluster and includes the private interconnect assigned to each node.
|
-v |
Logs cluster verification information in verbose mode.
|
- ocrcheck
OCRCHECK utility is used to verify the OCR integrity. The OCRCHECK utility displays the version of the OCR’s block format, total space available and used space, OCRID, and the OCR locations that you have configured. OCRCHECK performs a block-by-block checksum operation for all of the blocks in all of the OCRs that you have configured. It also returns an individual status for each file as well as result for the overall OCR integrity check. The following is a sample of the OCRCHECK output:
# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2528
Available space (kbytes) : 259592
ID : 1197072699
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
Note: Run this command as root in order to enable the Cluster registry integrity check, if you run this command with a non-privileged user this will be bypassed.
- ocrcheck -local
In Oracle Clusterware 11g release 2 (11.2), each node in a cluster has a local registry for node-specific resources, called an Oracle Local Registry (OLR), that is installed and configured when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, regardless of whether Oracle Clusterware is running or fully functional.
By default, OLR is located at Grid_home/cdata/host_name.olr on each node. Manage OLR using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities as root with the -local option.
# ./ocrcheck -local
Status of Oracle Local Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2184
Available space (kbytes) : 259936
ID : 1563540278
Device/File Name : /u01/app/11.2.0/grid/cdata/ol5-112-rac1.olr
Device/File integrity check succeeded
Local registry integrity check succeeded
Logical corruption check succeeded
- crsctl
CRSCTL is an interface between you and Oracle Clusterware, parsing and calling Oracle Clusterware APIs for Oracle Clusterware objects.
Oracle Clusterware 11g release 2 (11.2) introduces cluster-aware commands with which you can perform check, start, and stop operations on the cluster. You can run these commands from any node in the cluster on another node in the cluster, or on all nodes in the cluster, depending on the operation.
CRSCTL documentation is quite extensive and can be found in the below link:
(http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crsref.htm)
CRSCTL can be also used to check the state of the Voting disks:
# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 911b058da40b4f96bf456abb771dddc8 (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).
Thanks,
Alfred
- ocrcheck -local
In Oracle Clusterware 11g release 2 (11.2), each node in a cluster has a local registry for node-specific resources, called an Oracle Local Registry (OLR), that is installed and configured when Oracle Clusterware installs OCR. Multiple processes on each node have simultaneous read and write access to the OLR particular to the node on which they reside, regardless of whether Oracle Clusterware is running or fully functional.
By default, OLR is located at Grid_home/cdata/host_name.olr on each node. Manage OLR using the OCRCHECK, OCRDUMP, and OCRCONFIG utilities as root with the -local option.
# ./ocrcheck -local
Status of Oracle Local Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2184
Available space (kbytes) : 259936
ID : 1563540278
Device/File Name : /u01/app/11.2.0/grid/cdata/ol5-112-rac1.olr
Device/File integrity check succeeded
Local registry integrity check succeeded
Logical corruption check succeeded
- crsctl
CRSCTL is an interface between you and Oracle Clusterware, parsing and calling Oracle Clusterware APIs for Oracle Clusterware objects.
Oracle Clusterware 11g release 2 (11.2) introduces cluster-aware commands with which you can perform check, start, and stop operations on the cluster. You can run these commands from any node in the cluster on another node in the cluster, or on all nodes in the cluster, depending on the operation.
CRSCTL documentation is quite extensive and can be found in the below link:
(http://docs.oracle.com/cd/E11882_01/rac.112/e16794/crsref.htm)
CRSCTL can be also used to check the state of the Voting disks:
# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 911b058da40b4f96bf456abb771dddc8 (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).
Thanks,
Alfred
Oracle Clusterware 11gR2 Basics
In this blog will talk about Oracle Clusterware basics. It is really needed to know the basics before we can start talking about Clusterware installation, configuration and administration. The below information is an extract from the Oracle Clusterware, Administration and Deployment Guide 11g Release 2 (11.2) (http://docs.oracle.com/cd/E14072_01/rac.112/e10717.pdf).
What is Oracle Clusterware?
Oracle Clusterware enables servers to communicate with each other, so that they
appear to function as a collective unit. This combination of servers is commonly
known as a cluster. Although the servers are standalone servers, each server has
additional processes that communicate with other servers. In this way the separate
servers appear as if they are one system to applications and end users.
Oracle Clusterware provides the infrastructure necessary to run Oracle Real
Application Clusters (Oracle RAC). Oracle Clusterware also manages resources, such
as virtual IP (VIP) addresses, databases, listeners, services, and so on. These resources
are generally named ora.resource_name.host_name. Oracle does not support
editing these resources except under the explicit direction of Oracle support.
Additionally, Oracle Clusterware can help you manage your applications.
Oracle Clusterware has two stored components, besides the binaries: The voting disk
files, which record node membership information, and the Oracle Cluster Registry
(OCR), which records cluster configuration information. Voting disks and OCRs must
reside on shared storage available to all cluster member nodes.
Oracle Clusterware uses voting disk files to provide fencing and cluster node
membership determination. The OCR provides cluster configuration information. You can place the Oracle Clusterware files on either Oracle ASM or on shared common
disk storage. If you configure Oracle Clusterware on storage that does not provide file
redundancy, then Oracle recommends that you configure multiple locations for OCR
and voting disks. The voting disks and OCR are described as follows:
■ Voting Disks
Oracle Clusterware uses voting disk files to determine which nodes are members
of a cluster. You can configure voting disks on Oracle ASM, or you can configure
voting disks on shared storage.
If you configure voting disks on Oracle ASM, then you do not need to manually
configure the voting disks. Depending on the redundancy of your disk group, an
appropriate number of voting disks are created.
If you do not configure voting disks on Oracle ASM, then for high availability,
Oracle recommends that you have a minimum of three voting disks on physically
separate storage. This avoids having a single point of failure. If you configure a
single voting disk, then you must use external mirroring to provide redundancy.
You should have at least three voting disks, unless you have a storage device, such
as a disk array that provides external redundancy. Oracle recommends that you do
not use more than five voting disks. The maximum number of voting disks that is
supported is 15.
■ Oracle Cluster Registry
Oracle Clusterware uses the Oracle Cluster Registry (OCR) to store and manage
information about the components that Oracle Clusterware controls, such as
Oracle RAC databases, listeners, virtual IP addresses (VIPs), and services and any
applications. The OCR stores configuration information in a series of key-value
pairs in a tree structure. To ensure cluster high availability, Oracle recommends
that you define multiple OCR locations (multiplex). In addition:
– You can have up to five OCR locations
– Each OCR location must reside on shared storage that is accessible by all of the
nodes in the cluster
– You can replace a failed OCR location online if it is not the only OCR location
– You must update the OCR through supported utilities such as Oracle
Enterprise Manager, the Server Control Utility (SRVCTL), the OCR
configuration utility (OCRCONFIG), or the Database Configuration Assistant
(DBCA)
Oracle Clusterware Processes on Linux and UNIX Systems
Oracle Clusterware processes on Linux and UNIX systems include the following:
■ crsd: Performs high availability recovery and management operations such as
maintaining the OCR and managing application resources. This grid infrastructure
process runs as root and restarts automatically upon failure.
When you install Oracle Clusterware in a single-instance database environment
for Oracle ASM and Oracle Restart, ohasd manages application resources and
crsd is not used.
■ cssdagent: Starts, stops, and checks the status of the CSS daemon, ocssd. In
addition, the cssdagent and cssdmonitor provide the following services to
guarantee data integrity:
– Monitors the CSS daemon; if the CSS daemon stops, then it shuts down the
node
– Monitors the node scheduling to verify that the node is not hung, and shuts
down the node on recovery from a hang.
oclskd: (Oracle Clusterware Kill Daemon) CSS uses this daemon to stop
processes associated with CSS group members for which stop requests have come
in from other members on remote nodes.
■ ctssd: Cluster time synchronization service daemon: Synchronizes the time on all
of the nodes in a cluster to match the time setting on the master node but not to an
external clock.
■ diskmon (Disk Monitor daemon): Monitors and performs I/O fencing for HP
Oracle Exadata Storage Server storage. Because Exadata storage can be added to
any Oracle RAC node at any time, the diskmon daemon is always started when
ocssd starts.
■ evmd (Event manager daemon): Distributes and communicates some cluster
events to all of the cluster members so that they are aware of changes in the
cluster.
evmlogger (Event manager logger): This is started by EVMD at startup. This
reads a configuration file to determine what events to subscribe to from EVMD
and it runs user defined actions for those events. This facility maintains backward
compatibility only.
■ gpnpd (Grid Plug and Play daemon): Manages distribution and maintenance of
the Grid Plug and Play profile containing cluster definition data.
■ mdnsd (Multicast Domain Name Service daemon): Manages name resolution and
service discovery within attached subnets.
■ ocssd (Cluster Synchronization Service daemon): Manages cluster node
membership and runs as the oracle user; failure of this process results in a node
restart.
■ ohasd (Oracle High Availability Services daemon): Starts Oracle Clusterware
processes and also manages the OLR and acts as the OLR server.
In a cluster, ohasd runs as root. However, in an Oracle Restart environment,
where ohasd manages application resources, it runs as the oracle user
Next blog -> Basic Clusterware Administration Commands
Thanks,
Alfred
Cleanup After Failed Installation Oracle Clusterware 11gR2
In this blog I will talk about how to cleanup a failed installation of the Oracle Clusterware 11gR2 in one particular node. I was installing the new Oracle Clusterware 11.2.0.1 in my home lab by follwing Tom Kyte’s instructions (http://www.oracle-base.com/articles/11g/oracle-db-11gr2-rac-installation-on-oel5-using-virtualbox.php#install_grid_infrastructure), then went through the part that the installer GUI needs us to run the root.sh scripts.
The easy thought was to run the root.sh script in both nodes of my RAC (RAC1 & RAC2) at the same time, without reading the explicit instructions “Run the script on the local node first. After successful completion, you can run the script in parallel on all other nodes.”:
The script run successful in the RAC1 node but in the RAC2 node. So when the script finished in RAC1 tried to run it again in RAC2, but the same results “the output was that the script was already ran”.
What to do next?, do I have to start over from scratch?
Surfing the web found a good & useful article from Guenadi Jilevski (http://gjilevski.com/2010/08/12/how-to-clean-up-after-a-failed-11g-crs-install-what-is-new-in-11g-r2-2/), here shows how to perform a manual cleanup in 11gR1, but also shows the new features and scripts in 11gR2.
Summarized steps:
Deconfigure Oracle Clusterware 11.2.x.x without removing the binaries:
- Log in as root user on the node you encountered the error. Change directory to $GRID_HOME/crs/install.
- Run rootcrs.pl with the -deconfig -force flags on the node you have the issue.
- If you are deconfiguring Oracle Clusterware on all the nodes in the cluster, then you have to add the -lastnode flag on the last one in order to deconfigure OCR and Voting disks.
Historical SQL Statistics And Execution Plan Change
How many times have you received user’s phone call stating that the DB is slow?, well sometimes this has nothing to do with the DB’s overall performance itself, but with a user’s query.
Now the question is, is this query really slow?, the best way to find this out is comparing the actual execution time with the ones in the past. Here’s a SQL statement which search inside AWR repository (Caution!!! a special license is required!!!), it compares the execution plan of all SQL_ID’s against the previous snapshot.
awr_planchanges.sql
prompt enter the number of days in the past to scan
SET LINES 500
SELECT A.SNAP_ID,
BEGIN_INTERVAL_TIME,
SQL_ID,
HASH_VALUE1,
HASH_VALUE2,
VALUE1 as “ELAPSED TIME PER EXEC 1” ,
VALUE2 as “ELAPSED TIME PER EXEC 2”,
ROUND (CHANGE_PERCENT) as “CHANGE PERCENT”
FROM ( SELECT SNAP_ID,
SQL_ID,
SUM (pvalue1) HASH_VALUE1,
SUM (pvalue2) HASH_VALUE2,
SUM (value1) VALUE1,
SUM (value2) VALUE2,
(SUM (VALUE1) + 1) * 100 / (SUM (VALUE2) + 1)
AS CHANGE_PERCENT
FROM (SELECT snap_id,
0 AS snap2,
sql_id,
plan_hash_value AS pvalue1,
0 AS pvalue2,
ROUND (
elapsed_time_delta / executions_delta / 1000000,
3)
AS value1,
0 AS value2
FROM dba_hist_sqlstat sql
WHERE executions_delta > 0
UNION
SELECT snap_id + 1,
snap_id AS snap2,
sql_id,
0 AS pvalue1,
plan_hash_value AS pvalue2,
0 AS vaule,
ROUND (
elapsed_time_delta / executions_delta / 1000000,
3)
AS value2
FROM dba_hist_sqlstat sql
WHERE executions_delta > 0)
GROUP BY SNAP_ID, SQL_ID) A,
dba_hist_snapshot B
WHERE A.SNAP_ID = B.SNAP_ID
AND HASH_VALUE1 > 0
AND HASH_VALUE2 > 0
AND HASH_VALUE1 != HASH_VALUE2
AND BEGIN_INTERVAL_TIME > SYSDATE – (&days)
ORDER BY A.SNAP_ID
/
This is the output of the script:
SNAP_ID BEGIN_INTERVAL_TIME SQL_ID HASH_VALUE1 HASH_VALUE2 SEC PER EXE 1 SEC PER EXE 2 CHANGE %
——– ———————– ————- ———– ———– ————- ————- ——–
9547 01-OCT-12 01.00.07.841 AM 4urszd9dt9fjv 4862331523 891004645 .015 .007 101
9585 02-OCT-12 03.00.41.798 PM 4urszd9dt9fjv 4862331523 891004645 .009 .003 101
9586 02-OCT-12 04.00.36.393 PM 4urszd9dt9fjv 891004645 4862331523 .006 .009 100
9587 02-OCT-12 05.00.25.306 PM 1k30v0pyg32vu 414828074 878600859 .179 .157 102
9587 02-OCT-12 05.00.25.306 PM dsm86bzuqtd2r 2452407222 3005749068 .048 1.37 44
9616 03-OCT-12 10.00.34.499 PM 4urszd9dt9fjv 4862331523 891004645 .018 .023 100
9621 04-OCT-12 03.00.06.979 AM 4urszd9dt9fjv 891004645 4862331523 .03 .027 100
9640 04-OCT-12 09.00.40.250 PM 4urszd9dt9fjv 4862331523 891004645 .042 .012 103
9641 04-OCT-12 10.00.22.954 PM dfmu8nm1cscx7 3810296266 4308029399 .938 .703 114
9710 07-OCT-12 07.00.09.269 PM 4urszd9dt9fjv 4862331523 891004645 .046 .015 103
9750 09-OCT-12 11.00.59.162 AM 1k30v0pyg32vu 1634868183 414828074 .16 .155 100
9758 09-OCT-12 07.00.10.659 PM b70xavb9wv27v 1111647858 4256287279 5.937 6.475 93
12 rows selected.
You can clearly see how the HASH_VALUE changed and the execution time as well. If the CHANGE % is above 100% means that execution time decreased, on the other hand CHANGE % below 100% means the execution time increased.
ORAganism blog is having a really good script which searches by SQL_ID.
http://oraganism.wordpress.com/2011/12/14/a-dba_hist_sqlstat-query-that-i-am-very-fond-of/
Happy troubleshooting,
Alfred
Same post in Spanish here!