RAC - alfredokriegdba.com

CRS-2632: There are no more servers to try to place resource ‘resource_name’ on that would satisfy its placement policy

I just got this error from CRS while trying to start a database using srvctl. I find this error kind of missleading as the error doesn’t tell what is really happening with the resource.

I suggest you to manually check the logs of the resource. In this case it had to do with a cluster database that one of the instances was not able to start. I tried to start this instance manually and then I got the real error.

The instance was having a missconfiguration in the parameter file that prevented it from starting.

Hope this note will provide you a guidance while having this CRS-2632 error coming out from CRS.

Thanks,

Alfredo

WAIT_FOR_GAP. How to restore missing archivelogs from backup?

In a Dataguard configuration, Oracle’s RFS (Remote File Server) writes redo data to the standby. When for any reason it can’t write this data, MRP (Managed Recovery Process) will wait for the archivelog to be applied and have the status “WAIT_FOR_LOG”. This will lead the standby to be out-of-sync with the primary database.

Sometimes some archivelogs can’t be transferred from primary database to the standby leaving a gap in the archivelog sequence. The MRP process will have the status “WAIT_FOR_GAP”.

In order to fix the archivelog gap we have to manually transfer the archivelogs missing.

To find the gap you can query v$archive_gap (gv$archive_gap for RAC).

SELECT INST_ID, THREAD#, HIGH_SEQUENCE#, LOW_SEQUENCE# FROM GV$ARCHIVE_GAP;

INST_ID THREAD# HIGH_SEQUENCE# LOW_SEQUENCE#

————- ————— ———————— ————————

2 2 823 811

You can see that we are missing archivelogs from sequence 811 to 823 for thread 2. If these archivelogs are not available in the primary we have to restore them from backup.

RMAN> RESTORE ARCHIVELOG FROM SEQUENCE 811 UNTIL SEQUENCE 823 THREAD=2;

Keep in mind that parameter THREAD defaults to 1, so you must specify the thread number when you are trying to restore from a different thread.

After restoring these archivelogs the RFS process should transfer them automatically to the standby. Verify if the gap is fixed.

Thanks,

Alfredo

ALTER SYSTEM in a RAC environment

Few weeks ago I had to modify an instance parameter to a different value. In this case, I just wanted to modify it for only one instance and leave the other instance as it was.

I thought that if I didn’t specify the instance name using the SID clause, the default is going to be just that instance. I was really wrong about that and the command ended modifying both instances.

After looking at Oracle’s documentation here’s what I’ve found.

SID Clause

· Specify SID = ‘*’ if you want Oracle Database to change the value of the parameter for all instances.

· Specify SID = ‘sid’ if you want Oracle Database to change the value of the parameter only for the instance sid. This setting takes precedence over previous and subsequent ALTER SYSTEM SET statements that specify SID = ‘*’.

If you do not specify this clause:

· If the instance was started up with a pfile (client-side initialization parameter file), then Oracle Database assumes the SID of the current instance.

· If the instance was started up with an spfile (server parameter file), then Oracle Database assumes SID = ‘*’.

If you specify an instance other than the current instance, then Oracle Database sends a message to that instance to change the parameter value in the memory of that instance.

So now I know that in future I need to specify the SID in a RAC instance using spfile.

https://docs.oracle.com/cd/B12037_01/server.101/b10759/statements_2013.htm

Thanks,

Alfredo

Why my new Disk Group is not getting mounted automatically when ASM instance starts?

Today I faced the issue that a new ASM Disk Group created in instance +ASM1 is not getting automatically mounted in instance +ASM2.

The first step was to check DB dependencies:

[oracle@ol5-112-rac2 trace]$ srvctl config database -d RAC

Database unique name: RAC

Database name: RAC

Oracle home: /u01/app/oracle/product/11.2.0/db_1

Oracle user: oracle

Spfile: +DATA/RAC/spfileRAC.ora

Domain: localdomain

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: RAC

Database instances: RAC1,RAC2

Disk Groups: DATA

Services: reports

Database is administrator managed

You can see that only dependent Disk Groups are ‘DATA’.

Let us add ‘FRA’ also as dependent Disk Group:

[oracle@ol5-112-rac2 trace]$ srvctl modify database -d RAC -a ‘DATA,FRA’

[oracle@ol5-112-rac2 trace]$ srvctl config database -d RAC

Database unique name: RAC

Database name: RAC

Oracle home: /u01/app/oracle/product/11.2.0/db_1

Oracle user: oracle

Spfile: +DATA/RAC/spfileRAC.ora

Domain: localdomain

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: RAC

Database instances: RAC1,RAC2

Disk Groups: DATA,FRA

Services: reports

Database is administrator managed

The next step is to add the Disk Groups to ASM_DISKGROUPS parameter:

SQL> ALTER SYSTEM SET ASM_DISKGROUPS=’DATA’,’FRA’ scope=both sid=’+ASM1′;

System altered.

SQL> ALTER SYSTEM SET ASM_DISKGROUPS=’DATA’,’FRA’ scope=both sid=’+ASM2′;

System altered.

After restarting +ASM2 instance we can see that FRA Disk Group is mounted automatically:

[oracle@ol5-112-rac2 trace]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 12 15:44:44 2013

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – Production

With the Real Application Clusters and Automatic Storage Management options

SQL> set lines 1000

SQL> select * from V$ASM_DISKGROUP;

GROUP_NUMBER NAME SECTOR_SIZE BLOCK_SIZE ALLOCATION_UNIT_SIZE STATE TYPE TOTAL_MB FREE_MB HOT_USED_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS COMPATIBILITY DATABASE_COMPATIBILITY V

———— —————————— ———– ———- ——————– ———– —— ———- ———- ———– ———— ———————– ————– ————- ———————————————————— ———————————————————— –

1 DATA 512 4096 1048576 MOUNTED EXTERN 15342 12616 0 2726 0 12616 0 11.2.0.0.0 10.1.0.0.0 N

2 FRA 512 4096 1048576 MOUNTED EXTERN 6133 5962 0 171 0 5962 0 11.2.0.0.0 10.1.0.0.0 N

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – Production

With the Real Application Clusters and Automatic Storage Management options

Thanks,

Alfredo

ORA-15097: cannot SHUTDOWN ASM instance with connected client

In 11g R2 Clusterware configurations the OCR is located in ASM disk, that’s why ASM can’t be shutdown while the cluster is running. You have to shutdown your cluster in order to shutdown ASM instance.

Oracle support note 984663.1

Here’s a good blog about this:

http://divakarmehta.wordpress.com/2012/04/19/how-crs-accesses-ocr-in-oracle-asm-during-11-2-grid-infra-rac-startup/

Example, how to shutdown ASM instance on node ol5-112-rac1:

[oracle@ol5-112-rac1 ~]$ crsctl status resource -t

——————————————————————————–

NAME TARGET STATE SERVER STATE_DETAILS

——————————————————————————–

Local Resources

——————————————————————————–

ora.DATA.dg

ONLINE ONLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2

ora.FRA.dg

OFFLINE OFFLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2

ora.LISTENER.lsnr

ONLINE ONLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2

ora.asm

ONLINE ONLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2 Started

ora.eons

ONLINE ONLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2

ora.gsd

OFFLINE OFFLINE ol5-112-rac1

OFFLINE OFFLINE ol5-112-rac2

ora.net1.network

ONLINE ONLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2

ora.ons

ONLINE ONLINE ol5-112-rac1

ONLINE ONLINE ol5-112-rac2

——————————————————————————–

Cluster Resources

——————————————————————————–

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE ol5-112-rac1

ora.oc4j

1 OFFLINE OFFLINE

ora.ol5-112-rac1.vip

1 ONLINE ONLINE ol5-112-rac1

ora.ol5-112-rac2.vip

1 ONLINE ONLINE ol5-112-rac2

ora.rac.db

1 ONLINE ONLINE ol5-112-rac2 Open

2 ONLINE ONLINE ol5-112-rac1

ora.rac.reports.svc

1 ONLINE ONLINE ol5-112-rac1

ora.scan1.vip

1 ONLINE ONLINE ol5-112-rac1

[oracle@ol5-112-rac1 ~]$

Shutdown RAC1 instance first:

[oracle@ol5-112-rac1 ~]$ srvctl stop instance –d RAC –I RAC1 –o immediate

Then proceed to shutdown cluster (root required):

[oracle@ol5-112-rac1 ~]$ su –

[root@ol5-112-rac1 bin]#

[root@ol5-112-rac1 bin]# ./crsctl stop cluster

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘ol5-112-rac1’

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘ol5-112-rac1’

CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘ol5-112-rac1’

CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac1’ succeeded

CRS-2672: Attempting to start ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac2’

CRS-2676: Start of ‘ora.ol5-112-rac1.vip’ on ‘ol5-112-rac2’ succeeded

CRS-2677: Stop of ‘ora.DATA.dg’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.asm’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.asm’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.ons’ on ‘ol5-112-rac1’

CRS-2673: Attempting to stop ‘ora.eons’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.ons’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.net1.network’ on ‘ol5-112-rac1’ succeeded

CRS-2677: Stop of ‘ora.eons’ on ‘ol5-112-rac1’ succeeded

CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘ol5-112-rac1’ has completed

CRS-2677: Stop of ‘ora.crsd’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.cssdmonitor’ on ‘ol5-112-rac1’

CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘ol5-112-rac1’

CRS-2673: Attempting to stop ‘ora.evmd’ on ‘ol5-112-rac1’

CRS-2673: Attempting to stop ‘ora.asm’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.cssdmonitor’ on ‘ol5-112-rac1’ succeeded

CRS-2677: Stop of ‘ora.evmd’ on ‘ol5-112-rac1’ succeeded

CRS-2677: Stop of ‘ora.ctssd’ on ‘ol5-112-rac1’ succeeded

CRS-2677: Stop of ‘ora.asm’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.cssd’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.cssd’ on ‘ol5-112-rac1’ succeeded

CRS-2673: Attempting to stop ‘ora.diskmon’ on ‘ol5-112-rac1’

CRS-2677: Stop of ‘ora.diskmon’ on ‘ol5-112-rac1’ succeeded

[root@ol5-112-rac1 bin]# ps -ef|grep pmon

root 7072 5722 0 14:45 pts/2 00:00:00 grep pmon

[root@ol5-112-rac1 bin]# ps -ef|grep +ASM1

root 7074 5722 0 14:46 pts/2 00:00:00 grep +ASM1

[root@ol5-112-rac1 bin]#

At this point ASM & Cluster Ready Services managed resources are down.

In order to start them again just need to issue:

[root@ol5-112-rac1 bin]# ./crsctl start cluster

[root@ol5-112-rac1 bin]# exit

[oracle@ol5-112-rac1 ~]$ srvctl start instance –d RAC –I RAC1

Thanks,

Alfredo