Blog Feed - alfredokriegdba.com

Oracle Management Cloud – Managing the Oracle Cloud Agent

This post is about some useful commands to administer the OMC’s Cloud agent. The very first step is to install the OMC Cloud agent. You may refer to my previous post on how to install it here.

The omcli command is located at /agent_inst/bin.

I’m skipping these 3 basic commands start, stop and status.

Here we go!

– List the targets currently managed by the Cloud agent

$ ./omcli config agent listtargets

Oracle Management Cloud Agent

[hostname.localdomain:1302, Lama]

[hostname.localdomain, omc_host_linux]

– Get the TimeZone of the server where the Cloud agent is running

$ ./omcli config agent getTZ

Oracle Management Cloud Agent

US/Eastern

– Get a detailed output of the Cloud agent scheduler

$ ./omcli status agent scheduler

Oracle Management Cloud Agent

—————————————————————

Number of currently running scheduled items :: 0

Number of currently ready scheduled items :: 0

Number of scheduled items :: 57

Schedule Attempts Delay Percentage :: 0.000%

Running entries::

Ready entries::

Scheduled entries::

2017-03-31 14:32:58.968 : HeapMonitorTask

…

2017-03-31 22:34:14.667 : PurgeTargetEvents

2017-03-31 22:35:47.885 : PurgeADR

2017-04-01 13:34:20.810 : EventStatusPurge

—————————————————————

Agent is Running and Ready

– Get a summary of the Cloud agent’s scheduler

$ ./omcli status agent scheduler -summary

Oracle Management Cloud Agent

—————————————————————

Number of currently running scheduled items :: 0

Number of currently ready scheduled items :: 0

Number of scheduled items :: 57

Schedule Attempts Delay Percentage :: 0.000%

—————————————————————

Agent is Running and Ready

– Dump the internal state of the agent [option component name]

$ ./omcli dumpstate agent

Oracle Management Cloud Agent

Generated file: /u01/omc/infra/agent_inst/sysman/emd/dumps/SystemDump_1490985620656.xml

– Dump internal information of all components of the Cloud agent

$ ./omcli dumpstate agent -dump_full_state

Oracle Management Cloud Agent

Generated file: /u01/omc/infra/agent_inst/sysman/emd/dumps/SystemDump_1490985775823.xml

– Add entities to the Cloud agent using JSON template

$ ./omcli add_entity agent /home/oracle/omc_weblogic.json

Oracle Management Cloud Agent

Operation Succeeded: Accepted 2 of 2 entities for processing.

– Delete entities from the Cloud agent using JSON template

$ ./omcli delete_entity agent /home/oracle/omc_weblogic.json

Oracle Management Cloud Agent

Request for removing the entities was accepted for processing.

This is just a small taste of all the options built into the Oracle’s Cloud agent. For more information about all the options refer to Oracle’s documentation.

https://docs.oracle.com/en/cloud/paas/management-cloud/gfadg/omcli-command-options.html

Thanks,

Alfredo

Oracle Management Cloud – Install the Cloud Agent

This is my first experience with the Oracle Cloud, in particular with Oracle Management Cloud (OMC). In this first post I will guide you through the installation of the Cloud Agent in a Linux host.

First you have to login to OMC and click on the navigation icon below.

It’ going to open the navigation screen that contains 4 sections; Home, Cloud Services, Explorers and Administration.

In the Administration section click on Agents.

In this page we are looking into 2 important items for us. Registration Keys and Download.

The way I see the Registration Keys is like the Agent Registration password in OEM 13c. You need a valid Registration Key to deploy your Cloud Agent. So, click on that link and verify that you have a Registration Key; if not, you can create one.

Then proceed to download the agent by clicking on the Download link.

In order to install the Cloud Agent in the Linux host there are some pre-reqs to be met.

First verify that you have acl and libaio.i686 packages installed.

# rpm -q acl

acl-2.2.49-6.el6.x86_64

# rpm -qa|grep libaio

libaio-devel-0.3.107-10.el6.x86_64

libaio-0.3.107-10.el6.x86_64

libaio-0.3.107-10.el6.i686

Then move the AgentInstall.zip previously downloaded into an staging directory and unzip it.

$ mkdir –p /home/oracle/omc

$ mv AgentInstall.zip /home/oracle/omc

$ cd /home/oracle/omc

$ unzip AgentInstall.zip

$ $ ls

AgentInstall.bat AgentInstall.sh AgentInstall.zip README

Create a folder to install the Cloud Agent.

$ mkdir –p /u01/omc/infra

Verify that the server has the FQN configured (name.domain_name).

$ hostname

ol6-121-plug.localdomain

Finally install the Cloud Agent.

$ ./AgentInstall.sh AGENT_TYPE=cloud_agent AGENT_REGISTRATION_KEY=R******************1 AGENT_BASE_DIR=/u01/omc/infra AGENT_PORT=1302 ORACLE_HOSTNAME=ol6-121-plug.localdomain

WARNING: The current cURL version 7.19.7 does not support TLS1.2 protocol; the downloader script will attempt to continue but may not work. Please install 7.49.1 or later versions of cURL for error-free execution.

Downloading lama agent software …

Generating emaas.properties …

Extracting Agent Software …

Installing the Agent …

Registering the Agent …

Downloading Certificates …

Configuring the Agent …

Cleanup temporary files …

Then you have to run the root script required after the installation.

# /u01/omc/infra/core/1.15.0/root.sh

/etc exist

/u01/omc/infra/core/1.15.0

Check the status of the Cloud Agent.

$ ./omcli status agent

Oracle Management Cloud Agent

—————————————————————

Version : 1.15.0

State Home : /u01/omc/infra/agent_inst

Log Directory : /u01/omc/infra/agent_inst/sysman/log

Binaries Location : /u01/omc/infra/core/1.15.0

Process ID : 9828

Parent Process ID : 9790

URL : https://ol6-121-plug.localdomain:1302/emd/main/

Started at : 2017-03-20 20:49:15

Started by user : oracle

Operating System : Linux version 3.8.13-26.2.2.el6uek.x86_64 (amd64)

Data Collector enabled : false

Sender Status : FUNCTIONAL

Gateway Upload Status : FUNCTIONAL

Last successful upload : 2017-03-20 21:04:39

Last attempted upload : 2017-03-20 21:04:38

Pending Files (MB) : 0.01

Pending Files : 14

Backoff Expiration : (none)

—————————————————————

Agent is Running and Ready

Now if you go to the OMC portal, the agent will show up automatically.

Want to learn more about Oracle Management Cloud? There’s a free hands on lab “Cloud Experience” at Collaborate 2017 for all attendees on Wednesday 4/5/17.

Thanks,

Alfredo

OMS responded illegally [ERROR- Failed to Update Target Type Metadata] in 13c R2

I want to discuss about this OMS error coming from the agent’s heartbeat status. The agent shows with an unreachable status but is actually up and running.

$: emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 2

—————————————————————

Agent Version : 13.2.0.0.0

OMS Version : (unknown)

…

Last Reload : (none)

Last successful upload : (none)

Last attempted upload : (none)

Total Megabytes of XML files uploaded so far : 0

Number of XML files pending upload : 2,210

Size of XML files pending upload(MB) : 2.89

Available disk space on upload filesystem : 56.39%

Collection Status : Collections enabled

Heartbeat Status : OMS responded illegally [ERROR- Failed to Update Target Type Metadata]

Last attempted heartbeat to OMS : 2017-01-24 21:09:21

Last successful heartbeat to OMS : (none)

Next scheduled heartbeat to OMS : 2017-01-24 21:09:51

—————————————————————

Agent is Running and Ready

In this case the agent is not able to upload the metadata because is different in the OMS.

This is mainly due to a patch in the agent side that is missing in the OMS. By mistake I applied the latest bundle patch to the agent before applying it to the OMS.

Just another proof that any patches have to be installed to the OMS first then the agents.

Thanks,

Alfredo

Upgrade OPatch for OEM 13c R2 Agents

Opatch utility has to be updated often as Oracle delivers regular updates to it. I was used to version 12 and it only required an overwrite of the OPatch directory. Well, not anymore with version 13c R2.

I will walk you through an OPatch upgrade from 13.8 to 13.9 version.

First export your agent ORACLE_HOME:

$: export ORACLE_HOME=/u01/oracle/oemagent/agent_13.2.0.0.0/

Verify the current OPatch version:

$: $ORACLE_HOME/OPatch/opatch version

OPatch Version: 13.8.0.0.0

OPatch succeeded.

Download and unzip the 13.9 version in a temporary directory:

$: unzip p6880880_139000_Generic.zip

Archive: p6880880_139000_Generic.zip

creating: 6880880/

inflating: 6880880/README.txt

inflating: 6880880/opatch_generic.jar

inflating: 6880880/version.txt

As you can see, now we have some zip files and a java file. Execute java to upgrade OPatch as follows:

$: $ORACLE_HOME/jdk/bin/java -jar opatch_generic.jar -silent oracle_home=$ORACLE_HOME

Launcher log file is /tmp/OraInstall2017-01-19_08-47-21PM/launcher2017-01-19_08-47-21PM.log.

Extracting the installer . . . . Done

Checking if CPU speed is above 300 MHz. Actual 3000.000 MHz Passed

Checking swap space: must be greater than 512 MB. Actual 16383 MB Passed

Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)

Checking temp space: must be greater than 300 MB. Actual 18842 MB Passed

Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2017-01-19_08-47-21PM

Installation Summary

Disk Space : Required 6 MB, Available 5,229 MB

Feature Sets to Install:

Next Generation Install Core 13.9.1.0.0

OPatch 13.9.1.0.0

OPatch Auto OPlan 13.9.1.0.0

Session log file is /tmp/OraInstall2017-01-19_08-47-21PM/install2017-01-19_08-47-21PM.log

Loading products list. Please wait.

40%

Loading products. Please wait.

43%

…

97%

99%

Updating Libraries

Starting Installations

…

91%

92%

Install pending

Installation in progress

Component : oracle.swd.opatch 13.9.1.0.0

Copying files for ‘oracle.swd.opatch 13.9.1.0.0 ‘

Component : oracle.glcm.osys.core 13.9.1.0.0

Copying files for ‘oracle.glcm.osys.core 13.9.1.0.0 ‘

Component : oracle.glcm.oplan.core 13.9.1.0.0

Copying files for ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘

Component : oracle.glcm.opatch.common.api 13.9.1.0.0

Copying files for ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘

Component : oracle.glcm.opatchauto.core 13.9.1.0.0

Copying files for ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘

Install successful

Post feature install pending

Post Feature installing

Feature Set : apache_commons_cli_lib

Feature Set : oracle.glcm.opatchauto.core.actions.classpath

Post Feature installing ‘apache_commons_cli_lib’

Feature Set : oracle.glcm.opatchauto.core.binary.classpath

Feature Set : oracle.glcm.osys.core.classpath

Post Feature installing ‘oracle.glcm.opatchauto.core.binary.classpath’

Post Feature installing ‘oracle.glcm.osys.core.classpath’

Feature Set : apache_commons_compress_lib

Feature Set : oracle.glcm.opatchauto.core.wallet.classpath

Post Feature installing ‘oracle.glcm.opatchauto.core.actions.classpath’

Post Feature installing ‘apache_commons_compress_lib’

Feature Set : oracle.glcm.opatchauto.core.classpath

Post Feature installing ‘oracle.glcm.opatchauto.core.wallet.classpath’

Post Feature installing ‘oracle.glcm.opatchauto.core.classpath’

Feature Set : oracle.glcm.opatch.common.api.classpath

Feature Set : oracle.glcm.oplan.core.classpath

Post Feature installing ‘oracle.glcm.opatch.common.api.classpath’

Post Feature installing ‘oracle.glcm.oplan.core.classpath’

Post feature install complete

String substitutions pending

String substituting

Component : oracle.swd.opatch 13.9.1.0.0

String substituting ‘oracle.swd.opatch 13.9.1.0.0 ‘

Component : oracle.glcm.osys.core 13.9.1.0.0

String substituting ‘oracle.glcm.osys.core 13.9.1.0.0 ‘

Component : oracle.glcm.oplan.core 13.9.1.0.0

String substituting ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘

Component : oracle.glcm.opatch.common.api 13.9.1.0.0

String substituting ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘

Component : oracle.glcm.opatchauto.core 13.9.1.0.0

String substituting ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘

String substitutions complete

Link pending

Linking in progress

Component : oracle.swd.opatch 13.9.1.0.0

Linking ‘oracle.swd.opatch 13.9.1.0.0 ‘

Component : oracle.glcm.osys.core 13.9.1.0.0

Linking ‘oracle.glcm.osys.core 13.9.1.0.0 ‘

Component : oracle.glcm.oplan.core 13.9.1.0.0

Linking ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘

Component : oracle.glcm.opatch.common.api 13.9.1.0.0

Linking ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘

Component : oracle.glcm.opatchauto.core 13.9.1.0.0

Linking ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘

Linking in progress

Link successful

Setup pending

Setup in progress

Component : oracle.swd.opatch 13.9.1.0.0

Setting up ‘oracle.swd.opatch 13.9.1.0.0 ‘

Component : oracle.glcm.osys.core 13.9.1.0.0

Setting up ‘oracle.glcm.osys.core 13.9.1.0.0 ‘

Component : oracle.glcm.oplan.core 13.9.1.0.0

Setting up ‘oracle.glcm.oplan.core 13.9.1.0.0 ‘

Component : oracle.glcm.opatch.common.api 13.9.1.0.0

Setting up ‘oracle.glcm.opatch.common.api 13.9.1.0.0 ‘

Component : oracle.glcm.opatchauto.core 13.9.1.0.0

Setting up ‘oracle.glcm.opatchauto.core 13.9.1.0.0 ‘

Setup successful

Save inventory pending

Saving inventory

93%

Saving inventory complete

94%

Configuration complete

Logs successfully copied to /u01/app/oraInventory/logs.

Verify the new OPatch version

$: $ORACLE_HOME/OPatch/opatch version

OPatch Version: 13.9.1.0.0

OPatch succeeded.

Thanks,

Alfredo

ORA-01503: CREATE CONTROLFILE failed on RMAN duplicate between RAC databases

It was while since I performed an RMAN duplicate from a RAC database to another RAC database. You may find this post useless as there’s a lot of information out there about the error I faced, but all my posts are kind of my own little diary.

So, basically. In order to successfully duplicate one database to another you may need this.

– TNS setup on the server where the database to be duplicated is hosted. You need to be able to tnsping both, the target and the auxiliary database.

– Create a parameter file and password file for the auxiliary and startup nomount the first instance.

– Create a full backup of the target database including archivelogs.

– Connect to rman to both the target and the auxiliary databases from the auxiliary server:

rman target sys/*****@ auxiliary /

– Issue the duplicate command:

duplicate target database to ;

This should successfully finish but I got this error:

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of Duplicate Db command at 01/03/2017 18:29:58

RMAN-05501: aborting duplication of target database

RMAN-06136: ORACLE error from auxiliary database: ORA-01503: CREATE CONTROLFILE failed

ORA-12720: operation requires database is in EXCLUSIVE mode

I searched this and the culprit is that my parameter file for the auxiliary had this:

cluster_database = true

By MOS note “Rman duplicate fail with RMAN-06136, ORA-01503, ORA-12720, ORA-00494 enqueue [CF] (Doc ID 1335479.1)”. Duplicate from a RAC database to another RAC database is not supported, hence we need to modify the parameter cluster_database to false in the auxiliary database.

After doing this the duplicate finished correctly.

Thanks,

Alfredo

OEM 13c New Features – Corrective Actions

Today’s post is about another new feature of Oracle Enterprise Manager (OEM) 13c. Well, not a new feature itself but some awesome enhancements to Corrective Actions (CA).

In this new release 13c several new CA types were added, including “Add Space To Tablespace”, “Chef Recipe” for those who like automation and now can be added to Service Level Agreement (SLA) alerts and job status events.

I’m a big enthusiast of OEM jobs and now having the option to set a CA for them is just more than cool!

I wrote an article a year ago on IOUG’s Select journal about Corrective Actions and didn’t want to finish my day without setting a CA for job. But wait a minute, I know I can set CA for metrics using the “Metric and Collection Settings” page, but there’s no such thing for jobs.

Here’s how we setup a corrective action for jobs. First we need to enable events for jobs, this means that we as administrators decide what kind of job status will create an event.

In order to do this, you need to navigate to Setup -> Incidents -> Job Events

Step 1. Here you decide what job status and job severity generates an event.

Step 2. You decide if you want to generate events for jobs without targets.

Step 3. Choose the target you want to generate job events for.

Once you have the proper events setup, then the next step is to catch this event and do something with it. In this case fire a Corrective Action (CA). The only place we can catch these events and do something with them is an Incident Rule.

Navigate to Setup-> Incident -> Incident Rules

Then create or modify an existing Incident Rule Set. Inside this Rule Set create a new Rule to catch these job events. Inside the Rule Actions there’s a new section to Submit a Corrective Action.

Click on the “Select corrective action” and choose the CA from your CA Library.

Cool doesn’t it! One of the uses I can think of is with Database backup jobs. Sometimes the RMAN backup fails when is not able to find an archivelog and the solution is to run a crosscheck command and retry the backup job. This CA can do that for us and just notify us once it completes.

It still needs further testing but is another tool to save us time.

If you want information about the “Add Space To Tablespace” CA you better check Kellyn’s post about it.

http://dbakevlar.com/2016/04/em13c-corrective-actions/

Thanks,

Alfredo

OEM 13c New Features – System Broadcast

I can’t finish the year without a post about OEM 13c. I finally had some time to install it and from first hand go through its new features. I find this “System Broadcast” feature so useful, especially when you have tons of users using OEM.

You can try to email users, give them a call but they will always forget about the next maintenance window and call you right away when OEM is not available. This feature helps you to send a notification to all the users or a particular list of users.

System Broadcast messages (up to 200 characters) can only be sent using EMCLI, there’s no graphical option yet.

Here’s an example on how to send a message to a user named OEMADMIN:

$ emcli send_system_broadcast -toOption=”SPECIFIC” -to=”OEMADMIN” -message=”OEM will be down for maintenance Friday December 30th”

Successfully requested to send System Broadcast to users.

This is what you see once you login to OEM console:

Send the message to all the users:

$ emcli send_system_broadcast -toOption=”ALL” -message=”OEM will be down for maintenance Friday December 30th”

Successfully requested to send System Broadcast to users.

You need to be logged in EMCLI to be able to send these messages, if you are not you’ll get this error:

$ emcli send_system_broadcast -toOption=”ALL” -message=”OEM will be down for maintenance Friday December 30th”

Status:Unauthorized 401

As I said, you need to be logged in EMCLI:

$emcli login -username=sysman

Enter password:

$ emcli send_system_broadcast -toOption=”ALL” -message=”OEM will be down for maintenance Friday December 30th”

Successfully requested to send System Broadcast to users.

Here’s the verb syntax:

emcli send_system_broadcast

-toOption=”ALL|SPECIFIC”

[-to=”comma separated user names”]

[-messageType=”INFO|CONF|WARN|ERROR|FATAL” (default is INFO)]

-message=”message details”

[ ] indicates that the parameter is optional.

https://docs.oracle.com/cd/E63000_01/EMCLI/cli_verb_ref.htm#EMCLI1850

Thank you and happy new year!

Alfredo

ORA-01555 query duration 0 seconds with Dataguard

How many times we have calls from users complaining about their process that failed due to an ORA-01555 error?

We know that if the queries are not well tuned and they modify a lot of data, the image held in the UNDO Tablespace could not be consistent with the real data. But have you ever seen this error right away after executing a SQL statement against a table?

I just did couple of days ago. Here’s the story:

ORA-01555 error appeared in the alertlog’s database with a query duration of 0 seconds.

ORA-01555 caused by SQL statement below (SQL ID: d3rt4tyudufeu, Query Duration=0 sec, SCN: 0x034f.34f660b4)

Any queries plus an analyze table failed right away with ORA-01555:

ERROR at line 1: ORA-0155: snapshot too old: rollback segment number 10 with name “SYSSMU11_1072300523734$” too small

So weird.

After researching a bit on MOS, found a note regarding a bug.

Some minutes later we also started to receive ORA-600 errors related so scn numbers.

ORA-error stack (00600[ktbdchk1: bad dscn])

The MOS note mentions the ORA-01555 and the ORA-600 errors as part of bug 22241601 with a Dataguard configuration. Is worth to mention that yes, we were doing switchover testing recently in this 12.1.0.2 environment.

ALERT Bug 22241601 ORA-600 [kdsgrp1] / ORA-1555 / ORA-600 [ktbdchk1: bad dscn] / ORA-600 [2663] due to Invalid Commit SCN in INDEX (Doc ID 1608167.1)

The solution is to apply the patch but there’s also a tested workaround that is to rebuild online all the indexes of that table.

Hope this helps.

Alfredo

CRS-2632: There are no more servers to try to place resource ‘resource_name’ on that would satisfy its placement policy

I just got this error from CRS while trying to start a database using srvctl. I find this error kind of missleading as the error doesn’t tell what is really happening with the resource.

I suggest you to manually check the logs of the resource. In this case it had to do with a cluster database that one of the instances was not able to start. I tried to start this instance manually and then I got the real error.

The instance was having a missconfiguration in the parameter file that prevented it from starting.

Hope this note will provide you a guidance while having this CRS-2632 error coming out from CRS.

Thanks,

Alfredo

Tim Gorman is our guest speaker for September 30th NEOOUG’s quarterly meeting.

Tim currently works as a Technical Consultant at Delphix. Oracle ACE Director, Oak Table member and an advocate of the North East Ohio Oracle Users Group for many years. This time Tim brings two interesting sessions, “Split Brain Syndrome” and “Scaling To Infinity”.

Join us for these sessions, have lunch and learn from one of the most remarkable leader in technology.

Keep in mind that this time this quarterly meeting is to be held at the new Oracle’s facility in Beachwood, OH. We also require you to pre-register to this event using below URL:

Pre-register

More information here:

https://www.neooug.org/events/

Thanks,
Alfredo