Tag: Exadata

How To Discover ExaCC/ExaCS in Enterprise Manager – Detailed

In this post I want to share the required steps to discover ExaCC or ExaCS in Oracle Enterprise Manager (EM). I’ll provide as much detail as possible.

  1. Make sure your EM is at least on 13.5 RU 16. If not, please apply RU 16 to EM. Follow this link if you want to know how to patch EM. HERE!
  2. You need to designate a “monitoring” agent for the discovery. Is recommended that this agent sits outside your Exadata rack and has access to the OCI REST APIs. More information HERE!
  3. Make sure this agent has both the Database and the Exadata plugins deployed and is patched to the same RU level as the OMS. HERE!
  4. The agent must be able to reach the OCI REST APIs. There are 3 ways to achieve this.
    a) You can use a Proxy
    b) You can use the OCI Management Gateway
    c) You can have direct network connection

    Below is a list of APIs that you will need access to:
    Either grant access to (*.oci.oraclecloud.com) or individual URLS:
    https://query.<oci_region>.oci.oraclecloud.com
    https://identity.<oci_region>.oci.oraclecloud.com
    https://database.<oci_region>.oci.oraclecloud.com
    https://wss.exacc.<oci_region>.oci.oraclecloud.com
    https://management-agent.<oci_region>.oci.oraclecloud.com
    https://certificatesmanagement.<oci_region>.oci.oraclecloud.com
    https://certificates.<oci_region>.oci.oraclecloud.com
    https://telemetry-ingestion.<oci_region>.oci.oraclecloud.com
    https://auth.<oci_region>.oci.oraclecloud.com
    https://objectstorage.<oci_region>.oci.oraclecloud.com

    You may also test that connectivity by executing:
    $ curl -v https://query.<oci_region>.oci.oraclecloud.com
  5. Create or use an OCI account that has access to read your Exadata racks. These are the policies I have used:
    allow group <domain/group name> to read database-family in compartment <compartment name>
    You can find more information about the required policies HERE!
  6. Setup API Keys for authentication. Please follow instructions in the MOS note below:
    EM13.5: Manage OCI Connectivity Named Credentials Test Failed with Invalid Private Key. (Doc ID 2792126.1)
  7. Create an EM Named Credential using the API Keys created on the previous step. More details HERE!
  8. You also need the proper Storage Server credentials. If you don specify them during the discovery process, Storage Server targets will not be discovered. Instructions on how to retrieve this credential can be found HERE!
    If you can’t retrieve these credentials, please open a Service Request with support.
  9. Discover your Exadata Infrastructure following the discovery wizard. More information HERE!

After finishing the discovery, you should have your Exadata Cloud target in EM. Please wait between 15 to 20 minutes for the target information to be populated.

If your network setup does not allow direct connectivity to the OCI REST APIs then you will have to use an internal Proxy or use the OCI Management Gateway. More information about the OCI Management Gateway can be found HERE!

There’s also a very good post from Simone about setting up the Management Gateway for the EM agents.
https://blogs.oracle.com/observability/post/setup-oem-agents-cloud-mgmt-gw

Update:
You will also have to modify 2 parameters. One at the OMS level and one at the agent level. Please follow below MOS notes:

OEM 13c : SSH Key Credential Test On a Host Target Fails With “Remote operation timed out.” (Doc ID 2415262.1)
EM 13.2: Agent Patching From EM Console Fails With Error “concurrent job tasks limit (50) has been reached” (Doc ID 2476803.1)

Thanks,
Alfredo

Maximize Your Exadata Cloud at Customer Efficiency Using Oracle Enterprise Manager 13.5

Oracle Enterprise Manager (EM) it’s been the go-to tool to monitor and manage on-premises Exadata deployments for quite some time. By doing a quick search in this blog I found presentations dated back from 2019 about choosing EM 12c or 13c to monitor and manage Exadata. Time has gone quick and now it’s time to discuss a new generation of Cloud services.

During the recent years Exadata appliances have evolved into Exadata Cloud Service (ExaCS) and Exadata Cloud @ Customer (ExaCC). Customers have the same monitoring requirements though. It is more than necessary to have proper observability mechanisms in place for these cloud services.

This post is meant to guide you through the process of discovering your Exadata Cloud @ Customer services into Oracle Enterprise Manager 13.5

First things first. You can find all the required information in the EM 13.5 documentation.

click here…

One of the questions that I get from customers is, what are the features in EM for your ExaCC deployment? Here are some of them:

  • Automatically identify and organize related targets
  • Visualize the database and related targets associated with ExaCC
  • Monitor performance for ExaCC components like Storage Grids, ExaCC DomU, GI and database targets on a single pane of glass

For more information about these features, refer to the Features section in the Oracle documentation.

Let’s get now into the discovery process. There are some pre-reqs that you will have to meet before attempting the discovery process. Please read carefully Part II of the Oracle documentation. Here are some highlights:

  • Create Named Credentials for OCI
    In this step you will generate the required OCI API Signing Keys in order to create an EM Named Credential. Follow the instructions in this section but also look at below MOS note:
    EM13.5: Manage OCI Connectivity Named Credentials Test Failed with Invalid Private Key. (Doc ID 2792126.1)
  • Discover Exadata C@C
    On this step we will discover ExaCC in EM. Staring with EM 13.5 RU14 you can discover it using the UI (console). Before that the only available option was to use CLI. I strongly recommend you apply RU14 before attempting the discovery process.
    Follow the steps on the documentation in order to discover your ExaCC service.

During the discovery process you will have to designate or install an EM agent that will be responsible of monitoring your ExaCC service. This agent must have access to Oracle Cloud Infrastructure (OCI).

Open the Discovery Wizard in EM

On the “Add Targets Manually” page choose the guided process.

Select “Oracle Exadata Infrastructure” and click the “Add” button.

On the “Exadata Cloud Discovery” wizard select the required inputs:

  • Management Agent
  • Backup Agent (optional)
  • Agent’s Named Credential
  • OCI Cloud Home Region
  • OCI Named Credential (created on the pre-reqs section)
  • OCI Subscribed Region (where your tenancy resides)

After this, click on the “Next” button.

On the “Discovery” section of the wizard, EM will discover all the ExaCC related targets. On this page, set the “Storage Server Credentials” in order to monitor the Storage Servers. Click “Next”

On the next page a summary will be displayed. Verify all the targets and then click “Submit”.

At this point you will have your ExaCC service, discovered in EM. You can see all your ExaCC services by navigating to “Targets” -> “Exadata”.

The “Exadata” dashboard shows all your Exadata appliances and services. Click on the recently discovered ExaCC service.

The “Home Page” shows all ExaCC components along with resource utilization.

Use the “Overview Tab” in order to navigate on all the available monitoring sections. Below screenshot provides an insight into “CPU” utilization by selecting the “Resource” section.

Follow the Oracle documentation in order to understand all the available options and sections in the ExaCC Home Page.

At this point your ExaCC is discovered and ready to be managed and monitored.

Next steps are to setup your monitoring metrics, thresholds and rules in order to receive notifications. Also look at the available OAS reports available for your ExaCC.

We will cover the integration of this EM monitoring and management of ExaCC with OCI Observability & Management services on future posts. These O&M services like Operations Insights help you to forecast utilization, provide advanced capacity planning insights by using Machine Learning algorithms.

Update!!!: Royce Fu just created a wonderful post about EM and O&M integration for ExaCC. Please find the link below.

click here…

Thanks,
Alfredo

Oracle Enterprise Manager 13.3 problems discovering Exadata targets

If you are on the latest Exadata plugin for EM 13.3 (well, even the 13.2 version of the plugin). Be aware that there’s a bug where if the Exadata nodes’ names are longer than 7 characters, you cannot discover them.

EM just throws a weird error with below legend: 

SEVERE: makeTargetsXml for each rack#: 1 e=java.lang.StringIndexOutOfBoundsException: String index out of range: -1

The solution is to apply the latest (March 2019) Bundle Patch to the agents running on the Exadata’s compute nodes.

More info on below MOS note:  

Exadata Discovery Fails With “No New Component Is Discovered. Click Cancel” (Doc ID 2477624.1)  

Another good excuse to keep your EM system up to date.  

Happy patching.  

Thanks,
Alfredo

Oracle 19c Available for Exadata On-Premises

Oracle just released Oracle 19c for Exadata On-Premises. This is just close to a year when they released the 18c version for Exadata On-Premises on February 16th 2018.

If they follow the same schedule, we may expect 19c for the Database Cloud Service early next month.
Release dates can be found in the MOS Doc Id 742060.1
I did check the versions available in the Oracle Cloud for VM, Bare Metal and Exadata.
No 19c version as expected.

Now is time to start playing with the new features of 19c. Stay tuned!
Thanks,
Alfredo

Exadata/Exacloud CellServer health using AWR report

Exadata’s unique sauce relies on the fact that the DB instance can offload work to the CellServers to speed up I/O bound queries. But how do you monitor the health of the CellServers?
A quick and dirty way, especially if you don’t have access to the CellServers is to use AWR in a database that’s currently using those CellServers. Let’s get an AWR report and search for the SMART IO section. It should look like this.

What you want to see here is that % Total MB Requested is evenly distributed across all Cells. The same applies for Storage Index and Flash Cache metrics.
Now this is the interesting part. If you happen to see the Offload % Efficiency dropping for one of the cells, that means that the Cell is sending the complete blocks back to the DB and then the % Passthru metric is going to increase.
You don’t really want the % Passthru to increase as this will have a direct performance is the DBs being served by this Cell.
What are the reasons of this Passthru increasing?
There are 3 reasons for Passthru.
The first one is due to an issue with the Cell itself. The second one is due to issues with DST patches and the third one is related to user queries crashing the Cell.
You should look at the below MOS note in order to gather all the required information and open an SR immediately after you see this behavior.
SRDC – Exadata: Smart Scan Not Working Issues (Doc ID 2310422.1)
Thanks,
Alfredo