Integrate EM Metrics With Third-Party Tools Using REST APIs

I want to share how to use Enterprise Manager’s (EM) REST APIs to query metric data and integrate with 3rd party tools.

First of all, always follow the documentation. Below is the link to the EM’s REST API functionality.



Using the EM’s REST API functionality we can query the EM repository using a web service. Let’s follow below steps.

a) Create a new EM user who is going to have the privileges to query the EM’s repository. Navigate to Setup -> Security -> Administrators.

Administrators navigation

b) Create a new Repository User.

Administrator creation

c) I’m going to create a new Administrator named ABC.

Administrator creation

d) Create a new Named Credential. Navigate to Setup -> Security -> Named Credentials.

Named Credential creation

e) Click on Create.

Named Credential creation

f) I’m going to create NC_ABC Named Credential.

Named Credential creation

g) Enable the repository REST APIs in EM. For that execute below emctl command.

emctl set property -name oracle.sysman.db.restfulapi.executesql.repository.query.enable -value true -sysman_pwd "sysman"

Now is time to test our REST APIs.

I’m going to test them out using 2 different mechanisms. For the first one, let’s use CURL.

$ curl -k -u ABC -H "Content-Type: application/json" -d '{"sqlStatement": "SELECT * FROM sysman.MGMT$TARGET_METRIC_SETTINGS", "credential": {"DBCredsMonitoring":"NC_ABC"}, "maxRowLimit": 10, "maxColumnLimit": 2}' https://<EM_Host>:<EM Port>/em/websvcs/restful/emws/oracle.sysman.db/executesql/repository/query/v1

Here are the results based on my demo environment:

REST API results from CURL

The second test is by using Ansible. If you are not familiar on how to configure Ansible in order to work with EM, please follow the Oracle Live Labs that we created about this.



I’m going to create a new YAML file (test.yml) in order to test the REST API.

---
- name: Using a REST API
  become: false
  hosts: emserver
  gather_facts: false
  tasks:

    - name: Query Metric Settings
      uri:
        url: https://emcc.marketplace.com:7803/em/websvcs/restful/emws/oracle.sysman.db/executesql/repository/query/v1
        method: POST
        return_content: yes
        force_basic_auth: yes
        validate_certs: no
        body_format: json
        url_username: ABC
        url_password: *******
        headers:
           Content-Type: application/json
        body:
          {
            "sqlStatement": "SELECT * FROM sysman.MGMT$TARGET_METRIC_SETTINGS", "maxRowLimit": 2, "maxColumnLimit": 29
          }
      register: results

    - name: Print returned json dictionary
      debug:
        var: results.json

Let’s execute this using Ansible.

ansible-playbook /home/oracle/ansible/yml/test.yml -u oracle --private-key=~/.ssh/dtkey.ssh

Here the results:

REST API results from Ansible

With this you can get the desired results from the EM’s repository using this REST API.

Thanks,
Alfredo

Extend Your Oracle Monitoring With Configuration Extensions

As soon as you enable the Database Lifecycle Management Pack; Oracle Enterprise Manager will automatically collect hundreds of configuration items of your database estate. However if you need to customize this collection, Configuration Extensions provide a way to identify files and other configuration data that Cloud Control does not already collect.

In this blog, I’ll show you how I created a Configuration Extension (CE) in order to collect DB Services data.

CE’s are found under the Enterprise -> Configuration menus.

While in the CE dashboard, you can create, edit, delete and deploy CE’s using the available menu.

Let’s click the “Create” button.

Using the CE create wizard type a name for the CE, a target type and a Sample Target. As soon as you select “Database Instance” for the Target Type a new option will enable. This option is to select between “Files & Commands” or “SQL”.

In this example I’m going to choose “SQL” and then type the required SQL required to retrieve the data I need from the DB. Let’s type “select name as service_name from dba_services”. Also type “service_name” in the alias section and choose the “Database Query Parser”.

The entries should look similar to the screenshot above.

The data extracted from the DB in the CE is in XML format. We need to add parser rules in order to match and identify nodes in the parsed tree. Click on the number of “Rules” in the table. In this case click on the number “0”.

On the Parser Rule page click “Add” and type this condition and expression:

  • Condition: /root/row
  • Expression: SERVICE_NAME/text()

This will allow the parser to use the text inside the SERVICE_NAME node as the name of the row. Click the return button.

Now we can click on the “Preview” button from the menu.

It will show you the preview output. Similar like this:

Click “OK” and then click “Save”.

Select your newly created CE and click on “Manage Deployments” button. Select the targets where you want to deploy the CEs and click “Apply”.

It will look similar to the screen below:

Navigate to the configuration of your target and verify the CE is correctly pulling the desired configuration data.

Now we have our CE correctly gathering the desired configuration data of our DB.

You can find the documentation for Configuration Extensions below.



Hope this helps to provide some clarity on how to create Configuration Extensions in Oracle Enterprise Manager.

Thanks,
Alfredo

How To Discover ExaCC/ExaCS in Enterprise Manager – Detailed

In this post I want to share the required steps to discover ExaCC or ExaCS in Oracle Enterprise Manager (EM). I’ll provide as much detail as possible.

  1. Make sure your EM is at least on 13.5 RU 16. If not, please apply RU 16 to EM. Follow this link if you want to know how to patch EM. HERE!
  2. You need to designate a “monitoring” agent for the discovery. Is recommended that this agent sits outside your Exadata rack and has access to the OCI REST APIs. More information HERE!
  3. Make sure this agent has both the Database and the Exadata plugins deployed and is patched to the same RU level as the OMS. HERE!
  4. The agent must be able to reach the OCI REST APIs. There are 3 ways to achieve this.
    a) You can use a Proxy
    b) You can use the OCI Management Gateway
    c) You can have direct network connection

    Below is a list of APIs that you will need access to:
    Either grant access to (*.oci.oraclecloud.com) or individual URLS:
    https://query.<oci_region>.oci.oraclecloud.com
    https://identity.<oci_region>.oci.oraclecloud.com
    https://database.<oci_region>.oci.oraclecloud.com
    https://wss.exacc.<oci_region>.oci.oraclecloud.com
    https://management-agent.<oci_region>.oci.oraclecloud.com
    https://certificatesmanagement.<oci_region>.oci.oraclecloud.com
    https://certificates.<oci_region>.oci.oraclecloud.com
    https://telemetry-ingestion.<oci_region>.oci.oraclecloud.com
    https://auth.<oci_region>.oci.oraclecloud.com
    https://objectstorage.<oci_region>.oci.oraclecloud.com

    You may also test that connectivity by executing:
    $ curl -v https://query.<oci_region>.oci.oraclecloud.com
  5. Create or use an OCI account that has access to read your Exadata racks. These are the policies I have used:
    allow group <domain/group name> to read database-family in compartment <compartment name>
    You can find more information about the required policies HERE!
  6. Setup API Keys for authentication. Please follow instructions in the MOS note below:
    EM13.5: Manage OCI Connectivity Named Credentials Test Failed with Invalid Private Key. (Doc ID 2792126.1)
  7. Create an EM Named Credential using the API Keys created on the previous step. More details HERE!
  8. You also need the proper Storage Server credentials. If you don specify them during the discovery process, Storage Server targets will not be discovered. Instructions on how to retrieve this credential can be found HERE!
    If you can’t retrieve these credentials, please open a Service Request with support.
  9. Discover your Exadata Infrastructure following the discovery wizard. More information HERE!

After finishing the discovery, you should have your Exadata Cloud target in EM. Please wait between 15 to 20 minutes for the target information to be populated.

If your network setup does not allow direct connectivity to the OCI REST APIs then you will have to use an internal Proxy or use the OCI Management Gateway. More information about the OCI Management Gateway can be found HERE!

There’s also a very good post from Simone about setting up the Management Gateway for the EM agents.
https://blogs.oracle.com/observability/post/setup-oem-agents-cloud-mgmt-gw

Update:
You will also have to modify 2 parameters. One at the OMS level and one at the agent level. Please follow below MOS notes:

OEM 13c : SSH Key Credential Test On a Host Target Fails With “Remote operation timed out.” (Doc ID 2415262.1)
EM 13.2: Agent Patching From EM Console Fails With Error “concurrent job tasks limit (50) has been reached” (Doc ID 2476803.1)

Thanks,
Alfredo

Maximize Your Exadata Cloud at Customer Efficiency Using Oracle Enterprise Manager 13.5

Oracle Enterprise Manager (EM) it’s been the go-to tool to monitor and manage on-premises Exadata deployments for quite some time. By doing a quick search in this blog I found presentations dated back from 2019 about choosing EM 12c or 13c to monitor and manage Exadata. Time has gone quick and now it’s time to discuss a new generation of Cloud services.

During the recent years Exadata appliances have evolved into Exadata Cloud Service (ExaCS) and Exadata Cloud @ Customer (ExaCC). Customers have the same monitoring requirements though. It is more than necessary to have proper observability mechanisms in place for these cloud services.

This post is meant to guide you through the process of discovering your Exadata Cloud @ Customer services into Oracle Enterprise Manager 13.5

First things first. You can find all the required information in the EM 13.5 documentation.



One of the questions that I get from customers is, what are the features in EM for your ExaCC deployment? Here are some of them:

  • Automatically identify and organize related targets
  • Visualize the database and related targets associated with ExaCC
  • Monitor performance for ExaCC components like Storage Grids, ExaCC DomU, GI and database targets on a single pane of glass

For more information about these features, refer to the Features section in the Oracle documentation.

Let’s get now into the discovery process. There are some pre-reqs that you will have to meet before attempting the discovery process. Please read carefully Part II of the Oracle documentation. Here are some highlights:

  • Create Named Credentials for OCI
    In this step you will generate the required OCI API Signing Keys in order to create an EM Named Credential. Follow the instructions in this section but also look at below MOS note:
    EM13.5: Manage OCI Connectivity Named Credentials Test Failed with Invalid Private Key. (Doc ID 2792126.1)
  • Discover Exadata C@C
    On this step we will discover ExaCC in EM. Staring with EM 13.5 RU14 you can discover it using the UI (console). Before that the only available option was to use CLI. I strongly recommend you apply RU14 before attempting the discovery process.
    Follow the steps on the documentation in order to discover your ExaCC service.

During the discovery process you will have to designate or install an EM agent that will be responsible of monitoring your ExaCC service. This agent must have access to Oracle Cloud Infrastructure (OCI).

Open the Discovery Wizard in EM

On the “Add Targets Manually” page choose the guided process.

Select “Oracle Exadata Infrastructure” and click the “Add” button.

On the “Exadata Cloud Discovery” wizard select the required inputs:

  • Management Agent
  • Backup Agent (optional)
  • Agent’s Named Credential
  • OCI Cloud Home Region
  • OCI Named Credential (created on the pre-reqs section)
  • OCI Subscribed Region (where your tenancy resides)

After this, click on the “Next” button.

On the “Discovery” section of the wizard, EM will discover all the ExaCC related targets. On this page, set the “Storage Server Credentials” in order to monitor the Storage Servers. Click “Next”

On the next page a summary will be displayed. Verify all the targets and then click “Submit”.

At this point you will have your ExaCC service, discovered in EM. You can see all your ExaCC services by navigating to “Targets” -> “Exadata”.

The “Exadata” dashboard shows all your Exadata appliances and services. Click on the recently discovered ExaCC service.

The “Home Page” shows all ExaCC components along with resource utilization.

Use the “Overview Tab” in order to navigate on all the available monitoring sections. Below screenshot provides an insight into “CPU” utilization by selecting the “Resource” section.

Follow the Oracle documentation in order to understand all the available options and sections in the ExaCC Home Page.

At this point your ExaCC is discovered and ready to be managed and monitored.

Next steps are to setup your monitoring metrics, thresholds and rules in order to receive notifications. Also look at the available OAS reports available for your ExaCC.

We will cover the integration of this EM monitoring and management of ExaCC with OCI Observability & Management services on future posts. These O&M services like Operations Insights help you to forecast utilization, provide advanced capacity planning insights by using Machine Learning algorithms.

Update!!!: Royce Fu just created a wonderful post about EM and O&M integration for ExaCC. Please find the link below.



Thanks,
Alfredo

How To Monitor Oracle DBs On Docker Containers With EM 13c

This post continues the thread of DevOps and automation for the Oracle Database. During the last couple of months I’ve seen more interest in deploying Oracle Databases on Docker containers. Even more now that Oracle released full support of RAC Databases running on Docker.



After you deploy your first Oracle Database on Docker, several questions come into mind; like:

  • How do I monitor the status of the Oracle Database?
  • How do I manage the performance of the Oracle Database and the SQL’s being executed on it?
  • How do I alert on issues on a timely manner?

Well, if you have the same questions or concerns… you are not alone!

On this post, I’m trying to explain the process I followed in order to deploy an Oracle Enterprise Manager agent on a Docker container and how I do monitor and manage the Oracle Database running on it.

First things first. Just a quick recap of my environment.

  • Oracle Linux 7 host running Docker 19.03
  • Create a MACVLAN network on Docker. My network adapter is ens3 on the host running Docker.
sudo docker network create -d macvlan --subnet=192.168.56.0/24 --gateway=192.168.56.1 -o parent=ens3 orcl_nw
  • Create a Docker volume type “nfs” using an NFS share named /NFSSHARE
sudo docker volume create --driver local --opt type=nfs --opt o=addr=<nfs server>,rw,bg,hard,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 --opt device=:/NFSSHARE agentstorage
  • Pull the 19.3 Docker image from container-registry.oracle.com
sudo docker pull container-registry.oracle.com/database/enterprise:19.3.0.0
  • Create the Docker container using the recently pulled image and mounting the volume “agentstorage” into “/u01” inside the container
sudo docker create -t -i \
--name ol7-orcl --hostname ol7-orcl --user oracle --ip 192.168.56.101 \
-e ORACLE_SID=ORCL \
-e ORACLE_PDB=PDB1 \
-v agentstorage:/u01 \
container-registry.oracle.com/database/enterprise:19.3.0.0
  • Disconnect the bridged Docker network from the container
sudo docker network disconnect bridge ol7-orcl
  • Connect the container to the MACVLAN network
sudo docker network connect orcl_nw --ip 192.168.56.101 ol7-orcl
  • Start our Docker container
sudo docker start ol7-orcl
  • Review the startup process and the creation of the database
sudo docker logs -f ol7-orcl

At this point we have our Docker container and the ORCL database up and running. But now I need a way to connect my ORCL database to the external world. Because I’m running a MACVLAN, I can create another Docker container in the same network and install Oracle Enterprise Manager (EM) in order to monitor it. Or I can also install Oracle EM on the host running Docker and setup a network link and a route to my Docker container. Let’s do the second option.

sudo ip link add mydocker-net link ens3 type macvlan mode bridge
sudo ip addr add 192.168.56.1/32 dev mydocker-net
sudo ip link set mydocker-net up
sudo ip route add 192.168.56.0/24 dev mydocker-net

I now have connectivity between my host and the Docker container.

$ ping 192.168.56.101
PING 192.168.56.101 (192.168.56.101) 56(84) bytes of data.
64 bytes from 192.168.56.101: icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from 192.168.56.101: icmp_seq=2 ttl=64 time=0.070 ms
64 bytes from 192.168.56.101: icmp_seq=3 ttl=64 time=0.062 ms
64 bytes from 192.168.56.101: icmp_seq=4 ttl=64 time=0.073 ms
64 bytes from 192.168.56.101: icmp_seq=5 ttl=64 time=0.070 ms

sh-4.2$ ping 192.168.56.1
PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data.
64 bytes from 192.168.56.1: icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from 192.168.56.1: icmp_seq=2 ttl=64 time=0.079 ms
64 bytes from 192.168.56.1: icmp_seq=3 ttl=64 time=0.043 ms
64 bytes from 192.168.56.1: icmp_seq=4 ttl=64 time=0.082 ms
64 bytes from 192.168.56.1: icmp_seq=5 ttl=64 time=0.077 ms

The next step is to have name resolution setup between both hosts. For the Docker container it was easy to setup the full host name into the external DNS service. For the host resolution inside the Docker container you can use static entries in the /etc/hosts file or use the Docker embedded DNS server. For my exercise, I setup an entry in my /etc/hosts file providing the host name and the IP address of the host where EM is running.

The rest of the process is very straight forward. I use the silent install option for the Oracle EM agent. Below link has the steps to download the agent binaries for your specific platform and version.



Once you have the ZIP file of the agent binaries. Then just copy it inside the NFS share and connect to the Docker container.

sudo docker exec -it --user oracle ol7-orcl /bin/sh

After you login to the container just follow the steps in the provided agent install guide. Update the response file with the required information and run agentDeploy.sh. Make sure that the Agent Home resides in the Docker volume created. For this exercise will be inside /u01 directory.

After the agent deploy succeeds disconnect from the Docker session and re-connect as root.

sudo docker exec -it --user root ol7-orcl /bin/sh

Execute root.sh inside the Agent Home directory.

Let’s now verify that our EM has a new host target named ol7-orcl and is up and running.

Docker Container ol7-orcl on Oracle Enterprise Manager

The next step is to discover the ORCL database in EM. You can change the DBSNMP password by logging to the Docker container, then use SQLPlus to change the password and unlock the account.

alter user dbsnmp identified by <password> account unlock;

After the database discovery, verify the database target in the EM console.

ORCL database home page

Now that you have both the host target and the database target you can set metrics, thresholds and notifications on them.

Another important option is that if you have Diagnostics and Tuning packs, you can also monitor performance, get AWR/ASH reports and use EM to graphically analyze performance issues.

ORCL performance information

Hope this post helps understand the options you have in order to setup the required monitoring for the Oracle databases running on Docker containers.

Thanks,
Alfredo

Oracle Enterprise Manager 13c Series – Patching

I wanted to create a video series covering the basics of Oracle Enterprise Manager 13c. These series are designed to provide insights on how to install, configure and manage Oracle Enterprise Manager 13.5.

This week we are going to cover the patching of Oracle Enterprise Manager 13.5.

Oracle Enterprise Manager 13c – Patching

Hope this is useful.

Thanks,
Alfredo

Patch Oracle Databases With Ansible and Enterprise Manager 13c

In a previous post I show you how you can integrate DevOps automation and orchestration tools to provision Oracle databases by leveraging Enterprise Manager (EM) and the Cloud Management Pack (CMP). Once provisioned, databases need to be fully maintained in terms of monitoring but most precisely patching.

Patching databases decreases the risk of breach by timely mitigating vulnerabilities, but in can be a daunting task for organizations. Manual patching is time consuming and error prone. Home grown scripts are difficult to maintain and they increase maintenance cost. So the question is, how can I automate the patching process and even better, how can I integrate it with my current orchestration workflow?

Let me explain how you can achieve all this by making use of Oracle’s Database Lifecycle Management Pack (DBLM) and CMP. DBLM’s Fleet Maintenance help us patch and upgrade Oracle databases at scale with minimum downtime. It makes use of Gold Images and out-of-place patching in order to achieve this.

Fleet Maintenance Benefits

All this functionality can be integrated with CMP’s DBaaS in order to provide and end-to-end automation solution. DBaaS exposes REST APIs that we could then call using the automation tool of choice. Database Administrators, end users or 3rd party tools can then use these features to patch Oracle databases.

DBaaS Automation Diagram

Do you want to learn more about this and even be able to try it? We’ve created an Oracle LiveLabs that cover’s all this functionality. This lab will guide you through the request of a PDB, setup DBaaS configuration, setup Fleet Maintenance and finally patch the PDB.

Follow the link below for the Oracle LiveLabs workshop.



If you are planning on attending to Oracle Cloud World this year and you want to learn more about this consider attending my session.

LRN3519: Deploy and Manage Oracle Databases with Ansible, Oracle Enterprise Manager 

See you in Vegas!!!

Thanks,
Alfredo

Oracle Enterprise Manager 13.5 RU7 Now Available!

Before I start this post I want to give a big shoutout to FeedSpot for ranking my blog in the Top 100 Best Oracle Blogs and Websites. It’s honestly an honor. Since the start of this blog the main goal was and still is to share knowledge with the community.

Being an Oracle ACE Alumni and able to share all this information with you is a privilege.

If you want to take a look at the list click below.



Going back to this post. Last week RU7 for Oracle Enterprise Manager (EM) was released. In this post I’ll share information on new features that may want you think on applying this RU as soon as possible.

Oracle EM has a monthly release update model. With every RU there are bug fixes and new features released in the product.

Some of the features that I’m eager to use are:

  • Dynamic Runbooks. They allow you to add your SOP’s to EM for consistent incident resolution.
  • Migration Workbench TTS support. Migration Workbench now supports the option to control the incremental backups and the final migration phase.
  • Fleet Maintenance scheduling enhancement. Now EM allows you to better control the Fleet Maintenance scheduling steps.
  • New STIG standard version. STIG version 2 Release 3 is now available in RU7.

For additional information on this release update go to below Oracle blog:



The patching procedure is very straight forward and will cover it very soon in the next post.

In the meantime I strongly recommend to review the latest Oracle EM documentation and get familiar with the process.

Happy patching,
Alfredo

Oracle Enterprise Manager 13c: Extensibility and Interoperability for DevOps

DevOps adoption helps customers to automate and standardize processes, accelerate software delivery while keeping control of the operations, monitoring and lifecycle management of the environment. Enterprises select a tool or toolset in order to support desired business goals. However, these selected tools often require deeper automation from specialized management tools like Oracle Enterprise Manager 13c.

Oracle Enterprise Manager 13c by making use of the Database Lifecycle Management and the Cloud Management packs, enables Database Administrators, Developers and DevOps teams to automate the monitoring and management of the Oracle Database. Some of the use cases inlclude:

  • Monitoring Setup Automation
  • Provision Oracle Databases
  • Database Lifecycle Automation
  • Automated Database Cloning
  • Metering and Chargeback
  • Mask Sensitive Data
DevOps Tools and Oracle Enterprise Manager 13c

Monitoring Setup Automation

Monitoring begins as soon as you deploy the Oracle Enterprise Manager agent to the target environment. There are several tasks performed by the administrator in order to achieve the desired level of monitoring for a particular target. Using DevOps orchestration tools combined with EM 13c, customers can automate the monitoring setup right after the OS environment is provisioned.

In a previous post I went through the details on how to deploy the EM Agent using Ansible.



Provision Oracle Databases

DevOps orchestration tools like Ansible, Chef, Terraform, etc., can easily integrate with EM 13c in order to provision Oracle databases. These can be either single instance, multi-tenant container databases, pluggable databases, schemas or databases running on high availability clusters and Dataguard configurations. EM 13c implements pre-checks, best practices and processes to provision all these configurations in a secure, automated and controlled fashion.



Database Lifecycle Automation

Once provisioned, Oracle databases need to be well maintained and secured by regularly applying patches and eventually upgraded to a higher release.  All these database lifecycle activity tasks can be automated by integrating EM 13c with orchestration DevOps tools. Oracle’s Database Lifecycle Management pack enables customers to streamline these tasks using their DevOps workflows.



Automated Database Cloning

Oracle EM 13c help to automate the process of cloning databases using different options as a source. Like RMAN backups, snapshots or live data. EM 13c Snap Clone simplifies the cloning process by creating space efficient clones. DevOps orchestration tools can make use of this functionality to provision fully functional copies of Oracle databases in minutes by cloning databases while keeping the storage needs to a minimum.

Metering and Chargeback

DevOps tools automate the delivery of Oracle databases, however the lack of accurate costing of usage may result in no knowledge of consumption trends affecting the ability to measure the business value of IT investments. EM 13c Chargeback tools can be enabled so resource usage and allocation of databases provisioned in conjunction with DevOps tools is properly tracked and presented through comprehensive reports to business units. Chargeback enables consumers to adjust their IT consumption and utilization rates, hence driving consumer accountability.

Data Masking

DevOps teams often require databases to be provisioned with data cloned from production sources or asking for continuous refresh from these sources. Database and security organizations are required to limit exposure of sensitive information while delivering database environments. EM 13c enables administrators to provide to DevOps teams an automated and secure method that is easily integrated with orchestration tools. Data masking works on sensitive data by replacing it with realistic but scrubbed data based on masking rules.

I will cover more step by step examples of these use cases in future posts.

Thanks,
Alfredo