If you are migrating to multi-tenant architecture of the Oracle database, you’ll find out that in order to connect to the pluggable database (PDB) you must use the service name for the PDB. But, what if you have a legacy application that can only use SID instead of the service name?
For that, you can enable the parameter USE_SID_AS_SERVICE_LISTENER in the listener configuration, so the service name will act as an SID.
Problem solved, right? Well, the problem is now with applications that require the service name and not the SID. One of them is Oracle Enterprise Manager (EM). If you turn on this parameter at the listener level then, you won’t be able to connect to the databases and they will show down (red) in the EM console.
So, what can we do? In EM 13.5 RU16 or higher there’s functionality that allows EM to connect to the database by using an Agent preferred connect descriptor. This way, you can provide the connect descriptor using the service name.
One important note to mention is that you must have RU16 or higher applied to both, the OMS and the Agents in order to enable this functionality.
Follow the My Oracle Support Note “EM 13c: How To Enable Agent Preferred Connect Descriptor Support For RAC Instance, Single Instance DB (Doc ID 2963079.1) ” to enable this functionality. You will have to restart your OMS, so be prepared.
After you enable this setting, navigate to the Monitoring Configuration of your target and look for the new Agent preferred descriptor section. It looks like the image below.
Add the proper connect descriptor and viola! EM is now able to connect to your target.
Is always recommended to turn the USE_SID_AS_SERVICE_LISTENER parameter off after migrations and upgrades. Sometimes this is not an option if you have legacy applications though. This feature allows you connect your EM when the parameter USE_SID_AS_SERVICE_LISTENER must be turned on.
In this blog I want to provide some important steps if you want to monitor Oracle databases running on AWS RDS.
The first step is to have an Oracle Enterprise Manager (EM) installation. This can be on-prem or running on EC2. If you need to setup EM in EC2 you can use one of the pre-built images that already contains an Oracle database. You are going to need this database for the EM repository. One quick example can be the Oracle database 19c image by cloudimg running on Oracle Linux. Just make sure you add enough space in the filesystem so you can also install the EM software.
Once the EC2 instance is up and running, go ahead and patch the Oracle database with the latest RU (RU28 as of today).
Download the Oracle EM 13.5 software from oracle.com and also download the latest RU available (RU27 as of today).
Install EM 13.5 and patch it with RU27. Follow below videos if you need additional guidance on how to install and patch EM.
Next we are going to follow the instructions that RDS has available for the EM agent installation. Remember that because there’s no actual access to the RDS host, we can’t install the EM agent using the traditional way.
Verify all the network pre-reqs so your RDS instance can actually reach your EM installation running on EC2. If you EM is running on-prem then a VPN setup will be required.
Now is time to create an option group once all the network pre-reqs are met. The option group is how we actually ask the RDS console to provide us the option to deploy an EM agent. The option for the option group needs to be OEM_AGENT.
Before you deploy the option group to your RDS instance, you will have to provide information about your EM instance, including:
EM Host Name
EM upload port
EM registration key
Then deploy the option group to your RDS instance.
Once the deployment finishes, you will actually have the EM agent up and running in your RDS instance and showing up in the EM console.
The next step is to discover your DB instance and the listener. Follow below link for the documentation.
In this post I want to share the required steps to discover ExaCC or ExaCS in Oracle Enterprise Manager (EM). I’ll provide as much detail as possible.
Make sure your EM is at least on 13.5 RU 16. If not, please apply RU 16 to EM. Follow this link if you want to know how to patch EM. HERE!
You need to designate a “monitoring” agent for the discovery. Is recommended that this agent sits outside your Exadata rack and has access to the OCI REST APIs. More information HERE!
Make sure this agent has both the Database and the Exadata plugins deployed and is patched to the same RU level as the OMS. HERE!
The agent must be able to reach the OCI REST APIs. There are 3 ways to achieve this. a) You can use a Proxy b) You can use the OCI Management Gateway c) You can have direct network connection
Below is a list of APIs that you will need access to: Either grant access to (*.oci.oraclecloud.com) or individual URLS: https://query.<oci_region>.oci.oraclecloud.com https://identity.<oci_region>.oci.oraclecloud.com https://database.<oci_region>.oci.oraclecloud.com https://wss.exacc.<oci_region>.oci.oraclecloud.com https://management-agent.<oci_region>.oci.oraclecloud.com https://certificatesmanagement.<oci_region>.oci.oraclecloud.com https://certificates.<oci_region>.oci.oraclecloud.com https://telemetry-ingestion.<oci_region>.oci.oraclecloud.com https://auth.<oci_region>.oci.oraclecloud.com https://objectstorage.<oci_region>.oci.oraclecloud.com
You may also test that connectivity by executing: $ curl -v https://query.<oci_region>.oci.oraclecloud.com
Create or use an OCI account that has access to read your Exadata racks. These are the policies I have used: allow group <domain/group name> to read database-family in compartment <compartment name> You can find more information about the required policies HERE!
Setup API Keys for authentication. Please follow instructions in the MOS note below: EM13.5: Manage OCI Connectivity Named Credentials Test Failed with Invalid Private Key. (Doc ID 2792126.1)
Create an EM Named Credential using the API Keys created on the previous step. More details HERE!
You also need the proper Storage Server credentials. If you don specify them during the discovery process, Storage Server targets will not be discovered. Instructions on how to retrieve this credential can be found HERE! If you can’t retrieve these credentials, please open a Service Request with support.
Discover your Exadata Infrastructure following the discovery wizard. More information HERE!
After finishing the discovery, you should have your Exadata Cloud target in EM. Please wait between 15 to 20 minutes for the target information to be populated.
If your network setup does not allow direct connectivity to the OCI REST APIs then you will have to use an internal Proxy or use the OCI Management Gateway. More information about the OCI Management Gateway can be found HERE!
Update: You will also have to modify 2 parameters. One at the OMS level and one at the agent level. Please follow below MOS notes:
OEM 13c : SSH Key Credential Test On a Host Target Fails With “Remote operation timed out.” (Doc ID 2415262.1) EM 13.2: Agent Patching From EM Console Fails With Error “concurrent job tasks limit (50) has been reached” (Doc ID 2476803.1)
This post continues the thread of DevOps and automation for the Oracle Database. During the last couple of months I’ve seen more interest in deploying Oracle Databases on Docker containers. Even more now that Oracle released full support of RAC Databases running on Docker.
After you deploy your first Oracle Database on Docker, several questions come into mind; like:
How do I monitor the status of the Oracle Database?
How do I manage the performance of the Oracle Database and the SQL’s being executed on it?
How do I alert on issues on a timely manner?
Well, if you have the same questions or concerns… you are not alone!
On this post, I’m trying to explain the process I followed in order to deploy an Oracle Enterprise Manager agent on a Docker container and how I do monitor and manage the Oracle Database running on it.
First things first. Just a quick recap of my environment.
Oracle Linux 7 host running Docker 19.03
Create a MACVLAN network on Docker. My network adapter is ens3 on the host running Docker.
Review the startup process and the creation of the database
sudo docker logs -f ol7-orcl
At this point we have our Docker container and the ORCL database up and running. But now I need a way to connect my ORCL database to the external world. Because I’m running a MACVLAN, I can create another Docker container in the same network and install Oracle Enterprise Manager (EM) in order to monitor it. Or I can also install Oracle EM on the host running Docker and setup a network link and a route to my Docker container. Let’s do the second option.
sudo ip link add mydocker-net link ens3 type macvlan mode bridge
sudo ip addr add 192.168.56.1/32 dev mydocker-net
sudo ip link set mydocker-net up
sudo ip route add 192.168.56.0/24 dev mydocker-net
I now have connectivity between my host and the Docker container.
$ ping 192.168.56.101
PING 192.168.56.101 (192.168.56.101) 56(84) bytes of data.
64 bytes from 192.168.56.101: icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from 192.168.56.101: icmp_seq=2 ttl=64 time=0.070 ms
64 bytes from 192.168.56.101: icmp_seq=3 ttl=64 time=0.062 ms
64 bytes from 192.168.56.101: icmp_seq=4 ttl=64 time=0.073 ms
64 bytes from 192.168.56.101: icmp_seq=5 ttl=64 time=0.070 ms
sh-4.2$ ping 192.168.56.1
PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data.
64 bytes from 192.168.56.1: icmp_seq=1 ttl=64 time=0.064 ms
64 bytes from 192.168.56.1: icmp_seq=2 ttl=64 time=0.079 ms
64 bytes from 192.168.56.1: icmp_seq=3 ttl=64 time=0.043 ms
64 bytes from 192.168.56.1: icmp_seq=4 ttl=64 time=0.082 ms
64 bytes from 192.168.56.1: icmp_seq=5 ttl=64 time=0.077 ms
The next step is to have name resolution setup between both hosts. For the Docker container it was easy to setup the full host name into the external DNS service. For the host resolution inside the Docker container you can use static entries in the /etc/hosts file or use the Docker embedded DNS server. For my exercise, I setup an entry in my /etc/hosts file providing the host name and the IP address of the host where EM is running.
The rest of the process is very straight forward. I use the silent install option for the Oracle EM agent. Below link has the steps to download the agent binaries for your specific platform and version.
After you login to the container just follow the steps in the provided agent install guide. Update the response file with the required information and run agentDeploy.sh. Make sure that the Agent Home resides in the Docker volume created. For this exercise will be inside /u01 directory.
After the agent deploy succeeds disconnect from the Docker session and re-connect as root.
sudo docker exec -it --user root ol7-orcl /bin/sh
Execute root.sh inside the Agent Home directory.
Let’s now verify that our EM has a new host target named ol7-orcl and is up and running.
Docker Container ol7-orcl on Oracle Enterprise Manager
The next step is to discover the ORCL database in EM. You can change the DBSNMP password by logging to the Docker container, then use SQLPlus to change the password and unlock the account.
alter user dbsnmp identified by <password> account unlock;
After the database discovery, verify the database target in the EM console.
ORCL database home page
Now that you have both the host target and the database target you can set metrics, thresholds and notifications on them.
Another important option is that if you have Diagnostics and Tuning packs, you can also monitor performance, get AWR/ASH reports and use EM to graphically analyze performance issues.
ORCL performance information
Hope this post helps understand the options you have in order to setup the required monitoring for the Oracle databases running on Docker containers.
I wanted to create a video series covering the basics of Oracle Enterprise Manager 13c. These series are designed to provide insights on how to install, configure and manage Oracle Enterprise Manager 13.5.
This week we are going to cover the patching of Oracle Enterprise Manager 13.5.
I wanted to create a video series covering the basics of Oracle Enterprise Manager 13c. These series are designed to provide insights on how to install, configure and manage Oracle Enterprise Manager 13.5.
This week we are going to cover the installation phase of Oracle Enterprise Manager 13.5.
In a previous post I show you how you can integrate DevOps automation and orchestration tools to provision Oracle databases by leveraging Enterprise Manager (EM) and the Cloud Management Pack (CMP). Once provisioned, databases need to be fully maintained in terms of monitoring but most precisely patching.
Patching databases decreases the risk of breach by timely mitigating vulnerabilities, but in can be a daunting task for organizations. Manual patching is time consuming and error prone. Home grown scripts are difficult to maintain and they increase maintenance cost. So the question is, how can I automate the patching process and even better, how can I integrate it with my current orchestration workflow?
Let me explain how you can achieve all this by making use of Oracle’s Database Lifecycle Management Pack (DBLM) and CMP. DBLM’s Fleet Maintenance help us patch and upgrade Oracle databases at scale with minimum downtime. It makes use of Gold Images and out-of-place patching in order to achieve this.
Fleet Maintenance Benefits
All this functionality can be integrated with CMP’s DBaaS in order to provide and end-to-end automation solution. DBaaS exposes REST APIs that we could then call using the automation tool of choice. Database Administrators, end users or 3rd party tools can then use these features to patch Oracle databases.
DBaaS Automation Diagram
Do you want to learn more about this and even be able to try it? We’ve created an Oracle LiveLabs that cover’s all this functionality. This lab will guide you through the request of a PDB, setup DBaaS configuration, setup Fleet Maintenance and finally patch the PDB.
Follow the link below for the Oracle LiveLabs workshop.
Before I start this post I want to give a big shoutout to FeedSpot for ranking my blog in the Top 100 Best Oracle Blogs and Websites. It’s honestly an honor. Since the start of this blog the main goal was and still is to share knowledge with the community.
Being an Oracle ACE Alumni and able to share all this information with you is a privilege.
If you want to take a look at the list click below.
Going back to this post. Last week RU7 for Oracle Enterprise Manager (EM) was released. In this post I’ll share information on new features that may want you think on applying this RU as soon as possible.
Oracle EM has a monthly release update model. With every RU there are bug fixes and new features released in the product.
Some of the features that I’m eager to use are:
Dynamic Runbooks. They allow you to add your SOP’s to EM for consistent incident resolution.
Migration Workbench TTS support. Migration Workbench now supports the option to control the incremental backups and the final migration phase.
Fleet Maintenance scheduling enhancement. Now EM allows you to better control the Fleet Maintenance scheduling steps.
New STIG standard version. STIG version 2 Release 3 is now available in RU7.
For additional information on this release update go to below Oracle blog:
IT Security is popular topic nowadays! We constantly hear news about data breaches, ransomware, malware, unauthorized access to IT systems, etc. IT organizations are constantly looking to keep their systems, networks and data safe and secure.
Today’s blog is about how Oracle Enterprise Manager (EM) can help Database Administrators to secure and harden the Oracle Databases they manage along with the hosts those databases are running on.
First things first. I strongly recommend to review the Oracle Database 19c Security Guide. This guide provides guidelines on how to secure the Oracle Database, harden the DB access, secure and encrypt the DB data and so.
Now let’s discuss some areas that database administrators should also look at in order to improve their security posture:
Timely apply security patches
Monitor database configuration and detect misconfigurations
Use industry and regulatory standards like STIG and CIS for the Oracle Database
All the features that we will be discussing today are part of the Oracle Database Lifecycle Management pack. This pack requires an additional license.
Timely apply security patches
Fleet Maintenance (FM) enables administrators to automate lifecycle activities such as database patching and upgrades. FM is a gold image subscription based model that allows to patch databases with minimum downtime by using out-of-place patching mechanisms. In-place patching is also available if you need to apply an emergency on-off patch.
Administrators have the ability to customize the patching process by adding custom pre/post scripts to patching operations. FM supports single instance, RAC databases, Grid Infrastructure, Multitenant and Data Guard configurations.
One thing to mention is the ability to get security patch recommendations as soon as they are published. EM connects to My Oracle Support (MOS) and checks for the availability of new security patches. As soon as a new security patch is released EM will let you know if your DB estate is compliant or not in terms of these patches.
Monitor database configuration and detect misconfigurations
Configuration and Drift Management helps you monitor the configuration of your DB estate, the hosts on where those DB’s are running as well as the Oracle Homes (OH) for those installations. EM allows you to create your own configuration templates based on the configuration settings you need to enforce. Any misconfiguration or drift away of your template will be automatically reported via the Drift Management dashboard and you can also receive alerts if you choose to.
Corrective Actions (CA) can also be created to automatically fix this misconfigurations in order to comply with the templates and reduce security risks.
How many times administrators issued an ALTER SYSTEM command with SPFILE scope and forgot about it? Well, you will know next time you bring your DB up after maintenance. EM helps you detect these changes before they become a production issue. It also help you track the history of configuration changes, save configuration information at a given time and also allows you to use this configuration information to be compared between targets.
Have you wonder, how many OH’s we have with this specific one-off patch?
How many DB’s we have running on this specific OS version?
Well, EM can help you answer all these questions using this configuration data.
One thing worth mentioning is that EM comes with hundreds on configuration collections. If you need to gather a very specific configuration that is not available out-of-the-box, you can create your own configuration extension and collect this automatically.
Use industry and regulatory standards like STIG and CIS for the Oracle Database
EM provides compliance standards to help customers meet regulatory standards like STIG and CIS. Oracle’s best practices are also included within the compliance framework. There are two available options for analysis.
Rule based analysis
Real-time change
Each option allow administrators understand where attentions needs to be put in order to harden the DB estate.
Using the compliance framework, EM will provide a score to each associated target along with all the violations that need to be remediated after each evaluation.
DevOps adoption helps customers to automate and standardize processes, accelerate software delivery while keeping control of the operations, monitoring and lifecycle management of the environment. Enterprises select a tool or toolset in order to support desired business goals. However, these selected tools often require deeper automation from specialized management tools like Oracle Enterprise Manager 13c.
Oracle Enterprise Manager 13c by making use of the Database Lifecycle Management and the Cloud Management packs, enables Database Administrators, Developers and DevOps teams to automate the monitoring and management of the Oracle Database. Some of the use cases inlclude:
Monitoring Setup Automation
Provision Oracle Databases
Database Lifecycle Automation
Automated Database Cloning
Metering and Chargeback
Mask Sensitive Data
DevOps Tools and Oracle Enterprise Manager 13c
Monitoring Setup Automation
Monitoring begins as soon as you deploy the Oracle Enterprise Manager agent to the target environment. There are several tasks performed by the administrator in order to achieve the desired level of monitoring for a particular target. Using DevOps orchestration tools combined with EM 13c, customers can automate the monitoring setup right after the OS environment is provisioned.
In a previous post I went through the details on how to deploy the EM Agent using Ansible.
DevOps orchestration tools like Ansible, Chef, Terraform, etc., can easily integrate with EM 13c in order to provision Oracle databases. These can be either single instance, multi-tenant container databases, pluggable databases, schemas or databases running on high availability clusters and Dataguard configurations. EM 13c implements pre-checks, best practices and processes to provision all these configurations in a secure, automated and controlled fashion.
Once provisioned, Oracle databases need to be well maintained and secured by regularly applying patches and eventually upgraded to a higher release. All these database lifecycle activity tasks can be automated by integrating EM 13c with orchestration DevOps tools. Oracle’s Database Lifecycle Management pack enables customers to streamline these tasks using their DevOps workflows.
Oracle EM 13c help to automate the process of cloning databases using different options as a source. Like RMAN backups, snapshots or live data. EM 13c Snap Clone simplifies the cloning process by creating space efficient clones. DevOps orchestration tools can make use of this functionality to provision fully functional copies of Oracle databases in minutes by cloning databases while keeping the storage needs to a minimum.
Metering and Chargeback
DevOps tools automate the delivery of Oracle databases, however the lack of accurate costing of usage may result in no knowledge of consumption trends affecting the ability to measure the business value of IT investments. EM 13c Chargeback tools can be enabled so resource usage and allocation of databases provisioned in conjunction with DevOps tools is properly tracked and presented through comprehensive reports to business units. Chargeback enables consumers to adjust their IT consumption and utilization rates, hence driving consumer accountability.
Data Masking
DevOps teams often require databases to be provisioned with data cloned from production sources or asking for continuous refresh from these sources. Database and security organizations are required to limit exposure of sensitive information while delivering database environments. EM 13c enables administrators to provide to DevOps teams an automated and secure method that is easily integrated with orchestration tools. Data masking works on sensitive data by replacing it with realistic but scrubbed data based on masking rules.
I will cover more step by step examples of these use cases in future posts.