efm_sql_command: Postgres database extension for EFM (EDB Failover Manager 2.1.x)

 

EDB Failover manager (EFM) continues to gain popularity among Postgres DBAs. EDB customers are using this tool to build a highly available EDB Postgres platform.

EFM’s primarily role is to monitor Postgres Clusters, notify the DBA of any failed clusters and automatically promote the standby cluster to function as a master.

Here are some of the high-level features of EFM:

  1. Automatic failover
  2. Switchover to a standby (s) close to master in terms of transaction xlog
  3. Set priority of standby (s)
  4. Customize notifications/alerts
  5. Transfer VIP to the new master with failover (provided, the master has been configured with VIP)

Utility “efm” is used by the DBAs to gain control of these actions. However, the need to connect to EFM cluster servers/nodes using ssh continues to not optimize the DBA experience.

Keeping this pain point in mind, we have developed “efm_sql_command” extension for the EFM 2.1.x utility command using the inherent Postgres interface. DBAs no longer need to ssh to EFM node to allow/disallow new/old nodes for EFM clusters. Additionally, through the Postgres interface itself, EFM can also be integrated with other monitoring systems like EDB Postgres Enterprise Manager, Nagios, etc.

efm_sql_command extension provides following functions:

  • efm_sql_command.efm_allow_node(‘ip address’);

This functions takes IP address as an argument and notifies EFM agents to allow the specific IP address to be part of a EFM cluster. It returns status of EFM command in 0/1. 0 means successful and 1 means failed.DBAs now can use SQL interface to notify EFM for allowing the new standby (which they are still building) without ssh to one of the EFM nodes.

  • efm_sql_command.efm_disallow_node(‘ip address’);

Similar to efm_allow_node, this function notifies to EFM agents to disallow the IP address from EFM cluster. Function report the status of the command in 0 and 1, where 0 means successfully notified and 1 means failed.

  • efm_sql_command.efm_failover();

Function notifies the EFM cluster agents to perform manual failover of master to closest standby. The function returns 0/1 status, where 0 means successfully notified EFM cluster agents and 1 means failed to notify agents.

  • efm_sql_command.efm_resume_monitoring();

If an agent is not monitoring the local EDB Postgres database, this function makes EFM agent start monitoring the local EDB Postgres database.

  • efm_sql_command.efm_set_priority(‘ip address’, ‘priority’);

Set failover priority for standby and return the status of the command in 0/1, where 0 means success and 1 means failed.

  • efm_sql_command.efm_switchover();

This function performs the switchover to closest standby of master and reconfigures the master as a new standby.

  • efm_sql_command.efm_local_properties

This a view using which DBA can view the efm properties.

  • efm_sql_command.efm_nodes_details:

This view provides the details of each node in EFM cluster. A user can use this for other the purpose. For example checking the status of standby(s), How far are the standbys from a master? etc.

  • efm_sql_command.efm_cluster_status

This function takes following arguments as text:
‘text’: To print the status of EFM cluster in TEXT
‘json’: To print the status of EFM cluster in JSON format.

This extension also gives following GUC, which DBAs can set at cluster/database/user level

ALTER SYSTEM SET efm.cluster_name TO 'clustername';
ALTER DATABASE  SET efm.cluster_name TO 'clustername';
ALTER USER  SET efm.cluster_name TO 'clustername';

Following are some snapshots of efm_sql_command’s functions

CREATE EXTENSION efm_sql_command;
edb=# select efm_extension.efm_cluster_status('text');
INFO: efm command is available
efm_cluster_status
------------------------------------------------------------------------
Cluster Status: efm
VIP:
 
Agent Type Address Agent DB Info
--------------------------------------------------------------
Idle 172.17.0.2 UP UNKNOWN
 
Allowed node host list:
172.17.0.2
 
Membership coordinator: 172.17.0.2
 
Standby priority host list:
(List is empty.)
 
Promote Status:
 
 
Idle Node Status (idle nodes ignored in XLog location comparisons):
 
Address XLog Loc Info
--------------------------------------------------------------
172.17.0.2 0/35BFC10 DB is not in recovery.
(23 rows)
edb=# select jsonb_pretty(efm_extension.efm_cluster_status('json')::jsonb);
INFO:  efm command is available 
                         jsonb_pretty                         
--------------------------------------------------------------
 {                                                           +
     "VIP": "",                                              +
     "nodes": {                                              +
         "172.17.0.2": {                                     +
             "db": "UNKNOWN",                                +
             "info": " ",                                    +
             "type": "Idle",                                 +
             "xlog": "0/35BFC10",                            +
             "agent": "UP",                                  +
             "xloginfo": "DB is not in recovery."            +
         }                                                   +
     },                                                      +
     "messages": [                                           +
         "Did not find XLog location for any non-idle nodes."+
     ],                                                      +
     "allowednodes": [                                       +
         "(List",                                            +
         "is",                                               +
         "empty.)"                                           +
     ],                                                      +
     "minimumstandbys": 0,                                   +
     "failoverpriority": [                                   +
     ],                                                      +
     "membershipcoordinator": "172.17.0.2"                   +
 }
(1 row)
edb=# select efm_extension.efm_allow_node('172.17.0.2');
INFO: efm command is available
efm_allow_node
----------------
0
(1 row)
edb=# select efm_extension.efm_disallow_node('172.17.0.2');
INFO: efm command is available
efm_disallow_node
-------------------
0
(1 row)
edb=# select * from efm_extension.efm_nodes_details ;
INFO: efm command is available
node_ip | property | value
------------+----------+--------------------------
172.17.0.2 | db | "UNKNOWN"
172.17.0.2 | info | " "
172.17.0.2 | type | "Idle"
172.17.0.2 | xlog | "0/35BC388"
172.17.0.2 | agent | "UP"
172.17.0.2 | xloginfo | "DB is not in recovery."
(6 rows)
edb=# select * from efm_extension.efm_local_properties ;
           name            |              value              
---------------------------+----------------------------------
 efm.license               |
 db.user                   | efm
 db.password.encrypted     | 074b627bf50168881d246c5dd32fd8d0
 db.port                   | 5444
 db.database               | edb
 db.service.owner          | enterprisedb
 db.service.name           | edb-as-9.6
 db.bin                    | /usr/edb/as9.6/bin
 db.recovery.conf.dir      | /pgdata
 jdbc.ssl                  | false
 jdbc.ssl.mode             | verify-ca
 user.email                | vibhor.kumar@enterprisedb.com
 script.notification       |
 bind.address              | 172.17.0.2:5430
 admin.port                | 5431
 is.witness                | false
 local.period              | 10
 local.timeout             | 60
 local.timeout.final       | 10
 remote.timeout            | 10
 node.timeout              | 50
 pingServerIp              | 8.8.8.8
 pingServerCommand         | /bin/ping -q -c3 -w5
 auto.allow.hosts          | true
 db.reuse.connection.count | 0
 auto.failover             | true
 auto.reconfigure          | true
 promotable                | true
 minimum.standbys          | 0
 recovery.check.period     | 2
 auto.resume.period        | 0
 virtualIp                 |
 virtualIp.interface       |
 virtualIp.netmask         |
 script.fence              |
 script.post.promotion     |
 script.resumed            |
 script.db.failure         |
 script.master.isolated    |
 sudo.command              | sudo
 sudo.user.command         | sudo -u %u
 jgroups.loglevel          | INFO
 efm.loglevel              | INFO
 jvm.options               | -Xmx32m
(44 rows)

I look forward to your comments on this topic. Click here to request a full demo of EFM.

Meet BART – A New Tool for Backup And Recovery Management

EnterpriseDB recently launched a new tool for backup and recovery – named simply EDB Backup and Recovery Tool, or BART. This tool makes the DBA’s life easier by simplifying the tasks for managing their Postgres physical backup and recovery tasks, whether they are PostgreSQL or Postgres Plus Advanced Server deployments.

BART has the following advantages over custom scripts for managing backups:

1. It’s stable and it uses the tool pg_basebackup to take a physical backup. This tool has been well defined and is well-supported by the PostgreSQL community.

2. It catalogs all of the backups users are taking, which is important in terms of:
    i. Listing the type of backups used
   ii. Listing the status of those backups with server information.

3. BART also provides functionality to restore backups, with all required archived WAL files. So automation around this tool will make DBAs’ lives easier for restore and recovery.

4. BART provides an option to validate your backup by using checksum. This is useful for confirming you took a valid backup and it is not corrupted at disk level.

5. BART provides an option to define your retention policy around the backups you are keeping.

Given all of the above advantages, I decided to give this new tool a try and share some tips. To get started, you need the following prerequisites:

1. BART currently requires a Linux 64 bit platform, CentOS 6.x or RHEL 6.x
2. Need to have password-less, direct SSH access to the target machine where you want to restore backups as well as the database servers you want backed up
3. Install the Postgres Plus Advanced Server or PostgreSQL binaries for pg_basebackup

Yum or rpm

To install this tool, you have two options that I will explore below:

1. Yum command
2. Rpm command.

Using the yum command:

To perform a yum command installation, BART users can ask EDB for credentials to the EnterpriseDB yum repository and configure the their local yum repository as follows:

echo "[tools]
name=EnterpriseDB Tools
baseurl=http://username:password@yum.enterprisedb.com/tools/redhat/rhel-$releasever-$basearch
enabled=1
gpgcheck=0" > /etc/yum.repos.d/edbtools.repo

After creating the yum repo, the user can execute the following command to install BART:

 yum install edb-bart

If the user doesn’t want to install the EDB Backup and Recovery Tool using the yum command, then the user can download a free standing rpm using the link below from EDB’s website:

http://www.enterprisedb.com/downloads/postgres-postgresql-downloads

and then enter the rpm install command as follows:

rpm -ivh edb-bart-1.0.1-1.rhel6.x86_64.rpm

After installing BART using the above commands, the user can see the binaries in the directory:/usr/edb-bart-1.0/bin and a sample BART configuration file in /usr/edb-bart-1.0/etc

That’s a very easy installation.

For more information on configuring BART Host and Database Host, the following are some documents that will help:
1. pg_basebackup configuration for PostgreSQL:
http://www.postgresql.org/docs/current/static/app-pgbasebackup.html

2. For direct password less ssh configuration user can refer following link
http://www.enterprisedb.com/docs/en/1.0/bart/EDB_Backup_and_Recovery_Tool_Guide-17.htm#P1008_76316

After the installation of the BART binaries, the user also has to create a BART configuration file.

The following is a sample configuration file for BART:

[BART]
bart-host= enterprisedb@127.0.0.1
backup_path = /opt/backup
pg_basebackup_path = /usr/ppas-9.4/bin/pg_basebackup
logfile = /tmp/bart.log

[PG]
host = 127.0.0.1
port = 5432
user = postgres
description = "Postgres server"

[PPAS94]
host = 127.0.0.1
port = 5444
user = enterprisedb
description = "PPAS 94 server"

Global Configuration Settings

Content under the [BART] tag are called global configuration settings. Under this tag are the following:

1. bart-host: the IP address of the host on which BART is installed. The value for this parameter must be specified in the form: bart_user@bart_host_address, where bart_user is the operating system user account on the BART host that is used to run BART and owns the BART backup catalog directory. bart_host_address is the IP address of the BART host.

2. backup_path: specifies the file system parent directory where all BART database server base backups and archived WAL files are stored. This parameter is required.

3. pg_basebackup_path: specifies the path to the pg_basebackup program of the Postgres database server installed on the BART host.

4. log file: specifies the path to the BART log file. This parameter is optional. If no path to a log file is specified after logfile =, or if the parameter is commented out, BART does not create a log file.

The remaining part of configuration file is self-explanatory. The TAG: [PG]/[PPAS94] part is content for servers which the user wants to back up.

Pg_basebackup Settings

After performing the above configuration on the Backup Server, the user has to do set following settings on the servers that they want to back up. Below are the settings for enabling backup using pg_basebackup.

The user has to set a few parameters in PostgreSQL postgresql.conf file, which he wants to backup:

1. wal_level parameter to archive or hot_standby.
2. archive_mode=on
3. archive_command setting.
4. max_wal_senders to 1 or more than one, since pg_basebackup uses the replication protocol to copy data directory.

For more information on each setting please refer to the following:
1. wal_level:
http://www.postgresql.org/docs/9.4/static/runtime-config-wal.html

2. archive_mode and archive_command:
http://www.postgresql.org/docs/9.4/static/runtime-config-wal.html#RUNTIME-CONFIG-WAL-ARCHIVING

3. max_wal_senders:
http://www.postgresql.org/docs/9.4/static/runtime-config-replication.html
http://www.enterprisedb.com/docs/en/1.0/bart/EDB_Backup_and_Recovery_Tool_Guide-19.htm#TopOfPage

With the above settings, the user then needs to update the pg_hba.conf file for the replication connection.

Note: The above settings are for pg_basebackup to take backups using replication protocols. In case users need more information about pg_basebackup and settings, please use the above mentioned link

How BART Works

Now, since we have configured both servers, let’s have a look how BART works.

The following command executes a backup:

 bart -c bart.cfg BACKUP -s ppas94

And below is the output:

[bart@localhost ~]$ bart -c bart.cfg BACKUP -s ppas94

INFO:  creating backup for server 'ppas94'
INFO:  backup identifier: '1413852137762'
6394456/6394456 kB (100%), 1/1 tablespace

INFO:  backup checksum: 7f49ea9653511308710c174f22ec765d
INFO:  backup completed successfully
[bart@localhost ~]$ 

That was an easy way to take a backup. The DBA can also create a job to execute the above command to take backups.

If the user wants to list the backup using BART, the user can use the option SHOW-BACKUPS:

[bart@localhost ~]$ bart -c bart.cfg SHOW-BACKUPS -s ppas94
 Server Name   Backup ID       Backup Time           Backup Size  
                                                                  
 ppas94        1413852137762   2014-10-20 17:43:41   6244.59 MB   

This is useful for knowing what backups a user has available for recovery. The above command gives important information:

1.	Backup ID: It’s a unique ID for the physical backup
2.	Backup Time: Time when backup was taken
3.	Backup Size: Size of backup

This information is useful when a user wants to plan for recovery using backup. This way, the user can also plan for disk size.

Sometimes a user wants to verify their backup state. VERIFY-CHKSUM option is useful in this case:

[bart@localhost ~]$ bart -c bart.cfg VERIFY-CHKSUM -s ppas94 -i 1413852137762
 Server Name   Backup ID       Verify  
                                       
 ppas94        1413852137762   OK      

I have to say, after putting EDB BART through its paces, I think DBAs will enjoy having such a great tool for making Backup Management easy.

In my next post, I will blog about the Recovery process.

JAVA Program for JDBC Driver Version and Database Information

I had seen people ask Questions about finding the edb-jdbc/postgresql driver version with Database Version. So, I thought to give one java script which can be use to find Database Details with edb-jdbc version.

Following is a JAVA Code which can be use to find the Database Version and EDB-JDBC/postgresql-jdbc Version:

File Name: DBinfo.java

import java.sql.*;

public class DBinfo
{
  public static void main(String[] args)
  {
    try
    {
	  Class.forName("com.edb.Driver");
	  Connection con = 
DriverManager.getConnection("jdbc:edb://localhost:5444/edb",
	  "enterprisedb","edb"); // Advanced Server Database Connection Information	
 DatabaseMetaData dbmd = con.getMetaData();  
    
     System.out.println("=====  Database info =====");  
     System.out.println("DatabaseProductName: " + dbmd.getDatabaseProductName() );  
     System.out.println("DatabaseProductVersion: " + dbmd.getDatabaseProductVersion() );  
     System.out.println("DatabaseMajorVersion: " + dbmd.getDatabaseMajorVersion() );  
     System.out.println("DatabaseMinorVersion: " + dbmd.getDatabaseMinorVersion() );  
     System.out.println("=====  Driver info =====");  
     System.out.println("DriverName: " + dbmd.getDriverName() );  
     System.out.println("DriverVersion: " + dbmd.getDriverVersion() );  
     System.out.println("DriverMajorVersion: " + dbmd.getDriverMajorVersion() );  
     System.out.println("DriverMinorVersion: " + dbmd.getDriverMinorVersion() );  
     System.out.println("=====  JDBC/DB attributes =====");  
     System.out.print("Supports getGeneratedKeys(): ");  
     if (dbmd.supportsGetGeneratedKeys() )  
       System.out.println("true");  
     else  
       System.out.println("false");  
	  con.close();
	  System.out.println("Command successfully executed");
     }

     catch(ClassNotFoundException e)
     {
	   System.out.println("Class Not Found : " + e.getMessage()); 
     }
	
     catch(SQLException exp) {
	System.out.println("SQL Exception: " + exp.getMessage());
	System.out.println("SQL State: " + exp.getSQLState());
	System.out.println("Vendor Error: " + exp.getErrorCode());    
     }	
  }
}

To compile this program, user has to copy edb-jdbc14.jar file in $JAVA_HOME/jre/lib/ext directory .
After copying the edb-jdbc driver use following command to compile the program:

javac DBinfo.java

Following is output of above java program:

vibhor@ubuntu:~$ java DBinfo 
=====  Database info =====
DatabaseProductName: EnterpriseDB
DatabaseProductVersion: 9.0.4.14
DatabaseMajorVersion: 9
DatabaseMinorVersion: 0
=====  Driver info =====
DriverName: Postgres Plus Advanced Server Native Driver
DriverVersion: Postgres Plus Advanced Server 9.0 (9.0.4.14)
DriverMajorVersion: 9
DriverMinorVersion: 0
=====  JDBC/DB attributes =====
Supports getGeneratedKeys(): true
Command successfully executed

Above java code can also be use for find postgreSQL jdbc driver version.
For using with postgresql-jdbc driver, User has to change following

  Connection con = 
Class.forName("com.edb.Driver");
DriverManager.getConnection("jdbc:edb://localhost:5444/edb",
	  "enterprisedb","edb"); // Advanced Server Database Connection Information	

with

Class.forName("org.postgresql.Driver");
Connection con = 
DriverManager.getConnection("jdbc:postgresql://localhost:5432/postgres",
	  "postgres","postgres"); // PostgreSQL Database Connection Information	

And has copy the postgresql-jdbc driver in $JAVA_HOME/jre/lib/ext

Then compile as mentioned above.
Following is output:

=====  Database info =====
DatabaseProductName: PostgreSQL
DatabaseProductVersion: 9.0.4.14
DatabaseMajorVersion: 9
DatabaseMinorVersion: 0
=====  Driver info =====
DriverName: PostgreSQL Native Driver
DriverVersion: PostgreSQL 8.4 JDBC4 (build 701)
DriverMajorVersion: 8
DriverMinorVersion: 4
=====  JDBC/DB attributes =====
Supports getGeneratedKeys(): true
Command successfully executed