Set Up Cross-Region Replication

To set up cross-region replication (CRR), you set up Disaster Recover (DR) clusters that sync with the primary cluster. Changes on the primary cluster are copied over to the DR cluster.

When necessary, you can fail over to a DR cluster, making it the new primary cluster.

This setup assumes that you are setting up a DR cluster for an existing primary cluster. If you are setting up both the primary cluster and DR cluster from scratch, you only need perform Step 3 after TigerGraph is installed on both clusters.

1. Before you begin

  • Install TigerGraph 3.2 or higher on both the primary cluster and the DR cluster in the same version.

  • Make sure that your DR cluster has the same number of partitions as the primary cluster.

  • Make sure the username and password of the TigerGraph database user created on the DR cluster during installation matches one of the users on the primary cluster who have the superuser role.

  • If you choose to enable CRR and your DR cluster is in a different Virtual Private Cloud (VPC) than your primary cluster, make sure that TigerGraph is installed on your cluster with public IPs:

Make sure TigerGraph is not installed with a local loopback IP such as 127.0.0.1. You can verify if you are using loopback IP with gadmin config get System.HostList if this returns 127.0.0.1 then it means you have installed TigerGraph with loopback IP

2. Procedure

The following setup is needed in order to enable Cross Region Replication.

2.1. Backup primary data

Use GBAR to create a backup of the primary cluster. See Backup and Restore on how to create a backup.

If you are setting up both the primary cluster and the DR cluster from scratch, you can skip Steps 1, 2, and 4 and only perform Step 3.

2.2. Restore on the DR cluster

Copy the backup files from every node to every node on the new cluster. Restore the backup of the primary cluster on the DR cluster. See Backup and Restore on how to restore a backup.

2.3. Enable CRR on the DR cluster

Run the following commands on the DR cluster to enable CRR on the DR cluster.

# Enable Kafka Mirrormaker
$ gadmin config set System.CrossRegionReplication.Enabled true

# Kafka mirrormaker primary cluster's IPs, separator by ','
$ gadmin config set System.CrossRegionReplication.PrimaryKafkaIPs <PRIMARY_IP1,PRIMARY_IP2,PRIMARY_IP3>

# Kafka mirrormaker primary cluster's KafkaPort
$ gadmin config set System.CrossRegionReplication.PrimaryKafkaPort 30002

# The prefix of GPE/GUI/GSQL Kafka Topic, by default is empty.
$ gadmin config set System.CrossRegionReplication.TopicPrefix Primary

# Apply the config changes, init Kafka, and restart
$ gadmin config apply -y
$ gadmin init kafka -y
$ gadmin restart all -y

2.4. Force install queries on primary

Run the INSTALL QUERY -force ALL command on the primary cluster. After the command is finished, all other metadata operations on the primary cluster will start syncing to the DR cluster.

3. Restrictions on the DR cluster

After being set up, the DR cluster will be read-only and all data update operations will be blocked. This includes the following operations:

  • All metadata operations

    • Schema changes

    • User access management operations

    • Query creation, installation, and dropping

    • User-defined function operations

  • Data-loading operations

    • Loading job operations

    • RESTPP calls that modify graph data

  • Queries that modify the graph

4. Sync an outdated DR cluster

When the primary cluster executes an IMPORT, DROP ALL, or CLEAR GRAPH STORE GSQL command, or the gsql --reset bash command, the services on the DR cluster will stop syncing with the primary and become outdated.

To bring an outdated cluster back in sync, you need to generate a fresh backup of the primary cluster, and perform the setup steps detailed on this page again. However, you can skip Step 3: Enable CRR on the DR cluster, because CRR will have already been enabled.

5. Updating a CRR system

From time to time, you may want to update the TigerGraph software on a CRR system. To perform this correctly, follow this sequence of steps.

  1. Disable CRR on your DR cluster.

$ gadmin config set System.CrossRegionReplication.Enabled false
$ gadmin config apply -y
$ gadmin restart all -y

  1. Upgrade both the primary cluster and DR cluster.

  2. Enable CRR on the DR cluster.