KunlunDB指导手册(四)之从Oracle实时同步数

一、环境及软件需求 Oracle GoldenGate从11.2.1.0.2开始支持在Oracle数据库和KunlunDB之间做数据复制。
本文档提供数据复制过程的基本配置方案,涉及不同操作系统及数据库版本具体配置,请参考官方文档。
软件需求:

  • Oracle GoldenGate for Oracle
  • Oracle GoldenGate for Postgresql
  • 软件下载页面:https://www.oracle.com/middleware/technologies/goldengate.html
KunlunDB指导手册(四)之从Oracle实时同步数
文章图片

数据同步架构图
二、安装 2.1 安装&配置Oracle GoldenGate for Oracle
Oracle数据库服务器端环境变量。
ORACLE_HOME=/opt/oracle/product/19c/dbhome_1 ORACLE_SID=kunluntest LD_LIBRARY_PATH=$ORACLE_HOME/lib

Oracle数据库运行在归档模式并且开启附加日志。
alter system set log_archive_dest='LOCATION=USE_DB_RECOVERY_FILE_DEST'scope=both sid='*'; shutdown immediate startup mount alter database archivelog; alter database open; alterdatabaseaddsupplementallogdata;

安装Oracle GoldenGate for Oracle并设置GoldenGate软件目录到环境变量。
exportPATH=$PATH:/var/kunlun/ggs exportLD_LIBRARY_PATH=$ORACLE_HOME/lib:/var/kunlun/ggs/lib/LD_LIBRARY_PATH

为Goldengate 配置基本目录。
GGSCI (SOURCE.KUNLUN.COM)1> create subdirs Creatingsubdirectories under current directory /var/kunlun/ggs Parameterfiles/var/kunlun/ggs/dirprm:alreadyexists Reportfiles/var/kunlun/ggs/dirrpt:created Checkpointfiles/var/kunlun/ggs/dirchk:created processstatusfiles/var/kunlun/ggs/dirpcs:created SQLscriptfiles/var/kunlun/ggs/dirsql:created Databasedefinitionsfiles/var/kunlun/ggs/dirdef:created Extractdatafiles/var/kunlun/ggs/dirdat:created Temporaryfiles/var/kunlun/ggs/dirtmp:created Stdoutfiles/var/kunlun/ggs/dirout:created

创建OGG manager 参数文件。
GGSCI(SOURCE.KUNLUN.COM)2>editparammgr addPORT7809 totheparameterfile:

启动 OGG manager。
GGSCI(SOURCE.KUNLUN.COM)3>startmgr GGSCI(SOURCE.KUNLUN.COM)4>infoall ProgramStatusGroupLagatChkptTimeSinceChkpt addPORT7809 totheparameterfile: MANAGERRUNNING

2.2 安装&配置Oracle GoldenGate for KunlunDB
【KunlunDB指导手册(四)之从Oracle实时同步数】在PostgreSQL服务器端解压GoldenGate软件包并发lib路径配置到环境变量中。
mkdir ggs cd ggs unzip V34006-01.zip tar xvf *.tar [kunlun@centos7b ggs]$ export LD_LIBRARY_PATH=/var/kunlun/ggs/lib

GoldenGate通过ODBC连接 kunlunDB ,因此,需要配置ODBC 数据源。
[ODBC Data Sources] Kunlundb1=DataDirect 11.5 KUNLUNDB Wire Protocol [ODBC] IANAAppCodePage=106 InstallDir=/var/kunlun/ggs [kunlundb1] Driver=/var/kunlun/ggs/lib/GGpsql25.so Description=DataDirect 11.5 KUNLUNDB Wire Protocol Database=kunlundb HostName=192.168.0.130 PortNumber=5401 LogonID=abc Password=abc

将配置文件export到环境变量。
[kunlun@centos7bggs]$ export ODBCINI=/var/kunlun/ggs/odbc.ini

配置目标端Goldengate。
[kunlun@TARGET ggs]$ ./ggsci GGSCI (TARGET.KUNLUN.COM) 1> create subdirsCreating subdirectories under current directory /var/kunlun/ggsParameter files/var/kunlun/ggs/dirprm: already exists Report files/var/kunlun/ggs/dirrpt: created Checkpoint files/var/kunlun/ggs/dirchk: created Process status files/var/kunlun/ggs/dirpcs: created SQL script files/var/kunlun/ggs/dirsql: created Database definitions files/var/kunlun/ggs/dirdef: created Extract data files/var/kunlun/ggs/dirdat: created Temporary files/var/kunlun/ggs/dirtmp: created Stdout files/var/kunlun/ggs/dirout: createdcreate the Manager parameter file and start the manager: GGSCI (TARGET.KUNLUN.COM) 2> edit param mgr PORT 7809 GGSCI(TARGET.KUNLUN.COM) 3> start mgr Manager started.GGSCI (TARGET.KUNLUN.COM) 4> info all ProgramStatusGroupLag at Chkpt Time Since Chkpt MANAGERRUNNING

目标端新增checkpoint表。
-- ogg中执行dblogin sourcedb KUNLUNDB userid abc password abc add checkpointtable ogg.checkpointtab

三、准备测试表 分别在Oracle数据库和KunlunDB集群中建立结构相同的一张表。
Oracle DB
SQL> connect kunlun/kunlun Connected. SQL> create table kunluntest (col1 number, col2varchar2(20)); Table created. SQL> alter table kunluntest add primary key (col1); Table altered.

KunlunDB
KunlunDB>CREATE TABLE "public"."kunluntest" ( "col1"integer NOT NULL, "col2"varchar(20), CONSTRAINT"PK_Col111" PRIMARY KEY ("col1") )

3.1 验证到OGG到Oracle数据库的连接
在Oracle端执行ggsci。
GGSCI (SOURCE.KUNLUN.COM) 8> dblogin userid kunlun,password kunlun Successfully logged into database. GGSCI (SOURCE.KUNLUN.COM) 9> list tables * KUNLUN.KUNLUNTEST Found 1 tables matching list criteria.GGSCI (SOURCE.KUNLUN.COM) 10> capture tabledefKUNLUN.KUNLUNTEST Table definitions for KUNLUN.KUNLUNTEST: COL1NUMBER NOT NULL PK COL2VARCHAR (20)

3.2 验证到OGG到KunlunDB数据库的连接
在kunlunDB端执行ggsci。
GGSCI (TARGET.KUNLUN.COM) 4> dblogin sourcedb kunlundbuserid abc Password: 2013-04-06 16:51:18 INFO OGG-03036 Database character setidentified as UTF-8. Locale: en_US. 2013-04-06 16:51:18 INFO OGG-03037 Session character setidentified as UTF-8. Successfully logged into database. GGSCI (TARGET.KUNLUN.COM) 5> list tables * public.kunluntestFound 1 tables matching list criteria. GGSCI (TARGET.KUNLUN.COM) 3> capture tabledef"public"."kunluntest" Table definitions for public.kunluntest: col1NUMBER(10) NOT NULL PK col2VARCHAR (20)

四、配置抽取进程 配置一个抽取进程,抽取表Oracle数据库中的表kunluntest的数据增量到trail文件中。
首先配置MGR 参数:
GGSCI (SOURCE.ORACLE.COM) 4> edit param eporwith these parameters:EXTRACT epor USERID kunlun, PASSWORD kunlun RMTHOST 192.168.0.130, MGRPORT 7809 RMTTRAIL ./dirdat/ep TABLE kunlun.kunluntest;

启动抽取进程。
GGSCI (SOURCE.ORACLE.COM) 5> add extract epor, tranlog,begin now EXTRACT added. GGSCI (SOURCE.ORACLE.COM) 6> add exttrail ./dirdat/ep,extract epos, megabytes 5 EXTTRAIL added.GGSCI (SOURCE.ORACLE.COM) 7> start epor Sending START request to MANAGER ... EXTRACT EPOR startingGGSCI (SOURCE.ORACLE.COM) 8> info allProgramStatusGroupLag at Chkpt Time Since Chkpt MANAGERRUNNING EXTRACTRUNNINGEPOR00:00:0000:00:05

五、创建定义文件 在OGG中,异构数据库之间的数据同步需要为源端创建定义文件。
GGSCI (SOURCE.KUNLUN.COM) 10> edit param defgen DEFSFILE ./dirdef/KUNLUNTEST.def USERID kunlun, password kunlun TABLE KUNLUN.KUNLUNTEST; [kunlun@SOURCE ggs]$ ./defgen paramfile ./dirprm/defgen.prm*********************************************************************** Oracle GoldenGateTable Definition Generator for Oracle Version 11.2.1.0.314400833 OGGCORE_11.2.1.0.3_PLATFORMS_120823.1258 Copyright (C) 1995, 2012, Oracle and/or its affiliates. Allrights reserved.Starting at2022-03-15 18:32:10 ***********************************************************************Operating System Version: Linux Node: SOURCE.KUNLUN.COM Machine: x86_64 softlimithard limit Address Space Size :unlimitedunlimited Heap Size:unlimited unlimited File Size:unlimited unlimited CPU Time:unlimited unlimitedProcess id: 1546*********************************************************************** **Running withthe following parameters** *********************************************************************** DEFSFILE ./dirdef/KUNLUNTEST.def USERID postgres, password ******** TABLE KUNLUN.KUNLUNTEST; Retrieving definition for KUNLUN.KUNLUNTESTDefinitions generated for 1 table in ./dirdef/KUNLUNTEST.defContent of the Defgen File: [oracle@SOURCE ggs]$ more ./dirdef/KUNLUNTEST.def *+- Defgen version 2.0, Encoding UTF-8 * \* Definitions created/modified 2022-03-15 18:32 * \* Field descriptions for each column entry: * \*1Name \*2Data Type \*3External Length \*4Fetch Offset \*5Scale \*6Level \*7Null \*8Bump if Odd \*9Internal Length \*10Binary Length \*11Table Length \*12Most Significant DT \*13Least Significant DT \*14High Precision \*15Low Precision \*16Elementary Item \*17Occurs \*18Key Column \*19Sub Data Type * Database type: ORACLE Character set ID: UTF-8 National character set ID: UTF-16 Locale: neutral Case sensitivity: 14 14 14 14 14 14 14 14 14 14 14 14 11 1414 14 * Definition for table POSTGRES.GGTEST Record length: 262 Syskey: 0 Columns: 2 COL164500 0 0 1 0505050 0 0 0 0 10 1 2 COL26420056 0 0 1 020020000 0 0 0 10 0 0 End of definition

最后,将定义文件从oracle服务器/dirdef/KUNLUNTEST.def复制到kunlundb 服务器的./dirdef/KUNLUNTEST.def。
六、配置复制进程 在源端的抽取进程将数据的变更日志写入到trail日志中,日志可以通过dump进程或其他方式传输到目标端的OGG。复制进程将日志应用KunlunDB中去。
复制进程名称是rpor, 配置复制进程参数:
GGSCI (TARGET.KUNLUN.COM) 1> edit param rporwith the parameters:REPLICAT rpor SOURCEDEFS ./dirdef/GGTEST.def SETENV ( PGCLIENTENCODING = "UTF8" ) SETENV (ODBCINI="/var/kunlun/ggs/odbc.ini" ) SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8") TARGETDB GG_Postgres, USERID kunlun, PASSWORD kunlun DISCARDFILE ./dirrpt/diskg.dsc, purge MAP POSTGRES.GGTEST, TARGET public.kunluntest, COLMAP(COL1=col1,COL2=col2);

创建并启动复制进程。
GGSCI (ZKUPCHV119) 2> add replicat rpor, NODBCHECKPOINT,exttrail ./dirdat/ep REPLICAT added. GGSCI (edvmr1p0) 3> start rporSending START request to MANAGER ... REPLICAT REPKG startingGGSCI (TARGET.KUNLUN.COM) 2> info all ProgramStatusGroupLag at Chkpt Time Since Chkpt MANAGERRUNNING REPLICAT RUNNINGRPOR00:00:0000:00:07

复制配置完成。
七、测试 源库插入数据:
SQL> insert into KUNLUNTEST values (1,'hello world!'); 1 row created.SQL> commit; Commit complete.

目标库查看数据同步结果
-bash-3.2$ psql KUNLUNTEST Type "help" for help.KUNLUNTEST=# select * from kunluntest; col1 | col2 ------+--------- 10 | hello world!(1 rows)

目标库查看数据同步结果。
$ psql -h 192.168.0.130 -p 5401 -Uabc kunluntest Type "help" for help.KUNLUNTEST=# select * from kunluntest; col1 | col2 ------+--------- 10 | hello world!(1 rows)

八、说明 OGG on KunlunDB的更详细的配置说明及压力测试请参考《KunlunDB 快速入门(三)数据导入&同步》
OGG ON KunlunDB数据同步原理请参考《异构数据同步-Postgresql中国技术大会0109v4.pdf - 墨天轮文档 (modb.pro)》》")
KunlunDB项目已开源 【GitHub:】
https://github.com/zettadb
【Gitee:】
https://gitee.com/zettadb
END

    推荐阅读