基于IP|基于IP SAN的Oracle 11gR2 RAC构建（二）基于IPSAN的Oracle11gR2RAC构建（二

集群操作系统的安装
必要补丁的安装
网络配置
IPSAN 的iscsi initiator配置
分区的创建
其余集群节点配置和同步

系统基础软件的安装和配置
本集群，我们假设有两台服务器，分别是racnode1,racnode2,分别有两张网卡，一张用于对外提供服务，一张用于私网，提供到IPSAN的连接，并提供集群的Cache Fusion的私有连接。
1.集群节点操作系统的安装
最新安装包是7.6：
V980739-01.iso Oracle Linux Release 7 Update 6 for x86 (64 bit), 4.3 GB
安装过程此处省略。
2.必要的补丁安装
如下是手册中列出的补丁清单，我们需要首先将这些包全部安装完毕。这里把安装命令重复执行了一边。

[root@racnode1 Packages]# yum installbinutilselfutils-libelfglibcglibc-commonkshlibaiolibgcclibstdc++makeelfutils-libelf-develgccgcc-c++glibc-develglibc-headerslibstdc++-develunixODBCcompat-libstdc++libaio-devellibaio-develunixODBC-develsysstat Loaded plugins: langpacks, ulninfo Package binutils-2.27-34.base.0.1.el7.x86_64 already installed and latest version Package elfutils-libelf-0.172-2.el7.x86_64 already installed and latest version Package glibc-2.17-260.0.9.el7.x86_64 already installed and latest version Package glibc-common-2.17-260.0.9.el7.x86_64 already installed and latest version Package ksh-20120801-139.0.1.el7.x86_64 already installed and latest version Package libaio-0.3.109-13.el7.x86_64 already installed and latest version Package libgcc-4.8.5-36.0.1.el7.x86_64 already installed and latest version Package libstdc++-4.8.5-36.0.1.el7.x86_64 already installed and latest version Package 1:make-3.82-23.el7.x86_64 already installed and latest version Package elfutils-libelf-devel-0.172-2.el7.x86_64 already installed and latest version Package gcc-4.8.5-36.0.1.el7.x86_64 already installed and latest version Package gcc-c++-4.8.5-36.0.1.el7.x86_64 already installed and latest version Package glibc-devel-2.17-260.0.9.el7.x86_64 already installed and latest version Package glibc-headers-2.17-260.0.9.el7.x86_64 already installed and latest version Package libstdc++-devel-4.8.5-36.0.1.el7.x86_64 already installed and latest version Package unixODBC-2.3.1-11.el7.x86_64 already installed and latest version No package compat-libstdc++ available. Package libaio-devel-0.3.109-13.el7.x86_64 already installed and latest version Package libaio-devel-0.3.109-13.el7.x86_64 already installed and latest version Package unixODBC-devel-2.3.1-11.el7.x86_64 already installed and latest version Package sysstat-10.1.5-17.el7.x86_64 already installed and latest version Nothing to do

compat-libstdc++没有安装上，我们需要进入/mnt/cdrom/Package ,单独安装下。

[root@racnode1 Packages]# ls compat-libstdc* compat-libstdc++-33-3.2.3-72.el7.i686.rpmcompat-libstdc++-33-3.2.3-72.el7.x86_64.rpm [root@racnode1 Packages]# rpm -ivh compat-libstdc++-33-3.2.3-72.el7.* Preparing...################################# [100%] package compat-libstdc++-33-3.2.3-72.el7.x86_64 is already installed package compat-libstdc++-33-3.2.3-72.el7.i686 is already installed

3.网络配置
每个节点服务器有两张网卡，一张对外提供服务，一张私网的用于IPSAN和节点之间互联。
公网的网卡配置如下：

[root@racnode1 Packages]# cat /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE=Ethernet BOOTPROTO=static IPADDR=192.168.1.8 NETMASK=255.255.255.0 DEFROUTE=yes NAME=ens33 UUID=7f29b95d-7854-4069-acb9-cb60afbc88f4 DEVICE=ens33 ONBOOT=yes HWADDR=00:0C:29:53:4D:2D ZONE=public PREFIX=24 GATEWAY=192.168.1.1 DNS1=192.168.1.1

私网的网卡配置如下：

[root@racnode1 Packages]# cat /etc/sysconfig/network-scripts/ifcfg-ens37 TYPE=Ethernet BOOTPROTO=static IPADDR=192.168.237.126 NETMASK=255.255.255.0 NAME=ens37 DEVICE=ens37 ONBOOT=yes HWADDR=00:0C:29:53:4D:37

节点主机名配置：

[root@racnode1 Packages]# cat /etc/hostname racnode1

节点的hosts文件配置：

[root@racnode1 Packages]# cat /etc/hosts 127.0.0.1localhost localhost.localdomain localhost4 localhost4.localdomain4 #public network 192.168.1.8racnode1 192.168.1.9racnode2#private network 192.168.237.126 racnode1-priv 192.168.237.127 racnode2-priv#virtual network 192.168.1.18racnode1-vip 192.168.1.19racnode2-vip#scan 192.168.1.22racnode-cluster-scan#storage 192.168.237.128 storage-priv

预览一下配置生效后的结果：

[root@racnode1 Packages]# ifconfig ens33: flags=4163mtu 1500 inet 192.168.1.8netmask 255.255.255.0broadcast 192.168.1.255 inet6 240e:39a:344:3b10:18f0:d699:a52:715aprefixlen 64scopeid 0x0 inet6 fe80::6fb5:dd8b:90f:3edbprefixlen 64scopeid 0x20 inet6 fe80::7395:ca88:671:f19fprefixlen 64scopeid 0x20 ether 00:0c:29:53:4d:2dtxqueuelen 1000(Ethernet) RX packets 4360bytes 348580 (340.4 KiB) RX errors 0dropped 1302overruns 0frame 0 TX packets 2519bytes 427636 (417.6 KiB) TX errors 0dropped 0 overruns 0carrier 0collisions 0ens37: flags=4163mtu 1500 inet 192.168.237.126netmask 255.255.255.0broadcast 192.168.237.255 inet6 fe80::3431:565:9a78:4692prefixlen 64scopeid 0x20 ether 00:0c:29:53:4d:37txqueuelen 1000(Ethernet) RX packets 6057bytes 6882440 (6.5 MiB) RX errors 0dropped 0overruns 0frame 0 TX packets 2349bytes 215629 (210.5 KiB) TX errors 0dropped 0 overruns 0carrier 0collisions 0lo: flags=73mtu 65536 inet 127.0.0.1netmask 255.0.0.0 inet6 ::1prefixlen 128scopeid 0x10 looptxqueuelen 1000(Local Loopback) RX packets 0bytes 0 (0.0 B) RX errors 0dropped 0overruns 0frame 0 TX packets 0bytes 0 (0.0 B) TX errors 0dropped 0 overruns 0carrier 0collisions 0

4.IPSAN 的iscsi initiator配置
命令如下：

[root@racnode1 Packages]# iscsiadm -m discovery -t sendtargets -p storage-priv 192.168.237.128:3260,1 iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de [root@racnode1 Packages]# iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de -p storage-priv --login Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de, portal: 192.168.237.128,3260] (multiple) Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de, portal: 192.168.237.128,3260] successful. [root@racnode1 Packages]# lsblk NAMEMAJ:MIN RMSIZE RO TYPE MOUNTPOINT sdd8:4802G0 disk sdb8:16020G0 disk sr011:014.3G0 rom/mnt/cdrom sdc8:32015G0 disk sda8:0040G0 disk ├─sda28:2039G0 part │ ├─ol-swap 252:102G0 lvm[SWAP] │ └─ol-root 252:0037G0 lvm/ └─sda18:101G0 part /boot [root@racnode1 Packages]# iscsiadm -m node 192.168.237.128:3260,1 iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de

最后确认下，是否将防火墙关闭，selinux 是否关闭。

[root@racnode1 ~]# iptables -L Chain INPUT (policy ACCEPT) targetprot opt sourcedestinationChain FORWARD (policy ACCEPT) targetprot opt sourcedestinationChain OUTPUT (policy ACCEPT) targetprot opt sourcedestination [root@racnode1 ~]# [root@racnode1 ~]# getenforce Disabled

在系统启动后，自动启动iscsi initiator 登录

[root@racnode1 ~]# iscsiadm -m node 192.168.237.128:3260,1 iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de [root@racnode1 ~]# iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.storage.x8664:sn.b0a7e05a08de -p storage-priv --op update -n node.startup -v automatic [root@racnode1 ~]#

本小节需要注意以下，安装节点时如果是采用直接克隆虚拟机方式的，会有出现initiator-name相同的情况，后台会出现大量错误。

Jun8 17:57:50 racnode2 iscsid: connection1:0 is operational after recovery (1 attempts) Jun8 17:57:52 racnode2 kernel: connection1:0: detected conn error (1020) Jun8 17:57:52 racnode2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3) Jun8 17:57:54 racnode2 iscsid: connection1:0 is operational after recovery (1 attempts) Jun8 17:57:56 racnode2 kernel: connection1:0: detected conn error (1020) Jun8 17:57:56 racnode2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3) Jun8 17:57:58 racnode2 iscsid: connection1:0 is operational after recovery (1 attempts) Jun8 17:58:00 racnode2 kernel: connection1:0: detected conn error (1020) Jun8 17:58:00 racnode2 iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) sta

【基于IP|基于IP SAN的Oracle 11gR2 RAC构建（二）】我们需要避免这样的情况发生。所以需要在节点上修改IQN，避免重名。

[root@racnode1 ~]# cat /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1988-12.com.oracle:53507fdd1974

5.创建分区
依次创建分区：
首先是/dev/sdb

[root@racnode1 ~]# fdisk /dev/sdb Welcome to fdisk (util-linux 2.23.2).Changes will remain in memory only, until you decide to write them. Be careful before using the write command.Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0xb72735dd.Command (m for help): n Partition type: pprimary (0 primary, 0 extended, 4 free) eextended Select (default p): p Partition number (1-4, default 1): 1 First sector (8192-41943039, default 8192): Using default value 8192 Last sector, +sectors or +size{K,M,G} (8192-41943039, default 41943039): Using default value 41943039 Partition 1 of type Linux and of size 20 GiB is setCommand (m for help): w The partition table has been altered!Calling ioctl() to re-read partition table. Syncing disks.

其次是/dev/sdc：

[root@racnode1 ~]# fdisk /dev/sdc Welcome to fdisk (util-linux 2.23.2).Changes will remain in memory only, until you decide to write them. Be careful before using the write command.Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0xa2733da1.Command (m for help): n Partition type: pprimary (0 primary, 0 extended, 4 free) eextended Select (default p): Using default response p Partition number (1-4, default 1): First sector (8192-31457279, default 8192): Using default value 8192 Last sector, +sectors or +size{K,M,G} (8192-31457279, default 31457279): Using default value 31457279 Partition 1 of type Linux and of size 15 GiB is setCommand (m for help): w The partition table has been altered!Calling ioctl() to re-read partition table. Syncing disks.

最后是/dev/sdd：

[root@racnode1 ~]# fdisk /dev/sdd Welcome to fdisk (util-linux 2.23.2).Changes will remain in memory only, until you decide to write them. Be careful before using the write command.Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x74090209.Command (m for help): n Partition type: pprimary (0 primary, 0 extended, 4 free) eextended Select (default p): Using default response p Partition number (1-4, default 1): First sector (8192-4194303, default 8192): Using default value 8192 Last sector, +sectors or +size{K,M,G} (8192-4194303, default 4194303): Using default value 4194303 Partition 1 of type Linux and of size 2 GiB is setCommand (m for help): w The partition table has been altered!Calling ioctl() to re-read partition table. Syncing disks.

6.其余集群节点配置和同步
在另外一个节点服务器上重复1-4的步骤（可以直接采用虚拟机的克隆方式，这样更快）。需要注意的是相应的主机名和IP需要同步按照上文中的集群架构图中标识的那样进行修改。
这里的第5步不用在做。只需要同步即可。

[root@racnode2 ~]# partprobe Warning: Unable to open /dev/sr0 read-write (Read-only file system)./dev/sr0 has been opened read-only. [root@racnode2 ~]# lsblk NAMEMAJ:MIN RMSIZE RO TYPE MOUNTPOINT sdd8:4802G0 disk └─sdd18:4902G0 part sdb8:16020G0 disk └─sdb18:17020G0 part sr011:014.3G0 rom sdc8:32015G0 disk └─sdc18:33015G0 part sda8:0040G0 disk ├─sda28:2039G0 part │ ├─ol-swap 252:102G0 lvm[SWAP] │ └─ol-root 252:0037G0 lvm/ └─sda18:101G0 part /boot [root@racnode2 ~]#

下一节，我们继续配置工作。