联系：QQ(5163721)
标题：在Double-E之间配置基于Infiniband的SDP连接（比如，两个Exadata，或者1个Exadata和1个Exaligic之间）
作者：Lunar©版权所有[文章允许转载,但必须以链接方式注明源地址,否则追究法律责任.]
在Double-E之间（比如，两个Exadata，或者1个Exadata和1个Exaligic之间），我们可以使用IB来连接
比如两个Exadata之间互相做灾备（ADG）
再比如，Exalogic上有大量的空间，即使是1/4配置的Exalogic上也有几十T的容量空闲（没有部署应用的空间）。
有时候我们会见可以购买了Double-E的客户用这个空间来做Exadata上的数据库的ADG。
为了充分利用IB的传输速度块，低延迟和CPU消耗少等优点，Double-E之间的互联，一般建议通过SDP协议访问Exadata上的数据库
文档建议（Exadata X2和X3），Exadata和Exalogic使用8个IB线进行连接（4跟的话也可以，但是没有冗余，一般不建议）
Exadata 1/4 rack和Exalogic 1/4 rack上分别都有2个叶IB交换机（半配的会有第三个主IB交换机，从Exadata X3开始，出厂时缺省没有最下的主交换机了）
这4个交换机如下：
Exadata-IB1 Exadata–IB2
Exalogic-IB1 Exalogic–IB2
连接方式：
Exadata-IB1——-Exalogic–IB2
Exadata-IB1——-Exalogic–IB1
Exadata-IB2——-Exalogic–IB1
Exadata-IB2——-Exalogic–IB2
配置方式具体如下：
1、修改db node上的/etc/infiniband/openib.conf：
SDP_LOAD=yes
2、在/etc/ofed/libsdp.conf中添加：
use tcp server * *:* use tcp client * *:*
3、在/etc/modprobe.conf都添加并且reboot节点：
options ib_sdp sdp_zcopy_thresh=0 recv_poll=0
配置Double-E（比如Exadata和ExalogicIB，或者Exadata和Exadata之间的）网络上配置监听：
192.168.10.21 dm01db01-ibvip.lunar.com dm01db01-ibvip 192.168.10.22 dm01db02-ibvip.lunar.com dm01db02-ibvip
添加network2（for IB）：
[root@dm01db01 ~]# srvctl add network -k 2 -S 192.168.8.0/255.255.252.0/bondib0 [root@dm01db01 ~]#
验证添加结果：
[root@dm01db01 ~]# crsctl stat res -t | grep net ora.net1.network ora.net2.network -- Output indicating new Network resource [root@dm01db01 ~]# [root@dm01db01 ~]# srvctl config network -k 2 Network exists: 2/192.168.8.0/255.255.252.0/bondib0, type static [root@dm01db01 ~]#
添加vip：
[root@dm01db01 ~]# srvctl add vip -n dm01db01 -A dm01db01-ibvip/255.255.252.0/bondib0 -k 2 [root@dm01db01 ~]# srvctl add vip -n dm01db02 -A dm01db02-ibvip/255.255.252.0/bondib0 -k 2 [root@dm01db01 ~]#
检查vip是否可以正常启动：
[root@dm01db01 ~]# srvctl start vip -i dm01db01-ibvip [root@dm01db01 ~]# [root@dm01db02 ~]# srvctl start vip -i dm01db02-ibvip [root@dm01db02 ~]#
添加这个IB网络的监听（LISTENER_IB）：
[root@dm01db01 ~]# srvctl add listener -l LISTENER_IB -k 2 -p TCP:1522,/SDP:1522 [root@dm01db01 ~]# srvctl start listener -l LISTENER_IB [root@dm01db01 ~]# [grid@dm01db01 ~]$ lsnrctl status listener_ib LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 18-DEC-2012 12:43:15 Copyright (c) 1991, 2011, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_IB))) STATUS of the LISTENER ------------------------ Alias LISTENER_IB Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production Start Date 18-DEC-2012 12:41:03 Uptime 0 days 0 hr. 2 min. 11 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/app/11.2.0.3/grid/network/admin/listener.ora Listener Log File /u01/app/oracle/diag/tnslsnr/dm01db01/listener_ib/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_IB))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.10.21)(PORT=1522))) The listener supports no services The command completed successfully [grid@dm01db01 ~]$
配置tnsnames.ora
DBFS =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = dbfs)
)
)
LUNARDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = LUNARDB)
)
)
#the following are for IB
LISTENER_IBREMOTE=
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip.lunar.com)(PORT = 1522))
)
LISTENER_IBLOCAL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip.lunar.com)(PORT = 1522))
(ADDRESS = (PROTOCOL = SDP)(HOST = dm01db01-ibvip.lunar.com)(PORT = 1522))
)
LISTENER_IPLOCAL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm0101-vip.lunar.com)(PORT = 1521))
)
LISTENER_IPREMOTE=
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01-scan.lunar.com)(PORT = 1521))
)
LUNARDB_IB =
(DESCRIPTION =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = LUNARDB)
)
)
修改数据库参数listener_networks：
SQL> alter system set listener_networks='((NAME=network2) (LOCAL_LISTENER=LISTENER_IBLOCAL)(REMOTE_LISTENER=LISTENER_IBREMOTE))', '((NAME=network1)(LOCAL_LISTENER=LISTENER_IPLOCAL)(REMOTE_LISTENER=LISTENER_IPREMOTE))' scope=both; System altered. SQL>
测试连接，使用oracle用户：
conn sys/oracle@LUNARDB conn sys/oracle@LUNARDB_IB
在客户端（Exalogic或者其他需要连接db的主机上），需要配置 如下连接串：
LUNARDB_IB =
(DESCRIPTION =
(LOAD_BALANCE=on)
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db01-ibvip)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = dm01db02-ibvip)(PORT = 1522))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = LUNARDB)
)
)
好了，这样Exalogic就可以使用IB连接Exadata了。
