Lunar的oracle实验室

前不久，我同事修改standbalone的主机名，只是单纯的从主机层面修改了，has等并没有做调整，虽然数据库奇迹般的open了（有些时候，数据库在此情况下是不能open的，直接就会报错），但是数据库更改主机名以后，ASM和数据库的alert都有报错，而且分析后，觉得这个报错不解决，会在后期使用中造成数据库crash。
修复这个问题的程很简单，有几点说明：
1，8i和8i以前叫做OPS，Oracle Parallel Server，9.0.1~11.1之间叫做RAC（Real Application Cluster），11.2以后叫做GI（Grid Infrastructure）
2，Standbalone结构就是单实例数据库使用ASM的场景，他只需要db和asm通信的cssd等结构，不需要crs。
3，启动，关闭和管理has流程类似rac，但是千万别混淆，Standbalone的叫做crsctl start/stop has，RAC的叫做crsctl start/stop crs(整套架构中所有进程由ohasd创建)，如果弄错了，会出现一些异常情况，在其他案例中，我们遇到过直接ASM报错的，后续在总结了那个案例分享出来。
4，不管是Standbalone还是RAC，修改主机名和IP等都需要特别小心，因为他们不像单机数据库一样，单纯的主机层面就该就可以，他们需要分别在has和crs中进行重配或者修改。
5，无论发生了什么，只要没有动ASM和DB，那么不用担心丢数据。因为has或者crs的结构都跟db是独立的，一般不会丢数据，最差的结果，重装一下，也可以把asm和db拉起来。
6，很多时候，在做类似该ip或者主机名，或者升级crs或者has的版本，或者升级数据库软件时，都建议对现有环境进行备份，要么借助NBU之类的工具，要么使用tar命令。千万别用ftp或者直接copy，这类的否不靠谱，不能用于文件恢复。

具体过程如下：
ASM ALERT:

Tue Mar 03 14:58:34 2015
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.
Wed Mar 18 17:01:23 2015 
NOTE: client lunarogg:lunarogg deregistered
Wed Mar 18 17:07:18 2015 
NOTE: client lunarogg:lunarogg registered, osid 16085, mbr 0x1
~

数据库的日志，DB ALERT:

Fri Mar 20 11:29:44 2015
Archived Log entry 650 added for thread 1 sequence 107161 ID 0x9911db3 dest 1:
Fri Mar 20 11:30:15 2015
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 25695
Fri Mar 20 11:32:05 2015
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 25711
Fri Mar 20 11:32:54 2015
Thread 1 cannot allocate new log, sequence 107163
Private strand flush not complete
  Current log# 3 seq# 107162 mem# 0: /u01/oradata/redo/lunarogg/onlinelog/redo03.log
Fri Mar 20 11:32:55 2015
ERROR: unrecoverable error ORA-29701 raised in ASM I/O path; terminating process 25766
Thread 1 advanced to log sequence 107163 (LGWR switch)
  Current log# 4 seq# 107163 mem# 0: /u01/oradata/redo/lunarogg/onlinelog/redo04.log
Fri Mar 20 11:32:59 2015
Archived Log entry 651 added for thread 1 sequence 107162 ID 0x9911db3 dest 1:

CRS和ohasd也有明显报错：

2015-03-03 14:32:22.832: [UiServer][1135483200]{0:0:2} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:22.833: [UiServer][1135483200]{0:0:2} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:22.836: [UiServer][1135483200]{0:0:54} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:22.836: [UiServer][1135483200]{0:0:54} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:53.108: [UiServer][1135483200]{0:0:78} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:53.109: [UiServer][1135483200]{0:0:78} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:53.118: [UiServer][1135483200]{0:0:79} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:32:53.119: [UiServer][1135483200]{0:0:79} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:33:22.609: [  CRSSEC][1133381952]{0:0:95} Exception: setGroupIdentity failed while checking groupname: asmadmin
2015-03-03 14:33:22.615: [   CRSPE][1133381952]{0:0:95} Exception in setGroupIdentity (ignoring groupIdentity): Authorization Exception:The input identity user id does not belong to the group is invalid
2015-03-03 14:33:53.733: [UiServer][1135483200]{0:0:91} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:33:53.738: [UiServer][1135483200]{0:0:94} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:33:53.739: [UiServer][1135483200]{0:0:95} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-03 14:33:53.740: [UiServer][1135483200]{0:0:95} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-18 17:01:17.748: [  CRSSEC][1133381952]{0:0:8811} Exception: setGroupIdentity failed while checking groupname: asmadmin
2015-03-18 17:01:17.748: [   CRSPE][1133381952]{0:0:8811} Exception in setGroupIdentity (ignoring groupIdentity): Authorization Exception:The input identity user id does not belong to the group is invalid
2015-03-18 17:01:59.271: [  CRSSEC][1133381952]{0:0:8812} Exception: setGroupIdentity failed while checking groupname: asmadmin
2015-03-18 17:01:59.271: [   CRSPE][1133381952]{0:0:8812} Exception in setGroupIdentity (ignoring groupIdentity): Authorization Exception:The input identity user id does not belong to the group is invalid
2015-03-18 17:01:59.277: [UiServer][1135483200]{0:0:8812} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-18 17:01:59.295: [UiServer][1135483200]{0:0:8812} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-18 17:01:59.296: [UiServer][1135483200]{0:0:8812} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-18 17:34:14.510: [UiServer][1135483200]{0:0:8834} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.
2015-03-18 17:34:14.510: [UiServer][1135483200]{0:0:8834} Communication exception sending reply back to client.FatalCommsException : Failed to send response to client.

我猜测，此时如果做类似增加/删除文件数据库都会crash，果然，我让同事做rman备份，数据库就crash了。
处理的方法很简单，就是修改has的相关配置，讲新主机名配置进去。具体步骤如下：
首先记录当前的系统关键信息，因为修改完has后，需要将asm，db，diskgroup等关键信息重新注册到has中：

[grid@lunardb ~]$ srvctl config asm
ASM home: /u01/app/grid/product/11.2.0/grid
ASM listener: LISTENER
Spfile: +DATA/asm/asmparameterfile/registry.253.847035651
ASM diskgroup discovery string: /dev/mapper/*
[grid@lunardb ~]$ 
[oracle@lunardb ~]$ srvctl config database -d lunarogg
Database unique name: lunarogg
Database name: lunarogg
Oracle home: /u01/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/lunarogg/spfilelunarogg.ora
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Database instance: lunarogg
Disk Groups: DATA,DATA2
Services: 
[oracle@lunardb ~]$ 

这里还发现了一个问题，这个库的ASM的在has中的信息和实际使用的spfile不一致

[grid@lunardb ~]$ asmcmd spbackup +DATA/asm/asmparameterfile/registry.253.847035651 /tmp/ASMspfile.backup
ASMCMD-8303: invalid SPFILE '+DATA/asm/asmparameterfile/registry.253.847035651'
ORA-15056: additional error message
ORA-15173: entry 'asm' does not exist in directory '/'
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 322
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
[grid@lunardb ~]$ 
检查ASM实例，确实使用的是pfile启动的，这个从alert中也可以得到证实：
[grid@lunardb ~]$ ss

SQL*Plus: Release 11.2.0.4.0 Production on Mon Mar 23 13:56:01 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Automatic Storage Management option

SQL> show parameter pfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string
SQL> 

在所有的磁盘组中，我们发现实际上是找不到在has中注册的ASM实例使用的spfile的：

[grid@lunardb ~]$ asmcmd find --type ASMPARAMETERFILE +DATA2 "*"
[grid@lunardb ~]$ asmcmd find --type ASMPARAMETERFILE +DATA "*"

使用kfod检查会发现，大量磁盘的报错：

[grid@lunardb ~]$ kfod asm_diskstring='/dev/mapper/*' disks=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group   
================================================================================
   1:    1429504 Mb /dev/mapper/D_R5_01_LUN01                grid     oinstall
   2:    1429504 Mb /dev/mapper/D_R5_01_LUN02                grid     oinstall
   3:    1425408 Mb /dev/mapper/D_R5_02_LUN01                grid     oinstall
   4:    1425408 Mb /dev/mapper/D_R5_02_LUN02                grid     oinstall
   5:    1425408 Mb /dev/mapper/D_R5_03_LUN01                grid     oinstall
   6:    1425408 Mb /dev/mapper/D_R5_03_LUN02                grid     oinstall
   7:    1425408 Mb /dev/mapper/D_R5_04_LUN01                grid     oinstall
   8:    1425408 Mb /dev/mapper/D_R5_04_LUN02                grid     oinstall
KFOD-00301: Unable to contact Cluster Synchronization Services (CSS). Return code 2 from kgxgncin.
KFOD-00311: Error scanning device /dev/mapper/control
ORA-15025: could not open disk "/dev/mapper/control"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/VolGroup00-LogVol00
ORA-15025: could not open disk "/dev/mapper/VolGroup00-LogVol00"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/VolGroup00-LogVol03
ORA-15025: could not open disk "/dev/mapper/VolGroup00-LogVol03"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/VolGroup00-LogVol04
ORA-15025: could not open disk "/dev/mapper/VolGroup00-LogVol04"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/VolGroup00-LogVol02
ORA-15025: could not open disk "/dev/mapper/VolGroup00-LogVol02"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/VolGroup00-LogVol01
ORA-15025: could not open disk "/dev/mapper/VolGroup00-LogVol01"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/Vol_oradata01-lv_oradata1
ORA-15025: could not open disk "/dev/mapper/Vol_oradata01-lv_oradata1"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
KFOD-00311: Error scanning device /dev/mapper/lunarogg01lun1
ORA-15025: could not open disk "/dev/mapper/lunarogg01lun1"
Linux-x86_64 Error: 13: Permission denied
Additional information: 42
Additional information: 22621383
Additional information: 1598903119
[grid@lunardb ~]$ 

接下来，我就开始重建has了，首先是使用force都无法停止has，因为已经异常了，这个在意料之中的，没关系：

[root@lunardb ~]# crsctl stop has -f
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
[root@lunardb ~]# 
[root@lunardb ~]# crsctl stop crs
CRS-4013: This command is not supported in a single-node configuration.
CRS-4000: Command Stop failed, or completed with errors.
[root@lunardb ~]# ps -ef|grep d.bin
grid      9107     1  0 Mar03 ?        00:01:43 /u01/app/grid/product/11.2.0/grid/bin/evmd.bin
grid      9140  9107  0 Mar03 ?        00:00:00 /u01/app/grid/product/11.2.0/grid/bin/evmlogger.bin -o /u01/app/grid/product/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/grid/product/11.2.0/grid/evm/log/evmlogger.log
grid      9149     1  0 Mar03 ?        00:15:09 /u01/app/grid/product/11.2.0/grid/bin/cssdagent
grid      9172     1  0 Mar03 ?        00:01:50 /u01/app/grid/product/11.2.0/grid/bin/ocssd.bin 
grid     16751     1  0 Mar18 ?        00:00:17 /u01/app/grid/product/11.2.0/grid/bin/tnslsnr LISTENER -inherit
root     31046 30878  0 14:17 pts/8    00:00:00 grep d.bin
[root@lunardb ~]# 

这些进程已经停不掉了，于是只能重启主机（kill ocssd.bin 一样会导致主机重启，因此直接手工reboot了）。
起来后，先删除现有的has配置：

[root@lunardb ~]# $ORACLE_HOME/crs/install/roothas.pl -deconfig -force
Using configuration parameter file: /u01/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Delete failed, or completed with errors.
CLSU-00100: Operating System function: opendir failed with error data: 2
CLSU-00101: Operating System error message: No such file or directory
CLSU-00103: error location: scrsearch1
CLSU-00104: additional error information: cant open scr home dir scls_scr_getval
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
Successfully deconfigured Oracle Restart stack
[root@lunardb ~]# 

然后执行重新配置：

[root@lunardb ~]# $ORACLE_HOME/crs/install/roothas.pl
Using configuration parameter file: /u01/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
LOCAL ADD MODE 
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE 
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node lunardb successfully pinned.
Adding Clusterware entries to inittab

lunardb     2015/03/23 14:29:29     /u01/app/grid/product/11.2.0/grid/cdata/lunardb/backup_20150323_142929.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
You have new mail in /var/spool/mail/root
[root@lunardb ~]# 

这过程很快，比GI快多了，HAS还是结构简单啊，O(∩_∩)O哈哈~。
配置完成后，检查一下，服务都正常，只是需要将下面的服务改为自动启动：

[grid@lunardb ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ons
               OFFLINE OFFLINE      lunardb                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                                                   
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  ONLINE       lunardb                                    
[grid@lunardb ~]$ 
[grid@lunardb ~]$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1"
[grid@lunardb ~]$ crsctl modify resource "ora.evmd" -attr "AUTO_START=1"
[grid@lunardb ~]$ crsctl modify resource "ora.ons" -attr "AUTO_START=1"

使用NETCA重建监听:

[grid@lunardb ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunardb                                    
ora.ons
               OFFLINE OFFLINE      lunardb                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        OFFLINE OFFLINE                                                   
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  ONLINE       lunardb                                    
[grid@lunardb ~]$ 

添加asm：

[grid@lunardb ~]$ srvctl add asm
[grid@lunardb ~]$ crsctl modify resource "ora.asm" -attr "AUTO_START=1"
[grid@lunardb ~]$ srvctl config asm
ASM home: /u01/app/grid/product/11.2.0/grid
ASM listener: LISTENER
Spfile: 
ASM diskgroup discovery string: ++no-value-at-resource-creation--never-updated-through-ASM++
[grid@lunardb ~]$
修改asm和listener的配置：
[grid@lunardb dbs]$ srvctl modify asm -l LISTENER -p "/u01/app/grid/product/11.2.0/grid/dbs/init+ASM.ora" -d "/dev/mapper/*"
[grid@lunardb dbs]$ crsctl modify resource "ora.asm" -attr "AUTO_START=1"
[grid@lunardb dbs]$ srvctl config asm
ASM home: /u01/app/grid/product/11.2.0/grid
ASM listener: LISTENER
Spfile: /u01/app/grid/product/11.2.0/grid/dbs/init+ASM.ora
ASM diskgroup discovery string: /dev/mapper/*
[grid@lunardb dbs]$ 

在has中添加db：

[oracle@lunardb ~]$ srvctl add database -d lunarogg -n lunardb -o /u01/app/oracle/product/11.2.0/db_1 -p +DATA/lunarogg/spfilelunarogg.ora
[oracle@lunardb ~]$ srvctl modify database -d lunarogg -a "DATA,DATA2"
[oracle@lunardb ~]$

现在再次检查has的配置，都已经好了：

[grid@lunardb dbs]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       lunardb                                    
ora.DATA2.dg
               ONLINE  ONLINE       lunardb                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunardb                                    
ora.asm
               ONLINE  ONLINE       lunardb                Started             
ora.ons
               OFFLINE OFFLINE      lunardb                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       lunardb                                    
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  ONLINE       lunardb                                    
ora.lunarogg.db
      1        OFFLINE OFFLINE                                                   
[grid@lunardb dbs]$ 

在使用kfod检查asmdisk已经不抱错了：

[grid@lunardb dbs]$ kfod asm_diskstring='/dev/mapper/*' disks=all
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group   
================================================================================
   1:    1429504 Mb /dev/mapper/D_R5_01_LUN01                grid     oinstall
   2:    1429504 Mb /dev/mapper/D_R5_01_LUN02                grid     oinstall
   3:    1425408 Mb /dev/mapper/D_R5_02_LUN01                grid     oinstall
   4:    1425408 Mb /dev/mapper/D_R5_02_LUN02                grid     oinstall
   5:    1425408 Mb /dev/mapper/D_R5_03_LUN01                grid     oinstall
   6:    1425408 Mb /dev/mapper/D_R5_03_LUN02                grid     oinstall
   7:    1425408 Mb /dev/mapper/D_R5_04_LUN01                grid     oinstall
   8:    1425408 Mb /dev/mapper/D_R5_04_LUN02                grid     oinstall
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                                          
================================================================================
      +ASM /u01/app/grid/product/11.2.0/grid                                    
[grid@lunardb dbs]$ 

使用has重新启动，把所有资源和db都带起来：

[grid@lunardb dbs]$ crsctl status res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       lunardb                                    
ora.DATA2.dg
               ONLINE  ONLINE       lunardb                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunardb                                    
ora.asm
               ONLINE  ONLINE       lunardb                Started             
ora.ons
               OFFLINE OFFLINE      lunardb                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       lunardb                                    
ora.diskmon
      1        OFFLINE OFFLINE                                                   
ora.evmd
      1        ONLINE  ONLINE       lunardb                                    
ora.lunarogg.db
      1        ONLINE  ONLINE       lunardb                Open                
[grid@lunardb dbs]$ 

好了，都ok了。