在巡检时，发现cell的alert有如下告警：
[root@dm01cel02 ~]# cellcli -e list alerthistory 53 2013-09-09T16:59:32+08:00 info "This is a test trap" 54 2013-10-17T02:00:11+08:00 info "HDD disk controller battery on disk contoller at adapter 0 is going into a learn cycle. This is a normal maintenance activity that occurs quarterly and runs for approximately 1 to 12 hours. The disk controller cache might go into WriteThrough caching mode during the learn cycle. Disk write throughput might be temporarily lower during this time. The message is informational only, no action is required." 55_1 2013-10-29T22:52:11+08:00 critical "Hard disk failed. Status : CRITICAL Manufacturer : SEAGATE Model Number : ST360057SSUN600G Size : 600G Serial Number : E0P387 Firmware : 0805 Slot Number : 9 Cell Disk : CD_09_dm01cel02 Grid Disk : DATA_DM01_CD_09_dm01cel02, RECO_DM01_CD_09_dm01cel02, DBFS_DG_CD_09_dm01cel02" 55_2 2013-12-06T18:17:27+08:00 critical "Hard disk removed. Status : NOT PRESENT Manufacturer : SEAGATE Model Number : ST360057SSUN600G Size : 600G Serial Number : E0P387 Firmware : 0805 Slot Number : 9 Cell Disk : CD_09_dm01cel02 Grid Disk : DATA_DM01_CD_09_dm01cel02, RECO_DM01_CD_09_dm01cel02, DBFS_DG_CD_09_dm01cel02 Error Count : 0 Last Failure : unknown" 55_3 2013-12-06T18:18:21+08:00 clear "Hard disk replaced. Status : NORMAL Manufacturer : SEAGATE Model Number : ST360057SSUN600G Size : 600G Serial Number : E0SQFJ Firmware : 0805 Slot Number : 9 Cell Disk : CD_09_dm01cel02 Grid Disk : DATA_DM01_CD_09_dm01cel02, RECO_DM01_CD_09_dm01cel02, DBFS_DG_CD_09_dm01cel02" [root@dm01cel02 ~]#
我们注意到有这样的信息，就表示磁盘有损坏的情况，结合sundiag信息，可以发现磁盘确实损坏，需要更换。
另外，此时也可以通过直接看机柜，该磁盘为闪蓝色灯，表示进行拔出的操作了。
关键信息：
SEAGATE Model Number : ST360057SSUN600G
Serial Number : E0P387
Slot Number : 9
Cell Disk : CD_09_dm01cel02
换盘前，我们一般作比较细致的检查：
1.在db节点上grid用户登录，这是要确认一下asm disk是不是被drop掉。drop掉就可以直接更换，如果没有，就需要手动去drop了。
[grid@dm01db01 ~]$ sqlplus / as sysasm SQL*Plus: Release 11.2.0.2.0 Production on Fri Dec 6 18:04:35 2013 Copyright (c) 1982, 2010, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> select group_number,name,state from v$asm_diskgroup; GROUP_NUMBER NAME STATE ------------ ------------------------------ ----------- 1 DATA_DM01 MOUNTED 2 DBFS_DG MOUNTED 3 RECO_DM01 MOUNTED SQL> set linesize 132 SQL> col path format a50 SQL> select group_number,path,header_status,mount_status,mode_status,name from V$ASM_DISK where path like '%CD_09_dm01cel02'; no rows selected SQL> SQL> select * from gv$asm_operation where state='RUN'; no rows selected SQL>
这里表示磁盘celldisk：CD_09_dm01cel02已经被ASM自动删除了，且当前没有正在运行的rebalance操作。
2. 在相应的存储节点（dm01cel02）上进行确认检查：
[root@dm01cel02 ~]# cellcli -e list griddisk DATA_DM01_CD_00_dm01cel02 active DATA_DM01_CD_01_dm01cel02 active DATA_DM01_CD_02_dm01cel02 active DATA_DM01_CD_03_dm01cel02 active DATA_DM01_CD_04_dm01cel02 active DATA_DM01_CD_05_dm01cel02 active DATA_DM01_CD_06_dm01cel02 active DATA_DM01_CD_07_dm01cel02 active DATA_DM01_CD_08_dm01cel02 active DATA_DM01_CD_09_dm01cel02 not present -------------标记“not present”：gridisk DATA_DM01_CD_09_dm01cel02 DATA_DM01_CD_10_dm01cel02 active DATA_DM01_CD_11_dm01cel02 active DBFS_DG_CD_02_dm01cel02 active DBFS_DG_CD_03_dm01cel02 active DBFS_DG_CD_04_dm01cel02 active DBFS_DG_CD_05_dm01cel02 active DBFS_DG_CD_06_dm01cel02 active DBFS_DG_CD_07_dm01cel02 active DBFS_DG_CD_08_dm01cel02 active DBFS_DG_CD_09_dm01cel02 not present -------------标记“not present”：gridisk DBFS_DG_CD_09_dm01cel02 DBFS_DG_CD_10_dm01cel02 active DBFS_DG_CD_11_dm01cel02 active RECO_DM01_CD_00_dm01cel02 active RECO_DM01_CD_01_dm01cel02 active RECO_DM01_CD_02_dm01cel02 active RECO_DM01_CD_03_dm01cel02 active RECO_DM01_CD_04_dm01cel02 active RECO_DM01_CD_05_dm01cel02 active RECO_DM01_CD_06_dm01cel02 active RECO_DM01_CD_07_dm01cel02 active RECO_DM01_CD_08_dm01cel02 active RECO_DM01_CD_09_dm01cel02 not present -------------标记“not present”：gridisk RECO_DM01_CD_09_dm01cel02 RECO_DM01_CD_10_dm01cel02 active RECO_DM01_CD_11_dm01cel02 active [root@dm01cel02 ~]#
确认物理盘状态：
[celladmin@dm01cel02 ~]$ cellcli CellCLI: Release 11.2.2.3.5 - Production on Fri Dec 06 18:01:08 CST 2013 Copyright (c) 2007, 2009, Oracle. All rights reserved. Cell Efficiency Ratio: 4,314 CellCLI> list cell attributes cellsrvStatus,msStatus,rsStatus detail cellsrvStatus: running -----------cell状态时running msStatus: running rsStatus: running CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status LIKE ".*failure.*" DETAIL CellCLI> LIST PHYSICALDISK WHERE diskType=HardDisk AND status=critical DETAIL name: 20:9 ---------物理盘的设备名： 20:9 deviceId: 10 diskType: HardDisk enclosureDeviceId: 20 errMediaCount: 0 errOtherCount: 1 foreignState: false luns: 0_9 makeModel: "SEAGATE ST360057SSUN600G" ------磁盘型号 :SEAGATE ST360057SSUN600 physicalFirmware: 0805 physicalInsertTime: 2011-09-16T18:01:57+08:00 physicalInterface: sas physicalSerial: E0P387 --------Serial Number：E0P387 physicalSize: 558.9109999993816G slotNumber: 9 ----------Slot Number: 9 status: critical CellCLI> CellCLI> list griddisk where celldisk=CD_09_dm01cel02 DATA_DM01_CD_09_dm01cel02 not present -----状态“not present” DBFS_DG_CD_09_dm01cel02 not present RECO_DM01_CD_09_dm01cel02 not present CellCLI>
这里发现磁盘的报错信息跟alert是一致的：
SEAGATE Model Number : ST360057SSUN600G Serial Number : E0P387 Slot Number : 9 Cell Disk : CD_09_dm01cel02
磁盘的物理位置和编号如下：
我们这里是HDD9损坏。
此时从机柜边上观察，如果磁盘闪蓝灯则可以直接拔出，如果是闪橘色灯，那么需要手工关闭这个磁盘设备，然后再拔出：
alter physicaldisk 20:9 serviceled off
更换完成后需要检查：
1，磁盘的LED指示灯变为绿色
2，确认新换盘的celldisk，griddisk状态是normal
list celldisk where lun=0_9 detail
注意看： status: normal
CellCLI> list griddisk where celldisk=CD_09_dm01cel02 name: DATA_CD_09_dm01cel02 availableTo: cellDisk: CD_09_dm01cel02 。。。 errorCount: 0 。。。。。 status: active name: RECO_CD_09_dm01cel02 availableTo: cellDisk: CD_09_dm01cel02 。。。。。 errorCount: 0 。。。。。 status: active name: SYSTEMDG_CD_09_dm01cel02 availableTo: cellDisk: CD_09_dm01cel02 。。。。。 errorCount: 0 。。。。。。 status: active
在dm01cel02的alert中会有自动创建celldisk和griddisk的日志：
Fri Dec 06 18:18:20 2013 create CELLDISK CD_09_dm01cel02 on device /dev/sdj Fri Dec 06 18:18:20 2013 create GRIDDISK DATA_DM01_CD_09_dm01cel02 on CELLDISK CD_09_dm01cel02 Storage Index Allocation for GridDisk DATA_DM01_CD_09_dm01cel02 successful [code: 1] Griddisk DATA_DM01_CD_09_dm01cel02 - number is (21152) NOTE: Initiating ASM instance operation: Operation: DROP and ADD of ASM disks for 1 Grid disks guids. NOTE: Initiating ASM Instance operation: ASM DROP ADD disk on 1 disks Published 1 grid disk events ASM DROP ADD disk on DG DATA_DM01 to: ClientHostName = dm01db02.lunar.gov, ClientPID = 9994 Fri Dec 06 18:18:20 2013 create GRIDDISK RECO_DM01_CD_09_dm01cel02 on CELLDISK CD_09_dm01cel02 Storage Index Allocation for GridDisk RECO_DM01_CD_09_dm01cel02 successful [code: 1] Griddisk RECO_DM01_CD_09_dm01cel02 - number is (21156) NOTE: Initiating ASM instance operation: Operation: DROP and ADD of ASM disks for 1 Grid disks guids. NOTE: Initiating ASM Instance operation: ASM DROP ADD disk on 1 disks Published 1 grid disk events ASM DROP ADD disk on DG RECO_DM01 to: ClientHostName = dm01db01.lunar.gov, ClientPID = 10060 Fri Dec 06 18:18:20 2013 create GRIDDISK DBFS_DG_CD_09_dm01cel02 on CELLDISK CD_09_dm01cel02 Storage Index Allocation for GridDisk DBFS_DG_CD_09_dm01cel02 successful [code: 1] Griddisk DBFS_DG_CD_09_dm01cel02 - number is (21160) NOTE: Initiating ASM instance operation: Operation: DROP and ADD of ASM disks for 1 Grid disks guids. NOTE: Initiating ASM Instance operation: ASM DROP ADD disk on 1 disks Published 1 grid disk events ASM DROP ADD disk on DG DBFS_DG to: ClientHostName = dm01db01.lunar.gov, ClientPID = 10060
我们都知道，在存储节点上，有3个重要的服务：cellsrv，ms，restart service
其中，cellsrv负责跟计算节点通信，并执行相关的ioctl的操作
而ms负责监控服务，比如磁盘是否有add或者drop等，并把消息通知给cellsrv
而restart服务，主要是负责监控和重启其他的服务。
这里，我们清晰的看到，当新的磁盘插入后，cellsrv自动创建了celldisk，然后每创建一个griddisk就把先关信息通知到计算节点的ASM实例。
“Operation: DROP and ADD of ASM disks for 1 Grid disks guids.”
再看下ASM的日志：
计算节点的ASM实例收到来自cellsrv的通知，会启动XDWK进程（后面会讲），由该进程完成磁盘的drop和add等操作。
cellsrv通知ASM的XDMG进程，XDMG是负责监控磁盘的变化，捕获变化后，具体add和drop的工作是由XDWK完成的。
注意，XDMG和XDWK都是Exadata特有的，在计算节点的进程，其主要是能如下：
Exadata Automation Manager (XDMG) initiates automation tasks involved in managing Exadata storage. It monitors all configured storage cells for state changes, such as a failed disk getting replaced, and performs the required tasks for such events. Its primary tasks are to watch for inaccessible disks and cells and when they become accessible again, to initiate the ASM ONLINE operation.
Exadata Automation Manager (XDWK) performs automation tasks requested by XDMG.
It gets started when asynchronous actions such as disk ONLINE, DROP and ADD are requested by XDMG. After a 5 minute period of inactivity, this process will shut itself down.
cell上相关的配置文件位置：
$OSSCONF/cell_disk_config.xml 和 $OSSCONF/griddisk.owners.dat
ASM里面相关的初始化参数：
_AUTO_MANAGE_EXADATA_DISKS controls the auto disk management feature. To disable the feature set this parameter to FALSE. Range of values: TRUE [default] or FALSE.
_AUTO_MANAGE_NUM_TRIES controls the maximum number of attempts to perform an automatic operation. Range of values: 1-10. Default value is 2.
_AUTO_MANAGE_MAX_ONLINE_TRIES controls maximum number of attempts to ONLINE a disk. Range of values: 1-10. Default value is 3.
这两个进程都是可以kill的进程，也就是说某种意义上我们说的非核心进程，需要的时候，他们会自动启动。
这里，我们清晰的看到XDWK进程会自动每隔15分钟启动一次：
....... Sat Jun 08 15:21:57 2013 Starting background process XDWK Sat Jun 08 15:21:57 2013 XDWK started with pid=39, OS id=13234 Sat Jun 08 15:37:00 2013 Starting background process XDWK Sat Jun 08 15:37:00 2013 XDWK started with pid=39, OS id=27010 Sat Jun 08 15:52:03 2013 Starting background process XDWK Sat Jun 08 15:52:03 2013 XDWK started with pid=39, OS id=8806 Sat Jun 08 16:07:06 2013 Starting background process XDWK Sat Jun 08 16:07:06 2013 XDWK started with pid=39, OS id=23223 Sat Jun 08 16:22:09 2013 Starting background process XDWK Sat Jun 08 16:22:09 2013 XDWK started with pid=39, OS id=4888 Sat Jun 08 16:37:12 2013 Starting background process XDWK Sat Jun 08 16:37:12 2013 XDWK started with pid=39, OS id=19193 Sat Jun 08 16:52:15 2013 Starting background process XDWK Sat Jun 08 16:52:15 2013 XDWK started with pid=39, OS id=495 Sat Jun 08 17:07:18 2013 Starting background process XDWK Sat Jun 08 17:07:18 2013 XDWK started with pid=39, OS id=15095 Sat Jun 08 17:22:21 2013 Starting background process XDWK Sat Jun 08 17:22:21 2013 XDWK started with pid=39, OS id=29028 Sat Jun 08 17:37:24 2013 Starting background process XDWK Sat Jun 08 17:37:24 2013 XDWK started with pid=39, OS id=11007 ......
在有些老版本的11.2上，会有类似如下信息：
Wed Oct 30 13:18:33 2013 Starting background process XDWK Wed Oct 30 13:18:33 2013 XDWK started with pid=33, OS id=5049 WARNING: Exadata Auto Management: OS PID: 5049 Operation ID: 2782: DROP and ADD disk DBFS_DG_CD_07_xxx in diskgroup DBFSEXP Failed SQL : Cause : Action : Check alert log to see why this operation failed. Also check process trace file for matching Operation ID. WARNING: Exadata Auto Management: OS PID: 5049 Operation ID: 2786: DROP and ADD disk ARCH_CD_07_xxx in diskgroup ARCH Failed SQL : Cause : Action : Check alert log to see why this operation failed. Also check process trace file for matching Operation ID.
而相应的trace类似如下信息：
+ASM1_xdwk_2368 trace:
===============================
2013-10-30 10:33:01.075659 : kxdam_drop_add_disk: Disk DBFS_DG_CD_07_xxx dropped from Diskgroup DBFSEXP has a valid header. Cannot add it to diskgroup due to inconsistent state 2013-10-30 10:33:01.076217 : kxdam_drop_add_disk: Operation ID: 2672: DROP and ADD disk DBFS_DG_CD_07_xxx in diskgroup DBFSEXP Failed. SQL : Cause : Action : Check alert log to see why this operation failed. Also check process trace file for matching Operation ID. 2013-10-30 10:33:01.080131 : kxdam_validate_online: Operation ID: 2672: DROP and ADD disk DBFS_DG_CD_07_xxx in diskgroup DBFSEXP Failed. SQL : Cause : Action : Check alert log to see why this operation failed. Also check process trace file for matching Operation ID. 2013-10-30 10:33:01.087534 : kxdam_drop_add_disk: Disk ARCH_CD_07_xxx dropped from Diskgroup ARCH has a valid header. Cannot add it to diskgroup due to inconsistent state 2013-10-30 10:33:01.087602 : kxdam_drop_add_disk: Operation ID: 2676: DROP and ADD disk ARCH_CD_07_xxx in diskgroup ARCH Failed. SQL : Cause : Action : Check alert log to see why this operation failed. Also check process trace file for matching Operation ID.
也就是说，有可能会有自动添加磁盘失败的情况，那么可以kill掉xdmg和xdwk，然后手工添加磁盘：
ps -ef | grep -i xdmg kill -9 <xdmg PID> ps -ef | grep -i xdwk kill -9 <xdwk PID> sqlplus / as sysasm SQL> alter diskgroup <DG name> add disk <path of disk being added>;
XDWK添加了磁盘后，GMON进程会将新的磁盘信息等更新到ASM的metadata区域（更新ASM的元数据）
再之后，ARB0进程会负责具体的rebalance工作。
本次加盘的ASM日志如下：
Fri Dec 06 18:18:20 2013 Starting background process XDWK Fri Dec 06 18:18:20 2013 XDWK started with pid=32, OS id=12131 Fri Dec 06 18:18:23 2013 NOTE: disk validation pending for group 1/0x36c8734d (DATA_DM01) SUCCESS: validated disks for 1/0x36c8734d (DATA_DM01) NOTE: disk validation pending for group 1/0x36c8734d (DATA_DM01) NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_DM01_CD_09_dm01cel02) SUCCESS: validated disks for 1/0x36c8734d (DATA_DM01) NOTE: membership refresh pending for group 1/0x36c8734d (DATA_DM01) Fri Dec 06 18:18:32 2013 GMON querying group 1 at 63 for pid 19, osid 10052 NOTE: cache opening disk 21 of grp 1: DATA_DM01_CD_09_DM01CEL02 path:o/192.168.10.4/DATA_DM01_CD_09_dm01cel02 GMON querying group 1 at 64 for pid 19, osid 10052 SUCCESS: refreshed membership for 1/0x36c8734d (DATA_DM01) Fri Dec 06 18:18:34 2013 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_xdwk_12131.trc: Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_xdwk_12131.trc: SQL> /* Exadata Auto Mgmt: ADD ASM Disk in given FAILGROUP */ alter diskgroup RECO_DM01 add failgroup DM01CEL02 disk 'o/192.168.10.4/RECO_DM01_CD_09_dm01cel02' name RECO_DM01_CD_09_DM01CEL02 rebalance nowait NOTE: Assigning number (3,21) to disk (o/192.168.10.4/RECO_DM01_CD_09_dm01cel02) NOTE: requesting all-instance membership refresh for group=3 Fri Dec 06 18:18:36 2013 NOTE: Attempting voting file refresh on diskgroup DATA_DM01 NOTE: initializing header on grp 3 disk RECO_DM01_CD_09_DM01CEL02 NOTE: requesting all-instance disk validation for group=3 NOTE: skipping rediscovery for group 3/0x36d8734f (RECO_DM01) on local instance. NOTE: requesting all-instance disk validation for group=3 NOTE: skipping rediscovery for group 3/0x36d8734f (RECO_DM01) on local instance. Fri Dec 06 18:18:43 2013 GMON updating for reconfiguration, group 3 at 65 for pid 32, osid 12131 NOTE: group 3 PST updated. NOTE: initiating PST update: grp = 3 GMON updating group 3 at 66 for pid 32, osid 12131 NOTE: PST update grp = 3 completed successfully NOTE: membership refresh pending for group 3/0x36d8734f (RECO_DM01) GMON querying group 3 at 67 for pid 19, osid 10052 NOTE: cache opening disk 21 of grp 3: RECO_DM01_CD_09_DM01CEL02 path:o/192.168.10.4/RECO_DM01_CD_09_dm01cel02 Fri Dec 06 18:18:48 2013 NOTE: Attempting voting file refresh on diskgroup RECO_DM01 GMON querying group 3 at 68 for pid 19, osid 10052 SUCCESS: refreshed membership for 3/0x36d8734f (RECO_DM01) Fri Dec 06 18:18:48 2013 SUCCESS: /* Exadata Auto Mgmt: ADD ASM Disk in given FAILGROUP */ alter diskgroup RECO_DM01 add failgroup DM01CEL02 disk 'o/192.168.10.4/RECO_DM01_CD_09_dm01cel02' name RECO_DM01_CD_09_DM01CEL02 rebalance nowait NOTE: starting rebalance of group 3/0x36d8734f (RECO_DM01) at power 4 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_xdwk_12131.trc: Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_xdwk_12131.trc: Starting background process ARB0 SQL> /* Exadata Auto Mgmt: ADD ASM Disk in given FAILGROUP */ alter diskgroup DBFS_DG add failgroup DM01CEL02 disk 'o/192.168.10.4/DBFS_DG_CD_09_dm01cel02' name DBFS_DG_CD_09_DM01CEL02 rebalance nowait Fri Dec 06 18:18:48 2013 ARB0 started with pid=39, OS id=12553 NOTE: assigning ARB0 to group 3/0x36d8734f (RECO_DM01) with 4 parallel I/Os NOTE: Assigning number (2,17) to disk (o/192.168.10.4/DBFS_DG_CD_09_dm01cel02) NOTE: requesting all-instance membership refresh for group=2 NOTE: Attempting voting file refresh on diskgroup RECO_DM01 NOTE: initializing header on grp 2 disk DBFS_DG_CD_09_DM01CEL02 NOTE: requesting all-instance disk validation for group=2 NOTE: skipping rediscovery for group 2/0x36d8734e (DBFS_DG) on local instance. NOTE: requesting all-instance disk validation for group=2 NOTE: skipping rediscovery for group 2/0x36d8734e (DBFS_DG) on local instance. Fri Dec 06 18:18:58 2013 GMON updating for reconfiguration, group 2 at 69 for pid 32, osid 12131 Fri Dec 06 18:18:58 2013 NOTE: group 2 PST updated. NOTE: initiating PST update: grp = 2 GMON updating group 2 at 70 for pid 32, osid 12131 NOTE: PST update grp = 2 completed successfully Fri Dec 06 18:19:02 2013 NOTE: membership refresh pending for group 2/0x36d8734e (DBFS_DG) GMON querying group 2 at 71 for pid 19, osid 10052 NOTE: cache opening disk 17 of grp 2: DBFS_DG_CD_09_DM01CEL02 path:o/192.168.10.4/DBFS_DG_CD_09_dm01cel02 NOTE: Attempting voting file refresh on diskgroup DBFS_DG Fri Dec 06 18:19:16 2013 GMON querying group 2 at 72 for pid 19, osid 10052 Fri Dec 06 18:19:16 2013 SUCCESS: refreshed membership for 2/0x36d8734e (DBFS_DG) Fri Dec 06 18:19:16 2013 SUCCESS: /* Exadata Auto Mgmt: ADD ASM Disk in given FAILGROUP */ alter diskgroup DBFS_DG add failgroup DM01CEL02 disk 'o/192.168.10.4/DBFS_DG_CD_09_dm01cel02' name DBFS_DG_CD_09_DM01CEL02 rebalance nowait NOTE: Attempting voting file refresh on diskgroup DBFS_DG Fri Dec 06 18:33:49 2013 Starting background process XDWK Fri Dec 06 18:33:49 2013 XDWK started with pid=32, OS id=26304 。。。。。。。。。。。。
此时，你可以在ASM实例中检查rebanlance操作：
select * from gv$asm_operation;
通常在添加完磁盘后，我们会检查几件事情：
1，查看机柜的磁盘LED指示灯是否变成了绿色
2，用cellcli查看celldisk和griddisk等的信息
3，查看cell的alert日志
4，查看ASM的日志
5，查看ASM磁盘组和磁盘组信息
