ASM磁盘头被fdisk损坏的修复过程

发表于 2013 年 6 月 16 日 由 Lunar

联系：QQ(5163721)

标题：ASM磁盘头被fdisk损坏的修复过程

作者：Lunar©版权所有[文章允许转载,但必须以链接方式注明源地址,否则追究法律责任.]

一大早起来折腾昨天的12c（我装的是standalone），发现使用文件虚拟成设备的方法，磁盘IO效率很低（我猜是这个原因），于是铲掉打算重新安装
铲掉12c RAC跟铲掉11.2 RAC没啥区别，参考前面的文章 5分钟内搞定。

安装完grid，感觉磁盘不够用，于是把vm停了，加一块新的盘，然后启动后，fdisk /dev/sde
悲剧了，刚弄完就想起来，这个是ASM的DATADG…………于是，你懂的……

查看has的状态：
[root@lunar ~]# crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               OFFLINE OFFLINE      lunar                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunar                    STABLE
ora.asm
               ONLINE  OFFLINE      lunar                    Instance Shutdown,ST
                                                             ABLE
ora.ons
               OFFLINE OFFLINE      lunar                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  OFFLINE      lunar                    STARTING
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  INTERMEDIATE lunar                    STABLE
--------------------------------------------------------------------------------
[root@lunar ~]#

查看日志,ora.DATA.dg 资源状态异常：

[root@lunar ~]# vi /u01/app/12.1/grid/log/lunar/agent/ohasd/oraagent_grid/oraagent_grid.log

Oracle Database 12c Clusterware Release 12.1.0.1.0 - Production Copyright 1996, 2013 Oracle. All rights reserved.
2013-06-16 09:28:33.678: [    AGFW][274839072] Starting the agent: /u01/app/12.1/grid/log/lunar/agent/ohasd/oraagent_grid/
2013-06-16 09:28:33.678: [   AGENT][274839072] Agent framework initialized, Process Id = 7389
2013-06-16 09:28:33.693: [ USRTHRD][274839072] ConnectionPool::initMutex

...

2013-06-16 09:59:05.391: [ora.DATA.dg][1111324992] {0:0:123} [clean] (:CLSN00106:) clsn_agent::clean }
2013-06-16 09:59:05.391: [    AGFW][1111324992] {0:0:123} Command: clean for resource: ora.DATA.dg lunar 1 completed with status: SUCCESS
2013-06-16 09:59:05.392: [    AGFW][1100106048] {0:0:123} Agent sending reply for: RESOURCE_CLEAN[ora.DATA.dg lunar 1] ID 4100:606
2013-06-16 09:59:05.398: [    GPNP][1082648896]clsgpnpwu_walletfopen: [at clsgpnpwu.c:379] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP01603:)Wallet file not found: '/u01/app/12.1/grid/gpnp/lunar/wallets/prdr/cwallet.sso'
2013-06-16 09:59:05.398: [    GPNP][1082648896]clsgpnpwu_walletfopen: [at clsgpnpwu.c:497] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP01610:)Cannot open wallet: '/u01/app/12.1/grid/gpnp/lunar/wallets/prdr/cwallet.sso'
2013-06-16 09:59:05.399: [    GPNP][1082648896]clsgpnp_getCK: [at clsgpnp0.c:2472] (:GPNP00233:)Fatal error: failed to get local gpnp security keys (wallet). Gpnp profiles cannot be verified. ***LOCAL GPNP SETUP INVALID***.>
2013-06-16 09:59:05.399: [    GPNP][1082648896]clsgpnp_InitIdSetCtxCK: [at clsgpnp0.c:5151] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00267:)Cannot get stored certkey for id=1
2013-06-16 09:59:05.399: [    GPNP][1082648896]clsgpnp_Init: [at clsgpnp0.c:1003] Result: (66) CLSGPNP_WALLET_NONE. (:GPNP00211:)Error getting certkeys.
2013-06-16 09:59:05.399: [    GPNP][1082648896]clsgpnp_Init init failed. Error: CLSGPNP_ERR (1) .
2013-06-16 09:59:05.601: [ default][1082648896](:CLSUG0001:)Failed to initialize clsgpnp context grv CLSGPNP_ERR (1). Return GPNP [12]
2013-06-16 09:59:05.602: [ default][1082648896]clsugam:3: Failed to get gpnp profile. Return GPNP [12].
2013-06-16 09:59:05.602: [ora.DATA.dg][1082648896] {0:0:123} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 1 useFilter 0

使用kfed可以清晰的看到，盘头损坏了：

[grid@lunar ~]$ kfed dev=/dev/sde op=READ
kfbh.endian:                          0 ; 0x000: 0x00
kfbh.hard:                            0 ; 0x001: 0x00
kfbh.type:                            0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt:                          0 ; 0x003: 0x00
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:                       0 ; 0x008: file=0
kfbh.check:                           0 ; 0x00c: 0x00000000
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
000000000 00000000 00000000 00000000 00000000  [................]
        Repeat 30 times
0000001F0 00000000 00000000 00000000 AA550000  [..............U.]
000000200 00000000 00000000 00000000 00000000  [................]
  Repeat 223 times
KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

[grid@lunar ~]$ 

正常的盘头应该是：   kfbh.type=KFBTYP_DISKHEAD

在ASM中也可以看到，/dev/sdb原本是DATA DG的设备（HEADER_STATU应为 MEMBER），现在确变成“CANDIDATE”：

SQL> set pages 999
SQL> set linesize 122
SQL> select DISK_NUMBER,HEADER_STATUS,substr(PATH,1,20),label from v$asm_disk;

DISK_NUMBER HEADER_STATU SUBSTR(PATH,1,20)    LABEL
----------- ------------ -------------------- -------------------------------
          1 CANDIDATE    /dev/sde
          0 CANDIDATE    /dev/sdb

SQL>

我们知道，每个ASM磁盘的UNIT 1，块254（Allocation unit# 1, Block# 254）是盘头的备份，因此查看下，这个块是否是好的：

[grid@lunar ~]$ kfed read /dev/sde aun=1 blkn=254 |grep KFBTYP
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
[grid@lunar ~]$ 

行了，可以使用repair进行修复了（10.2.0.5以后的版本可以使用这个方法）：
1
[grid@lunar ~]$  kfed  repair /dev/sde
[grid@lunar ~]$ 

修复后，再次查看，已经正常了：
[grid@lunar ~]$ kfed read /dev/sde aun=0 blkn=0 | grep KFBTYP
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
[grid@lunar ~]$ 

SQL> select DISK_NUMBER,HEADER_STATUS,substr(PATH,1,20),label from v$asm_disk;

DISK_NUMBER HEADER_STATU SUBSTR(PATH,1,20)    LABEL
----------- ------------ -------------------- -------------------------------
          1 MEMBER       /dev/sde
          0 CANDIDATE    /dev/sdb

SQL> 

[grid@lunar ~]$ asmcmd lsdsk -p
Group_Num  Disk_Num      Incarn  Mount_Stat  Header_Stat  Mode_Stat  State   Path
        1         0  3916001478  CACHED      MEMBER       ONLINE     NORMAL  /dev/sde
[grid@lunar ~]$ 

[grid@lunar ~]$ kfed dev=/dev/sde op=READ
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483648 ; 0x008: disk=0
kfbh.check:                  2821357814 ; 0x00c: 0xa82a80f6
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr:         ORCLDISK ; 0x000: length=8
kfdhdb.driver.reserved[0]:            0 ; 0x008: 0x00000000
kfdhdb.driver.reserved[1]:            0 ; 0x00c: 0x00000000
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                202375168 ; 0x020: 0x0c100000
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
kfdhdb.grptyp:                        1 ; 0x026: KFDGTP_EXTERNAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:               DATA_0000 ; 0x028: length=9
kfdhdb.grpname:                    DATA ; 0x048: length=4
kfdhdb.fgname:                DATA_0000 ; 0x068: length=9
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32987657 ; 0x0a8: HOUR=0x9 DAYS=0x10 MNTH=0x6 YEAR=0x7dd
kfdhdb.crestmp.lo:           2046686208 ; 0x0ac: USEC=0x0 MSEC=0x37d SECS=0x1f MINS=0x1e
kfdhdb.mntstmp.hi:             32987657 ; 0x0b0: HOUR=0x9 DAYS=0x10 MNTH=0x6 YEAR=0x7dd
kfdhdb.mntstmp.lo:           2072964096 ; 0x0b4: USEC=0x0 MSEC=0x3bb SECS=0x38 MINS=0x1e
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.mfact:                    113792 ; 0x0c0: 0x0001bc80
kfdhdb.dsksize:                   10240 ; 0x0c4: 0x00002800
kfdhdb.pmcnt:                         3 ; 0x0c8: 0x00000003
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                     10 ; 0x0d4: 0x0000000a
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000
kfdhdb.dbcompat:              168820736 ; 0x0e0: 0x0a100000
kfdhdb.grpstmp.hi:             32987657 ; 0x0e4: HOUR=0x9 DAYS=0x10 MNTH=0x6 YEAR=0x7dd
kfdhdb.grpstmp.lo:           2046528512 ; 0x0e8: USEC=0x0 MSEC=0x2e3 SECS=0x1f MINS=0x1e
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
kfdhdb.spfile:                       59 ; 0x0f4: 0x0000003b
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001
kfdhdb.flags:                         1 ; 0x0fc: 0x00000001
kfdhdb.f1b1fcn.base:                  0 ; 0x100: 0x00000000
kfdhdb.f1b1fcn.wrap:                  0 ; 0x104: 0x00000000
kfdhdb.ub4spare[0]:                   0 ; 0x108: 0x00000000
kfdhdb.ub4spare[1]:                   0 ; 0x10c: 0x00000000
kfdhdb.ub4spare[2]:                   0 ; 0x110: 0x00000000
kfdhdb.ub4spare[3]:                   0 ; 0x114: 0x00000000
kfdhdb.ub4spare[4]:                   0 ; 0x118: 0x00000000
kfdhdb.ub4spare[5]:                   0 ; 0x11c: 0x00000000
kfdhdb.ub4spare[6]:                   0 ; 0x120: 0x00000000
kfdhdb.ub4spare[7]:                   0 ; 0x124: 0x00000000
kfdhdb.ub4spare[8]:                   0 ; 0x128: 0x00000000
kfdhdb.ub4spare[9]:                   0 ; 0x12c: 0x00000000
kfdhdb.ub4spare[10]:                  0 ; 0x130: 0x00000000
kfdhdb.ub4spare[11]:                  0 ; 0x134: 0x00000000
kfdhdb.ub4spare[12]:                  0 ; 0x138: 0x00000000
kfdhdb.ub4spare[13]:                  0 ; 0x13c: 0x00000000
kfdhdb.ub4spare[14]:                  0 ; 0x140: 0x00000000
kfdhdb.ub4spare[15]:                  0 ; 0x144: 0x00000000
kfdhdb.ub4spare[16]:                  0 ; 0x148: 0x00000000
kfdhdb.ub4spare[17]:                  0 ; 0x14c: 0x00000000
kfdhdb.ub4spare[18]:                  0 ; 0x150: 0x00000000
kfdhdb.ub4spare[19]:                  0 ; 0x154: 0x00000000
kfdhdb.ub4spare[20]:                  0 ; 0x158: 0x00000000
kfdhdb.ub4spare[21]:                  0 ; 0x15c: 0x00000000
kfdhdb.ub4spare[22]:                  0 ; 0x160: 0x00000000
kfdhdb.ub4spare[23]:                  0 ; 0x164: 0x00000000
kfdhdb.ub4spare[24]:                  0 ; 0x168: 0x00000000
kfdhdb.ub4spare[25]:                  0 ; 0x16c: 0x00000000
kfdhdb.ub4spare[26]:                  0 ; 0x170: 0x00000000
kfdhdb.ub4spare[27]:                  0 ; 0x174: 0x00000000
kfdhdb.ub4spare[28]:                  0 ; 0x178: 0x00000000
kfdhdb.ub4spare[29]:                  0 ; 0x17c: 0x00000000
kfdhdb.ub4spare[30]:                  0 ; 0x180: 0x00000000
kfdhdb.ub4spare[31]:                  0 ; 0x184: 0x00000000
kfdhdb.ub4spare[32]:                  0 ; 0x188: 0x00000000
kfdhdb.ub4spare[33]:                  0 ; 0x18c: 0x00000000
kfdhdb.ub4spare[34]:                  0 ; 0x190: 0x00000000
kfdhdb.ub4spare[35]:                  0 ; 0x194: 0x00000000
kfdhdb.ub4spare[36]:                  0 ; 0x198: 0x00000000
kfdhdb.ub4spare[37]:                  0 ; 0x19c: 0x00000000
kfdhdb.ub4spare[38]:                  0 ; 0x1a0: 0x00000000
kfdhdb.ub4spare[39]:                  0 ; 0x1a4: 0x00000000
kfdhdb.ub4spare[40]:                  0 ; 0x1a8: 0x00000000
kfdhdb.ub4spare[41]:                  0 ; 0x1ac: 0x00000000
kfdhdb.ub4spare[42]:                  0 ; 0x1b0: 0x00000000
kfdhdb.ub4spare[43]:                  0 ; 0x1b4: 0x00000000
kfdhdb.ub4spare[44]:                  0 ; 0x1b8: 0x00000000
kfdhdb.ub4spare[45]:                  0 ; 0x1bc: 0x00000000
kfdhdb.ub4spare[46]:                  0 ; 0x1c0: 0x00000000
kfdhdb.ub4spare[47]:                  0 ; 0x1c4: 0x00000000
kfdhdb.ub4spare[48]:                  0 ; 0x1c8: 0x00000000
kfdhdb.ub4spare[49]:                  0 ; 0x1cc: 0x00000000
kfdhdb.ub4spare[50]:                  0 ; 0x1d0: 0x00000000
kfdhdb.acdb.aba.seq:                  0 ; 0x1d4: 0x00000000
kfdhdb.acdb.aba.blk:                  0 ; 0x1d8: 0x00000000
kfdhdb.acdb.ents:                     0 ; 0x1dc: 0x0000
kfdhdb.acdb.ub2spare:                 0 ; 0x1de: 0x0000
[grid@lunar ~]$ 

[grid@lunar ~]$ kfed dev=/dev/sde op=READ| egrep "kfdhdb.vf|ausize"
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
[grid@lunar ~]$

手工把ASM磁盘组DATA挂载上：

[root@lunar ~]# srvctl start diskgroup -diskgroup data
[root@lunar ~]# crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       lunar                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       lunar                    STABLE
ora.asm
               ONLINE  ONLINE       lunar                    Started,STABLE
ora.ons
               OFFLINE OFFLINE      lunar                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       lunar                    STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.evmd
      1        ONLINE  ONLINE       lunar                    STABLE
--------------------------------------------------------------------------------
[root@lunar ~]#

好了，可以安装db软件，然后建库了…………

此条目发表在 ASM, backup&recovery 分类目录，贴了 , 标签。将固定链接加入收藏夹。

《ASM磁盘头被fdisk损坏的修复过程》有 1 条评论

  1. 小鹏 说：
    2013 年 6 月 16 日18:52

    女神的博客干货真多。喜欢

    回复

发表评论

电子邮件地址不会被公开。 必填项已用 * 标注