解决Exadata上IB检查脚本infinicheck的报错过程

今天检查Exadata的IB网络时,使用 infinicheck 检查,发现db节点有报错,cell节点正常。
当前主机是Exadata X5-2:

[root@dm01db01 ibdiagtools]# /opt/oracle.SupportTools/CheckHWnFWProfile -c loose
[SUCCESS] The hardware and firmware matches supported profile for server=ORACLE_SERVER_X5-2
[root@dm01db01 ibdiagtools]# 

infinicheck的执行结果(该命令可以有很丰富的参数,但是也可以不带任何参数,缺省就可以):

                        INFINICHECK
                [Network Connectivity, Configuration and Performance]
                [Version IBD VER 2.d ]

 Verifying User Equivalance of user=root to all hosts.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)

 Verifying User Equivalance of user=root to all cells.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)


                    ####  CONNECTIVITY TESTS  ####
                    [COMPUTE NODES -> STORAGE CELLS]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All  can talk to all storage cells

        Verifying Subnet Masks on all nodes

[SUBNET MASKS DIFFER].....2 entries found

         Prechecking for uniformity of rds-tools on all nodes
[SUCCESS].... rds-tools version is the same across the cluster

        Checking for bad links in the fabric
[SUCCESS].......... No bad fabric links found

                    [COMPUTE NODES -> COMPUTE NODES]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All hosts can talk to all other nodes


                    ####  PERFORMANCE TESTS  ####

                    [(1) Storage Cell to Compute Node]
                                 (375 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off

                    [(2) Every COMPUTE NODE to another COMPUTE NODE]
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.2 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.1 and 192.168.10.4 has been started.
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off

                    [(3) Every COMPUTE NODE to ALL STORAGE CELLS]
                        (looking for SymbolErrors)
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.3  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off
[CRITICAL].............192.168.10.4  rds-stress commands did not run as expected on this host.PLEASE run [./infinicheck -z] to cleanup before re-run.PLEASE ensure that user equivalence for root is setup (./infinicheck -s) Also ensure all other workloads are turned off

[SUCCESS]....... No port errors found
Infinicheck failures reported.. please check log files

从这里我们看到,凡是到db节点的都报错。

infinicheck命令底层是调用的rds-stress命令,例如: rds-stress -r 192.168.10.1 -p 10584

当然,除了infinicheck意外,还有其他很多检查方法,比如rds-ping(ExaWatcher和OSWatcher中调用的这个命令)。
很奇怪,为什么就db节点报错?
于是,使用infinicheck 带参数-b -g 来检查和配置一下DB节点的IB的SSH连通性:
这里我犯了个错误:这个命令需要配置IB的基于IP的SSH(root),而不是主机名


                        INFINICHECK
                [Network Connectivity, Configuration and Performance]
                [Version IBD VER 2.d ]
ping: unknown host dm01db01-priv
[FAILURE] Host dm01db01-priv is Unreachable and is excluded from testing
ping: unknown host dm01db02-priv
[FAILURE] Host dm01db02-priv is Unreachable and is excluded from testing

 Please supply Infiniband IP addresses only in cell_ib_group

这里很清晰的告诉我们,ping不通,O(∩_∩)O哈哈~,这个就好办了。
接下来,我们手工ping看看:

[root@dm01db01 ~]# ping  dm01db02-priv
ping: unknown host 
[root@dm01db01 ~]# 

那么ping第2个节点的主机名试试看,证实一下是不是解析的问题:

[root@dm01db01 ~]# ping 192.168.10.3
PING 192.168.10.3 (192.168.10.3) 56(84) bytes of data.
64 bytes from 192.168.10.3: icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from 192.168.10.3: icmp_seq=2 ttl=64 time=0.025 ms
^C
--- 192.168.10.3 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1902ms
rtt min/avg/max/mdev = 0.025/0.025/0.026/0.005 ms
[root@dm01db01 ~]# 

这里我们看到,果然是解析的问题。
由于IB网络是Exadata内部互联用的,因此没有在DNS解析,只在/etc/hosts中解析。
而/etc/hosts文件是由onecommand配置的(除非手工安装,否则使用了onecommand后,所有配置文件都由onecommand根据配置xml文件自动生成)

[root@dm01db01 ~]# cat /etc/hosts
#### BEGIN Generated by Exadata. DO NOT MODIFY ####
127.0.0.1 localhost.localdomain localhost

# 192.168.10.1 dm01db01-priv1.lunar.com dm01db01-priv1
# 192.168.10.2 dm01db01-priv2.lunar.com dm01db01-priv2
此处略去了vip和scan ip
#### END Generated by Exadata ####
#### BEGIN Added by Configuration Utility #### 
192.168.10.1 dm01db01-priv1 dm01db01-priv1.800best.com
192.168.10.10 dm01cel03-priv2 dm01cel03-priv2.800best.com
192.168.10.2 dm01db01-priv2 dm01db01-priv2.800best.com
192.168.10.3 dm01db02-priv1 dm01db02-priv1.800best.com
192.168.10.4 dm01db02-priv2 dm01db02-priv2.800best.com
192.168.10.5 dm01cel01-priv1 dm01cel01-priv1.800best.com
192.168.10.6 dm01cel01-priv2 dm01cel01-priv2.800best.com
192.168.10.7 dm01cel02-priv1 dm01cel02-priv1.800best.com
192.168.10.8 dm01cel02-priv2 dm01cel02-priv2.800best.com
192.168.10.9 dm01cel03-priv1 dm01cel03-priv1.800best.com
#### END Added by Configuration Utility ####
[root@dm01db01 ~]# 

从这里我们看到,IB网络的IP配置格式是错误的,正确的是:
127.0.0.1 localhost.localdomain localhost
错误的是:
192.168.10.1 dm01db01-priv1.lunar.com dm01db01-priv1
修改了上述hosts文件后,
纠正hosts文件后,发现ping主机名的问题解决了:

[root@dm01db01 ~]# ping dm01db01-priv1
PING dm01db01-priv1.800best.com (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=2 ttl=64 time=0.010 ms
^C
--- dm01db01-priv1.800best.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1586ms
rtt min/avg/max/mdev = 0.010/0.010/0.010/0.000 ms
[root@dm01db01 ~]# ping dm01db01-priv1.800best.com
PING dm01db01-priv1.800best.com (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=1 ttl=64 time=0.011 ms
64 bytes from dm01db01-priv1.800best.com (192.168.10.1): icmp_seq=2 ttl=64 time=0.008 ms
^C
--- dm01db01-priv1.800best.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1525ms
rtt min/avg/max/mdev = 0.008/0.009/0.011/0.003 ms
[root@dm01db01 ~]# 

这里还有个问题很奇怪,cell节点的hosts文件也是错误的,但是却可以ping通,怀疑跟DNS缓存有关系:

[root@dm01cel02 ~]# cat /etc/hosts
#### BEGIN Generated by Exadata. DO NOT MODIFY ####
127.0.0.1 localhost.localdomain localhost

# 192.168.10.7 dm01cel02-priv1.800best.com dm01cel02-priv1
# 192.168.10.8 dm01cel02-priv2.800best.com dm01cel02-priv2
10.45.1.194 dm01cel02.800best.com dm01cel02
#### END Generated by Exadata ####
#### BEGIN Added by Configuration Utility #### 
192.168.10.1 dm01db01-priv1 dm01db01-priv1.800best.com
192.168.10.10 dm01cel03-priv2 dm01cel03-priv2.800best.com
192.168.10.2 dm01db01-priv2 dm01db01-priv2.800best.com
192.168.10.3 dm01db02-priv1 dm01db02-priv1.800best.com
192.168.10.4 dm01db02-priv2 dm01db02-priv2.800best.com
192.168.10.5 dm01cel01-priv1 dm01cel01-priv1.800best.com
192.168.10.6 dm01cel01-priv2 dm01cel01-priv2.800best.com
192.168.10.7 dm01cel02-priv1 dm01cel02-priv1.800best.com
192.168.10.8 dm01cel02-priv2 dm01cel02-priv2.800best.com
192.168.10.9 dm01cel03-priv1 dm01cel03-priv1.800best.com
#### END Added by Configuration Utility ####
[root@dm01cel02 ~]# ping dm01db01-priv1
PING dm01db01-priv1 (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=1 ttl=64 time=0.056 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=2 ttl=64 time=0.064 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=3 ttl=64 time=0.051 ms
^C
--- dm01db01-priv1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2637ms
rtt min/avg/max/mdev = 0.051/0.057/0.064/0.005 ms
[root@dm01cel02 ~]# ping dm01db01-priv1.800best.com
PING dm01db01-priv1 (192.168.10.1) 56(84) bytes of data.
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=2 ttl=64 time=0.027 ms
64 bytes from dm01db01-priv1 (192.168.10.1): icmp_seq=3 ttl=64 time=0.029 ms
^C
--- dm01db01-priv1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2833ms
rtt min/avg/max/mdev = 0.027/0.033/0.043/0.007 ms
[root@dm01cel02 ~]# 

现在,再次使用infinicheck 带参数-b -g 来检查一下DB节点的IB的SSH连通性,这次没问题了(过程很长,而且无需用户输入和交互,因此不列举在这里了)。
配置完成后,测试一下:

[root@dm01db01 oracle.SupportTools]# dcli -g all_ibip_group -l root 'date'
192.168.10.1: Sun Apr  5 08:14:58 CST 2015
192.168.10.2: Sun Apr  5 08:14:58 CST 2015
192.168.10.3: Sun Apr  5 08:14:59 CST 2015
192.168.10.4: Sun Apr  5 08:14:58 CST 2015
192.168.10.5: Sun Apr  5 08:14:58 CST 2015
192.168.10.6: Sun Apr  5 08:14:59 CST 2015
192.168.10.7: Sun Apr  5 08:14:58 CST 2015
192.168.10.8: Sun Apr  5 08:14:58 CST 2015
192.168.10.9: Sun Apr  5 08:14:58 CST 2015
192.168.10.10: Sun Apr  5 08:14:58 CST 2015
[root@dm01db01 oracle.SupportTools]# 

这里我们看到,每个节点有2个IP,是因为从11.2.3.3.0以后,Exadata为了增加带宽,不再绑定IB了,所有的IB私有网络类似一个大的VLAN。

下面我们再次执行一下infinicheck ,因为网络不好,infinicheck执行时间大概要5分钟以上,因此使用screen后台执行:

[root@dm01db01 ~]# screen -S lunar

[root@dm01db01 ~]# date;/opt/oracle.SupportTools/ibdiagtools/infinicheck;date
Sun Apr  5 08:33:13 CST 2015

                        INFINICHECK
                [Network Connectivity, Configuration and Performance]
                [Version IBD VER 2.d ]

 Verifying User Equivalance of user=root to all hosts.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)

 Verifying User Equivalance of user=root to all cells.
(If it isn't setup correctly, an authentication prompt will appear to push keys to all the nodes)


                    ####  CONNECTIVITY TESTS  ####
                    [COMPUTE NODES -> STORAGE CELLS]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All  can talk to all storage cells

        Verifying Subnet Masks on all nodes
[SUCCESS] ......... Subnet Masks is same across the network

         Prechecking for uniformity of rds-tools on all nodes
[SUCCESS].... rds-tools version is the same across the cluster

        Checking for bad links in the fabric
[SUCCESS].......... No bad fabric links found

                    [COMPUTE NODES -> COMPUTE NODES]
                           (30 seconds approx.)
[SUCCESS]..............Results OK

[SUCCESS]....... All hosts can talk to all other nodes


                    ####  PERFORMANCE TESTS  ####

                    [(1) Storage Cell to Compute Node]
                                 (375 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[SUCCESS]..............Results OK

                    [(2) Every COMPUTE NODE to another COMPUTE NODE]
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.2 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.1 and 192.168.10.4 has been started.
[SUCCESS]..............Results OK

                    [(3) Every COMPUTE NODE to ALL STORAGE CELLS]
                        (looking for SymbolErrors)
                                 (195 seconds approx)
[  INFO  ].............Performance test between 192.168.10.5 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.6 and 192.168.10.4 has been started.
[  INFO  ].............Performance test between 192.168.10.7 and 192.168.10.2 has been started.
[  INFO  ].............Performance test between 192.168.10.8 and 192.168.10.1 has been started.
[  INFO  ].............Performance test between 192.168.10.9 and 192.168.10.3 has been started.
[  INFO  ].............Performance test between 192.168.10.10 and 192.168.10.4 has been started.
[SUCCESS]..............Results OK

[SUCCESS]....... No port errors found
INFINICHECK REPORTS SUCCESS FOR NETWORK CONNECTIVITY and PERFORMANCE


----------DIAGNOSTICS -----------

6 Cell ips found: ..
192.168.10.5 | 192.168.10.6 | 192.168.10.7 | 192.168.10.8 | 192.168.10.9 | 192.168.10.10  
4 Host ips found: ..
192.168.10.3 | 192.168.10.4 | 192.168.10.2 | 192.168.10.1  

##########   Host to Cell Connectivity   ##########

Analyzing cells_conntest.log...


[SUCCESS]..... All nodes can talk to all other nodes


Now Analyzing Compute Node-Compute Node connectivity 

##########    Inter-Host Connectivity     ##########

Analyzing hosts_conntest.log...


[SUCCESS]..... All hosts can talk to all its peers

##########     Performance Diagnostics     ##########

###   [(1) STORAGE CELL to COMPUTE NODE    ######

Analyzing perf_cells.log.* logfile(s)....

        --------Throughput results using rds-stress --------
        2300 MB/s and above is expected for runs on quiet machines
dm01db02( 192.168.10.3 ) to dm01cel01( 192.168.10.5 ) : 3987 MB/s...OK
dm01db02( 192.168.10.4 ) to dm01cel01( 192.168.10.6 ) : 4204 MB/s...OK
dm01db01( 192.168.10.2 ) to dm01cel02( 192.168.10.7 ) : 3848 MB/s...OK
dm01db01( 192.168.10.1 ) to dm01cel02( 192.168.10.8 ) : 3876 MB/s...OK
dm01db02( 192.168.10.3 ) to dm01cel03( 192.168.10.9 ) : 3868 MB/s...OK
dm01db02( 192.168.10.4 ) to dm01cel03( 192.168.10.10 ) : 3971 MB/s...OK

##########     Performance Diagnostics     ##########

####      [(2) Every DBNODE to its PEER        ######

Analyzing perf_hosts.log.* logfile(s)....

        --------Throughput results using rds-stress --------
        2300 MB/s and above is expected for runs on quiet machines
dm01db02( 192.168.10.3 ) to dm01db01( 192.168.10.2 ) : 3958 MB/s...OK
dm01db02( 192.168.10.4 ) to dm01db01( 192.168.10.1 ) : 3865 MB/s...OK

-------------------------
Results are available in the file diagnostics.output
Sun Apr  5 08:39:30 CST 2015
[root@dm01db01 ~]#

这里我们看到,每秒的吞吐量在3.8GB/s~4GB左右(注意是大B),很强悍,O(∩_∩)O哈哈~

发表在 故障诊断 | 标签为 , | 留下评论

CPU E5 / E5 v2 / E7 v2的运行超过208天后,热重启后系统异常

昨天朋友(感谢bbq和很多热心朋友)告知一个Linux的bug:
E5 / E5 v2 / E7 v2的运行超过208天后,热重启后(比如 shutdown -r),TSC时钟不能被clearout,因此在重启后TSC会继续之前的计数,造成系统异常。
(涉及的Linux内核版本列举在最下面红帽说明中)
.
红帽的官方说明:
https://access.redhat.com/solutions/433883
.
由于Linux时钟机制引起的bug很多,
Linux中有3种timer:
1、Real Time Clock(RTC):RTC是位于CMOS中的
2、Programmalbe Interval Timer(PIT):PIT主要由8254时钟芯片实现的
3、Time Stamp Counter.(TSC):TSC的主体是位于CPU里面的一个64位的TSC寄存器。每个CPU时钟周期其值加一

类似的时间引起宕机的问题很多跟Linux的TSC时钟机制有关系,比如:


无标题


.
今天看见飞总也在讨论, 具体参见:
http://www.xifenfei.com/5760.html
.

Exadata X5目前没有发现这个问题:
Exadata的db node: CPU E5 2699 v3; cell node: CPU E5 2630 v3; db node和cell node都采用OEL 6.6
.
针对这个Linux 在E5 V2和E7 v2(RHEL 6.1~6.4)的bug,我补充一下Oracle对此的官方说明:
SYMPTOMS

Oracle Linux server bootup hangs upon the clock initialization stage after warm-reboot of the system. This issue could occur under the following conditions:

The system contains Intel Xeon E5 / E5v2 / E7v2 family processors, affected by Intel Errata BT81.
Approximately 208.5 days or more passed after the previous power-cycle.
Running either Oracle Linux 5.x with UEK, Oracle Linux 6.4 or earlier.
x86_64 system
Followings have no impact:

Any 32-bit system
Oracle Linux 5.x running Red Hat Kernel (2.6.18-x)
Oracle Linux 7.x
.
.
CHANGES

The operating system (OS) was recently rebooted without power-cycling.

.
.
CAUSE

The root cause of this issue the combination of these two below:

Intel Xeon processor E5 / E5 v2 / E7 v2 has the issue where TSC is not cleared upon receipt of reset signal. This makes TSC continues counting up when the operating system issued the warm-reboot.
x86 kernel booting code contains the clock initialization. In there, when set_cyc2ns_scale() is called, improper handling of possible overflow due to unexpectedly large value from TSC. This is similar to the overflow issue in cyc2ns_offset(), where returning value could overflow if TSC value is too large, which could occur approximately 208.5 days after power-cycle.
The cause is the same as the issue reported in the following documents, however, the previous fix did not cover all the potential overflows.

Linux OS reboots every 208 days (Doc ID 1482279.1)
System Crash After 208 Days Of Continuous Uptime (Doc ID 1496633.1)
SOLUTION

In a long term, please consider updating to the following versions of kernel:

For Oracle Linux 5.6, 5.7, and 5.8 with Unbreakable Enterprise Kernel (UEK),
Updating to the latest version of Unbreakable Enterprise Kernel Release 2 (UEK2) is the best solution. UEK2 version 2.6.39-300.0.4 or later covers the fix.
For Exadata 11.2.3.2.x with Oracle Linux 5.8, update to 11.2.3.2.2 which contains UEK1 backport fix as 2.6.32-400.34.1. Refer to Doc 1613511.1.
For Oracle Linux 6.0 – 6.4 with Unbrealable Enterprise Kernel Release 1 or 2 (UEK1/UEK2), update to the latest version of UEK2. In UEK2, version 2.6.39-300.0.4 or later contains the fix. Also consider updating to Unbreakable Enterprise Kernel Release 3 (UEK3), which is not affected by this issue.
For Oracle Linux 6.0 – 6.4 with Red Hat Compatible Kernel, update to kernel 2.6.32-358.23.2.el6 or later. Another option is to update Oracle Linux 6.5 (equivalent to 2.6.32-431.el6) or later.
For Oracle Linux customer with Long Term Support Contract, please consult with Oracle Linux Support.

同时,针对这个bug,Oracle给出了一个internal的workround:
For a short-term solution, power-cycling the system will resolve the issue, because this will forcibly reset TSC.
demo代码:
Example code for 2.6.32-358.23.2.el6 (arch/x86/kernel/tsc.c)
static void set_cyc2ns_scale(unsigned long cpu_khz, int cpu)
{
unsigned long long tsc_now, ns_now, *offset;
unsigned long flags, *scale;

local_irq_save(flags);
sched_clock_idle_sleep_event();

scale = &per_cpu(cyc2ns, cpu);
offset = &per_cpu(cyc2ns_offset, cpu);

rdtscll(tsc_now);
ns_now = __cycles_2_ns(tsc_now);

if (cpu_khz) {
*scale = ((NSEC_PER_MSEC << CYC2NS_SCALE_FACTOR) + cpu_khz / 2) / cpu_khz; *offset = ns_now - mult_frac(tsc_now, *scale, <<<<<<<<< (1UL << CYC2NS_SCALE_FACTOR)); <<<<<< } sched_clock_idle_wakeup_event(0); local_irq_restore(flags); } 红帽的官方说明:
https://access.redhat.com/solutions/433883

下面摘出来受影响的kernal列表:
Environment

Red Hat Enterprise Linux 6.1 (kernel-2.6.32-131.26.1.el6 and newer)
Red Hat Enterprise Linux 6.2 (kernel-2.6.32-220.4.2.el6 and newer)
Red Hat Enterprise Linux 6.3 (kernel-2.6.32-279 series)
Red Hat Enterprise Linux 6.4 (kernel-2.6.32-358 series)
Any Intel® Xeon® E5, Intel® Xeon® E5 v2, or Intel® Xeon® E7 v2 series processor
The issue has been observed in the following environments with 64-bit kernels. Notice that 32-bit kernels of the above mentioned versions are prone to the issue too.
.


无标题


发表在 Linux | 留下评论

Exadata X5 软件和硬件的新特性概览

Exadata X5的新特性,很有意思,软硬件都有很多改变,个人感觉最突出的硬件改变是X5只有两种磁盘配置:
1,全闪配置(EF)
2,SSD(Flash F160 NVMe PCIe card with 1.6 TB)+4T(HC DISK)
然后就是内存增加,CPU升级等等。。。

下面的信息来源于强大的MOS:
Oracle Exadata Database Machine X5-2 Support
……New Exadata X5-2 Database Server
.
Exadata X5-2 updates the 2-socket database servers to use the latest and fastest Intel Xeon E5-2699 v3 “Haswell” processors with 18 cores (vs. 12 cores in X4-2) for 50% greater performance. An X5-2 database server uses the faster DDR4 memory, and the default 256 GB configuration can now be expanded up to 768 GB, a 50% increase over the X4-2 maximum. The HBAs no longer depend on batteries to retain cached data, hence eliminating the need for preventive maintenance.
.
……New Exadata X5-2 High Capacity (HC) Storage Server
Exadata X5-2 updates the 2-socket storage server to an eight-core (33% increase) Intel Xeon E5-2630 v2 processor and the flash card to the Sun Accelerator Flash F160 NVMe PCIe card with 1.6 TB (100% increase). The HBAs in the storage servers also no longer use batteries which eliminates the need for planned maintenance. The disk drives remain at 4 TB, for a total of 48 TB per High Capacity Storage Server.
.
……Introducing Exadata X5-2 Extreme Flash (EF) Storage Server
Exadata X5-2 replaces the X4-2 High Performance Storage Server with an all PCI flash based storage server. Each Extreme Flash storage server has eight 1.6 TB NVMe PCIe flash drives, with a total raw capacity of 12.8 TB. Extreme Flash storage delivers consistently low latency and the highest levels of performance for scans and OLTP read/write IOPS. The Extreme Flash storage server uses the same eight-core Intel Xeon processor as the High Capacity Storage Server.
.
Oracle Exadata Database Machine Elastic Configurations
.
Elastic configurations allow Oracle Exadata Racks to have customer-defined combinations of database servers and Exadata Storage Servers. This allows the hardware configuration to be tailored to specific workloads such as Database In-Memory, OLTP, Data Warehousing, or Data Retention.
.
Sparse Grid Disks
.
Sparse grid disks allocate space as new data is written to the disk, and therefore have a virtual size that can be much larger than the actual physical size. Sparse grid disks can be used to create a sparse disk group to store database files that will use a small portion of their allocated space.
.
Snapshot Databases for Test and Development
.
Space efficient database snapshots can now be quickly created for test and development purposes. Snapshots start with a shared read-only copy of the production database (or pluggable database (PDB)) that has been cleansed of any sensitive information. As changes are made, each snapshot writes the changed blocks to a sparse disk group.
.
Columnar Flash Caching
Oracle Exadata Storage Server Software release 12.1.2.1.0 can efficiently support mixed workloads, delivering optimal performance for both OLTP and analytics. This is possible due to the dual format architecture of Smart Flash Cache that enables the data to be stored in hybrid columnar for transactional processing and pure columnar optimized for analytical processing.
Oracle Exadata Storage Server Software I/O Latency Capping for Write Operations
.
Oracle Exadata Storage Server Software automatically redirects high latency write I/O operations to another healthy flash device on the same cell. After the write is successfully completed on another healthy flash device, the write I/O is acknowledged as successful to the database thereby eliminating the write outlier. This feature requires write back flash cache to be enabled on the cell.
.
Elimination of False Drive Failures
.
In the event of an apparent hard drive failure on X5-2 High Capacity cell or an apparent flash drive failure on X5-2 Extreme Flash cell, Oracle Exadata Storage Server Software automatically redirects I/Os to other drives, and then power cycles the drive. If the drive returns to normal status after the power cycle, then it will be re-enabled and resynchronized. If the drive continues to fail after being power cycled, then it will be dropped. This feature allows Oracle Exadata Storage Server Software to eliminate false-positive disk failures and therefore helps to preserve data redundancy and reduce management.
.
Flash and Disk Life Cycle Management Alert
.
Oracle Exadata Storage Server Software now monitors Oracle ASM rebalance operations due to disk failure and replacement. Management Server sends an alert when a rebalance operation completes successfully, or encounters an error.
.
Performance Optimization for SQL Queries with Minimum or Maximum Functions
.
SQL queries using minimum or maximum functions are designed to take advantage of the storage index column summary that is cached in Exadata Storage Server memory. As a query is processed, a running minimum an a maximum value is tracked. Before issuing an I/O, the minimum/maximum value cached in the storage index for the data region is checked in conjunction with the running minimum/maximum value to decide whether that I/O should be issued or can be pruned. Overall, this optimization can result in significant I/O pruning during the course of a query and improves query performance.
.
Oracle Exadata Storage Server Software Performance Statistics in AWR Reports
.
Exadata Storage Server configuration and performance statistics are collected in an Automatic Workload Repository (AWR) report. The AWR report provides storage level performance statistics, not restricted to a specific instance or a database. This is useful for analyzing cases where one database can affect the performance of another database.
.
Exafusion Direct-to-Wire Protocol
.
Exafusion Direct-to-Wire protocol allows database processes to read and send Oracle Real Applications Cluster (Oracle RAC) messages directly over the Infiniband network bypassing the overhead of entering the OS kernel, and running the normal networking software stack. This improves the response time and scalability of the Oracle RAC environment on Oracle Exadata Database Machine.
.
Management Server on Database Servers
.
Oracle Exadata Database Machine Command-Line Interface (DBMCLI) utility is the command-line administration tool for managing the database servers. DBMCLI runs on each server to enable you to manage an individual database server.
SQL Operators for JSON and XML
Oracle Exadata Storage Server Software supports offload of many SQL operators for predicate evaluation.
.
I/O Resource Management for Flash
.
I/O Resource Management (IORM) now manages flash drive I/Os in addition to disk drive I/Os to control I/O contention between databases, pluggable databases, and consumer groups. Since it is very rare for Oracle Exadata environments to be limited by OLTP I/Os, IORM automatically prioritizes OLTP flash I/Os over smart scan flash I/Os, ensuring fast OLTP response times with little cost to smart scan throughput.
.
Flash Cache Space Resource Management
.
Flash Cache Space Resource Management allows users to specify the minimum and maximum sizes a database can use in the flash cache using interdatabase IORM plans. Flash Cache Space Resource Management also allows users to specify the minimum and maximum sizes a pluggable database can use in the flash cache using database resource plans.
.
I/O Resource Management Profiles
.
IORM interdatabase plans now support profiles that reduce management of interdatabase plans for environments with many databases.
.
Write Back Flash Cache on Extreme Flash Cells
.
On Extreme Flash cells, flash cache runs in write back mode by default, and takes 5 percent of the flash space. It is useful for columnar caching, write I/O latency capping and fast data file creation.
Secure Erase for 1.6 TB Flash Drives in Extreme Flash and High Capacity Systems
Oracle Exadata Storage Server Software supports secure erase for 1.6 TB flash drives in the Extreme Flash and High Capacity systems. The 1.6 TB flash drives take approximately 5.5 hours to securely erase.
.
Increased Exadata Cell Connection Limit
.
Oracle Exadata X5-2 and X4-2 cells can now support up to 120,000 simultaneous connections originating from one or more database servers that is using active-active bonding. This implies that at most 120,000 processes can simultaneously remain connected to a cell and perform I/O operations.
.
Support for SNMP v3
.
Oracle Exadata Database Machine database and storage servers support SNMP v3 for sending alerts. SNMP v3 provides authentication and encryption for alerts sent from the servers to administrators and Auto Service Request.
.
Federal Information Processing Standards (FIPS) 140-2 Compliant Smart Scan
.
The U.S. Federal Information Processing Standard (FIPS) 140-2 specifies security requirements for cryptographic modules. To support customers with FIPS 140-2 requirements, Oracle Exadata version 12.1.2.1.0 can be configured to use FIPS 140-2 validated cryptographic modules. These modules provide cryptographic services such as Oracle Database password hashing and verification, network encryption (SSL/TLS and native encryption), as well as data at rest encryption (Transparent Data Encryption).
.
Oracle Exadata Virtual Machines
.
Consolidated environments can now use Oracle Virtual Machine (OVM) on X5-2, X4-2, X3-2, and X2-2 database servers to deliver higher levels of isolation between workloads. Virtual machine isolation is desirable for workloads that cannot be trusted to restrict their security, cpu, or memory usage in a shared environment.
.
Notable firmware updates
.
Aura 2.0 flash package update to 09.05.42.00 in X4270-M3 (X3-2L) storage servers.
Aura 2.1 flash package update to 09.05.43.00 in X4-2L storage servers.
ILOM update to 3.2.4.18 r93492 for X4-8 (X4-8) database servers
ILOM update to 3.0.16.20.g r93325 for X4800-M2 (X3-8) database servers.
ILOM update to 3.0.16.16.e r93323 for X4800 (X2-8) database servers.
ILOM update to 3.2.4.20.a r94217 for X4-2 (X4-2) database servers.
ILOM update to 3.2.4.22.a r94470 for X4-2L (X4-2L) storage servers.
ILOM update to 3.1.2.42.a r93409 for X4170-M3 (X3-2) database servers.
ILOM update to 3.1.2.44.a r93411 for X4270-M3 (X3-2L) storage servers.
ILOM update to 3.1.2.20.e r93424 for X4170-M2 (X2) database servers and X4270-M2 (X2) storage servers.
ILOM update to 3.0.16.15.h r93405 for X4170 (V2) database servers and X4275 (V2) storage servers.
Software Release Requirements
.

Exadata Storage Server Software 12.1.2.1.0 supports the following Oracle Database software releases:
.

Oracle Database 12c Release 1 (12.1.0.2.0)
Oracle Database 12c Release 1 (12.1.0.1.0)
Oracle Database 11g Release 2 (11.2.0.4.0)
Oracle Database 11g Release 2 (11.2.0.3.0) BP 20 or Newer BP
Oracle Enterprise Manager requires Exadata Plug-in version 12.1.0.5.0 or later to manage Exadata Storage Server Software 12.1.1.1.0.

Oracle Enterprise Manager requires Exadata Plug-in version 12.1.0.6.0 or later to manage Exadata Storage Server Software 12.1.2.1.0.

.

发表在 性能指标 | 标签为 | 留下评论

oracle一些块损坏和常见数据库损坏的相关概念和处理

最近帮朋友做了一个公开课(大概2小时吧),大概介绍了一下oracle一些块损坏和常见数据库损坏的相关概念和处理。
这里谈到的东西很少,很多内容细讲都是一门学问,我这里介绍的只是大概的概念,冰山一角。对于oracle大牛们来说,不算是什么,O(∩_∩)O哈哈~
下周打算给公司的同事们介绍一下。
目前pdf可以下载了Oracle常见错误处理-lunar
后续朋友整理好录音等东西,也会上传到这里,与你共勉,请多指教 :)。

本次公开交流的内容已经放到优酷了:

发表在 backup&recovery, ORA-600/7445 | 标签为 , | 留下评论

ORA-00704 ORA-00604 ORA-01502: index ‘SYS.I_ACCESS1’ or partition of such index is in unusable state

如何处理下面的错误?
ORA-00704 ORA-00604 ORA-01502: index ‘SYS.I_ACCESS1’ or partition of such index is in unusable state

方法跟ORA-00704 ORA-00604 ORA-01502 SYS.I_DEPENDENCY1损坏或者不可用的是一样的,这里不赘述,简单模拟损坏并修复的过程如下:
I_DEPENDENCY2损坏的处理

11.2中I_DEPENDENCY1损坏的处理

例如:

Mon Dec 22 11:38:44 2014
SMON: enabling cache recovery
Mon Dec 22 11:38:45 2014
Deleted Oracle managed file /home/oracle/oracle/product/flash_recovery_area/ORCL/archivelog/2009_11_21/o1_mf_1_48_5jgg8461_.arc
Mon Dec 22 11:38:45 2014
Errors in file /home/oracle/oracle/product/admin/orcl/udump/orcl_ora_27945.trc:
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_ACCESS1' or partition of such index is in unusable state
Mon Dec 22 11:38:45 2014
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 27945
ORA-1092 signalled during: ALTER DATABASE OPEN...

相应的trace如下:

PARSING IN CURSOR #5 len=56 dep=1 uid=0 oct=3 lid=0 tim=1385956621228493 hv=3993603298 ad='29b047b0'
select order#,columns,types from access$ where d_obj#=:1
END OF STMT
PARSE #5:c=2000,e=1891,p=1,cr=24,cu=0,mis=1,r=0,dep=1,og=4,tim=1385956621228490
=====================
PARSING IN CURSOR #2 len=84 dep=2 uid=0 oct=3 lid=0 tim=1385956621229181 hv=2686874206 ad='29b03e00'
select o.name, u.name from obj$ o, user$ u  where o.obj# = :1 and o.owner# = u.user#
END OF STMT
PARSE #2:c=1000,e=531,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,tim=1385956621229176
BINDS #2:
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=b72384c8  bln=22  avl=03  flg=05
  value=124
EXEC #2:c=0,e=710,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,tim=1385956621229957
WAIT #2: nam='db file sequential read' ela= 11 file#=1 block#=98 blocks=1 obj#=-1 tim=1385956621230031
WAIT #2: nam='db file sequential read' ela= 13 file#=1 block#=90 blocks=1 obj#=-1 tim=1385956621230124
FETCH #2:c=1000,e=182,p=2,cr=5,cu=0,mis=0,r=1,dep=2,og=4,tim=1385956621230156
STAT #2 id=1 cnt=1 pid=0 pos=1 obj=0 op='NESTED LOOPS  (cr=5 pr=2 pw=0 time=187 us)'
STAT #2 id=2 cnt=1 pid=1 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=0 pw=0 time=28 us)'
STAT #2 id=3 cnt=1 pid=2 pos=1 obj=36 op='INDEX UNIQUE SCAN I_OBJ1 (cr=2 pr=0 pw=0 time=16 us)'
STAT #2 id=4 cnt=1 pid=1 pos=2 obj=22 op='TABLE ACCESS CLUSTER USER$ (cr=2 pr=2 pw=0 time=146 us)'
STAT #2 id=5 cnt=1 pid=4 pos=1 obj=11 op='INDEX UNIQUE SCAN I_USER# (cr=1 pr=1 pw=0 time=96 us)'
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_ACCESS1' or partition of such index is in unusable state
EXEC #1:c=144978,e=1804415,p=119,cr=759,cu=0,mis=0,r=0,dep=0,og=1,tim=1385956622211453
ERROR #1:err=1092 tim=433705993

解决问题:

SYS@lunar>alter database open upgrade;

Database altered.

SYS@lunar>select obj# from obj$ where name='I_ACCESS1';

      OBJ#
----------
       124

SYS@lunar>update ind$ set flags=1024 where obj#=124;

1 row updated.

SYS@lunar>commit;

Commit complete.

SYS@lunar>

然后将数据库启动到限制模式,rebuild该index:

SYS@lunar>alter index I_ACCESS1 rebuild;

Index altered.

SYS@lunar>
发表在 backup&recovery, ORA-600 or ORA-7445 | 标签为 , | 留下评论

I_DEPENDENCY2损坏的处理

我们知道,在10.2中如果SYS.I_DEPENDENCY1损坏或者不可用时,数据库open时就会报如下类似错误:
10g中I_DEPENDENCY1损坏的处理

11.2中I_DEPENDENCY1损坏的处理

Mon Dec 22 09:13:32 2014
Errors in file /home/oracle/oracle/product/admin/orcl/udump/orcl_ora_19783_lunar.trc:
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_DEPENDENCY1' or partition of such index is in unusable state

那么如果I_DEPENDENCY2有问题时,什么结果呢?
我们还是在10.2.0.1中测试:

[oracle@lunar oracle]$ ss

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 22 11:13:21 2014

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to an idle instance.

SYS@lunar>startup
ORACLE instance started.

Total System Global Area  167772160 bytes
Fixed Size                  1218316 bytes
Variable Size              71305460 bytes
Database Buffers           92274688 bytes
Redo Buffers                2973696 bytes
Database mounted.
Database opened.
SYS@lunar>
SYS@lunar>alter index i_dependency2 unusable;

Index altered.

SYS@lunar>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER           TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
--------------- ------------------------------ ------------------------------ ------------------------------ --------
SYS             DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         VALID
SYS             DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         UNUSABLE

SYS@lunar>

测试结果,数据库可以正常open,但是alert后报错,显示如下:

Mon Dec 22 11:12:17 2014
Successfully onlined Undo Tablespace 1.
Mon Dec 22 11:12:17 2014
SMON: enabling tx recovery
Mon Dec 22 11:12:17 2014
Errors in file /home/oracle/oracle/product/admin/orcl/bdump/orcl_smon_19955.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_DEPENDENCY2' or partition of such index is in unusable state
Mon Dec 22 11:12:17 2014
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
WARNING: AQ_TM_PROCESSES is set to 0. System operation might be adversely affected.
Mon Dec 22 11:12:20 2014
db_recovery_file_dest_size of 2048 MB is 43.37% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Mon Dec 22 11:12:21 2014
Starting background process CJQ0
CJQ0 started with pid=20, OS id=19979
Mon Dec 22 11:12:21 2014
Completed: alter database open

修复的方法很简单,直接将该索引的状态设置为失效,然后rebuild,具体操作,参考ORA-00704 ORA-00604 ORA-01502 SYS.I_DEPENDENCY1损坏或者不可用

SYS@lunar>alter index i_dependency2 rebuild
  2  ;

Index altered.

SYS@lunar>analyze table dependency$ validate structure cascade;

Table analyzed.

SYS@lunar>select obj#,flags from ind$ where obj#=123;

      OBJ#      FLAGS
---------- ----------
       123          2

SYS@lunar>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER                          TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
------------------------------ ------------------------------ ------------------------------ ------------------------------ --------
SYS                            DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         VALID
SYS                            DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID

SYS@lunar>
发表在 backup&recovery, ORA-600 or ORA-7445 | 标签为 | 留下评论

11.2中I_DEPENDENCY1损坏的处理

上一篇,我们测试了10.2中,i_dependency1失效的过程和处理方法。
参考10g中I_DEPENDENCY1损坏的处理

I_DEPENDENCY2损坏的处理

这里,我们在11.2中模拟i_dependency1失效(测试环境使用11.2.0.3):

11.2中:先看一下,数据库中跟依赖关系相关的表都有哪些:

SQL> select owner,object_id,object_name,object_type from dba_objects where object_name like '%DEPENDENCY%';


OWNER            OBJECT_ID OBJECT_NAME                    OBJECT_TYPE
--------------- ---------- ------------------------------ -------------------
SYS                    104 DEPENDENCY$                    TABLE
SYS                    106 I_DEPENDENCY1                  INDEX
SYS                    107 I_DEPENDENCY2                  INDEX
SYS                   1511 V_$OBJECT_DEPENDENCY           VIEW
SYS                   2200 GV_$OBJECT_DEPENDENCY          VIEW
SYS                   2837 GV_$SQL_FEATURE_DEPENDENCY     VIEW
SYS                   2839 V_$SQL_FEATURE_DEPENDENCY      VIEW
SYS                   2857 GV_$RESULT_CACHE_DEPENDENCY    VIEW
SYS                   2859 V_$RESULT_CACHE_DEPENDENCY     VIEW
SYS                   4841 PUBLIC_DEPENDENCY              VIEW
PUBLIC                1512 V$OBJECT_DEPENDENCY            SYNONYM
PUBLIC                2201 GV$OBJECT_DEPENDENCY           SYNONYM
PUBLIC                2838 GV$SQL_FEATURE_DEPENDENCY      SYNONYM
PUBLIC                2840 V$SQL_FEATURE_DEPENDENCY       SYNONYM
PUBLIC                2858 GV$RESULT_CACHE_DEPENDENCY     SYNONYM
PUBLIC                2860 V$RESULT_CACHE_DEPENDENCY      SYNONYM
PUBLIC                4842 PUBLIC_DEPENDENCY              SYNONYM
SYSMAN               14862 MGMT_INV_DEPENDENCY_RULE       TABLE
SYSMAN               15200 MGMT_METRIC_DEPENDENCY_DEF     TABLE
SYSMAN               15201 PK_MGMT_METRIC_DEPENDENCY_DEF  INDEX
SYSMAN               15202 MGMT_METRIC_DEPENDENCY         TABLE
SYSMAN               15203 PK_MGMT_METRIC_DEPENDENCY      INDEX
SYSMAN               15204 MGMT_METRIC_DEPENDENCY_DETAILS TABLE
SYSMAN               15851 MGMT_METRIC_DEPENDENCY_IDX_01  INDEX

24 rows selected.

SQL> 

可以看到,10g是18个,11.2是24个。

SQL> select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER           TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
--------------- ------------------------------ ------------------------------ ------------------------------ --------
SYS             DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID
SYS             DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         VALID

SQL> 

10g和11g中,DEPENDENCY$表都存在两个索引,这两个索引损坏时对数据库的影响是不同的,他们分别定义如下:

create table dependency$                                 /* dependency table */
( d_obj#        number not null,                  /* dependent object number */
  d_timestamp   date not null,   /* dependent object specification timestamp */
  order#        number not null,                             /* order number */
  p_obj#        number not null,                     /* parent object number */
  p_timestamp   date not null,      /* parent object specification timestamp */
  d_owner#      number,                           /*  dependent owner number */
  property      number not null,                   /* 0x01 = HARD dependency */
                                                   /* 0x02 = REF  dependency */
                                          /* 0x04 = FINER GRAINED dependency */
  d_attrs       raw("M_CSIZ"), /* Finer grain attr. numbers if finer grained */
  d_reason      raw("M_CSIZ"))  /* Reason mask of attrs causing invalidation */
storage (initial 10k next 100k maxextents unlimited pctincrease 0)
/

create unique index i_dependency1 on
  dependency$(d_obj#, d_timestamp, order#)
  storage (initial 10k next 100k maxextents unlimited pctincrease 0)
/

create index i_dependency2 on
  dependency$(p_obj#, p_timestamp)
  storage (initial 10k next 100k maxextents unlimited pctincrease 0)
/

他们存储在哪些block中:

SQL> col segment_name for a30
SQL> select owner,segment_name,segment_type,extent_id,file_id,block_id from DBA_EXTENTS where segment_name like '%DEPENDENCY%';

OWNER           SEGMENT_NAME                   SEGMENT_TYPE        EXTENT_ID    FILE_ID   BLOCK_ID
--------------- ------------------------------ ------------------ ---------- ---------- ----------
SYS             I_DEPENDENCY2                  INDEX                       0          1        864
SYS             I_DEPENDENCY2                  INDEX                       1          1       8584
SYS             I_DEPENDENCY2                  INDEX                       2          1       8856
SYS             I_DEPENDENCY2                  INDEX                       3          1       9000
SYS             I_DEPENDENCY2                  INDEX                       4          1      10072
SYS             I_DEPENDENCY2                  INDEX                       5          1      12792
SYS             I_DEPENDENCY2                  INDEX                       6          1      16128
SYS             I_DEPENDENCY2                  INDEX                       7          1      18752
SYS             I_DEPENDENCY2                  INDEX                       8          1      19960
SYS             I_DEPENDENCY2                  INDEX                       9          1      21328
SYS             I_DEPENDENCY2                  INDEX                      10          1      21584
SYS             I_DEPENDENCY2                  INDEX                      11          1      22288
SYS             I_DEPENDENCY2                  INDEX                      12          1      22888
SYS             I_DEPENDENCY2                  INDEX                      13          1      23408
SYS             I_DEPENDENCY2                  INDEX                      14          1      25616
SYS             I_DEPENDENCY2                  INDEX                      15          1      25672
SYS             I_DEPENDENCY2                  INDEX                      16          1      28672
SYS             I_DEPENDENCY1                  INDEX                       0          1        856
SYS             I_DEPENDENCY1                  INDEX                       1          1       8720
SYS             I_DEPENDENCY1                  INDEX                       2          1       8984
SYS             I_DEPENDENCY1                  INDEX                       3          1      10712
SYS             I_DEPENDENCY1                  INDEX                       4          1      14968
SYS             I_DEPENDENCY1                  INDEX                       5          1      19248
SYS             I_DEPENDENCY1                  INDEX                       6          1      20760
SYS             I_DEPENDENCY1                  INDEX                       7          1      21312
SYS             I_DEPENDENCY1                  INDEX                       8          1      21608
SYS             I_DEPENDENCY1                  INDEX                       9          1      22344
SYS             I_DEPENDENCY1                  INDEX                      10          1      23368
SYS             I_DEPENDENCY1                  INDEX                      11          1      25608
SYS             I_DEPENDENCY1                  INDEX                      12          1      25688
SYS             I_DEPENDENCY1                  INDEX                      13          1      29192
SYS             I_DEPENDENCY1                  INDEX                      14          1      32760
SYS             I_DEPENDENCY1                  INDEX                      15          1      34240
SYS             I_DEPENDENCY1                  INDEX                      16          1      37504
SYS             DEPENDENCY$                    TABLE                       0          1        840
SYS             DEPENDENCY$                    TABLE                       1          1       8624
SYS             DEPENDENCY$                    TABLE                       2          1       8800
SYS             DEPENDENCY$                    TABLE                       3          1       8968
SYS             DEPENDENCY$                    TABLE                       4          1       9640
SYS             DEPENDENCY$                    TABLE                       5          1      10080
SYS             DEPENDENCY$                    TABLE                       6          1      12992
SYS             DEPENDENCY$                    TABLE                       7          1      15544
SYS             DEPENDENCY$                    TABLE                       8          1      18728
SYS             DEPENDENCY$                    TABLE                       9          1      19848
SYS             DEPENDENCY$                    TABLE                      10          1      20768
SYS             DEPENDENCY$                    TABLE                      11          1      21296
SYS             DEPENDENCY$                    TABLE                      12          1      21576
SYS             DEPENDENCY$                    TABLE                      13          1      21832
SYS             DEPENDENCY$                    TABLE                      14          1      22808
SYS             DEPENDENCY$                    TABLE                      15          1      23344
SYS             DEPENDENCY$                    TABLE                      16          1      24704
SYS             DEPENDENCY$                    TABLE                      17          1      45440
SYSMAN          PK_MGMT_METRIC_DEPENDENCY_DEF  INDEX                       0          2      14608
SYSMAN          MGMT_METRIC_DEPENDENCY_DEF     TABLE                       0          2      14600

54 rows selected.

SQL>

10g和11g的DEPENDENCY$ 和 I_DEPENDENCY1、I_DEPENDENCY2定义都一致的,只是处理方法不同。
现在我们在11.2.0.3模拟i_dependency1失效,看看什么效果:


SYS@lunarp>alter index i_dependency1 unusable;

Index altered.

Elapsed: 00:00:00.42
SYS@lunarp>commit;

Commit complete.

Elapsed: 00:00:00.00
SYS@lunarp>

相应的alert.log信息如下:

Wed Mar 05 01:50:18 2014
Index SYS.I_DEPENDENCY1 or some [sub]partitions of the index have been marked unusable

说明已经标示 SYS.I_DEPENDENCY1 为无效了。

现在启动数据库看看:

SYS@lunarp>startup mount
Welcome Lunar's oracle world!

Love you , baby !

ORACLE instance started.

Total System Global Area  626327552 bytes
Fixed Size                  2230952 bytes
Variable Size             243271000 bytes
Database Buffers          373293056 bytes
Redo Buffers                7532544 bytes
Database mounted.
Welcome Lunar's oracle world!

Love you , baby !

SYS@lunarp>oradebug setmypid
Statement processed.
SYS@lunarp>alter session set db_file_multiblocK_read_count=1;

Session altered.

Elapsed: 00:00:00.00
SYS@lunarp>alter session set tracefile_identifier='lunar';

Session altered.

Elapsed: 00:00:00.01
SYS@lunarp>oradebug event 10046 trace name context forever,level 12;
Statement processed.
SYS@lunarp>oradebug tracefile_name
/u01/app/oracle/diag/rdbms/lunarp/lunarp/trace/lunarp_ora_8257_lunar.trc
SYS@lunarp>alter database open;   ----------注意这里,直接open

Database altered.

Elapsed: 00:00:08.91
SYS@lunarp>oradebug event 10046 trace name context off
Statement processed.
SYS@lunarp>oradebug close_trace
Statement processed.
SYS@lunarp>

没错,就是这个结果,从11.2开始,类似I_DEPENDENCY1这样的index的失效不会数据库不能open了。
而在10.2中,需要open upgrade模式。
现在修复这个索引:

SYS@lunarp>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER                          TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
------------------------------ ------------------------------ ------------------------------ ------------------------------ --------
SYS                            DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID
SYS                            DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         UNUSABLE

Elapsed: 00:00:00.16
SYS@lunarp>alter index sys.I_DEPENDENCY1 rebuild online;

Index altered.

Elapsed: 00:00:02.76
SYS@lunarp>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER                          TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
------------------------------ ------------------------------ ------------------------------ ------------------------------ --------
SYS                            DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID
SYS                            DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         VALID

Elapsed: 00:00:00.00
SYS@lunarp>

注意 :11.2跟10.2还有一个小区别(感觉无关紧要):11.2多取了一个字段“type#”:

select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,type#,d_attrs 
from dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#

那么11.2为什么可以直接open呢?
首先,我们来看一下,当 I_DEPENDENCY1 正常时,数据库open的相关动作:

PARSING IN CURSOR #140636885644848 len=185 dep=1 uid=0 oct=3 lid=0 tim=1394010624511477 hv=1850944673 ad='850425a8' sqlid='3ktacv9r56b51'
select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,type#,d_attrs from dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#
END OF STMT
PARSE #140636885644848:c=5999,e=20725,p=2,cr=41,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1394010624511472
BINDS #140636885644848:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7fe8939c71e8  bln=22  avl=03  flg=05
  value=426
EXEC #140636885644848:c=1000,e=1100,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=4184428695,tim=1394010624512679
WAIT #140636885644848: nam='db file sequential read' ela= 4373 file#=1 block#=857 blocks=1 obj#=106 tim=1394010624517146
WAIT #140636885644848: nam='db file sequential read' ela= 299 file#=1 block#=858 blocks=1 obj#=106 tim=1394010624517612
FETCH #140636885644848:c=1000,e=4978,p=2,cr=2,cu=0,mis=0,r=0,dep=1,og=4,plh=4184428695,tim=1394010624517698
STAT #140636885644848 id=1 cnt=0 pid=0 pos=1 obj=0 op='SORT ORDER BY (cr=2 pr=2 pw=0 time=5006 us cost=0 size=0 card=0)'
STAT #140636885644848 id=2 cnt=0 pid=1 pos=1 obj=0 op='NESTED LOOPS OUTER (cr=2 pr=2 pw=0 time=4974 us)'
STAT #140636885644848 id=3 cnt=0 pid=2 pos=1 obj=104 op='TABLE ACCESS BY INDEX ROWID DEPENDENCY$ (cr=2 pr=2 pw=0 time=4956 us)'
STAT #140636885644848 id=4 cnt=0 pid=3 pos=1 obj=106 op='INDEX RANGE SCAN I_DEPENDENCY1 (cr=2 pr=2 pw=0 time=4936 us)'
STAT #140636885644848 id=5 cnt=0 pid=2 pos=2 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=0 pr=0 pw=0 time=0 us)'
STAT #140636885644848 id=6 cnt=0 pid=5 pos=1 obj=36 op='INDEX RANGE SCAN I_OBJ1 (cr=0 pr=0 pw=0 time=0 us)'
CLOSE #140636885644848:c=0,e=90,dep=1,type=0,tim=1394010624517803
BINDS #140636885393152:

该语句执行7次,七次的执行计划都是上面这样的。

当I_DEPENDENCY1为UNUSABLE时,该语句仍然执行7次,但是7次的执行计划都类似下面的,走“TABLE ACCESS FULL DEPENDENCY$”。
这说明oracle 11.2的二进制代码多了一个判断的功能,当发现I_DEPENDENCY1为UNUSABLE时自动修改了执行计划(没有验证其他类似索引是否都是这样的):

select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, 
  nvl(property,0),subname,type#,d_attrs 
from
 dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse       33      0.00       0.02          2         41          0           0
Execute     33      0.00       0.00          0          0          0           0
Fetch      127      0.11       0.55        285       8764          0          94
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total      193      0.12       0.58        287       8805          0          94

Misses in library cache during parse: 2
Misses in library cache during execute: 2
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)
Number of plan statistics captured: 7

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
         0          2          6  SORT ORDER BY (cr=263 pr=37 pw=0 time=42164 us cost=0 size=0 card=0)
         0          2          6   NESTED LOOPS OUTER (cr=263 pr=37 pw=0 time=41858 us)
         0          2          6    TABLE ACCESS FULL DEPENDENCY$ (cr=259 pr=37 pw=0 time=38319 us)
         0          2          6    TABLE ACCESS BY INDEX ROWID OBJ$ (cr=4 pr=1 pw=0 time=3497 us)
         0          2          6     INDEX RANGE SCAN I_OBJ1 (cr=3 pr=0 pw=0 time=995 us)(object id 36)

推测:11.2的代码里面写死了一个判断当open数据库时,如果遇到类似DEPENDENCY$这样的表的索引失效了,那么直接对DEPENDENCY$进行全表扫描。

发表在 backup&recovery, ORA-600 or ORA-7445 | 标签为 , | 留下评论

一些好玩的现象(表区分大小写,同一个owner下有同名的表名存在等等)

一些好玩的现象:
1、谁说同一个用户下面不能有两个表是同名的,看看下面,O(∩_∩)O哈哈~:

[oracle@lunar ~]$ ss

SQL*Plus: Release 11.2.0.3.0 Production on Sat Jan 3 14:17:47 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Welcome Lunar's oracle world!

Love you , baby !


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning and Real Application Testing options

ERROR:
ORA-22303: type "SYS"."DBMSOUTPUT_LINESARRAY" not found
ORA-00942: table or view does not exist
ORA-06508: PL/SQL: could not find program unit being called: "SYS.DBMS_OUTPUT"
ORA-06512: at line 1



Session altered.

Elapsed: 00:00:00.02
SYS@lunarp>select owner,segment_name,HEADER_FILE,HEADER_BLOCK from dba_segments where SEGMENT_NAME='LUNAR_BOOTSTRAP';

OWNER                          SEGMENT_NAME                                       HEADER_FILE HEADER_BLOCK
------------------------------ -------------------------------------------------- ----------- ------------
SYS                            LUNAR_BOOTSTRAP                                             1          520
SYS                            LUNAR_BOOTSTRAP                                             4          434

Elapsed: 00:00:00.20
SYS@lunarp>

2、再看下面,我的库没有引导表了,O(∩_∩)O哈哈~:

SYS@lunarp>select object_id,object_name from dba_objects where object_name='%BOOTSTRAP%';

no rows selected

Elapsed: 00:00:00.15
SYS@lunarp>select owner,segment_name,HEADER_FILE,HEADER_BLOCK from dba_segments where SEGMENT_NAME='%BOOTSTRAP%';

no rows selected

Elapsed: 00:00:00.15
SYS@lunarp>select owner,segment_name from dba_segments where SEGMENT_NAME like '%BOOTSTRAP%';

OWNER                          SEGMENT_NAME
------------------------------ ---------------------------------------------------------------------------------
SYSMAN                         MGMT_RT_BOOTSTRAP_TIMES
SYSMAN                         MGMT_RT_BOOTSTRAP_TIMES_PK

Elapsed: 00:00:00.21
SYS@lunarp>

3、看这里,表名区分大小写了,你知道为什么么?——— O(∩_∩)O哈哈~:

SYS@lunarp>select dbms_rowid.rowid_relative_fno(rowid) file#,
  2        dbms_rowid.rowid_block_number(rowid) blk#
  3   from BOOTSTRAP$;
 from BOOTSTRAP$
      *
ERROR at line 3:
ORA-00942: table or view does not exist


Elapsed: 00:00:00.00
SYS@lunarp>select dbms_rowid.rowid_relative_fno(rowid) file#,
  2        dbms_rowid.rowid_block_number(rowid) blk#
  3   from bootstrap$;

     FILE#       BLK#
---------- ----------
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        521
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        522
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523
         1        523

60 rows selected.

Elapsed: 00:00:00.13
SYS@lunarp>

4、你知道这是为什么么?————很好猜,O(∩_∩)O哈哈~

SYS@lunarp>select owner,segment_name,HEADER_FILE,HEADER_BLOCK from dba_segments where SEGMENT_NAME='LUNAR_BOOTSTRAP';

OWNER                          SEGMENT_NAME                                       HEADER_FILE HEADER_BLOCK
------------------------------ -------------------------------------------------- ----------- ------------
SYS                            LUNAR_BOOTSTRAP                                             1          520
SYS                            LUNAR_BOOTSTRAP                                             4          434

Elapsed: 00:00:00.15
SYS@lunarp>select obj#,name from obj$ where name like '%BOOTSTRAP%';

      OBJ# NAME
---------- ------------------------------
     16077 IP_BOOTSTRAP_PRIMARY_KEY
     16078 IP_DIST_BOOTSTRAP_PRIMARY_KEY
     15554 MGMT_RT_BOOTSTRAP_TIMES
     16072 MGMT_RT_BOOTSTRAP_TIMES_PK
     15563 MGMT_RT_DOMAIN_BOOTSTRAP
     15566 MGMT_RT_DOMAIN_DIST_BOOTSTRAP
     16075 DOMAIN_BOOTSTRAP_PRIMARY_KEY
     15569 MGMT_RT_IP_BOOTSTRAP
     15572 MGMT_RT_IP_DIST_BOOTSTRAP
     15557 MGMT_RT_URL_BOOTSTRAP
     15560 MGMT_RT_URL_DIST_BOOTSTRAP
     16073 URL_BOOTSTRAP_PRIMARY_KEY
     16074 URL_DIST_BOOTSTRAP_PRIMARY_KEY

13 rows selected.

Elapsed: 00:00:00.01
SYS@lunarp>

5、被删除的表名被占用了,不能重用,这是为什么?
这是整个玩耍过程中,最郁闷的问题了………………………………

SYS@lunarp>drop table lunar_bootstrap purge ;

Table dropped.

Elapsed: 00:00:01.19
SYS@lunarp>create table lunar_bootstrap tablespace system as select * from bootstrap$ ;
create table lunar_bootstrap tablespace system as select * from bootstrap$
             *
ERROR at line 1:
ORA-00955: name is already used by an existing object


Elapsed: 00:00:00.06
SYS@lunarp>col segment_name for a30
SYS@lunarp>select owner,segment_name,segment_type,extent_id,file_id,block_id from DBA_EXTENTS where segment_name like 'BOOTSTRAP$';

no rows selected

Elapsed: 00:00:00.34
SYS@lunarp>select owner,segment_name,HEADER_FILE,HEADER_BLOCK from dba_segments where SEGMENT_NAME='%BOOTSTRAP%';

no rows selected

Elapsed: 00:00:00.09
SYS@lunarp>select owner,segment_name from dba_segments where SEGMENT_NAME like '%BOOTSTRAP%';

OWNER                          SEGMENT_NAME
------------------------------ ------------------------------
SYS                            LUNAR_BOOTSTRAP
SYSMAN                         MGMT_RT_BOOTSTRAP_TIMES
SYSMAN                         MGMT_RT_BOOTSTRAP_TIMES_PK

Elapsed: 00:00:00.13
SYS@lunarp>select object_id,object_name from dba_objects where object_name='%BOOTSTRAP%';

no rows selected

Elapsed: 00:00:00.14
SYS@lunarp>

好了,不玩了,还原回来了,O(∩_∩)O哈哈~:

SYS@lunarp>select owner,segment_name,HEADER_FILE,HEADER_BLOCK from dba_segments where SEGMENT_NAME='BOOTSTRAP$';

OWNER                          SEGMENT_NAME                                                                      HEADER_FILE HEADER_BLOCK
------------------------------ --------------------------------------------------------------------------------- ----------- ------------
SYS                            BOOTSTRAP$                                                                                  1          520

Elapsed: 00:00:00.15
SYS@lunarp>select owner,segment_name,HEADER_FILE,HEADER_BLOCK from dba_segments where SEGMENT_NAME='LUNAR_BOOTSTRAP';

OWNER                          SEGMENT_NAME                                                                      HEADER_FILE HEADER_BLOCK
------------------------------ --------------------------------------------------------------------------------- ----------- ------------
SYS                            LUNAR_BOOTSTRAP                                                                            4          434

Elapsed: 00:00:00.02
SYS@lunarp>
发表在 未分类 | 留下评论

ORA-00704 ORA-00604 ORA-01502 SYS.I_DEPENDENCY1损坏或者不可用

有朋友问起ORA-00704 ORA-00604 ORA-01502: index ‘SYS.I_DEPENDENCY1’ or partition of such index is in unusable state错误怎么处理。

11.2中I_DEPENDENCY1损坏的处理
I_DEPENDENCY2损坏的处理

当SYS.I_DEPENDENCY1损坏或者不可用时,就会报如下类似错误:

Mon Dec 22 09:13:32 2014
Errors in file /home/oracle/oracle/product/admin/orcl/udump/orcl_ora_19783_lunar.trc:
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_DEPENDENCY1' or partition of such index is in unusable state

首先说一下,10g中的处理方式,有两种:
1,直接在10g中修复
2,使用11.2将数据库拉起来到upgrade模式,然后修复

先看一下,数据库中跟依赖关系相关的表都有哪些:

[oracle@lunar oracle]$ ss

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 22 08:10:07 2014

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

SYS@lunar>select owner,object_id,object_name,object_type from dba_objects where object_name like '%DEPENDENCY%' and owner='SYS';

OWNER            OBJECT_ID OBJECT_NAME                    OBJECT_TYPE
--------------- ---------- ------------------------------ -------------------
SYS                     92 DEPENDENCY$                    TABLE
SYS                  46827 GENDEPENDENCYBLOCKSEQUENCE     TYPE
SYS                  46828 GENDEPENDENCYBLOCKSEQUENCESEQU TYPE
SYS                  46824 GENDEPENDENCYBLOCKSTRUCT       TYPE
SYS                  46826 GENDEPENDENCYBLOCKUNION        TYPE
SYS                  46825 GENINCLUDEDDEPENDENCYBLOCKSTRU TYPE
SYS                   1691 GV_$OBJECT_DEPENDENCY          VIEW
SYS                    122 I_DEPENDENCY1                  INDEX
SYS                    123 I_DEPENDENCY2                  INDEX
SYS                   3950 PUBLIC_DEPENDENCY              VIEW
SYS                  47048 SQLDEPENDENCYBLOCKSEQ          TYPE
SYS                  47049 SQLDEPENDENCYBLOCKSEQUENCE     TYPE
SYS                  47050 SQLDEPENDENCYBLOCKSEQUENCESEQ  TYPE
SYS                  47051 SQLDEPENDENCYBLOCKSEQUENCESEQU TYPE
SYS                  47045 SQLDEPENDENCYBLOCKSTRUCT       TYPE
SYS                  47047 SQLDEPENDENCYBLOCKUNION        TYPE
SYS                  47046 SQLINCLUDEDDEPENDENCYBLOCKSTRU TYPE
SYS                   1066 V_$OBJECT_DEPENDENCY           VIEW

18 rows selected.

SYS@lunar>

这里,我们关心的是DEPENDENCY$的索引信息,可以看到,DEPENDENCY$表存在两个索引。
这两个索引损坏时对数据库的影响是不同的,他们分别定义如下:


SYS@lunar>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER           TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
--------------- ------------------------------ ------------------------------ ------------------------------ --------
SYS             DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         VALID
SYS             DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID

SYS@lunar>

9i、10g、11.1以前,一些核心对象的定义就在$ORACLE_HOME/rdbms/admin/sql.bsq中。
比如tab$,obj$,ind$等等的定义都在其中。
从11.2开始,创建一些重要的核心基表的脚本仍然是sql.bsq,但是已经将各种对象的创建分类到不同的脚本:

dcore.bsq
dsqlddl.bsq
dmanage.bsq
dplsql.bsq
dtxnspc.bsq
dfmap.bsq
denv.bsq
drac.bsq
dsec.bsq
doptim.bsq
dobj.bsq
djava.bsq
dpart.bsq
drep.bsq
daw.bsq
dsummgt.bsq
dtools.bsq
dexttab.bsq
ddm.bsq
dlmnr.bsq
ddst.bsq

dependency$,i_dependency1和i_dependency1的定义如下(10.2和11.2没有区别):

create table dependency$                                 /* dependency table */
( d_obj#        number not null,                  /* dependent object number */
  d_timestamp   date not null,   /* dependent object specification timestamp */
  order#        number not null,                             /* order number */
  p_obj#        number not null,                     /* parent object number */
  p_timestamp   date not null,      /* parent object specification timestamp */
  d_owner#      number,                           /*  dependent owner number */
  property      number not null,                   /* 0x01 = HARD dependency */
                                                   /* 0x02 = REF  dependency */
                                          /* 0x04 = FINER GRAINED dependency */
  d_attrs       raw("M_CSIZ"), /* Finer grain attr. numbers if finer grained */
  d_reason      raw("M_CSIZ"))  /* Reason mask of attrs causing invalidation */
storage (initial 10k next 100k maxextents unlimited pctincrease 0)
/

create unique index i_dependency1 on
  dependency$(d_obj#, d_timestamp, order#)
  storage (initial 10k next 100k maxextents unlimited pctincrease 0)
/

create index i_dependency2 on
  dependency$(p_obj#, p_timestamp)
  storage (initial 10k next 100k maxextents unlimited pctincrease 0)
/

下面我们看一下他们使用了哪些block:

SYS@lunar>select owner,segment_name,segment_type,extent_id,file_id,block_id from DBA_EXTENTS where segment_name like '%DEPENDENCY%';

OWNER           SEGMENT_NAME                   SEGMENT_TYPE        EXTENT_ID    FILE_ID   BLOCK_ID
--------------- ------------------------------ ------------------ ---------- ---------- ----------
SYS             DEPENDENCY$                    TABLE                       0          1        689
SYS             DEPENDENCY$                    TABLE                       1          1       6329
SYS             DEPENDENCY$                    TABLE                       2          1       6665
SYS             DEPENDENCY$                    TABLE                       3          1       6945
SYS             DEPENDENCY$                    TABLE                       4          1       8481
SYS             DEPENDENCY$                    TABLE                       5          1      10505
SYS             DEPENDENCY$                    TABLE                       6          1      12825
SYS             DEPENDENCY$                    TABLE                       7          1      13425
SYS             DEPENDENCY$                    TABLE                       8          1      14753
SYS             DEPENDENCY$                    TABLE                       9          1      15513
SYS             DEPENDENCY$                    TABLE                      10          1      17169
SYS             DEPENDENCY$                    TABLE                      11          1      19209
SYS             DEPENDENCY$                    TABLE                      12          1      19897
SYS             DEPENDENCY$                    TABLE                      13          1      26689
SYS             DEPENDENCY$                    TABLE                      14          1      28153
SYS             DEPENDENCY$                    TABLE                      15          1      29377
SYS             DEPENDENCY$                    TABLE                      16          1      31241
SYS             DEPENDENCY$                    TABLE                      17          1      45321
SYS             DEPENDENCY$                    TABLE                      18          1      51465
SYS             DEPENDENCY$                    TABLE                      19          1      55433
SYSMAN          MGMT_INV_DEPENDENCY_RULE       TABLE                       0          3      24009
SYSMAN          MGMT_METRIC_DEPENDENCY_DEF     TABLE                       0          3      26193
SYSMAN          MGMT_METRIC_DEPENDENCY         TABLE                       0          3      26209
SYSMAN          MGMT_METRIC_DEPENDENCY_DETAILS TABLE                       0          3      26225
SYS             I_DEPENDENCY1                  INDEX                       0          1        929
SYS             I_DEPENDENCY1                  INDEX                       1          1       6313
SYS             I_DEPENDENCY1                  INDEX                       2          1       6777
SYS             I_DEPENDENCY1                  INDEX                       3          1       8905
SYS             I_DEPENDENCY1                  INDEX                       4          1      12185
SYS             I_DEPENDENCY1                  INDEX                       5          1      13433
SYS             I_DEPENDENCY1                  INDEX                       6          1      14761
SYS             I_DEPENDENCY1                  INDEX                       7          1      15537
SYS             I_DEPENDENCY1                  INDEX                       8          1      18425
SYS             I_DEPENDENCY1                  INDEX                       9          1      19273
SYS             I_DEPENDENCY1                  INDEX                      10          1      26705
SYS             I_DEPENDENCY1                  INDEX                      11          1      28297
SYS             I_DEPENDENCY1                  INDEX                      12          1      30609
SYS             I_DEPENDENCY1                  INDEX                      13          1      32297
SYS             I_DEPENDENCY1                  INDEX                      14          1      34353
SYS             I_DEPENDENCY1                  INDEX                      15          1      37129
SYS             I_DEPENDENCY1                  INDEX                      16          1      38665
SYS             I_DEPENDENCY1                  INDEX                      17          1      47113
SYS             I_DEPENDENCY1                  INDEX                      18          1      51721
SYS             I_DEPENDENCY2                  INDEX                       0          1        937
SYS             I_DEPENDENCY2                  INDEX                       1          1       6337
SYS             I_DEPENDENCY2                  INDEX                       2          1       6657
SYS             I_DEPENDENCY2                  INDEX                       3          1       6961
SYS             I_DEPENDENCY2                  INDEX                       4          1       9545
SYS             I_DEPENDENCY2                  INDEX                       5          1      11481
SYS             I_DEPENDENCY2                  INDEX                       6          1      13281
SYS             I_DEPENDENCY2                  INDEX                       7          1      14369
SYS             I_DEPENDENCY2                  INDEX                       8          1      14841
SYS             I_DEPENDENCY2                  INDEX                       9          1      16617
SYS             I_DEPENDENCY2                  INDEX                      10          1      18409
SYS             I_DEPENDENCY2                  INDEX                      11          1      19889
SYS             I_DEPENDENCY2                  INDEX                      12          1      26681
SYS             I_DEPENDENCY2                  INDEX                      13          1      28129
SYS             I_DEPENDENCY2                  INDEX                      14          1      29369
SYS             I_DEPENDENCY2                  INDEX                      15          1      30649
SYS             I_DEPENDENCY2                  INDEX                      16          1      32137
SYS             I_DEPENDENCY2                  INDEX                      17          1      45449
SYS             I_DEPENDENCY2                  INDEX                      18          1      51593
SYS             I_DEPENDENCY2                  INDEX                      19          1      59785
SYSMAN          PK_MGMT_METRIC_DEPENDENCY_DEF  INDEX                       0          3      26201
SYSMAN          PK_MGMT_METRIC_DEPENDENCY      INDEX                       0          3      26217

65 rows selected.

SYS@lunar>

下面我们在10.2总模拟i_dependency1失效(测试环境使用10.2.0.1):


[oracle@lunar oracle]$ ss

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 22 08:10:07 2014

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

SYS@lunar>alter index i_dependency1 unusable;

Index altered.

SYS@lunar>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER           TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
--------------- ------------------------------ ------------------------------ ------------------------------ --------
SYS             DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         UNUSABLE
SYS             DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID

SYS@lunar>select obj#,flags,to_char(flags,'xxxxxxxxxxxxx') from ind$ where obj# in (92,122,123);

      OBJ#      FLAGS TO_CHAR(FLAGS,
---------- ---------- --------------
       122       2051            803   ------ 2000+10+40+1=====>/* unusable (dls) : 0x01 ,这里也可以看出来被为unusable了
       123       2050            802

SYS@lunar>

执行alter index i_dependency1 unusable后,alert.log的信息如下:

Mon Dec 22 09:08:37 2014
Index SYS.I_DEPENDENCY1 or some [sub]partitions of the index have been marked unusable

[oracle@lunar bdump]$ date
Mon Dec 22 09:11:29 CST 2014
[oracle@lunar bdump]$ 

ind$的flags字段的定义(该定义可以查询基表定义,10.2中在sql.bsq,11.1以后,在dcore.bsq):

  flags         number not null,      																												
                /* mutable flags: anything permanent should go into property */
                                                    /* unusable (dls) : 0x01 */
                                                    /* analyzed       : 0x02 */
                                                    /* no logging     : 0x04 */
                                    /* index is currently being built : 0x08 */
                                     /* index creation was incomplete : 0x10 */
                                           /* key compression enabled : 0x20 */
                                              /* user-specified stats : 0x40 */
                                            /* secondary index on IOT : 0x80 */
                                      /* index is being online built : 0x100 */
                                    /* index is being online rebuilt : 0x200 */   
                                                /* index is disabled : 0x400 */   ---------注意这里,设置一个index失效的标示(转换成10进制是1024)
                                                     /* global stats : 0x800 */
                                            /* fake index(internal) : 0x1000 */
                                       /* index on UROWID column(s) : 0x2000 */
                                            /* index with large key : 0x4000 */
                             /* move partitioned rows in base table : 0x8000 */
                                 /* index usage monitoring enabled : 0x10000 */   
                      /* 4 bits reserved for bitmap index version : 0x1E0000 */
                                      /* Delayed Segment Creation: 0x4000000 */

通过对比,可以发现实际上对于ind$.flags的定义并没有改变,只是内部操作有所变化。
这些标示很有用,比如orcacle 9204有监控SYS的XXX索引的bug会导致数据库不能open,可以通过这个标示来修复。
再比如,10g以前online 创建索引中断后很麻烦,也可以通过修改这个来修复。
10.2以后可以通过dbms_repair.online_index_clean来清理索引状态。不过该功能没有记录在10.2的官方文档中(但是用desc dbms_repair可以看到该功能)。
Oracle官方把这个在文档漏掉如此优秀功能的做法解释为Bug 3965042 : NEW FUNCTION DBMS_REPAIR.ONLINE_INDEX_CLEAN NEEDS TO BE DOCUMENTED)
ALTER TABLE MOVE ONLINE一个IOT表,也可能有类似问题,处理方法也雷同。


这时候,启动数据库会收到ORA-00704 ORA-00604 ORA-01502: index ‘SYS.I_DEPENDENCY1’ or partition of such index is in unusable state报错:

SYS@lunar>shutdown abort
ORACLE instance shut down.
SYS@lunar>startup mount
ORACLE instance started.

Total System Global Area  167772160 bytes
Fixed Size                  1218316 bytes
Variable Size              71305460 bytes
Database Buffers           92274688 bytes
Redo Buffers                2973696 bytes
Database mounted.
SYS@lunar>oradebug setmypid
alter session set db_file_multiblocK_read_count=1;
alter session set tracefile_identifier='lunar';
Statement processed.
SYS@lunar>
Session altered.

SYS@lunar>oradebug event 10046 trace name context forever,level 12;

Session altered.

SYS@lunar>Statement processed.
SYS@lunar>oradebug tracefile_name
/home/oracle/oracle/product/admin/orcl/udump/orcl_ora_19783_lunar.trc
SYS@lunar>alter database open; 
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced


SYS@lunar>

alert.log中信息如下

Mon Dec 22 09:13:29 2014
SMON: enabling cache recovery
Mon Dec 22 09:13:29 2014
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=17, OS id=19789
Mon Dec 22 09:13:32 2014
Deleted Oracle managed file /home/oracle/oracle/product/flash_recovery_area/ORCL/archivelog/2009_11_21/o1_mf_1_42_5jg30x9m_.arc
Mon Dec 22 09:13:32 2014
Errors in file /home/oracle/oracle/product/admin/orcl/udump/orcl_ora_19783_lunar.trc:
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_DEPENDENCY1' or partition of such index is in unusable state
Mon Dec 22 09:13:32 2014
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 19783
ORA-1092 signalled during: alter database open...

trace文件信息:

PARSING IN CURSOR #5 len=179 dep=1 uid=0 oct=3 lid=0 tim=1385948058756927 hv=2812844157 ad='29b07834'
select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,d_attrs from dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#
END OF STMT
PARSE #5:c=6999,e=70263,p=8,cr=40,cu=0,mis=1,r=0,dep=1,og=4,tim=1385948058756919
=====================
PARSING IN CURSOR #2 len=84 dep=2 uid=0 oct=3 lid=0 tim=1385948058757830 hv=2686874206 ad='29b0652c'
select o.name, u.name from obj$ o, user$ u  where o.obj# = :1 and o.owner# = u.user#
END OF STMT
PARSE #2:c=1000,e=266,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,tim=1385948058757826
BINDS #2:
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=b72f40bc  bln=22  avl=03  flg=05
  value=122  ---------》》obj#=122的正是I_DEPENDENCY1
EXEC #2:c=0,e=658,p=0,cr=0,cu=0,mis=1,r=0,dep=2,og=4,tim=1385948058758543
WAIT #2: nam='db file sequential read' ela= 15160 file#=1 block#=98 blocks=1 obj#=-1 tim=1385948058773774
WAIT #2: nam='db file sequential read' ela= 481 file#=1 block#=90 blocks=1 obj#=-1 tim=1385948058774587
FETCH #2:c=2000,e=16086,p=2,cr=5,cu=0,mis=0,r=1,dep=2,og=4,tim=1385948058774648
STAT #2 id=1 cnt=1 pid=0 pos=1 obj=0 op='NESTED LOOPS  (cr=5 pr=2 pw=0 time=16090 us)'
STAT #2 id=2 cnt=1 pid=1 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=0 pw=0 time=28 us)'
STAT #2 id=3 cnt=1 pid=2 pos=1 obj=36 op='INDEX UNIQUE SCAN I_OBJ1 (cr=2 pr=0 pw=0 time=16 us)'
STAT #2 id=4 cnt=1 pid=1 pos=2 obj=22 op='TABLE ACCESS CLUSTER USER$ (cr=2 pr=2 pw=0 time=16050 us)'
STAT #2 id=5 cnt=1 pid=4 pos=1 obj=11 op='INDEX UNIQUE SCAN I_USER# (cr=1 pr=1 pw=0 time=15264 us)'
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_DEPENDENCY1' or partition of such index is in unusable state
EXEC #1:c=252962,e=5328707,p=155,cr=730,cu=0,mis=0,r=0,dep=0,og=1,tim=1385948059778098
ERROR #1:err=1092 tim=432829200

可以看到,下面的递归SQL导致报错和数据库不能open:
select owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,d_attrs
from dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#

这里绑定变量为122的正是I_DEPENDENCY1。

现在我们把数据库启动到upgrade然后修改flags=1024(即,标记该索引为无效)试试看:

[oracle@lunar oracle]$ ss

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 22 09:22:06 2014

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to an idle instance.

SYS@lunar>startup mount
ORACLE instance started.

Total System Global Area  167772160 bytes
Fixed Size                  1218316 bytes
Variable Size              71305460 bytes
Database Buffers           92274688 bytes
Redo Buffers                2973696 bytes
Database mounted.
SYS@lunar>oradebug setmypid
Statement processed.
SYS@lunar>alter session set db_file_multiblocK_read_count=1;

Session altered.

SYS@lunar>alter session set tracefile_identifier='lunar';

Session altered.

SYS@lunar>oradebug event 10046 trace name context forever,level 12;
Statement processed.
SYS@lunar>oradebug tracefile_name
/home/oracle/oracle/product/admin/orcl/udump/orcl_ora_19834_lunar.trc
SYS@lunar>alter database open upgrade;      

Database altered.

SYS@lunar>

alert.log信息如下:

Mon Dec 22 09:23:02 2014
Successfully onlined Undo Tablespace 1.
Mon Dec 22 09:23:02 2014
SMON: enabling tx recovery
Mon Dec 22 09:23:03 2014
Database Characterset is ZHS16GBK
Mon Dec 22 09:23:06 2014
Stopping background process MMNL
Mon Dec 22 09:23:06 2014
Stopping background process MMON
Starting background process MMON
MMON started with pid=10, OS id=19842
Mon Dec 22 09:23:06 2014
Starting background process MMNL
MMNL started with pid=11, OS id=19844
Mon Dec 22 09:23:06 2014
ALTER SYSTEM enable restricted session;
Mon Dec 22 09:23:07 2014
ALTER SYSTEM SET _system_trig_enabled=FALSE SCOPE=MEMORY;
Mon Dec 22 09:23:07 2014
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
Mon Dec 22 09:23:07 2014
ALTER SYSTEM SET resource_manager_plan='' SCOPE=MEMORY;
replication_dependency_tracking turned off (no async multimaster replication found)
kwqiconfy: Warning AQ Kchunk open notifier failed with 1502 
 XDB UNINITIALIZED: XDB$SCHEMA not accessible 
Completed: alter database open upgrade
1



可以看到数据库可以启动到upgrade模式,找到刚才报错的语句,发现在upgrade模式下
1
=====================
PARSING IN CURSOR #4 len=221 dep=2 uid=0 oct=3 lid=0 tim=1385948615586776 hv=1926936385 ad='29b04538'
select /* ordered use_nl(o) index(d) index(o) */ owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,d_attrs from dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#
END OF STMT
PARSE #4:c=2999,e=3438,p=4,cr=35,cu=0,mis=1,r=0,dep=2,og=4,tim=1385948615586774
=====================
PARSING IN CURSOR #2 len=84 dep=3 uid=0 oct=3 lid=0 tim=1385948615587498 hv=2686874206 ad='29b0396c'
select o.name, u.name from obj$ o, user$ u  where o.obj# = :1 and o.owner# = u.user#
END OF STMT
PARSE #2:c=0,e=279,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=4,tim=1385948615587494
BINDS #2:
kkscoacd
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=b725753c  bln=22  avl=03  flg=05
  value=122
EXEC #2:c=1000,e=534,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=4,tim=1385948615588087
FETCH #2:c=0,e=56,p=0,cr=5,cu=0,mis=0,r=1,dep=3,og=4,tim=1385948615588159
STAT #2 id=1 cnt=1 pid=0 pos=1 obj=0 op='NESTED LOOPS  (cr=5 pr=0 pw=0 time=60 us)'
STAT #2 id=2 cnt=1 pid=1 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=3 pr=0 pw=0 time=26 us)'
STAT #2 id=3 cnt=1 pid=2 pos=1 obj=36 op='INDEX UNIQUE SCAN I_OBJ1 (cr=2 pr=0 pw=0 time=15 us)'
STAT #2 id=4 cnt=1 pid=1 pos=2 obj=22 op='TABLE ACCESS CLUSTER USER$ (cr=2 pr=0 pw=0 time=23 us)'
STAT #2 id=5 cnt=1 pid=4 pos=1 obj=11 op='INDEX UNIQUE SCAN I_USER# (cr=1 pr=0 pw=0 time=6 us)'

具体SQL:

select /* ordered use_nl(o) index(d) index(o) */ owner#,name,namespace,remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,d_attrs 
from dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#

查询一下dba_objects就会报ORA-00604 ORA-01502: index ‘SYS.I_DEPENDENCY1’ or partition of such index is in unusable state错误。
道理跟上面说的一样,索引损坏了,但是Oracle傻乎乎的非要走这个索引,因此就会报错:

SYS@lunar>select owner,object_id,object_name,object_type from dba_objects where object_id in (92,122,123);
select owner,object_id,object_name,object_type from dba_objects where object_id in (92,122,123)
                                                    *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01502: index 'SYS.I_DEPENDENCY1' or partition of such index is in unusable state
SYS@lunar>

那么Oracle启动时为什么可以启动呢?
这是因为Oracle在Upgrade模式下会有几项不同于普通open的动作。
猜测正常启动数据库时因为执行递归sql来获取OBJ#和USER#时,执行计划中首先根据I_DEPENDENCY1来对DEPENDENCY1进行扫描。
因此该索引的不可用状态导致了数据库不能open。而upgrade模式启动数据库时,使用了/* ordered use_nl(o) index(d) index(o) */ hint。
因此获取OBJ#时使用了oid$来获取obj#,而没有使用SYS.I_DEPENDENCY1索引,因此,可以upgrade模式打开库。
通过tkprof可以看到这个递归sql的执行计划:


select /* ordered use_nl(o) index(d) index(o) */ owner#,name,namespace,
  remoteowner,linkname,p_timestamp,p_obj#, nvl(property,0),subname,d_attrs 
from
 dependency$ d, obj$ o where d_obj#=:1 and p_obj#=obj#(+) order by order#


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        5      0.00       0.00          0          0          0           0
Execute      0      0.00       0.00          0          0          0           0
Fetch        0      0.00       0.00          0          0          0           0
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      0.00       0.00          0          0          0           0

Misses in library cache during parse: 5
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 2)
********************************************************************************

select obj# 
from
 oid$ where user#=:1 and oid$=:2


call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      2      0.00       0.00          0          0          0           0
Fetch        2      0.00       0.01          3          5          0           1
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      0.00       0.02          3          5          0           1

Misses in library cache during parse: 1
Misses in library cache during execute: 1
Optimizer mode: CHOOSE
Parsing user id: SYS   (recursive depth: 1)

Rows     Row Source Operation
-------  ---------------------------------------------------
      0  TABLE ACCESS BY INDEX ROWID OID$ (cr=2 pr=2 pw=0 time=7433 us)
      0   INDEX UNIQUE SCAN I_OID1 (cr=2 pr=2 pw=0 time=7419 us)(object id 179)


Elapsed times include waiting on following events:
  Event waited on                             Times   Max. Wait  Total Waited
  ----------------------------------------   Waited  ----------  ------------
  db file sequential read                         3        0.00          0.01


oid$的定义如下:

create table oid$                    /* OID mapping table for schema objects */
(
  user#         number not null,   /* user this mapping is for (user$.user#) */
  oid$          raw(16) not null,        /* OID for typed table/view or type */
  obj#          number not null)         /* target object number (obj$.obj#) */
                                                       /* key: (user#, oid$) */
/

现在我们来修复:

[oracle@lunar oracle]$ ss

SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 22 08:10:07 2014

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options

SYS@lunar>select obj# from obj$ where name='I_DEPENDENCY1';

      OBJ#
----------
       122


SYS@lunar>update ind$ set flags=1024 where obj#=122;

1 row updated.

SYS@lunar>select flags,obj# from ind$ where obj#=122;

     FLAGS       OBJ#
---------- ----------
      1024        122

SYS@lunar>commit;

Commit complete.

SYS@lunar>

重启数据库到限制模式:
SYS@lunar>alter index i_dependency1 rebuild
  2  ;

Index altered.

SYS@lunar>analyze table dependency$ validate structure cascade;

Table analyzed.

SYS@lunar>select owner,table_name,index_name,tablespace_name,status from dba_indexes where table_name='DEPENDENCY$';

OWNER           TABLE_NAME                     INDEX_NAME                     TABLESPACE_NAME                STATUS
--------------- ------------------------------ ------------------------------ ------------------------------ --------
SYS             DEPENDENCY$                    I_DEPENDENCY1                  SYSTEM                         VALID
SYS             DEPENDENCY$                    I_DEPENDENCY2                  SYSTEM                         VALID

SYS@lunar>

至此,数据库已经修复

发表在 backup&recovery, ORA-600 or ORA-7445 | 标签为 , , | 留下评论

根据一个给定的rowid手工推算他的file#,block#,obj#,row#

从Oracle 8i开始使用扩展rowid标识行物理地址,扩展rowid使用base64编码行的物理地址,编码字符包含A-Z, a-z, 0-9, +, 和/。
扩展rowid由四部分组成:OOOOOOOFFFBBBBBBRRR。其中:
OOOOOO:数据对象编号(6位显示)
FFF:相关数据文件编号(3位显示)
BBBBBB:数据块编号(6位显示)
RRR:数据块中行编号(3位显示)
8i以后,rowid采用base64编码(基于64位的编码)的扩展rowid.
.
关于64bit编码表,可以搜索Google或者Baidu,关键字“Base64编码表”。

无标题


将64位编码转换为十进制:
file#: AAB —–> 0 0 1 —–>0*64^2+0*64^1+1*64^0 =1
block#: AAAAMh —–> 0 0 0 0 12 33 —–>0*64^5+0*64^4+0*64^3+0*64^2+12*64^1+33*64^0 =801
obj#: AAAABi —–> 0 0 0 0 1 34 —–>0*64^5+0*64^4+0*64^3+0*64^2+1*64^1+34*64^0 =98
row#:AAf —–> 0 0 31 —–>0*64^2+0*64^4+0*64^1+31*64^0 =31
注意这里,row#是31行(我的环境是Oracle 11.2.0.3),记住row#是从0开始计数的,这个是固定位置(10g中是第25行)

验证一下:
SYS@lunarp>select rownum,substr(rowid,1,6) “object”,
2 substr(rowid,7,3) “file”,
3 substr(rowid,10,6) “block”,
4 substr(rowid,16,3) “row”
5 from props$
6 where name = ‘GLOBAL_DB_NAME’;

ROWNUM object file block row
———- ———————— ———— ———————— ————
1 AAAABi AAB AAAAMh AAf

Elapsed: 00:00:00.01
SYS@lunarp>select distinct dbms_rowid.rowid_relative_fno(rowid) file#,
2 dbms_rowid.rowid_block_number(rowid) blk#,
3 dbms_rowid.rowid_object(rowid) obj#,
4 dbms_rowid.rowid_row_number(rowid) row#
5 from props$
6 where name = ‘GLOBAL_DB_NAME’;

FILE# BLK# OBJ# ROW#
———- ———- ———- ———-
1 801 98 31

Elapsed: 00:00:00.02
SYS@lunarp>

data_block_dump,data header at 0x76fa205c
===============
tsiz: 0x1fa0
hsiz: 0x5a
pbl: 0x76fa205c
76543210
flag=——–
ntab=1
nrow=36
frre=-1
fsbo=0x5a
fseo=0x16b4
avsp=0x1798
tosp=0x179f

发表在 FAQ | 标签为 | 留下评论