昨天客户的一个重要应用切换到新的系统环境上，今天观察，发现部分异常等待：
从OS的CPU负载来看，定期会出现一个峰值，从ASH中可以看出，这个峰值对应的等待事件跟AWR的完全吻合。
因此，主要怀疑两个东西：
1，应用的SQL和对象的属性（比如table或者index的统计信息，并行度等等……）
2，系统的AIO设置
上面的第一条，已经提交给开发相应的SQL和其他信息
第二条，因为系统以前是11.2 RAC，使用了ASM，而现在是单机文件系统.
因此对比了这两种环境下AIO的异同，结论如下：
1，Linux下，ASM数据库和文件系统数据库的AIO设置差别：
(1). ASM的AIO属性是不受 FILESYSTEMIO_OPTIONS 参数的影响（因为ASM会绕过文件系统buffer），只跟DISK_ASYNCH_IO有关系
(2). 文件系统的AIO属性跟 FILESYSTEMIO_OPTIONS 和 DISK_ASYNCH_IO 都有关系
2，FILESYSTEMIO_OPTIONS=NONE ： Bug 6733627 – Unaccounted Wait Time on “Direct Path” operations with FILESYSTEM_IO_OPTIONS=NONE (Doc ID 6733627.8)
3， db file async I/O submit’相关内容：
‘db file async I/O submit’ when FILESYSTEMIO_OPTIONS=NONE (Doc ID 1274737.1) —详细讲述了 ‘db file async I/O submit’ 和 FILESYSTEMIO_OPTIONS=NONE的关系
当文件系统设置了FILESYSTEMIO_OPTIONS=NONE时，会出现“db file async I/O submit”后台等待事件，而正常应该是出现“db file parallel write”
可以通过设置 FILESYSTEMIO_OPTIONS=SETALL，启用AIO，然后在AWR中会出现db file parallel write，而不再是db file async I/O submit
4，FILESYSTEMIO_OPTIONS=DIRECTIO ： Wrong FILESYSTEMIO_OPTIONS Settings Can Cause a Corrupted Block to be Returned at the First Read (Doc ID 1918825.1)
5，一般，linux建议设置为 FILESYSTEMIO_OPTIONS=SETALL
具体测试过程如下：
先看看官方文档的描述：
FILESYSTEMIO_OPTIONS
--------------------------------
Property Description
Parameter type String
Syntax FILESYSTEMIO_OPTIONS = { none | setall | directIO | asynch }
Default value Varies by database version and operating system.
Modifiable No
Basic No
DISK_ASYNCH_IO
------------------------------
Property Description
Parameter type Boolean
Default value true
Modifiable No
Range of values true | false
Basic No
此处是文件系统：filesystemio_options=none disk_asynch_io=true（缺省值），使用strace发现，没有启用AIO：
15:21:06 SYS@ Lunar> show parameter filesystemio_options NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ filesystemio_options string none 15:21:16 SYS@ Lunar> show parameter DISK_ASYNCH_IO NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ disk_asynch_io boolean TRUE 15:21:30 SYS@ Lunar>
在系统上看，目前并没有启用AIO：
[oracle@Lunar tmp]$ cat /proc/slabinfo | grep kio kioctx 140 384 320 12 1 : tunables 54 27 8 : slabdata 31 32 22 kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 [oracle@Lunar tmp]$
使用strace对后台进程DBWR进行跟踪，佐证了上述观点，当前文件系统的数据库没有启用AIO：
[oracle@Lunar tmp]$ ps -ef|grep dbw
oracle 16540 30146 0 14:50 pts/3 00:00:00 grep dbw
oracle 20618 1 0 Sep24 ? 00:02:39 ora_dbw0_Lunar
oracle 20620 1 0 Sep24 ? 00:02:55 ora_dbw1_Lunar
oracle 20622 1 0 Sep24 ? 00:02:47 ora_dbw2_Lunar
oracle 20624 1 0 Sep24 ? 00:02:29 ora_dbw3_Lunar
oracle 20626 1 0 Sep24 ? 00:02:48 ora_dbw4_Lunar
oracle 20628 1 0 Sep24 ? 00:02:41 ora_dbw5_Lunar
oracle 20630 1 0 Sep24 ? 00:02:44 ora_dbw6_Lunar
oracle 20632 1 0 Sep24 ? 00:02:55 ora_dbw7_Lunar
oracle 20634 1 0 Sep24 ? 00:02:06 ora_dbw8_Lunar
oracle 20636 1 0 Sep24 ? 00:01:46 ora_dbw9_Lunar
oracle 20638 1 0 Sep24 ? 00:01:56 ora_dbwa_Lunar
oracle 20640 1 0 Sep24 ? 00:01:58 ora_dbwb_Lunar
oracle 20642 1 0 Sep24 ? 00:01:52 ora_dbwc_Lunar
oracle 20644 1 0 Sep24 ? 00:01:57 ora_dbwd_Lunar
oracle 20646 1 0 Sep24 ? 00:01:50 ora_dbwe_Lunar
oracle 20648 1 0 Sep24 ? 00:01:50 ora_dbwf_Lunar
[oracle@Lunar tmp]$
[oracle@Lunar ~]$ tail -f /tmp/20620.log
20620 0.000030 pwrite(264, "\6\242\0\0\330a\350\20\241\363;\201\241\5\1\6\331\242\0\0\2\0\f\0M.\1\0s\363;\201"..., 8192, 21680029696) = 8192
20620 0.000088 times({tms_utime=7086, tms_stime=10442, tms_cutime=0, tms_cstime=0}) = 454220443
20620 0.000032 pwrite(281, "\6\242\0\0\311O+\25\2475=\201\241\5\1\0061\223\0\0\2\0\34\0S.\1\0\326-=\201"..., 8192, 23252770816) = 8192
20620 0.000090 times({tms_utime=7086, tms_stime=10442, tms_cutime=0, tms_cstime=0}) = 454220443
20620 0.000031 pwrite(282, "\6\242\0\0\206\201&\27\270\364;\201\241\5\1\6{\300\0\0\2\0\6\0M.\1\0\243\364;\201"..., 8192, 20672724992) = 8192
20620 0.000115 times({tms_utime=7086, tms_stime=10442, tms_cutime=0, tms_cstime=0}) = 454220443
20620 0.000038 pwrite(286, "\6\242\0\0v#\5\24\363J<\201\241\5\1\6J\315\0\0\2\0\26\0M.\1\0\tP\306\200"..., 24576, 2758721536) = 24576
20620 0.000116 times({tms_utime=7086, tms_stime=10442, tms_cutime=0, tms_cstime=0}) = 454220443
20620 0.000026 times({tms_utime=7086, tms_stime=10442, tms_cutime=0, tms_cstime=0}) = 454220443
20620 0.000022 semtimedop(557058, 0x7fffdef50660, 1, {2, 990000000}
strace中，没有io_submi函数，因为 filesystemio_options = none
如果 filesystemio_options = setall ，那么会出现 io_submi函数
下面的测试是使用ASM的数据库的参数：
15:24:25 SYS@ Lunardb1> show parameter FILESYSTEMIO_OPTIONS NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ filesystemio_options string none 15:24:29 SYS@ Lunardb1> show parameter DISK_ASYNCH_IO NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ disk_asynch_io boolean TRUE 15:24:39 SYS@ Lunardb1>
在系统上看，虽然设置了FILESYSTEMIO_OPTIONS=NONE，但是由于DISK_ASYNCH_IO=TRUE（缺省值），因此，ASM下的数据库依然使用AIO：
[root@Lunardb1 ~]# cat /proc/slabinfo | grep kio kioctx 238 340 384 10 1 : tunables 54 27 8 : slabdata 34 34 0 kiocb 3656 4845 256 15 1 : tunables 120 60 8 : slabdata 323 323 180 [root@Lunardb1 ~]# [oracle@Lunardb1 ~]$ ps -ef|grep dbw oracle 82860 82820 0 15:25 pts/1 00:00:00 grep dbw grid 85795 1 0 Sep25 ? 00:00:36 asm_dbw0_+ASM1 grid 86406 1 0 Sep25 ? 00:01:32 /u01/app/11.2.0/grid/jdk/jre//bin/java -server -Xcheck:jni -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -Dport.rmi=23792 -jar /u01/app/11.2.0/grid/oc4j/j2ee/home/oc4j.jar -config /u01/app/11.2.0/grid/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -out /u01/app/11.2.0/grid/oc4j/j2ee/home/log/oc4j.out -err /u01/app/11.2.0/grid/oc4j/j2ee/home/log/oc4j.err oracle 87014 1 0 Sep25 ? 00:10:02 ora_dbw0_Lunardb1 oracle 87018 1 0 Sep25 ? 00:11:22 ora_dbw1_Lunardb1 oracle 87022 1 0 Sep25 ? 00:09:28 ora_dbw2_Lunardb1 oracle 87026 1 0 Sep25 ? 00:08:50 ora_dbw3_Lunardb1 oracle 87030 1 0 Sep25 ? 00:09:43 ora_dbw4_Lunardb1 oracle 87041 1 0 Sep25 ? 00:09:47 ora_dbw5_Lunardb1 oracle 87048 1 0 Sep25 ? 00:08:52 ora_dbw6_Lunardb1 oracle 87052 1 0 Sep25 ? 00:08:59 ora_dbw7_Lunardb1 oracle 87056 1 0 Sep25 ? 00:08:26 ora_dbw8_Lunardb1 oracle 87060 1 0 Sep25 ? 00:12:26 ora_dbw9_Lunardb1 oracle 87064 1 0 Sep25 ? 00:09:35 ora_dbwa_Lunardb1 oracle 87068 1 0 Sep25 ? 00:09:25 ora_dbwb_Lunardb1 oracle 87072 1 0 Sep25 ? 00:08:35 ora_dbwc_Lunardb1 oracle 87076 1 0 Sep25 ? 00:09:22 ora_dbwd_Lunardb1 oracle 87080 1 0 Sep25 ? 00:08:16 ora_dbwe_Lunardb1 [oracle@Lunardb1 ~]$
可以看到，使用ASM数据库的dbw0进程，即使FILESYSTEMIO_OPTIONS设置为NONE，只要disk_asynch_io设置为true（缺省值），DBWR也可以使用到AIO：
[root@Lunardb1 ~]# cat /tmp/87014.log|grep io_submit|wc -l
82
[root@Lunardb1 ~]#
[oracle@Lunardb1 ~]$ tail -f /tmp/87014.log
87014 0.000056 times({tms_utime=45414, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051459
87014 0.000089 times({tms_utime=45414, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051459
87014 0.000054 times({tms_utime=45414, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051459
87014 0.001498 io_submit(140281973911552, 96, {{0x7f95f0a56d18, 0, 1, 0, 260}, {0x7f95f0e5cf50, 0, 1, 0, 262}, {0x7f95f0d97080, 0, 1, 0, 264}, {0x7f95f0a5a7b0, 0, 1, 0, 258}, {0x7f95f09703a0, 0, 1, 0, 256}, {0x7f95f0a5f2b0, 0, 1, 0, 266}, {0x7f95f0a42e58, 0, 1, 0, 259}, {0x7f95f0982d48, 0, 1, 0, 260}, {0x7f95f0e6a240, 0, 1, 0, 266}, {0x7f95f09734d8, 0, 1, 0, 267}, {0x7f95f071b2a8, 0, 1, 0, 269}, {0x7f95f0a5d438, 0, 1, 0, 259}, {0x7f95f0719430, 0, 1, 0, 256}, {0x7f95f0747420, 0, 1, 0, 269}, {0x7f95f0effa40, 0, 1, 0, 266}, {0x7f95f0d86550, 0, 1, 0, 256}, {0x7f95f071cec8, 0, 1, 0, 256}, {0x7f95f0a43560, 0, 1, 0, 256}, {0x7f95f0e780e8, 0, 1, 0, 265}, {0x7f95f074b5c0, 0, 1, 0, 262}, {0x7f95f095a668, 0, 1, 0, 260}, {0x7f95f0957e90, 0, 1, 0, 266}, {0x7f95f094adf8, 0, 1, 0, 258}, {0x7f95f0a8a070, 0, 1, 0, 258}, {0x7f95f097bf20, 0, 1, 0, 258}, {0x7f95f0e5c398, 0, 1, 0, 265}, {0x7f95f0aa2c88, 0, 1, 0, 263}, {0x7f95f0743730, 0, 1, 0, 269}, {0x7f95f0740850, 0, 1, 0, 260}, {0x7f95f0939968, 0, 1, 0, 264}, {0x7f95f0e5c5f0, 0, 1, 0, 264}, {0x7f95f0a4ec30, 0, 1, 0, 264}, {0x7f95f0d7ae80, 0, 1, 0, 256}, {0x7f95f0959ab0, 0, 1, 0, 256}, {0x7f95f0f01fc0, 0, 1, 0, 260}, {0x7f95f095c4e0, 0, 1, 0, 271}, {0x7f95f0aa3cf0, 0, 1, 0, 264}, {0x7f95f0d9c288, 0, 1, 0, 264}, {0x7f95f07494f0, 0, 1, 0, 258}, {0x7f95f0edef98, 0, 1, 0, 268}, {0x7f95f0a2f8f8, 0, 1, 0, 256}, {0x7f95f0709e18, 0, 1, 0, 269}, {0x7f95f0d7caa0, 0, 1, 0, 261}, {0x7f95f0e830b0, 0, 1, 0, 256}, {0x7f95f0a967a8, 0, 1, 0, 270}, {0x7f95f0ede890, 0, 1, 0, 262}, {0x7f95f0e7ab18, 0, 1, 0, 259}, {0x7f95f0a202e0, 0, 1, 0, 266}, {0x7f95f09778d0, 0, 1, 0, 266}, {0x7f95f0d781f8, 0, 1, 0, 266}, {0x7f95f074c3d0, 0, 1, 0, 269}, {0x7f95f0740f58, 0, 1, 0, 269}, {0x7f95f0dc0570, 0, 1, 0, 265}, {0x7f95f097a558, 0, 1, 0, 265}, {0x7f95f07204b0, 0, 1, 0, 265}, {0x7f95f0973be0, 0, 1, 0, 270}, {0x7f95f0a53988, 0, 1, 0, 270}, {0x7f95f0e934d8, 0, 1, 0, 270}, {0x7f95f0745350, 0, 1, 0, 266}, {0x7f95f0a56ac0, 0, 1, 0, 263}, {0x7f95f0964118, 0, 1, 0, 263}, {0x7f95f0727c38, 0, 1, 0, 263}, {0x7f95f096ace8, 0, 1, 0, 260}, {0x7f95f0d7f980, 0, 1, 0, 260}, {0x7f95f0f05f08, 0, 1, 0, 258}, {0x7f95f0d88f80, 0, 1, 0, 261}, {0x7f95f0a262f8, 0, 1, 0, 265}, {0x7f95f0a209e8, 0, 1, 0, 263}, {0x7f95f0ea2898, 0, 1, 0, 266}, {0x7f95f0720000, 0, 1, 0, 264}, {0x7f95f0d9c4e0, 0, 1, 0, 264}, {0x7f95f0e84cd0, 0, 1, 0, 262}, {0x7f95f0f10a20, 0, 1, 0, 268}, {0x7f95f095e808, 0, 1, 0, 260}, {0x7f95f0d809e8, 0, 1, 0, 258}, {0x7f95f0e69430, 0, 1, 0, 261}, {0x7f95f0efaf40, 0, 1, 0, 261}, {0x7f95f0941f00, 0, 1, 0, 261}, {0x7f95f0ea0c78, 0, 1, 0, 267}, {0x7f95f0a983c8, 0, 1, 0, 269}, {0x7f95f0a2be60, 0, 1, 0, 258}, {0x7f95f0a615d8, 0, 1, 0, 259}, {0x7f95f0ef68f0, 0, 1, 0, 259}, {0x7f95f0742470, 0, 1, 0, 268}, {0x7f95f0e64228, 0, 1, 0, 260}, {0x7f95f0dc07c8, 0, 1, 0, 260}, {0x7f95f0a48e70, 0, 1, 0, 260}, {0x7f95f0daace8, 0, 1, 0, 259}, {0x7f95f09498e0, 0, 1, 0, 259}, {0x7f95f0715740, 0, 1, 0, 267}, {0x7f95f0f071c8, 0, 1, 0, 271}, {0x7f95f0aac030, 0, 1, 0, 265}, {0x7f95f0e62ab8, 0, 1, 0, 261}, {0x7f95f093ba38, 0, 1, 0, 260}, {0x7f95f0723cf0, 0, 1, 0, 260}, {0x7f95f096de20, 0, 1, 0, 258}}) = 96
87014 0.007424 io_getevents(140281973911552, 7, 128, {{0x7f95f0a56d18, 0x7f95f0a56d18, 8192, 0}, {0x7f95f0e5cf50, 0x7f95f0e5cf50, 8192, 0}, {0x7f95f0d97080, 0x7f95f0d97080, 8192, 0}, {0x7f95f0a5a7b0, 0x7f95f0a5a7b0, 8192, 0}, {0x7f95f09703a0, 0x7f95f09703a0, 8192, 0}, {0x7f95f0a5f2b0, 0x7f95f0a5f2b0, 8192, 0}, {0x7f95f0a42e58, 0x7f95f0a42e58, 8192, 0}, {0x7f95f0982d48, 0x7f95f0982d48, 8192, 0}, {0x7f95f0e6a240, 0x7f95f0e6a240, 8192, 0}, {0x7f95f09734d8, 0x7f95f09734d8, 8192, 0}, {0x7f95f071b2a8, 0x7f95f071b2a8, 8192, 0}, {0x7f95f0a5d438, 0x7f95f0a5d438, 8192, 0}, {0x7f95f0719430, 0x7f95f0719430, 8192, 0}, {0x7f95f0747420, 0x7f95f0747420, 8192, 0}, {0x7f95f0effa40, 0x7f95f0effa40, 8192, 0}, {0x7f95f0d86550, 0x7f95f0d86550, 8192, 0}, {0x7f95f071cec8, 0x7f95f071cec8, 8192, 0}, {0x7f95f0a43560, 0x7f95f0a43560, 8192, 0}, {0x7f95f0e780e8, 0x7f95f0e780e8, 8192, 0}, {0x7f95f074b5c0, 0x7f95f074b5c0, 8192, 0}, {0x7f95f095a668, 0x7f95f095a668, 8192, 0}, {0x7f95f0957e90, 0x7f95f0957e90, 8192, 0}, {0x7f95f094adf8, 0x7f95f094adf8, 8192, 0}, {0x7f95f0a8a070, 0x7f95f0a8a070, 8192, 0}, {0x7f95f097bf20, 0x7f95f097bf20, 8192, 0}, {0x7f95f0e5c398, 0x7f95f0e5c398, 8192, 0}, {0x7f95f0aa2c88, 0x7f95f0aa2c88, 8192, 0}, {0x7f95f0743730, 0x7f95f0743730, 8192, 0}, {0x7f95f0740850, 0x7f95f0740850, 8192, 0}, {0x7f95f0939968, 0x7f95f0939968, 8192, 0}, {0x7f95f0e5c5f0, 0x7f95f0e5c5f0, 8192, 0}, {0x7f95f0a4ec30, 0x7f95f0a4ec30, 8192, 0}, {0x7f95f0d7ae80, 0x7f95f0d7ae80, 8192, 0}, {0x7f95f0959ab0, 0x7f95f0959ab0, 8192, 0}, {0x7f95f0f01fc0, 0x7f95f0f01fc0, 8192, 0}, {0x7f95f095c4e0, 0x7f95f095c4e0, 8192, 0}, {0x7f95f0aa3cf0, 0x7f95f0aa3cf0, 8192, 0}, {0x7f95f0d9c288, 0x7f95f0d9c288, 8192, 0}, {0x7f95f07494f0, 0x7f95f07494f0, 8192, 0}, {0x7f95f0a2f8f8, 0x7f95f0a2f8f8, 8192, 0}, {0x7f95f0edef98, 0x7f95f0edef98, 8192, 0}, {0x7f95f0709e18, 0x7f95f0709e18, 8192, 0}, {0x7f95f0d7caa0, 0x7f95f0d7caa0, 8192, 0}, {0x7f95f0e830b0, 0x7f95f0e830b0, 8192, 0}, {0x7f95f0a967a8, 0x7f95f0a967a8, 8192, 0}, {0x7f95f0ede890, 0x7f95f0ede890, 8192, 0}, {0x7f95f0e7ab18, 0x7f95f0e7ab18, 8192, 0}, {0x7f95f0d781f8, 0x7f95f0d781f8, 8192, 0}, {0x7f95f09778d0, 0x7f95f09778d0, 32768, 0}, {0x7f95f0a202e0, 0x7f95f0a202e0, 65536, 0}, {0x7f95f074c3d0, 0x7f95f074c3d0, 8192, 0}, {0x7f95f0740f58, 0x7f95f0740f58, 8192, 0}, {0x7f95f0dc0570, 0x7f95f0dc0570, 16384, 0}, {0x7f95f097a558, 0x7f95f097a558, 24576, 0}, {0x7f95f07204b0, 0x7f95f07204b0, 8192, 0}, {0x7f95f0973be0, 0x7f95f0973be0, 8192, 0}, {0x7f95f0a53988, 0x7f95f0a53988, 8192, 0}, {0x7f95f0e934d8, 0x7f95f0e934d8, 8192, 0}, {0x7f95f0745350, 0x7f95f0745350, 8192, 0}, {0x7f95f0a56ac0, 0x7f95f0a56ac0, 8192, 0}, {0x7f95f0964118, 0x7f95f0964118, 8192, 0}, {0x7f95f0727c38, 0x7f95f0727c38, 8192, 0}, {0x7f95f096ace8, 0x7f95f096ace8, 8192, 0}, {0x7f95f0d7f980, 0x7f95f0d7f980, 8192, 0}, {0x7f95f0f05f08, 0x7f95f0f05f08, 8192, 0}, {0x7f95f0d88f80, 0x7f95f0d88f80, 8192, 0}, {0x7f95f0a262f8, 0x7f95f0a262f8, 8192, 0}, {0x7f95f0a209e8, 0x7f95f0a209e8, 8192, 0}, {0x7f95f0ea2898, 0x7f95f0ea2898, 8192, 0}, {0x7f95f0720000, 0x7f95f0720000, 8192, 0}, {0x7f95f0d9c4e0, 0x7f95f0d9c4e0, 8192, 0}, {0x7f95f0e84cd0, 0x7f95f0e84cd0, 8192, 0}, {0x7f95f0f10a20, 0x7f95f0f10a20, 8192, 0}, {0x7f95f095e808, 0x7f95f095e808, 8192, 0}, {0x7f95f0d809e8, 0x7f95f0d809e8, 8192, 0}, {0x7f95f0e69430, 0x7f95f0e69430, 8192, 0}, {0x7f95f0efaf40, 0x7f95f0efaf40, 8192, 0}, {0x7f95f0ea0c78, 0x7f95f0ea0c78, 8192, 0}, {0x7f95f0941f00, 0x7f95f0941f00, 8192, 0}, {0x7f95f0a983c8, 0x7f95f0a983c8, 8192, 0}, {0x7f95f0a2be60, 0x7f95f0a2be60, 8192, 0}, {0x7f95f0a615d8, 0x7f95f0a615d8, 8192, 0}, {0x7f95f0ef68f0, 0x7f95f0ef68f0, 8192, 0}, {0x7f95f0742470, 0x7f95f0742470, 8192, 0}, {0x7f95f0e64228, 0x7f95f0e64228, 8192, 0}, {0x7f95f0dc07c8, 0x7f95f0dc07c8, 24576, 0}, {0x7f95f0a48e70, 0x7f95f0a48e70, 8192, 0}, {0x7f95f0daace8, 0x7f95f0daace8, 8192, 0}, {0x7f95f09498e0, 0x7f95f09498e0, 8192, 0}, {0x7f95f0715740, 0x7f95f0715740, 16384, 0}, {0x7f95f0f071c8, 0x7f95f0f071c8, 8192, 0}, {0x7f95f0aac030, 0x7f95f0aac030, 8192, 0}, {0x7f95f0e62ab8, 0x7f95f0e62ab8, 8192, 0}, {0x7f95f093ba38, 0x7f95f093ba38, 8192, 0}, {0x7f95f0723cf0, 0x7f95f0723cf0, 8192, 0}, {0x7f95f096de20, 0x7f95f096de20, 8192, 0}}, {600, 0}) = 96
87014 0.000321 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051459
87014 0.000459 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051460
87014 0.000060 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051460
87014 0.000036 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051460
87014 0.000029 semtimedop(26935363, {{25, -1, 0}}, 1, {2, 990000000}) = -1 EAGAIN (Resource temporarily unavailable)
87014 2.990020 getrusage(RUSAGE_SELF, {ru_utime={454, 151000}, ru_stime={151, 178000}, ...}) = 0
87014 0.000104 getrusage(RUSAGE_SELF, {ru_utime={454, 151000}, ru_stime={151, 178000}, ...}) = 0
87014 0.000076 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051759
87014 0.000092 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051759
87014 0.000057 times({tms_utime=45415, tms_stime=15117, tms_cutime=0, tms_cstime=0}) = 439051759
87014 0.001685 io_submit(140281973911552, 91, {{0x7f95f096de20, 0, 1, 0, 266}, {0x7f95f0723cf0, 0, 1, 0, 262}, {0x7f95f093ba38, 0, 1, 0, 262}, {0x7f95f0e62ab8, 0, 1, 0, 267}, {0x7f95f0aac030, 0, 1, 0, 256}, {0x7f95f0f071c8, 0, 1, 0, 263}, {0x7f95f0715740, 0, 1, 0, 266}, {0x7f95f09498e0, 0, 1, 0, 267}, {0x7f95f0daace8, 0, 1, 0, 269}, {0x7f95f0a48e70, 0, 1, 0, 258}, {0x7f95f0dc07c8, 0, 1, 0, 269}, {0x7f95f0e64228, 0, 1, 0, 268}, {0x7f95f0742470, 0, 1, 0, 256}, {0x7f95f0ef68f0, 0, 1, 0, 271}, {0x7f95f0a615d8, 0, 1, 0, 271}, {0x7f95f0a2be60, 0, 1, 0, 267}, {0x7f95f0a983c8, 0, 1, 0, 267}, {0x7f95f0941f00, 0, 1, 0, 268}, {0x7f95f0ea0c78, 0, 1, 0, 266}, {0x7f95f0efaf40, 0, 1, 0, 261}, {0x7f95f0e69430, 0, 1, 0, 264}, {0x7f95f0d809e8, 0, 1, 0, 267}, {0x7f95f095e808, 0, 1, 0, 270}, {0x7f95f0f10a20, 0, 1, 0, 258}, {0x7f95f0e84cd0, 0, 1, 0, 269}, {0x7f95f0d9c4e0, 0, 1, 0, 260}, {0x7f95f0720000, 0, 1, 0, 270}, {0x7f95f0ea2898, 0, 1, 0, 256}, {0x7f95f0a209e8, 0, 1, 0, 258}, {0x7f95f0a262f8, 0, 1, 0, 268}, {0x7f95f0d88f80, 0, 1, 0, 256}, {0x7f95f0f05f08, 0, 1, 0, 263}, {0x7f95f0d7f980, 0, 1, 0, 263}, {0x7f95f096ace8, 0, 1, 0, 258}, {0x7f95f0727c38, 0, 1, 0, 259}, {0x7f95f0964118, 0, 1, 0, 259}, {0x7f95f0a56ac0, 0, 1, 0, 259}, {0x7f95f0745350, 0, 1, 0, 258}, {0x7f95f0e934d8, 0, 1, 0, 271}, {0x7f95f0a53988, 0, 1, 0, 269}, {0x7f95f0973be0, 0, 1, 0, 269}, {0x7f95f07204b0, 0, 1, 0, 265}, {0x7f95f097a558, 0, 1, 0, 265}, {0x7f95f0dc0570, 0, 1, 0, 265}, {0x7f95f0740f58, 0, 1, 0, 270}, {0x7f95f074c3d0, 0, 1, 0, 270}, {0x7f95f0a202e0, 0, 1, 0, 260}, {0x7f95f09778d0, 0, 1, 0, 266}, {0x7f95f0d781f8, 0, 1, 0, 262}, {0x7f95f0e7ab18, 0, 1, 0, 267}, {0x7f95f0ede890, 0, 1, 0, 270}, {0x7f95f0a967a8, 0, 1, 0, 269}, {0x7f95f0e830b0, 0, 1, 0, 262}, {0x7f95f0d7caa0, 0, 1, 0, 267}, {0x7f95f0709e18, 0, 1, 0, 267}, {0x7f95f0edef98, 0, 1, 0, 260}, {0x7f95f0a2f8f8, 0, 1, 0, 267}, {0x7f95f07494f0, 0, 1, 0, 264}, {0x7f95f0d9c288, 0, 1, 0, 260}, {0x7f95f0aa3cf0, 0, 1, 0, 270}, {0x7f95f095c4e0, 0, 1, 0, 260}, {0x7f95f0f01fc0, 0, 1, 0, 260}, {0x7f95f0959ab0, 0, 1, 0, 258}, {0x7f95f0d7ae80, 0, 1, 0, 267}, {0x7f95f0a4ec30, 0, 1, 0, 266}, {0x7f95f0e5c5f0, 0, 1, 0, 258}, {0x7f95f0939968, 0, 1, 0, 258}, {0x7f95f0740850, 0, 1, 0, 258}, {0x7f95f0743730, 0, 1, 0, 258}, {0x7f95f0aa2c88, 0, 1, 0, 260}, {0x7f95f0e5c398, 0, 1, 0, 260}, {0x7f95f097bf20, 0, 1, 0, 259}, {0x7f95f0a8a070, 0, 1, 0, 259}, {0x7f95f094adf8, 0, 1, 0, 259}, {0x7f95f0957e90, 0, 1, 0, 259}, {0x7f95f095a668, 0, 1, 0, 259}, {0x7f95f074b5c0, 0, 1, 0, 259}, {0x7f95f0e780e8, 0, 1, 0, 259}, {0x7f95f0a43560, 0, 1, 0, 259}, {0x7f95f071cec8, 0, 1, 0, 267}, {0x7f95f0d86550, 0, 1, 0, 267}, {0x7f95f0effa40, 0, 1, 0, 264}, {0x7f95f0747420, 0, 1, 0, 261}, {0x7f95f0719430, 0, 1, 0, 269}, {0x7f95f0a5d438, 0, 1, 0, 261}, {0x7f95f071b2a8, 0, 1, 0, 264}, {0x7f95f09734d8, 0, 1, 0, 264}, {0x7f95f0e6a240, 0, 1, 0, 260}, {0x7f95f0982d48, 0, 1, 0, 260}, {0x7f95f0a42e58, 0, 1, 0, 260}, {0x7f95f0a5f2b0, 0, 1, 0, 261}}) = 91
87014 0.005871 io_getevents(140281973911552, 14, 128, {{0x7f95f096de20, 0x7f95f096de20, 8192, 0}, {0x7f95f0723cf0, 0x7f95f0723cf0, 8192, 0}, {0x7f95f093ba38, 0x7f95f093ba38, 8192, 0}, {0x7f95f0e62ab8, 0x7f95f0e62ab8, 8192, 0}, {0x7f95f0aac030, 0x7f95f0aac030, 8192, 0}, {0x7f95f0f071c8, 0x7f95f0f071c8, 8192, 0}, {0x7f95f0715740, 0x7f95f0715740, 8192, 0}, {0x7f95f09498e0, 0x7f95f09498e0, 8192, 0}, {0x7f95f0daace8, 0x7f95f0daace8, 8192, 0}, {0x7f95f0a48e70, 0x7f95f0a48e70, 8192, 0}, {0x7f95f0dc07c8, 0x7f95f0dc07c8, 8192, 0}, {0x7f95f0e64228, 0x7f95f0e64228, 8192, 0}, {0x7f95f0742470, 0x7f95f0742470, 8192, 0}, {0x7f95f0ef68f0, 0x7f95f0ef68f0, 8192, 0}, {0x7f95f0a615d8, 0x7f95f0a615d8, 8192, 0}, {0x7f95f0a2be60, 0x7f95f0a2be60, 8192, 0}, {0x7f95f0a983c8, 0x7f95f0a983c8, 8192, 0}, {0x7f95f0941f00, 0x7f95f0941f00, 8192, 0}, {0x7f95f0ea0c78, 0x7f95f0ea0c78, 8192, 0}, {0x7f95f0efaf40, 0x7f95f0efaf40, 8192, 0}, {0x7f95f0e69430, 0x7f95f0e69430, 8192, 0}, {0x7f95f0d809e8, 0x7f95f0d809e8, 8192, 0}, {0x7f95f095e808, 0x7f95f095e808, 8192, 0}, {0x7f95f0f10a20, 0x7f95f0f10a20, 8192, 0}, {0x7f95f0e84cd0, 0x7f95f0e84cd0, 8192, 0}, {0x7f95f0d9c4e0, 0x7f95f0d9c4e0, 8192, 0}, {0x7f95f0720000, 0x7f95f0720000, 8192, 0}, {0x7f95f0ea2898, 0x7f95f0ea2898, 8192, 0}, {0x7f95f0a209e8, 0x7f95f0a209e8, 8192, 0}, {0x7f95f0a262f8, 0x7f95f0a262f8, 8192, 0}, {0x7f95f0d88f80, 0x7f95f0d88f80, 8192, 0}, {0x7f95f0f05f08, 0x7f95f0f05f08, 8192, 0}, {0x7f95f0d7f980, 0x7f95f0d7f980, 8192, 0}, {0x7f95f096ace8, 0x7f95f096ace8, 8192, 0}, {0x7f95f0727c38, 0x7f95f0727c38, 8192, 0}, {0x7f95f0964118, 0x7f95f0964118, 8192, 0}, {0x7f95f0a56ac0, 0x7f95f0a56ac0, 8192, 0}, {0x7f95f0745350, 0x7f95f0745350, 8192, 0}, {0x7f95f0e934d8, 0x7f95f0e934d8, 8192, 0}, {0x7f95f0973be0, 0x7f95f0973be0, 8192, 0}, {0x7f95f0a53988, 0x7f95f0a53988, 49152, 0}, {0x7f95f07204b0, 0x7f95f07204b0, 32768, 0}, {0x7f95f0dc0570, 0x7f95f0dc0570, 8192, 0}, {0x7f95f097a558, 0x7f95f097a558, 24576, 0}, {0x7f95f0740f58, 0x7f95f0740f58, 8192, 0}, {0x7f95f0a202e0, 0x7f95f0a202e0, 8192, 0}, {0x7f95f074c3d0, 0x7f95f074c3d0, 24576, 0}, {0x7f95f09778d0, 0x7f95f09778d0, 8192, 0}, {0x7f95f0d781f8, 0x7f95f0d781f8, 8192, 0}, {0x7f95f0e7ab18, 0x7f95f0e7ab18, 8192, 0}, {0x7f95f0ede890, 0x7f95f0ede890, 8192, 0}, {0x7f95f0a967a8, 0x7f95f0a967a8, 8192, 0}, {0x7f95f0e830b0, 0x7f95f0e830b0, 8192, 0}, {0x7f95f0d7caa0, 0x7f95f0d7caa0, 8192, 0}, {0x7f95f0709e18, 0x7f95f0709e18, 8192, 0}, {0x7f95f0edef98, 0x7f95f0edef98, 8192, 0}, {0x7f95f0a2f8f8, 0x7f95f0a2f8f8, 8192, 0}, {0x7f95f07494f0, 0x7f95f07494f0, 8192, 0}, {0x7f95f0d9c288, 0x7f95f0d9c288, 8192, 0}, {0x7f95f0aa3cf0, 0x7f95f0aa3cf0, 8192, 0}, {0x7f95f095c4e0, 0x7f95f095c4e0, 8192, 0}, {0x7f95f0f01fc0, 0x7f95f0f01fc0, 8192, 0}, {0x7f95f0959ab0, 0x7f95f0959ab0, 8192, 0}, {0x7f95f0d7ae80, 0x7f95f0d7ae80, 8192, 0}, {0x7f95f0a4ec30, 0x7f95f0a4ec30, 8192, 0}, {0x7f95f0e5c5f0, 0x7f95f0e5c5f0, 8192, 0}, {0x7f95f0939968, 0x7f95f0939968, 8192, 0}, {0x7f95f0740850, 0x7f95f0740850, 8192, 0}, {0x7f95f0743730, 0x7f95f0743730, 8192, 0}, {0x7f95f0e5c398, 0x7f95f0e5c398, 16384, 0}, {0x7f95f0aa2c88, 0x7f95f0aa2c88, 32768, 0}, {0x7f95f097bf20, 0x7f95f097bf20, 16384, 0}, {0x7f95f0a8a070, 0x7f95f0a8a070, 8192, 0}, {0x7f95f094adf8, 0x7f95f094adf8, 24576, 0}, {0x7f95f0957e90, 0x7f95f0957e90, 8192, 0}, {0x7f95f074b5c0, 0x7f95f074b5c0, 8192, 0}, {0x7f95f095a668, 0x7f95f095a668, 16384, 0}, {0x7f95f0e780e8, 0x7f95f0e780e8, 8192, 0}, {0x7f95f0a43560, 0x7f95f0a43560, 16384, 0}, {0x7f95f071cec8, 0x7f95f071cec8, 8192, 0}, {0x7f95f0d86550, 0x7f95f0d86550, 8192, 0}, {0x7f95f0effa40, 0x7f95f0effa40, 8192, 0}, {0x7f95f0747420, 0x7f95f0747420, 8192, 0}, {0x7f95f0719430, 0x7f95f0719430, 8192, 0}, {0x7f95f0a5d438, 0x7f95f0a5d438, 8192, 0}, {0x7f95f071b2a8, 0x7f95f071b2a8, 8192, 0}, {0x7f95f09734d8, 0x7f95f09734d8, 8192, 0}, {0x7f95f0e6a240, 0x7f95f0e6a240, 8192, 0}, {0x7f95f0982d48, 0x7f95f0982d48, 8192, 0}, {0x7f95f0a42e58, 0x7f95f0a42e58, 8192, 0}, {0x7f95f0a5f2b0, 0x7f95f0a5f2b0, 8192, 0}}, {600, 0}) = 91
87014 0.000308 times({tms_utime=45415, tms_stime=15118, tms_cutime=0, tms_cstime=0}) = 439051759
87014 0.000356 times({tms_utime=45415, tms_stime=15118, tms_cutime=0, tms_cstime=0}) = 439051759
87014 0.000103 times({tms_utime=45415, tms_stime=15118, tms_cutime=0, tms_cstime=0}) = 439051759
87014 0.000160 times({tms_utime=45415, tms_stime=15118, tms_cutime=0, tms_cstime=0}) = 439051759
再看ASM实例的dbw进程，也是用了AIO：
[root@Lunardb1 ~]# strace -fr -o /tmp/asm-dbw0-85795.log -p 85795
Process 85795 attached - interrupt to quit
^CProcess 85795 detached
[root@Lunardb1 ~]#
[root@Lunardb1 ~]# cat /tmp/asm-dbw0-85795.log|grep io_submit
85795 0.000088 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
85795 0.000043 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
85795 0.000039 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
85795 0.000062 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 270}}) = 1
85795 0.000038 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
85795 0.000060 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
85795 0.000048 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 270}}) = 1
85795 0.000145 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
85795 0.000053 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 270}}) = 1
85795 0.000095 io_submit(140349435969536, 2, {{0x7fa5a63589b8, 0, 1, 0, 270}, {0x7fa5a6357248, 0, 1, 0, 257}}) = 2
85795 0.000042 io_submit(140349435969536, 1, {{0x7fa5a63589b8, 0, 1, 0, 257}}) = 1
[root@Lunardb1 ~]#
