最近帮助用户恢复了一个由于存储故障导致的数据库灾难，整个数据库容量4TB，浪潮的存储设备，控制器损坏导致数据库故障。

    从启动信息来看，首先出现故障的是控制文件：

    Read from controlfile member \'oradata/ORCL/control01.ctl\' has found a fractured block (blk# 25, seq# 1909664)

    Read from controlfile member \'oradata/ORCL/control02.ctl\' has found a fractured block (blk# 25, seq# 1909664)

    Read from controlfile member \'oradata/ORCL/control03.ctl\' has found a fractured block (blk# 25, seq# 1909664)

    我们可以用dbv来检查一下控制文件的损坏情况，控制文件块大小是16K，检查3个控制文件，都出现了不一致：

    dbv file=control01.ctl

    DBVERIFY: Release 10.2.0.1.0 - Production on Thu Dec 2 21:40:27 2010

    Copyright (c) 1982, 2005, Oracle.  All rights reserved.

    DBV-00103: Specified BLOCKSIZE (8192) differs from actual (16384)

    [oracle@oradbserver ORCL]$ dbv file=control01.ctl blocksize=16384

    DBVERIFY: Release 10.2.0.1.0 - Production on Thu Dec 2 21:40:44 2010

    Copyright (c) 1982, 2005, Oracle.  All rights reserved.

    DBVERIFY - Verification starting : FILE = control01.ctl

    Page 19 is influx - most likely media corrupt

    Corrupt block relative dba: 0x00000013 (file 0, block 19)

    Fractured block found during dbv:

    Data in bad block:

    type: 21 format: 2 rdba: 0x00000013

    last change scn: 0xffff.001d238a seq: 0x1 flg: 0x04

    spare1: 0x0 spare2: 0x0 spare3: 0x0

    consistency value in tail: 0x23791501

    check value in block header: 0x6ad5

    computed block checksum: 0xf3

    Page 25 is marked corrupt

    Corrupt block relative dba: 0x00000019 (file 0, block 25)

    Bad check value found during dbv:

    Data in bad block:

    type: 21 format: 2 rdba: 0x00000019

    last change scn: 0xffff.001d2388 seq: 0x1 flg: 0x04

    spare1: 0x0 spare2: 0x0 spare3: 0x0

    consistency value in tail: 0x23881501

    check value in block header: 0xbe7a

    computed block checksum: 0x52d0

    Page 27 is influx - most likely media corrupt

    Corrupt block relative dba: 0x0000001b (file 0, block 27)

    Fractured block found during dbv:

    Data in bad block:

    type: 21 format: 2 rdba: 0x0000001b

    last change scn: 0xffff.001d238b seq: 0x1 flg: 0x04

    spare1: 0x0 spare2: 0x0 spare3: 0x0

    consistency value in tail: 0x23501501

    check value in block header: 0x9663

    computed block checksum: 0x5fc4

    Page 316 is influx - most likely media corrupt

    Corrupt block relative dba: 0x0000013c (file 0, block 316)

    Fractured block found during dbv:

    Data in bad block:

    type: 21 format: 2 rdba: 0x0000013c

    last change scn: 0xffff.001d2393 seq: 0x1 flg: 0x04

    spare1: 0x0 spare2: 0x0 spare3: 0x0

    consistency value in tail: 0x23811501

    check value in block header: 0xc99d

    computed block checksum: 0x12

    DBVERIFY - Verification complete

    Total Pages Examined         : 662

    Total Pages Processed (Data) : 0

    Total Pages Failing   (Data) : 0

    Total Pages Processed (Index): 0

    Total Pages Failing   (Index): 0

    Total Pages Processed (Other): 319

    Total Pages Processed (Seg)  : 0

    Total Pages Failing   (Seg)  : 0

    Total Pages Empty            : 339

    Total Pages Marked Corrupt   : 4

    Total Pages Influx           : 3

    Highest block SCN            : 1909665 (65535.1909665)

    注意控制文件也是有一致性校验的。

    但是我仍然可以通过strings命令从损坏的控制文件中提取文件名，重建控制文件。

    剩下的是UNDO表空间损坏，这个不太可怕，强制来启动数据库，出现2662错误：

    Errors in file /ORCL/udump/orcl_ora_4792.trc:

    ORA-00600: internal error code, arguments: [2662], [1], [1574405318], [1], [1574541355], [8388617], [], []

    Errors in file /ORCL/udump/orcl_ora_4792.trc:

    ORA-00600: internal error code, arguments: [2662], [1], [1574405318], [1], [1574541355], [8388617], [], []

    这里我设置了_minimum_giga_scn参数来推进SCN

    _minimum_giga_scn=1047

    最后启动数据库后，遇到了以下错误：

    ORA-00604: error occurred at recursive SQL level 1

    ORA-08102: index key not found, obj# 239, file 1, block 1674 (2)

    ORA-12012: error on auto execute of job 1

    ORA-08102: index key not found, obj# 239, file 1, block 1674 (2)

    这是索引的一致性问题，找到这个对象：

    SQL> select object_name from dba_objects where object_id=239;

    OBJECT_NAME

    --------------------------------------------------------------------------------

    I_JOB_NEXT

    直接DROP掉了重建：

    SQL> drop index I_JOB_NEXT;

    Index dropped.

    SQL> create index i_job_next on job$ (next_date);

    Index created.

    错误消除，参考Metalink：1036858.6

建议继续学习：

1. 用 LD_PRELOAD 挽救被误删的 libc.so.6    (阅读：4163)
2. MySQL从压缩文件恢复数据    (阅读：3700)
3. 参数_smon_internal_errlimit与数据库恢复    (阅读：3196)
4. MySQL 备份和其恢复机制原理简述    (阅读：3134)
5. 用bin日志中恢复MySQL数据库    (阅读：2688)
6. 用CloneZilla制作紧急恢复分区    (阅读：2642)
7. Zmanda让MySQL的备份与恢复更加方便快捷灵活    (阅读：2331)
8. 当logfile被误删除后    (阅读：2190)
9. EXT文件系统误删除数据恢复指南    (阅读：2188)
10. Oracle数据库恢复：归档日志损坏案例一则    (阅读：1917)