天天看點

ORACLE RAC叢集 RMAN自動備份controlfile失敗之ORA-00245&ORA-27037

    近期oracle資料庫巡檢中,廣州汕頭一RAC叢集有告警日志顯示RMAN自動備份控制檔案失敗,原因是控制檔案自動備份打開并且備份目錄在本地,涉及的備份目錄隻在節點2上有,而在節點1上沒有,導緻控制檔案自動備份失敗!

    作業系統:RHEL6.7

    資料庫版本:11.2.0.4

    告警日志資訊:

    節點1:

Fri Jan 06 20:04:06 2017

Control file backup creation failed:

  failure to open backup target file /u01/app/oracle/product/11.2.0/dbs/snapcf_orcl2.f.

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ckpt_35202.trc:

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

[oracle@rac01 trace]$ more  /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ckpt_35202.trc

Trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ckpt_35202.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,

Data Mining and Real Application Testing options

ORACLE_HOME = /u01/app/oracle/product/11.2.0

System name:    Linux

Node name:      rac01

Release:        4.1.12-37.4.1.el6uek.x86_64

Version:        #2 SMP Tue May 17 07:23:38 PDT 2016

Machine:        x86_64

Instance name: orcl1

Redo thread mounted by this instance: 1

Oracle process number: 21

Unix process pid: 35202, image: oracle@rac01 (CKPT)

*** 2016-09-21 22:04:41.855

*** SESSION ID:(153.1) 2016-09-21 22:04:41.855

*** CLIENT ID:() 2016-09-21 22:04:41.855

*** SERVICE NAME:(SYS$BACKGROUND) 2016-09-21 22:04:41.855

*** MODULE NAME:() 2016-09-21 22:04:41.855

*** ACTION NAME:() 2016-09-21 22:04:41.855

Redo thread 2 internally enabled

Control file enqueue hold time tracking dump at time: 5304

    節點2:

[oracle@rac02 trace]$ 

Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_63100.trc:

ORA-00245: control file backup failed; target is likely on a local file system

[oracle@rac02 trace]$ more  /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_63100.trc

Trace file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_ora_63100.trc

Node name:      rac02

Instance name: orcl2

Redo thread mounted by this instance: 2

Oracle process number: 67

Unix process pid: 63100, image: oracle@rac02 (TNS V1-V3)

*** 2017-01-06 20:04:06.406

*** SESSION ID:(288.7239) 2017-01-06 20:04:06.406

*** CLIENT ID:() 2017-01-06 20:04:06.406

*** SERVICE NAME:(SYS$USERS) 2017-01-06 20:04:06.406

*** MODULE NAME:(backup archivelog) 2017-01-06 20:04:06.406

*** ACTION NAME:(0000096 STARTED111) 2017-01-06 20:04:06.406

Starting control autobackup

Got error: 245

Autobackup failed with following error

    檢視叢集控制檔案位置(叢集控制檔案應該在共享存儲,如果不在共享存儲可能備份控制檔案會失敗)

[oracle@rac02 trace]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Mon Jan 9 14:10:24 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:

SQL> select * from v$controlfile;

STATUS  NAME                                                                             IS_ BLOCK_SIZE FILE_SIZE_BLKS

-------     --------------------------------------------------------------------------------      --- ---------- --------------

          +DATA01/orcl/controlfile/current.260.923176917                         NO       16384           3384

SQL> quit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

[oracle@rac02 trace]$

    在1節點嘗試備份控制檔案

[oracle@rac01 trace]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Mon Jan 9 14:14:57 2017

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1450957652)

RMAN> backup current controlfile;

Starting backup at 2017-01-09 14:15:12

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=462 instance=orcl1 device type=DISK

channel ORA_DISK_1: starting full datafile backup set

channel ORA_DISK_1: specifying datafile(s) in backup set

including current control file in backup set

channel ORA_DISK_1: starting piece 1 at 2017-01-09 14:15:14

channel ORA_DISK_1: finished piece 1 at 2017-01-09 14:15:15

piece handle=/u01/app/oracle/product/11.2.0/dbs/i5rpjijh_1_1 tag=TAG20170109T141513 comment=NONE

channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01

Finished backup at 2017-01-09 14:15:15

Starting Control File and SPFILE Autobackup at 2017-01-09 14:15:15

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-03009: failure of Control File and SPFILE Autobackup command on ORA_DISK_1 channel at 01/09/2017 14:15:16

ORA-19504: failed to create file "/u01/backup/data/c-1450957652-20170109-01"

ORA-27040: file create error, unable to create file

Additional information: 1

    檢視rman備份設定(看到的備份設定是備份叢集控制檔案到本地)

[oracle@rac02 trace]$ rman target /

RMAN> show all;

RMAN configuration parameters for database with db_unique_name ORCL are:

CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 7 DAYS;

CONFIGURE BACKUP OPTIMIZATION OFF; # default

CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default

CONFIGURE CONTROLFILE AUTOBACKUP ON;

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/u01/backup/data/%F';

CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default

CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default

CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default

CONFIGURE MAXSETSIZE TO UNLIMITED; # default

CONFIGURE ENCRYPTION FOR DATABASE OFF; # default

CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default

CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default

CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default

CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/11.2.0/db_1/dbs/snapcf_orcl.f'; # default

RMAN> 

    檢視節點1的備份目錄是否存在(果然,節點2存放的備份,備份目錄一定存在,而節點1上沒有建立備份目錄):

[oracle@rac01 ~]$ cd /u01/backup/data

-bash: cd: /u01/backup/data: No such file or directory

[oracle@rac01 ~]$

    節點1上建立備份目錄并賦予權限:

[root@rac01 u01]# mkdir -p backup/data

[root@rac01 u01]# ls -l

total 36

drwxr-xr-x. 6 root   oinstall  4096 Sep 21 20:00 app

drwxr-xr-x  3 root   root      4096 Jan  9 14:18 backup

drwxrwxr-x  7 oracle oinstall  4096 Sep 21 22:00 database

drwxrwxrwx  2 oracle oinstall  4096 Sep 22 02:52 dump

drwxrwxr-x  7 oracle oinstall  4096 Aug 27  2013 grid

drwxrwxr-x. 2 oracle oinstall 16384 Sep 21 18:53 lost+found

[root@rac01 u01]# chown -R oracle:oinstall backup

[root@rac01 u01]# chmod -R 777 backup

drwxrwxrwx  3 oracle oinstall  4096 Jan  9 14:18 backup

[root@rac01 u01]#

    在1節點重新嘗試備份控制檔案(成功了):

Recovery Manager: Release 11.2.0.4.0 - Production on Mon Jan 9 14:22:50 2017

RMAN>  backup current controlfile;

Starting backup at 2017-01-09 14:19:43

using channel ORA_DISK_1

channel ORA_DISK_1: starting piece 1 at 2017-01-09 14:19:44

channel ORA_DISK_1: finished piece 1 at 2017-01-09 14:19:45

piece handle=/u01/app/oracle/product/11.2.0/dbs/i7rpjirv_1_1 tag=TAG20170109T141943 comment=NONE

Finished backup at 2017-01-09 14:19:45

Starting Control File and SPFILE Autobackup at 2017-01-09 14:19:45

piece handle=/u01/backup/data/c-1450957652-20170109-02 comment=NONE

Finished Control File and SPFILE Autobackup at 2017-01-09 14:19:46

RMAN> list backup of controlfile;

List of Backup Sets

===================

BS Key  Type LV Size       Device Type Elapsed Time Completion Time    

------- ---- -- ---------- ----------- ------------ -------------------

1501    Full    52.95M     DISK        00:00:01     2017-01-01 20:01:15

        BP Key: 1501   Status: AVAILABLE  Compressed: NO  Tag: TAG20170101T200114

        Piece Name: /u01/backup/data/c-1450957652-20170101-00

  Control File Included: Ckp SCN: 143516927    Ckp time: 2017-01-01 20:01:14

1506    Full    52.95M     DISK        00:00:01     2017-01-02 20:01:11

        BP Key: 1506   Status: AVAILABLE  Compressed: NO  Tag: TAG20170102T200110

        Piece Name: /u01/backup/data/c-1450957652-20170102-00

  Control File Included: Ckp SCN: 144147774    Ckp time: 2017-01-02 20:01:10

1555    Full    52.95M     DISK        00:00:00     2017-01-03 20:20:26

        BP Key: 1555   Status: AVAILABLE  Compressed: NO  Tag: TAG20170103T202026

        Piece Name: /u01/backup/data/c-1450957652-20170103-00

  Control File Included: Ckp SCN: 155236810    Ckp time: 2017-01-03 20:20:26

1566    Full    52.95M     DISK        00:00:01     2017-01-05 20:03:16

        BP Key: 1566   Status: AVAILABLE  Compressed: NO  Tag: TAG20170105T200315

        Piece Name: /u01/backup/data/c-1450957652-20170105-00

  Control File Included: Ckp SCN: 157603651    Ckp time: 2017-01-05 20:03:15

1586    Full    52.95M     DISK        00:00:01     2017-01-07 01:29:12

        BP Key: 1586   Status: AVAILABLE  Compressed: NO  Tag: TAG20170107T012911

        Piece Name: /u01/backup/data/c-1450957652-20170107-00

  Control File Included: Ckp SCN: 161699766    Ckp time: 2017-01-07 01:29:11

1591    Full    52.95M     DISK        00:00:01     2017-01-07 20:01:11

        BP Key: 1591   Status: AVAILABLE  Compressed: NO  Tag: TAG20170107T200110

        Piece Name: /u01/backup/data/c-1450957652-20170107-01

  Control File Included: Ckp SCN: 162284897    Ckp time: 2017-01-07 20:01:10

1596    Full    52.95M     DISK        00:00:01     2017-01-08 20:01:21

        BP Key: 1596   Status: AVAILABLE  Compressed: NO  Tag: TAG20170108T200120

        Piece Name: /u01/backup/data/c-1450957652-20170108-00

  Control File Included: Ckp SCN: 162881404    Ckp time: 2017-01-08 20:01:20

1597    Full    52.92M     DISK        00:00:01     2017-01-09 10:52:12

        BP Key: 1597   Status: AVAILABLE  Compressed: NO  Tag: TAG20170109T105211

        Piece Name: /u01/app/oracle/product/11.2.0/dbs/i3rpj6mr_1_1

  Control File Included: Ckp SCN: 163348183    Ckp time: 2017-01-09 10:52:11

1598    Full    52.95M     DISK        00:00:00     2017-01-09 10:52:13

        BP Key: 1598   Status: AVAILABLE  Compressed: NO  Tag: TAG20170109T105213

        Piece Name: /u01/backup/data/c-1450957652-20170109-00

  Control File Included: Ckp SCN: 163348190    Ckp time: 2017-01-09 10:52:13

1599    Full    52.92M     DISK        00:00:01     2017-01-09 14:15:14

        BP Key: 1599   Status: AVAILABLE  Compressed: NO  Tag: TAG20170109T141513

        Piece Name: /u01/app/oracle/product/11.2.0/dbs/i5rpjijh_1_1

  Control File Included: Ckp SCN: 163446650    Ckp time: 2017-01-09 14:15:13

1600    Full    52.92M     DISK        00:00:01     2017-01-09 14:19:44

        BP Key: 1600   Status: AVAILABLE  Compressed: NO  Tag: TAG20170109T141943

        Piece Name: /u01/app/oracle/product/11.2.0/dbs/i7rpjirv_1_1

  Control File Included: Ckp SCN: 163447041    Ckp time: 2017-01-09 14:19:43

1601    Full    52.95M     DISK        00:00:00     2017-01-09 14:19:45

        BP Key: 1601   Status: AVAILABLE  Compressed: NO  Tag: TAG20170109T141945

        Piece Name: /u01/backup/data/c-1450957652-20170109-02

  Control File Included: Ckp SCN: 163447048    Ckp time: 2017-01-09 14:19:45

RMAN> quit

Recovery Manager complete.

[oracle@rac01 trace]$