Skip to content

block corruption – Indepth article for the experienced dba





PURPOSE

This article discusses block corruptions in Oracle and how they are related

to the underlying operating system and hardware. To better illustrate the

discussion, Unix is taken as the operating system of reference.

SCOPE & APPLICATION

For users requiring further understanding as to how a block could become

corrupted.

Block corruption has been a common occurrence on most UNIX based systems and

relational databases for many years. It is one of the most frequent ways to

lose data and cause serious business impact. Through a survey of literary

technical sources, this document will discuss several ways that block

corruptions can occur, provide conclusions and possible solutions.

To fully comprehend all the reasons for block corruption’s, it is necessary to

understand how I/O device subsystems work, how memory buffers are used to

support the reading and writing of data blocks, how blocks are sized on both

UNIX and Oracle, and how these three objects work together to maintain data

consistency.

I/O devices are designed specifically for host machines and there have been

few attempts to standardize a particular interface across the industry. Most

software, including Oracle, on UNIX machines uses standard C program calls that

in turn perform system calls to support the reading and writing of data to

disk. These system calls access I/O device software that retrieves or writes

data on disk.

The UNIX system contains two types of devices, block devices and raw or

character devices. Block devices look like random access storage devices to

the rest of the system while character devices include all other devices such

as terminals and network media. (Bach, 1990 314). These device types are

important to understand because different combinations can increase corruptions.

Device drivers are configured by the operating system and the configuration

procedure generates or populates tables that form part of the code of the

kernel. This kernel to device driver interface is described by the block

device switch table and the character device switch table. Each device type

has entries in these tables that direct the kernel to the appropriate driver

interfaces for the system calls. The open and close system calls of a device

file funnel through the two device switch tables, according to file type. The

mount and umount system calls also invoke the device open and close procedures

for block devices. Read and write system calls of character special files pass

through the respective procedures in the character device switch tables. Read

and write system calls of block devices and of files on mounted file systems

invokes the algorithms of the buffer cache, which invoke the device strategy

procedure. (Bach, 1990 314). This buffer cache plays an important role in

block corruptions since it is the location where data blocks are the most

vulnerable.

The difference between the two disk interfaces is whether they deal with the

buffer cache. When accessing the block device interface, the UNIX kernel

follows the same algorithm as for regular files, except that after converting

the logical byte offset into a logical block offset, it treats the logical

block offset as a physical block number in the file system. It then accesses

the data via the buffer cache and, ultimately, the driver strategy interface.

However, when accessing the disk via the raw interface, the kernel does not

convert the byte offset into the file but passes the offset immediately to the

driver. The driver’s read or write routine converts the byte offset to a

block offset and copies the data directly to the user address space, bypassing

kernel buffers.

Thus, if one process writes a block device and a second process then reads a

raw device at the same address, the second process may not read the data that

the first process had written, because the data may still be in the buffer

cache and not on disk. However, if the second process had read the block

device, it would automatically pick up the new data, as it exists in the

buffer cache. (Bach, 1990 328).

Use of the raw interface may also introduce strange behavior. If a process

reads or writes a raw device in units smaller than the block size, results are

driver-dependent. For instance, when issuing 1-byte writes to a tape drive,

each byte may appear in different tape blocks. (Bach 1990)

The advantage of using the raw interface is speed, assuming there is no

advantage to caching data for later access. Processes accessing block devices

transfer blocks of data whose size are constrained by the file system logical

block size. Furthermore, use of the block interface entails an extra copy of

data between user address space and kernel buffers, which is avoided in the

raw interface. For example, if a file system has a logical block size 1K

bytes, at most 1K bytes are transferred per I/O operation. However, processes

accessing the disk as a raw device can transfer many disk blocks during a disk

operation, subject to the capabilities of the disk controller.

Disk controllers are hardware devices that control the I/O actions of one or

more disks. These controllers can also create a bottleneck in a system.

(Corey, Abbey, Dechichio 1995). Controllers are the most frequent piece of

hardware to have and cause problems on many systems. When a system has

multiple disks controlled by one controller, the results can be fatal. The

bottleneck on controllers is a common cause of write error.

It is important to remember that Oracle and other products use these device

access methods to perform their work. It is also important to note the added

complexity that the Oracle kernel adds to the I/O game.

The Oracle Relational Database Management System (RDBMS) keeps its

information, including data, in block format. However, the Oracle data block

can be, and in most cases is, composed of several operating system blocks.

An Oracle database block is the physical unit of storage in which all Oracle

database data are stored in files. The Oracle database block size is

determined by setting a parameter called db_block_size when the

database is created. (Millsap, 1995).

The most common UNIX block is 512 bytes but the Oracle block size can range

from 512 to 32K. The difference in block sizing between the operating system

and the Oracle kernel are beneficial for Oracle; boosting performance gains

while allowing UNIX to maintain small files with minimal wasted space. The

Oracle block can be considered a superset of the UNIX file system block size.

Each block of an Oracle data file is formatted with a fixed header that

contains information about the particular block. This information provides a

means to ensure the integrity for each block and in turn, the entire Oracle

database. One component of the fixed header of a data block is called a Relative

Data Block Address (DBA). This DBA is a 4 bytes that stores the relative file

number of the Oracle database file and the Oracle block number offset relative

to the beginning of the file. (Presley, 1993).

Whenever there is a problem with the RDBA, Oracle may signal an Oracle error

ORA-1578: Data block corrupted in file # block #. This error provide information that point to where the

corruption exists.

Oracle uses the standard C system function calls to read and write blocks to

its database files. Once the block has been read it is mapped to shared

memory by the operating system, After the block has been read into shared

memory, the Oracle kernel does verification checks on the block to ensure the

integrity of the fixed header. The RDBA check is the first verification made

on the fixed header. So why do RDBAs become corrupt and how can we identify

and correct them?

Case One

——–

The first case of block corruption occurs when the block has been zeroed out. If the Oracle block

is completely zeroed out, sql statements may generate an ORA-8103 as the block is invalid

and it is not formatted as an empty block. In this case the dbverify utility (dbv) can detect it

and will produce an error message. Dbv output example:

DBVERIFY – Verification starting : FILE = /oradata/data_01.dbf

Page 307161 is marked corrupt

***

Corrupt block relative dba: 0x0644afd9 (file 0, block 307161)

Completely zero block found during dbv:

Usually the first operating system block of an Oracle block is zeroed out when

there was a software error on disk and the operating system attempted to repair

its block. In addition, disk repair utility programs have caused this zeroing out effect.

Programs that read from and write to the disk directly can destroy the

consistency of file system data. The file system algorithms coordinate disk

I/O operation to maintain a consistent view of disk data structures, including

linked lists of free disk blocks and pointer from inodes to direct and

indirect data blocks. Processes that access the disk directly bypass these if

they run while other file system activity is going on. For this reason, these

programs should not be run on an active file system. (Bach, 1990 328).

Case Two

——–

The RDBA in the physical block on disk is incorrect. It can generate an error ORA-1578

and a message in the alert.log with message "Data in bad block" as next:

***

Corrupt block relative dba: 0x56c07ac1 (file 347, block 31425)

Bad header found during buffer read

Data in bad block –

type: 6 format: 2 rdba: 0x06407ac1

last change scn: 0x0000.00a02808 seq: 0x1 flg: 0x02

consistency value in tail: 0x28080601

check value in block header: 0x0, block checksum disabled

spare1: 0x0, spare2: 0x0, spare3: 0x0

***

Reread of rdba: 0x56c07ac1 (file 347, block 31425) found same corrupted data

Corrupt block relative dba: 0x2310d545 (file 140, block 1103173)

Bad header found during backing up datafile

Data in bad block –

type: 0 format: 0 rdba: 0x00000002

last change scn: 0x0000.00000010 seq: 0x0 flg: 0x18

consistency value in tail: 0x26740602

check value in block header: 0x0, block checksum disabled

spare1: 0x0, spare2: 0x0, spare3: 0x20

***

Reread of blocknum=1103173, file=/gl/cdb1/oracle/cs/data1/cs_dm_ext.dbf. found same corrupt data

Blocks are sometimes written into the wrong places in the data file. This is

called "write blocks out of sequence." This typically happens when the operating system

I/O device driver fails to write the block in the proper location that Oracle

requested via the lseek() system call.

The lseek() system call is one of the most important calls related to block

corruption. The calculations that lseek() performs are often the cause of

block problems. To understand lseek() a brief discussion of byte positioning

is necessary.

Every open file has a "current byte position" associated with it. This is

measured as the number of bytes from the start of the file. The create system

call sets the file’s position to the beginning of the file, as does the open

system call. The read and write system calls update the file’s position by

the number of bytes read or written. Before a read or write, an open file can

be positioned using lseek(). The format is:

lseek(int fildes, long offset, int whence);

The offset and whence arguments are interpreted as follows: If whence is 0,

the file’s position is set to offset bytes from the beginning of the file. If

whence is 1, the file’s position is set to its current position plus the

offset. If whence is 2, the file’s position is set to the size of the file

plus the offset. The file’s offset can be greater than the file’s current

size, in which case the next write to the file will extend the file. Lseek()

returns a long integer byte offset of the file. (Stevens, 1990 40).

There is great opportunity for miscalculation of an offset based on the

lseek() system call. Though lseek is not the only system call culprit in the

block corruption problem, it is a major contributor.

This may also happen if the block was corrupted in memory but was written to disk.

This situation is quite rare and in most cases it is usually caused by memory

faults that go undetected. The RDBA found in the block is usually garbage and

not a valid RDBA.

If there is a possibility of memory problems on the system, the database

administrator can enable further sanity block checking by placing the

following parameters in the database instance init.ora parameter file:

db_block_checking=true

db_block_checksum=true

_db_block_cache_protect= true

db_block_checking force the Oracle RDBMS kernel to call functions that check

the block. Oracle checks a block by going through the data on the block, making

sure it is self-consistent. Block checking can often prevent memory and data corruption

db_block_checksum determines whether DBWn and the direct loader will calculate

a checksum (a number calculated from all the bytes stored in the block) and

store it in the cache header of every data block when writing it to disk.

Checksums are verified when a block is read only if this parameter is true and the

last write of the block stored a checksum.

The _db_block_cache_protect=true protects the cache layer from becoming corrupted.

This parameter will prevent certain corruption from getting to disk, although

it may crash the foreground of the database instance. It will help catch

stray writes in the cache. When a process tries to write past the buffer size

in the SGA, it will fail first with a stack violation.

If the database writer process detects a corrupted block in cache prior to

writing the block to disk, it will signal an error and will crash the

database instance. The block that is corrupted is never written to disk.

After receiving such an error, simply attempt to restart the database instance.

There is no doubt that this can be a costly workaround to avoid block

corruptions. However, the workaround once a corruption has occurred can be

even costlier.

Case 3

——

A third cause for block corruption is the requested I/O not being serviced by

the operating system. The calls that Oracle makes to lseek() and read() are checked for

return error codes. In addition, Oracle checks to see the number of bytes read in by the read()

system call to ensure that the block size or a multiple of the block size was

read. Since these checks appeared to have been successful, Oracle assumes

that the direct read succeeded. Upon sanity checking, the RDBA is incorrect

and the database operation request fails. Therefore, the I/O read request

really never took place. In this case, the RDBA found can point to a block of

a different file.

Case 4

——

Another reason for block corruption is reading the wrong block from the same

device. Typically, this is caused by a very busy disk. In some cases, the

block read was off by 1 block but can range into several hundreds of blocks.

Since this occurs when the disk is very busy and under lots of

stress, try spreading datafiles across multiple disks and ensure that the disk

drive can support the load.

In the third and fourth situations, the database files will not be physically

corrupted and the operation can be tried again with success. Most diagnostics

testing will not reveal anything wrong with either the operating system or the

hardware. However, the problem is due to operating system or hardware related

problems. (Velpuri, 1995).

So what causes the operating system calls to behave the way they do and how

can companies try to minimize their risk? To evaluate these questions,

another look into how UNIX works is required.

UNIX vendors, in a attempt to speed performance, have implemented many

features into the filesystem. The filesystem manages a large cache of I/O

buffers, called the buffer cache. This cache allows UNIX to optimize read and

write operations. When a program writes data, the filesystem stores the data

in a buffer rather that writing it to disk immediately. At some later point

in time, the system will send this data to the disk driver, together with

other data that has accumulated in the cache. In other words, the buffer

cache lets the disk driver schedule disk operations in batches. It can make

larger transfers and use techniques such as seek optimization to make disk

access more efficient. This is called write-behind.

When a program reads data, the system first checks the buffer cache to see if

the desired data is already there. If the data is already in the buffer

cache, the filesystem does not need to access the disk for those blocks. It

just gives the user the data it found in its buffer, eliminating the need to

wait for a disk drive. The filesystem only needs to read the disk if the data

isn’t already in the cache. To increase efficiency even further, the

filesystem assumes the program will read the file consecutively and read

several blocks from the disk at once. This increases the likelihood that the

data for future read operations will already be in the cache. (Loukides, M.,

1990) This also increases the chance of block corruption.

As a filesystem gets busy and buffers are being read, modified, written, and

aged out of the cache the chance of the kernel reading or writing the wrong

block increases. Also, the more complex the scheme to read from and write to

disk, the greater the likelihood of function failure.

The UNIX kernel uses the strategy interface to transmit data between the

buffer cache and a device, although the read and write procedures of character

devices sometime use their block counterpart strategy procedure to transfer

data directly between the device and the user address space. The strategy

procedure may queue I/O jobs for a device on a work list or do more

sophisticated processing to schedule I/O jobs. Drivers can set up data

transmission for one physical address or many, as appropriate. The UNIX

kernel passes a buffer header address to the driver strategy procedure. The

header contains a list of addresses and sizes for transmission of data to or

from the device. This is also how the swapping operations work. For the

buffer cache, the kernel transmits data from one address; when swapping, the

kernel transmits data from many data addresses. If data is being copied to or

from the user’s address space, the driver must lock the process in memory

until the I/O transfer is complete.

The kernel loses control over a buffer only when it waits for the completion

of I/O between the buffer and the disk. It is conceivable that a disk drive

is corrupt so that it cannot interrupt the CPU, preventing the kernel from

ever releasing the buffer. There are processes that monitor the hardware for

such cases and zero out the block and return an error to the kernel for a bad

disk job. (Bach, 1990 52).

On the UNIX level there are several utilities that will check for bad disk

blocks and zero out any blocks they find corrupted. These utilities do not

realize that the block in question may be an Oracle RDBMS block and zero out

the block by mistake.

In (Silberschatz, Galvin, 1994), the authors consider the possible effect of a

computer crash. In this case, the table of opened files is generally lost,

and with it any changes in the directories of opened files. This event can

leave the file system in an inconsistent structure. Frequently, a special

program is run at reboot time to check for and correct disk inconsistencies.

The consistency checker compares the data in the directory structure with the

data blocks on disk, and tries to fix and inconsistencies it finds.

(Silberschatz, Galvin, 1994) This will often result in the reformatting of

blocks which will cause the Oracle block information to be removed. This will

definitely cause Oracle corruption.

It is important to realize that monitoring of hardware is required for all

operating systems. Hardware monitors can sense electrical signals on the

busses and can accurately record them even at high speed. A hardware monitor

keeps observing the system even when it is malfunctioning, and thus, it can be

used to debug the system. (Jain, 1991 99) These tools can help determine the

cause of the problem and detect problems like controller error and media

faulting which are frequent corruption contributors.

In any case, there are many opportunities for blocks, either on disk or in the

buffer cache, to become corrupt. Fixing the corruption can sometimes provide

even greater opportunities.

Conclusion

———-

Data block corruption is an ongoing problem on all operating systems,

especially UNIX. There are many types and causes of corruptions to consider.

Advanced system configurations can increase the chance and hardware problems

are a common source of corruptions. When receiving block corruption errors,

remember that a couple of them are not physical corruptions but memory

corruptions that are never written to disk.

Oracle Customer Support provides a number of bulletins on block corruption

problems that help recover what is left of the data once corruption has

occurred. If block corruption occurs on a machine, be sure to identify the

type of corruption and establish a plan for its correction.

VN:F [1.9.22_1171]
Rating: 0.0/10 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Post a Comment

You must be logged in to post a comment.