Serving a Demanding Client While Short on Resources

stolle@zib.de
http://www.zib.de

ABSTRACT:

• DMF on UNICOS, UNICOS/MK and IRIX is well suited to handle a local file system but it gets rather poor in a distributed environment with files larger than the DMF controlled file system. Using a small DMF controlled file system a `file system full` error is not supposed to occur due to poor migration performance or too large external files.

  dmscp-agent is a tool supporting the transfer of data streams to/from a file server with DMF related features.
  

• A DMF controlled file system with a long history shows problems concerning the fast and safe restoration of data located on damaged tapes. dmtaperepair is a tool usable to efficiently restore bitfiles of a damaged tape in the same MSP.

KEYWORDS:

Utilities, Data Migration, Systems

Introduction

The 'Konrad Zuse Zentrum fuer Informationstechnik Berlin' (ZIB - URL: http://www.zib.de) uses the CRAY Data Migration Facility (DMF) since 1991 first on a CRAY-X/M. Currently version 2.5 on CRAY-J90, T3D (Y/MP), T3E and version 2.6 on a SGI Origin 200 are used.

Although DMF is a very reliable and easy to handle HSM (Hierarchical Storage Management) software some features are missing.

This paper is going to show solutions to some problems ZIB ran into.

An infinite file system is not supposed to run out of space

Writing a file larger than the DMF controlled file system will fail due to an 'ENOSPC' error. DMF has no means to migrate parts of a file, the file system has at least to be larger than the largest file to be expected.

There are more reasons for a DMF controlled file system to run out of space.

• Busy tape drives may lead to a bottleneck in the migration process. So disc space may not be available soon enough while a process writes to the DMF file system. Usually the writing process will handle the 'ENOSPC' error with an error exit.
• The data stream written to a DMF controlled file system not seldom is initiated by a process on a remote client. Another reason for a lack of disc space is the inadequate triggering for migration (or release of disc space). More sophisticated means to cause the migration of a file can prevent this kind of situations.

If the data to be stored to the DMF controlled file system (or to be retrieved) is accessed sequentially dmscp-agent is a solution to the problems shown above.

dmscp-agent

• reads a data stream from any kind of UNIX file (stdin, named pipe, ordinary file ..) and transmits the data to a (remote) DMF server (or any remote file system). The remote dmscp-agent receives the data stream and writes it to a remote object.

  A remote object can be

  • an ordinary file or
  • a set of ordinary files (fragments).
  In the second case the source is split into fragments of equal size.
  Exceptions are the first fragment (stub) and the last fragment (rest of the data stream).
  The size of the stub can be defined separately from the size of the remaining fragments.
  This is used to prevent the first part of the data stream from migration.

• retrieves a data stream to any kind of UNIX file. The remote dmscp-agent is asked to read and transmit the remote object. The local dmscp-agent writes the data stream to the file (stdout, named pipe, ordinary file ..).

In both cases the amount of the data transmitted is compared with the size of the remote object (success check) by the local dmscp-agent.

The data can be encrypted prior to transmission.

When using standard UNIX means the user is confronted with the following disadvantages:

• rcp, scp, ftp: The local file has to be an ordinary file (stdin/stdout is not supported),
• rsh, ssh: The success of the transmission is not checked.

Fragmentation, ENOSPC handling and DMF related functions (see below) are not supported by standard UNIX means.

If the remote object is located on a DMF controlled file system the following additional features of dmscp-agent apply:

• The user can demand the migration (with or without freeing the disc space) immediately after each fragment is closed or after the last fragment is closed.
• A timeout for opening a migrated file can be defined (default 3600 sec). The same timeout interval applies for transmission of a buffer (8 MB).
• If an ENOSPC error occurs the write operation is tried repeatedly using the same timeout interval.

dmscp-agent can be used in local mode if the client process is on the same host as the remote object and the UNIX access permissions allow the access to the remote object.

In this case the local dmscp-agent and the remote dmscp-agent are the same process.

In the standard (remote) mode dmscp-agent uses the secure shell (ssh) to establish the connection to the remote server (It calls the remote dmscp-agent by ssh).
So it is necessary that:

• ssh is in the local search path,
• ssh can establish a connection to the remote server (by sshd or rshd),
• dmscp-agent is in the search path on the remote server.

In the standard (plain) transmission mode the remote dmscp-agent searches an (additional) free port that is used for the transmission.

In the encrypted transmission mode the ssh transmission is used.

The server/client code is symmetric (The same program is used as server or client process).

The source is one C file of 2000 lines of code.

dmscp-agent doesn't require any privileges. Just compile it and store it in any directory of your search path.

Makefiles for CRAY (UNICOS, UNICOS/mk), IRIX, IRIX64 and Solaris are provided. The port to other UNIX platforms should be done without major complications.

You can find the source code, the man page and the Makefiles on the www server at ZIB (http://www.zib.de/rz/backup-service/CUG/dmscp)

Examples:

tar -cf - . \
| dmscp-agent -w -s servername -u login -d /coffin \ 

-f mytar.tar -F 2g -frelease

• stores the tar file on login@servername:/coffin/mytar.tar_[STUBB|FRAG#]
• each fragment is migrated immediately after close (dmput -r)

dmscp-agent -r -s servername -u login -d /coffin \ 

-f mytar.tar -F 1 -frelease \

|tar -xf -

• retrieves this tar image, releases (frees the disc space of) each fragment immediately after close.

Tapes get damaged

Quite a lot of situations in a living HSM with a tape robot system can cause tape damage.

The question is not whether tapes get damaged (they get!!!) but how to handle damaged tapes without losing data.

First of all: one MSP (media specific process) per migrated file is not enough.

At the ZIB each migrated file is written to two MSPs. So in case of a tape damage at least one file (usually thousands) exist only in one copy.

Immediate action is required, the next tape error can lead to data loss! The task is to get a second copy of each file involved as soon as possible.

The standard DMF means don't support the solution of this problem sufficiently.

dmtaperepair is a (perl5) script that provides a (second) copy of each file located on the damaged tape in the same MSP while (logically) clearing the damaged tape.

dmtaperepair is

• easy to handle - the only parameter is the VSN (volume serial number) of the damaged tape,
• reliable - even a reboot of the DMF server while running dmtaperepair can be handled by only restarting with the same command.
• portable - only perl5 is required, the same script can be used for DMF versions 2.4-2.6 (UNICOS and IRIX).

How does dmtaperepair work?

• First it makes a list of the handles/bitfile-ids of the valid (not soft deleted) files on the damaged tape,
• Then it finds out the tape-VSN/chunk# of these bitfiles on the other MSPs and puts them into this list.
• After sorting the list by MSP_VSN_chunk# worklists for (logically) deleting these bitfiles from the damaged tape and for copying them to the MSP containing the damaged tape (including restart files) are provided using the DMF tool dmselect.
• These worklists are the input data of the DMF tool dmmove in two generated Korn shell scripts. By running these scripts the bitfiles are deleted from the damaged tape resp. copied to the MSP containing the damaged tape.
• Soft deleted files are erased by the DMF tool dmcatadm.
• After clearing the damaged tape and making the new second copies the success of these operations is checked.

You can find the perl scriptdmtaperepair and a README file on the www server of ZIB (http://www.zib.de/rz/backup-service/CUG/dmtaperepair)

For more information see the README.dmtaperepair file provided and the comments in the perl code.

dmtaperepair handles 2 copies of each bitfile on any number (>1) of MSPs. On an installation using more than 2 copies per bitfile dmtaperepair will cause the loss of the additional copies!

Although dmtaperepair is very easy to operate it has to be installed very carefully. (See README) An improper installation can cause serious damage.

Make sure that no locking flag is set on any tape of the MSP's not containing the damaged tape! Otherwise dmtaperepair may be unable to copy bitfiles and so the task my be fulfilled incompletely.

Repairing a Redwood tape containing 67 GB in 2148 valid (not soft deleted) files with 29 tapes involved on a CRAY-J90 (DMF 2.5) took 6 hours including 30 minutes preparation and final check.

dmtaperepair doesn't process files from DMF versions prior to 2.2 properly.

These files may have different file handles on different MSPs (looking like 'aaalpyw').
In order to get rid of these old style filehandles you have to copy each of these files to another inode.
A sufficient effective method to do so is described in http://www.zib.de/rz/backup-service/CUG/clnup_DMF_fs

perl scripts to support this process are provided.

The cleanup of a file system containing 62,000 files with old style handles out of 140,000 migrated files on a CRAY-J90 took 33 days.

Table of Contents | Author Index | CUG Home Page | Home