Red Hat GLOBAL FILE SYSTEM 4.7 Betriebsanweisung Seite 1

HP StorageWorks Scalable File ShareClient Installation and User GuideVersion 2.2Product Version: HP StorageWorks Scalable File Share Version 2.2Publis

x

Troubleshooting7–4• The interconnect may not be functioning correctly.If all of the MDS and OST services associated with the file system are available

File system mounting issues 7–5To configure Lustre to use a different port on the client node when using a Myrinet interconnect, perform the following

Troubleshooting7–67.2.5 Troubleshooting stalled mount operationsIf a mount operation stalls, you can troubleshoot the problem on the HP SFS system. Re

Operational issues 7–7 obdidx objid objid group 0 1860 0x744 0

Troubleshooting7–8In a situation where only one or two client nodes have crashed and a lock is needed, there is a pause of 6 to 20 seconds while the c

Operational issues 7–9Preventing and correcting the problemYou can take action to prevent access to files hanging as described above; however, if you

Troubleshooting7–10In the following example, the output from a fully dual-connected configuration is shown; in this example delta6 is the client node:

A–1A Using the sfsconfig commandThe sfsconfig command is a tool that you can use to automatically perform the following tasks on client nodes:• Config

Using the sfsconfig commandA–2The sfsconfig command also adds the lquota setting (which is needed to allow the client node to use quotas functionality

A–3The target can be one or more of the following:conf Specifies that the /etc/modprobe.conf or /etc/modules.conf file is to be updated. The command c

1–11OverviewHP StorageWorks Scalable File Share Version 2.2 (based on Lustre® technology) is a product from HP that uses the Lustre File System (from

Using the sfsconfig commandA–4The following command adds the required options lnet setting to the /etc/modprobe.conf or /etc/modules.conf file; it als

B–1B Options for Lustre kernel modulesThis appendix is organized as follows:• Overview (Section B.1)• Setting the options lnet settings (Section B.2)•

Options for Lustre kernel modulesB–2B.1 OverviewTo support the functionality provided in HP SFS Version 2.2, the /etc/modprobe.conf and /etc/modprobe.

Setting the options lnet settings B–3B.2 Setting the options lnet settingsThe options lnet settings are critical in ensuring both connectivity and per

Options for Lustre kernel modulesB–4• Two (non-bonded) Gigabit Ethernet interconnects, which the client node uses to access two different HP SFS syste

Setting the options lnet settings B–5b. When you have identified an appropriate server for the test, enter the following command on that server, to id

Options for Lustre kernel modulesB–6B.3 Modifying the /etc/modprobe.conf file on Linux Version 2.6 client nodes manuallyTIP: You can restrict the Giga

Modifying the /etc/modules.conf file on Linux Version 2.4 client nodes manually B–74. To configure the options lnet settings on the client node, add a

Options for Lustre kernel modulesB–8

C–1C Building an HP SFS client kit manuallyThis appendix describes how to build an HP SFS client kit manually (that is, not using the sample script pr

Overview1–21.1 Overview of the Lustre file systemLustre is a design for a networked file system that is coherent, scalable, parallel, and targeted tow

Building an HP SFS client kit manuallyC–2C.1 OverviewThe build_SFS_client.sh example script provided on the HP StorageWorks Scalable File Share Client

Building the HP SFS client kit manually C–36. If you wish to build RPM files, it is best to create an rpmmacros file (if one does not already exist fo

Building an HP SFS client kit manuallyC–411. If your client has additional Lustre patches listed in the client_enabler/src/arch/distro/lustre_patches/

Building the HP SFS client kit manually C–5d. Add the additional required patches to the kernel.spec file in the same way that you applied the Lustre

Building an HP SFS client kit manuallyC–6To create a built kernel tree, perform the following steps:a. Extract the kernel sources from the src/ direct

Building the HP SFS client kit manually C–715. Build the interconnect driver trees.• If you are building the HP SFS client kit with support for a Volt

Building an HP SFS client kit manuallyC–8b. Generate the spec file, by entering the following commands:$ m4 -D_VERSION=2.2 -D_RELEASE=0 -D_LICENSE=com

Output from the SFS Client Enabler C–9C.3 Output from the SFS Client EnablerWhen you build an HP SFS client kit manually, the output directories for t

Building an HP SFS client kit manuallyC–10• hpls-diags-clientIf possible, use the version of the hpls-diags-client package that you built when you cre

Glossary–1Glossaryadministration server The ProLiant DL server that the administration service runs on. Usually the first server in the system.See als

Overview of HP SFS 1–3A typical Lustre file system consists of multiple Object Storage Servers that have storage attached to them. At present, the Obj

Glossary–2Internet address A unique 32-bit number that identifies a host’s connection to an Internet network. An Internet address is commonly represen

Glossary–3Object Storage Server A ProLiant DL server that OST services run on.See also OST serviceOST service The Object Storage Target software subsy

Glossary–4

Index–1Cclient configurationsadditional tested configurations 1-6configurations that do not work with HP SFS 1-9untested configurations 1-7client e

Index–2Vviewing file system state information 4-16XXC client nodesdowngrading HP SFS software 2-12installing HP SFS software 2-2, 3-12upgrading HP

Overview1–4HP SFS Version 2.2-0 software has been tested successfully with the following interconnect types:• Gigabit Ethernet interconnect• Quadrics

HP SFS client configurations 1–51.3.2.1 Supported upgrade paths for HP SFS and HP XC configurationsThe supported upgrade paths for HP SFS and HP XC co

Overview1–61.3.3 HP SFS with RHEL and SLES 9 SP3 client configurationsIn addition to HP XC systems (as described in Section 1.3.2), the HP SFS Version

HP SFS client configurations 1–71.3.3.2 Untested client configurationsTable 1-3 lists a number of client configurations that are likely to work succes

Overview1–8• ia32e RHEL 4 Update 2 2.6.9-22.0.2.EL • Gigabit Ethernet interconnect• Quadrics interconnect (QsNetII) (from Quadrics, Ltd.) Version 5.23

HP SFS client configurations 1–91.3.4 Client configurations that do not work with HP SFSIt is either not possible to build or not possible to run HP S

© Copyright 2005, 2006 Hewlett-Packard Development Company, L.P.Lustre® is a registered trademark of Cluster File Systems, Inc.Linux is a U.S. registe

Overview1–10

2–12 Installing and configuring HP XC systemsTo allow client nodes to mount the Lustre file systems on an HP SFS system, the HP SFS client software an

Installing and configuring HP XC systems2–22.1 HP SFS client software for HP XC systemsThe prebuilt packages that you will need for installing the HP

Installing the HP SFS client software on HP XC systems (new installations) 2–33. The binary distribution directory contains a number of subdirectories

Installing and configuring HP XC systems2–4When you have finished configuring the options lnet and lquota settings, proceed to Section 2.2.3 to comple

Installing the HP SFS client software on HP XC systems (new installations) 2–52.2.3.2 Configuring the NTP serverFor the HP SFS diagnostics to work cor

Installing and configuring HP XC systems2–6In both cases, the name of the HP SFS system must resolve on the HP XC system. HP recommends you do this in

Installing the HP SFS client software on HP XC systems (new installations) 2–7• The mount operation may stall for up to ten minutes. Do not interrupt

Installing and configuring HP XC systems2–8Example 2-1 Sample /etc/sfstab.proto file#% n1044lnet://10.0.128.2@vib0,10.0.128.1@vib0:/south-mds10/client

Upgrading HP SFS client software on existing HP XC systems 2–92.3 Upgrading HP SFS client software on existing HP XC systemsThe HP XC version on the c

iiiAbout this guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii1 Overview1.1 Overv

Installing and configuring HP XC systems2–104. Remove all of the existing HP SFS RPM files on the head node in the order in which they were installed,

Upgrading HP SFS client software on existing HP XC systems 2–11NOTE: The sfsconfig command uses the http: protocol to get configuration information fr

Installing and configuring HP XC systems2–12specifically the section titled Disabling Portals compatibility mode (when client nodes have been upgraded

Downgrading HP SFS client software on HP XC systems 2–132. Unmount all Lustre file systems on the head node, as follows:# sfsumount -a3. Remove all of

Installing and configuring HP XC systems2–14

3–13 Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systemsTo allow client nodes to mount the Lustr

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–23.1 HP SFS client software for RHEL and SL

Building your own client kit 3–3• ia32e/RHEL3.0_U8 • ia64/RHEL3.0_U8 • x86_64/RHEL3.0_U8 • i686/RHEL4_U1 • ia32e/RHEL4_U1 • ia64/RHEL4_U1 • x86_64/RHE

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–4• Locating the python-ldap and hpls-diags-

Building your own client kit 3–5• Additional Lustre patches (some distributions only).Where appropriate, you will find additional Lustre patches in th

iv3.3.2 Step 2: Installing the client software ...3-133.3.3 Step 3:

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–6• If you are building on a SLES 9 SP3 syst

Building your own client kit 3–7• Myrinet interconnect:• To add support for the Myrinet interconnect driver, add the following to the command line:--c

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–83. Edit the bootloader configuration file

Building your own client kit 3–9When a Voltaire InfiniBand Version 3.4.5 interconnect driver is used, the ARP resolution parameter on each of the clie

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–10• hpls-diags-clientUse the version of the

Building your own client kit 3–11• listman-2.4.21-chaos.patchAdds 2.6 kernel-compatible list utilities for use in Lustre.• bug2707_fixed-2.4.21-rh.pat

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–123.3 Installing the HP SFS client software

Installing the HP SFS client software on RHEL and SLES 9 SP3 systems (new installations) 3–133.3.2 Step 2: Installing the client softwareTo install th

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–14Note the following points:• In kits where

Installing the HP SFS client software on RHEL and SLES 9 SP3 systems (new installations) 3–15hpls-lustre-client-version_number.rpm \hpls-diags-client-

v6.2 Dealing with ENOSPC or EIO errors... 6-46.2.1 Determining

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–16TIP: Alternatively, you can use the ib-se

Installing the HP SFS client software on RHEL and SLES 9 SP3 systems (new installations) 3–17When the script has completed, examine the /etc/modprobe.

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–18It is possible to restrict the interfaces

Upgrading HP SFS client software on existing RHEL and SLES 9 SP3 systems 3–193.3.4.3 Configuring the NTP serverFor the HP SFS diagnostics to work corr

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–203.4.1 Step 1: Upgrading the HP SFS client

Upgrading HP SFS client software on existing RHEL and SLES 9 SP3 systems 3–21If a modified version of the python-ldap package is not provided for your

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–22• Examine the /etc/sfstab and /etc/sfstab

Downgrading HP SFS client software on RHEL and SLES 9 SP3 systems 3–23 user3 22513 ..c.. csh user3 31

Installing and configuring Red Hat Enterprise Linux and SUSE Linux Enterprise Server 9 SP3 client systems3–247. Replace or edit the /etc/sfstab.proto

4–14 Mounting and unmounting Lustre file systems on client nodesThis chapter provides information on mounting and unmounting file systems on client no

vi

Mounting and unmounting Lustre file systems on client nodes4–24.1 OverviewNOTE: Before you attempt to mount a Lustre file system on a client node, mak

Mounting Lustre file systems using the sfsmount command with the lnet: protocol 4–3that you convert existing systems to use the lnet: protocol. The pr

Mounting and unmounting Lustre file systems on client nodes4–4TIP: If the client node has access to the HP SFS system on a TCP/IP network, you can gen

Mount options 4–5mds_service is the name of the MDS service on the HP SFS system (as shown by the sfsmgr show filesystem command on the HP SFS server)

Mounting and unmounting Lustre file systems on client nodes4–6[no]repeat sfsmount Specifies whether repeated attempts are to be made to mount the file

Unmounting file systems on client nodes 4–74.6 Unmounting file systems on client nodesYou can use either the standard umount(8) command or the sfsumou

Mounting and unmounting Lustre file systems on client nodes4–8filesystemSpecifies the name of the Lustre file system that is to be unmounted.mountpoin

Using the SFS service 4–9An alternative method of unmounting Lustre file systems on the client node is to enter the service sfs stop command, as descr

Mounting and unmounting Lustre file systems on client nodes4–10To configure a client node to automatically mount a file system at boot time, perform t

Using the SFS service 4–11CAUTION: When you move an entry from a client node’s /etc/sfstab file to the /etc/sfstab.proto file, you must delete the ent

viiAbout this guideThis guide describes how to install and configure the HP StorageWorks Scalable File Share (HP SFS) client software on client nodes

Mounting and unmounting Lustre file systems on client nodes4–12After the /etc/sfstab.proto file shown above is processed by the SFS service, the /etc/

Using the SFS service 4–134.7.6 The service sfs status commandThe service sfs status command shows information on the status (mounted or unmounted) of

Mounting and unmounting Lustre file systems on client nodes4–144.8 Alternative sfsmount modesIn addition to supporting the standard mount command with

Alternative sfsmount modes 4–154.8.2 Mounting Lustre file systems using the sfsmount command with the ldap: protocolNOTE: Lustre file systems must be

Mounting and unmounting Lustre file systems on client nodes4–164.9 Restricting interconnect interfaces on the client nodeWhen a Gigabit Ethernet inter

File system service information and client communications messages 4–17• If a server in the HP SFS system is shut down or crashes, or if the file syst

Mounting and unmounting Lustre file systems on client nodes4–18• The following message shows that the client node is attempting to connect to a server

5–15 Configuring NFS and Samba servers to export Lustre file systemsHP SFS allows client systems to use the NFS or SMB (using Samba) protocols to acce

Configuring NFS and Samba servers to export Lustre file systems5–25.1 Configuring NFS serversSome legacy client systems can only use the NFS protocol;

Configuring NFS servers 5–35.1.2 Configuration factors for NFS serversWhen configuring HP SFS client nodes as NFS servers, consider the following poin

viiiHP SFS documentationThe HP StorageWorks Scalable File Share documentation set consists of the following documents:• HP StorageWorks Scalable File

Configuring NFS and Samba servers to export Lustre file systems5–45.1.3 Configuration factors for multiple NFS serversNFS services may be configured t

Configuring NFS servers 5–55.1.3.2 NFS performance scaling exampleFigure 5-2 illustrates how performance is affected when the number of NFS servers an

Configuring NFS and Samba servers to export Lustre file systems5–65.1.6 Optimizing NFS server performanceTo optimize NFS performance, consider the fol

Configuring Samba servers 5–7• The functionality that allows Lustre file systems to be exported via Samba is intended for interoperability purposes. W

Configuring NFS and Samba servers to export Lustre file systems5–8

6–16 User interaction with Lustre file systemsThis chapter is organized as follows:• Defining file stripe patterns (Section 6.1)• Dealing with ENOSPC

User interaction with Lustre file systems6–26.1 Defining file stripe patternsLustre presents a POSIX API as the file system interface; this means that

Defining file stripe patterns 6–36.1.2 Using a C program to create a fileThe following C program fragment shows an example of how to create a file wit

User interaction with Lustre file systems6–46.1.3 Setting a default stripe size on a directoryIf you want to create many files with the same stripe at

Dealing with ENOSPC or EIO errors 6–56.2.1 Determining the file system capacity using the lfs df commandYou can use the lfs df command to determine if

ixNaming conventionsThis section lists the naming conventions used for an HP SFS system in this guide. You are free to choose your own name for your H

User interaction with Lustre file systems6–6b. Check the MDS service by entering the command shown in the following example on the client node:# cat /

Using Lustre file systems — performance hints 6–7south-ost51_UUID 2113787820 681296236 1432491584 32 /mnt/data[OST:2]south-ost52_UUID 21137878

User interaction with Lustre file systems6–86.3.1.1 Improving the performance of the rm -rf commandIf the rm -rf command is issued from a single clien

Using Lustre file systems — performance hints 6–96.3.3 Variation of file stripe count with shared file accessWhen multiple client processes are access

User interaction with Lustre file systems6–10Server-side timeoutsServer-side timeouts can occur as follows:• When client nodes are connected to MDS an

Using Lustre file systems — performance hints 6–11The parameters that control client operation interact as shown in the following example. In this exa

User interaction with Lustre file systems6–126.3.5 Using a Lustre file system in the PATH variableHP strongly recommends that you do not add a Lustre

7–17TroubleshootingThis chapter provides information for troubleshooting possible problems on client systems. The topics covered include the following

Troubleshooting7–27.1 Installation issuesThis section deals with issues that may arise when the HP SFS software is being installed on the client nodes

File system mounting issues 7–37.1.2 Client node still boots the old kernel after installationIf a client node does not boot the new kernel after the