Storage with batch jobs

Aim: Provide basic information about how to write files to the Stoomboot dCache storage.

Target audience: Users of the Stoomboot cluster.

Introduction

dCache is filesystem designed for storing large amounts of data. The data is stored on many server nodes, accessed through a single virtual filesystem by a variety of methods. The dCache filesystem is available on Stoomboot and can be found under the directory /dcache.

There are no backups made of the dCache file system.

Files written to dCache are fixed and not changeable, see Immutable Files. Other than this limitation dCache can be used pretty much as a normal file system.

Issues and problems with the file system should be sent to stbc-admin@nikhef.nl.

Prerequisites

A Nikhef account;
An ssh client;
(Optional) Nikhef eScience IGTF-MICS certificate

Usage

Getting a dCache directory

To get a storage directory, please send a request to stbc-admin@nikhef.nl, including which group you are working with and where to create your directory (as the directory organization within each group sub-directory is up to each group). And finally don't forget a name for the new directory. Access

dCache can be accessed from any of the Stoomboot nodes, both the interactive and compute nodes. The file system is mounted via an NFS mount point /dcache. This mount point is the same across all the Stoomboot nodes. Running jobs can also read and write to dCache as jobs run under the identity of the user who submitted the job and so have the same access rights as the same user in an interactive shell. There is no access to the /dcache mount from outside Stoomboot.

Immutable Files

Once a file is written into dCache the contents of the file can not be changed.

This means a file CAN NOT be overwritten:

[stbc-i1 /dcache/test]$ echo 123 > test.file
[stbc-i1 /dcache/test]$ echo 1234 > test.file
-bash: test.file: Permission denied

and CAN NOT be appended to:

[stbc-i2 /dcache/test]$ echo 123 > test.file
[stbc-i2 /dcache/test]$ echo 4 >> test.file
-bash: test.file: Permission denied

and when deleting a file is always treated as being write protected (even if the writable bit is set):

[stbc-i2 /dcache/test]$ ll test.file
-rw-r--r--. 1 andrewp datagrid 4 May  4 09:57 test.file
[stbc-i2 /dcache/test]$ rm test.file
rm: remove write-protected regular file `test.file'? y

These limitations mean that dCache is NOT suited for such things as compiling code on, storing large files being actively edited (e.g. word documents) or storing files under a version control system.

No Backups

There are NO BACKUPS of files stored on the dCache file system.

Storage Space Organization

The total storage space on the dCache storage nodes is a couple of petabytes. Each group has a directory under the main dCache directory:

[stbc-i4 ~]$ ll /dcache/
total 2
drwxr-xr-x. 4 root alice    512 Apr  1 11:46 alice
drwxr-sr-x. 5 root atlas    512 May  4 09:48 atlas
drwxrwsr-x. 3 root datagrid 512 Nov 13  2015 datagrid

User storage directories are under the appropriate group directory:

[stbc-i4 ~]$ ll /dcache/datagrid/
total 1
drwxr-sr-x. 1094 andrewp datagrid 512 May  4 09:59 andrewp

Storage space is assigned to each group under there group directory so all people within a group share the same group storage space allocation.

Note df doesn't give useful results for dCache so a web page with current data usage will be setup.

Information about the dCache usage can be found (while on the Nikhef eduVPN or Nikhef network) at https://steker.nikhef.nl:3000/d/PO3OmtkZk/dcache-stoomboot?orgId=1&refresh=5m.

Remote access (certificate required)

Remote access to the dCache storage using grid certificate authorization is also available. See this page on getting started with grid certificates for information.

In order to access dCache, your need to link your certificate to your Nikhef account in the SSO system: log on at https://sso.nikhef.nl/ and select 'Connect your certificate' from the menu on the left hand side and follow the instructions.

You may need to wait several minutes before the registration comes through to the dCache systems.

Once this is done, the xrootd door can be accessed at dcache.nikhef.nl. The directory paths are the same as for NFS access.

access using the XRootD protocol

The directory /dcache/test on Stoomboot can be seen via xrootd at root://dcache.nikhef.nl/dcache/test. To list the directory use the command xrd for example:

[bosui ~]$ xrdfs dcache.nikhef.nl ls -l /dcache/test

OR

[bosui ~]$ xrd dcache.nikhef.nl dirlist /dcache/test

And to copy a file:

[bosui ~]$ xrdcp root://dcache.nikhef.nl/dcache/test/test1 test1

Access using the WebDAV protocol

The WebDAV protocol is based on HTTP, which means you can access your files directly from your web browser (provided that you have imported your certificate in your browser).

Point your browser at https://dcache.nikhef.nl:2880/dcache/test to see some test files.

Using WebDAV from the command line and in scripts is also possible, e.g. with the open source DaviX client.

The best way to do this is by generating an X509 proxy certificate and using that for authentication, as a proxy does not have a passphrase so you are not prompted for one on every interaction.

Some examples (please modify to your situation):

davix-ls --cert $X509_USER_PROXY --key $X509_USER_PROXY https://dcache.nikhef.nl:2880/dcache/test

davix-ls --cert ${HOME}/.globus/usercert.pem --key ${HOME}/.globus/userkey.pem   --capath /etc/grid-security/certificates/ https://dcache.nikhef.nl:2880/dcache/test