Skip to content

Directory /dcache (Stoomboot)

Aim: Describe the Stoomboot dCache storage.

Target audience: Users who want to store large amounts of data at Nikhef.

Introduction

The dCache store can hold your largest data files. It’s a distributed storage system, designed to cope with the load of Stoomboot running on all cylinders (or nodes, in this case). Access looks like a normal file system (/dcache/...), but once written a file cannot be modified - only re-written in full. It’s the perfect place for analysis results, your ntuples, and MC samples.

Usage

Intended use

Large scale data files, files access from many (Stoomboot) systems at once

Examples of data types that are better put elsewhere

  • your software and scripts (these should be in /project or your home directory);
  • intermediate log files (these should be in $TMPDIR or /localstore).

What are the limitations?

This filesystem is not backed up, so if the system fails catastrophically: too bad. You will have to re-create or re-download the data. Also, dCache files cannot be modified (or appended to), and are written ‘in one go’, but it still looks like a regular filesystem. Out of space? Ask your group leader to discuss with the CT-PDP team.

The files are distributed across several storage system, but all meta-data is held together in one massive PostgreSQL database. The system is optimised to deal with large files (several GB), not for millions of small files. Avoid intensive metadata operations like du or find. Contact the system administrators if you need to know how much disk space is in use.

Did you know...

Did you know that files in dCache can also be read remotely through the GridFTP, WebDAV, and xrootd protocols? Register your certificate distinguished name for access.

There is a public Grafana dashboard to see the operational status.

Contact