The CANARI large ensemble, investigating climate variability and change across the Arctic–North Atlantic region and its impacts on the UK, presents substantial challenges in high-performance computing, data management and distribution. With 40 ensemble members generating several petabytes of data and millions of files, conventional approaches to managing and accessing simulation output become increasingly difficult to sustain.

Presentation: pdf (2.6 MB)

This talk describes innovations developed to address these challenges, drawing on experience from running the ensemble on ARCHER2 and exploiting advances in data management from several European and UK infrastructure projects. A particular focus is the use of Climate Forecast Aggregation (CFA) to provide a unified, metadata-rich view of large collections of fragmented datasets, while supporting flexible management of data across disk and tape storage.

The talk also explores emerging approaches to distributed data analysis, particularly Active Storage, which moves computation closer to where data resides rather than requiring large volumes of data to be transferred. Together, these developments offer a route towards more scalable and efficient exploitation of increasingly large, high-resolution climate simulations.