Origins2

Hardware

  • Dell PowerEdge R750

  • Processor: Intel® Xeon® Gold 5317 3G, 12C/24T, 11.2GT/s, 18M Cache, Turbo, HT (150W) DDR4-2933 x 2

  • Memory: 8GB RDIMM, 3200MT/s, Single Rank x 16

  • Disk: 1.92TB SSD vSAS Read Intensive 12Gbps 512e 2.5in Hot-Plug ,AG Drive SED, 1DWPD x 4 + 2.4TB 10K RPM SAS ISE 12Gbps 512e 2.5in Hot-Plug Hard Drive x 4

    • partition

      disk space configuration
      home 1.7T RAID1
      scratch 3.5T RAID0
      storage1 2.2T RAID1
      storage2 2.2T RAID1

  • Graphic Card: NVIDIA Ampere A30, PCIe, 165W, 24GB Passive, Double Wide, Full Height GPU x 1

Operating System

  • Ubuntu 22.04

User Resources

Directories and Quota

directory path soft limit hard limit
home /home/<account> 8 GB 10 GB
scratch /scratch/users/<account> 250 GB 300 GB
storage /storage1/users/<account> or /storage2/users/<account> 150 GB 200 GB
  • To check your own quota: quota -s

Connection

  • Domain name: origins2.rss.chalmers.se

Softwares

Modules

Origins2 support using environment module to dynamically load required modules. The basic usage can be found in many documents from other HPC center, e.g. C3SE

To check the available modules, use module avail or module key <keyword> search the software you want to use

Containers

Some softwares are installed in containers, e.g. CASA. They can be found in /apps/containers. There are several ways you can use containers

  • apptainer run <path/to/container>: execute the runscript defined in the container
  • apptainer shell <path/to/container>: launch a shell with the environment of the container
  • apptainer exec <path/to/container> <whatever/command/you/would/like/to/execute/with/arguments>: execute the command with the container

For more information, please check Apptainer document

Others

Running Enzo on origins2

Prerequisite

To build Enzo, you may need load several modules on origins2. Here are some examples assuming using intel compilers

~ $ module purge && module load intel/2022a HDF5/1.12.2-iimpi-2022a
### if you are using grackle
~ $ module purge && module load intel/2022a HDF5/1.12.2-iimpi-2022a Grackle/3.2dev-intel-2022a
### if you are using naunet and grackle
~ $ module purge && module load SUNDIALS/5.7.0-intel-2022a HDF5/1.12.2-iimpi-2022a Grackle/3.2dev-intel-2022a

Makefile configuration

The following is an example Makefile configuration file for Enzo:

Make.mach.origins2 
#=======================================================================
#
# FILE:        Make.mach.origins2
#
# DESCRIPTION: Makefile settings on origins2
#
# AUTHOR:      Chia-Jung Hsu
#
# DATE:        2023-05-24
#
# Revised from Make.mach.linux-gnu
#
#=======================================================================

MACH_VALID = 1
MACH_FILE  = Make.mach.origins2

#-----------------------------------------------------------------------
# Install paths (local variables)
#-----------------------------------------------------------------------

LOCAL_MPI_INSTALL         =
LOCAL_HDF5_INSTALL        =
LOCAL_GRACKLE_INSTALL     =
LOCAL_NAUNET_INSTALL      = <path/to/any/naunet/library>
LOCAL_SUNDIALS_INSTALL    =
LOCAL_SUITESPARSE_INSTALL =
LOCAL_HYPRE_INSTALL       =

#-----------------------------------------------------------------------
# Compiler settings
#-----------------------------------------------------------------------

MACH_CPP       = cpp # C preprocessor command

# With MPI

MACH_CC_MPI    = mpiicc # C compiler when using MPI
MACH_CXX_MPI   = mpiicpc # C++ compiler when using MPI
MACH_FC_MPI    = mpiifort # Fortran 77 compiler when using MPI
MACH_F90_MPI   = mpiifort # Fortran 90 compiler when using MPI
MACH_LD_MPI    = mpiicpc # Linker when using MPI

# Without MPI

MACH_CC_NOMPI  = icc # C compiler when not using MPI
MACH_CXX_NOMPI = icpc # C++ compiler when not using MPI
MACH_FC_NOMPI  = ifort # Fortran 77 compiler when not using MPI
MACH_F90_NOMPI = ifort # Fortran 90 compiler when not using MPI
MACH_LD_NOMPI  = icpc # Linker when not using MPI

#-----------------------------------------------------------------------
# Machine-dependent defines
#-----------------------------------------------------------------------

MACH_DEFINES   = -DLINUX -DH5_USE_16_API

#-----------------------------------------------------------------------
# Compiler flag settings
#-----------------------------------------------------------------------


MACH_CPPFLAGS = -P -traditional
MACH_CFLAGS   =
MACH_CXXFLAGS =
MACH_FFLAGS   = -132
MACH_F90FLAGS =
MACH_LDFLAGS  = -mkl

#-----------------------------------------------------------------------
# Optimization flags
#-----------------------------------------------------------------------

MACH_OPT_WARN        = -Wall -g
MACH_OPT_DEBUG       = -g
MACH_OPT_HIGH        = -O2
MACH_OPT_AGGRESSIVE  = -O3 -g

#-----------------------------------------------------------------------
# Includes
#-----------------------------------------------------------------------

LOCAL_INCLUDES_MPI         =
LOCAL_INCLUDES_HDF5        =
LOCAL_INCLUDES_HYPRE       =
LOCAL_INCLUDES_PAPI        = # PAPI includes
LOCAL_INCLUDES_GRACKLE     =
LOCAL_INCLUDES_NAUNET      = -I$(LOCAL_NAUNET_INSTALL)/include
LOCAL_INCLUDES_SUNDIALS    =
LOCAL_INCLUDES_SUITESPARSE =

MACH_INCLUDES             = $(LOCAL_INCLUDES_HDF5)
MACH_INCLUDES_MPI         = $(LOCAL_INCLUDES_MPI)
MACH_INCLUDES_HYPRE       = $(LOCAL_INCLUDES_HYPRE)
MACH_INCLUDES_PAPI        = $(LOCAL_INCLUDES_PAPI)
MACH_INCLUDES_GRACKLE     = $(LOCAL_INCLUDES_GRACKLE)
MACH_INCLUDES_NAUNET      = $(LOCAL_INCLUDES_NAUNET)
MACH_INCLUDES_SUNDIALS    = $(LOCAL_INCLUDES_SUNDIALS)
MACH_INCLUDES_SUITESPARSE = $(LOCAL_INCLUDES_SUITESPARSE)

#-----------------------------------------------------------------------
# Libraries
#-----------------------------------------------------------------------

LOCAL_LIBS_MACH       = -lifcore -lifport # Machine-dependent libraries
LOCAL_LIBS_MPI        =
LOCAL_LIBS_HDF5       = -lhdf5 -lz
LOCAL_LIBS_HYPRE      =
LOCAL_LIBS_PAPI       = # PAPI libraries
LOCAL_LIBS_GRACKLE    = -lgrackle
LOCAL_LIBS_NAUNET     = -L$(LOCAL_NAUNET_INSTALL)/lib -lnaunet


MACH_LIBS             = $(LOCAL_LIBS_HDF5) $(LOCAL_LIBS_MACH)
MACH_LIBS_MPI         = $(LOCAL_LIBS_MPI)
MACH_LIBS_HYPRE       = $(LOCAL_LIBS_HYPRE)
MACH_LIBS_PAPI        = $(LOCAL_LIBS_PAPI)
MACH_LIBS_GRACKLE     = $(LOCAL_LIBS_GRACKLE)
MACH_LIBS_NAUNET      = $(LOCAL_LIBS_NAUNET)
MACH_LIBS_SUNDIALS    = $(LOCAL_LIBS_SUNDIALS)
MACH_LIBS_SUITESPARSE = $(LOCAL_LIBS_SUITESPARSE)

If you are building Enzo with naunet, you may need to override some parameters

Make.config.override 
PARAMETER_MAX_BARYONS = 145
CONFIG_GRACKLE = yes
CONFIG_NAUNET = yes
CONFIG_OPT = high
CONFIG_FFTW = no

A complete example of building Enzo with naunet lib and grackle on origins2

~ $ module purge && module load SUNDIALS/5.7.0-intel-2022a \
                                HDF5/1.12.2-iimpi-2022a \
                                CMake/3.23.1-GCCcore-11.3.0 \
                                Grackle/3.2dev-intel-2022a
~ $ git clone https://github.com/appolloford/naunet_klu_cloud_custom.git # if you are using the chemical network from GMC Collision IX.
~ $ cd naunet_klu_cloud_custom
(naunet_cloud_custom) $ cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release -DMAKE_PYTHON=ON -DCMAKE_INSTALL_PREFIX=.
(naunet_cloud_custom) $ cmake --build build -j4
(naunet_cloud_custom) $ cd build && ctest
(build) $ make install
(build) $ cd 
~ $ git clone https://github.com/appolloford/enzo-dev.git 
~ $ cd enzo-dev
(enzo-dev) $ ./configure
(enzo-dev) $ cd src/enzo && git checkout klu_cloud

### Copy the make.mach.origins2 and Make.config.override here
### Replace the value of LOCAL_NAUNET_INSTALL by `$HOME/naunet_klu_cloud_custom`

(enzo) $ make machine-origins2
(enzo) $ make -j8

### You should get an executale: enzo.exe here