Origins2

Hardware

  • Dell PowerEdge R750

  • Processor: Intel® Xeon® Gold 5317 3G, 12C/24T, 11.2GT/s, 18M Cache, Turbo, HT (150W) DDR4-2933 x 2

  • Memory: 8GB RDIMM, 3200MT/s, Single Rank x 16

  • Disk: 1.92TB SSD vSAS Read Intensive 12Gbps 512e 2.5in Hot-Plug ,AG Drive SED, 1DWPD x 4 + 2.4TB 10K RPM SAS ISE 12Gbps 512e 2.5in Hot-Plug Hard Drive x 4

    • partition

      disk space configuration
      home 1.7T RAID1
      scratch 13T RAID6
      storage1 2.2T RAID1
      storage2 2.2T RAID1

  • Graphic Card: NVIDIA Ampere A30, PCIe, 165W, 24GB Passive, Double Wide, Full Height GPU x 1

Operating System

  • Ubuntu 22.04

User Resources

Directories and Quota

directory path soft limit hard limit
home /home/<account> 8 GB 10 GB
scratch /scratch/users/<account> 250 GB 300 GB
storage /storage1/users/<account> or /storage2/users/<account> 150 GB 200 GB
  • To check your own quota: quota -s

Connection

  • Domain name: origins2.rss.chalmers.se

Softwares

Modules

Origins2 support using environment module to dynamically load required modules. The basic usage can be found in many documents from other HPC center, e.g. C3SE

To check the available modules, use module avail or module key <keyword> search the software you want to use

Containers

Some softwares are installed in containers, e.g. CASA. They can be found in /apps/containers. There are several ways you can use containers

  • apptainer run <path/to/container>: execute the runscript defined in the container
  • apptainer shell <path/to/container>: launch a shell with the environment of the container
  • apptainer exec <path/to/container> <whatever/command/you/would/like/to/execute/with/arguments>: execute the command with the container

For more information, please check Apptainer document

Others

Running Enzo on origins2

Prerequisite

To build Enzo, you may need load several modules on origins2. Here are some examples assuming using intel compilers

~ $ module purge && module load HDF5/1.14.0-gompi-2023a buildenv/default-foss-2023a
### if you are using grackle
~ $ module purge && module load grackle/3.3.0-gompi-2023a buildenv/default-foss-2023a
### if you are using naunet and grackle
~ $ module purge && module load SUNDIALS/6.6.0-foss-2023a-CUDA-12.1.1 grackle/3.3.0-gompi-2023a buildenv/default-foss-2023a

Makefile configuration

The following is an example Makefile configuration file for Enzo:

Make.mach.origins2 
#=======================================================================
#
# FILE:        Make.mach.origins2
#
# DESCRIPTION: Makefile settings on origins2
#
# AUTHOR:      Chia-Jung Hsu
#
# DATE:        2025-03-22
#
#=======================================================================

MACH_TEXT  = ORIGINS2
MACH_VALID = 1
MACH_FILE  = Make.mach.origins2

#-----------------------------------------------------------------------
# Install paths (local variables)
#-----------------------------------------------------------------------

LOCAL_MPI_INSTALL         =
LOCAL_HDF5_INSTALL        =
LOCAL_GRACKLE_INSTALL     =
LOCAL_NAUNET_INSTALL      = <path/to/any/naunet/library>
LOCAL_SUNDIALS_INSTALL    =
LOCAL_SUITESPARSE_INSTALL =
LOCAL_HYPRE_INSTALL       =
LOCAL_FFTW_INSTALL        =

#-----------------------------------------------------------------------
# Compiler settings
#-----------------------------------------------------------------------

MACH_CPP       = cpp # C preprocessor command

# With MPI

MACH_CC_MPI    = mpicc
MACH_CXX_MPI   = mpicxx
MACH_FC_MPI    = mpif90
MACH_F90_MPI   = mpif90
MACH_LD_MPI    = mpicxx

# Without MPI

MACH_CC_NOMPI  = gcc      # C compiler when not using MPI
MACH_CXX_NOMPI = g++      # C++ compiler when not using MPI
MACH_FC_NOMPI  = gfortran # Fortran 77 compiler when not using MPI
MACH_F90_NOMPI = gfortran # Fortran 90 compiler when not using MPI
MACH_LD_NOMPI  = g++      # Linker when not using MPI

#-----------------------------------------------------------------------
# Machine-dependent defines
#-----------------------------------------------------------------------

# Note: When compiling against HDF5 version 1.8 or greater, you need to
# compile HDF5 with --with-default-api-version=v16, or Enzo with
# -DH5_USE_16_API.

MACH_DEFINES   = -DLINUX -DH5_USE_16_API

#-----------------------------------------------------------------------
# Compiler flag settings
#-----------------------------------------------------------------------

MACH_CPPFLAGS = -P -traditional
MACH_CFLAGS   =
MACH_CXXFLAGS =
MACH_FFLAGS   = -std=legacy -fno-second-underscore -ffixed-line-length-132
MACH_F90FLAGS = -std=legacy -fno-second-underscore
MACH_LDFLAGS  =

#-----------------------------------------------------------------------
# Optimization flags
#-----------------------------------------------------------------------

MACH_OPT_WARN        = -Wall -g
MACH_OPT_DEBUG       = -g
MACH_OPT_HIGH        = -O2 -g -march=native -mcmodel=medium
MACH_OPT_AGGRESSIVE  = -O3 -g -march=native -mcmodel=medium
#MACH_OPT_AGGRESSIVE  = -ipo -O3

#-----------------------------------------------------------------------
# Includes
#-----------------------------------------------------------------------

LOCAL_INCLUDES_MPI         =
LOCAL_INCLUDES_HDF5        =
LOCAL_INCLUDES_HYPRE       =
LOCAL_INCLUDES_PAPI        = # PAPI includes
LOCAL_INCLUDES_GRACKLE     =
LOCAL_INCLUDES_NAUNET      = -I$(LOCAL_NAUNET_INSTALL)/include
LOCAL_INCLUDES_SUNDIALS    =
LOCAL_INCLUDES_SUITESPARSE =
LOCAL_INCLUDES_FFTW        =

MACH_INCLUDES             = $(LOCAL_INCLUDES_HDF5)

MACH_INCLUDES_MPI         = $(LOCAL_INCLUDES_MPI)
MACH_INCLUDES_HYPRE       = $(LOCAL_INCLUDES_HYPRE)
MACH_INCLUDES_PAPI        = $(LOCAL_INCLUDES_PAPI)
MACH_INCLUDES_GRACKLE     = $(LOCAL_INCLUDES_GRACKLE)
MACH_INCLUDES_NAUNET      = $(LOCAL_INCLUDES_NAUNET)
MACH_INCLUDES_SUNDIALS    = $(LOCAL_INCLUDES_SUNDIALS)
MACH_INCLUDES_SUITESPARSE = $(LOCAL_INCLUDES_SUITESPARSE)


#-----------------------------------------------------------------------
# Libraries
#-----------------------------------------------------------------------

LOCAL_LIBS_MACH       = -lgfortran
LOCAL_LIBS_HDF5       = -lhdf5 -lsz -lz
LOCAL_LIBS_GRACKLE    = -lgrackle
LOCAL_LIBS_NAUNET     = -L$(LOCAL_NAUNET_INSTALL)/lib -lnaunet
LOCAL_LIBS_SUNDIALS   = -L$(LOCAL_SUNDIALS_INSTALL)/lib64 -lsundials_cvode \
                        -lsundials_sunlinsolklu -lsundials_sunmatrixsparse
LOCAL_LIBS_FFTW       = -lfftw3

MACH_LIBS             = $(LOCAL_LIBS_HDF5) $(LOCAL_LIBS_MACH)
MACH_LIBS_MPI         = $(LOCAL_LIBS_MPI)
MACH_LIBS_HYPRE       = $(LOCAL_LIBS_HYPRE)
MACH_LIBS_PAPI        = $(LOCAL_LIBS_PAPI)
MACH_LIBS_GRACKLE     = $(LOCAL_LIBS_GRACKLE)
MACH_LIBS_NAUNET      = $(LOCAL_LIBS_NAUNET)
MACH_LIBS_SUNDIALS    = $(LOCAL_LIBS_SUNDIALS)
MACH_LIBS_SUITESPARSE = $(LOCAL_LIBS_SUITESPARSE)
MACH_LIBS_FFTW        = $(LOCAL_LIBS_FFTW)

If you are building Enzo with naunet, you may need to override some parameters

Make.config.override 
PARAMETER_MAX_BARYONS = 145
CONFIG_GRACKLE = yes
CONFIG_NAUNET = yes
CONFIG_OPT = high
CONFIG_FFTW = no

A complete example of building Enzo with naunet lib and grackle on origins2

~ $ module purge && module load SUNDIALS/6.6.0-foss-2023a-CUDA-12.1.1 \
                                grackle/3.3.0-gompi-2023a \
                                buildenv/default-foss-2023a \
                                CMake/3.26.3-GCCcore-12.3.0 \
                                pybind11/2.11.1-GCCcore-12.3.0  # only needed for building the python package of the model
~ $ git clone https://github.com/appolloford/naunet_klu_cloud_custom.git # if you are using the chemical network from GMC Collision IX.
~ $ cd naunet_klu_cloud_custom
(naunet_cloud_custom) $ cmake -S. -Bbuild -DCMAKE_INSTALL_PREFIX=.
(naunet_cloud_custom) $ cmake --build build -j4
(naunet_cloud_custom) $ cd build && ctest
(build) $ make install
(build) $ cd 
~ $ git clone https://github.com/appolloford/enzo-dev.git 
~ $ cd enzo-dev
(enzo-dev) $ ./configure
(enzo-dev) $ cd src/enzo && git checkout klu_cloud

### Copy the make.mach.origins2 and Make.config.override here
### Replace the value of LOCAL_NAUNET_INSTALL by `$HOME/naunet_klu_cloud_custom`

(enzo) $ make machine-origins2
(enzo) $ make -j8

### You should get an executale: enzo.exe here