Origins

Don’t save your data in your home directory!

Change your password at the first time you login: use passwd!

Directories

directory path soft limit hard limit comments
home /home/<account> 2.5 GB 3 GB save your configuration files
scratch /scratch/<account> - - save your program output
storage /storage/<account> - - save important data, automatically back up
  • General idea:
    1. Compile you codes in /home
    2. Copy the binary executable and parameter files to /scratch
    3. Save production run data to /storage

Connection

  • domain name: origins.rss.chalmers.se

Softwares, Compilers, and Libraries

General Concepts:

  1. If you cannot find a compiler, you should add the path to the compiler into the PATH environment variable
  2. If a shared library is missing, you should add the path to the library into the LD_LIBRARY_PATH environment variable

Softwares

CASA-6.2.0

  • Path: /opt/casa/casa-6.2.0-124
  • To use casa, add the executable path into PATH environmental variable:
~ $ echo 'export PATH=/opt/casa/casa-6.2.0-124/bin:$PATH' >> ~/.bashrc

Anaconda-2022.05

  • Path: /opt/anaconda3
  • To use conda and other commands, add the executable path into PATH environmental variable:
~ $ echo 'export PATH=/opt/anaconda3/bin:$PATH' >> ~/.bashrc

Compilers

GCC-7.5.0

  • Default system compiler. The associated MPI (including openmpi and mpich) are also installed in system path. They should work anyway.

Intel oneAPI HPC Toolkit

  • Path: /opt/intel/oneapi
  • To use intel compiler (icc, icpc, ifort), modify the PATH and LD_LIBRARY_PATH environmental variables:
~ $ echo 'export PATH=/opt/intel/oneapi/compiler/latest/linux/bin/intel64:$PATH' >> ~/.bashrc
~ $ echo 'export LD_LIBRARY_PATH=/opt/intel/oneapi/compiler/latest/linux/compiler/lib/intel64:$LD_LIBRARY_PATH' >> ~/.bashrc
  • To use intel mpi (mpiicc, mpiicpc, mpiifort), modify the PATH and LD_LIBRARY_PATH environmental variables:
~ $ echo 'export PATH=/opt/intel/oneapi/mpi/latest/bin:$PATH' >> ~/.bashrc
~ $ echo 'export LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/latest/lib:/opt/intel/oneapi/mpi/latest/lib/debug:$LD_LIBRARY_PATH' >> ~/.bashrc
  • Notes:
    • LD_LIBRARY_PATH should have been added into system path. If it does not work, try execute this block
    • Simple check: which icpc, which mpiicpc

CUDA-10.0.0

  • Path: /usr/local/cuda
  • To use nvcc, modify the PATH and LD_LIBRARY_PATH environmental variables:
~ $ echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
  • Notes:
    • LD_LIBRARY_PATH should have been added into system path. If it does not work, try execute this block
    • Simple check: which nvcc
    • Check drivers on: nvidia-smi

CUDA-11.7.0

  • Path: /usr/local/cuda-11.7
  • To use nvcc, modify the PATH and LD_LIBRARY_PATH environmental variables:
~ $ echo 'export PATH=/usr/local/cuda-11.7/bin:$PATH' >> ~/.bashrc
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11,7/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc

Other libraries

HDF5-1.8.21

  • Path: /usr/local/hdf5
  • To use command line tool, modify PATH:
~ $ echo 'export PATH=/usr/local/hdf5/bin:$PATH' >> ~/.bashrc
  • Notes:
    • Simple check: which h5ls --version
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/hdf5/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

HDF5-1.10.7

  • Path: /usr/local/hdf5_1.10.7
  • To use command line tool, modify PATH:
~ $ echo 'export PATH=/usr/local/hdf5_1.10.7/bin:$PATH' >> ~/.bashrc
  • Notes:
    • Simple check: which h5ls --version
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/hdf5_1.10.7/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

FFTW-2.1.5

  • Path: /usr/local/fftw2
  • To load libraries, modify LD_LIBRARY_PATH: shell ~ $ echo 'export LD_LIBRARY_PATH=/usr/local/fftw2/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

FFTW-3.3.9

  • Path: /usr/local/fftw3
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/fftw3/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

Grackle-2.2

  • Path: /usr/local/grackle2.2
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/grackle2.2/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

Grackle-3.2

  • Path: /usr/local/grackle3.2
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/grackle3.2/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

SuiteSparse-5.9.0

  • Path: /usr/local/suitesparse-5.9.0
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/suitesparse-5.9.0/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

Sundials-5.7.0

  • Path: /usr/local/sundials-5.7.0
  • To load libraries, modify LD_LIBRARY_PATH:
~ $ echo 'export LD_LIBRARY_PATH=/usr/local/sundials-5.7.0/lib:$LD_LIBRARY_PATH' >> ~/.bashrc

Others

Running Enzo on origins

For Enzo users, the following Makefile setting should work on origins. (You may have to check whether icpc and mpiicpc have been in your path. Check the previous Intel oneAPI HPC Toolkit section). Copy the file into src/enzo and execute make machine-origins.

Make.mach.origins 
    #=======================================================================
    #
    # FILE:        Make.mach.origins
    #
    # DESCRIPTION: Makefile settings on Origins
    #
    # AUTHOR:      Chia-Jung Hsu
    #
    # DATE:        2021-07-11
    #
    # Update from Make.mach.linux-gnu
    #
    #=======================================================================

    MACH_VALID = 1
    MACH_FILE  = Make.mach.origins

    #-----------------------------------------------------------------------
    # Install paths (local variables)
    #-----------------------------------------------------------------------

    LOCAL_MPI_INSTALL         = /opt/intel/oneapi/mpi/latest
    LOCAL_HDF5_INSTALL        = /usr/local/hdf5
    # Uncomment the next line if you are using grackle. For enzo > 2.5
    # LOCAL_GRACKLE_INSTALL     = /usr/local/grackle3.2
    # LOCAL_GRACKLE_INSTALL     = /usr/local/grackle2.2 # For older version enzo
    LOCAL_HYPRE_INSTALL       = 

    #-----------------------------------------------------------------------
    # Compiler settings
    #-----------------------------------------------------------------------

    MACH_CPP       = cpp # C preprocessor command

    # With MPI

    MACH_CC_MPI    = mpiicc # C compiler when using MPI
    MACH_CXX_MPI   = mpiicpc # C++ compiler when using MPI
    MACH_FC_MPI    = mpiifort # Fortran 77 compiler when using MPI
    MACH_F90_MPI   = mpiifort # Fortran 90 compiler when using MPI
    MACH_LD_MPI    = mpiicpc # Linker when using MPI

    # Without MPI

    MACH_CC_NOMPI  = gcc # C compiler when not using MPI
    MACH_CXX_NOMPI = g++ # C++ compiler when not using MPI
    MACH_FC_NOMPI  = gfortran # Fortran 77 compiler when not using MPI
    MACH_F90_NOMPI = gfortran # Fortran 90 compiler when not using MPI
    MACH_LD_NOMPI  = g++ # Linker when not using MPI

    #-----------------------------------------------------------------------
    # Machine-dependent defines
    #-----------------------------------------------------------------------

    MACH_DEFINES   = -DLINUX -DH5_USE_16_API 

    #-----------------------------------------------------------------------
    # Compiler flag settings
    #-----------------------------------------------------------------------

    MACH_CPPFLAGS = -P -traditional 
    MACH_CFLAGS   = 
    MACH_CXXFLAGS =
    MACH_FFLAGS   = -nofor-main
    MACH_F90FLAGS = -nofor-main
    # if using gfortran
    # MACH_FFLAGS   = -nofor_main -fno-second-underscore -ffree-line-length-none
    # MACH_F90FLAGS = -nofor_main -fno-second-underscore
    MACH_LDFLAGS  = -lifcore

    #-----------------------------------------------------------------------
    # Optimization flags
    #-----------------------------------------------------------------------

    MACH_OPT_WARN        = -Wall -g
    MACH_OPT_DEBUG       = -g
    MACH_OPT_HIGH        = -O2
    MACH_OPT_AGGRESSIVE  = -O3 -g

    #-----------------------------------------------------------------------
    # Includes
    #-----------------------------------------------------------------------

    LOCAL_INCLUDES_MPI         = -I$(LOCAL_MPI_INSTALL)/include 
    LOCAL_INCLUDES_HDF5        = -I$(LOCAL_HDF5_INSTALL)/include # HDF5 includes
    LOCAL_INCLUDES_HYPRE       = -I$(LOCAL_HYPRE_INSTALL)/include
    LOCAL_INCLUDES_PAPI        = # PAPI includes
    LOCAL_INCLUDES_GRACKLE     = -I$(LOCAL_GRACKLE_INSTALL)/include

    MACH_INCLUDES             = $(LOCAL_INCLUDES_HDF5)
    MACH_INCLUDES_MPI         = $(LOCAL_INCLUDES_MPI)
    MACH_INCLUDES_HYPRE       = $(LOCAL_INCLUDES_HYPRE)
    MACH_INCLUDES_PAPI        = $(LOCAL_INCLUDES_PAPI)
    MACH_INCLUDES_GRACKLE     = $(LOCAL_INCLUDES_GRACKLE)

    #-----------------------------------------------------------------------
    # Libraries
    #-----------------------------------------------------------------------

    LOCAL_LIBS_MPI        = -L$(LOCAL_MPI_INSTALL)/lib -L$(LOCAL_MPI_INSTALL)/lib/debug
    LOCAL_LIBS_HDF5       = -L$(LOCAL_HDF5_INSTALL)/lib -lhdf5 -lz
    LOCAL_LIBS_HYPRE      = -L$(LOCAL_HYPRE_INSTALL)/lib -lHYPRE
    LOCAL_LIBS_PAPI       = # PAPI libraries
    LOCAL_LIBS_MACH       = -lgfortran # Machine-dependent libraries
    LOCAL_LIBS_GRACKLE    = -L$(LOCAL_GRACKLE_INSTALL)/lib -lgrackle

    MACH_LIBS             = $(LOCAL_LIBS_HDF5) $(LOCAL_LIBS_MACH)
    MACH_LIBS_MPI         = $(LOCAL_LIBS_MPI)
    MACH_LIBS_HYPRE       = $(LOCAL_LIBS_HYPRE)
    MACH_LIBS_PAPI        = $(LOCAL_LIBS_PAPI)
    MACH_LIBS_GRACKLE     = $(LOCAL_LIBS_GRACKLE)