Compile OpenMPI with PBSpro 14.1.10


#1

Attempting to compile OpenMPI for use with my cluster running PBSPro 14.1.0 and receiving the following error from the configure command:

— MCA component ess:tm (m4 configuration macro)
checking for MCA component ess:tm compile mode… static
checking --with-tm value… sanity check ok (/opt/pbs)
checking for pbs-config… not found
checking tm.h usability… yes
checking tm.h presence… yes
checking for tm.h… yes
looking for library in lib
checking for library containing tm_init… no
looking for library in lib64
checking for library containing tm_init… no
looking for library in lib
checking for library containing tm_init… no
looking for library in lib64
checking for library containing tm_init… no
configure: error: TM support requested but not found. Aborting

Excerpt from config.log:

configure:133689: checking for library containing tm_init
configure:133720: gcc -std=gnu99 -o conftest -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -pthread -I/usr/local/src/openmpi-1.10.1/opal/mca/hwloc/hwloc191/hwloc/include -I/usr/local/src/openmpi-1.10.1/opal/mca/event/libevent2021/libevent -I/usr/local/src/openmpi-1.10.1/opal/mca/event/libevent2021/libevent/include -I/opt/pbs/include -L/opt/pbs/lib conftest.c -lrt -lm -lutil >&5
/tmp/ccPqRZ2s.o: In function main':** **conftest.c:(.text.startup+0x7): undefined reference totm_init’
collect2: error: ld returned 1 exit status
configure:133720: $? = 1

My configure line is as follows:
./configure --prefix=/usr/local/software/OpenMPI/ --with-tm=/opt/pbs --with-verbs

The /opt/pbs/include does have the tm.h file

Anyone have an idea as to what I’m missing here?

-SS-


#2

Forgot to include that I’m attempting this on a CentOS 7.2 server.


#3

That seems odd. Here’s what I’m seeing:

$ nm -D /opt/pbs/lib/libpbs.so | grep tm_init
00000000000398d0 T tm_init

It looks as though the exported symbol is present in the shared library. Do you see the same output on your system?

What version of OpenMPI are you using?


#4

Looking at the call to the compiler in the config.log you will notice that “-lpbs” is not one of the arguments provided. It looks like the method used to check for tm_init() defined in config/orte_check_tm.m4 relies on a script called pbs-config being present. It uses this script to gather information like CPPFLAGS, LDFLAGS, and LIBS that are required for linking the library. I think the easiest approach would be to create a new version of pbs-config for PBS Pro, rather than attempt to change OpenMPI.

Are you able to file a ticket in our JIRA system? It is located here: https://pbspro.atlassian.net/secure/Dashboard.jspa


TM api on pbspro does not work for me?
#5

This script should get you past the problem you are seeing. You should be able to install it anywhere in your path. I installed it as /opt/pbs/bin/pbs-config for my testing. Remember to make it executable.

#!/bin/sh
#
#  Copyright (C) 1994-2016 Altair Engineering, Inc.
#  For more information, contact Altair at www.altair.com.
#
#  This file is part of the PBS Professional ("PBS Pro") software.
#
#  Open Source License Information:
#
#  PBS Pro is free software. You can redistribute it and/or modify it under the
#  terms of the GNU Affero General Public License as published by the Free
#  Software Foundation, either version 3 of the License, or (at your option) any
#  later version.
#
#  PBS Pro is distributed in the hope that it will be useful, but WITHOUT ANY
#  WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
#  PARTICULAR PURPOSE.  See the GNU Affero General Public License for more details.
#
#  You should have received a copy of the GNU Affero General Public License along
#  with this program.  If not, see <http://www.gnu.org/licenses/>.
#
#  Commercial License Information:
#
#  The PBS Pro software is licensed under the terms of the GNU Affero General
#  Public License agreement ("AGPL"), except where a separate commercial license
#  agreement for PBS Pro version 14 or later has been executed in writing with Altair.
#
#  Altair’s dual-license business model allows companies, individuals, and
#  organizations to create proprietary derivative works of PBS Pro and distribute
#  them - whether embedded or bundled with other software - under a commercial
#  license agreement.
#
#  Use of Altair’s trademarks, including but not limited to "PBS™",
#  "PBS Professional®", and "PBS Pro™" and Altair’s logos is subject to Altair's
#  trademark licensing policies.

. ${PBS_CONF_FILE:-/etc/pbs.conf}

case "$1" in
--prefix)
        [ -d "$PBS_EXEC" ] || exit 1
        echo $PBS_EXEC
        ;;
--cflags)
        [ -d "$PBS_EXEC/include" ] || exit 1
        echo "-I$PBS_EXEC/include"
        ;;
--libs)
        libdir="$PBS_EXEC/lib"
        [ -d "$libdir" ] || libdir="$PBS_EXEC/lib64"
        [ -d "$libdir" ] || exit 1
        echo "-L$libdir -lpbs -lpthread -lcrypto"
        ;;
*)
        exit 1
        ;;
esac

Libpbs does not depend on libcrypto nor libpthread
#6

Filed JIRA ticket PP-360.

https://pbspro.atlassian.net/browse/PP-360


#7

Michael,
Thank you for the quick analysis and script to try and address the issue. I will be attempting the recompile later. One question though is why is there only a /opt/pbs/lib directory, but not a /opt/pbs/lib64 which you would expect to find with x86_64 compiles?

-SS-


#8

Script worked. I was able to successfully compile OpenMPI 2.0.0.


#9

WRT lib64, it depends on the Linux distro. On CentOS it uses lib by default, but on openSUSE it uses lib64. I don’t know why that is the case, but it has been my observation. We could force it to use one or the other by specifying parameters to the configure script, but we have chosen not to. It could cause problems if someone ever tried to port PBS Pro to a non-Linux OS.


#10

Thank you for the followup.


#11

Adding a pbs_config script is one way of making openmpi’s configure to succeed, but here also a simple manual way to do it:

export LD_LIBRARY_PATH=/opt/pbs/lib:$LD_LIBRARY_PATH
export LDFLAGS="-L/opt/pbs/lib -lpbs -lpthread -lcrypto"
./configure --with-tm=/opt/pbs

This should simply configure openmpi with PBS tm support. I tried with 2.0.2 as well.


#12

Hi mkaro,

  1. I have tried to compile OpenMpi (v1.10.6, v2.0.2, v2.1.0) but
    keep on encountering following error:
    “configure: error: TM support requested but not found. Aborting”

    pbspro: pbspro-server-14.1.0-13.1.x86_64 (install via rpm)
    openmpi: openmpi-2.1.0.tar.gz
    OS: Centos 7.1.1503

  2. 1st attempt:

    $ cd ~/openmpi-2.1.0
    $ ./configure --prefix=/opt1/openmpi-2.1.0 --with-tm=/opt/pbs

    — MCA component ess:tm (m4 configuration macro)
    checking for MCA component ess:tm compile mode… dso
    checking --with-tm value… sanity check ok (/opt/pbs)
    checking for pbs-config… not found <-- *** note this!!
    checking tm.h usability… yes
    checking tm.h presence… yes
    checking for tm.h… yes
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    configure: error: TM support requested but not found. Aborting

    <-- initial attempt failed!

  3. 2nd attempt:

    Ref:
    Compile OpenMPI with PBSpro 14.1.10

    <-- confirmed that the ‘tm_init’ is in ‘libpbs.so’ library
    $ nm -D /opt/pbs/lib/libpbs.so | grep tm_init
    00000000000397e0 T tm_init

    Follow “mkaro solution” to create “/opt/pbs/bin/pbs-config” script,
    then re-config the openmpi again

    $ cd ~/openmpi-2.1.0
    $ ./configure --prefix=/opt1/openmpi-2.1.0 --with-tm=/opt/pbs

    — MCA component ess:tm (m4 configuration macro)
    checking for MCA component ess:tm compile mode… dso
    checking --with-tm value… sanity check ok (/opt/pbs)
    checking for pbs-config… /opt/pbs/bin/pbs-config <-- *** note this!!
    checking tm.h usability… yes
    checking tm.h presence… yes
    checking for tm.h… yes
    checking for tm_finalize… no
    checking tm.h usability… yes
    checking tm.h presence… yes
    checking for tm.h… yes
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    looking for library in lib
    checking for library containing tm_init… no
    looking for library in lib64
    checking for library containing tm_init… no
    configure: error: TM support requested but not found. Aborting

    $ vi config.log

    configure:161051: result: /opt/pbs/bin/pbs-config
    configure:161064: orte_check_tm_CPPFLAGS from pbs-config: -I/opt/pbs/include
    configure:161145: orte_check_tm_LDFLAGS from pbs-config: -L/opt/pbs/lib
    configure:161226: orte_check_tm_LIBS from pbs-config: -lpbs -lpthread -lcrypto
    configure:161241: checking tm.h usability


    configure:161553: checking for library containing tm_init
    configure:161584:
    gcc -std=gnu99 -o conftest -O3 -DNDEBUG -finline-functions
    -fno-strict-aliasing -pthread
    -I/home/richardl/openmpi-2.1.0/opal/mca/hwloc/hwloc1112/hwloc/include
    -I/home/richardl/openmpi-2.1.0/opal/mca/event/libevent2022/libevent
    -I/home/richardl/openmpi-2.1.0/opal/mca/event/libevent2022/libevent/include
    -I/opt/pbs/include
    -L/opt/pbs/lib conftest.c -lrt -lm -lutil >&5/tmp/cc4fiK8N.o:
    In function main': conftest.c:(.text.startup+0x7): undefined reference totm_init’
    collect2: error: ld returned 1 exit status
    configure:161584: $? = 1

    <-- the script “/opt/pbs/bin/pbs-config” did execute properly
    <-- Looking at the call to the compiler in the config.log will
    notice that “-lpbs” is still not one of the arguments provided

    <-- So, your solution of creating "/opt/pbs/bin/pbs-config"
    did not solve the problem!!

    Is it the problem with the pbspro-14.1.0 rpm package? Any idea?
    Hope that you can point me to the right direction!

Rgds,
Richardl.


#13

In addition to the pbs-config and the LDFLAGS / LD_LIBRARY_PATH above, the error about “TM support requested but not found” can still occur.

Looking in the config.log will show:

configure:6266: gcc -L/opt/pbs/lib -lpbs -lpthread -lcrypto conftest.c >&5
/usr/bin/ld: cannot find -lcrypto
collect2: error: ld returned 1 exit status

The solution to this is to construct a symbolic link if libcrypto.so does not exist. (Thanks Alexis/Prasanna)
Check the listing first:
ls -al /usr/lib64/libcrypto*
19 May 12 13:43 /usr/lib64/libcrypto.so -> libcrypto.so.1.0.1e
1440448 Jun 24 2014 /usr/lib64/libcrypto.so.0.9.8e
19 Aug 20 2016 /usr/lib64/libcrypto.so.10 -> libcrypto.so.1.0.1e
2017168 May 9 2016 /usr/lib64/libcrypto.so.1.0.1e
19 May 27 2015 /usr/lib64/libcrypto.so.6 -> libcrypto.so.0.9.8e

In ‘this’ case, create a symbolic link:
cd /usr/lib64
ln -s libcrypto.so.1.0.1e libcrypto.so

Then configure should work, followed by the remaining process:
./configure --with-tm=/opt/pbs
make
make install


#14

To Compile an openMPI program:
mpiCC -g wave_mpi.cpp -o wave_mpi

Compiling a previously working MPI program will FAIL with the following:
… error: ‘MPI’ has not been declared

The reason is that a change from OpenMPI 1.x to 2.x correspond roughly to change in impementation of MPI v2 to MPI v3

Apparently MPI-3.x deleted C++ bindings so OpenMPI is not build by default with C++
https://github.com/open-mpi/ompi/issues/3173

Solution is:
make clean
make distclean
./configure --with-tm=/opt/pbs --enable-mpi-cxx
make
make install

Then to compile program:
mpiCC -g wave_mpi.cpp -o wave_mpi

THis can be submitted using a script:

(replace HASH_ with symbol #)

Submitting to pbs:
qsub wave_pbs.sh