PBS Pro (public version) Installation on Ubuntu 18.04.2 LTS (Bionic Beaver)

Hi,
Are you still getting following error while starting PBS.

or while executing any pbs commands like ‘pbsnodes -av’ ?

Dear kjakkali,

A the moment, I dont see any error when I start PBS and it says even it connected to dataserver@hotname. However, when I use a PBS command like qstat -B, I face a Error. The log file is as follows:

bash$ . /etc/profile.d/pbs.sh
bash$ sudo /etc/init.d/pbs start
Starting PBS
/opt/pbs/sbin/pbs_comm ready (pid=32097), Proxy Name:ehsan:17001, Threads:4
PBS comm
Creating usage database for fairshare.
PBS sched
Connecting to PBS dataservice…connected to PBS dataservice@Ehsan
Licenses valid for 10000000 Floating hosts
PBS server
bash$ qstat -B
Connection refused
qstat: cannot connect to server Ehsan (errno=111)

@ehsanam : One issue might be due to loop back ip-address.
Can you try adding non loop back IP address of the hostname entry into /etc/hosts file and try restart PBS?

Dear kjakkali,

Tnx again for your help. Now I can run the PBS commands using non-loop IP address. However, I face a problem regarding default queue:

bash$ echo “sleep 60” | qsub
bash$ qsub: No default queue specified.

I am wondering the default queue should be specified during installation or after installation?

@ehsanam : Good to know that.
Please run the pbs_postinstall (ex: libexec/pbs_postinstall) script which will create default queue.

@kjakkali As you suggested, I used the pbs_postinstall script and it solved the issue. Thanks a lot!
However, the job is not executed by qsub and its status is in Q mode. For example:

bash$ echo “sleep 60” | qsub
bash$ qstat -a

S = Q

Should I configure something else in PBS Pro?

@kjakkali

I guess it might be due to lack of resources. When I use pbsnodes -a command, it gives:

pbsnodes: Server has no node list

P.S: I am trying to install PBS Pro on my workstation which has 8 cores.

@kjakkali It got solved by:

root$ # source /etc/profile.d/pbs.sh
root$ # qmgr -c “create node hostname”

Dear ehsanam,
I also have the same problem. I don’t understand how to add non loop back IP address, can you show me more what exactly this means?
Here is my /etc/hosts
127.0.0.1 localhost
127.0.1.1 amax-server
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Please find this example:
cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
 192.168.100.101 pbsosserver.pbspro.org  pbsosserver 

If this does not help, please share your issues with /etc/hosts and PBS Pro OSS.

Dear adarsh,
Thank you very much for your help. By doing so, I run “qstat -B”, get the following result:
Server Max Tot Que Run Hld Wat Trn Ext Status


amax-server 0 0 0 0 0 0 0 0 Active
(1) **But I get the new error when I run: $ qmgr

ybwang@amax-server : ~ $ qmgr

Max open servers: 49

Qmgr: set queue batch queue_type = Execution

qmgr obj=batch svr=default: Unauthorized Request

qmgr: Error (15007) returned from server

Qmgr: set queue batch enabled = True

qmgr obj=batch svr=default: Unauthorized Request

qmgr: Error (15007) returned from server

Qmgr: set queue batch started = True

qmgr obj=batch svr=default: Unauthorized Request

qmgr: Error (15007) returned from server

Qmgr: set server default_queue = batch

qmgr obj= svr=default: Unauthorized Request

qmgr: Error (15007) returned from server

Qmgr:

(2) The error when I run “qsub”:
ybwang@amax-server : ~ $ echo “slepp 222” | qsub

qsub: No default queue specified

before run this, other output as following:

ybwang@amax-server:~$ sudo /opt/pbs/libexec/pbs_postinstall
[sudo] password for ybwang:
*** PBS Installation Summary


*** Postinstall script called as follows:
*** /opt/pbs/libexec/pbs_postinstall ‘’


*** Existing configuration file found: /etc/pbs.conf


*** Saving /etc/pbs.conf as /etc/pbs.conf.pre.19.0.0.20190926155700
*** Replacing /etc/pbs.conf with /etc/pbs.conf.19.0.0
*** /etc/pbs.conf has been modified.
*** The original contents have been saved to /etc/pbs.conf.pre.19.0.0.20190926155700


*** Registering PBS Pro as a service.
Synchronizing state of pbs.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable pbs


*** PBS_HOME is /var/spool/pbs
*** Existing environment file left unmodified: /var/spool/pbs/pbs_environment


*** The PBS Pro server has been installed in /opt/pbs/sbin.
*** The PBS Pro scheduler has been installed in /opt/pbs/sbin.


*** The PBS Pro communication agent has been installed in /opt/pbs/sbin.


*** The PBS Pro MOM has been installed in /opt/pbs/sbin.


*** The PBS commands have been installed in /opt/pbs/bin.


*** End of /opt/pbs/libexec/pbs_postinstall
ybwang@amax-server:~$ pbsnodes -a
amax-server
Mom = amax-server
ntype = PBS
state = free
pcpus = 24
resources_available.arch = linux
resources_available.host = amax-server
resources_available.mem = 32912876kb
resources_available.ncpus = 24
resources_available.vnode = amax-server
resources_assigned.accelerator_memory = 0kb
resources_assigned.hbmem = 0kb
resources_assigned.mem = 0kb
resources_assigned.naccelerators = 0
resources_assigned.ncpus = 0
resources_assigned.vmem = 0kb
resv_enable = True
sharing = default_shared
last_state_change_time = Thu Sep 26 15:47:23 2019

Sincerely looking forward to your reply, thanks a lot.

Please try these commands:

qmgr -c "create queue workq queue_type=e,enabled=t,started=t"

Run qmgr command and copy paste the below contents

set server scheduling = True
set server default_queue = workq
set server log_events = 511
set server mail_from = adm
set server query_other_jobs = True
set server resources_default.ncpus = 1
set server default_chunk.ncpus = 1
set server scheduler_iteration = 60
set server flatuid = True
set server resv_enable = True
set server node_fail_requeue = 310
set server max_array_size = 10000
set server eligible_time_enable = False
set server job_history_enable = True
set server max_concurrent_provision = 5
set server max_job_sequence_id = 9999999

Share the output of the below commands

qstat -Bf
pbsnodes -aSj
qmgr -c 'p s'
echo “sleep 222” | qsub
qstat -answ1 

Hope this helps, thank you

Thank you very much for your kindness, adarsh.

**When I run the commands: ** , the error still there. It seems qmgr doesn’t work.

ybwang@amax-server : ~ $ qmgr -c “create queue workq queue_type=e,enabled=t,started=t”

qmgr obj=workq svr=default: Unauthorized Request

qmgr: Error (15007) returned from server

You are welcome and Thank you.
Please share the output of the below commands:

ps -ef | grep pbs_
cat /etc/hosts
ifconfig 
ping <pbs server hostname>
nslookup  pbs-server-hostname
nslookup  pbs-server-ipaddress
cat /etc/pbs.conf

Dear adarsh, here are the output:

(1) ybwang@amax-server : ~ $ ps -ef | grep pbs_

root 7857 1 0 17:18 ? 00:00:00 /opt/pbs/sbin/pbs_comm
root 7892 1 0 17:18 ? 00:00:00 /opt/pbs/sbin/pbs_mom
root 7904 1 0 17:18 ? 00:00:00 /opt/pbs/sbin/pbs_sched
root 8039 1 0 17:18 ? 00:00:00 /opt/pbs/sbin/pbs_ds_monitor monitor
postgres 8075 8055 0 17:18 ? 00:00:00 postgres: postgres pbs_datastore 192.168.0.8(36996) idle
root 8076 1 0 17:18 ? 00:00:00 /opt/pbs/sbin/pbs_server.bin
ybwang 8349 8286 0 17:28 pts/1 00:00:00 grep --color=auto pbs_

(2) ybwang@amax-server : ~ $ cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.8 amax-server

#127.0.0.1 localhost

#localhost

#127.0.1.1 amax-server

The following lines are desirable for IPv6 capable hosts

#::1 ip6-localhost ip6-loopback

#fe00::0 ip6-localnet

#ff00::0 ip6-mcastprefix

#ff02::1 ip6-allnodes

#ff02::2 ip6-allrouters

(3) ybwang@amax-server : ~ $ ifconfig

enp6s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.0.8 netmask 255.255.255.0 broadcast 192.168.0.255
inet6 fe80::225:90ff:fe98:be68 prefixlen 64 scopeid 0x20
ether 00:25:90:98:be:68 txqueuelen 1000 (Ethernet)
RX packets 17528 bytes 1773726 (1.7 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2917 bytes 361518 (361.5 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xdfd20000-dfd3ffff

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 8307 bytes 1290865 (1.2 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8307 bytes 1290865 (1.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

wlx14cf92d8c354: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether 14:cf:92:d8:c3:54 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

(4) ybwang@amax-server : ~ $ ping amax-server

PING amax-server (192.168.0.8) 56(84) bytes of data.

64 bytes from amax-server (192.168.0.8): icmp_seq=1 ttl=64 time=0.033 ms

64 bytes from amax-server (192.168.0.8): icmp_seq=2 ttl=64 time=0.032 ms

64 bytes from amax-server (192.168.0.8): icmp_seq=3 ttl=64 time=0.032 ms

64 bytes from amax-server (192.168.0.8): icmp_seq=4 ttl=64 time=0.032 ms

(5) ybwang@amax-server : ~ $ nslookup pbs-server-hostname

Server: 127.0.0.53

Address: 127.0.0.53#53

** server can’t find pbs-server-hostname: SERVFAIL

(6) ybwang@amax-server : ~ $ nslookup pbs-server-ipaddress

Server: 127.0.0.53

Address: 127.0.0.53#53

** server can’t find pbs-server-ipaddress: SERVFAIL

(7) ybwang@amax-server : ~ $ cat /etc/pbs.conf

PBS_SERVER=amax-server

PBS_START_SERVER=1

PBS_START_SCHED=1

PBS_START_COMM=1

PBS_START_MOM=1

PBS_EXEC=/opt/pbs

PBS_HOME=/var/spool/pbs

PBS_CORE_LIMIT=unlimited

PBS_SCP=/usr/bin/scp

Thank you for this information

Please share the output of the below :
nslookup amax-server
nslookup 192.168.0.8

Please check

  • SELinux is disabled and system is rebooted , if it is disabled now
  • Firewall turned off and disabled ( 15001 to 15009 , 17001 ports are not blocked and open for communication )
  • 15007 is pbs_datastore port

Thank you

whoops,sorry for that.
(1) ybwang@amax-server : ~ $ nslookup amax-server

Server: 127.0.0.53

Address: 127.0.0.53#53

Non-authoritative answer:

Name: amax-server

Address: 192.168.0.8

ybwang@amax-server : ~ $ nslookup 192.168.0.8

8.0.168.192.in-addr.arpa name = amax-server.

Authoritative answers can be found from:

ybwang@amax-server : ~ $

(2.1) The system I used is Ubuntu1804, which should not install SELinux.

root@amax-server:~# /usr/sbin/sestatus -v

-su: /usr/sbin/sestatus: No such file or directory

(2.2) ybwang@amax-server : ~ $ sudo ufw status

Status: inactive

ybwang@amax-server : ~ $ sudo ufw allow 15001

Rules updated

Rules updated (v6)

Similar, I updated 15001-15009, and 17001 ports.

Please run the qmgr commands that displays unauthorized request
Please check and share the server logs $PBS_HOME/server_logs/YYYYMMDD

Dear adarsh, thanks a lot for your time and attention.
(1) root@amax-server:~# qmgr -c “create queue workq queue_type=e,enabled=t,started=t”

Unknown Host.

qmgr: cannot connect to server queue

Unknown Host.

qmgr: cannot connect to server workq

Unknown Host.

qmgr: cannot connect to server queue_type=e,enabled=t,started=t”

(2) root@amax-server:~# qmgr

Max open servers: 49

Qmgr: set queue batch queue_type = Execution

qmgr obj=batch svr=default: Unknown queue

qmgr: Error (15018) returned from server

Qmgr: set queue batch enabled = True

qmgr obj=batch svr=default: Unknown queue

qmgr: Error (15018) returned from server

Qmgr: set queue batch started = True

qmgr obj=batch svr=default: Unknown queue

qmgr: Error (15018) returned from server

Qmgr: set server default_queue = batch

qmgr obj= svr=default: Unknown queue

qmgr: Error (15018) returned from server

Qmgr:

(3) /var/spool/pbs/server_logs/20190928 as follows:

09/28/2019 09:48:04;0002;Server@amax-server;Svr;Server@amax-server;pbs_status_db exit code 0
09/28/2019 09:48:14;0002;Server@amax-server;Svr;Server@amax-server;Starting PBS dataservice
09/28/2019 09:48:45;0002;Server@amax-server;Svr;Server@amax-server;connected to PBS dataservice@amax-server
09/28/2019 09:48:46;0086;Server@amax-server;Svr;pbs_python_ext_quick_start_interpreter;–> Python Interpreter quick started, compiled with version:‘3.6.8 (default, Aug 20 2019, 17:12:48)
[GCC 8.3.0]’ <–
09/28/2019 09:48:46;0086;Server@amax-server;Svr;pbs_python_ext_quick_start_interpreter;–> Inserted Altair PBS Python modules dir ‘/opt/pbs/lib/python/altair’ ‘/opt/pbs/lib/python/altair/pbs/v1’<–
09/28/2019 09:48:46;0002;Server@amax-server;n/a;setup_env;read environment from /var/spool/pbs/pbs_environment
09/28/2019 09:48:46;0004;Server@amax-server;Svr;Server@amax-server;node_fail_requeue value changed to 310
09/28/2019 09:48:46;0004;Server@amax-server;Svr;Server@amax-server;svr_max_job_sequence_id set to val 9999999
09/28/2019 09:48:46;0004;Server@amax-server;Svr;Server@amax-server;Licenses valid for 10000000 Floating hosts
09/28/2019 09:48:46;0002;Server@amax-server;Svr;Act;Account file /var/spool/pbs/server_priv/accounting/20190928 opened
09/28/2019 09:48:46;0002;Server@amax-server;Svr;Server@amax-server;Expected 0, recovered 0 queues
09/28/2019 09:48:46;0080;Server@amax-server;Svr;Server@amax-server;No jobs to open
09/28/2019 09:48:46;0086;Server@amax-server;Svr;Server@amax-server;Found hook PBS_translate_mpp type=pbs
09/28/2019 09:48:46;0086;Server@amax-server;Svr;Server@amax-server;Found hook PBS_power type=pbs
09/28/2019 09:48:46;0086;Server@amax-server;Svr;Server@amax-server;Found hook PBS_cray_jacs type=pbs
09/28/2019 09:48:46;0086;Server@amax-server;Svr;Server@amax-server;Found hook pbs_cgroups type=site
09/28/2019 09:48:46;0086;Server@amax-server;Svr;Server@amax-server;Found hook PBS_alps_inventory_check type=pbs
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;ALLHOOKS hook[0] = {PBS_translate_mpp, order=1000, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(queuejob,resvsub), alarm=90, freq=120}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;ALLHOOKS hook[1] = {PBS_power, order=2000, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(periodic,execjob_begin,execjob_prologue,execjob_epilogue,execjob_end,exechost_periodic,exechost_startup), alarm=180, freq=300}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;ALLHOOKS hook[2] = {PBS_cray_jacs, order=100, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(execjob_begin,execjob_end), alarm=300, freq=120}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;ALLHOOKS hook[3] = {pbs_cgroups, order=100, type=0, enabled=0 user=0, debug=(0) fail_action=(2), event=(execjob_begin,execjob_epilogue,execjob_end,execjob_launch,execjob_attach,execjob_resize,execjob_abort,execjob_postsuspend,execjob_preresume,exechost_periodic,exechost_startup), alarm=90, freq=120}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;ALLHOOKS hook[4] = {PBS_alps_inventory_check, order=1, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(exechost_periodic), alarm=90, freq=300}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;queuejob hook[0] = {PBS_translate_mpp, order=1000, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(queuejob,resvsub), alarm=90, freq=120}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;resvsub hook[0] = {PBS_translate_mpp, order=1000, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(queuejob,resvsub), alarm=90, freq=120}
09/28/2019 09:48:46;0080;Server@amax-server;Hook;print_hook;periodic hook[0] = {PBS_power, order=2000, type=1, enabled=0 user=0, debug=(0) fail_action=(1), event=(periodic,execjob_begin,execjob_prologue,execjob_epilogue,execjob_end,exechost_periodic,exechost_startup), alarm=180, freq=300}
09/28/2019 09:48:46;0086;Server@amax-server;Svr;pbs_python_ext_quick_shutdown_interpreter;–> Stopping Python interpreter <–
09/28/2019 09:48:46;0d80;Server@amax-server;TPP;Server@amax-server(Main Thread);TPP set to use reserved port authentication
09/28/2019 09:48:46;0c06;Server@amax-server;TPP;Server@amax-server(Main Thread);TPP leaf node names = 192.168.0.6:15001,127.0.0.1:15001,192.168.0.6:15001
09/28/2019 09:48:46;0d80;Server@amax-server;TPP;Server@amax-server(Main Thread);Initializing TPP transport Layer
09/28/2019 09:48:46;0d80;Server@amax-server;TPP;Server@amax-server(Main Thread);Max files allowed = 16384
09/28/2019 09:48:46;0d80;Server@amax-server;TPP;Server@amax-server(Main Thread);TPP initialization done
09/28/2019 09:48:46;0c06;Server@amax-server;TPP;Server@amax-server(Main Thread);Single pbs_comm configured, TPP Fault tolerant mode disabled
09/28/2019 09:48:46;0d80;Server@amax-server;TPP;Server@amax-server(Main Thread);Connecting to pbs_comm amax-server:17001
09/28/2019 09:48:46;0002;Server@amax-server;Svr;Server@amax-server;Server pid = 9910 ready; using ports Server:15001 Scheduler:15004 MOM:15002 RM:15003
09/28/2019 09:48:46;0c06;Server@amax-server;TPP;Server@amax-server(Thread 0);Thread ready
09/28/2019 09:48:46;0c06;Server@amax-server;TPP;Server@amax-server(Thread 0);Registering address 192.168.0.6:15001 to pbs_comm
09/28/2019 09:48:46;0c06;Server@amax-server;TPP;Server@amax-server(Thread 0);Connected to pbs_comm amax-server:17001
09/28/2019 09:48:46;0100;Server@amax-server;Svr;Server@amax-server;–> Python Interpreter started, compiled with version:‘3.6.8 (default, Aug 20 2019, 17:12:48)
[GCC 8.3.0]’ <–
09/28/2019 09:48:46;0100;Server@amax-server;Svr;Server@amax-server;sys.modules= {‘builtins’: <module ‘builtins’ (built-in)>, ‘sys’: <module ‘sys’ (built-in)>, ‘_frozen_importlib’: <module ‘_frozen_importlib’ (frozen)>, ‘_imp’: <module ‘_imp’ (built-in)>, ‘_warnings’: <module ‘_warnings’ (built-in)>, ‘_thread’: <module ‘_thread’ (built-in)>, ‘_weakref’: <module ‘_weakref’ (built-in)>, ‘_frozen_importlib_external’: <module ‘_frozen_importlib_external’ (frozen)>, ‘_io’: <module ‘io’ (built-in)>, ‘marshal’: <module ‘marshal’ (built-in)>, ‘posix’: <module ‘posix’ (built-in)>, ‘zipimport’: <module ‘zipimport’ (built-in)>, ‘encodings’: <module ‘encodings’ from ‘/usr/lib/python3.6/encodings/init.py’>, ‘codecs’: <module ‘codecs’ from ‘/usr/lib/python3.6/codecs.py’>, ‘_codecs’: <module ‘_codecs’ (built-in)>, ‘encodings.aliases’: <module ‘encodings.aliases’ from ‘/usr/lib/python3.6/encodings/aliases.py’>, ‘encodings.ascii’: <module ‘encodings.ascii’ from ‘/usr/lib/python3.6/encodings/ascii.py’>, ‘_signal’: <module ‘_signal’ (built-in)>, ‘main’: <module ‘main’ (built-in)>, ‘encodings.utf_8’: <module ‘encodings.utf_8’ from ‘/usr/lib/python3.6/encodings/utf_8.py’>, ‘encodings.latin_1’: <module ‘encodings.latin_1’ from ‘/usr/lib/python3.6/encodings/latin_1.py’>, ‘io’: <module ‘io’ from ‘/usr/lib/python3.6/io.py’>, ‘abc’: <module ‘abc’ from ‘/usr/lib/python3.6/abc.py’>, ‘_weakrefset’: <module ‘_weakrefset’ from ‘/usr/lib/python3.6/_weakrefset.py’>, ‘pbs’: <module ‘pbs’ from ‘/opt/pbs/lib/python/altair/pbs/init.py’>, ‘_pbs_v1’: <module ‘_pbs_v1’ (built-in)>, ‘pbs.v1’: <module ‘pbs.v1’ from ‘/opt/pbs/lib/python/altair/pbs/v1/init.py’>, ‘pbs.v1._export_types’: <module ‘pbs.v1._export_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_export_types.py’>, ‘pbs.v1._base_types’: <module ‘pbs.v1._base_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_base_types.py’>, ‘math’: <module ‘math’ (built-in)>, ‘pbs.v1._exc_types’: <module ‘pbs.v1._exc_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_exc_types.py’>, ‘pbs.v1._svr_types’: <module ‘pbs.v1._svr_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_svr_types.py’>, ‘os’: <module ‘os’ from ‘/usr/lib/python3.6/os.py’>, ‘errno’: <module ‘errno’ (built-in)>, ‘stat’: <module ‘stat’ from ‘/usr/lib/python3.6/stat.py’>, ‘_stat’: <module ‘_stat’ (built-in)>, ‘posixpath’: <module ‘posixpath’ from ‘/usr/lib/python3.6/posixpath.py’>, ‘genericpath’: <module ‘genericpath’ from ‘/usr/lib/python3.6/genericpath.py’>, ‘os.path’: <module ‘posixpath’ from ‘/usr/lib/python3.6/posixpath.py’>, ‘_collections_abc’: <module ‘_collections_abc’ from ‘/usr/lib/python3.6/_collections_abc.py’>, ‘pbs.v1._pmi_types’: <module ‘pbs.v1._pmi_types’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_pmi_types.py’>, ‘pbs.v1._pmi_utils’: <module ‘pbs.v1._pmi_utils’ from ‘/opt/pbs/lib/python/altair/pbs/v1/_pmi_utils.py’>}
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;BEGIN setting up all resource attributes
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;DONE setting up all resource attributes, number set <58>
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;BEGIN setting up all queue attributes
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;DONE setting up all queue attributes, number set <56>
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;BEGIN setting up all job attributes
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;DONE setting up all job attributes, number set <109>
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;BEGIN setting up all server attributes
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;DONE setting up all server attributes, number set <94>
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;BEGIN setting up all reservation attributes
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;DONE setting up all reservation attributes, number set <43>
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;BEGIN setting up all vnode attributes
09/28/2019 09:48:46;0106;Server@amax-server;Svr;Server@amax-server;DONE setting up all vnode attributes, number set <35>
09/28/2019 09:48:49;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:48:52;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:48:55;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:48:55;0d80;Server@amax-server;TPP;Server@amax-server(Main Thread);net restore handler called
09/28/2019 09:48:58;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:48:58;0002;Server@amax-server;Node;amax-server;node down: communication closed
09/28/2019 09:49:01;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:04;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:07;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:11;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:14;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:17;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:20;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:23;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:26;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:29;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:32;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:35;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:38;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:41;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:44;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:47;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:50;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:54;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:49:57;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:00;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:03;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:06;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:09;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:09;0100;Server@amax-server;Req;;Type 0 request received from root@amax-server, sock=15
09/28/2019 09:50:12;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:15;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:15;0100;Server@amax-server;Req;;Type 49 request received from root@amax-server, sock=16
09/28/2019 09:50:18;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:21;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:24;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:27;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:30;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:33;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:37;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:40;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:43;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:46;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:49;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:52;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:55;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:50:58;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:01;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:04;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:04;0100;Server@amax-server;Req;;Type 0 request received from root@amax-server, sock=15
09/28/2019 09:51:07;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:10;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:10;0100;Server@amax-server;Req;;Type 49 request received from root@amax-server, sock=16
09/28/2019 09:51:13;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:16;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:16;0100;Server@amax-server;Req;;Type 9 request received from root@amax-server, sock=15
09/28/2019 09:51:16;0080;Server@amax-server;Req;req_reject;Reject reply code=15018, aux=0, type=9, from root@amax-server
09/28/2019 09:51:20;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:23;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:24;0100;Server@amax-server;Req;;Type 9 request received from root@amax-server, sock=15
09/28/2019 09:51:24;0080;Server@amax-server;Req;req_reject;Reject reply code=15018, aux=0, type=9, from root@amax-server
09/28/2019 09:51:26;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:29;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:30;0100;Server@amax-server;Req;;Type 9 request received from root@amax-server, sock=15
09/28/2019 09:51:30;0080;Server@amax-server;Req;req_reject;Reject reply code=15018, aux=0, type=9, from root@amax-server
09/28/2019 09:51:32;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:35;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:37;0100;Server@amax-server;Req;;Type 9 request received from root@amax-server, sock=15
09/28/2019 09:51:38;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:41;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:44;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:47;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:50;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:53;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:56;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:51:59;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:03;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:06;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:09;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:12;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:15;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:18;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:21;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:24;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:27;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:30;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:33;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:36;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:39;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:43;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:46;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:49;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:52;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:55;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:52:58;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:01;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:04;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:07;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:10;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:13;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:16;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:19;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:22;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:26;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:29;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:32;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:35;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:38;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:41;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:44;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:47;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:50;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:53;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:56;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:53:59;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:54:02;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler
09/28/2019 09:54:05;0001;Server@amax-server;Svr;Server@amax-server;Operation now in progress (115) in contact_sched, Could not contact Scheduler

Thank you for the above information.

The below log message refers to this issue

https://pbspro.atlassian.net/browse/PP-1083

The IP Address registered seems to have 192.168.0.6, which is not seen mapped to any hostname in the /etc/hosts

Would it be possible to disable the firewall complete, do a uninstall and fresh install of PBS Pro OSS.