A quick word to share with you work that has been done on using Spark and Dask with PBS at CNES (French Spatial Agency). PBS scripts to launch Spark or Dask based cluster are available in this repo: https://github.com/guillaumeeb/big-data-frameworks-on-pbs.
Has anyone already done this? Do you have something to share?
Are there plan to add similar functionnality in PBS? I’ve already discussed a bit with @subhasisb some times ago, but I don’t know what is he current situation.
I will be happy to have any feedback on this, so fill free to answer or ask anything.