OSG-SEC-2023-09-26 CRITICAL PMIx race condition vulnerability affecting Slurm
Dear OSG Security Contacts,
A CRITICAL risk vulnerability concerning PMIx has been discovered, which may permit a malicious user to obtain ownership of an arbitrary file and could thus result in privilege escalation to root.    This is a critical issue for all sites that are using Slurm built with PMIx support.
OpenPMIx PMIx before 4.2.6 and 5.0.x before 5.0.1
WHAT ARE THE VULNERABILITIES:
CVE-2023-41915 A filesystem race condition could permit a malicious user to obtain ownership of an arbitrary file on the filesystem when parts of the PMIx library is called by a process running as uid 0. This may happen under the default configuration of certain workload managers, including Slurm. 
WHAT YOU SHOULD DO:
First check if Slurm was built with PMIx support by running:
$ srun --mpi=list
Typical output will be something like
MPI plugin types are... cray_shasta none pmi2 pmix specific pmix plugin versions available: pmix_v3,pmix_v4
This will tell you whether Slurm is built with PMIx support and if so which version of PMIx.
If the command returns no pmix option, your Slurm installation can be assumed to be unaffected by this CVE.
Otherwise, upgrade PMIx to the fixed releases v4.2.6 or v5.0.1 .
Note that there is at least one ABI-breaking change between versions 4.2.3 and 4.2.6, which means that upgrading to 4.2.6 requires Slurm packages to be rebuilt.
If Slurm upgrade isn't an option, you can disable PMIx support by removing the mpi_pmix*.so libraries on the compute nodes and adjusting the MpiDefault setting in your Slurm configuration.
You can also patch your current PMIx version by replacing the chown function with lchown in the source and rebuilding the PMIx rpm . After installing the patched PMIx, this command shouldn't return any result:
objdump -t /usr/lib64/libpmi*.so* | grep chown
After the installation, slurmd on the compute nodes needs to be restarted.
Note that all versions prior to PMIx 4.2.6 are vulnerable, but some older PMIx versions are no longer supported and will not be patched.
Thanks to the EGI CSIRT for their alert and write up on this issue. 
OSG was alerted to this vulnerability by the EGI CSIRT.
Please contact the OSG security team at [email protected] if you have any questions or concerns.
OSG Security Team