OSG-SEC-2024-10-03 IDTOKEN Signing Key Present In OSG Hosted-CE Container Images

Dear OSG Security Contacts,

OSG has discovered a security issue with the OSG Hosted-CE container images [1] where a default IDTOKEN signing key was generated each time the images were built. This key could have been used to submit local jobs to the Hosted-CEs until a new image, containing a new key, was generated.

Upon discovery of the issue, we investigated our audit logs and found no evidence of job submission using this key. We have made changes to our container infrastructure to mitigate this issue and prevent the automatically generated key from being used.

We are investigating further improvements to harden the Hosted-CEs to make access to an IDTOKEN signing key less impactful. Additionally, we are investigating methods and tools to implement automated secret scanning for OSG container images and other release artifacts to reduce the likelihood of future secrets being included in release artifacts.

While we have no evidence that this issue was ever exploited, out of an abundance of caution we are rotating ALL SSH keys used by the Hosted-CEs to connect back to sites. OSG is working with the affected sites to minimize any disruptions caused by this credential rotation.

Please contact the OSG Security team at [email protected] if you have any questions or concerns.

OSG Security Team

REFERENCES

[1] https://hub.docker.com/r/opensciencegrid/hosted-ce