OSG 23 News¶
Supported OS Versions: EL8, EL9 (see this document for details)
OSG 23 is the first release series that follows our new support policy, which aims to increase the regularity of the OSG Software Stack lifecycle. Moving forward, we plan to distribute a new release series each year, supporting each release series for approximately 2 years total. As with OSG 3.6, we will continue to release OSG 23 package updates in a rolling fashion.
Additionally, OSG 23 aligns the OSG and HTCondor Software Suite (HTCSS) release cycles:
- OSG 23 main Yum repositories will contain HTCSS LTS series (HTCondor 23.0, HTCondor-CE 23.0)
- OSG 23 upcoming Yum repositories will contain HTCSS feature series (HTCondor 23.x, HTCondor-CE 23.x)
Known Issues¶
The following issues are known to currently affect packages distributed in OSG 23:
CA Certificates on EL9¶
EL9 operating systems have a tighter default cryptographic policy that can cause services to reject certificates issued
by SHA-1 signed CAs.
Some CAs in the igtf-ca-certs
and osg-ca-certs
packages are affected and you may see service issues if your server
certificate or certificates presented by clients are issued by these CAs.
The Software Team is investigating solutions but in the meantime, we recommend running the following command on XRootD
hosts to accept certificates issued by SHA-1 signed CAs:
root@host # update-crypto-policies --set DEFAULT:SHA1
Do I need to run this on my Compute Entrypoint (CE) hosts?
No. At this time, the Software Team believes that CE hosts are unaffected since their clients only present tokens and token issuers present modern CAs.
Latest News¶
Incoming changes to osg-ca-certs
To address an issue with certificates issued by SHA1 CAs on EL9 systems and EL8 systems configured to use the
FUTURE
system crypto policy,
the OSG Software Team intends to release a fix to the osg-ca-certs
package in the near future.
-
If you are running OSG-supported software (e.g., HTCondor, XRootD), update to the new version of
osg-ca-certs
. -
If you are running non-OSG-supported, Java-based software (e.g., dCache) with
osg-ca-certs
, repleaceosg-ca-certs
withosg-ca-certs-java
with the following command:s
root@host # yum swap osg-ca-certs osg-ca-certs-java
osg-ca-certs-java
is compatible with Java-based software but does not include the aforementioned fix.
November 30, 2023: VO Package v133, IGTF 1.125¶
- VO Package v133
- Update certificates for FNAL and GlueX VOMS servers
- CA certificates based on IGTF 1.125
- Updated root certificate ArmeSFo CA with extended validity (AM)
November 16, 2023: VO Package v132, XRootD 5.6.3, osg-ce 23-2, HTCondor-CE 23.0.1, osg-system-profiler 1.7.0¶
- VO Package v132
- Update certificates for FNAL and SLAC VOMS servers
- Update certificates for CLAS12, EIC, GLOW, and HCC
- Drop stale certificates for nanohub, STAR, and wisc.edu lz
- XRootD 5.6.3
- Fix parsing of chunked PUT requests
- Add HTTP TPC packet marking
- Differentiate between push and pull TPC error messages
- Use configured CA path for the SciTokens plugin
- osg-ce meta package
- Correctly set value of
OSG_RELEASE_SERIES
attribute for OSG 23
- Correctly set value of
- HTCondor-CE 23.0.1
- Add
condor_ce_test_token
command
- Add
- osg-system-profiler 1.7.0
- Add system cryptographic policy
- Better XRootD configuration information for generated profile
November 2, 2023: IGTF 1.124, CVMFS 2.11.2, cvmfs-x509-helper 2.4¶
- CA certificates based on IGTF 1.124
- Updated contact meta-data for ArmeSFo authority (AM)
- Removed discontinued AEGIS authority (RS)
- Removed suspended KENET Root and issuing CAs (KE)
- Removed suspended SDG-G2 authority (CN)
- Removed suspended CNIC authority (CN)
- Removed all four discontinued DigitalTrust CAs operated by their issuer (AE)
- CVMFS 2.11.2
- Bug fix release
- cvmfs-x509-helper 2.4
- Important bug fix for reading credentials from within an unprivileged user namespace such as unprivileged apptainer users. This is needed due to a change in recent el8 & el9 kernels.
October 31, 2023: HTCondor 23.0.1 LTS; Upcoming: HTCondor 23.1.0¶
- HTCondor 23.0.1 LTS
- Update to apptainer version 1.2.4 in the HTCondor tarballs
- Fix 10.6.0 bug that broke PID namespaces
- Fix bug where execution times for ARC CE jobs were 60 times too large
- Fix bug where a failed 'Service' node would crash DAGMan
- Condor-C and Job Router jobs now get resources provisioned updates
- Upcoming: HTCondor 23.1.0
- Enhanced filtering with
condor_watch_q
- The Access Point can now be told to use a non-standard ssh port when sending jobs to a remote scheduling system (such as Slurm)
- Laid groundwork to allow an Execution Point running without root access to accurately limit the job's usage of CPU and Memory in real time via Linux kernel cgroups; this is particularly interesting for glidein pools
- HTCondor file transfers using HTTPS can now utilize CA certificates in a non-standard location
- All the fixes from HTCondor 23.0.1
- Enhanced filtering with
October 26, 2023: CVMFS 2.11.1-1.3, XRootD 5.6.2-2.3, osg-update-vos 1.4.2-2¶
- CVMFS 2.11.1-1.3
- Important fix to bug impacting osgstorage.org repositories introduced in 2.11.0 -- all 2.11.0 installations should upgrade urgently
- Fix race conditions on concurrent fuse3 mounts
- XRootD 5.6.2-2.3
- Update to -2.3 release to avoid confusion with upstream -2 release
- Fix a bug with parsing compound IDs in authfiles
- osg-update-vos 1.4.2-2
- tarballs now contain cpio, so osg-update-vos will work
October 3, 2023: Initial Release¶
OSG 3.6 retirement
As part of our transition to our new series release cadence, we are planning to end support for OSG 3.6 on 30 June 2024 to align with the EL7 end-of-life. See our release series life-cycle table for details.
The initial release of OSG 23 contains major package updates, package removals, and new container images. All other packages may have received minor version and/or packaging updates compared to OSG 3.6.
Major package updates¶
This release contains the following major package updates compared to the current OSG 3.6 release:
-
HTCondor 23.0.0: an update from 10.0.8 in OSG 3.6 main and 10.8.0 in OSG 3.6 upcoming.
- New features
- A
condor_startd
without any slot types defined will now default to a single partitionable slot rather than a number of static slots equal to the number of cores as it was in previous versions. The configuration template useFEATURE : StaticSlots
was added for admins wanting the old behavior. - The
TargetType
attribute is no longer a required attribute in most Classads. It is still used for queries to thecondor_collector
and it remains in the Job ClassAd and the Machine ClassAd because of older versions of HTCondor require it to be present. - The
-dry-run option
ofcondor_submit
will now print the output of aSEC_CREDENTIAL_STORER script
. This can be useful when developing such a script. - Added ability to query epoch history records from the python bindings.
- The default value of
SEC_DEFAULT_AUTHENTICATION_METHODS
will now be visible incondor_config_val
. The default forSEC_*_AUTHENTICATION_METHODS will
inherit from this value, and thus noREAD
andCLIENT
will no longer automatically haveCLAIMTOBE
. - Added new tool
condor_test_token
, which will create a SciToken with configurable contents (including issuer) which will be accepted for a short period of time by the local HTCondor daemons.
- A
- Bug fixes
- Fixed a bug that would cause the condor_startd to crash in rare cases when jobs go on hold
- Fixed a bug where if a user-level checkpoint could not be transferred from the starter to the AP, the job would go on hold Now it will retry, or go back to idle
- Fixed a bug where the
CommittedTime
attribute was not set correctly for Docker Universe jobs doing user level check-pointing - Fixed a bug where
condor_preen
was deleting files named 'OfflineAds' in the spool directory - Fixed a bug where the blahpd would incorrectly believe that an LSF batch scheduler was not working
- Fixed the Execution Point’s detection of whether libvirt is working properly for the VM universe
- Fixed a bug where container universe did not work for late materialization jobs submitted to the
condor_schedd
- Fixed a bug where the condor_startd could crash if a new match is made at the end a drain request
- New features
-
HTCondor-CE 23.0.0: an update from 6.0.0 in OSG 3.6 main.
Job router configuration deprecation
The configuration macros
JOB_ROUTER_DEFAULTS
,JOB_ROUTER_ENTRIES
,JOB_ROUTER_ENTRIES_CMD
, andJOB_ROUTER_ENTRIES_FILE
are deprecated and will be removed for V24 of HTCondor. New configuration syntax for the job router is defined usingJOB_ROUTER_ROUTE_NAMES
andJOB_ROUTER_ROUTE_<name>
. Note: The removal will occur during the lifetime of the HTCondor V23 feature series, i.e. the versions that will be available in OSG upcoming repositories.- Adds deprecation warnings for old job router configuration syntax
- Adds grid CA and host certificate/key locations to default SSL search paths
- Verifies that HTCondor-CE can access the local HTCondor's
SPOOL
directory on startup condor_ce_upgrade_check
checks compatibility with HTCondor 23- Adds an option to allow running
condor_ce_trace
without a SciToken for testing batch system integration
-
XRootD 5.6.2: an update from XRootD 5.5.5 in OSG 3.6 main.
- New Features
- Add xrdfs cache subcommand to allow for cache evictions
- Better handling of unicode strings in the API
- Add gsi option to display DN when it differs from entity name
- Allow specfication of minimum and maximum creation mode
- Make maxfd be configurable (default is 256k)
- Include token information in the monitoring stream (phase 1)
- Implement a file evict function
- Increase default number of parallel event loops to 10
- xrdcp: number of parallel copy jobs increased from 4 to 128
- Allow XRootD to return trailers indicating failure
- Denote Accept-Ranges in HEAD response
- Report cache object age for caching proxy mode
- Allow origin to be a directory of a locally mounted file system
- Implement ability to have the token username as a separate claim
- Use SHA-256 for signatures, and message digest algorithm
- Allow option '-tokenlib none' to disable token validation
- Allow to point to a token file using CGI
'?xrd.ztn=tokenfile'
- Major bug fixes
- Fix SEGV in case request has object for opaque data but no content
- Fix memory leaks in GSI authentication
- Fix chunked PUT creating empty files
- New Features
-
GlideinWMS 3.10.5: an update from 3.10.1 in OSG 3.6 main
If you are using custom setup scripts...
If you are using custom setup scripts please change the use of
glidein_config
:- Custom scripts should always read values via
gconfig_get()
. The only exception is the parsing of the line to get theadd_config_line
source file. add_config_line
is deprecated in favor ofgconfig_add
.add_config_line
will be removed from future versions.add_config_line_safe
is deprecated in favor ofgconfig_add_safe
.gconfig_add
is the recommended method to use also in concurrent scripts.- Custom scripts in Python should
import gconfig.py
(compatible with both Python 2 and 3) and use the provided functions or classes:gconfig_get
,gconfig_add
, etc.
- This release completes EL9 and Python 3.9 support
- Added structured logging
- Various OSG_Autoconf improvements
- Fixed bugs with Python 3.9 and rrdtools failures with missing ClassAds and monitoring
- Custom scripts should always read values via
Package Removals¶
The following packages were removed from OSG 23 compared to OSG 3.6:
blahp
: available as part of thecondor
packageoidc-agent
: available in EPELpython-jwt
: available in EPELpython-scitokens
: available in EPELrrdtool
available from OS repositories
Container Images¶
The following container images have new tags for OSG 23:
Image name | Tags |
---|---|
hub.opensciencegrid.org/opensciencegrid/atlas-xcache |
23-release , 23-testing |
hub.opensciencegrid.org/opensciencegrid/cms-xcache |
23-release , 23-testing |
hub.opensciencegrid.org/opensciencegrid/frontier-squid |
23-release , 23-testing |
hub.opensciencegrid.org/opensciencegrid/oidc-agent |
23-release , 23-testing |
hub.opensciencegrid.org/opensciencegrid/osgvo-docker-pilot |
23-release , 23-testing |
hub.opensciencegrid.org/opensciencegrid/stash-cache |
23-release , 23-testing |
hub.opensciencegrid.org/opensciencegrid/stash-origin |
23-release , 23-testing |
For example, to retreive an OSG 23 backfill container image, run the following command:
docker pull hub.opensciencegrid.org/opensciencegrid/osgvo-docker-pilot:23-release
For more details on OSG container images, see our policy document.
Announcements¶
Updates to critical packages also announced by email and are sent to the following recipients and lists: