Skip to content

Updating to OSG 3.5

OS Version Support

OSG 3.5 only supports EL7

If you have an existing installation based on OSG release version <= 3.4 (which will be referred to as the old series), and want to upgrade to 3.5 (the new series), we recommend the following procedure:

  1. First, remove the old series yum repositories:

    [email protected] # rpm -e osg-release

    This step ensures that any local modifications to *.repo files will not prevent installing the new series repos. Any modified *.repo files should appear under /etc/yum.repos.d/ with the *.rpmsave extension. After installing the new OSG repositories (the next step) you may want to apply any changes made in the *.rpmsave files to the new *.repo files.

  2. Install the OSG repositories:

    [email protected] # yum install https://repo.opensciencegrid.org/osg/3.5/osg-3.5-el7-release-latest.rpm

  3. Clean yum cache:

    [email protected] # yum clean all --enablerepo=*

  4. Update software:

    [email protected] # yum update

    Info

    • Please be aware that running yum update may also update other RPMs. You can exclude packages from being updated using the --exclude=[package-name or glob] option for the yum command.
    • Watch the yum update carefully for any messages about a .rpmnew file being created. That means that a configuration file had been edited, and a new default version was to be installed. In that case, RPM does not overwrite the edited configuration file but instead installs the new version with a .rpmnew extension. You will need to merge any edits that have made into the .rpmnew file and then move the merged version into place (that is, without the .rpmnew extension). Watch especially for /etc/lcmaps.db, which every site is expected to edit.

  5. Remove any deprecated packages that were previously installed:

    [email protected] # yum remove osg-version \
                       osg-control \
                       'rsv*' \
                       glite-ce-cream-client-api-c \
                       glite-lbjp-common-gsoap-plugin \
                       xacml

    If you did not have any of the above packages installed, Yum will not remove any packages:

    No Match for argument: osg-version
No Match for argument: osg-control
No Match for argument: rsv*
No Match for argument: glite-ce-cream-client-api-c
No Match for argument: glite-lbjp-common-gsoap-plugin
No Match for argument: xacml
No Packages marked for removal

  6. If you are updating an HTCondor-CE host, please consult the manual HTCondor and OSG Configure instructions below.

Running into issues?

If you are not having the expected result or having problems with Yum please see the Yum troubleshooting guide

Updating to HTCondor-CE 4.x

The OSG 3.5 release series contains HTCondor-CE 4, a major version upgrade from the previously released versions in the OSG. See the HTCondor-CE 4.0.0 release notes for an overview of the changes. In particular, this version includes a major reorganization of the default configuration so updates will require manual intervention. To update your HTCondor-CE host(s), perform the following steps:

  1. Update all CE packages:

    [email protected] # yum update osg-ce 'osg-ce*'

  2. The new default condor_mapfile is sufficient since HTCondor-CE no longer relies on GSI authentication between its daemons. If /etc/condor-ce/condor_mapfile.rpmnew exists, replace your old condor_mapfile with the .rpmnew version:

    [email protected] # mv /etc/condor-ce/condor_mapfile.rpmnew /etc/condor-ce/condor_mapfile

  3. Merge any *.rpmnew files in /etc/condor-ce/config.d/

  4. Additionally, you may wish to make one or more of the following optional changes:

    • HTCondor-CE now disables batch system job retries by default. To re-enable job retries, set the following configuration in /etc/condor-ce/config.d/99-local.conf:

      ENABLE_JOB_RETRIES = True

    • For non-HTCondor sites that use remote CE requirements, the new version of HTCondor-CE accepts a simplified format. For example, a snippet from an example job route in the old format:

      set_default_remote_cerequirements = strcat("Walltime == 3600 && AccountingGroup =="", x509UserProxyFirstFQAN, "\"");

      May be rewritten as the following:

      set_WallTime = 3600;
set_AccountingGroup = x509UserProxyFirstFQAN;
set_default_CERequirements = "Walltime,AccountingGroup";

  5. Reload and restart the HTCondor-CE daemons:

    [email protected] # systemctl daemon-reload
[email protected] # systemctl restart condor-ce

Updating to HTCondor 8.8.x

The OSG 3.5 release series contains HTCondor 8.8, a major version upgrade from the previously released versions in the OSG. See the detailed update instructions below to update to HTCondor 8.8.

Updating to OSG Configure 3

The OSG 3.5 release series contains OSG-Configure 3, a major version upgrade from the previously released versions in the OSG. See the OSG Configure release notes for an overview of the changes. To update OSG Configure on your HTCondor-CE, perform the following steps:

  1. If you haven't already, update to OSG 3.5.

  2. If you have site_name set in /etc/osg/config.d/40-siteinfo.ini, delete it and specify resource instead. resource should match the resource name that's registered in OSG Topology.

  3. Set resource_group in /etc/osg/config.d/40-siteinfo.ini to the resource group registered in OSG Topology, i.e. the name of the .yaml file in OSG Topology that contains the registered resource above.

  4. Set host_name to the host name that is registered in OSG Topology. This may be different from the FQDN of the host if you're using a DNS alias, for example.

  5. OSG Configure will warn about config options that it does not recognize; delete these options from the config to get rid of the warnings.

Updating to HTCondor-CE 5

HTCondor-CE 5 is a major version upgrade from HTCondor-CE 4, available through the OSG upcoming repository. HTCondor-CE 5 provides improved support for SciTokens and WLCG Tokens as well as support for a new Job Router configuration syntax. To update HTCondor-CE 5, perform the following steps:

  1. Update your HTCondor-CE packages:

    [email protected] # yum update htcondor-ce --enablerepo=osg-upcoming

    Updating to HTCondor 9.0.x

    This update may also pull in updates to HTCondor 9.0. To update, consult the upgrade section below, especially if you use HTCondor for your local batch system.

  2. If you support the OSG, GLOW, or ATLAS VOs and map their jobs to non-standard local Unix accounts add SciTokens mappings to a file in /etc/condor-ce/mapfiles.d/:

    # OSG
SCITOKENS /^https\:\/\/scitokens\.org\/osg\-connect,/ osg
# GLOW
SCITOKENS /^https\:\/\/chtc\.cs\.wisc\.edu,/ glow
# ATLAS production
SCITOKENS /^https:\/\/atlas-auth.web.cern.ch\/,7dee38a3-6ab8-4fe2-9e4c-58039c21d817/ usatlas1
# ATLAS analysis
SCITOKENS /^https:\/\/atlas-auth.web.cern.ch\/,750e9609-485a-4ed4-bf16-d5cc46c71024/ usatlas3
# ATLAS SAM/ETF
SCITOKENS /^https:\/\/atlas-auth.web.cern.ch\/,5c5d2a4d-9177-3efa-912f-1b4e5c9fb660/ usatlas2

    Replacing the third field with the local Unix account.

  3. Also consult the upgrade documentation for other required configuration updates.

Updating to HTCondor 8.8.x

The OSG 3.5 release series contains HTCondor 8.8, a major version upgrade from the previously released versions in the OSG. See the HTCondor 8.8 manual for an overview of the changes. To update HTCondor on your HTCondor-CE and/or HTCondor pool hosts, perform the following steps:

  1. Update all HTCondor packages:

    [email protected] # yum update 'condor*'

  2. HTCondor pools only:

    • The DAEMON_LIST, and CONDOR_HOST configuration changed in HTCondor 8.8. Additionally in OSG 3.5, the default security was changed to use FS and pool password. If you are experiencing issues with communication between hosts in your pool after the upgrade, the default OSG configuration is listed in /etc/condor/config.d/00-osg_default_*.config: ensure that any default configuration is overridden with your own DAEMON_LIST, CONDOR_HOST, and/or security configuration in subsequent files.

    • As of HTCondor 8.8, MOUNT_UNDER_SCRATCH has default values of /tmp and /var/tmp, which may cause issues if your OSG_WN_TMP is a subdirectory of either of these directories. If the partition containing your execute directories is large enough, we recommend setting your OSG_WN_TMP to /tmp or /var/tmp. If that partition is not large enough, we recommend setting your OSG_WN_TMP variable to a directory outside of /tmp or /var/tmp.

  3. HTCondor-CE hosts only: The HTCondor 8.8 series changed the default job route matching order from round-robin to first matching route. To use the old round-robin matching order, add the following configuration to /etc/condor-ce/config.d/99-local.conf:

    JOB_ROUTER_ROUND_ROBIN_SELECTION = True

  4. Clean-up deprecated packages:

    [email protected] # yum remove 'rsv*' glite-ce-cream-client-api-c

Updating to HTCondor 9.0.0+

Where to find HTCondor 9.0

The HTCondor 9.0 series is available through the OSG upcoming repository.

For HTCondor hosts < 8.9.7 using the SciTokens CredMon, updates to HTCondor 8.9.7+ require manual intervention and a corresponding update to python2-scitokens-credmon, available in the OSG 3.5 release repository. If you do not have the python2-scitokens-credmon package installed, you may skip these instructions. Otherwise, follow these steps for a seamless update to HTCondor 8.9.7+:

  1. Determine if your HTCondor installation is configured to use the SciTokens CredMon:

    # condor_config_val -v DAEMON_LIST

    If CREDD and SEC_CREDENTIAL_MONITOR are in the output of the above command, continue onto the next step. Otherwise, your installation is not configured to use SciTokens CredMon and you should skip the rest of these instructions.

  2. Add the following to a file in /etc/condor/config.d/:

    SEC_CREDENTIAL_DIRECTORY_OAUTH = /var/lib/condor/oauth_credentials
CREDMON_OAUTH = /usr/bin/condor_credmon_oauth
SEC_CREDENTIAL_MONITOR_OAUTH_LOG = $(LOG)/CredMonOAuthLog

if version < 8.9.7
  CREDMON_OAUTH = /usr/bin/scitokens_credmon
endif

  3. Update your DAEMON_LIST configuration from:

    DAEMON_LIST = $(DAEMON_LIST), CREDD, SEC_CREDENTIAL_MONITOR

    to

    DAEMON_LIST = $(DAEMON_LIST), CREDD, CREDMON_OAUTH

  4. Turn off the SchedD and CredMon daemons:

    # condor_off -daemon SCHEDD
# condor_off -daemon SEC_CREDENTIAL_MONITOR
# condor_off -daemon CREDMON_OAUTH

  5. Move the existing credential directory and set up a temporary symlink:

    # mv $(condor_config_val SEC_CREDENTIAL_DIRECTORY) $(condor_config_val SEC_CREDENTIAL_DIRECTORY_OAUTH)
# ln -s $(condor_config_val SEC_CREDENTIAL_DIRECTORY_OAUTH) $(condor_config_val SEC_CREDENTIAL_DIRECTORY)

  6. Update HTCondor and SciTokens CredMon packages:

    # yum -y upgrade python2-scitokens-credmon condor

  7. If you are running Apache on this host, reload the Apache configuration:

    # systemctl reload httpd.service

  8. Reconfigure HTCondor:

    # condor_reconfig

  9. Turn the SchedD and CredMon daemons back on:

    # condor_on -daemon CREDMON_OAUTH
# condor_on -daemon SCHEDD

  10. Clean up old CredMon configuration. Remove the following entries from your HTCondor configuration:

    SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credentials
SEC_CREDENTIAL_MONITOR = /usr/bin/scitokens_credmon
SEC_CREDENTIAL_MONITOR_LOG = $(LOG)/CredMonLog

  11. To allow for seamless HTCondor downgrades, update the if version < 8.9.7 block that you added in step 2.

    if version < 8.9.7
  CREDMON_OAUTH = /usr/bin/scitokens_credmon
  SEC_CREDENTIAL_DIRECTORY = $(SEC_CREDENTIAL_DIRECTORY_OAUTH)
endif

  12. Remove the symlink to the old credential directory that you created in step 5. This is whatever you had set SEC_CREDENTIAL_DIRECTORY to before. For example:

    # rm /var/lib/condor/credentials

Updating to XRootD 5

Known issues with XRootD 5.1.1

  • The XRootD team is evaluating solutions for a memory leak in the HTTP Third-Party Copy (HTTP-TPC) use case related to libcurl and NSS. These leaks appear to exist in libcurl for all versions of XRootD and their impact depends on the transfer load at each site.
  • Incompatibility with the multi-user plugin: users of the XRootD multi-user plugin will be unable to update to XRootD 5.1.x until a fixed version of XRootD multi-user is released into the OSG repositories
  • In some cases, XCaches using the Rucio plug-in may crash due to malformed URLs generated by the plug-in.

Use the OSG XRootD meta-package!

The osg-xrootd and osg-xrootd-standalone meta-packages provide default XRootD configurations in /etc/xrootd/config.d/ with the ability to easily enable or disable features (such as HTTP or LCMAPS) through simple configuration flags. Additionally, the configurations provided by osg-xrootd and osg-xrootd-standalone are designed to work with the OSG-released versions of XRootD, reducing the number of necessary manual configuration updates.

The OSG 3.5 upcoming repositories contain XRootD 5, a major version upgrade from the previously released versions in the OSG. See the upstream release notes for an overview of the changes. To update XRootD on your StashCache, XCache, XRootD clustered, and XRootD standalone hosts, perform the following steps:

  1. Update all XRootD packages:

    [email protected] # yum update --enablerepo=osg-upcoming 'xrootd'* xcache

  2. Determine whether or not you are using OSG XRootD meta-packages

    [email protected] # rpm -q --verify xrootd-server | \
            egrep "/etc/xrootd/xrootd-(standalone|clustered).cfg"

  3. If the above command does not return any output, skip to step 6.

  4. If you are not using OSG XRootD meta-packaging and are using xrootd-lcmaps, update the -authzfunparams to the new key=value syntax. For example, the following configuration:

    -authzfunparms:--lcmapscfg,/etc/xrootd/lcmaps.cfg,--loglevel,0,--policy,authorize_only

    Should be turned into:

    -authzfunparms:lcmapscfg=/etc/lcmaps.db,loglevel=0,policy=authorize_only

  5. If you are not using OSG XRootD meta-packaging and are loading multiple ofs.authlib libraries, separate out each library into its own ofs.authlib directive. For example, the following configuration:

    ofs.authlib libXrdMacaroons.so libXrdAccSciTokens.so

    Should be re-ordered and turned into:

    # Enable SciTokens-based mappings; if no token is present, then the GSI certificate will be used.
ofs.authlib ++ libXrdAccSciTokens.so
ofs.authlib ++ libXrdMacaroons.so

  6. Restart the relevant XRootD services:

    If you are running a(n)... Restart the service with...
    ATLAS XCache systemctl restart [email protected]
    CMS XCache systemctl restart [email protected]
    Stash Cache systemctl restart [email protected]
    Stash Origin systemctl restart [email protected]
    XRootD Standalone systemctl restart [email protected]
    XRootD Clustered systemctl restart [email protected]

Getting Help

To get assistance, please use the this page.

Back to top