Installing the OSDF Origin by RPM¶

OSG 24

This installation guide requires OSG 24

Upgrading from a non-Pelican origin?

This installation guide also walks you through upgrading an origin that you installed prior to OSG 24. See this section for more details.

This document describes how to install an Open Science Data Federation (OSDF) Origin service via RPMs. This service, based on the Pelican Platform, allows an administrator to serve data from a POSIX filesystem or S3 endpoint through the global OSDF infrastructure.

Before Starting¶

Before starting the installation process, consider the following requirements:

Operating system: A RHEL 8 or RHEL 9 or compatible operating system.
User IDs: If it does not exist already, the installation will create the Linux user named xrootd for running daemons.
Host certificate: Required for authentication. See note below.
Network ports: The origin service requires the following ports open:
- Inbound TCP port 8443 for file access via the HTTP(S) and XRoot protocols.
- (Optional) Inbound TCP port 8444 for access to the web interface for monitoring and configuration; if enabled, access to this port should be restricted to the LAN.
Service requirements: An origin in the OSDF should have at least:
- 1 core
- 1 Gbps connectivity
- 12 GB of RAM
- 10 GB of local disk space in /var/log for log files, although some logging verbosity can be reduced.

As with all OSG software installations, there are some one-time steps to prepare in advance:

Obtain root access to the host
Prepare the required Yum repositories; the OSG 24 repositories should be used.
Upgrading to a Pelican origin

If you are upgrading from a pre-Pelican OSDF origin, update all of your OSG 23 packages before installing the OSG 24 repositories:
```
root@host # yum update
```

Host certificates

Origins are accessed by users through browsers, meaning origins need a certificate from a CA acceptable to a standard browser. Examples include Let's Encrypt or the InCommon RSA CA. Origins without a valid certificate for the browser cannot be added to the OSDF. Note that, unlike legacy grid software, the public certificate file will need to contain the "full chain", including any intermediate CAs (if you're unsure about your setup, try accessing your origin from your browser).

The following locations should be used (note that they are in separate directories):

Host Certificate Chain: /etc/pki/tls/certs/pelican.crt
Host Key: /etc/pki/tls/private/pelican.key

Upgrading a Non-Pelican Origin¶

If you are running a non-Pelican origin, e.g. one that was installed before OSG 24, there are special considerations for the upgrade to ensure minimal downtime for your users. This document will guide you through the upgrade process by installing and configuring a Pelican origin alongside your non-Pelican origin.

Using different hosts

You may install your new Pelican origin on a separate host if your underlying data store is shared between hosts.

First, determine if you have an active non-Pelican origin service running:

user@host $ systemctl status xrootd@stash-origin-auth \
                             xrootd@stash-origin \
                             xrootd-privileged@stash-origin-auth \
            | grep -F 'Active: active'
   Active: active (running) since Wed 2024-12-04 17:46:17 CST; 1 weeks 1 days ago

If you do not see any output from the above command, you may proceed with the rest of the documentation.
If you see any output from the above command, you may proceed with the rest of the documentation but keep an eye out for special instructions related to the upgrade:

Upgrading to a Pelican origin

You will find upgrade-specific instructions here.

Installing the Origin¶

The origin service is provided by the osdf-origin RPM. Install it using the following command:

root@host # yum install osdf-origin

osdf-origin 7.11.1

This document covers versions 7.11.1 and later of the osdf-origin package; ensure the above installation results in an appropriate version.

Configuring the Origin Server¶

Create a file named /etc/pelican/config.d/20-origin.yaml

You must tell Pelican the data to export to the federation. An origin may export one or more directory trees, or one or more S3 buckets -- follow one of the sections below. A single origin cannot export both a bucket and a directory tree.

Configuring POSIX (directory) export¶

Set these options to export one or more directory trees to the federation.

Origin:
  StorageType: "posix"
  Exports:
    # You may have one or more of the following block:
    - FederationPrefix: "<EXTERNAL OSDF NAMESPACE>"
      StoragePrefix:    "<LOCAL FILESYSTEM DIRECTORY>"
      Capabilities:    # Add or remove as desired
        - Reads        # Enable authenticated reading of objects from under the directory tree through a cache
        - PublicReads  # Enable unauthenticated reading of objects from under the directory tree through a cache
        - DirectReads  # Enable reading objects from under the directory tree without going through a cache
        - Listings     # Enable directory listings
        - Writes       # Enable writing to files in the directory tree

Configuring S3 export¶

To configure your origin to serve objects from an S3 endpoint, see the upstream documentation.

Preparing for Initial Startup¶

The origin identifies itself to the federation via public key authentication; before starting the origin for the first time, it is recommended to generate a keypair.
```
root@host$ cd /etc/pelican
root@host$ pelican generate keygen
```
The newly created files, issuer.jwk and issuer-pub.jwks are the private and public keys, respectively.
Save these files; if you lose the issuer.jwk, your origin will need to be re-approved.

Validating the Origin Installation¶

Do the following steps to verify that the origin is functional:

Start the origin using the following command:
```
root@host$ systemctl start osdf-origin
```
Download a test file (POSIX) or object (S3) from your origin (replacing ORIGIN_HOSTNAME with the host name of your origin, and TEST_PATH with the OSDF path to the test file or object
```
user@host$ curl -L https://ORIGIN_HOSTNAME:8443/TEST_PATH -o /tmp/testfile
```
Verify the contents of /tmp/testfile match the test file or object your origin was serving.

If the download fails, debugging information is located in /var/log/pelican/osdf-origin.log. See this page for requesting assistance; please include the log file in your request.

Joining the Origin to the Federation¶

Upgrading to a Pelican origin

Once registered, all OSDF clients of your namespace will be directed to your Pelican origin. Before initiating this process, ensure that your Pelican origin is functioning and that you are ready to migrate production transfers.

The origin must be registered with the OSG prior to joining the data federation. Send mail to help@osg-htc.org requesting registration; provide the following information:

Origin hostname
Administrative and security contact(s)
Institution that the origin belongs to

OSG Staff will register the origin and respond with the Resource Name.

Once you have that information, edit /etc/pelican/config.d/15-osdf.yaml, and set XRootD.Sitename:

XRootD:
  Sitename: <RESOURCE NAME REGISTERED WITH OSG>

Then, restart the origin by running

root@host$ systemctl restart osdf-origin

Validating the Origin Through the Federation¶

Once your origin has been registered in the federation:

Download a test file (POSIX) or object (S3) from your origin (replacing ORIGIN_HOSTNAME with the host name of your origin, and TEST_PATH with the OSDF path to the test file or object:
```
user@host $ curl -L https://osdf-director.osg-htc.org:8443/TEST_PATH -o /tmp/testfile
```
Verify the contents of /tmp/testfile match the test file or object your origin was serving.

If the download fails, debugging information is located in /var/log/pelican/osdf-origin.log. See this page for requesting assistance; please include the log file in your request.
Verify that your test is running against your Pelican origin:
```
user@host $ grep <TEST_PATH> /var/log/pelican/osdf-origin.log
```
Replacing <TEST PATH> with the same path that you used in step (1). If you see output, then the OSDF is directing client requests to your Pelican origin! If you do not see output, please contact us.

Upgrading to a Pelican origin

Congratulations, you have fully verified the functionality of your Pelican origin! You may uninstall the non-Pelican origin:

root@host $ yum remove stash-origin

Managing the Origin Service¶

Use the following SystemD commands as root to start, stop, enable, and disable the OSDF Origin.

To...	Run the command...
Start the origin	`systemctl start osdf-origin`
Stop the origin	`systemctl stop osdf-origin`
Enable the origin to start on boot	`systemctl enable osdf-origin`
Disable the origin from starting on boot	`systemctl disable osdf-origin`

Getting Help¶

To get assistance, please use the this page.