Skip to content

Installing the OSDF Origin by RPM

OSG 24

This installation guide requires OSG 24

This document describes how to install an Open Science Data Federation (OSDF) Origin service via RPMs. This service, based on the Pelican Platform, allows an administrator to serve data from a POSIX filesystem or S3 endpoint through the global OSDF infrastructure.

Before Starting

Before starting the installation process, consider the following requirements:

  • Operating system: A RHEL 8 or RHEL 9 or compatible operating system.
  • User IDs: If it does not exist already, the installation will create the Linux user named xrootd for running daemons.
  • Host certificate: Required for authentication. See note below.
  • Network ports: The origin service requires the following ports open:
    • Inbound TCP port 8443 for file access via the HTTP(S) and XRoot protocols.
    • (Optional) Inbound TCP port 8444 for access to the web interface for monitoring and configuration; if enabled, access to this port should be restricted to the LAN.
  • Service requirements: An origin in the OSDF should have at least:
    • 1 core
    • 1 Gbps connectivity
    • 12 GB of RAM
    • Several GB of local disk space be available for log files, although some logging verbosity can be reduced.

As with all OSG software installations, there are some one-time steps to prepare in advance:

Host certificates

Origins are accessed by users through browsers, meaning origins need a certificate from a CA acceptable to a standard browser. Examples include Let's Encrypt or the InCommon RSA CA. Origins without a valid certificate for the browser cannot be added to the OSDF. Note that, unlike legacy grid software, the public certificate file will need to contain the "full chain", including any intermediate CAs (if you're unsure about your setup, try accessing your origin from your browser).

The following locations should be used (note that they are in separate directories):

  • Host Certificate Chain: /etc/pki/tls/certs/pelican.crt
  • Host Key: /etc/pki/tls/private/pelican.key

Installing the Origin

The origin service is provided by the osdf-origin RPM. Install it using the following command:

OSG 24: 

root@host # yum install osdf-origin

osdf-origin 7.11.1

This document covers versions 7.11.1 and later of the osdf-origin package; ensure the above installation results in an appropriate version.

Configuring the Origin Server

Create a file named /etc/pelican/config.d/20-origin.yaml

You must tell Pelican the data to export to the federation. An origin may export one or more directory trees, or one or more S3 buckets -- follow one of the sections below. A single origin cannot export both a bucket and a directory tree.

Configuring POSIX (directory) export

Set these options to export one or more directory trees to the federation.

Origin:
  StorageType: "posix"
  Exports:
    # You may have one or more of the following block:
    - FederationPrefix: "<EXTERNAL OSDF NAMESPACE>"
      StoragePrefix:    "<LOCAL FILESYSTEM DIRECTORY>"
      Capabilities:    # Add or remove as desired
        - Reads        # Enable authenticated reading of objects from under the directory tree through a cache
        - PublicReads  # Enable unauthenticated reading of objects from under the directory tree through a cache
        - DirectReads  # Enable reading objects from under the directory tree without going through a cache
        - Listings     # Enable directory listings
        - Writes       # Enable writing to files in the directory tree

Configuring S3 export

To configure your origin to serve objects from an S3 endpoint, see the upstream documentation.

Preparing for Initial Startup

  1. The origin identifies itself to the federation via public key authentication; before starting the origin for the first time, it is recommended to generate a keypair.

    root@host$ cd /etc/pelican
root@host$ pelican generate keygen

    The newly created files, issuer.jwk and issuer-pub.jwks are the private and public keys, respectively.

  2. Save these files; if you lose the issuer.jwk, your origin will need to be re-approved.

Validating the Origin Installation

Do the following steps to verify that the origin is functional:

  1. Start the origin using the following command:

    root@host$ systemctl start osdf-origin

  2. Download a test file (POSIX) or object (S3) from your origin (replacing ORIGIN_HOSTNAME with the host name of your origin, and TEST_PATH with the OSDF path to the test file or object

    user@host$ curl -L https://ORIGIN_HOSTNAME:8443/TEST_PATH -o /tmp/testfile

    Verify the contents of /tmp/testfile match the test file or object your origin was serving.

    If the download fails, debugging information is located in /var/log/pelican/osdf-origin.log. See this page for requesting assistance; please include the log file in your request.

Joining the Origin to the Federation

The origin must be registered with the OSG prior to joining the data federation. Send mail to help@osg-htc.org requesting registration; provide the following information:

  • Origin hostname
  • Administrative and security contact(s)
  • Institution that the origin belongs to

OSG Staff will register the origin and respond with the Resource Name that the origin was registered as.

Once you have that information, edit /etc/pelican/config.d/15-osdf.yaml, and set XRootD.Sitename: 

XRootD:
  Sitename: <RESOURCE NAME REGISTERED WITH OSG>

Then, restart the origin by running

root@host$ systemctl restart osdf-origin

Let OSG Staff know that you have restarted the origin with the updated sitename, so they can approve the new origin.

Managing the Origin Service

Use the following SystemD commands as root to start, stop, enable, and disable the OSDF Origin.

To... Run the command...
Start the origin systemctl start osdf-origin
Stop the origin systemctl stop osdf-origin
Enable the origin to start on boot systemctl enable osdf-origin
Disable the origin from starting on boot systemctl disable osdf-origin

Getting Help

To get assistance, please use the this page.

Back to top