Skip to content


This is a technology preview document and will probably change content and location withouth notice.

Installation of FileBeats for Access Points

This document is for frontend administrators. It describes the installation of Filebeats to continuously upload the HTCondor access point transfer log to Elastic Search.


An access point (HTCondor schedd) is a login node where users submit jobs to distributed computing pools. One interesting log that it produces is the TransferLog. The TransferLogs report all the transfers of files between compute node and access points. In this guide we describe the installation of Filebeats to upload this log to Elastic Search.


FileBeat Installation

For the installation of filebeats follow the official instruction to set up the repositories and install filebeats as described here.


Configuration of Filebeats

The configuration of filebeats revolves around this file /etc/filebeat/filebeat.yml. Bellow are the steps to modify the different sections of this file

  1. The Filebeat Inputs section, the input should look like this:

    - type: log
      enabled: true
        - /var/log/condor/XferStatsLog
  2. The output logstash section should look like:

    #----------------------------- Logstash output --------------------------------
      # The Logstash hosts
      hosts: [""]
      # Optional SSL. By default is off. 
      # List of root certificates for HTTPS server verifications
      ssl.certificate_authorities: ["/etc/grid-security/certificates/InCommon-IGTF-Server-CA.pem"]
      #  Certificate for SSL client authentication
      ssl.certificate: "/etc/grid-security/hostcert.pem"
      # Client Certificate Key
      ssl.key: "/etc/grid-security/hostkey.pem"
  3. Comment out all of the Elasticsearch output since we are using LogStash

    #-------------------------- Elasticsearch output ------------------------------
    # Array of hosts to connect to.
    #hosts: ["localhost:9200"]
    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "elastic"
    #password: "changeme"
  4. The general section should look like this, where <HOSTNAME> should be replaced by the hostname of the machine you are installing filebeats on.

    #================================ General =====================================
    name: <HOSTNAME>
    tags: ["xfer-log"]
  5. Test that the configuration is correct by running:

    root@host # filebeat test config
  6. Start the filebeats services:

    root@host # service filebeat start

Configuration of HTCondor

For the configuration of the HTCondor access point to use the TransferLog follow the next instructions:


The transfer metrics was introduced in HTCondor 8.6 series. You need to be running a version equal or greater than 8.6.1 to enable it.

  1. Create a file named /etc/condor/config.d/50-transferLog.config with the following contents:

    SHADOW_STATS_LOG = $(LOG)/XferStatsLog
    STARTER_STATS_LOG = $(LOG)/XferStatsLog
  2. Reconfigure condor:

    root@host # condor_reconfig
  3. Make sure that after a couple of minutes the new log /var/log/condor/XferStatsLog is present.

Back to top