Skip to content

🔭 Researcher — Architecture & Analytics

Understand the OSG/WLCG network monitoring system, access data, and explore insights.

System Architecture

High-Level Overview

Architecture Overview — components, responsibilities, and system design

The perfSONAR network consists of:

  • Testpoints — distributed measurement agents at OSG/WLCG sites

  • Collection Pipeline — HTTP-Archiver ingestion and Logstash processing

  • Data Storage — Central Elasticsearch instances (distributed for

resilience)

  • Configuration Services — pSConfig for centralized test mesh management

  • Monitoring — PSETF for infrastructure health and visibility

Data Flow: From Measurement to Insight

  1. Measurement (2-minute intervals)

  2. perfSONAR testpoints run periodic latency, bandwidth, traceroute tests

  3. Tests configured by central mesh at psconfig.opensciencegrid.org

  4. Collection (near real-time)

  5. Results sent to central Elasticsearch via HTTP-Archiver

  6. Logstash processes and enriches measurement metadata

  7. Storage (permanent)

  8. OSG Network Datastore — distributed Elasticsearch storage

  9. JSON API for direct programmatic access

  10. Analysis (on-demand)

  11. OSG Analytics Platform — Kibana dashboards + Jupyter notebooks

  12. Custom queries, time-series analysis, anomaly detection

  13. Visualization (real-time dashboards)

  14. WLCG Dashboards — performance monitoring

  15. Site-to-site path performance, latency trends, bandwidth utilization

Accessing & Analyzing Data

Real-Time Dashboards

WLCG Grafana Dashboards

  • Network performance by site and path

  • Latency, bandwidth, packet loss trends

  • Time-series filtering and drill-down

OSG PSETF Monitoring

  • perfSONAR infrastructure health

  • Testpoint availability and service status

  • Test execution success rates

OSG Analytics Platform

  • Custom Kibana queries

  • Ad-hoc measurement exploration

  • Jupyter notebooks for advanced analysis

Programmatic Access

Elasticsearch API

  • OSG Network Datastore — detailed API documentation

  • JSON endpoints for direct queries

  • Available at: University of Chicago and University of Nebraska instances

Example query

curl -X GET "elasticsearch-server:9200/perfsonar-testpoint/_search" \
  -H 'Content-Type: application/json' \
  -d '{"query": {"match": {"agent": "testpoint.example.com"}}}'

Jupyter Notebooks

  • Available on analytics platform

  • Pre-built examples for common analysis tasks

  • Python libraries: pandas, numpy, matplotlib for data science workflows

pSConfig API

  • Central Mesh Configuration — test definitions and schedule

  • JSON endpoints for retrieving test configurations

  • Dynamic mesh membership and test parameters

Analyzing Network Measurements

Questions You Can Answer

Performance Characterization:

  • Which network paths have persistent latency issues?

  • What's the peak and sustained bandwidth between sites?

  • How has network performance trended over the past month/year?

  • Are there time-of-day or day-of-week patterns?

Infrastructure Health:

  • Which perfSONAR testpoints are most active?

  • What's the geographic distribution of measurement agents?

  • Are there coverage gaps (missing paths)?

Root Cause Analysis:

  • Did network performance degrade after a specific event?

  • Correlate measurements with known network changes

  • Identify bottlenecks in multi-hop paths

Tools & Resources

Data Analysis:

  • Kibana — query, filter, and visualize Elasticsearch data

  • Jupyter — Python/pandas for advanced statistical analysis

  • Grafana — time-series visualization and alerting

Measurement Understanding:

Community Resources:

Contributing & Development

Adding New Measurements or Tests

Propose new tests to the WLCG Mesh Configuration:

  • Define measurement parameters and schedule

  • Request inclusion in production mesh

  • Community review and approval

Or deploy local tests:

  • Add custom tests via your testpoint's pSConfig web interface

  • Share configurations with the community

Improving the Infrastructure

Source code and development:

Contributing:

Architecture & Documentation

Want to contribute diagrams, data pipeline notes, or architecture updates?

  • Add diagrams or notes to personas/research/

  • Submit via GitHub PR or email networking-team@osg-htc.org

  • All contributions welcome and attributed

Infrastructure & Services

Foundational Concepts

Tools & Technical Details