OSG Networking Area
Welcome to OSG Networking ! This is an entry point for those interested in Networking in OSG/WLCG or for those OSG/WLCG users experiencing network problems. It provides an overview of the networking goals, plans and various activities and subtopics underway regarding networking in the Open Science Grid (OSG) and World-wide LHC Computing Grid (WLCG), operated as a joint project. This area started in June 2012 with initial focus on the network monitoring as monitoring is critical to provide needed visibility into existing networks and site connectivity. OSG is working to provide needed networking information and tools for users, sites and experiments/VOs.
This documentation is divided into several sub-sections, each covering a specific area of activities.
Network Monitoring in WLCG and OSG (perfSONAR)
WLCG and OSG jointly operate a network of
perfSONAR agents deployed world-wide, which provides an open platform that can be used to baseline network performance and debug any potential issues. The following subsections provide details on the motivation, deployment and operations of the perfSONARs in WLCG/OSG:
- Motivation - overview, core concepts, motivation
- Deployment Guide - deployment models and options, hardware requirements
- Installation and Administration Guide - installation, configuration and maintanance
- Frequently Asked Questions
Users with network issues should check the troubleshooting link below for initial guidance on how best to get their issue resolved. In addition, you can refer to the ESNet network performance guide for a detailed instructions on how to identify and isolate network performance issues using perfSONAR.
OSG operates an advanced platform to collect, store, publish and analyse the network monitoring data it gathers from perfSONAR and other locations. All measurements are collected and available via streaming or through APIs. The following services are available:
- perfSONAR infrastructure monitoring - monitors state of perfSONAR network and reports on availability of core services
- OSG Distributed Network Datastore - distributed datastore based on ElasticSearch holding all the network measurements and providing an API to expose them via JSON is available at two locations (University of Chicago and University of Nebraska). In addition, all network data is backed up on tape at Fermi National Accelerator Laboratory. The central OSG datastore that was hosted at OSG/GOC was retired in June 2018.
- OSG Network Stream - access to network measurements in near realtime is provided by the OSG RabbitMQ and CERN ActiveMQ messaging brokers.
- OSG pSConfig Web Admin (PWA) - centralized configuration of the tests performed by the OSG/WLCG perfSONAR infrastructure . In case you'd like to start/manage particular mesh, please contact our support channels to get access.
- OSG Dashboards - set of dashboards showing an overview of the network state as seen by the perfSONAR infrastructure (NOTE: this instance was previously the prototype instance at AGLT2 and no longer uses the deprecated OSG central measurement archive).
- WLCG Dashboards - set of dashboards showing WLCG and OSG network performance by combining multiple sources of data including perfSONAR, FTS, ESNet/LHCOPN traffic, etc.
University of Chicago has setup an analytics platform using
Kibana4 as well as
Jupyter that can be used to access and analyse all the existing network measurements.
Support and Feedback
If you suspect a network problem and wish to follow up on it, please open a ticket with the appropriate support unit: For
OSG sites please open a ticket with GOC; For
WLCG sites please open a GGUS ticket to
WLCG Network Throughput support unit. If you'd like to get help in setting up or debugging perfSONAR instance please open a ticket with GOC or via GGUS to WLCG perfSONAR support. For any other requests please open a ticket at GGUS and mention OSG networking.
- ESNet network performance tuning and debugging https://fasterdata.es.net/
- perfSONAR toolkit is part of the perfSONAR project.
- OSG/WLCG mesh configuration interface is available at http://psconfig.opensciencegrid.org
- Information on ESmond is available at http://software.es.net/esmond/
- Information on querying the perfSONAR data from ESmond is at http://software.es.net/esmond/perfsonar_client_rest.html
- OSG dashboard instance http://psmad.opensciencegrid.org/maddash-webui/index.cgi
- OSG perfSONAR infrastructure monitoring https://psetf.opensciencegrid.org/etf/check_mk/
- OSG Analytics platform http://atlas-kibana.mwt2.org/s/networking/app/kibana
- WLCG dashboards http://monit-grafana-open.cern.ch/dashboard/db/home?orgId=16
- SAND (an OSG Satellite project) focusing on analyzing perfSONAR and related network metrics: https://sand-ci.org