Open Science Grid Management Plan
The Open Science Grid (OSG) is a Consortium of institutions and organizations with the goal to design, deploy and operate a shared distributed infrastructure and services in support of US science.
"The Open Science Grid (OSG) advances science through open distributed computing. The OSG is a multi-disciplinary partnership to federate local, regional, community and national cyberinfrastructures to meet the needs of research and academic communities at all scales."
An important aspect of OSG is the close collaboration across the scientific user organizations, the resource administrators and the technology providers in all aspects of the program of work. The OSG governing structure consists of two parts: An element which oversees the Open Science Grid as a whole and a set of projects (including satellite projects) with technical leaders managing well defined deliverables towards common goals. The organization of the OSG consists of the fol-lowing:
- Consortium - all participants.
- Council - governs the Consortium ensuring effective benefit to the scientific mission of the stakeholders.
- Scientific Advisory Group - an invited, appointed group representing a diverse scientific community that provides advice to the Council and Executive Team.
- Executive Team - manages the OSG project and program of work, including the OSG infrastructure, services and software, and oversees the financial arrangements.
- Executive Board - provides coordination across the areas of work.
This management plan for OSG describes the role and composition of the organizational bodies and the way they interact with the projects. Our goal is to have an organization able to make decisions quickly, clearly and with a common understanding of expectations. We aim to be flexible and to make timely reactions to changes in, or new, user requirements, opportunities and available technologies.
Any group, institution, or organization may join the OSG Consortium by joining in the ongoing program of work. Contributions can be making resources accessible to the infrastructure, use of or providing OSG services and/or software, participating in projects to enhance and/or benefit the OSG. The process is described in more detail in the OSG By-Laws.
The OSG program of work to operate, expand and extend the distributed high throughput computing infrastructure is defined via the governance of the OSG Consortium. The content of this program of work is ultimately owned by the OSG Council and managed by the Executive Director. The effort and funds are organized via a set of projects, which invariably depend on both OSG and externally provided effort and funds. The project deliverables include targeted deliverables to both the Consortium and to the stakeholder organizations.
To extend the capacity and capability of the OSG infrastructure we depend crucially on a mix of effort that is internal and external to the OSG. Internal effort is directly under the fiscal control of the Executive Team. External effort is generally embedded into organizations whose scope extends well beyond the OSG, e.g. the scientific programs of the stakeholder organizations, computer sci-ence groups, as well as the universities and laboratories participating in the OSG Consortium. These contributions are organized as part of the OSG program of work via the set of projects, with the list of projects changing over time.
The OSG staff are mentored both by their local organization and the OSG management. Attention is paid to the technical and professional growth of the staff to enable them to gain and communicate their expertise and foster a productive, improving organization.
The OSG Council is the governing body of the Consortium responsible for the program of work and ensuring effective benefit to the scientific mission of the stakeholders. The Consortium includes all contributors to OSG and the Council is made of those Consortium members making significant contributions and commitments to the success of OSG. The Council endorses all strategic decisions, which are typically recommended by the Executive Team. The basic rules for operation of the OSG Council are described in the OSG By-Laws and summarized here.
The OSG council appoints a Chair following the procedure outlined in the OSG bylaws. The Council Chair is an ex-officio member of Executive Board. The Council Chairperson may form a team to assist in the preparation of Council business. The composition of such an advisory group is at the discretion of the Council Chairperson. Council membership includes Institution Representatives selected by the institutions and all members of the Executive Board ex-officio (as non-voting members).
The council elects the Executive Director following the procedures outlined in the OSG bylaws. The Executive Director reports regularly to the OSG Council. Decisions are to be agreed by consensus whenever possible. If consensus cannot be reached, the relevant arguments will be documented and distributed to the OSG Council, and a meeting date set to decide the issue. A simple majority of those present who are eligible to vote will decide the issue through an open ballot. In cases where the OSG Council disagrees with a recommendation from the Executive Team the issue, together with a statement of concerns, will be returned to the Executive Team for further discussion. The issue will finally be decided through negotiation between the Executive Team and Council.
The Council elects the Executive Director for 2 years. The Executive Director appoints the Technical Director, the Resources Manager, the Production Coordinator and the Applications Coordinator. The Council confirms all appointees. The Executive Director may appoint co-coordinators and/or deputies in any of these positions. The appointment of a new Executive Director and his/her team takes place 6 months prior to their taking office. Submissions for the post are submitted 2 months before the election. Between the submission and appointment, the Council arranges for presentations by the candidates for Executive Director. During the 6 months following the appointment the new team works with the equivalent members of the current team. If the six month overlap includes any of the months May through July, when planning for the following fiscal year is done, the new ED and team will lead the planning activity. At the transition, the (new) Executive Director and team will be up to speed on all responsibilities and will take over from the previous team at the transition date.
Scientific Advisory Group
A Scientific Advisory Group consisting of leading scientists provides guidance to the Council. The Scientific Advisory Group reviews the effectiveness of the benefits of OSG to the scientific mission of the stakeholders and provides feedback on the program of work to achieve this. Members of the Scientific Advisory Group are appointed by the Council Chair and confirmed by the Council.
OSG Project PI and Co-PIs
The OSG Project delivers core services to the goals of the Consortium. The PI is responsible for the contractual obligations to the funding agencies, including the management of funds, reporting and contracts between the OSG Project and the funding agencies. The PI distributes funds through a sub-contract with each institution, as decided by the Executive Director guided by the Resources Manager and Executive Board. The Co-PIs assist the PI in their responsibilities. The PI is a member of the Council and Executive Board. The current organizational structure of the OSG is shown in Figure 2.
The Executive Director, the Technical Director, the Applications Coordinators, the Associate Executive Director, the Production Coordinator, and the Project Manager form the Executive Team. This team has responsibility and authority for the ongoing program of work of the Project and reporting from the project managers. It supervises and reviews progress covering technical progress, schedule progress, milestones, planning and resources. All strategic Executive Team decisions are submitted to the OSG Council for ratification.
The Executive Team leads the implementation and management of the OSG infrastructure, services and projects. The Executive Team liaises with the contributing projects and follows the evolution, integration, technical and scientific impact of project deliverables and stakeholder needs. The team coordinates and oversees problem solving and encourages development and adoption of common solutions. The team follows the scientific benefit provided by the distributed facility to ensure its effectiveness. The Executive Team monitors closely the performance and operations of the OSG infrastructure.
The organization of projects is communicated through the Executive Team. The Executive Team resolves controversies within or between projects that impact the OSG. The Executive Team is supported by the program office that includes staff for administration, assessment, communications, documentation, education and outreach.
The OSG Executive Director chairs the OSG Executive Team and the Executive Board. The Ex-ecutive Director matches the program of work to the scientific needs of the stakeholders from the OSG. The Executive Director leads the external relations and outreach parts of the OSG project, communicating between with OSG and external bodies and partners. The Executive Director, together with the Executive Team, decides on staff resource allocations across the projects. He/she runs periodic reviews of operations efficiencies and delivery of resources to the stakeholders. Decisions are reached by consensus wherever possible. The Executive Director may propose a compromise statement to try to reach consensus. If none is reached, the Executive Director may take an operational decision on immediate issues but shall forward the issue to the OSG Council for final decision.
The Technical Director manages the technical and software capabilities and evolution of the OSG program of work. The Technical Director leads the Technology Group that is responsible for the architecture and blueprint of the software, software tools and functionality of the OSG. He/she is responsible for overseeing the quality and evolution of the technical aspects of the program of work.
Associate Executive Director
The OSG Associate Executive Director acts as the liaison to the associated projects, such as Institutes, that make needed contributions to the scope of work of the OSG project. The Associate Executive Director is a member of the Executive Team. The Associate Executive Director ensures effective communication of the (technical, schedule, process) requirements of OSG and its stakeholders to the associated projects, agreement and delivery of the contributions from the associated projects, and ensure an effective matching of the ongoing program of work to the scientific needs of the stakeholders from the OSG.
The Applications Coordinators provide ongoing technical input to and assessment of the project deliverables for the applications and advise the rest of the Executive Team on the program of work and its effectiveness in contributing to the scientific missions.
The Production Coordinator oversees the production usage, operations, and on-campus activities of the OSG. The Production Coordinator tracks production usage and follows up on short-term problems and issues identified. He/she identifies, extracts, and socializes best practices as they are identified and coordinates these with the community to execute best practices in a manner that minimizes disruptions in the productivity. The production coordinator brings computing and storage resource needs to the Executive Team for discussion and forwarding as needed to the Council (whose members contribute the resources). The Operations and Campus leads report to the Production Coordinator:
- The Operations Lead coordinates the monitoring and support and operates the OSG infra-structure.
- The Campus Lead coordinates the extension of the production services into the campuses.
The Project Manager defines and tracks the deliverables, milestones, costs and resources of the OSG program of work. The Project Manager defines and tracks the annual program of work and the day-to-day deliverables and schedule of the program. He/she tracks and recommends the dis-tribution of costs and effort. The Project Manager coordinates reporting of the Projects as they im-pact the OSG.
The Executive Board plans, decides, and executes the detailed program of work of the project. The Executive Board consists of the Executive Team, the Resources Manager, the Area Leads and the Council Co-Chairs (Ex-officio). The area leads are responsible for the deliverables and services in their technical area of work and also the interfaces between OSG and their users. The Area Leads are Security, Software, Campus, Operations, User Support, and Technology.
The Resources Manager oversees the finances within the OSG, as well as the contributions from external organizations. The Executive Team discusses and decides on issues related to OSG fi-nances - including funding, distribution of charges, income and expenditure. The Executive Team, Executive Board, or stakeholder organizations can make requests for a change in the financial allocations to the Resources Manager. The Resources Manager is a member of the Executive Board.
The Software Infrastructure Lead coordinates the packaging, distribution and support of the OSG infrastructure middleware. Particular responsibilities include: making timely releases of the OSG middleware in response to project and stakeholder requirements; testing and validation of the integrated set of components in the OSG middleware releases; coordinating support for the OSG middleware; determining the effectiveness of the OSG middleware to the scientific benefit to the Consortium stakeholders. The software infrastructure lead also leads the Software Tools Group that oversees the development and delivery of all software tools and utilities owned by the OSG. OSG staff writes (and supports) or acquires, utilities and tools to provide new capabilities to the infrastructure in support of the program of work. The Software Coordinator is a member of the Executive Board.
The Security Officer coordinates, monitors and supports the security of the OSG infrastructure. Par-ticular responsibilities include: providing and operating the operational security processes of the distributed facility; identifying and communicating security threats and vulnerabilities; coordinating the response to and mitigation of security threats, vulnerabilities and incidents; overseeing the trust, authentication and authorization infrastructure. The Security Office coordinates security to enable effective benefit of OSG to the scientific missions of the stakeholders. The Security Officer is a member of the Executive Board.
The User Support Lead facilitates and supports engaging new scientific communities to use the OSG and assists existing user communities in meeting their needs in developing and running applications on the OSG. Particular responsibilities include: identifying target groups and organizing collaborative work; liaising between the OSG applications, extension development projects, and the user communities; coordinating the support for integration and/or interoperation of new users and new approaches with the existing teams and activities. The User Support group runs task forces or 'campaigns' targeted to increase the effectiveness of and benefit from OSG for a particular community. The User Support Coordinator is a member of the Executive Board.
The Technical Director works with the Technology Lead to document the current high-level architecture, articulate the end-to-end design of the Open Science Grid software distribution, create a roadmap of future functional and performance goals of the OSG software system, and assist the area teams in having a coherent, consistent end-to-end system. The Technology Investigations Coordinator assesses technology changes and new capabilities as requested by the stakeholders. The Technology Group and Investigations Coordinators are members of the Executive Board.
Projects and Satellites
Projects are the organizational means by which deliverables that OSG depends on are defined, agreed to, executed and delivered. . Each project is managed by a project management team, and may be funded completely through the Consortium, include contributions both from within and external to the OSG, or be completely externally funded. Projects are Satellites (https://twiki.grid.iu.edu/bin/view/Management/OSGSatelliteProjects) if OSG was involved in the planning process, OSG commits support for future collaborative action and if the project agrees to be considered an OSG Satellite project.
Changes to this Management Plan
Changes to this management plan are proposed by the Council or Executive Director in consultation with the Executive Team and approved by both bodies.
The following standing meetings are part of the OSG management. More details are available on the OSG TWiki.
- Production Meeting. Chair: Production Coordinator: Deals with immediate problems with the production facility and services, i.e. issues with a one or two week timeframe. (Tuesday, weekly)
- Executive Board Area Coordinators. Chair: Project Manager: Presentation of areas status, deliverables and milestones and discussion of issues between the areas. (Wednesday, biweekly)
Consortium Level Meetings:
- Council Meeting. Chair: Council Co-Chairs: Discussion of issues, plans and decisions affecting the Consortium. (Second Tuesday, monthly)
- All Hands Meeting. (Annual several day event, usually in March)
Internal Work Meetings:
These meetings include specific stakeholders and contributors as needed:
- Blueprint, Technology Group: Chair: Technology Group Coordinator. Discuss architecture, principles and potential technology changes. (Quarterly)
- Executive Team. Chair: Executive Director: Presentation of plans and decisions for sign off by the Executive Team. Discussion of longer term issues affecting the OSG project. (Thursday, bi-weekly)
- Integration: Discussion, planning, tracking of integration, validation and testbed activities, priorities and issues. (Thursday bi-weekly)
- Operations Meeting. Chair: Operations Coordinator: Coverage of high priority tickets open at the GOC; Introduction of contacts of new VOs and Sites. (Monday, weekly)
- Project, Production and Operations management. Chair: Production Coordinator. Internal meeting to manage change in operations services. (Tuesday, weekly)
- Security Team meeting. Chair: Security Officer: Discussion, planning, tracking of security activities, priorities and issues. (Tuesday, weekly)
- Sites. Communication with resources administrators. (Thursday as needed)
- Software Tools Group, Chair: Software Coordinator. (Tuesday as needed)
- Software, Production and Project Management coordination meetings. Chair: Executive Director. (Friday as needed)
- Storage team meeting. Chair: Storage Software Coordinator: Discussion, planning, tracking of Storage Team activities, priorities and issues. (Wednesday, weekly)
- VO/User Group. Chair: User Support Coordinator: Discussion with specific VOs on their needs, campaigns to help specific issues associated with users use of OSG. (Thursday, weekly)
- WLCG Operations call. Attendee from OSG: Operations Coordinator: Day to day operational issues for the WLCG. (Daily during Mon-Friday)
- WLCG Management Board. Attendee from OSG: Executive Director. Supervision and tracking of the WLCG. (Tuesday bi-weekly)
- Interoperation between OSG and the WLCG NGIs/EGI (monthly)
- Grid Deployment Board (monthly).
V7, Feb. 22, 2011
V8, July 15, 2011