SKA telescope manager: a status update

The international Square Kilometre Array (SKA) project to build two radio interferometers is approaching the end of its design phase, and gearing up for the beginning of formal construction. A key part of this distributed Observatory is the overall software control system: the Telescope Manager (TM). The two telescopes, a Low frequency dipole array to be located in Western Australia (SKA-Low) and a Mid-frequency dish array to be located in South Africa (SKA-Mid) will be operated as a single Observatory, with its global headquarters (GHQ) based in the United Kingdom at Jodrell Bank. When complete it will be the most powerful radio observatory in the world. The TM software must combine the observatory operations based at the GHQ with the monitor and control operations of each telescope, covering the range of domains from proposal submission to the coordination and monitoring of the subsystems that make up each telescope. It must also monitor itself and provide a reliable operating platform. This paper will provide an update on the design status of TM, covering the make-up of the consortium delivering the design, a brief description of the key challenges and the top level architecture, and its software development plans for tackling the construction phase of the project. It will also briefly describe the consortium’s response to the SKA Project’s decision in the second half of 2016 to adopt the processes set out by the Software Engineering Institute (SEI) for system architecture design and documentation, including a re-evaluation of its deliverables, documentation and approach to internal reviews.


INTRODUCTION
The Square Kilometre Array (SKA) will be the world's most advanced radio observatory, many times more sensitive and hundreds of times faster at mapping the sky than today's best radio telescopes.The SKA is to be designed and built in two phases of construction, SKA1 and SKA2.
The SKA1 observatory comprises a global headquarters and two telescopes on two continents: a 197 dish interferometer in South Africa (SKA1-Mid) operating between 350 and 13800 MHz, and a low-frequency antenna interferometer in Australia (SKA1-Low), operating from 50 to 350 MHz, utilising ∼131000 dipole antennas.Operating as a single observatory these telescopes will offer advanced capabilities capable of addressing several fundamental questions of cosmology, astronomy and astrophysics.
Beginning in 2014 the design of SKA1 is being undertaken by a world-wide community of experts in the field, with the various aspects being tackled by nine consortia.One of these, Telescope Manager (TM), is designing the observatory and telescope software control systems, from science proposal preparation, through the execution of observations, configuring and monitoring the sub-systems, and to tracking the state of observations and projects through the system.A previous paper 1 introduced the Telescope Manager and the consortium, this paper provides a status report on the Telescope Manager, both the system and the consortium.

THE CONSORTIUM
The TM architecture described in this paper is the output of a consortium led by the National Centre for Radio Astrophysics (NCRA), India.The consortium consists of institutes and industry partners from five main contributing countries -India, United Kingdom, South Africa, Italy and Portugal, and two supporting countries -Australia and Canada.At the beginning of the pre-construction phase four key work packages had already been identified in the Work Breakdown Structure (WBS): Observation Management, Telescope Management; Local Monitor and Control, and Local Infrastructure, along with the supporting items of Project Management and Systems Engineering.Each functional team also performed exploratory prototyping.Analysis of User Interfaces (UI) and Authentication, Authorization and Accounting (AAA) were added later (in 2015).AAA is an additional, SKA-wide package taken on by the TM consortium.
Each of the partner countries took responsibility for key areas in the work breakdown structure, as indicated in this table, though contributions and team membership are often shared amongst the partners.

SCOPE AND PROBLEM DESCRIPTION
TM can be perceived as the central coordinator of each SKA telescope.It supports the use of the SKA Observatory and Telescopes to fulfill the objectives of important stakeholders such as the Science Principal Investigators and their teams, the SKA Office, SKA operations and the engineering teams spread across the partnering countries of the SKA.
The main responsibilities of TM are: 1. Management of astronomical observations: TM facilitates the activities towards the utilization of the SKA Observatory for conducting science.It is responsible for accepting science proposals, designing astronomical observations, scheduling them and then preparing the system for performing and executing those observations.The TM monitors the telescope health during astronomical observations and evaluates conditions that may impact observation activities.2. Management of the telescope operations: It is responsible for integrating and interfacing with various SKA systems and subsystems and facilitates the operational life cycle of each telescope.It orchestrates these systems to perform the operations of the SKA, and it also supports the assembly, integration and verification (AIV) of the telescopes, and commissioning, maintenance, and the troubleshooting of the entire SKA observatory and telescope systems.For carrying out scientific operations using the SKA telescopes, TM performs domain specific computations such as delay calculations and so on.3. Management of data to support SKA operations and stakeholders: TM is responsible for capturing and storing the health monitoring data of the SKA systems and subsystems.It is also responsible for providing data pertaining to the Telescope state to other interested systems in SKA.It provides the necessary support for performing diagnostics based on the archived data.It also delivers the data to the interested stakeholders through user interfaces.

TM Products
TM fulfills its responsibilities highlighted above through three main products, namely TM Observatory, TM Low and TM Mid.These products are derived to reflect the structure of the Observatory and Telescopes as set out in the Operational Concept Document 4 , to reflect their key responsibilities and to some extent where they will be deployed.TM Observatory is envisaged to incorporate capabilities enabling observation management related functions, many of which will be common between the two telescopes and executed primarily at the Global Headquarters (GHQ), though others will be telescope-specific and deployed at the individual telescope sites.TM Low and TM Mid will primarily incorporate the functionalities to operate each of the two telescopes SKA1-Low and SKA1-Mid for their scientific exploitation and engineering operations.The implementation of these products will be done based on three software modules, namely Observatory Science Operations (OSO), TM Control (TMC) and TM Services (SER).As is described here these modules do not map directly, one-to-one, to the products, but are derived from the initial analysis.An overview of the overall TM architecture can be found in section 4.

Initial analysis
As described above, the TM has three core responsibilities: 1. Management of astronomical observations; 2. Management of the telescope hardware and software systems in order to perform those observations; 3. Management of the data to support operators, maintainers, engineers and science users in achieving operational, maintenance and engineering goals.
TM does not have responsibility for the management of the science data products (such as visibilities, images, catalogues), which are the responsibility of the Science Data Processor (SDP) System, being designed by a different consortium.
The management of astronomical observations is largely the domain of the OSO module.This system provides tools to support proposal handling and the creation of observing projects corresponding to approved proposals.These projects consist of Scheduling Blocks (SBs).The projects will normally be independent of each other, but some observations may be identified as candidates for Commensal Observing, where data is shared between projects.The SBs contain the information needed to schedule and execute observations for a project.Further tools can create plans of SB execution and then also execute the SBs themselves.All data relating to observation management is stored in a dedicated and distributed Observation Data Archive (ODA).Observation Management, and thus the OSO Software Module, straddles all three products, TM Observatory, TM Mid and TM Low, as while much of its function is common and Observatorywide, the dynamic scheduling and execution of the observations is specific to each telescope product.
Before the execution of an observation on a telescope, it is necessary to configure the telescope appropriately to the needs of the observation.A key step in this configuration involves creating a 'sub-array', or a subset of the telescope resources that are sufficient for the observation.This sub-array has allocated to it the required capabilities from the different systems of the telescope, Receptors, Beam formers and so on.Once the configuration is done, the observation is executed on the sub-array.While executing observations and performing telescope management, the TMC module orchestrates the appropriate telescope systems, collects monitoring data that is used to track the status of all of the telescope systems, and responds to faults, threats and opportunities.Much of this functionality is common to TM Mid and TM Low, thus there are not separate top-level modules for each, rather the differences are handled via configuration and specialized instances of common components.The TMC manages monitoring and configuration data as a system model that describes the status of the telescope at any one time.The TM continually collects telescope configuration, telescope dynamic status and environmental data.The data are time-stamped and stored.The TMC is responsible for hosting and distributing the data carrying information related to the telescope state for the purpose of support and maintenance.TMC is responsible for storing and distributing aspects that are essential for carrying out the science such as delay calculations, pointing calculations and so on.
All three TM products include a self-monitoring and control function, delivered by the SER module that configures the rest of Telescope Manager, monitors its function, performs fault management as needed, and provides lifecycle support functionality including upgrades and shutdown.The presence of this independent function contributes greatly to the reliability of Telescope Manager.Dependability is also a major consideration in the design of the TM infrastructure, including redundancy and failover both for computation and communications in specific, critical areas.In particular, every TM application can specify a specific SLA (Service level agreement) with the Virtualization service in order to support these qualities.The high level functions of TM are organized firstly based on the level 2 (that is one below the level 1, or SKA level) product system.As the high-level functions are generic, no distinction is made between the TM Mid and TM Low products and here is referred simply as TM.The TM Observatory system has a separate set of functions.The functions are firstly grouped into related set of capabilities.Each of the high level functions is decomposed further by means of a functional analysis into sub-functions owned exclusively and respectively by parts of the Level 2 system.

Key Non-functional Requirements
The TM undertook a number of Mission Thread and Quality Attribute workshops 2 which, when combined with more direct requirements provided in the SKA1 Level 1 requirements 5 resulted in the identification of the following key areas for non-functional requirements and constraints on the whole TM system: • Availability -The software system must be available 24/7 and should not impact overall system availability; • Modifiability -The observatory will have a long life, it must be possible to upgrade, replace, evolve systems; • Usability -Minimize training time, minimize errors and their consequences; maximize productivity (and user satisfaction); • Scalability and extensibility to SKA2 -the system must be able to scale to the larger number of receptors proposed for the SKA Phase 2 project; • Security -user data must be kept secure and protected (Note: network security not in scope, provided by another consortium); • Testability -software components must be testable as far as is possible without access to the telescope hardware or specialized equipment; • Standardization -monitoring and control interfaces should be standized; • Location of equipment -the observatory is at distributed locations, with some challenging environments; • Constraints on power consumption -there are remote locations, and power costs must be kept low; • Construction cost -there is a capped construction budget; • Operations cost -on-going costs must be controlled.
Additional, more specific, quality attributes that drive the OSO, TMC, and SER architectures were also captured and addressed in each SAD respectively.Each SAD also contains information related to various architectural decisions made to achieve the quality attributes.

Conceptual Domain Analysis
From a domain perspective the Telescope Manager problem can be decomposed into a Control domain and a Monitor domain.Each of these decomposes further: Figure 2 (element description at Table 2) shows the Control domain dependencies, while Figure 3 (element description at   The control logic dealing with planning, definition, scheduling and executing of actions resulting in products useful in the scientific domain.This layer relates closely to the science that needs to be carried out.
Telescope Control Layer The control logic dealing with processes and configurations of telescope resources in order to produce the products required by the science management layer.

Telescope Subsystem Control Layer
The control logic dealing with processes and configurations to realise specific and detailed yet partial telescope subsystem outputs.Assigned to specific deliverable subsystems, not part of Telescope Manager.

Alarm Management
The logic (including rules) for creation of alarms, the notification of alarms and the management of alarm states.

Telescope Health Monitoring
Monitoring and bookkeeping health state of resources.

Telescope Subsystem Monitoring and Alarm Triggering Layer
The control logic for detecting alarm triggering events in individual subsystems.The logic for updating telescope subsystem state and publishing changes to the corresponding attributes and health status.

Module Decomposition
The domain analysis and the functional analysis of the problem (highlighted in the sections above) allows the derivation of four main software modules (shown in Figure 4) that will comprise the Telescope Manager software system and through which it will be possible to build the products described in section 3.1.Two of these modules (OSO and TMC) map closely to the top two domain layers in Figure 2, while the other two (SER and AAA) provide cross-cutting services for both in support of their domain functionality.The OSO software module is mostly dedicated to the management of astronomical observations and their translation into scripts that can be executed by the TMC module.The TMC software module is dedicated to the management of the telescope hardware and software.The SER software module contains important TM services that support both OSO and TMC: component monitoring and lifecycle management, logging and the virtualization service.
OSO, TMC and SER are also dedicated to the management of the data that supports operators, maintainers, engineers and science users in achieving operational, maintenance and engineering goals excluding management of the science data products (such as visibilities, images, catalogues), which are the responsibility of a separate Science Data Processor (SDP) consortium.
The AAA software module is dedicated to the authentication and authorization (plus auditing) and though it may be part of the TM work package, it is seen as external to the TM product.
Every software module directly involved in the management of the Telescope has to be built on top of a common framework: the TANGO Controls Framework 6 (identified in Figure 4 as the TANGO Core).It is expected that an SKAspecific Software Development Toolkit (SDK) that will harmonize use of TANGO across all systems will be developed in collaboration between various groups.
It is important to notice that the relation between TMC and AAA is only for user authentication and authorization at the application level.It is assumed that security for the telescope control and monitor systems is realized in many ways (i.e. with Trusted Zones, use of VPN and TANGO ACL) and is dependent on collaborative users and preventing mistakes.Network security is not the responsibility of TM.

User Interfaces
Not shown in Figure 4 or listed in Table 4 but very important are the User Interface (UI) modules.UI modules implement UI frontends and UI portals.UI frontends may be implemented using web technologies and/or, for TMC components, frameworks within the TANGO ecosystem.
UI frontends are used by users of the telescope, including engineers, commissioning staff, operators, astronomers on duty, operations staff in various roles, and the external investigators.
UI portals act as an intermediate layer between the telescope core components and the UI frontends.There are two strategic goals to be achieved with portals, to support high levels of quality in use for the TM, and sustainability during development, deployment and maintenance.Therefore UI portals should: • integrate information and user operations across all the TM products • support a flexible conceptual model to be exploited by UI Frontends • hide technologies used within the telescope core components and provide a uniform and lightweight API • support extendability of UI Frontends • support testability at service and system test levels.5, relations description at Table 6) shows an overview of the TM Observatory product.In TM.Observatory the OSO software module is split into five sub-modules and the SER software module is split into three sub-modules.Each of them is summarised in the following section.PTT Project Tracking Tool (PTT) allows the tracking of the execution of Observing Projects and allows updates from staff.There is also a component that updates the status of Observing Projects according to information from the telescope systems.
OPT Observation Planning Tool (OPT) is designed to aid the management of the observatory's science programme at a high level by analysing the SB definitions submitted through the Observation Design Tool and generating a plan for their observation.
ODA Observation Data Archive (ODA) is an archive to store observation management artefacts.

LM
The Lifecycle Manager (LM) is an IT automation tool that supports the control of a software application lifecycle.

SSM
The Software System Monitor (SSM) is a software component used to monitor resources and performance in a computer system.
Logging Service Centralized logging service to collect log data and support analysis of them.All Modules SSM Each OSO software module sends monitoring information to the SSM.

All Modules Logging Service
Each OSO software module sends log information through the normal TM Logging Service.
ODT PHT PTT PPT ODA The OSO software modules persist and exchange Proposal and Project data via the ODA.

TM Mid/Low View
Figure 6 (element description at Table 7, relations description at Table 8) shows a simplified top level of the TM Mid and Low products described in section 3.1.For these products there are three OSO software sub-modules.The SER software module is split in the same way as in TM Observatory and the TMC software module comprises five software modules.OST Tool that supports the near real-time scheduling of Scheduling Blocks from a list of schedulable SBs, and in response to telescope and other environmental conditions.

UI Module
The User Interface module is a joint OSO and TMC module concerned with receiving inputs from the SKA operators, which are processed by appropriate OSO or TMC components, and displaying output back to the operators.It supports real-time update of data, receiving and processing user gestures, exposes application functionality and notifies views of changes.
TM Monitor Provides generic monitoring information to the M&C Module

M&C Module
The M&C module is composed of a hierarchical structure which consists of various controllers: Central node, Sub-array nodes and Leaf nodes.Leaf nodes are sub-system Controllers primarily responsible for configuring the sub-systems, gathering data from all sub-systems and processing it to obtain a real-time view of the state, health status and performance parameters of each sub-system, and the Telescope as a whole.

External Interface Manager
External Interface Manager is responsible for acquiring data required for telescope operations from various systems (such as IERS, SIS, FIS, and IPS) that are external to the SKA1 Observatory.

Fault Manager
Fault Manager is responsible for detecting and handling Telescope Alarms and Alerts, TM Alerts, and alarm lifecycle management.It also allows users to configure alarm conditions and administer them.

Configuration Manager
Configuration Manager manages instrumental configuration data as well as calibration and telescope data.

Resource Manager
Resource Manager is responsible for the resource management and maintaining the resource matrix.The resource matrix provides current, up-to-date availability status of all sub-systems and their capabilities.Resource Manager also validates the resource allocation and deallocation requests.

EDA
The Engineering Data Archive, responsible for storing current and historical telescope state information and related engineering data.It also provides analytics capabilities to perform analysis of the stored data for diagnostics and evaluation of current and historic system state.

Reporting System
Report Generation System will generate report for various stakeholders based on historical data stored in EDA.

ODA
The Observation Data Archive is available for each telescope product.See also Table 5 LM Lifecycle Manager.Same description as is found in Table 5 SSM Software System Monitor.Same description as is found in Table 5 Logging Service Same description as is found in Table 5.The UI Module provides the User Interface for a number of entities of the system.

Bringing it together -a normal workflow
The sections above describe some detail of the TM architecture.However, it is useful to provide an overview of how it all fits together in a "normal" observing process.

THE PROCESS
In late 2016 the SKAO adopted the Software Engineering Institute (SEI) approach to software architecture design and documentation for all of its software 2,3 .At the same time the SKAO started to make it much clearer that SKA software would be constructed and delivered using an agile process (subsequently the Scaled Agile Framework 7 , SAFe©, has been adopted).
As a major software component of the SKA system, and the overall control system for the, this decision had a major impact on the processes and work to date of the TM consortium.This change affected all areas: project management had to negotiate a revised set of documentation deliverables with the SKAO; systems engineering had to accommodate a new approach to capturing non-functional requirements (as Quality Attributes) and individual software teams had to change their approach to documenting their designs.However, it is very clear that this was a beneficial change, resulting in a better architecture, and, most importantly, a better documented architecture.

Restructuring
At a meeting in early 2017 the TM Consortium made the decision to create a TM Architecture Team (TMAT).In fact this was not the first time such a team existed, but this time it was created with a much clearer remit and composed of a small group (four) of individuals chosen from across the consortium who were in good positions to understand the entirety of the TM problem and to drive forward its solution.The TMAT took the leadership role of trying to resolve a number of technical challenges, both for the overall architecture of TM and in assisting individual teams with issues they had, providing alternative "informed outsiders" insights.Some challenges of the TMAT are outlined in a companion paper 8 .
One of the very first challenges addressed by the TMAT was in resolving areas of responsibility for some functions, and this also determining internal boundaries and interfaces.At the time of the creation of the TMAT some of these boundaries were in danger of succumbing to Conway's Law 9 , and thus being somewhat artificial.The TMAT took a structured and reasoned approach to resolving these issues.The outcomes were usually not much different from the initial thinking, but now had reasoned, rational, analysis behind them and the individual teams could thus be slightly modified to reflect the TM Architecture, rather than the consortium make-up.This was a great improvement not only in the architecture and its documentation, but also in the level of communication between the teams.
the SKAO, in collaboration with the TM team and other consortia.The SKA System CDR is scheduled for March 2019.
In parallel progress towards the beginning of SKA construction is ongoing.For the software teams utilizing an agile approach, that construction is about to begin.

Figure 4 :
Figure 4: Top level Telescope Manager software modules and dependencies.Each of the TMC, OSO and SER modules include UI sub-modules.

Figure 5 (
Figure5(element description at Table5, relations description at Table6) shows an overview of the TM Observatory product.In TM.Observatory the OSO software module is split into five sub-modules and the SER software module is split into three sub-modules.Each of them is summarised in the following section.

Figure 5 :
Figure 5: TM Observatory uses module.Table 5: TM Observatory uses module, element descriptions.Element Description ODT Observation Design Tool (ODT) allows the design of Observing Projects (including Scheduling Blocks).PHT Proposal Handling Tool (PHT) allows the preparation and submission of Observing Proposals by external scientists, and the review and management of these proposals by Observatory staff.

Figure 6 :
Figure 6: TM Mid/Low uses module.Table 7: TM Mid/Low uses module, element descriptions.Element Description OET Tool supporting the execution of concurrent Scheduling Blocks.

Figure 7 :
Figure 7: A standard observing workflow

Table 1 :
Countries responsibilityFollowing a top-level analysis that derived the three top-level products and the associated software modules identified in Section 3 these modules were mapped to the key team responsibilities with the Observation Management team delivering the Observatory Science Operations (OSO) architecture, the Telescope Management team delivering the Telescope Monitor and Control (TMC), and the Local Monitor and Control team delivering the Telescope Manager Services (SER).The local infrastructure contributions were absorbed into the SER architectural design.Following the methodology prescribed by the Software Engineering Institute (SEI) 2,3each top-level software module is described by a Software Architecture Document (SAD).
The UI work is being delivered as a supporting analysis of the user interface challenges presented by the SKA, covering the areas of Design Principles, Usage Scenarios, User Level Use Cases and Storyboards.To support this work at the beginning of 2016 a small team consolidated and carried out activities aimed at gathering information from SKA precursors, at analyzing users needs and at defining UI design principles.AAA is a separate activity supporting the whole of the SKA covering the Authentication and Authorization of all users accessing the resources of the SKA Observatory and Telescopes.It is being delivered as a supporting set of Requirements Specification, Design Report and Interface Control Documents.There are also supporting documents on the Criticality, Testability and Dependability Analysis including Reliability, Availability, Maintainability (RAM) and Integrated Logistics, EMC Control Procedures, and Compute Infrastructure Deployment being delivered by the Local Infrastructure team.

Table 3
) shows the Monitor domain dependencies.

Table 2 :
Control domain dependencies, element descriptions

Table 3 :
Monitor domain dependencies, element descriptions

Table 4 :
Top level Telescope Manager software module descriptions.
SKA SDKPart of the TANGO controls framework development could be done by the SKA community.