introduction.tex

\chapter{Introduction}
The main goal of the bookkeeping system of A Large Ion Collider Experiment (ALICE) is to keep the accounting for the different operational activities of the experiment. Different workflows and operational activities are performed daily and it is extremely important to register on which conditions they are done and keep an historical record. 

The purpose of this Software Requirements Specifications document is to have a central point for all the requirements of the bookkeeping system. It is not written with the goal to have a document with definite and final requirements for this system. During the process of developing this system requirements will be added and modified.

Although this document aims to comply with IEEE Std 830-1998, the first chapter deviates from this standard. The reason for this is the need to describe the contest of the envisioned system. It was also deemed important to present a description of the system currently in use. This will give a view on the scope and general functionality of the system to develop.

In this introductory chapter ALICE itself is presented. Then the update of ALICE and the bookkeeping system is elaborated. Moreover, the actual situation or how bookkeeping has been done during Runs 1 and 2 is presented. In Section 6 definitions of terms used are given and an explanation of acronyms is provided. The chapter concludes with references to literature about the bookkeeping system and an overview of Chapter 2 and 3.

\section{The ALICE experiment}
The ALICE experiment is a collaborative experiment of approximate 1800 persons from 176 institutes in 41 countries. The collaborative work of this scientific and technical community is largely distributed over the planet and covering it from UTC-8 till UTC+9. The experiment is taking data and producing physics results since 2009. During the Long Shutdown 2 (LS2) in 2019 and 2020 a major upgrade of certain key detectors and the experiment's computing system will be done. 

\section{The ALICE upgrade}
During LS2, ALICE will be upgraded. The upgrade will be done on many subsystems:
\begin{description}
  \item[CTP] A new Central Trigger Processor will be installed.
  \item[FIT] New Fast Interaction Trigger detector providing minimum bias trigger to the experiment.
  \item[ITS] A new Inner Tracking System will be installed. This new system will have an improved pointing precision. It will also be made of less material than before and become the thinnest tracker at the LHC.
  \item[MFT] A new Silicon tracker located between the ITS and the front absorber to improve Muon pointing precision.
  \item[MCH] The Muon arm will get new continuous readout electronics.
  \item[O2] A new computing system will be installed, combining the traditional functions of Data Acquisition (DAQ), High Level Trigger (HLT) and Offline.
  \item[TOF] The Time Of Flight detector will have new and faster readout electronics.
  \item[TPC] The Time Projection Chamber will use new GEM technology for the readout chambers. The readout will be continuous and it will use faster readout electronics.
  \item[TRD] The Transition Radiation Detector will be read out faster.
  \item[ZDC] The Zero Degree Calorimeter will be read out faster.
\end{description}

\begin{figure}[h]
  \begin{center}
    \includegraphics[scale=0.4]{./images/alice_upgrade.jpg}
    \caption{An overview of ALICE}
    \label{fig:overview}
  \end{center}
\end{figure}
The 50 kHz collision rate will produce about 100 times more data than was produced in 2010. This will give the physicists the opportunity to detect very rare processes with very small signal over background ratio. Because the traditional triggering techniques are very inefficient if not impossible, most detectors will be continuously read-out. Therefore, a new computing system, $O^2$, is being developed. It will read-out the data of all interactions and compress these data intelligently by online reconstruction. The system will merge the traditional Online and Offline functions.

Unmodified raw data of all interactions will be shipped (via 9000 GBT links) at an aggregated data rate of 3.4 TB/s from the detectors to a first farm of 270 First Level Processor (FLP) nodes equipped with FPGA PCIe cards. In this farm, baseline correction and zero suppression will be applied to reduce the data volume to 500 GB/s. The resulting data will be shipped via a switching network to a second farm of 1500 Event Processing Nodes (EPN) equipped with GPUs, where further data volume reduction will be achieved by online tracking to reach a data rate of 100 GB/s to permanent storage. In the data storage one year of compressed data will be kept. A few hours later, a subsequent reconstruction done either in the EPNs (2/3) or on Grid sites (1/3) with the final calibration will be done.

After LS2, the LHC will produce Pb–Pb collisions at up to $L = 6.10^{27} cm^{-2} s^{-1}$, corresponding to an interaction rate of 50kHz. During Run 3 (2021-2023), pp collisions will be done at 14 TeV and Pb-Pb collisions at 5.5 TeV. In 2023, no Pb-Pb collisions are scheduled. After LS3 (2024-2026) the same schedule will be realised except for 2028 when p-Pb will be run at 8.8 TeV.

\begin{figure}[h]
  \begin{center}
    \includegraphics[scale=0.3]{./images/timetable.png}
    \caption{Time table for the LHC}
    \label{fig:timetable}
  \end{center}
\end{figure}


\subsection{Work packages}
To manage the $O^2$ upgrade, several work packages were created:
\begin{description}
  \item[WP 1] Data Model
  \item[WP 2] Data Flow and System Simulation
  \item[WP 3] Common Tools and Software
  \item[WP 4] $O^2$ Software Framework– Common tools and infrastructure
  \item[WP 5] Data distribution and load balancing
  \item[WP 6] Detector readout
  \item[WP 7] Quality Control
  \item[WP 8] Control, Configuration and Monitoring
  \item[WP 9] Event Display
  \item[WP 10] Constant and Calibration DB
  \item[WP 11] ALFA
  \item[WP 12] Simulation
  \item[WP 13] Reconstruction and Calibration
  \item[WP 14] Analysis framework and facility infrastructure
  \item[WP 15] Data Management
  \item[WP 16] Computing Room CR1 (FLP)
  \item[WP 17] Computing Room CR0 (EPN)
\end{description}

The bookkeeping system is part of WP 8. WP 8 has once every two weeks a meeting at Tuesday morning at 10.00 AM. Once every two months the $O^2$ technical board meets. An $O^2$ data model for bookkeeping should be available in January 2018.


\section{ALICE activities and workflows}

This section describes the main workflows that the computing system of the experiment will have to handle.

\subsection{Synchronous data taking}
The synchronous data taking will consist of reading out the data of all detector channels, assembling the different pieces first into subtimeframes (STF) in the FLPs and later into timeframes (EPNs), reducing the data volume via fast reconstruction with temporary calibration and finally writing the resulting compressed timeframes (CTF) to storage.  All this is done in memory and synchronous with LHC operations, therefore downtime in this workflow will results in lost data.

\subsection{Asynchronous reconstruction}
Asynchronous reconstruction will be performed a few hours after the synchronous data taking is finished, once the final calibration sets are made available. Due to lack of resources on the O2 Farm, one third of this workflow will be executed on the Grid. 

On the same original raw data set multiple reconstruction passes can be done. This can be due to a bug in earlier reconstruction software or to the availability of a better calibration set. It may happen that the same data will be re-calibrated 20 times. The bookkeeping system should keep track of this recalibration/reconstruction cycles.

\subsection{Simulation}
To know which collisions are of interest and what we can learn from these collisions, simulations are created using among other methods the Monte Carlo (MC) statistical method. These simulations will run on the Grid and will therefore be completely decoupled from the LHC operations. Normally such simulations are associated with or anchored to an actual data set taken from a run, i.e. the conditions for the run are replicated for the simulations. There is no limit to the number of MC productions for a particular run. A new MC production can be done because of new software releases or because the chosen parameters changed.

\subsection{Analysis}
Physicists from all over the world are doing analyses on the reconstructed data and produce papers about their findings. The physicists are the end user of ALICE and the bookkeeping system. The analyses are run on the Grid which is completely decoupled from the LHC operations. Normally these analyses run as trains, i.e. multiple analysis jobs using the same data sets are packed together (as trains consisting of various wagons).


\section{Purpose}
In this section the purpose of the application is described. Also, the reasons for and ideas on the update of the bookkeeping system will be elaborated.

\subsection{Upgrade of the bookkeeping}
As mentioned before, ALICE will do a major upgrade in 2019/2020 during which the new Online-Offline ($O^2$) computer system will be implemented and deployed. This gives an opportunity to upgrade the bookkeeping facility currently in use, which consists of two separate systems developed since 2009 and improved throughout the years: the ALICE Electronic Logbook and AliMonitor. In \ref{section:Overview of the current bookkeeping} a description of the current system is given. 

The motivation to do an upgrade is threefold. First it makes sense to unify the two systems now used to bookkeeping for ALICE. In this respect it should be mentioned that some users of AliMonitor themselves use separate systems to store information which arguably should be part of the bookkeeping system, e.g. JIRA or Twiki pages. Secondly, although in the years of its existing a lot of features were added, this has not been done anymore since there was the possibility that the system would be replaced. Features which have been mentioned the last few years were only implemented when really needed. This means that there is a list of features not implemented but still wished for. Lastly, because the system was developed starting around 2009 it is based on relatively old technologies. And because of its scale and its importance to the ALICE operations, such a large change in the bookkeeping system of the experiment can only be done together with the LS2 upgrade. This means that, if not upgraded, the bookkeeping system would be 20 years old by the end of Run 4 in 2029.

The goal is to develop a bookkeeping system which is compatible with different media and has a unified look and feel. It should be able to import the data of the current system and be accessible from inside and outside of CERN. The authorisation and authentication should be integrated with the central Single Sign On systems available at CERN. It should be supported as long as it is in use by ALICE, which according to the current schedule means until 2029.

There are several challenges for the update of the bookkeeping system. The bookkeeping system is critical to the ALICE operations because it ensures that all interventions are properly documented and all workflows are properly accounted. These workflows which should be registered in the system are quite complex. Because the Amsterdam University of Applied Sciences (AUAS) is a newcomer to the High Energy Physics community a lot has to be learned about vocabulary and jargon. The schedule for the project is also a challenge. At the end of 2019, a first version of the system has to be delivered while the whole system should be in production in 2021. Another difficulty for the project is found in the migration from the existing to the new system. A part of the bookkeeping system concerns the offline activities for data reconstruction, calibration and simulation. This work will not stop during LS2 and therefore a continuous use of the system will take place. Lastly, support for the system will have to be done until 2029.

During Run 1 and 2 a lot of experience concerning the use of the bookkeeping system was accumulated. Developers of the new system have the opportunity to elicit the experience of the past users and use this to their advantage. Modern technologies should make it more easy to develop a bookkeeping system. Because the AUAS is the task owner to create the bookkeeping system it is expected this should generate a lot of energy to make it happen.

\subsection{Vision}
Provide unified bookkeeping experience for operations, run catalogue and management.

\subsection{Needs}
Access in a single place all metadata related with operational activities. Keep historical record of used configurations, statistics and data quality. Produce reports for operational teams and management. 

\subsection{Product}
Dashboards for run metadata with different levels of detail. Search for data sets that match given criteria. Forms for creating textual log entries. Notifications for interventions, main events and summary reports. REST API for read/write access to metadata repository. 


\subsection{Business goals}
Adapt to new $O^2$ data model. Consolidate existing ALICE Electronic Logbook and Run Conditions Table in a single product. Refresh used technologies and make the product more future oriented. Integrate gathered experience and introduce missing features. 



\section{Scope of the Bookkeeping System}
The scope of the project to develop the bookkeeping system is restricted to keeping track of the configuration of ALICE, data produced by ALICE, and computations on this data. The bookkeeping system is not a monitoring system for ALICE.


\subsection{Overview of the current bookkeeping}

The bookkeeping is important because of several reasons. In the bookkeeping system accounting is done for:
\begin{itemize}
  \item configurations of the detectors during the data taking runs,
  \item the conditions which existed during the data taking runs,
  \item the statistics which are generated using the data.
\end{itemize}

The bookkeeping system is also used to search and select specific information concerning these runs. Questions such as `which runs included detector X?' or `which data sets should I use for my analysis?'. Furthermore, it is used for manually and automatically generate reports about for instance how much data was collected during a given period or the efficiency during a period.

Momentarily there are two systems which are part of the bookkeeping system:
\begin{itemize}
  \item the electronic logbook (http://cern.ch/alice-logbook),
  \item AliMonitor (http://alimonitor.cern.ch/).
\end{itemize}
A few scattered repositories are also used. These are mainly human-curated web pages.


\subsubsection{The ALICE Electronic Logbook}
The ALICE Electronic Logbook stores text based reports on interventions and incidents called Log Entries. These Log Entries are typically created by:
\begin{itemize}
  \item run coordinator (and deputy), shift leader and shifters which are 24/7 present in the control room,
  \item technical coordination, for cooling, infrastructure etc,
  \item detector teams for expert interventions,
  \item automatically by software elements of the different subsystems.
\end{itemize}

The logbook also stores online configurations and statistics of the:
\begin{itemize}
  \item Data Acquisition system (DAQ),
  \item High Level Trigger (HLT),
  \item Trigger,
  \item Large Hadron Collider (LHC),
  \item Detectors (TPC etc.).
\end{itemize}

In all the logbook has approximately
\begin{itemize}
  \item 37 GB of metadata,
  \item 195,000 log entries,
  \item 20,000 file attachments,
  \item 280,000 runs
  \item 5,000 LHC fills
\end{itemize}

For this it uses a MySQL database with 27 tables and 312 fields. There is a single instance for online usage and a daily copy for external uses. Back-ups are sent to CERN-IT. The logbook has several APIs:
\begin{itemize}
  \item C for core software,
  \item DIM for detector teams,
  \item REST for other systems.
\end{itemize}
The Web GUI was created by using PHP on the server side. For this no framework is used. On the client side Javascript is used.


\subsubsection{AliMonitor}

aliMonitor is used for bookkeeping of the Grid operations done for data reconstruction, simulation, calibration and Quality Assurance.  The Grid is composed of Tier 0 and Tier 1 computer farms located at CERN and Budapest (Tier 0) and at 10 other places around the world (such as SARA and Nikhef in Amsterdam) for Tier 1. Users of AliMonitor are:
\begin{itemize}
  \item Data Preparation Group (DPG) to organize reconstruction and simulation,
  \item Physics Working Groups for simulation and analysis
  \item ALICE physicists for simulation and analysis.
\end{itemize}

The activities done on the Grid are called jobs. A masterjob is loosely coupled to a run. Each masterjob is composed of subjobs which are loosely coupled on raw files. These jobs can be reconstruction passes and Monte Carlo (MC) productions. Reconstruction is done to adjust the data and correct biases present due to imperfections of the detectors. Although a lot of these jobs are automated, human intervention is possible and sometimes necessary.

The calibration and correction of raw data is not a one time event but can be done several times because of several reasons. It is possible that it is discovered that there is a specific bias in the data which should be corrected or it is possible that someone creates a better algorithm, etc. The re-calibration is currently being done within months. In the future it might be possible to do it within hours because a part of the reconstruction will already be done online. Because it is possible that data will be re-calibrated maybe even as often as 20 time the new bookkeeping system should be flexible enough to handle various scenarios.

When different jobs are being done on the same data, they are put together in so called trains. These trains are a sequence of wagons each representing a job and all jobs process the same data. From the viewpoint of efficiency this makes a lot of sense. Currently, a tool called Lightweight Process Manager (LPM) initiates this sort of operations.

In all there is 97 GB of metadata. There are 12.9 million master jobs (1 million simulation, 225,000 reconstruction, 5 million organised analysis and 6.5 million user jobs). There are 1 billion subjobs which are not accounted for permanently but summarized per master job. In the LPM there are 56,000 processing chains which are grouped in 13,000 job types (aka productions).

AliMonitor is using a PostgreSQL database with about 30 tables. It has two instances which are periodically backed up. There are APIs for JDBC used by core processes and HTTP for csv- and XML-files. The web GUI runs on a Tomcat webserver using Java Server Pages.

\subsubsection{Scattered repositories}
There are a few scattered repositories such as:
\begin{itemize}
  \item TWiki
  \item JIRA
\end{itemize}


\subsection{Time constraints}

The bookkeeping system consists of several distinct subsystems with separate features and deadlines:
\begin{itemize}
  \item At 1 January 2019 the electronic logbook feature of the bookkeeping system has to be in place, to be of assistance for the commissioning of several detectors. 
  \item the $O^2$ software has to be in place later. This concerns functionalities such as:
  \begin{itemize}
    \item QA
    \item statistics for runs
  \end{itemize}
  \item bookkeeping of the synchronous data taking
  \item bookkeeping of the asynchronous reconstruction
  \item reporting
\end{itemize}
In the middle of 2020 a global commission is planned. By then the bookkeeping of synchronous data taking should be possible. The generation of reports should be possible starting January 2021.

\begin{tabular}{cc}
\hline
Feature & deadline\\
\hline
\hline
 Reconstruction and callibration  & 2020 commissioning \\
 \hline
Simulation   & 2020\\
\hline
Analysis & 2020\\
\hline
\end{tabular}

\subsection{Migration}
Data taking with the current system stops end 2018. Then a migration of the data in the ALICE Electronic Logbook can be done. Old statistics from the current system should be combined with the new system. And the data stored in the old database should be migrated to the new database or the old database could be kept in place and have an interface with the new system. 

The offline activities will continue during LS2. A date has to be set to migrate to the new bookkeeping system. It should be carefully prepared which part will be migrated.

\section{Definitions, Acronyms, and Abbreviations}

\subsection{Definitions}
\subsubsection{Log Entry}
A Log Entry is a text message that describe an intervention or an event that happened. It can be generated either by humans (e.g. a shifter enters his/her end-of-shift report) or by machines (e.g. a detector logs some abnormal situation automatically). 

\subsubsection{Run}
A Run is a unique ID that identifies a synchronous data processing session in the $O^2$ computing system with a specific and well-defined configuration. It normally ranges from a few minutes to tens of hours. It is generated and managed by the $O^2$ system. 

\subsubsection{Period}
Periods contain runs under the same conditions.

\subsubsection{LHC Fill}
An LHC Fill is a unique ID that identifies a period of activity in the LHC accelerator. It normally ranges from a few minutes to tens of hours. It is generated and managed by the LHC system and published via DIP protocol. 

\subsubsection{Production}
All (master) jobs of a given type are grouped together. They get a unique tag, eg. LHC17g8a\_fast. Here is `17' the year of production, `g' a symbol for Monte Carlo jobs and `8a' a number given by the bookkeeping system. 

\subsubsection{Job}
A job is a wrapper around an AliEn masterjob ID. It belongs to a production ID. There is an owner and an output directory. The parent job ID should also be given to chain processes. 

\subsubsection{Subjobs}
Jobs can be split into subjobs. These jobs are executed independently.

\subsubsection{Train}
A train is a considerable amount of jobs which all use the same data set. This is to reduce the I/O for all these jobs.

\subsection{Acronyms} 
In this section most of the acronyms used in this document are mentioned, explained and presented in Table \ref{tab:acronyms}.
\begin{table}[h]
\begin{center}
\begin{longtable}{ll}
    \hline
    ALICE & A Large Ion Collider Experiment\\
    \hline
    ADC & \\
    \hline
    API & Application Programming Interface\\
    \hline
    DAQ & Data Acquisition subsystem \\
    \hline
    DPG & Data Preparation Group\\
    \hline
    ID & Identity Document\\
    \hline
    IEEE & Institute of Electrical and Electronics Engineers\\
    \hline
     LHC  & Large Hadron Collider\\
     \hline
     LPM & Lightweight Process Manager \\
     \hline
     MC & Monte Carlo (statistical method)\\
     \hline
     $O^2$ & Online and Offline\\
     \hline
     SAMS & Shift Accounting Management System \\
     \hline
    \end{longtable}
      \caption{Acronyms}
  \label{tab:acronyms}
  \end{center}
  
\end{table}

\section{References}
The information offered in this SRS originates from several sources. Firstly from interviews done in December 2017 at CERN. Secondly from the ongoing process of requirements engineering started in 2018. Also several documents produced by Vasco Barosso in 2017. Furthermore, several documents produced before have been studied:
\begin{itemize}
  \item ALICE $O^2$ Technical Design Report November 30, 2015
  \item Barroso, Vasco et al, "The ALICE Electronic Logbook," in IEEE-NPSS Real Time Conference, Lisbon, 2010.
  \item "ALICE Electronic Logbook," ALICE Collaboration, [Online]. Available: https://cern.ch/alice-logbook. [Accessed 01 November 2017].
  \item Bagnasco, Stefano et al, "Alien: ALICE environment on the GRID," in Computing in High Energy Physics (CHEP), Victoria, Canada., 2007.
  \item "Alien - ALICE Environment Grid Framework," ALICE Collaboration, [Online]. Available: http://alien.web.cern.ch. [Accessed 01 November 2017].
  \item Grigoras, Costin et al, "A tool for optimization of the production and user analysis on the Grid," in Computing in High Energy Physics (CHEP), Taipei, 2010. 
  \item https://alice-o2.web.cern.ch/
\end{itemize}


\section{Overview}
This document is structured according to the IEEE standard 830-1998 Recommended Practice for Software Requirements Specifications. In Chapter 2 a not very detailed description of the functional requirements is given. This is elaborated in Chapter 3.