This project is a WRENCH-based simulator of the Pegasus WMS. Since Pegasus relies on HTCondor, first we have implemented the HTCondor services as simulated core CI services, which together form a new Compute Service that exposes the WRENCH Developer API. This makes HTCondor available to any WMS implementation that is to be simulated using WRENCH.
HTCondor is composed of six main service daemons (startd, starter, schedd, shadow, negotiator, and collector). In addition, each host on which one or more of these daemons is spawned must also run a master daemon, which controls the execution of all other daemons (including initialization and completion). The bottom part of the simulator architecture figure below depicts the components of our simulated HTCondor implementation, where daemons are shown in red-bordered boxes. In our simulator we implement the 3 fundamental HTCondor services, implemented as particular sets of daemons, as depicted in the bottom part of the figure shown below in borderless white boxes. The Job Execution Service consists of a startd daemon, which adds the host on which it is running to the HTCondor pool, and of a starter daemon, which manages task executions on this host. The Central Manager Service consists of a collector daemon, which collects information about all other daemons, and of a negotiator daemon, which performs task/resource matchmaking. The Job Submission Service consists of a schedd daemon, which maintains a queue of tasks, and of several instances of a shadow daemon, each of which corresponds to a task submitted to the Condor pool for execution.
We implemented the simulated Pegasus WMS, including the DAGMan workflow engine, using the WRENCH Developer API. This implementation instantiates all services and parses the workflow description file, the platform description file, and a Pegasus-specific configuration file. DAGMan orchestrates the workflow execution, and monitors the status of tasks submitted to the HTCondor pool using a pull model. The top part of the architecture figure below depicts the components of our simulated Pegasus implementation (each shown in a red box).
WRENCH Research Paper:
- H. Casanova, R. Ferreira da Silva, R. Tanaka, S. Pandey, G. Jethwani, W. Koch, S. Albrecht, J. Oeth, and F. Suter, Developing Accurate and Scalable Simulators of Production Workflow Management Systems with WRENCH, Future Generation Computer Systems, 2020.
WRENCH-Pegasus is fully developed in C++. The code follows the C++11 standard, and thus older compilers tend to fail the compilation process. Therefore, we strongly recommend users to satisfy the following requirements:
- CMake - version 3.5 or higher
And, one of the following:
- g++ - version 5.4 or higher
- clang - version 3.8 or higher
If all dependencies are installed, compiling and installing WRENCH-Pegasus is as simple as running:
cmake .
make
make install # try "sudo make install" if you do not have the permission to writeThe main channel to reach the WRENCH-Pegasus team is via the support email: [email protected].
Bug Report / Feature Request: our preferred channel to report a bug or request a feature is via
WRENCH-Pegasus' Github Issues Track.