SIDGRAPH

Modern applications with increasingly connected domain topologies require processing and programming abstractions that reflect the network structure inherent to these applications. At the same time, data-intensive applications necessitate more and more online processing capabilities when consuming incoming streams of events to execute continuous computations and provide fast results. Only very few systems have taken into account the combined challenge of executing graph processing on a dynamically evolving graph. However, this is a critical capability as timely computations enable reactive application behaviors upon graph changes. In addition, no existing system supports processing on a live graph and on past version of that evolving graph at the same time. The distinct characteristics of event processing and graph computing, as well as batch processing and live processing yield specific requirements for any computing platform that unifies these approaches. Solutions require (i) data management strategies to keep track of the continuous graph evolution, (ii) appropriate graph processing models that can simultaneously handle computations and graph updates, and (iii) an efficient platform implementation that provides the necessary performance at runtime. To this end, this thesis suggests a combination of an adapted actor model, an event-sourced persistence layer, and a vertex-based, asynchronous live programming model. The event-sourced actor model enables highly concurrent computations in which the full application history is implicitly persisted. This model is then refined into a live graph processing model with a particular focus on asynchronicity, liveness, and parallel execution support. At the same time, the use of event sourcing enables the model to reconstruct global and consistent graph representations from arbitrary points of the timeline. These graph representations form the basis for decoupled, non-live graph processing models. The Chronograph platform represents an implementation of the event-sourced graph model. The platform ingests external update streams and maintains a live graph representation as part of a user-specified graph application. It thus enables live and offline computations on event-driven, history-aware graphs and supports different processing models on the evolving graph. This thesis provides the following contributions: (i) a distributed computing approach with history support based on the actor model and event sourcing, as wall as corresponding and supplementary concepts, (ii) a data management approach for evolving graphs that builds on the event-sourced actor model, (iii) a set of novel and adapted programming and processing models that integrate well with event-sourced graphs, (iv) a distributed platform architecture that implements the event-sourced graph model; and (v) an evaluation framework for such live graph processing systems.

State changes over time are inherent characteristics of stateful applications. So far, there are almost no attempts to make the past application history programmatically accessible or even modifiable. This is primarily due to the complexity of temporal changes and a difficult alignment with prevalent programming primitives and persistence strategies. Retroactive computing enables powerful capabilities though, including computations and predictions of alternate application timelines, post-hoc bug fixes, or retroactive state explorations. We propose an event-driven programming model that is oriented towards serverless computing and applies retroaction to the event sourcing paradigm. Our model is deliberately restrictive, but therefore keeps the complexity of retroactive operations in check. We introduce retro-λ, a runtime platform that implements the model and provides retroactive capabilites to its applications. While retro-λ only shows negligible performance overheads compared to similar solutions for running regular applications, it enables its users to execute retroactive computations on the application histories as part of its programming model.

Distributed event-sourced systems adopt a fairly new architectural style for data-intensive applications that maintains the full history of the application state. However, the performance implications of such systems are not yet well explored, let alone how the performance of these systems can be improved. A central issue is the lack of systematic performance engineering approaches that take into account the specific characteristics of these systems. To address this problem, we suggest a methodology for performance engineering and performance analysis of distributed event-sourced systems based on specific measurements and subsequent, targeted optimizations. The methodology blends in well into existing software engineering processes and helps developers to identify bottlenecks and to resolve performance issues. Using our structured approach, we improved an existing event-sourced system prototype and increased its performance considerably.

Event sourcing is increasingly used and implemented in event-based systems for maintaining the evolution of application state. However, unbounded event logs are impracticable for many systems, as it is difficult to align scalability requirements and long-term runtime behavior with the corresponding storage requirements. To this end, we explore the design space of log pruning approaches suitable for event-sourced systems. Furthermore, we survey specific log pruning mechanisms for event-sourced logs. In a brief evaluation, we point out the trade-offs when applying pruning to event logs and highlight the applicability of log pruning to event-sourced systems.

Stream-based graph systems continuously ingest graph-changing events via an established input stream, performing the required computation on the corresponding graph. While there are various benchmarking and evaluation approaches for traditional, batch-oriented graph processing systems, there are no common procedures for evaluating stream-based graph systems. We, therefore, present GraphTides, a generic framework which includes the definition of an appropriate system model, an exploration of the parameter space, suitable workloads, and computations required for evaluating such systems. Furthermore, we propose a methodology and provide an architecture for running experimental evaluations. With our framework, we hope to systematically support system development, performance measurements, engineering, and comparisons of stream-based graph systems.

An increasing number of distributed, event-based systems adopt an architectural style called event sourcing, in which entities keep their entire history in an event log. Event sourcing enables data lineage and allows entities to rebuild any previous state. Restoring previous application states is a straight-forward task in event-sourced systems with a global and totally ordered event log. However, the extraction of causally consistent snapshots from distributed, individual event logs is rendered non-trivial due to causal relationships between communicating entities. High dynamicity of entities increases the complexity of such reconstructions even more. We present approaches for retrospective and global state extraction of event-sourced applications based on distributed event logs. We provide an overview on historical approaches towards distributed debugging and breakpointing, which are closely related to event log-based state reconstruction. We then introduce and evaluate our approach for non-local state extraction from distributed event logs, which is specifically adapted for dynamic and asynchronous event-sourced systems.

Several data-intensive applications take streams of events as a continuous input and internally map events onto a dynamic, graph-based data model which is then used for processing. The differences between event processing, graph computing, as well as batch processing and near-realtime processing yield a number of specific requirements for computing platforms that try to unify theses approaches. By combining an altered actor model, an event-sourced persistence layer, and a vertex-based, asynchronous programming model, we propose a distributed computing platform that supports event-driven, graph-based applications in a single platform. Our Chronograph platform concept enables online and offline computations on event-driven, history-aware graphs and supports different processing models on the evolving graph.

Many data-driven applications require mechanisms for processing interconnected or graph-based data sets. Several platforms exist for offline processing of such data and fewer solutions address online computations on dynamic graphs. We combined a modified actor model, an event-sourced persistence layer, and a vertex-based, asynchronous programming model in order to unify event-driven and graph-based computations. Our distributed chronograph platform supports both near-realtime and batch computations on dynamic, event-driven graph topologies, and enables full history tracking of the evolving graphs over time.

The actor model is an established programming model for distributed applications. Combining event sourcing with the actor model allows the reconstruction of previous states of an actor. When this event sourcing approach for actors is enhanced with additional causality information, novel types of actor-based, retroactive computations are possible. A globally consistent state of all actors can be reconstructed retrospectively. Even retroactive changes of actor behavior, state, or messaging are possible, with partial recomputations and projections of changes in the past. We believe that this approach may provide beneficial features to actor-based systems, including retroactive bugfixing of applications, decoupled asynchronous global state reconstruction for recovery, simulations, and exploration of distributed applications and algorithms.

The processing of large-scale data sets and streaming data is challenging traditional computing platforms and lacks increasingly relevant features such as data lineage and inherent support for retrospective and predictive analytics. By combining concepts from event processing and graph computing, an Actor-related programming model, and an event-based, time-aware persistence approach into a unified distributed processing solution, we suggest a novel processing approach that embraces the idea of graph-based computing with built-in support for application history.

Systems for highly interconnected application domains are increasingly taking advantage of graph-based computing platforms. Existing platforms employ a batch-oriented computing model and neglect near-realtime processing or temporal analysis. We suggest an extended conceptual model for event-driven computing on graphs. It takes into account the evolution of a graph and enables temporal analyses, processing on previous graph states, and retroactive modifications.

Discrete event simulations (DES) represent the status quo for many different types of simulations. There are still open challenges, such as designing distributed simulation architectures, providing development and debugging support, or analyzing and evaluating simulation runs. In the area of scalable, distributed application architectures exists an architectural style called event sourcing, which shares the same inherent idea as DES. We believe that both approaches can benefit from each other and provide a comparison of both approaches. Next, we point out how event sourcing concepts can address DES issues. Finally, we suggest a hybrid architecture that allows to mutually execute simulations and real applications, enabling seamless transitions between both.