New Projecting System

[dgafka]

Hey,
Starting new topic here to collect plan and ideas for new way of rebuilding projections. This is also follow up on #254.

Current State of Projections:

Currently Projections are based on Prooph's implementation. The problem with this solution is, it does not scale and requires recomponesation actions like Gap Detection or Projection State which does not allow to run it in parallel.

--- TODO > describe current design in details

Future Projections should follow three principles:

1. Being Resilient - It means that there should no way to end up in inconsistent state or having a need for do recompensation actions like Gap Detection.
2. Being Scalable - It means we should be able to rebuild projection in parallel. The process should be split in this way, that the only limitation are the system resources, not the design.
3. Being Available - No down time should be required. This means we should be able to register new projection of the same projection and allow for blue/green deployment, when we switch when the second projection is done with rebuilt.

--- TODO > describe future design in details

[lifinsky]

Hi. We can split and parallelize in 3 projection types:

• for inserts
• for updates
• for deletions

No problem with scale for inserts and deletions

[lifinsky]

We can group updates by event and process the same event types sequentially, but different in parallel

[lifinsky]

Let's consider an example of a batch of messages fetched from event stream - letter represents stream id and number is the sequence number of the event within that stream:

A1 - A2 - B1 - A3 - B2 - C1 - A4 - C2 - C3 - C4 - C5 - C6
If we then group messages using stream id we will end up with 3 sub streams:

A1 - A2 - A3 - A4
B1 - B2
C1 - C2 - C3 - C4 - C5 - C6
Now we can parallelize processing of each group of messages but within each group the messages will be processed sequentially, which will guarantee ordering (assuming events were not shuffled during grouping).

[dgafka]

@lifinsky for now I am into different matter, but I will get back on this. I had in my head something like you're proposing in your last message, so in our case aggregate id being a stream id and aggregate version being the number :)

@unixslayer will describe how it works currently with Prooph Projection integration. So we can keep the same high level API between new and current projecting system.

[lifinsky]

Hi, thanks. I'm more interested in whether it will be conceptually similar.

[unixslayer]

Prooph provides 3 types of persistence strategies:

1. SimpleStreamStrategy
2. SingleStreamStrategy
3. AggregateStreamStrategy

SimpleStreamStrategy is not meant to be used with Event Sourcing due its stream does not contain any constraints. This strategy is usefull if you have to combine events from other streams outside of ES scope.

SingleStreamStrategy lets you store events in a single stream with constraint using aggregate type and aggregate version. This strategy lets you persist events from a single or multiple aggregate(s).

AggregateStreamStrategy lets you store events from single instance of an aggregate into single stream. Every aggregate will have it's own stream names as {aggregate_type}-{aggregate_id} with {aggregate_type} as category.

Each strategy lets you use projections over their streams. Prooph provides following configuration for projections:

• fromAll - projection will load all stream names that exists in Event Store.
• fromStream(s) - you can define one or more streams from which projection will load Events by their names. With AggregateStreamStrategy you can define projection for a single instance of your aggregate.
• fromCategor(y/ies) - this is useful when you are using AggregateStreamStrategy and want to have a single projection. You can use one or more category names.

Events will be loaded from each of the streams that applies to from* using the same configuration, e.g. limits. All loaded streams will be merged into single one for processing.