Added sample configuration of sending EventStoreDB logs to Elastic together with Kibana integration.

That includes:

• Filebeat,
• Logstash,
• Filebeat and Logstash.

See also related documentation PR: https://github.com/EventStore/EventStore/pull/3488.

While the exact layout of a streamId is pretty arbitrary, in general it works better to have the tenant to the right of the '-' in the streamId as the default category projection treats anything to the left of the first '-' as the category (while this is configurable, this is the default, so Pit Of Success principles apply)

Protecting Sensitive Data in Event-Sourced Systems with Crypto Shredding

This sample is showing an example of using the Crypto Shredding pattern with EventStoreDB. This can be a solution for handling, e.g. European General Data Protection Regulation.

Read more in the Diego Martin article "Protecting Sensitive Data in Event-Sourced Systems with Crypto Shredding".

Description

• shows how to Protecting Sensitive Data (e.g. for European General Data Protection Regulation) in Event-Sourced Systems.
• shows how to use the .NET System.Security.Cryptography library with AES algorithm to encrypt and decrypt events' data.
• uses EventStoreDB.

Other changes:

Updated the CI configuring for showing test results to:

• not override each other if there are other changes
• be able to be run by external PR. Unfortunately, that part is only to be verified once it's merged because of how GH is running the workflow triggers (the workflow file has to be on the main branch already).

Added E-Commerce sample showing the classic Event Sourcing and CQRS flow with EvenStoreDB in .NET.

Overview

• typical Event Sourcing with CQRS flow.
• stores events to EventStoreDB.
• shows how to organise the write model and read model handling.
• Builds read models using Subscription to $all.
• Read models are stored as ElasticSearch documents.
• Shows how to unit and integration test solution.

Write Model

• Provides the basic boilerplate together with Core projects,
• EventStoreDBRepository repository to load and store aggregate state,
• IProjection interface to handle the same way stream aggregation and materialised projections,
• Thanks to that added dedicated AggregateStream method for stream aggregation
• See sample Aggregate and base class

Read Model

• Read models are rebuilt with eventual consistency using subscribe to all EventStoreDB feature,
• Added hosted service SubscribeToAllBackgroundWorker to handle subscribing to all. It handles checkpointing and simple retries if the connection was dropped.
• Added ISubscriptionCheckpointRepository for handling Subscription checkpointing.
• Added checkpointing to EventStoreDB stream with EventStoreDBSubscriptionCheckpointRepository and dummy in-memory checkpointer InMemorySubscriptionCheckpointRepository,
• Added ElasticSearchProjection as a sample how to project with left-fold into external storage. Another (e.g. ElasticSearch, EntityFramework) can be implemented the same way.

Tests

• Added sample of unit testing in Carts.Tests:
  • Aggregate unit tests
  • Command handler unit tests
• Added sample of integration testing in Carts.Api.Tests
  • API integration tests

Other

• EventTypeMapper class to allow both convention-based mapping (by the .NET type name) and custom to handle event versioning,
• StreamNameMapper class for convention-based id (and optional tenant) mapping based on the stream type and module,
• IoC registration helpers for EventStoreDB configuration,

Besides that:

• Added README.md with repository description
• Added CONTRIBUTING.md with contribution guidelines
• set MIT Licence
• Github Action with build, test and acceptance tests for the sample and test report

• Updated .NET CQRS flow to .NET 6, bumped packages to the latest version
• Moved to file-scoped namespaces.
• Updated obsolete usage of AesManaged to Aes.Create. From: https://www.meziantou.net/cryptography-in-dotnet.htm#aes:

  "AESManaged is fully implemented in .NET, however, the implementation is not FIPS compliant. AESCryptoServiceProvider uses the Windows implementation (CryptoAPI) which is FIPS compliant. Or you can create an instance of the best available provider using AES.Create()."

Note for reviewers: Review with whitespace hidden, there is ten time less line change https://github.com/EventStore/samples/pull/13/files?diff=split&w=1

• Updated .NET CQRS flow to .NET 6, bumped packages to the latest version
• Moved to file-scoped namespaces.
• Refactored and simplified Subscription to all processing.
• Fixed bug in the ElasticSearchProjection processing: Entity wasn't getting from the database because of missing index name.
• Added tests for:
  • adding product item,
  • removing product item,
  • confirming shopping cart.
• Added StreamEvent wrapper to pass event metadata (e.g. to support idempotency in the projections).
• Updated ElasticSearchProjection to generically support idempotency.
• Hidden generic Publish method in EventBus, as it may be getting the wrong Type (e.g. base class instead of the exact one). It's also redundant to keep it public, as the object has a method.
• Updated SubscribeToAll logic to not fail when it wasn't possible to deserialise even.
• Removed test report workflow, as it's flaky and not adding much value.
• Disabled Test parallelization for API tests to not have deadlocks on starting subscription host.

Note for reviewers: Review with whitespace hidden, there is ten timeless line change https://github.com/EventStore/samples/pull/12/files?diff=split&w=1

• [x] Removed MediatR library not to have a redundant dependency from the external package (no advanced features like pipelines were used)
• [x] Removed marker interface for Commands, Queries and Events to reduce potential ceremony

The logic in StreamNameMapper.ToStreamPrefix: https://github.com/EventStore/samples/blob/285a71700202166af78028d570d3d6b080c75c42/CQRS_Flow/.NET/Core/Core/Events/StreamNameMapper.cs#L23-L27 is problematic at present as it appears to map MyCompany.Domain.Cart to MyCompany_ItemAdded

While I'm not a fan of seeing the naming of events as a global thing in the first instance*, I feel that this code is worse than useless as you'll inevitably end up with multiple MyCompany_Added things if you start from this code, which will conceal the objective of the code

I'd be more than happy to make a PR to remove the prefixing logic, but would settle for any resolution that's more likely to end up with a useful convention as people inevitably paste this code around the world ;)

• (for me, the event types should be relative to a Category and not even be subject to a global uniqueness constraint within a given store)

I was porting your C# code to F# when I've realized that something didn't add up and most notably how merging quantities is achieved in the current implementation:

https://github.com/EventStore/samples/blob/a552bc5d564cd4121807bc0fba958f5e01e4627e/CQRS_Flow/.NET/Carts/Carts/Carts/Cart.cs#L92-L110

https://github.com/EventStore/samples/blob/a552bc5d564cd4121807bc0fba958f5e01e4627e/CQRS_Flow/.NET/Carts/Carts/Carts/Products/PricedProductItem.cs#L46-L52

https://github.com/EventStore/samples/blob/a552bc5d564cd4121807bc0fba958f5e01e4627e/CQRS_Flow/.NET/Carts/Carts/Carts/Products/ProductItem.cs#L30-L36

Not only there is a redundant check as part of the current implementation:

https://github.com/EventStore/samples/blob/a552bc5d564cd4121807bc0fba958f5e01e4627e/CQRS_Flow/.NET/Carts/Carts/Carts/Products/PricedProductItem.cs#L41-L44

https://github.com/EventStore/samples/blob/a552bc5d564cd4121807bc0fba958f5e01e4627e/CQRS_Flow/.NET/Carts/Carts/Carts/Products/ProductItem.cs#L46-L49

but actually considering how data are stored:

https://github.com/EventStore/samples/blob/a552bc5d564cd4121807bc0fba958f5e01e4627e/CQRS_Flow/.NET/Carts/Carts/Carts/Cart.cs#L21

I think it would be relevant to use a map or a dictionary to store the priced items.

It would make the whole consistency check a lot simpler.

Also why bother throwing exceptions when you can add a product when the same product isn't already part of the collection and then merge quantities when it is.

Wdyt?