Event Sourcing for Micro Services

This is a hypothetical & high level architecture for Event Sourcing.

Key Architectural Considerations

1. Beyond Eventual Consistency.
2. Asynchronous.
3. Command Query Responsibility Segregation (CQRS).
4. Fault tolerant & Fail safe
5. Scalability.
6. Zero event lost & Exactly once.

Brief Explanation.

1. Micro-services application will publish events to Kafka.
2. Kafka consumer (sink connector) will consume and insert into Mysql Event Source table within a transactional context.
3. Mysql Change Data Capture (source connector) which runs within Kafka Connect will detect changes.
4. Kafka source connector will transform events and publish to Kafka.
5. Kafka connect will consume events from Kafka.
6. Elasticsearch sink connector will insert events into ES and update relevant MySql record activeness within transactional context.
7. Micro-Service query data from API.
8 & 9. API return latest data unless it's an DELETE event.

Events Cleaning

1. There are schedulers to clean/delete both MySQL and ES active index.
2. All the events will be rest in historical index.
3. MySQL events will be partitioned based on it's activeness.