Ordering data stream events in a streaming data storage platform

A streaming data storage system maintains streamed events based on external ordering (position) data. Each segment of the stream has minimum position maximum position metadata values based on the events' positions therein. When an ordered event is appended to an ordered segment, an index segmen...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
1. Verfasser: Paduroiu, Andrei
Format: Patent
Sprache:eng
Schlagworte:
Online-Zugang:Volltext bestellen
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:A streaming data storage system maintains streamed events based on external ordering (position) data. Each segment of the stream has minimum position maximum position metadata values based on the events' positions therein. When an ordered event is appended to an ordered segment, an index segment maintains offset information that tracks the event's position order relative to offsets of other previously appended events. When a segment is scaled (split or merged), the predecessor segment set's maximum position is the basis for the successor segment set's minimum position. An event is written to any predecessor segment or successor segment based on the event's routing key and position with respect to a matching segment's routing key and minimum and maximum positions. A data stream can be truncated based on a specified position. Reads can be for events between a specified range of positions, including a range that spans predecessor and successor segments.