Source to searchable index. Every branch below is a scalability / resilience / cost decision, not a fixed recommendation — trace the path that matches your document volume and update frequency.
SCALABILITYRESILIENCECOST
Read every branch point in the diagram as a trade-off across these three forces — not as "the right way."
1
Data Ingestion
source → trigger → raw store
2
Parsing & Extraction
route by file type → extract → classify → human-in-the-loop
3
Preprocessing & Chunking
clean → chunk() interface → tag
4
Embedding
adapter → batch / retry / rate-limit
5
Storage & Indexing
cost vs. maintenance decision
Ingestion
Parsing / Extraction
Chunking
Embedding
Storage / Indexing
⋯ dashed = conditional path
◆ diamond = decision point
Hover any box for its role. Moving dots trace the direction of data flow through each layer.