Building Data Pipelines for Search: From Raw Content to Vector-Ready Indices

When building a modern semantic search system, the data pipeline is where everything begins. You can have the most sophisticated retrieval algorithms in the world, but if your data isn't properly ingested, parsed, normalized, and enriched, your search quality will suffer.

This post explores the principles and patterns for building production data pipelines that transform raw content into semantically-enriched search indices.

The Challenge

Search applications face several data engineering challenges:

Diverse source formats: Content might come from structured databases, semi-structured XML/JSON, unstructured text, or legacy systems with their own formats.

Hierarchical content: Documents often have natural structure—titles, sections, subsections—that should be preserved for retrieval.

Enrichment needs: Modern search benefits from embeddings, extracted entities, geographic coordinates, and other derived data.

Scale and reliability: Pipelines must handle large volumes reliably, with the ability to resume after failures.

Pipeline Architecture Patterns

A well-designed data pipeline separates concerns into distinct stages, each independently resumable:

┌─────────────────────────────────────────────────┐
│             DATA PIPELINE STAGES                 │
├─────────────────────────────────────────────────┤
│                                                  │
│   ┌──────────┐     ┌──────────────┐             │
│   │  Source  │────▶│  Extraction  │             │
│   │  System  │     │  & Migration │             │
│   └──────────┘     └──────────────┘             │
│                           │                      │
│                           ▼                      │
│                    ┌──────────────┐             │
│                    │ Base Index   │             │
│                    │ (Normalized) │             │
│                    └──────────────┘             │
│                           │                      │
│         ┌─────────────────┼─────────────────┐   │
│         ▼                 ▼                 ▼   │
│   ┌──────────┐     ┌──────────────┐  ┌────────┐│
│   │ Embedding│     │  Geocoding   │  │ Entity ││
│   │Generation│     │  Enrichment  │  │Extract ││
│   └──────────┘     └──────────────┘  └────────┘│
│         │                 │                 │   │
│         └─────────────────┼─────────────────┘   │
│                           ▼                      │
│                    ┌──────────────┐             │
│                    │   Enriched   │             │
│                    │    Index     │             │
│                    └──────────────┘             │
│                                                  │
└─────────────────────────────────────────────────┘

Stage 1: Data Extraction and Migration

The first stage moves data from source systems to your search platform, handling format conversion and schema normalization.

Pagination Strategies

When extracting large datasets, pagination approach matters:

Offset-based pagination is simple but has problems: deep offsets are slow, and if data changes during extraction, you might miss or duplicate records.

Cursor-based pagination maintains consistency and performs well regardless of depth. The cursor is a stable marker that points to a specific position in the result set. Most modern systems support cursor-based pagination—use it when available.

Key principle: Always use a stable sort order (typically by ID) with cursor pagination to ensure consistent results.

Schema Normalization

Source systems often use different naming conventions than your target. Common transformations include:

• Case convention changes (camelCase to snake_case)
• Date format standardization
• Type conversions (strings to proper types)
• Null handling

Normalize at ingestion time rather than query time. Consistent schemas simplify everything downstream.

Checkpointing for Resumability

Any production pipeline processing large datasets needs checkpointing. Save progress periodically:

• Current position (cursor mark, last processed ID)
• Count of processed documents
• Timestamps
• Error summaries

When restarting after a failure, load the checkpoint and resume from where you left off. This is essential for pipelines that take hours to complete.

Stage 2: Embedding Generation

Embeddings transform text into dense vectors that enable semantic search. The embedding strategy should respect your content's natural structure.

Hierarchical Content Considerations

Documents often have natural hierarchy:

• Document level: Title + abstract/summary
• Section level: Individual sections or chapters
• Chunk level: Overlapping segments of long sections

Embedding at multiple levels enables different retrieval strategies:

• Document-level embeddings for high-level topic matching
• Section-level embeddings for targeted retrieval within documents
• Chunk-level embeddings for precise passage retrieval

The right granularity depends on your use case. For question-answering, section or chunk-level often works best. For topic exploration, document-level may suffice.

Chunking Strategies

Long content needs to be split into chunks for embedding. Common strategies:

Fixed-size chunks: Split by character or token count with overlap. Simple but may break semantic units.

Semantic chunking: Split at natural boundaries (paragraphs, sections). Preserves meaning but produces variable-length chunks.

Hierarchical chunking: Maintain parent-child relationships between document, section, and chunk embeddings.

Overlap between chunks (typically 10-20%) helps avoid losing context at boundaries.

Parallel Processing

Embedding generation is I/O-bound (API calls to embedding services), making it ideal for parallel execution:

• Use thread pools for concurrent API calls
• Implement rate limiting to respect API quotas
• Use locks for thread-safe progress tracking
• Batch documents for efficiency

The right parallelism level depends on your API's rate limits and your latency requirements. Start with 4-8 workers and adjust based on observed throughput and error rates.

Index Mapping for Vectors

Search engines need to know about vector fields at index creation time. Key considerations:

• Dimensionality: Must match your embedding model's output dimensions
• Similarity metric: Cosine similarity is most common; dot product works for normalized vectors
• Indexing strategy: HNSW is the standard for approximate nearest neighbor search
• Nested vectors: If embedding at multiple levels, use nested document structure

Elasticsearch's dense_vector type and similar features in other search engines handle these requirements.

Stage 3: Geographic Enrichment

Location-aware search requires coordinate data. If your source data has addresses but not coordinates, geocoding is necessary.

Address Quality Challenges

Real-world address data is messy:

• Inconsistent formatting
• Multiple locations per record
• Partial addresses
• Outdated or incorrect information

Building robust geocoding pipelines requires handling these realities.

Fallback Strategies

Not every address geocodes successfully on the first try. Implement fallback strategies:

1. Full address: Street, city, state, postal code
2. Partial address: Drop postal code (often wrong or outdated)
3. City-level: Fall back to city centroid when street-level fails

Track the precision level achieved (rooftop, street, city centroid) so downstream systems know how accurate the coordinates are.

Caching

Geocoding APIs are rate-limited and often billed per request. Implement caching:

• Hash addresses to cache keys
• Persist cache between runs
• Periodically save cache to avoid losing work

Re-processing a partially completed geocoding job shouldn't re-geocode already-processed addresses.

Geo Point Indexing

Store coordinates in a format your search engine understands. Elasticsearch's geo_point type, for example, enables:

• Distance-based filtering
• Distance decay scoring
• Geo aggregations

Include precision metadata alongside coordinates so queries can filter for address-level vs city-level accuracy when needed.

Operational Considerations

Command-Line Interface Design

Pipeline scripts benefit from consistent CLI patterns:

• Dry run mode: See what would happen without making changes
• Sample mode: Process a subset for testing
• Resume mode: Continue from last checkpoint
• Recreate mode: Start fresh, rebuilding indices
• Verbose mode: Detailed logging for debugging

These options make pipelines easier to test, debug, and operate.

Logging and Observability

Production pipelines need observability:

Progress logging: Periodically log progress with rate, ETA, and error count.

Structured logs: Use JSON or structured logging for easier analysis.

Error tracking: Log failed documents with enough context to diagnose issues.

Metrics: Track throughput, error rates, and latency for monitoring.

Error Handling Patterns

Pipelines should continue despite individual document failures:

Log and continue: For non-critical failures, log the error and move on. Don't let one bad document stop the entire pipeline.

Retry with backoff: For transient failures (rate limits, network issues), implement exponential backoff.

Batch fallback: If a batch fails, retry documents individually to identify which specific document caused the failure.

Critical error handling: For infrastructure failures (lost database connection), save checkpoint and exit gracefully.

The goal is maximum progress while preserving the ability to diagnose and fix issues.

Data Lineage

Maintain clear lineage for debugging and auditing:

• Store raw source data before transformation
• Store intermediate results at each stage
• Use consistent naming conventions for artifacts
• Track timestamps and version information

When search results seem wrong, you need to trace back to the exact input data that produced them.

Index Design Principles

The target index schema significantly impacts search quality and performance.

Field Configuration

Different fields need different treatment:

• Text fields: Choose appropriate analyzers (language-specific, custom)
• Keyword fields: For exact match filtering and aggregations
• Nested objects: For arrays of complex objects (like sections with embeddings)
• Dense vectors: For semantic search with appropriate dimensions and similarity

Nested vs Flattened

For hierarchical content, decide between:

Nested documents: Preserve relationships (section-level matches return parent document), but more complex queries.

Flattened structure: Simpler queries, but may lose important structure.

For search applications, nested usually wins—you want to match specific sections while returning full documents.

Key Takeaways

Building production data pipelines for search requires attention to reliability, efficiency, and data quality:

1. Cursor-based pagination for reliable large-scale data extraction

2. Hierarchical embedding generation respecting natural content structure

3. Parallel processing with rate limiting for API-bound operations

4. Checkpointing at every stage enabling resumable operations

5. Fallback strategies for handling imperfect source data

6. Strong error handling that maximizes progress while enabling debugging

The result is a search index that combines the best of keyword-based retrieval with semantic vector search—the foundation for hybrid retrieval systems.

---

For implementation details, consult your search engine's documentation. The Elasticsearch documentation covers index design and the Python client for pipeline implementation. For embeddings, see your provider's API documentation for best practices on batching and rate limiting.

Note: The patterns discussed here are intentionally generalized, drawn from industry experience but presented as transferable concepts rather than specific proprietary implementations.