What is a Data Pipeline?
A Data Pipeline is a structured flow of data from source to AI execution. It ensures that:- Raw data is properly ingested
- Data is cleaned and structured
- Information is stored and indexed
- AI agents receive consistent, high-quality inputs
Data Sources
Pipelines can ingest data from multiple sources depending on your use case.API (Batch Processing)
APIs are used for scheduled or batch data ingestion.- Periodic data sync (e.g., every hour, daily)
- Large dataset ingestion
- Historical data processing
Webhooks (Real-Time Processing)
Webhooks enable real-time data ingestion.- Triggered instantly on events
- Suitable for live systems (user activity, transactions, updates)
- Enables real-time AI responses
Integrations
Pipelines can also connect to external systems via integrations such as:- CRMs (e.g., Salesforce)
- E-commerce platforms (e.g., Shopify)
- Internal tools and databases
Custom Sources
You can also define custom data inputs depending on your system requirements.Pipeline Creation Modes
Avigrah supports multiple pipeline structures depending on how data needs to be processed and used.1. Full Pipeline (Recommended)
Data Source → Lakehouse → Vector Database → Ready for AI This is the most complete and structured pipeline.- Data is ingested from source
- Stored in the lakehouse (structured storage)
- Vectorized and indexed in the vector database
- Made available for AI agents
- Large-scale systems
- Historical + real-time analysis
- High reliability and traceability
2. Direct Vector Pipeline
Data Source → Vector Database → Ready for AI- Data is directly vectorized and stored
- Skips structured storage layer
- Fast setup
- Lightweight use cases
- Retrieval-focused AI systems
3. Hybrid Pipeline
Data Source → Lakehouse + Vector Database- Data is stored in both structured and vector formats
- Supports both analytical queries and AI retrieval
- Systems requiring both reporting and AI reasoning
Pipeline Schema (Critical Component)
Every pipeline requires a Pipeline Schema. This is a structured definition of how data is organized and interpreted.What is a Schema?
A schema defines:- Field structure
- Data types
- Relationships between data points
Why Schema is Mandatory
AI agents rely on schema to understand data correctly. Without schema:- Data interpretation becomes inconsistent
- AI outputs become unreliable
- Agents receive structured context
- Outputs are more accurate and aligned
- Reasoning becomes consistent across executions
Schema as AI Context
The pipeline schema is directly used by AI agents as context. This means:- Agents understand what each field represents
- Data relationships are preserved
- Outputs are aligned with business meaning
Unique Identifier & Upsert Behavior
Each pipeline must define a unique identifier. This identifier is used to manage data updates.Upsert Model
Avigrah pipelines use upsert by default. This means:- Existing records are updated if they already exist
- New records are inserted if they do not exist
- No duplicate data
- Always up-to-date datasets
- Consistent state for AI processing
How Pipelines Connect to AI
Once a pipeline is ready:- Data is structured and stored
- Vector representations are created
- AI agents can query and reason over this data