> ## Documentation Index
> Fetch the complete documentation index at: https://docs.avigrah.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Pipelines

Data Pipelines are the foundation of how AI systems operate in Avigrah.

They are responsible for **collecting, transforming, structuring, and delivering data** in a format that AI agents can reliably understand and act upon.

***

## What is a Data Pipeline?

A Data Pipeline is a **structured flow of data** from source to AI execution.

It ensures that:

* Raw data is properly ingested
* Data is cleaned and structured
* Information is stored and indexed
* AI agents receive consistent, high-quality inputs

AI systems in Avigrah do not operate on raw data — they operate on **pipeline-processed data**.

***

## Data Sources

Pipelines can ingest data from multiple sources depending on your use case.

### API (Batch Processing)

APIs are used for **scheduled or batch data ingestion**.

* Periodic data sync (e.g., every hour, daily)
* Large dataset ingestion
* Historical data processing

***

### Webhooks (Real-Time Processing)

Webhooks enable **real-time data ingestion**.

* Triggered instantly on events
* Suitable for live systems (user activity, transactions, updates)
* Enables real-time AI responses

***

### Integrations

Pipelines can also connect to external systems via integrations such as:

* CRMs (e.g., Salesforce)
* E-commerce platforms (e.g., Shopify)
* Internal tools and databases

These integrations allow AI systems to operate on **live operational data**.

***

### Custom Sources

You can also define custom data inputs depending on your system requirements.

***

## Pipeline Creation Modes

Avigrah supports multiple pipeline structures depending on how data needs to be processed and used.

***

### 1. Full Pipeline (Recommended)

**Data Source → Lakehouse → Vector Database → Ready for AI**

This is the most complete and structured pipeline.

* Data is ingested from source
* Stored in the lakehouse (structured storage)
* Vectorized and indexed in the vector database
* Made available for AI agents

Best for:

* Large-scale systems
* Historical + real-time analysis
* High reliability and traceability

***

### 2. Direct Vector Pipeline

**Data Source → Vector Database → Ready for AI**

* Data is directly vectorized and stored
* Skips structured storage layer

Best for:

* Fast setup
* Lightweight use cases
* Retrieval-focused AI systems

***

### 3. Hybrid Pipeline

**Data Source → Lakehouse + Vector Database**

* Data is stored in both structured and vector formats
* Supports both analytical queries and AI retrieval

Best for:

* Systems requiring both reporting and AI reasoning

***

## Pipeline Schema (Critical Component)

Every pipeline requires a **Pipeline Schema**.

This is a structured definition of how data is organized and interpreted.

***

### What is a Schema?

A schema defines:

* Field structure
* Data types
* Relationships between data points

It ensures that data is **consistent, predictable, and usable by AI agents**.

***

### Why Schema is Mandatory

AI agents rely on schema to understand data correctly.

Without schema:

* Data interpretation becomes inconsistent
* AI outputs become unreliable

With schema:

* Agents receive structured context
* Outputs are more accurate and aligned
* Reasoning becomes consistent across executions

***

### Schema as AI Context

The pipeline schema is directly used by AI agents as **context**.

This means:

* Agents understand what each field represents
* Data relationships are preserved
* Outputs are aligned with business meaning

***

## Unique Identifier & Upsert Behavior

Each pipeline must define a **unique identifier**.

This identifier is used to manage data updates.

***

### Upsert Model

Avigrah pipelines use **upsert by default**.

This means:

* Existing records are updated if they already exist
* New records are inserted if they do not exist

This ensures:

* No duplicate data
* Always up-to-date datasets
* Consistent state for AI processing

***

## How Pipelines Connect to AI

Once a pipeline is ready:

* Data is structured and stored
* Vector representations are created
* AI agents can query and reason over this data

Pipelines act as the **data backbone** for all AI systems.

***

## Best Practices

### Always Define a Clear Schema

This directly impacts AI output quality.

***

### Choose Pipeline Mode Based on Use Case

Do not over-engineer — use full pipelines only when needed.

***

### Use Webhooks for Real-Time Systems

Critical for responsiveness and live insights.

***

### Ensure Unique Identifiers are Stable

This is essential for correct upsert behavior.

***

### Keep Pipelines Focused

Each pipeline should serve a clear purpose within the project.

***