Skip to main content
Executions represent how your data pipelines and AI systems run in real time. Observability ensures you can track, understand, and trust every step of that execution. Together, they provide full visibility into how your AI systems operate.

​
What is an Execution?

An Execution is a single run of a data pipeline. Each time a pipeline is triggered:
  • Data is processed through defined stages
  • Transformations and vectorization occur
  • Outputs are generated and stored
  • AI systems receive updated data
Every run is tracked as a separate execution.

​
Execution Lifecycle

Each execution follows a structured flow: Pending → Running → Completed / Failed / Stopped During execution:
  • Data moves through pipeline stages
  • Metrics are recorded in real time
  • Logs are generated for traceability

​
Pipeline Stages (Execution Flow)

Executions are broken down into clear stages. Typical stages include:
  • Extract → Data ingestion from sources
  • Transform → Cleaning and structuring
  • Vectorize → Converting data into embeddings
  • Load → Storing in databases and making it ready for AI
Each stage is:
  • Independently tracked
  • Measured for duration and status
  • Logged for debugging and auditing

​
Real-Time Observability

Avigrah provides full visibility into execution through multiple layers.

​
Execution Status

At any time, you can see:
  • Current status (Running, Completed, Failed, Paused)
  • Start time and duration
  • Progress across stages

​
Stage-Level Tracking

Each pipeline stage shows:
  • Current state (Pending, Running, Completed, Failed)
  • Execution duration
  • Stage-specific metrics
This allows precise identification of where issues occur.

​
Logs (System-Level Visibility)

Every execution generates structured logs. Logs provide:
  • Step-by-step system activity
  • Errors and warnings
  • Execution messages and events
This ensures full traceability of what happened during execution.

​
Metrics (Performance Insights)

Executions track key system metrics such as:
  • API calls made
  • Data processed
  • Vectors generated
  • Retry attempts
These metrics help evaluate:
  • System performance
  • Data volume handled
  • Operational efficiency

​
Progress Tracking

Execution progress is calculated based on:
  • Number of completed stages
  • Total pipeline stages
  • Weighted execution flow
This provides a clear view of how far the pipeline has progressed.

​
Failure Handling & Retries

If an execution fails:
  • The system captures the failure stage
  • Logs highlight the root cause
  • Retry attempts are tracked
Pipelines can be:
  • Re-run
  • Restarted from execution controls
This ensures reliability in production environments.

​
Execution History

Each pipeline maintains a history of executions. This allows you to:
  • Review past runs
  • Compare performance
  • Identify patterns or failures
Execution history is critical for:
  • Debugging
  • Auditing
  • System optimization

​
Observability as a System Layer

Observability in Avigrah is not an add-on — it is built into the system. Every execution is:
  • Tracked
  • Logged
  • Measured
  • Auditable
This ensures:
  • Transparency in AI operations
  • Confidence in outputs
  • Control over system behavior

​
How This Connects to AI

Executions directly impact AI performance.
  • Pipelines prepare structured data
  • Executions ensure data is up-to-date
  • AI agents consume the latest processed data
Reliable execution = reliable AI outputs.

​
Best Practices

​
Monitor Active Executions

Track running pipelines to ensure smooth operation.

​
Use Logs for Debugging

Always refer to logs for understanding failures.

​
Watch Metrics for Scaling

High usage or retries may indicate scaling needs.

​
Maintain Healthy Pipelines

Consistent execution ensures consistent AI performance.

​
Review Execution History

Use past runs to improve pipeline efficiency.