README

Stop writing spaghetti code for imports.

Laravel Ingest is a robust, configuration-driven ETL (Extract, Transform, Load) framework for Laravel. It replaces fragile, procedural import scripts with elegant, declarative configuration classes.

Whether you are importing 100 rows or 10 million, Laravel Ingest handles the heavy lifting: streaming, chunking, queueing, validation, relationships, and error reporting.

⚡ Why use this?

Most import implementations suffer from the same issues: memory leaks, timeouts, lack of validation, and messy controllers.

Laravel Ingest solves this by treating imports as a first-class citizen:

• ♾️ Infinite Scalability: Uses Generators and Queues to process files of any size with flat memory usage.
• 📝 Declarative Syntax: Define what to import, not how to loop over it.
• 🧪 Dry Runs: Simulate imports to find validation errors without touching the database.
• 🔗 Auto-Relations: Automatically resolves BelongsTo and BelongsToMany relationships (e.g., finding IDs by names).
• 🛡️ Robust Error Handling: Tracks every failed row and allows you to download a CSV of only the failures to fix and retry.
• 🔌 API & CLI Ready: Comes with auto-generated API endpoints and Artisan commands.

📚 Documentation

Full documentation is available at zappzerapp.github.io/laravel-ingest.

🚀 Quick Start

1. Installation

composer require zappzerapp/laravel-ingest

# Publish config & migrations
php artisan vendor:publish --provider="LaravelIngest\IngestServiceProvider"

# Create tables
php artisan migrate

2. Define an Importer

Use the Artisan generator to scaffold a new importer:

php artisan make:importer UserImporter --model=User

Alternatively, create a class implementing IngestDefinition manually:

namespace App\Ingest;

use App\Models\User;
use LaravelIngest\Contracts\IngestDefinition;
use LaravelIngest\IngestConfig;
use LaravelIngest\Enums\SourceType;
use LaravelIngest\Enums\DuplicateStrategy;

class UserImporter implements IngestDefinition
{
    public function getConfig(): IngestConfig
    {
        return IngestConfig::for(User::class)
            ->fromSource(SourceType::UPLOAD)
            ->keyedBy('email') // Identify records by email
            ->onDuplicate(DuplicateStrategy::UPDATE) // Update if exists
            
            // Map CSV columns to DB attributes
            ->map('Full Name', 'name')
            ->map(['E-Mail', 'Email Address'], 'email') // Supports aliases
            
            // Handle Relationships automatically
            ->relate('Role', 'role', Role::class, 'slug', createIfMissing: true)
            
            // Validate rows before processing
            ->validate([
                'email' => 'required|email',
                'Full Name' => 'required|string|min:3'
            ]);
    }
}

3. Register it

In App\Providers\AppServiceProvider:

use LaravelIngest\IngestServiceProvider;

public function register(): void
{
    $this->app->tag([UserImporter::class], IngestServiceProvider::INGEST_DEFINITION_TAG);
}

4. Run it!

You can now trigger the import via CLI or API.

Via Artisan (Backend / Cron):

php artisan ingest:run user-importer --file=users.csv

Via API (Frontend / Upload):

curl -X POST \
  -H "Authorization: Bearer <token>" \
  -F "file=@users.csv" \
  https://your-app.com/api/v1/ingest/upload/user-importer

💡 Demo Project

Want to see Laravel Ingest in action? Check out our Laravel Ingest Demo repository for a complete working example.

# Clone the demo
git clone https://github.com/zappzerapp/Laravel-Ingest-Demo.git
cd Laravel-Ingest-Demo

# Start and benchmark
docker compose up -d
docker compose exec app php artisan benchmark:ingest

🛠 Features in Depth

Monitoring & Management

Ingest runs happen in the background. You can monitor and manage them easily:

API Endpoints

The package automatically exposes endpoints for building UI integrations (e.g., React/Vue progress bars).

• GET /api/v1/ingest - List recent runs.
• GET /api/v1/ingest/{id} - Get status and progress.
• GET /api/v1/ingest/{id}/errors/summary - Get aggregated error stats (e.g., "50x Email invalid").
• GET /api/v1/ingest/{id}/failed-rows/download - Download a CSV of failed rows to fix & re-upload.

Events

Hook into the lifecycle to send notifications (e.g., Slack) or trigger downstream logic.

• LaravelIngest\Events\IngestRunStarted
• LaravelIngest\Events\ChunkProcessed
• LaravelIngest\Events\RowProcessed
• LaravelIngest\Events\IngestRunCompleted
• LaravelIngest\Events\IngestRunFailed

Pruning

To keep your database clean, logs are prunable. Add this to your scheduler:

$schedule->command('model:prune', [
    '--model' => [LaravelIngest\Models\IngestRow::class],
])->daily();

🧩 Configuration Reference

The IngestConfig fluent API handles complex scenarios with ease.

IngestConfig::for(Product::class)
    // Sources: UPLOAD, FILESYSTEM, URL, FTP, SFTP
    ->fromSource(SourceType::FTP, ['disk' => 'erp', 'path' => 'daily.csv'])
    
    // Performance
    ->setChunkSize(1000)
    ->atomic() // Wrap chunks in transactions
    
    // Logic
    ->keyedBy('sku')
    ->onDuplicate(DuplicateStrategy::UPDATE_IF_NEWER)
    ->compareTimestamp('last_modified_at', 'updated_at')
    
    // Transformation
    ->mapAndTransform('price_cents', 'price', fn($val) => $val / 100)
    ->resolveModelUsing(fn($row) => $row['type'] === 'digital' ? DigitalProduct::class : Product::class);

See the Documentation for all available methods.

✨ Advanced Features

Beyond the basics, Laravel Ingest supports advanced capabilities for complex import scenarios:

• Validators — Rule-based and custom validators to reject rows before processing.
• Transformers & Pipelines — Chain multiple transformations on field values.
• Conditional Mappings — Apply mappings only when row conditions are met.
• Custom Sources — Define your own source types (e.g., database, API).
• Import Event Handlers — React to lifecycle events (start, progress, complete, failure).
• Nested Mappings — Handle nested data structures and one-to-many relationships.
• Schema Validation — Validate the structure of your file before processing.
• Tracing & Debugging — Inspect row-level decision trees and resolve issues fast.

🧪 Testing

We provide a Docker-based test environment to ensure consistency.

# Start Docker
composer docker:up

# Run Tests
composer docker:test

# Check Coverage
composer docker:coverage

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

📄 License

The MIT License (MIT). Please see License File for more information.