Custom Ingestion Providers¶

The bundle ships with providers for Sulu pages and articles, but you can implement your own to feed content from any source — external APIs, databases, file systems, or non-Sulu CMS content types — into the same intelligent-search index.

Implementing a Provider¶

Create a class implementing IngestProviderInterface:

<?php

namespace App\Ingestion;

use Sulu\Bundle\AiPlatformBundle\Ingestion\IngestProviderInterface;
use Sulu\Bundle\AiPlatformBundle\Store\Api\IngestChunk;
use Sulu\Bundle\AiPlatformBundle\Store\Api\IngestDocument;

final readonly class ProductIngestProvider implements IngestProviderInterface
{
    public function __construct(
        private ProductRepository $productRepository,
    ) {
    }

    public static function getName(): string
    {
        return 'products';
    }

    public function count(array $options): int
    {
        return $this->productRepository->countPublished($options['locale'] ?? 'en');
    }

    public function provide(array $options): iterable
    {
        $locale = $options['locale'] ?? 'en';

        foreach ($this->productRepository->findPublished($locale) as $product) {
            yield new IngestDocument(
                id: 'product-' . $product->getId(),
                title: $product->getTitle(),
                url: '/products/' . $product->getSlug(),
                bucket: 'products',
                locale: $locale,
                chunks: [
                    new IngestChunk(content: $product->getDescription()),
                    new IngestChunk(
                        content: $product->getSpecifications(),
                        metadata: ['section' => 'specifications'],
                    ),
                ],
                metadata: [
                    'category' => $product->getCategory(),
                    'sku' => $product->getSku(),
                ],
                updatedAt: $product->getUpdatedAt()->format(\DateTimeInterface::ATOM),
            );
        }
    }
}

Interface Contract¶

interface IngestProviderInterface
{
    public static function getName(): string;

    public function count(array $options): int;

    public function provide(array $options): iterable;
}

`getName(): string`¶

A unique name for the provider, used as the bucket-style filter on the CLI:

php bin/console sulu:ai:intelligent-search:ingest --provider=products

It is also what the example UI lists as an available bucket filter.

`count(array $options): int`¶

Total number of documents this provider will yield. The IngestionRunner uses this to render a progress bar — returning a rough estimate is fine; it does not have to match exactly.

`provide(array $options): iterable`¶

Yields IngestDocument instances. Use yield to keep memory usage low for large datasets.

Options¶

Both count() and provide() receive the same options array, populated from the CLI:

Key	Type	Description
`locale`	`?string`	Content locale, or `null` when `--locale` is omitted
`webspace`	`?string`	Optional Sulu webspace filter
`host`	`?string`	Optional host override for URLs
`scheme`	`?string`	URL scheme (`https` by default)

Use or ignore any of them depending on what your data source needs.

Registration¶

Tag your provider with sulu_ai_platform.ingest_provider. The ingest command derives the iterator key from your static getName() method, so no extra tag attribute is needed:

# config/services.yaml
services:
    App\Ingestion\ProductIngestProvider:
        autowire: true
        tags: ['sulu_ai_platform.ingest_provider']

The value returned by getName() (e.g. 'products') is what --provider <name> matches on.

IngestDocument Structure¶

Each IngestDocument represents a single piece of content to index:

Property	Type	Description
`id`	`string`	Unique document identifier (must be stable across re-ingestion)
`title`	`string`	Document title (shown in search results)
`url`	`string`	URL to the source content (used for citations)
`bucket`	`string`	Category/group name for filtering search results
`locale`	`string`	Content locale
`chunks`	`list<IngestChunk>`	Text content split into chunks
`metadata`	`array<string, mixed>`	Optional metadata (default: `[]`)
`updatedAt`	`?string`	ISO-8601 update timestamp (default: `null`)

Each IngestChunk has string $content and an optional array<string, mixed> $metadata.

Chunking¶

Split long content into focused chunks — each chunk is indexed separately, so the platform can retrieve the most relevant section instead of returning entire documents. A practical chunk size sits around 500–1000 characters: enough context for the LLM, small enough for precise retrieval.

The built-in Sulu providers delegate to the configured ContentChunkerInterface (markdown by default). Custom providers can chunk however makes sense for their source. If you want to reuse the same chunker as the Sulu providers, inject the sulu_ai_platform.ingestion.content_chunker service.

// Single chunk for short content
$chunks = [new IngestChunk($product->getDescription())];

// Multiple chunks with per-chunk metadata
$chunks = [
    new IngestChunk($product->getDescription(), ['section' => 'overview']),
    new IngestChunk($product->getSpecifications(), ['section' => 'specs']),
    new IngestChunk($product->getFaq(), ['section' => 'faq']),
];

See Ingestion → Chunking for chunker configuration.

Buckets¶

The bucket field groups documents for filtered search. The CLI --bucket option and the public API buckets query parameter restrict results to a subset.

Pick bucket names that match your content domains: pages, articles, products, faq, docs, etc.

Testing¶

Use --dry-run to verify your provider produces correct documents without sending them to the platform:

php bin/console sulu:ai:intelligent-search:ingest --provider=products --dry-run

count() and provide() still execute, so this is also a quick way to catch loader errors before a real ingest.