README

Manticore Engine for Laravel Scout

Installation

Via Composer

$ composer require romanstruk/manticore-scout-engine

Configuration

After installing Manticore Scout Engine, you should publish the Manticore configuration file using the vendor:publish Artisan command. This command will publish the manticore.php configuration file to your application's config directory:

php artisan vendor:publish --provider="RomanStruk\ManticoreScoutEngine\ManticoreServiceProvider"

php artisan vendor:publish --provider="Laravel\Scout\ScoutServiceProvider"

Configuring Search Driver

Set up your search driver manticore in .env file

SCOUT_DRIVER=manticore

There is a choice between two ways to connect to the manticore

• http-client - \Manticoresearch\Client github
• mysql-builder - \RomanStruk\ManticoreScoutEngine\Builder use mysql connection

Set up your engine in .env file

MANTICORE_ENGINE=http-client

Configuring Driver Connection

For http-client in .env file

MANTICORE_HOST=127.0.0.1
MANTICORE_PORT=9308

For mysql-builder in .env file

MANTICORE_MYSQL_HOST=127.0.0.1
MANTICORE_MYSQL_PORT=9306

Configuring Model Migration

To create a migration, specify the required fields in the searchable model

public function scoutIndexMigration(): array
{
    return [
        'fields' => [
            'id' => ['type' => 'bigint'],
            'name' => ['type' => 'text'],
            'category' => ['type' => 'string stored indexed'],// string|text [stored|attribute] [indexed]
        ],
        'settings' => [
            'min_prefix_len' => '3',
            'min_infix_len' => '3',
            'prefix_fields' => 'name',
            'expand_keywords' => '1',
            //'engine' => 'columnar', // [default] row-wise - traditional storage available in Manticore Search out of the box; columnar - provided by Manticore Columnar Library
        ],
    ];
}

Configuring query options

max_matches - Maximum amount of matches that the server keeps in RAM for each index and can return to the client. Default is 1000.

For queries with pagination, you can specify automatic parameter calculation max_matches Set up your paginate_max_matches in manticore.php config file

'paginate_max_matches' => 1000,

Set null for calculate offset + limit

As some characters are used as operators in the query string, they should be escaped to avoid query errors or unwanted matching conditions. Set up your auto_escape_search_phrase in manticore.php config file

'auto_escape_search_phrase' => true,

Set false for disable auto escape special symbols ! " $ ' ( ) - / < @ \ ^ | ~

Other parameters for queries can be specified in the model

public function scoutMetadata(): array
{
    return [
        'cutoff' => 0,
        'max_matches' => 1000,
    ];
}

Config paginate_max_matches has higher priority than scoutMetadata max_matches option

Usage

Documentation for Scout can be found on the Laravel website.

Run artisan command for create Manticore index

php artisan manticore:index "App\Models\Product"

Manticore allows you to add "whereRaw" methods to your search queries.

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

$products = Product::search('Brand Name', function (Builder $builder) {
    return $builder
        ->whereAny('category_id', ['1', '2', '3'])
        ->where('column', '=', 'value')
//        ->whereIn('column', ['1', '2'])
//        ->whereNotIn('column', ['3', '4'])
//        ->whereAll('column', ['3', '4'])
//        ->whereNotAll('column', ['5', '6'])
//        ->whereAllMva('column', 'in', ['1', '2'])
//        ->whereAnyMva('column', 'not in', ['1', '2'])
        ->facet('category_id')
        ->inRandomOrder();
})->get();

Quorum matching operator

Quorum matching operator introduces a kind of fuzzy matching. It will only match those documents that pass a given threshold of given words. The example above ("the world is a wonderful place"/3) will match all documents that have at least 3 of the 6 specified words.

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

$products = Product::search('the world is a wonderful place', function (Builder $builder) {
    return $builder->setQuorumMatchingOperator(3);
})->get();

Proximity distance is specified in words, adjusted for word count, and applies to all words within quotes. For instance, "cat dog mouse"~5 query means that there must be less than 8-word span which contains all 3 words, ie.

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

$products = Product::search('cat dog mouse', function (Builder $builder) {
    return $builder->setProximitySearchOperator(5);
})->get();

Autocomplete

Autocomplete (or word completion) is a feature in which an application predicts the rest of a word a user is typing. On websites, it's used in search boxes, where a user starts to type a word, and a dropdown with suggestions pops up so the user can select the ending from the list.

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

//[doc] My cat loves my dog. The cat (Felis catus) is a domestic species of small carnivorous mammal.

$autocomplete = Product::search('my*',function (Builder $builder) {
    return $builder->autocomplete(['"','^'], true); // "" ^ * allow full-text operators; stats - Show statistics of keywords, default is 0
})->raw();
// $autocomplete<array> "my", "my cat", "my dog"

Spell correction

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

//[doc] Crossbody Bag with Tassel
//[doc] microfiber sheet set
//[doc] Pet Hair Remover Glove

$result = Product::search('bagg with tasel',function (Builder $builder) {
    return $builder->spellCorrection(true) // correct first word
})->raw();
// $result<array> 0 => ['suggest' => "bag"]

$result = Product::search('bagg with tasel',function (Builder $builder) {
    return $builder->spellCorrection() // correct last word
})->raw();
// $result<array> 0 => ['suggest' => "tassel"]

$result = Product::search('bagg with tasel',function (Builder $builder) {
    return $builder->spellCorrection(false, true) // correct last word and return sentence
})->raw();
// $result<array> 0 => ['suggest' => "bagg with tassel"]

Highlighting

Highlighting enables you to obtain highlighted text fragments (referred to as snippets) from documents containing matching keywords.

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

//[doc] My cat loves my dogs.

$highlight = Product::search('dogs',
    fn(Builder $builder) => $builder->highlight()->select(['id', 'name'])
)->raw();
// $highlight['hits']<array> [id => 1, name => 'My cat loves my dogs.', 'highlight' => 'My cat loves my <b>dogs</b>.']

or

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

//[doc] title => My cat loves my dogs. id => 1000

$highlight = Product::search('dogs',
    fn(Builder $builder) => $builder->highlight()->select(['id', 'title'])
)->get();
// $highlight->getHighlight()[1000] => 'My cat loves my <b>dogs</b>.'

Percolate Query

To create a migration, specify the required fields in the searchable model

public function scoutIndexMigration(): array
{
    return [
            'fields' => [
                'title' => ['type' => 'text'],
                'color' => ['type' => 'string'],
            ],
            'settings' => [
                'type' => 'pq'
            ],
    ];
}

public function toSearchableArray(): array
{
    return array_filter([
        'id' => $this->name,
        'query' => "@title {$this->title}",
        'filters' => $this->color ? "color='{$this->color}'" : null,
    ]);
}

Percolate queries are also known as Persistent queries, Prospective search, document routing, search in reverse, and inverse search. https://manual.manticoresearch.com/Searching/Percolate_query#Percolate-Query

use RomanStruk\ManticoreScoutEngine\Mysql\Builder;

$products = PercolateProduct::search(json_encode(['title' =>'Beautiful shoes']),
    fn(Builder $builder) => $builder->percolateQuery(docs: true, docsJson: true)
)->get();

KNN

K-nearest neighbor vector search https://manual.manticoresearch.com/Searching/KNN#K-nearest-neighbor-vector-search

Added the ability to create records with the float_vector field type

use RomanStruk\ManticoreScoutEngine\Mysql\ManticoreVector;

public function scoutIndexMigration(): array
{
    return [
        'fields' => [
            'id' => ['type' => 'bigint'],
            'name' => ['type' => 'text'],
            'vector' => ['type' => "float_vector knn_type='hnsw' knn_dims='4' hnsw_similarity='l2'"],
        ],
        'settings' => [],
    ];
}


public function toSearchableArray(): array
{
    return [
        'id' => $this->id,
        'name' => $this->name,
        'vector' => new ManticoreVector(...$this->vector), // $this->vector is array
    ];
}

Currently, the implementation is only available using whereRaw.

$results = SimilarProduct::search('bar', function (Builder $query) {
    return $query->whereRaw("knn ( vector, 5, (0.286569,-0.031816,0.066684,0.032926), 2000 )");
})->get();

Please note that when using the "Find similar docs by id" syntax, you need to discard meta discardMeta(). Exact information about the number of results is not available

$results = SimilarProduct::search('foo', function (Builder $query) {
    return $query->whereRaw("knn ( vector, 5, 1 )")->discardMeta();
})->get();

Join Table

Table joins in Manticore Search enable you to combine documents from two tables by matching related columns. This functionality allows for more complex queries and enhanced data retrieval across multiple tables.

https://manual.manticoresearch.com/Searching/KNN#K-nearest-neighbor-vector-search

use App\Models\Category;
use App\Models\Product;

$categoryTable = (new Category)->searchableAs();

$searchable = Product::search('some search query', static fn(Builder $builder)
    => $builder
    ->select(['id', DB::raw($categoryTable.'.name as name')])
    ->join($categoryTable, $categoryTable.'.id', '=', $builder->index.'.category_id') // inner join table categories
    ->leftJoin($categoryTable, $categoryTable.'.id', '=', $builder->index.'.category_id') // left join table categories
    ->orderBy('id'),
)->raw();

Manticore Docs LEFT JOIN: Returns all rows from the left table and the matched rows from the right table. If there is no match, NULL values are returned for the right table's columns.

WARNING there is a problem that "LEFT JOIN" does not return "NULL" in the classic php sense but returns a string with the text "NULL"

Change log

Please see the changelog for more information on what has changed recently.

Testing

$ composer test

Contributing

Please see contributing.md for details and a todolist.

Security

If you discover any security related issues, please email romanuch4@gmail.com instead of using the issue tracker.

License

MIT. Please see the license file for more information.