Symfony meets Elasticsearch — Implement a search as you type feature

Stefan Pöltl

Sep 22, 2020·10 min read

What do we build?

In this post we’re gonna implement a search as you type feature with Symfony. We will feed product information into Elasticsearch, build a user interface to search for a product by it’s title and manage the Symfony integration to query Elasticsearch.

What is Elasticsearch?

Elasticsearch is an open source search and analysis engine based on Apache Lucene developed in Java. Feed it with your data using a REST interface and explore it with powerful queries and aggregation possibilities. Use it as a base for analytical dashboards or realtime systems.

Elasticsearch is designed as a distributed system from the ground up to build a resilient system with replicas spread over several nodes.

You can feed data to Elasticsearch with parallel requests, because it has multiple write threads with a queue to buffer not processed requests. This allows to accelerate the data import dramatically.

Development setup

As you might know from my other posts, I like Docker pretty much. So here is a simple docker-compose based setup to run Elasticsearch and Symfony in a beautiful development environment. First install Symfony in a directory like this:

docker run --rm -it -v $PWD:/app composer create-project symfony/website-skeleton symfony_elasticsearch

Now add a docker-compose.yml with the following content:

version: '3'
services:
  php:
    image: webdevops/php-nginx-dev:7.4
    working_dir: /app
    environment:
      - WEB_DOCUMENT_ROOT=/app/public
      - PHP_DISPLAY_ERRORS=1
      - PHP_MEMORY_LIMIT=2048M
      - PHP_MAX_EXECUTION_TIME=-1
      - XDEBUG_REMOTE_AUTOSTART=1
      - XDEBUG_REMOTE_PORT=9000
      - XDEBUG_PROFILER_ENABLE=0
      - XDEBUG_REMOTE_CONNECT_BACK=0
      - XDEBUG_REMOTE_HOST=docker.for.mac.localhost
      - php.xdebug.idekey=PHPSTORM
      - php.xdebug.remote_enable=1
      - php.xdebug.max_nesting_level=1000
    ports:
      - "8080:80"
    volumes:
      - ./:/app:rw,cached
  es:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.9.0
    environment:
      - "discovery.type=single-node"
      - "bootstrap.memory_lock=true"
      - "ES_JAVA_OPTS=-Xms1G -Xmx1G"
      - "xpack.security.enabled=false"
      - "http.cors.enabled=true"
      - "http.cors.allow-origin=*"
    ports:
      - 9201:9200

This docker-compose setup boots a NGINX webserver, a PHP-FPM setup and one Elasaticsearch node that runs in single mode.

Discover Elasticsearch via the Head plugin

If you use Google Chrome it’s really easy to get an UI to play around with Elasticsearch. There’s an extension called Elasticsearch Head. You can get it here:

Now you are able to click the lens icon in your browser and set http://localhost:9201 as the Elasticsearch url.

It should look like this:

The plugin shows the single node Elasticsearch instance and a green health state.

Symfony integration

To handle Elasticsearch within Symfony, we use the official PHP library to interact with it. We do this because there’s no real bundle out there that brings us any benefit. Most of the bundles are not up to date and miss a lot of maintenance. With Symfony we can manage the integration part really easy, thanks to the amazing DI-Container and it’s autowiring feature. To install the PHP Elasticsearch client we run the following:

docker run --rm -it -v $PWD:/app composer require elasticsearch/elasticsearch

Now we need to configure an Elasticsearch client as a service in the services.yaml file like this:

parameters:
    es_config: {'hosts': ['http://es:9200']}

services:
    # lots of service definitions in between    Elasticsearch\ClientBuilder: ~

    Elasticsearch\Client:
        factory:   ['@Elasticsearch\ClientBuilder', fromConfig]
        arguments: ['%es_config%']

This configuration enables the possibility to get a preconfigured Elasticserach Client injected into our code. We will see how this works in the Feeder Command in the next step.

Feed Elasticsearch

Let’s create a command to import product data to Elasticsearch. We can generate a command template file with the make command:

Let’s start our coding. Checkout the file below, it contains everything we need to feed the data to elasticsearch:

As you can see in the code, we get the Elasticsearch client injected in the constructor and setup a schema/mapping when we create the index for the product documents. In the mapping part we define a title property as a analyzed text field. The title property of the uploaded documents will be analyzed by our defined autocomplete analyzer. Our analyzer transforms the incoming title strings to lower case and builds tokens with an edge_ngram tokenizer based on letters and digits of the incoming strings. The next step is to run this command.

Execute the feeder command

We can run this command vai Docker like this:

docker-compose exec php bin/console app:feed_products

When the command finished successfully, we can see the created index in the Head plugin:

Check the analyzer

We are able to check examples of how our analyzer performs now:

As you can see the analyzer starts with tokens from left to right. As defined above in the code with a minimum of 2 chars per token. We also defined a search analyzer that uses the standard setting which means for the search input string:

As you can see the standard analyzer tokenizes the string on whitespace and applies a lowercase filter. From the Elasticsearch documentation you get the information that the tokenizer works grammar based for a lot of languages. Thats absolutely fine to map the tokens above to get a good match.

You can use the Head plugin to debug the Elasticsearch analyzers like this:

You can also test filters that you can add to an analyzer:

Analyze the index feeding code

As you can see in the github project, we parse a CSV file with data I crawled from Google Shopping:

Whenever you can, use the OOP approach instead of fopen! Go for SplFileObject to handle files, because it’s much more readable and easier to write testable code. It fits much better in your already OOP codebase. So first of all we’re gonna ignore the headline of the CSV file and iterate over the rows of the file. For each row we build a document as an array structure and send it with the client to Elasticsearch. That’s it, pretty easy!

Autocomplete UI with Webpack Encore

To get a feeling for the search results and the matching, it’s much easier to test it in a real user interface(UI). As we are used to from Symfony, the framework offers a great UI solution based on Node.js. This means using the Webpack Encore library. Webpack Encore simplifies the handling of CSS and Javascript files a lot. Basically, it’s a wrapper of the well known module bundler called Webpack to build preprocessed and minified CSS and Javascript files.

To setup Webpack Encore we need to install the corresponding bundle and install it’s dependencies with yarn. With the help of Docker you don’t need to have any Node.js tool installed on your machine:

docker run --rm -it -v $PWD:/app composer require symfony/webpack-encore-bundledocker run --rm -it -v $PWD:/app -w /app node:alpine yarn install

To build a simple autocomplete search bar feature we use Bootstrap and autocomplete.js:

docker run --rm -it -v $PWD:/app -w /app node:alpine yarn add bootstrap sass-loader autocomplete.js node-sass --dev

Now we start with the UI setup and create a Controller to render a template with a search field:

docker-compose exec php php bin/console make:controller AutoCompleteController

This generates a Controller with a Template. Let’s take a look at the files in the project structure:

We know now where the Template lives so we start to build the markup and add the needed styles and scripts:

{% extends 'base.html.twig' %}

{% block title %}Hello AutoCompleteController!{% endblock %}

{% block stylesheets %}
    {{ encore_entry_link_tags('app') }}
{% endblock %}
{% block javascripts %}
    {{ encore_entry_script_tags('app') }}
{% endblock %}

{% block body %}
    <div class="container">
        <div class="row justify-content-md-center">
            <div class="col-sm-6 col-sm-offset-3 mt-5">
                <form action="#" class="form">
                    <h2>Elasticsearch search as you type</h2>
                    <input class="form-control" id="search-input" name="contacts" type="text" placeholder='Search by name' />
                </form>
            </div>
        </div>
    </div>
{% endblock %}

As you can see we inherit first of all from the base Template that defines the basic Twig blocks and structure of the html document, which we will render in the end. We pass a basic title to the title block and start adding the assets. We will build an assets.js and assets.css file for the frontend logic and style. To get the asset links in the template, we need to use the encore_entry_link_tags and encore_entry_script_tags Twig functions to get the paths. After this a basic Bootstrap responsive container with a form and search input field is placed in the body Twig block. If you want to learn more about Bootstrap take a look at: https://getbootstrap.com/

Code the frontend

As you saw in the project structure above, there is also an assets folder in the document root. There we place a Javascript and a corresponding SASS file:

We start with the styling in the app.scss file like this:

// customize some Bootstrap variables
$primary: darken(#428bca, 20%);

// the ~ allows you to reference things in node_modules
@import "~bootstrap/scss/bootstrap";

.algolia-autocomplete {
  width: 100%;
}
.algolia-autocomplete .aa-input, .algolia-autocomplete .aa-hint {
  width: 100%;
}
.algolia-autocomplete .aa-hint {
  color: #999;
}
.algolia-autocomplete .aa-dropdown-menu {
  width: 100%;
  background-color: #fff;
  border: 1px solid #999;
  border-top: none;
}
.algolia-autocomplete .aa-dropdown-menu .aa-suggestion {
  cursor: pointer;
  padding: 5px 4px;
}
.algolia-autocomplete .aa-dropdown-menu .aa-suggestion.aa-cursor {
  background-color: #B2D7FF;
}
.algolia-autocomplete .aa-dropdown-menu .aa-suggestion em {
  font-weight: bold;
  font-style: normal;
}

We import the main bootstrap sass file that loads all the boostrap components. It’s a bit of an overhead but for this example it’s absolutely fine. Then we adjusted already the primary bootstrap color before loading Bootstrap itself to override the !default value. To change the primary color, we use the SASS darken function. The basic Bootstrap color variables look like this:

For more details on the Bootstrap theming take a look here: https://getbootstrap.com/docs/4.0/getting-started/theming/

After this adjustment we start adding a default style copied from the basic example of the autocomplete.js lib: https://github.com/algolia/autocomplete.js/blob/master/examples/basic.html

Once the assets.scss file is done, we need to implement the javascript part in app.js and import the app.scss file as a dependency:

// any CSS you import will output into a single css file (app.css in this case)
import '../css/app.scss';

import autocomplete from 'autocomplete.js';

autocomplete('#search-input', { hint: false }, [{
    source: function(query, cb) {
        fetch("/ac/search?q="+query)
            .then(response => response.json())
            .then(data => cb(data.products));
    }
}]);

It’s really important to mention how less code is necessary to have a working autocomplete input field. We use the imported autocomplete lib, set the input field id, disable any top result styles by setting hint to false and query via the fetch API our own Symfony Controller.

Let’s build the assets with Docker and yarn:

docker run --rm -it -v $PWD:/app -w /app node:alpine yarn run build

After the build we should get the following content in the public folder:

Besides the front Controller(index.php) there is a build folder containing the app.js and app.css file. The files contain a hash value in the middle. But for what is this for? If you set .enableVersioning() in webpack.config.js, a hash based on the file content is generated and added to the file name to avoid browsers to continue using an older version of your assets. To map these hash based file names, encore generates a manifest.json like this:

Also a file called entrypoints.json is generated to enable these Twig functions: encore_entry_script_tags/encore_entry_link_tags

The runtime.js file is a general Webpack file to manage and resolve the connections and interactions of your modules.

The final step is to code the controller that renders the template and executes the search:

<?php

namespace App\Controller;

use Elasticsearch\Client;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Routing\Annotation\Route;

class AutoCompleteController extends AbstractController
{
    /**
     * @Route("/ac", name="auto_complete")
     */
    public function index()
    {
        return $this->render('auto_complete/index.html.twig', [
            'controller_name' => 'AutoCompleteController',
        ]);
    }

    /**
     * @Route("/ac/search", name="auto_complete_search", methods="GET")
     */
    public function search(Client $client, array $indexDefinition, Request $request)
    {
        $query = $request->query->get('q');

        $result = $client->search(
            array_merge(
                $indexDefinition,
                ['body' => [
                    'query' => [
                        'match' => [
                            'title' => [
                                'query' => $query,
                                "operator" => "and",
                                "fuzziness" => 2,
                                "analyzer" => "standard"
                            ]
                        ],
                    ],
                    'size' => 3
                ]]
            ));

        $data = array_map(function ($item) {
            return ['value' => $item['_source']['title']];
        }, $result['hits']['hits']);

        return $this->json([
            'products' => $data
        ]);
    }
}

The index function simply renders the UI and the search function fetches the passed “q” param and triggers Elasticsearch to find the best 3 results and returns them in json format. The used match query searches for all title fields in the documents with a fuzziness of 2. This means a levenshtein distance is computed, so all strings match who need not more than 2 character changes to fit the searched string.

Browse the UI

Finally all the coding is done so we should be able to browse the UI and test the search. Start browsing the UI with this link: http://localhost:8080/ac

Now you can test your search as shown below:

Summary

It was a lot of work, but in the end you gathered all the knowledge about how to integrate Elasticsearch with Symfony. You are able to build a search as you type application yourself and you are ready to deep dive into other features of Elasticsearch. Even handling the development of an user interface is no magic anymore. So go ahead and build your Elasticsearch driven Symfony app now!