How to Add Search to a Hugo Site with Lunr and Gulp

I just recently added search to Circuit Dojo but it was not as straight forward as I had hoped. There was plenty of people who had done it but there were some important details that were left out.

So, like anyone else in this situation, I hacked on it a bit to understand how things work under the hood. Here’s the process from beginning to end on how to get it done using a Gulpfile, and some Hugo templates generated on compile time.

Create the template file

The first thing that needs to be done is to create the template file that will generate a manifest of all the posts. The problem is, I happened to be using the default JSON template for Critical.

So after some research, I found that you can create an unlimited amount of output formats which then can be generated for different page types.

In the Hugo config, I put these lines to do just that.

[outputFormats]
  [outputFormats.Critical]
    mediaType = "application/json"
    baseName = "critical"
    isPlainText = true

[outputs]
    home = [ "HTML", "JSON", "RSS", "CRITICAL" ]
    page = [ "HTML" ]

So, I create a custom output format for Critical while also generating the default JSON. The Critical file is a manifest that Gulp uses on compilation to help with page load times while the default index.json will generate the complete manifest of all the pages on the site so they can be indexed.

This won’t work though the template files are in the right place. So, navigating to the layouts folder I created an index.critical.json and also created index.json.

For Critical, that template looks like the following:

{{- $.Scratch.Add "index" slice -}}
{{- range where .Site.Pages "Type" "not in"  (slice "page" "json") -}}
{{- if ne .File.BaseFileName "" -}}
{{- $url := printf "%s%s%s" .File.Dir .File.BaseFileName ".html" -}}
{{- $.Scratch.Add "index" (dict "file" $url) -}}
{{- end -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

Whereas the main index file looks like the following:

{{- $.Scratch.Add "index" slice -}}
{{- range where .Site.Pages "Type" "not in"  (slice "page" "json") -}}
{{- $link := .Permalink -}}
{{- $.Scratch.Add "index" (dict "uri" .Permalink "title" .Title "content" .Plain "summary" .Summary) -}}
{{- range sort (.Resources.ByType "page") "Params.order" -}}
{{- $title := replace (lower .Title) " " "-" -}}
{{- $anchor :=  printf "%s#%s" $link $title -}}
{{- $.Scratch.Add "index" (dict "uri" $anchor "title" .Title "content" .Plain "summary" .Summary) -}}
{{- end -}}
{{- end -}}
{{- $.Scratch.Get "index" | jsonify -}}

When hugo is run index.json and critical.json will be generated and placed in the public folder.

My critical.json looks something like this:

[{"file":"fundamentals/index.html"},{"file":"latching-load-switch/index.html"},{"file":"jeep-project/index.html"},{"file":"wire-gauge-calculator/index.html"}]

Whereas my index.json looks something like this:

[{"content":"Fundamentals are all about learning the small tips and trick that will make life easier when building a project. From ways to preventing errors to faster ways to prototype; it\u0026rsquo;s all here.\nCheck out the ever-growing table of contents below and start absorbing!\n","summary":"Fundamentals are all about learning the small tips and trick that will make life easier when building a project. From ways to preventing errors to faster ways to prototype; it\u0026rsquo;s all here.\nCheck out the ever-growing table of contents below and start absorbing!","title":"Fundamentals for Electronics Design","uri":"https://www.circuitdojo.org/fundamentals/"}
...

index.json is not in the form that Lunr expects to do speedy searches though. So, in the next step we’ll get Lunr to generate an index file that can be downloaded client side which will lead to quick searches and happy site goers.

Creating an Index Using Gulp

Now that the template files are in place, it’s time to generate the index file that will be used by Lunr client side. It’s much better to do it here and host that file statically than to have client side generate that file every time they load the page.

At a minimum you’ll need

var gulp = require("gulp");
var lunr = require("lunr");
var fs = require("fs");

My gulp file has a bunch more packages that it uses but i’ve removed them now for simplicity.

Next, I create a gulp task and called it search:

gulp.task("search", () => {
  const documents = JSON.parse(fs.readFileSync("public/index.json"));

  var store = {};

  let lunrIndex = lunr(function() {
    this.field("title", {
      boost: 10
    });
    this.field("content");
    this.ref("uri");

    documents.forEach(function(doc) {
      // console.log(doc);
      this.add(doc);

      //add info to store
      store[doc.uri] = { title: doc.title, summary: doc.summary };
    }, this);
  });

  fs.writeFileSync(
    "public/lunr-index.json",
    JSON.stringify({ index: lunrIndex, store: store })
  );
});

What this task does is read the manifest generated by Hugo (yes, you’ll need to compile your site before running this), look for the fields that you want to index, then adds each post to the Lunr index.

In order to display the results client side with actual content, you’ll need to use an extra key to store the raw content. In my case I used store to store both the title and summary generated by Hugo.

store[doc.uri] = { title: doc.title, summary: doc.summary };

I tied this to the uri of the document as that’s the only piece of information that Lunr spits back at you once a search is complete.

Now, when gulp serach is run, it will generate and place a file called lunar-index.json into your public folder. If you are developing locally, you may need to redirect this file to your static folder so your local Hugo instance will copy it over automagically.

Loading the Index Client Side Using JQuery and Lunr

To get the easy stuff out of the way, I created a partial for the javascript includes. This one is called scripts-search.html

<script src="https://unpkg.com/lunr/lunr.js"></script>
<script src="{{ "/js/search.js" }}"></script>

I created search.json and kept it in the static/js/ folder of the site. More on this in a second.

I also created another partial and called it search.html. The most important thing in this file is the search box and the unordered list that will display the results. A basic version of this would look like the following.

	<input type="search" placeholder="Search here.." id="search">
  <div class="results">
  <ul id="results"></ul>
  </div>

I’m using Bootstrap on Circuit Dojo so there’s lots more supporting HTML to make things look good that I’ll leave out for now.

Now back to search.js. I adopted this file completely from what was done in this gist. The only changes involved loading the indexed JSON file rather than the raw JSON manifest. In the init function it looks like this:

    $.getJSON("/lunr-index.json")
      .done(function(index) {
        console.log(index);
        store = index.store;
        lunrIndex = lunr.Index.load(index.index);
      })
      .fail(function(jqxhr, textStatus, error) {
        var err = textStatus + ", " + error;
        console.error("Error getting Hugo index flie:", err);
      });

As you can notice I’ve also copied the raw content into the store variable so it can be used a little later on.

I also simplified the search by just using the plain old .search() method that Lunr provides.

var results = lunrIndex.search(query);

In the render step, I limit the results to 20 and also reference the store for the post title and the post summary. Without those things the results would simply be links. (Boring!) If you want to see the full search.jsscroll to the bottom of this post.

After some styling everything came together and looks like this:

Now my Hugo site has full text search which requires no API calls or database lookups. It couldn’t have been possible without all the good people that have done the hard work to make it all possible. Be sure to check out the Lunr project and Hugo for more information.

By the way, my full modified search.js is below:

var store, lunrIndex, $results;

// Initialize lunrjs using our generated index file
function initLunr() {
  // First retrieve the index file

  $.getJSON("/lunr-index.json")
    .done(function(index) {
      store = index.store;
      lunrIndex = lunr.Index.load(index.index);
    })
    .fail(function(jqxhr, textStatus, error) {
      var err = textStatus + ", " + error;
      console.error("Error getting Hugo index flie:", err);
    });
}

// Nothing crazy here, just hook up a listener on the input field
function initUI() {
  $results = $("#results");
  $("#search").keyup(function() {
    $results.empty();

    // Only trigger a search when 2 chars. at least have been provided
    var query = $(this).val();
    if (query.length < 2) {
      return;
    }

    var results = lunrIndex.search(query);

    renderResults(results);
  });
}

/**
 * Display the 10 first results
 *
 * @param  {Array} results to display
 */
function renderResults(results) {
  if (!results.length) {
    return;
  }

  // Only show the ten first results
  results.slice(0, 20).forEach(function(result) {
    var $result = $("<li>");
    $result.append(
      $("<a>", {
        href: result.ref,
        text: store[result.ref].title
      })
    );
    $result.append($("<p>").html(store[result.ref].summary));
    $results.append($result);
  });
}

// Let's get started
initLunr();

$(document).ready(function() {
  initUI();
});

Thanks for reading! If you’re interested in more subscribe to my mailing list below. 👇👇👇

Last Modified: 2020.3.7