Searchlight is a Views-driven approach to pluggable search in Drupal. Its goals include

• Clean pluggability with search services like Sphinx and Apache Solr

• Per-site or per-application customization of the search schema for fine-grained control of what gets indexed and how

• Support for any Views base tables (e.g. nodes, users, comments, data tables)

• Leverage Views API integration already provided by contrib modules for indexing, faceting, and display

• Multisite deployment, including integration with Aegir.

In this screencast, I explain the high level concepts of Searchlight and do a quick demo of the module:

What we wanted out of Searchlight

We’ve been interested in the integration of advanced search solutions with Drupal for quite some time. Fast, relevant search has a huge potential for shaping our products and helping our clients tell their stories. In the Drupal contrib space big strides have been made by the Apache Solr project and contributions to Views that make swappable query plugins in Views 3.x possible.

At the same time, we started taking note of limitations that were particular pain points for our projects. We knew that we needed support for more than Drupal’s nodes, users, and comments. Some of our advanced data projects like Afghanistan Election Data rely on custom data tables for speed and flexibility. Our projects can have very niche use cases so configurable indexes was a must. Strong Views integration was also critical. Views powers many of our data visualizations and it was simply not an option to build a completely separate stack to power visualizations using search.

Finally, we simply couldn’t make up our minds between two of the heavyweights in open source search today. Apache Solr has great community adoption, extremely flexible indexing, and a grab-bag of features like file content search. On the other hand Sphinx blazes through even the biggest datasets and has a hardcore developer base hell-bent on improving performance and maintaining thorough documentation.

Where we are and where we’re going

In the past few weeks we’ve gotten a lot closer to meeting a lot of Searchlight’s goals. We have working Solr and Sphinx backend plugins, good Views 3.x integration and a Views 2.x compatibility layer, a facet plugin system, and Aegir/multisite deployment for the Sphinx backend. That said, we still have a ways to go before our first real alpha or beta tag. We want to explore Aegir deployment for our Solr backend, provide clean fallback to core search if other search services are not available, and provide good test coverage.

The commits are currently flying fast and furious in the Searchlight github repo. If you’re interested in checking this work out make sure to read the docs, roll up your sleeves, and expect to get dirty. We promise that Searchlight will make its way back to Drupal.org as soon as the dust settles.