Indexing and Searching Using Lucene

Have a huge database and need to access data from it in a short span of time? Meet Lucene. Lucene works by creating documents of the data from the database, and these documents are stored in an index. When data is required and needs to be searched, it uses the index to get faster results by fetching the required documents.

Using Lucene.NET in C#

In C# we can use Lucene.NET as a dll integrated with the project and it does not require the installation of any other packages.
It basically has two parts.
1) Creating the Lucene index.
2) Searching through the Lucene index.

1. Creating the Lucene Index

Lucene.NET creates text indexes of the data that needs to be stored and later searched through.
The following are the steps to creating the Lucene Index:

Create a Directory:
– The directory ‘LuceneIndex’ is created to store the Lucene index. This will store the data in the form of documents.
– Directory directory = FSDirectory.Open(“C:\LuceneIndex”);

Create an Analyzer:
– The analyzer is used to process the text before it is written to the directory. It breaks down the text into single words/tokens and removes the stopwords like ‘a’, ‘the’, ‘and’, ‘is’.
– Lucene.Net.Util.Version LuceneVersion = Lucene.Net.Util.Version.LUCENE_30;
– Analyzer analyzer = new StandardAnalyzer(LuceneVersion);

Create the Index Writer:
– The writer works between the analyzer and the directory. It first uses the analyzer to first break down the text and then writes this result to the directory.
– IndexWriter writer = new IndexWriter(directory, analyzer, IndexWriter.MaxFieldLength.UNLIMITED)

Create a Document:
– The data written to the directory is in the form of documents. You can consider a document as an entity, for example, a Student.
– Document doc = new Document();

Add fields to the Document:
– Fields are name-value pairs. Each document will contain fields. For example, the document ‘Student’ can have the fields Id, FirstName, and LastName. While creating a field, we can specify whether and how a field should be stored, and be indexed.

• doc.Add(new Field(“Id”, “123”, Field.Store.YES, Field.Index.NOT_ANALYZED));
  
• doc.Add(new Field(“FirstName”, “John”, Field.Store.YES, Field.Index.ANALYZED));
  
• doc.Add(new Field(“LastName”, “Doe”, Field.Store.YES, Field.Index.ANALYZED));

Write the Document to the Index:
– This document is stored in the directory specified in the first step.
– writer.AddDocument(doc);
Now we can check the directory C:\LuceneIndex and we will see that the index files have been created. We can now use the index to perform search and retrieval of data.

ALSO READ: Learning REACT JS – New Technology

2. Searching Through the Lucene Index

To search for a name in the field FirstName, we do the following:

Form the Query:
– We then parse the value ‘John’ that we are searching for through the query parser.
Query mainQuery = queryParser.Parse(“John”);

Search for the Results
– The searcher takes the directory and the query to get the result. A limit is used to specify how many of the matching results should be returned.
– Searcher searcher = new IndexSearcher(IndexReader.Open(directory, true));
int limit = 5;
ScoreDoc[] results = searcher.Search(mainQuery, limit).ScoreDocs;

Lucene.NET can be easily used in a project for searching data in a large database. Once the index is created, retrieving any data is made easy with the use of various search queries available. Hence, saving a lot of time and calls to the actual DB.

Melody Fernandes