Lucene : avoiding locking (incremental indexing)
2004-11-15 - By Luke Francl BackThis is how I implemented incremental indexing. If anyone sees anything
wrong, please let me know.
Our motivation is similar to John Eichel‘s. We have a digital asset
management system and when users update, delete or create a new asset,
they need to see their results immediately.
The most important thing to know about incremental indexing that
multiple threads cannot share the same IndexWriter, and only one
IndexWriter can be open on an index at a time.
Therefore, what I did was control access to the IndexWriter through a
singleton wrapper class that synchronizes access to the IndexWriter and
IndexReader (for deletes). After finishing writing to the index, you
must close the IndexWriter to flush the changes to the index.
If you do this you will be fine.
However, opening and closing the index takes time so we had to look for
some ways to speed up the indexing.
The most obvious thing is that you should do as much work as possible
outside of the synchronized block. For example, in my application, the
creation of Lucene Document objects is not synchronized. Only the part
of the code that is between your IndexWriter.open() and
IndexWriter.close() needs to be synchronized.
The other easy thing I did to improve performance was batch changes in a
transaction together for indexing. If a user changes 50 assets, that
will all be indexed using one Lucene IndexWriter.
So far, we haven‘t had to explore further performance enhancements, but
if we do the next thing I will do is create a thread that gathers assets
that need to be indexed and performs a batch job every five minutes or
so.
Hope this is helpful,
Luke
