ISP Problem Solving Blog, Software Development and More

Mor-Pah.net

Tag Archives: solr 5

Dovecot 2.2 with Solr 6 (or 5)

Posted on by Posted in Linux 11 Comments

I found very little (if anything) out there with people trying to setup Solr 5 and Dovecot 2 so thought i’d post this up.  Perhaps it will help someone else along the way.

What i’m running:-

  • FreebSD 10.3
  • Openjdk 8
  • Dovecot 2.2.24 (installed from pkgs)
  • Apache-solr (downloaded direct from apache)

I’m not going to cover how to install and configure dovecot here but in short I installed it using “pkg install dovecot2” on FreeBSD and configured based on my needs in /usr/local/etc/dovecot/dovecot.conf along with the following entries to enable FTS and Solr

 

...
mail_plugins = $mail_plugins quota fts fts_solr
...
plugin {
fts_autoindex=yes
fts = solr
fts_solr = url=http://localhost:8983/solr/dovecot/
}
...
protocol imap {
mail_plugins = quota imap_quota fts fts_solr
}
protocol lmtp {
mail_plugins = quota fts fts_solr
}
...

The above assumes that you’re running solr on localhost of course.

Now, in Solr version 5+ you no longer need tomcat or jetty installed separately to run solr so forget reading any articles that advise you to install these unless you’re running solr version 4 or older.  Reason being, solr now ships with a copy of jetty and all the startup scripts you need.

  1. First thing I did was install apache-solr from packages and then remove it so that FreeBSD will install all the prerequisites needed to run solr.  I don’t have a list of the java/jre packages so this was a bit of a cheating way to achieve it really. But all you need to install is OpenJDK 8 JRE and make sure that your java version output is at least version 1.8 as below 
    # java -version
openjdk version "1.8.0_92"
OpenJDK Runtime Environment (build 1.8.0_92-b14)
OpenJDK 64-Bit Server VM (build 25.92-b14, mixed mode)

    Version 1.7 (OpenJDK 7) will not work with solr and you will find that solr will not start

    pkg install apache-solr (make sure it's apache-solr not version 3)

    This will prompt you to install numerous java packages and other bits and bobs.   Once completed, remove just apache-solr because we don’t need it.

    pkg remove apache-solr

    In theory, this should leave all the other packages including jre installed.

  2. Download a copy of apache solr and extract to a suitable locationmkdir /opt 
    cd /opt
fetch http://www-eu.apache.org/dist/lucene/solr/6.1.0/solr-6.1.0.tgz
tar -xvzf  solr-6.1.0.tgz
mv solr-6.1.0 solr
  3. Solr should now just start without issue by running the below 
    bin/solr start

    once started it will drop to the background

  4. Now solr should be running you can create a brand new collection for dovecot 
    bin/solr create -c dovecot -n dovecot
  5. If all went well you’ll have a new collection made, so lets navigate to the configuration folder 
    cd /opt/solr/server/solr/dovecot/conf/
  6. Once there you’ll need to download a create a file named schema.xml and load it with the following 
    <?xml version="1.0" encoding="UTF-8" ?>

<!--
For fts-solr:

This is the Solr schema file, place it into solr/conf/schema.xml. You may
want to modify the tokenizers and filters.
-->
<schema name="dovecot" version="1.5">
<types>
<!-- IMAP has 32bit unsigned ints but java ints are signed, so use longs -->
<fieldType name="string" class="solr.StrField" />
<fieldType name="long" class="solr.TrieLongField" />
<fieldType name="boolean" class="solr.BoolField" />

<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.EnglishMinimalStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="lang/stopwords_en.txt"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPossessiveFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
<filter class="solr.EnglishMinimalStemFilterFactory"/>
</analyzer>
</fieldType>
</types>
<fields>
<field name="id" type="string" indexed="true" stored="true" required="true" />
<field name="uid" type="long" indexed="true" stored="true" required="true" />
<field name="box" type="string" indexed="true" stored="true" required="true" />
<field name="user" type="string" indexed="true" stored="true" required="true" />

<field name="hdr" type="text" indexed="true" stored="false" />
<field name="body" type="text" indexed="true" stored="false" />

<field name="from" type="text" indexed="true" stored="false" />
<field name="to" type="text" indexed="true" stored="false" />
<field name="cc" type="text" indexed="true" stored="false" />
<field name="bcc" type="text" indexed="true" stored="false" />
<field name="subject" type="text" indexed="true" stored="false" />

<!-- Used by Solr internally: -->
<field name="_version_" type="long" indexed="true" stored="true"/>
</fields>

<uniqueKey>id</uniqueKey>
</schema>
  7. Now you need to delete the managed schema file that comes with solr 5/6 as we will not be using a managed schema 
    rm managed-schema
  8. Next, open up the below file in a text editor, i.e. 
    vi solrconfig.xml

    and completely remove this section

    <processor class="solr.AddSchemaFieldsUpdateProcessorFactory">
<str name="defaultFieldType">strings</str>
<lst name="typeMapping">
<str name="valueClass">java.lang.Boolean</str>
<str name="fieldType">booleans</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.util.Date</str>
<str name="fieldType">tdates</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Long</str>
<str name="valueClass">java.lang.Integer</str>
<str name="fieldType">tlongs</str>
</lst>
<lst name="typeMapping">
<str name="valueClass">java.lang.Number</str>
<str name="fieldType">tdoubles</str>
</lst>
</processor>

    Now add the following schema type change to the file, I’ve included the comment section that was above it to show where I had added it at around line 1260

    ........
See http://wiki.apache.org/solr/GuessingFieldTypes-->
<schemaFactory class="ClassicIndexSchemaFactory">
</schemaFactory>
  9. With a bit of luck that’s everything you need to do so go ahead and restart solr by issuing /opt/solr/bin/solr restart and head straight to the logs file and you should hopefully see the following. 
    tail -f /opt/solr/server/logs/solr.log
......
2016-08-15 14:40:37.515 INFO (coreLoadExecutor-6-thread-1) [ x:dovecot] o.a.s.c.CoreContainer registering core: dovecot

    If you see any errors about failing to register field types, double check over the steps above and make sure you didn’t miss anything or something managed to creep back in

  10. finally open up another terminal and tail the solr log file as above whilst running the following command in dovecot and you should have all your emails indexed into solr 🙂 
    doveadm index -A inbox

    Good luck!