Running Solr 6.6 in Production

I thought I’d share our experiences of running Solr in production. So far Solr 6.6.6 has been very stable provided GC, memory and other system resources are carefully managed. Here are some of the technical details which I hope will be useful to anyone considering this platform.

We have been running Solr 6.6.6 with Java 9 in production for about a year now with the data index size of about 1.5 TB per node.

This cluster is a 4 node cluster indexing about 30-50GB of data per day with time to live (TTL) ranging from 14 to 90 days.

Here is how we configured Solr 6 with Java 9

1) Default Solr GC settings caused OOM exceptions as index size grew. We had to switch to G1 collector to improve stability. Solr would periodically stop with OOM with default GC settings.

2) Index size will impact max heap size requirement for Solr. Of course, a lot depends on your data and your data & SOLR configuration. We disabled most caches and even then heap size would grow with growth in index size. We found that ratio of index to heap to be around 23-25x.

Meaning estimated heap size per Solr node: Size of Index per node divided by 23. For 1TB index, you would expect to have heap at around 44GB. Experiment with your configuration for optimum settings.

3) G1 collector settings had to be tweaked carefully. Here are our settings for data index size of around 1.5TB:

SOLR_JAVA_MEM=”-Xmx64g -Xms64g”
GC_TUNE=” \
-XX:+UseG1GC \
-XX:+AggressiveOpts \
-XX:+PerfDisableSharedMem \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=32m \
-XX:MaxGCPauseMillis=500 \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/opt/nastel/solr/var/logs

Below are Max Heap and G1HeapRegionSize combinations:

4GB — 2MB
8GB — 4MB
16GB — 8MB
32GB — 16MB
64GB — 32MB

4) Separate filesystems for Solr binaries and /opt/nastel/solr/var where all logs and data are stored. We mounted /opt/nastel/solr/var file system on high performance direct attached SSD drives.

Solr VAR filesystem is sized as follows:
Total expected index size * 2. Solr will need significantly more space during restart recovery. You will run out of disk space if you don’t have enough free space available during restart recovery.

5) VM RAM size is Solr max heap size * 2 (at least 2x) to make sure Solr can store as much index into memory as possible. We typically run on 128GB RAM Linux VM if Solr heap size is 64GB.