ElasticSearch--性能优化相关的配置

ElasticSearch--性能优化相关的配置

原文网址：

简介

本文介绍ElasticSearch中性能优化有关的配置。

相关的配置有：

内存大小批量写的数量索引数量

内存大小

说明

ElasticSearch是个Java进程，JVM默认的最大内存是1G。一般需要增大这个配置。

配置方法

配置文件：config/jvm.options

原来是：
##-Xms4g
##-Xmx4g

改为：
-Xms4g
-Xmx4g

报错信息

如果这个值不够大，会报的错如下：

主要错误信息：

Caused by: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]]

详细错误信息：

Caused by: org.springframework.data.elasticsearch.UncategorizedElasticsearchException: Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]; nested exception is ElasticsearchStatusException[Elasticsearch exception [type=circuit_breaking_exception, reason=[parent] Data too large, data for [] would be [991614656/945.6mb], which is larger than the limit of [986061209/940.3mb], real usage: [991390584/945.4mb], new bytes reserved: [224072/218.8kb], usages [request=0/0b, fielddata=0/0b, in_flight_requests=224072/218.8kb, model_inference=0/0b, accounting=11298560/10.7mb]]]
	at org.springframework.data.elasticsearch.core.ElasticsearchExceptionTranslator.translateExceptionIfPossible(ElasticsearchExceptionTranslator.java:67)
	at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:370)
	at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:353)
	at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.doBulkOperation(ElasticsearchRestTemplate.java:238)
	at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.bulkIndex(ElasticsearchRestTemplate.java:191)
	at org.springframework.data.elasticsearch.core.documentOperations.bulkIndex(documentOperations.java:152)
	at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.save(AbstractElasticsearchTemplate.java:175)
	at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.save(AbstractElasticsearchTemplate.java:159)

错误的解析

Data too large, data for [] would be [991614656/945.6mb]    //A
which is larger than the limit of [986061209/940.3mb]       //B
real usage: [991390584/945.4mb]                             //C
new bytes reserved: [224072/218.8kb]                        //D

这里有4个数值：

A：本次将会消耗的总内存B：上限，超过这个就报错。（缺省值是ES最大内存的95%)C：本机上ES进程已使用的内存大小D：你本次 *** 作（或者说执行当前的任务）所需要的内存

本次 *** 作中，C + D = A > B，所以报错了。

批量写的数量

说明

提高ES的批量写入数据的性能。

配置方法

配置文件：config/elasticsearch.yml

添加下边一行：

thread_pool.write.queue_size: 1000

报错信息

如果这个值不够大，会报429错误：

主要错误信息：

{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[elasticsearch-0][10.16.9.130:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService$7@19a5cf02 on EsThreadPoolExecutor[name = elasticsearch-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@389297ad[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 147611]]"},"status":429}

详细错误信息：

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:9200], URI [/service_instance_inventory/type/6_tcc-app-gateway-77b98ff6ff-crblx.cards_0_0/_update?refresh=true&timeout=1m], status line [HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[elasticsearch-0][10.16.9.130:9300][indices:data/write/update[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of org.elasticsearch.transport.TransportService@19a5cf02 on EsThreadPoolExecutor[name = elasticsearch-0/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@389297ad[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 147611]]"},"status":429}
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch

索引数量

详见：ElasticSearch--解决this action would add [5] total shards, but this cluster currently has [1000]/[1000_IT利刃出鞘的博客-CSDN博客

说明

默认最大允许的索引数是1000个，如果业务需要大于这个数，就需要修改配置。

配置方法

法1：head工具

无需重启ES。

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "max_shards_per_node":10000
    }
  }
}

persistent：永久生效，transient：临时生效。  

法2：CURL命令

无需重启ES。

curl --location --request PUT 'http://127.0.0.1:9200/_cluster/settings' 
--header 'Content-Type: application/json' 
--data '{"persistent":{"cluster":{"max_shards_per_node":10000}}}'

persistent：永久生效，transient：临时生效。  

报错信息

Elasticsearch exception [type=validation_exception, reason=Validation Failed: 1: this action would add [5] total shards, but this cluster currently has [1000]/[1000] maximum shards open;]