java - org.apache.http.ContentTooLongException: entity content is too long [105539255] for the configured buffer limit [104857600] - Stack Overflow

在生产环境批量同步数据的时候，我写了一个查询请求，然后直接报错：entity content is too long [142501157] for the configured buffer limit [104857600]

具体内容如下：

Caused by: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600] at org.elasticsearch.client.HeapBufferedAsyncResponseConsumer.onEntityEnclosed(HeapBufferedAsyncResponseConsumer.java:76) at org.apache.http.nio.protocol.AbstractAsyncResponseConsumer.responseReceived(AbstractAsyncResponseConsumer.java:137) at org.apache.http.impl.nio.client.MainClientExec.responseReceived(MainClientExec.java:315) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseReceived(DefaultClientExchangeHandlerImpl.java:151) at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.responseReceived(HttpAsyncRequestExecutor.java:315) at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:255) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) at java.lang.Thread.run(Thread.java:748) Suppressed: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600] ... 16 more Suppressed: org.apache.http.ContentTooLongException: entity content is too long [142501157] for the configured buffer limit [104857600] ... 16 more

在HttpAsyncResponseConsumerFactory类中：

在HeapBufferedAsyncResponseConsumer类中有一个判断：返回的内容长度大于限定的100MB就会抛出ContentTooLongException异常。

我这里用的依赖是：

<dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-client</artifactId> <version>6.6.2</version> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>transport</artifactId> <version>6.6.2</version> </dependency> <dependency> <groupId>org.elasticsearch.client</groupId> <artifactId>elasticsearch-rest-high-level-client</artifactId> <version>6.6.2</version> </dependency> <dependency> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> <version>6.6.2</version> </dependency>

而我们对es进行查询操作，或者其他增删改操作是，都会默认使用如下的RequestOptions.DEFAULT

进入RequestOptions.DEFAULT看它的代码：这里用了HeapBufferedResponseConsumerFactory.DEFAULT

而这个HeapBufferedResponseConsumerFactory.DEFAULT,就是我们之前看到的，已经被限制了100MB返回值的HeapBufferedResponseConsumerFactory。

所以有两种方式可以解决这个bug。

方式一：

不使用默认的RequestOptions.DEFAULT，而通过使用自定义RequestOptions的方式（ES官方api已经给我们开放出来了）：

RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder(); builder.setHttpAsyncResponseConsumerFactory( new HttpAsyncResponseConsumerFactory //修改为500MB .HeapBufferedResponseConsumerFactory(500 * 1024 * 1024));

RequestOptions requestOptions=builder.build();

方式二：

如果不方便修改怎么办，例如我们使用的是Spring Data ElasticSearch或者是其他的一些框架，而它默认使用的也是RequestOptions.DEFAULT

这时候我们可以通过java反射进行修改：

//设置es查询buffer大小 RequestOptions requestOptions = RequestOptions.DEFAULT; Class<? extends RequestOptions> reqClass = requestOptions.getClass(); Field reqField = reqClass.getDeclaredField("httpAsyncResponseConsumerFactory"); reqField.setAccessible(true); //去除final Field modifiersField = Field.class.getDeclaredField("modifiers"); modifiersField.setAccessible(true); modifiersField.setInt(reqField, reqField.getModifiers() & ~Modifier.FINAL); //设置默认的工厂 reqField.set(requestOptions, new HttpAsyncResponseConsumerFactory() { @Override public HttpAsyncResponseConsumer<HttpResponse> createHttpAsyncResponseConsumer() { //500MB return new HeapBufferedAsyncResponseConsumer(5 * 100 * 1024 * 1024); } });

如果你使用的是springmvc，那么可以把这段代码加入到拦截器中，这样可以拦截需要自定义返回值大小的es请求

---

1.本站遵循行业规范，任何转载的稿件都会明确标注作者和来源；2.本站的原创文章，会注明原创字样,如未注明都非原创，如有侵权请联系删除！；3.作者投稿可能会经我们编辑修改或补充；4.本站不提供任何储存功能只提供收集或者投稿人的网盘链接。