SpringCloud-Gateway:reactor.netty.channel.AbortedException: Connection has been

网络 02-07 5292

项目场景：

项目采用了SpringCloud 2020.0.4?SpringCloudAlibaba 2021.1 SpringBoot 2.5.6 SpringCloudGateway 3.0.4 这种微服务架构前期流量小没出什么问题但是最近业务量上去了?

SpringCloudGateway这个微服务偶发?

reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation

以及链接被拒错误信息：

2021-12-18 10:23:00.319 WARN 1 --- [or-http-epoll-2] r.netty.http.client.HttpClientConnect : [id: 0x189f9dab, L:/10.200.78.85:31838 - R:<*******our host / IP **********>:8081] The connection observed an error

这个异常信息特此记录一下

问题描述：

服务运行一段时间后网关会抛出异常信息，网关下面的微服务也会报一些异常：

网关异常信息： 2021-12-20 03:51:32.405 ERROR 1 --- [or-http-epoll-8] reactor.core.publisher.Operators : Operator called default onErrorDropped reactor.netty.channel.AbortedException: Connection has been closed BEFORE send operation at reactor.netty.channel.AbortedException.beforeSend(AbortedException.java:60) at reactor.netty.http.HttpOperations.then(HttpOperations.java:172) at reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:679) at reactor.netty.ReactorNetty$OutboundThen.<init>(ReactorNetty.java:672) at reactor.netty.NettyOutbound.then(NettyOutbound.java:359) at reactor.netty.http.HttpOperations.send(HttpOperations.java:110) at org.springframework.http.server.reactive.ReactorServerHttpResponse.writeWithInternal(ReactorServerHttpResponse.java:92) at org.springframework.http.server.reactive.AbstractServerHttpResponse.lambda$null$7(AbstractServerHttpResponse.java:240) at org.springframework.http.server.reactive.AbstractServerHttpResponse.doCommit(AbstractServerHttpResponse.java:297) at org.springframework.http.server.reactive.AbstractServerHttpResponse.lambda$writeWith$8(AbstractServerHttpResponse.java:240) at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:186) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:199) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120) at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:280) at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:389) at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:405) at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:707) at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:94) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 2021-12-20 05:59:26.175 WARN 1 --- [or-http-epoll-7] r.netty.http.client.HttpClientConnect : [id:12748923-11159, L:/172.17.0.5:47870 - R:113.75.42.235/113.75.42.235:7071] The connection observed an error reactor.netty.channel.AbortedException: Connection has been closed at reactor.netty.http.server.HttpServerOperations.onInboundClose(HttpServerOperations.java:575) at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:74) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) 网关下面的微服务偶发异常信息如下： 2021-12-22 13:02:33.211 ERROR 6 --- [ XNIO-1 task-49] c.l.g.c.s.h.GlobalBizExceptionHandler : 全局异常信息 ex=I/O error while reading input message; nested exception is java.io.IOException: UT000128: Remote peer closed connection before all data could be read ?

原因分析：

参考博客地址：

readAddress(..) failed: Connection reset by peer · Issue #2114 · spring-cloud/spring-cloud-gateway · GitHubHI , we are getting the below issue in Production environments when we are proxying request to tomcats 2021-01-18 10:23:00.319 WARN 1 --- [or-http-epoll-2] r.netty.http.client.HttpClientConnect : [id: 0x189f9dab, L:/10.200.78.85:32818 - ...https://github.com/spring-cloud/spring-cloud-gateway/issues/2114Got Connection prematurely closed BEFORE response · Issue #796 · reactor/reactor-netty · GitHubhttps://github.com/reactor/reactor-netty/issues/796Connection has been closed BEFORE response异常_荒野哞的专栏-CSDN博客最近我组使用的spring cloud gateway 线上偶发返回500，后台查看报错日志信息，发现有一条下面的异常：reactor.netty.channel.AbortedException: Connection has been closed BEFORE response网上狂搜，以我以往的解决思路，欲解决问题，首先复现问题，然后再解决问题，这种最放心，若是实在无法复...https://blog.csdn.net/songjinbin/article/details/103537581关于使用Spring Cloud Gateway网关网络连接的4个重要的参数配置_ankeway的博客-CSDN博客_spring.cloud.gateway.httpclient整理出这四个重要的参数，说起来很不易，来源于一次网络时候事故后的调查，对于平时使用Spring Cloud Gateway(简称scg)来说这些参数几乎很少会关注到，从网上也很少能看到讲解的文章，表面上是SCG的问题，实则都是和SCG的底层网络通信框架Netty有关系。率先曝光一下这4个参数System.setProperty("reactor.netty.pool.leasingStrategy","lifo");spring.cloud.gateway.httpclient.pool.max.https://blog.csdn.net/ankeway/article/details/108745751

解决方案：

按照各位大佬配置的方式加了相关配置：

#网关gateway加配置并且启动也加了-Dreactor.netty.pool.leasingStrategy=lifo spring: cloud: gateway: httpclient: pool: max-idle-time: 10000 max-connections: 500 #网关下面的微服务加配置： server: tomcat: connection-timeout: 20000

发现问题并没有得到解决，流量高峰期还是偶发这个错误~~

目前问题还在解决中欢迎进群踩坑交流（新建的群~）：738720661