长安网站建设制作价格,乐清网站,ppt做的比较好的网站有哪些,软文发布平台基本信息 客户名称#xff1a;xxx 产品名称#xff1a; 版本号#xff1a; 问题分类#xff1a;编码问题 环境类型#xff1a;实体机 问题现象 保险公司测试环境爆内存#xff0c;机器挂掉。总共64g的内存#xff0c;在没有开始测试tomcat部署war包前内存使用率为25%左… 基本信息 客户名称xxx 产品名称 版本号 问题分类编码问题 环境类型实体机 问题现象 保险公司测试环境爆内存机器挂掉。总共64g的内存在没有开始测试tomcat部署war包前内存使用率为25%左右。开始测试tomcat后内存使用率一直上涨内存使用率从12-08下午起就维持在98%的水平知道最后机器down掉才发现存在的问题。 问题原因 tomcat执行shutdown后虽然端口占用被释放tomcat也已经停止。但tomcat的进程并未被杀死仍旧占用着一开始分配给他的xmx的jvm内存空间。 Tomcat是一个Servlet容器用于部署Serlvet程序我们通常写的各种Java Web应用本质上就是一个Servlet程序。也就说在停止Tomcat时不仅仅需要释放Tomcat进程本身所占用的资源还需要释放Serlvet程序所占用的资源。而出现“停止Tomcat之后进程依然存在”这种现象的主要原因就是我们自己写的Java Web应用在Tomcat容器停止时没有正常释放所占用的系统资源比如线程池未关闭输入输出流未关闭等等。这些系统资源没有被完全释放导致tomcat进程一直存在且无法被正常回收。 下面是tomcat的shutdown执行的catalina命令可以看到是程序是先调用Bootstrap类的方法释放Tomcat进程所占用的资源。再使用kill命令停止Tomcat进程kill -15 tomcat_process_id。到在这里使用kill命令发送的信号为SIGTERM15kill -15在进程未释放系统资源时不能停止Tomcat进程如程序的线程池未关闭。
eval $_RUNJAVA $LOGGING_MANAGER $JAVA_OPTS-Djava.endorsed.dirs$JAVA_ENDORSED_DIRS -classpath $CLASSPATH-Dcatalina.base$CATALINA_BASE-Dcatalina.home$CATALINA_HOME-Djava.io.tmpdir$CATALINA_TMPDIRorg.apache.catalina.startup.Bootstrap $ stop
# stop failed. Shutdown port disabled? Try a normal kill.if [ $? ! 0 ]; thenif [ ! -z $CATALINA_PID ]; thenecho The stop command failed. Attempting to signal the process to stop through OS signal.kill -15 cat $CATALINA_PID /dev/null 21fifi 排查过程 察觉到问题发生时机器已经down掉了。服务器管理员已经执行了重启的操作时间点为10:29。因此无法直接得知机器当时的内存占用的情况。只能通过一些命令获取重启前机器的部分信息 1、查看系统日志查看宕机时内存和cpu的情况
1.1、查看是否有杀内存的情况发生cat /var/log/messages |grep -i kill 可以看到在宕机时发生了杀内存的情况证明在机器出问题之前内存已经相当吃紧。基本上可以确认问题就出在机器内存满了。 1.2、查看历史cpu占用情况sar -u -f /var/log/sa/sa09
cup看起来没什么异常 1.3、查看历史内存sar -r -f /var/log/sa/sa09
可以看到内存占用一直处在99.45%的超高状态用户占有率高达181% 1.4、再向前看一天 sar -r -f /var/log/sa/sa09 可以观察到8时50分和9时00分内存激增10g左右。这个有明显的异常且内存激增的时间点和tomcat开始测试的时间点吻合。因此怀疑是tomcat导致的问题。 2、本地测试尝试还原问题原因
2.1、启动tomcat容器和容器中的luna-project-hub服务查看内存占用 netstat -tnlp | grep 8455port ps -ef | grep 2730pid 2.2、关闭tomcat
执行shutdown命令后项目已经关闭。 重新执行ps -ef | grep 2730发现进程居然仍旧存在没有被杀死。 2.3、jstack查看tomcat关闭后进程中仍旧存在的线程 发现存在的都是一些ScheduledExecutorService类创建的线程。这些线程没有无法关闭导致了这次的问题。
[rootlocalhost bin]# jstack -l 2730
2022-12-09 17:16:57
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.301-b09 mixed mode):Attach Listener #215 daemon prio9 os_prio0 tid0x00007f3608001800 nid0xc4e waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneDestroyJavaVM #214 prio5 os_prio0 tid0x00007f364c5fe800 nid0xb27 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneCACHE_LOAD_POOL1 #184 prio5 os_prio0 tid0x00007f35d829e800 nid0xbe1 waiting on condition [0x00007f359bfdf000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000b0a627b8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.beat.sender #183 daemon prio5 os_prio0 tid0x00007f35b8008000 nid0xbe0 waiting on condition [0x00007f359c0e0000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000ab99ee30 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneDictionaryCacheRefreshThreadPool1 #181 prio5 os_prio0 tid0x00007f35d8296000 nid0xbde waiting on condition [0x00007f359c1e1000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000b0a8f210 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneCurator-Framework-0 #178 daemon prio5 os_prio0 tid0x00007f35a899f800 nid0xbdb waiting on condition [0x00007f359c4e2000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000b16411d8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000b16413d8 (a java.util.concurrent.ThreadPoolExecutor$Worker)localhost-startStop-1-EventThread #177 daemon prio5 os_prio0 tid0x00007f35a899d000 nid0xbda waiting on condition [0x00007f359c5e3000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000b1647918 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)Locked ownable synchronizers:- Nonelocalhost-startStop-1-SendThread(10.60.44.187:2181) #176 daemon prio5 os_prio0 tid0x00007f35a899b800 nid0xbd9 runnable [0x00007f359c6e4000]java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked 0x00000000b1618ba8 (a sun.nio.ch.Util$3)- locked 0x00000000b1618b98 (a java.util.Collections$UnmodifiableSet)- locked 0x00000000b16186f0 (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)Locked ownable synchronizers:- NoneCurator-ConnectionStateManager-0 #175 daemon prio5 os_prio0 tid0x00007f35a892e800 nid0xbd8 waiting on condition [0x00007f359c7e5000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000b1641938 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000b1641b30 (a java.util.concurrent.ThreadPoolExecutor$Worker)TaskCommandCacheRefreshThreadPool1 #173 prio5 os_prio0 tid0x00007f35a81ff000 nid0xbd6 waiting on condition [0x00007f359c8e6000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000aedc4b70 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.beat.sender #172 daemon prio5 os_prio0 tid0x00007f35a81fd000 nid0xbd5 waiting on condition [0x00007f359c9e7000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000ab99ee30 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneTHREAD_POOL_ALARM_POOL-1 #169 prio5 os_prio0 tid0x00007f35a80fc800 nid0xbd3 waiting on condition [0x00007f359d7e9000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000b0e8b0e8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneTimer-3 #162 daemon prio5 os_prio0 tid0x00007f35a8b0b000 nid0xbcc in Object.wait() [0x00007f359e6ee000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.lang.Object.wait(Object.java:502)at java.util.TimerThread.mainLoop(Timer.java:526)- locked 0x00000000ad465348 (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.push.receiver #127 daemon prio5 os_prio0 tid0x00007f35a80a7800 nid0xba9 runnable [0x00007f35a3a19000]java.lang.Thread.State: RUNNABLEat java.net.PlainDatagramSocketImpl.receive0(Native Method)- locked 0x00000000ab99e588 (a java.net.PlainDatagramSocketImpl)at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)- locked 0x00000000ab99e588 (a java.net.PlainDatagramSocketImpl)at java.net.DatagramSocket.receive(DatagramSocket.java:812)- locked 0x00000000abe96300 (a java.net.DatagramPacket)- locked 0x00000000ab99e548 (a java.net.DatagramSocket)at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:83)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000ab99e2d8 (a java.util.concurrent.ThreadPoolExecutor$Worker)com.alibaba.nacos.naming.failover #126 daemon prio5 os_prio0 tid0x00007f35a80a5000 nid0xba8 waiting on condition [0x00007f35a3b1a000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000ab9b4980 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.naming.updater #125 daemon prio5 os_prio0 tid0x00007f35a80a3000 nid0xba7 waiting on condition [0x00007f35a3c1b000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000ab9b3c60 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.naming.updater #124 daemon prio5 os_prio0 tid0x00007f35a80a1000 nid0xba6 waiting on condition [0x00007f35a3d1c000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000ab9b3c60 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.client.listener #123 daemon prio5 os_prio0 tid0x00007f35a809e800 nid0xba5 waiting on condition [0x00007f35a3e1d000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000ab99eb90 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:163)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000ab99e918 (a java.util.concurrent.ThreadPoolExecutor$Worker)AsyncAppender-Worker-ConsoleAppender #117 daemon prio5 os_prio0 tid0x00007f35e5959000 nid0xb9f waiting on condition [0x00007f35a5722000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000aa244020 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.Worker.fixed-10.60.52.109_8848-insurance-ats-upgrade-urtrust #114 daemon prio5 os_prio0 tid0x00007f35e59d2000 nid0xb9c runnable [0x00007f35a501f000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000aa250eb8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.config.security.updater #113 daemon prio5 os_prio0 tid0x00007f35e59d0000 nid0xb9b waiting on condition [0x00007f35a5120000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000aa2506b0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneTimer-2 #112 daemon prio5 os_prio0 tid0x00007f35e59c9800 nid0xb9a in Object.wait() [0x00007f35a5421000]java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.util.TimerThread.mainLoop(Timer.java:552)- locked 0x00000000aa251358 (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- NoneAsyncAppender-Worker-ConsoleAppender #108 daemon prio5 os_prio0 tid0x00007f35e5945800 nid0xb96 waiting on condition [0x00007f35a5b24000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000aa0d74c8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- NoneCACHE_LOAD_POOL1 #100 prio5 os_prio0 tid0x00007f3618022000 nid0xb8c waiting on condition [0x00007f35a6a27000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a771aee0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.beat.sender #99 daemon prio5 os_prio0 tid0x00007f35ec004000 nid0xb8b waiting on condition [0x00007f35a6d28000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a3300ea8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneDictionaryCacheRefreshThreadPool1 #96 prio5 os_prio0 tid0x00007f3618003000 nid0xb88 waiting on condition [0x00007f35a6e29000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a771abb8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneCurator-Framework-0 #93 daemon prio5 os_prio0 tid0x00007f35e7630000 nid0xb85 waiting on condition [0x00007f35a7ffe000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a83f4b90 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000a83f4d90 (a java.util.concurrent.ThreadPoolExecutor$Worker)localhost-startStop-1-EventThread #92 daemon prio5 os_prio0 tid0x00007f35e70f6000 nid0xb84 waiting on condition [0x00007f35ac550000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a83f4490 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)Locked ownable synchronizers:- Nonelocalhost-startStop-1-SendThread(10.60.44.187:2181) #91 daemon prio5 os_prio0 tid0x00007f35e70f4800 nid0xb83 runnable [0x00007f35ac651000]java.lang.Thread.State: RUNNABLEat sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)- locked 0x00000000a83e2b00 (a sun.nio.ch.Util$3)- locked 0x00000000a83e2af0 (a java.util.Collections$UnmodifiableSet)- locked 0x00000000a83e2648 (a sun.nio.ch.EPollSelectorImpl)at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)Locked ownable synchronizers:- NoneCurator-ConnectionStateManager-0 #90 daemon prio5 os_prio0 tid0x00007f35e749c800 nid0xb82 waiting on condition [0x00007f35ac952000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a83f52d0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000a83f54c8 (a java.util.concurrent.ThreadPoolExecutor$Worker)TaskCommandCacheRefreshThreadPool1 #88 prio5 os_prio0 tid0x00007f35e6b33800 nid0xb80 waiting on condition [0x00007f35aca53000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a48584c0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.beat.sender #86 daemon prio5 os_prio0 tid0x00007f35e6b2f800 nid0xb7e waiting on condition [0x00007f35acc55000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a3300ea8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneTHREAD_POOL_ALARM_POOL-1 #83 prio5 os_prio0 tid0x00007f35e7185800 nid0xb7c waiting on condition [0x00007f35ad257000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a7f8c2e0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneJava2D Disposer #81 daemon prio10 os_prio0 tid0x00007f35e76f4800 nid0xb7b in Object.wait() [0x00007f3630892000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)- waiting on 0x00000000a7a92958 (a java.lang.ref.ReferenceQueue$Lock)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)- locked 0x00000000a7a92958 (a java.lang.ref.ReferenceQueue$Lock)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)at sun.java2d.Disposer.run(Disposer.java:148)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneTimer-1 #74 daemon prio5 os_prio0 tid0x00007f35e7ab2000 nid0xb74 in Object.wait() [0x00007f35af408000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)- waiting on 0x00000000a420e9f0 (a java.util.TaskQueue)at java.lang.Object.wait(Object.java:502)at java.util.TimerThread.mainLoop(Timer.java:526)- locked 0x00000000a420e9f0 (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.push.receiver #39 daemon prio5 os_prio0 tid0x00007f35e6d09800 nid0xb51 runnable [0x00007f362adfa000]java.lang.Thread.State: RUNNABLEat java.net.PlainDatagramSocketImpl.receive0(Native Method)- locked 0x00000000a3300670 (a java.net.PlainDatagramSocketImpl)at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)- locked 0x00000000a3300670 (a java.net.PlainDatagramSocketImpl)at java.net.DatagramSocket.receive(DatagramSocket.java:812)- locked 0x00000000a34f2448 (a java.net.DatagramPacket)- locked 0x00000000a3300630 (a java.net.DatagramSocket)at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:83)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000a33001e8 (a java.util.concurrent.ThreadPoolExecutor$Worker)com.alibaba.nacos.naming.failover #38 daemon prio5 os_prio0 tid0x00007f35e6d04800 nid0xb50 waiting on condition [0x00007f362b0fb000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a33029e8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.naming.updater #37 daemon prio5 os_prio0 tid0x00007f35e6d01000 nid0xb4f waiting on condition [0x00007f362b1fc000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a3301ed8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.naming.updater #36 daemon prio5 os_prio0 tid0x00007f35e6cff000 nid0xb4e waiting on condition [0x00007f362b2fd000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a3301ed8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.naming.client.listener #35 daemon prio5 os_prio0 tid0x00007f35e6cf8800 nid0xb4d waiting on condition [0x00007f362b3fe000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a3300c08 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:163)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- 0x00000000a3300918 (a java.util.concurrent.ThreadPoolExecutor$Worker)AsyncAppender-Worker-ConsoleAppender #29 daemon prio5 os_prio0 tid0x00007f35e6059800 nid0xb47 waiting on condition [0x00007f3631f2d000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a20439f8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.Worker.fixed-10.60.52.109_8848-insurance-ats-upgrade-urtrust #26 daemon prio5 os_prio0 tid0x00007f35e6387000 nid0xb44 runnable [0x00007f3630690000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a20a3b98 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- Nonecom.alibaba.nacos.client.config.security.updater #25 daemon prio5 os_prio0 tid0x00007f35e6380800 nid0xb43 waiting on condition [0x00007f3630791000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a20a3090 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)Locked ownable synchronizers:- NoneTimer-0 #23 daemon prio5 os_prio0 tid0x00007f35e634c800 nid0xb41 in Object.wait() [0x00007f3630b93000]java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.util.TimerThread.mainLoop(Timer.java:552)- locked 0x00000000a2856cf0 (a java.util.TaskQueue)at java.util.TimerThread.run(Timer.java:505)Locked ownable synchronizers:- NoneAsyncAppender-Worker-ConsoleAppender #19 daemon prio5 os_prio0 tid0x00007f35e61a3800 nid0xb3d waiting on condition [0x00007f3631296000]java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a1e7a940 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)Locked ownable synchronizers:- NoneGC Daemon #12 daemon prio2 os_prio0 tid0x00007f364c523000 nid0xb36 in Object.wait() [0x00007f36381c9000]java.lang.Thread.State: TIMED_WAITING (on object monitor)at java.lang.Object.wait(Native Method)- waiting on 0x00000000a0f796d8 (a sun.misc.GC$LatencyLock)at sun.misc.GC$Daemon.run(GC.java:117)- locked 0x00000000a0f796d8 (a sun.misc.GC$LatencyLock)Locked ownable synchronizers:- NoneAsyncFileHandlerWriter-1259475182 #11 daemon prio5 os_prio0 tid0x00007f364c1b6000 nid0xb35 waiting on condition [0x00007f3638520000]java.lang.Thread.State: TIMED_WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for 0x00000000a0f796e8 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:160)Locked ownable synchronizers:- NoneService Thread #8 daemon prio9 os_prio0 tid0x00007f364c0cd000 nid0xb33 runnable [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneC1 CompilerThread2 #7 daemon prio9 os_prio0 tid0x00007f364c0c2000 nid0xb32 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneC2 CompilerThread1 #6 daemon prio9 os_prio0 tid0x00007f364c0c0000 nid0xb31 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneC2 CompilerThread0 #5 daemon prio9 os_prio0 tid0x00007f364c0bd000 nid0xb30 waiting on condition [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneSignal Dispatcher #4 daemon prio9 os_prio0 tid0x00007f364c0bb800 nid0xb2f runnable [0x0000000000000000]java.lang.Thread.State: RUNNABLELocked ownable synchronizers:- NoneFinalizer #3 daemon prio8 os_prio0 tid0x00007f364c088800 nid0xb2e in Object.wait() [0x00007f3639351000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)- locked 0x00000000a0f79d30 (a java.lang.ref.ReferenceQueue$Lock)at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)Locked ownable synchronizers:- NoneReference Handler #2 daemon prio10 os_prio0 tid0x00007f364c084000 nid0xb2d in Object.wait() [0x00007f3639452000]java.lang.Thread.State: WAITING (on object monitor)at java.lang.Object.wait(Native Method)at java.lang.Object.wait(Object.java:502)at java.lang.ref.Reference.tryHandlePending(Reference.java:191)- locked 0x00000000a0f79d70 (a java.lang.ref.Reference$Lock)at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)Locked ownable synchronizers:- NoneVM Thread os_prio0 tid0x00007f364c07a000 nid0xb2c runnableGC task thread#0 (ParallelGC) os_prio0 tid0x00007f364c020800 nid0xb28 runnableGC task thread#1 (ParallelGC) os_prio0 tid0x00007f364c022800 nid0xb29 runnableGC task thread#2 (ParallelGC) os_prio0 tid0x00007f364c024800 nid0xb2a runnableGC task thread#3 (ParallelGC) os_prio0 tid0x00007f364c026000 nid0xb2b runnableVM Periodic Task Thread os_prio0 tid0x00007f364c0d2000 nid0xb34 waiting on conditionJNI global references: 3541
2.4、tomcat关闭时的报错信息 tomcat关闭时其实有线程无法关闭的报错信息。但当时认为端口已经释放tomcat程序已经关闭就没有意识到这个问题。 解决方案
手动关停无法关闭的tomcat的进程
./shutdown.sh
ps -ef |grep tomcat
kill -9 pid 排查内存问题时可能用到的其他命令
查看当前内存占用高低排序ps aux --sort -%mem查看内存占用最高的10个ps aux | head -1;ps aux |grep -v PID |sort -rn -k 4 | head -10查看当前内存占用free
参考文档
排查思路https://www.cnblogs.com/zcyy/p/16033677.html
tomcat关闭时nacos线程报错关闭的原因tomcat无法正常关闭问题分析及解决 - 码农教程
tomcat无法正常关闭的原因记一次关于tomcat关闭时清理线程时的警告日志问题排查。_org.apache.catalina.loader.webappclassloaderbase:1-CSDN博客
