seo中文全称是什么,360搜索怎么做网站自然优化,500万网官网,网站后台上传图片显示运行错误为什么前言这篇文章是一位大神在实际项目中遇到问题并分析总结出来的#xff0c;作为新手#xff0c;能接触到这类文章应该是受益匪浅#xff0c;这位同学现在在魅族工作#xff0c;以后也会一直在魅族工作#xff0c;是Linux 方面的专家#xff0c;「魅族还有另一个Linux 大神… 前言这篇文章是一位大神在实际项目中遇到问题并分析总结出来的作为新手能接触到这类文章应该是受益匪浅这位同学现在在魅族工作以后也会一直在魅族工作是Linux 方面的专家「魅族还有另一个Linux 大神知道的自然知道了」不知道大家发现没有最近文章底部的小guanggao被我撤掉了我就是想让大家看文章的时候能有一种家的感觉如果觉得这位同学文章不错的支持一下支持方式不做限制好了不吹牛了看下面的正文。Description一个项目中偶现几十上百个 D 进程卡住在 shrink_inactive_list导致卡顿/卡死/android SWT 等问题前前后后提交了 3 次修复还没有彻底解决。山重水复疑无路LOG [149459.897408] [3:2065:watchdog] Binder:1042_16 D 0 9917 635 0x00000008 [149459.897427] [3:2065:watchdog] Call trace: [149459.897435] [3:2065:watchdog] [ffffff8bf28852d4] _switch_to0xb4/0xc0 [149459.897452] [3:2065:watchdog] [ffffff8bf3a1f6a0] _schedule0x7f0/0xad0 [149459.897468] [3:2065:watchdog] [ffffff8bf3a1f9f0] schedule0x70/0x90 [149459.897485] [3:2065:watchdog] [ffffff8bf3a23b00] schedule_timeout0x548/0x668 [149459.897502] [3:2065:watchdog] [ffffff8bf2959028] msleep0x28/0x38 [149459.897517] [3:2065:watchdog] [ffffff8bf2a1ff38] shrink_inactive_list0x118/0x998 [149459.897534] [3:2065:watchdog] [ffffff8bf2a1cb10] shrink_node_memcg0xa18/0x1100 [149459.897552] [3:2065:watchdog] [ffffff8bf2a1f0b0] shrink_node0x108/0x2f8 [149459.897568] [3:2065:watchdog] [ffffff8bf2a1bcb0] do_try_to_free_pages0x178/0x380 [149459.897586] [3:2065:watchdog] [ffffff8bf2a1b9d0] try_to_free_pages0x370/0x4d8 [149459.897605] [3:2065:watchdog] [ffffff8bf2a071b8] _alloc_pages_nodemask0x868/0x1380 [149459.897623] [3:2065:watchdog] [ffffff8bf2a13784] __do_pagecache_readahead0xbc/0x358 [149459.897640] [3:2065:watchdog] [ffffff8bf29fde4c] filemapfault0x11c/0x600 [149459.897647] [3:2065:watchdog] [ffffff8bf2b479f8] ext4_filemap_fault0x30/0x50 [149459.897664] [3:2065:watchdog] [ffffff8bf2a47f38] handle_pte_fault0xb38/0xfa8 [149459.897681] [3:2065:watchdog] [ffffff8bf2a485c8] handle_mm_fault0x1d0/0x328 [149459.897699] [3:2065:watchdog] [ffffff8bf28a3668] do_page_fault0x2a0/0x3e0 [149459.897716] [3:2065:watchdog] [ffffff8bf28a3364] do_translation_fault0x44/0xa8 [149459.897732] [3:2065:watchdog] [ffffff8bf2880b74] do_mem_abort0x4c/0xd0 [149459.897750] [3:2065:watchdog] [ffffff8bf2882c78] el0_da0x20/0x24 [149459.897767] [3:2065:watchdog] Binder:1042_19 D 0 11188 635 0x00000008 [149459.897786] [3:2065:watchdog] Call trace: [149459.897797] [3:2065:watchdog] [ffffff8bf28852d4] _switch_to0xb4/0xc0 [149459.897804] [3:2065:watchdog] [ffffff8bf3a1f6a0] _schedule0x7f0/0xad0 [149459.897820] [3:2065:watchdog] [ffffff8bf3a1f9f0] schedule0x70/0x90 [149459.897835] [3:2065:watchdog] [ffffff8bf3a23b00] schedule_timeout0x548/0x668 [149459.897853] [3:2065:watchdog] [ffffff8bf2959028] msleep0x28/0x38 [149459.897868] [3:2065:watchdog] [ffffff8bf2a1ff38] shrink_inactive_list0x118/0x998 [149459.897887] [3:2065:watchdog] [ffffff8bf2a1cb10] shrink_node_memcg0xa18/0x1100 [149459.897904] [3:2065:watchdog] [ffffff8bf2a1f0b0] shrink_node0x108/0x2f8 [149459.897922] [3:2065:watchdog] [ffffff8bf2a1bcb0] do_try_to_free_pages0x178/0x380 [149459.897940] [3:2065:watchdog] [ffffff8bf2a1b9d0] try_to_free_pages0x370/0x4d8 [149459.897957] [3:2065:watchdog] [ffffff8bf2a071b8] __alloc_pages_nodemask0x868/0x1380 [149459.897977] [3:2065:watchdog] [ffffff8bf2a13784] _do_page_cache_readahead0xbc/0x358 [149459.897996] [3:2065:watchdog] [ffffff8bf29fde4c] filemap_fault0x11c/0x600 [149459.898013] [3:2065:watchdog] [ffffff8bf2b479f8] ext4_filemap_fault0x30/0x50 [149459.898031] [3:2065:watchdog] [ffffff8bf2a47f38] handle_pte_fault0xb38/0xfa8 [149459.898048] [3:2065:watchdog] [ffffff8bf2a485c8] handle_mm_fault0x1d0/0x328 [149459.898065] [3:2065:watchdog] [ffffff8bf28a3668] do_page_fault0x2a0/0x3e0 [149459.898083] [3:2065:watchdog] [ffffff8bf28a3364] do_translation_fault0x44/0xa8 [149459.898100] [3:2065:watchdog] [ffffff8bf2880d18] do_el0_ia_bp_hardening0xc0/0x158 [149459.898118] [3:2065:watchdog] [ffffff8bf2882c98] el0_ia0x1c/0x20现象大量进程从缺页异常入口调用内存回收接口shrink_inactive_list - msleep 使得该进程状态变为 D.void msleep(unsigned int msecs)
{ unsigned long timeout msecs_to_jiffies(msecs) 1; while (timeout) timeout schedule_timeout_uninterruptible(timeout);
}signed long __sched schedule_timeout_uninterruptible(signed long timeout)
{ __set_current_state(TASK_UNINTERRUPTIBLE); return schedule_timeout(timeout);
}D 进程就是被设置了 TASK_UNINTERRUPTIBLE 进程状态不可中断的睡眠状态。不可中断指的并不是 CPU 不响应外部硬件的中断而是指进程不响应异步信号信号只会挂到信号队列而没有机会去立即执行。它不占用 CPU 也不能被杀掉很直观的现象就是kill -9 一个 D 进程是没有效果的只有等进程获得资源被唤醒才处理信号才处理 SIGKILL。「进程是很有脾气的不知道你们有没有遇到那种钻牛角尖的人拿我儿子来举例一下有时候他想找到他的玩具火箭就一直在那里闹一定要我们帮他找到他的玩具火箭位置其他事情就是不干你用坦克哄他也不行哄他看小猪佩奇也不行。D进程也是一样必须要等有他等到的那个事件为止」 static noinline_for_stack unsigned long shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec, struct scan_control *sc, enum lru_list lru) {
...... while (unlikely(too_many_isolated(pgdat, file, sc, stalled))) { if (stalled) return 0; /* wait a bit for the reclaimer. */ msleep(100); // 卡在这里 stalled true; /* We are about to die and free our memory. Return now. */ if (fatal_signal_pending(current)) return SWAP_CLUSTER_MAX; }
......初步定位该函数已经有跳出功能不会一直卡住最多 2 次就会退出去。说明是大量的进程疯狂地调用 shrink_inactive_list 又被阻塞了一下子又退出去又掉进来。所以不是一直卡死而是性能瓶颈拥堵在这个地方congestion 「拥挤堵车的意思」。从上层 systrace 也能看到很有规律的大概 110ms 一段的 D 状态一个进程甚至可以持续几十秒。说明隔离页面过多sleep 100ms猜测目的是给时间处理隔离页面回写文件页到磁盘 是控制并发也许另一个 cpu 也在同样的回收流程导致隔离页在时刻变大。所以初步定了两个方向和疑点一是内存回收瓶颈内存回收不及时内存需求量巨大而 LMK 没触发内存有很多匿名页都在回收和回写文件页等。二是 io 读写瓶颈io 速率慢某个时间段速率变慢ufs 频率低上层读写大量数据io 占用率过高等。需要澄清这些疑点。插播一些背景知识page cache导致这个情况的原因是进程在申请内存的时候发现该 zone 的 freelist 上已经没有足够的内存可用所以不得不去从该 zone 的 LRU 链表里回收 inactive 的page这种情况就是 direct reclaim直接回收。direct reclaim 会比较消耗时间的原因是如果回收的是 dirty page就会触发磁盘 IO 的操作它会首先把 dirty page 里面的内容给回写到磁盘作同步再去把该 page 给放到 freelist 里。下图来看下 memorypage cacheDisk I/O 的关系。举个简单的例子比如我们 open 一个文件时如果没有使用 O_DIRECT 这个flag那就是 File I/O, 所有对磁盘文件的访问都要经过内存内存会把这部分数据给缓存起来但是如果使用了 O_DIRECT 这个flag那就是 Direct I/O, 它会绕过内存而去直接访问磁盘访问的这部分数据也不会被缓存起来自然性能上会降低很多。page reclaim在直观上我们有一个认知我们现在读了一个文件它会被缓存到内存里面如果接下来的一个月我们一直都不会再次访问它而且我们这一个月都不会关闭或者重启机器那么在这一个月之后该文件就不应该再在内存里头了。这就是内核对 page cache 的管理策略LRU最近最少使用。即把最近最少使用的 page cache 给回收为 free pages。页框回收算法 PFRA 远没有这么简单内核的页回收机制有两种后台周期性回收和直接回收。后台回收是有一个内核线程 kswapd 来做当内存里 free 的 pages 低于一个水位page_low时就会唤醒该内核线程然后它从 LRU 链表里回收 page cache 到内存的 free_list 里头它会一直回收直至 free 的 pages 达到另外一个水位 page_high 才停止. 如下图所示直接回收则是在发生 page fault/alloc memory 时没有足够可用的内存于是线程就自己直接去回收内存它一次性的会回收 32 个 pages。逻辑过程如下图所示所以在内存优化上1、抬高 watermark 可以间接减少内存回收的并发量减轻卡在 shrink_inactive_list. 2、提高回收效率如 LMK 的效率。然而还是没彻底解决这个问题所以我们把疑点再次指向 io。尝试抓取更多的信息来了解触发瓶颈的微观过程。1、跑 monkey 增加 io 使用率、io 读写速度监控以时间片为 100ms监控连续 D 状态并收集 D 进程堆栈信息、内存信息等。2、打开 ftarce 的 vmscan 和 writeback 两个监控点apk 监控到持续 D 状态就进dump从 dump 解析 ftrace再使用 kernelshark 来观察一些数据。echo 1 /sys/kernel/debug/tracing/events/writeback/enable
echo 1 /sys/kernel/debug/tracing/events/vmscan/enable
echo 1 /sys/kernel/debug/tracing/tracing_on为了准备再深入上述的微观过程需要再补充一些代码和流程图注释的代码不贴了受微信公众号篇幅限制。ftrace kernelshark 辅助分析执行页面回收中页面状态ftrace 会抓取下面这些信息统计所以提前了解下。struct reclaim_stat { unsigned nr_dirty;// page_list中脏页数 unsigned nr_unqueued_dirty;// page_list中脏页但是没有放入块设备请求队列中的页数 unsigned nr_congested;// page_list中阻塞的页数 unsigned nr_writeback; // page_list中处于回写中但是不是被回收的页数 unsigned nr_immediate; //page_list中即回写中而且即将被回收的页数 unsigned nr_activate;// page_list中近期被访问过需要添加到 activate list 的页数 unsigned nr_ref_keep;// page_list中近期被访问过的页数 unsigned nr_unmap_fail;//解除映射失败的页数
}; 经过一段时间的老化测试测试同学终于抓到 log 了。图中显示 nr_dirtynr_congestednr_writeback 几乎都是 0只有零星 nr_activate 被再访问的页面要添加回 active list.说明现场不存在 dirty 页面很多回写 io 遇到瓶颈的情况。这个猜想不成立了。图中显示在 34 秒内所有在 pageout() 中的页面全是 anon 页面没有 file ?查看 writeback trace event。同样没有很多 writeback 量从测试结果看到1.apk 监控到的 io 使用率不高2.从 ftrace 看到回写量不大通过最新的数据信息回到之前的两个大方向•一是内存紧缺内存回收不及时内存需求量大。LMK 没触发内存有很多匿名页都在回收和回写文件页等。抬高水位、加速 LMK 触发还有复现不能彻底解决•二是 io 速率慢某个时间段速率变慢ufs 频率低上层读写大量数据io 占用率过高等。数据证明io 量不多没有瓶颈那么之前的两个方向猜想都落空了。那会是什么意想不到的原因那回去看看卡住的代码too_many_isolated 代码。 static int __too_many_isolated(struct pglist_data *pgdat, int file, struct scan_control *sc, bool stalled) { unsigned long inactive, isolated; if (file) { if (stalled) { inactive node_page_state_snapshot(pgdat, NR_INACTIVE_FILE); isolated node_page_state_snapshot(pgdat, NR_ISOLATED_FILE); } else { inactive node_page_state(pgdat, NR_INACTIVE_FILE); isolated node_page_state(pgdat, NR_ISOLATED_FILE); } } else { if (stalled) { inactive node_page_state_snapshot(pgdat, NR_INACTIVE_ANON); isolated node_page_state_snapshot(pgdat, NR_ISOLATED_ANON); } else { inactive node_page_state(pgdat, NR_INACTIVE_ANON); isolated node_page_state(pgdat, NR_ISOLATED_ANON); } } /* * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so they * wont get blocked by normal direct-reclaimers, forming a circular * deadlock. */ if ((sc-gfp_mask (__GFP_IO | __GFP_FS)) (__GFP_IO | __GFP_FS)) inactive 3; return isolated inactive; }没有很复杂的逻辑只有简单的 isolated 和 inactive 统计计数比较。所以只能是更直接的猜想isolated file 统计一直偏大导致一直判断 too_many_isolated 为真卡在 shrink_inactive_list。根据这个猜想从 log 中打印的 mem info也看到 isolated file 一直偏大一直在增加不会减少。好像印证了猜想似的。LOG6[95299.607369] isolated(anon):0kB isolated(file):37880kB
6[95318.568833] isolated(anon):0kB isolated(file):37752kB
6[95323.773350] isolated(anon):0kB isolated(file):37752kB
6[97520.184804] isolated(anon):0kB isolated(file):44604kB
6[97525.658037] isolated(anon):0kB isolated(file):44604kB
6[97754.256431] isolated(anon):0kB isolated(file):44604kB
6[97759.418172] isolated(anon):0kB isolated(file):44604kB
6[97764.574908] isolated(anon):0kB isolated(file):44604kB
6[97769.735128] isolated(anon):0kB isolated(file):44604kB
6[98543.638667] isolated(anon):0kB isolated(file):44684kB
6[98548.905397] isolated(anon):0kB isolated(file):44684kB
6[98554.209671] isolated(anon):0kB isolated(file):44684kB
6[99996.798031] isolated(anon):0kB isolated(file):51572kB
6[100002.122853] isolated(anon):0kB isolated(file):51572kB
6[100007.359023] isolated(anon):0kB isolated(file):51572kB
6[100146.079882] isolated(anon):0kB isolated(file):51700kB
6[100151.313065] isolated(anon):0kB isolated(file):51572kB
6[100156.587622] isolated(anon):0kB isolated(file):51572kB
6[100328.483071] isolated(anon):0kB isolated(file):51700kB
6[100520.245217] isolated(anon):0kB isolated(file):51572kB
6[100550.688429] isolated(anon):0kB isolated(file):51572kB
6[100555.913634] isolated(anon):0kB isolated(file):51572kB
6[100669.226582] isolated(anon):0kB isolated(file):51572kB
6[100935.069661] isolated(anon):0kB isolated(file):51688kB
6[100940.240279] isolated(anon):0kB isolated(file):51572kB
6[100945.476071] isolated(anon):0kB isolated(file):51828kB
6[103104.120921] isolated(anon):0kB isolated(file):53344kB
6[103121.900214] isolated(anon):0kB isolated(file):53344kB
6[103481.197823] isolated(anon):0kB isolated(file):53412kB
6[103486.555528] isolated(anon):0kB isolated(file):53412kB
6[103721.346234] isolated(anon):0kB isolated(file):53412kB
6[103726.655700] isolated(anon):0kB isolated(file):53540kB
6[103731.961321] isolated(anon):0kB isolated(file):53540kB
6[103737.236295] isolated(anon):0kB isolated(file):53540kB
6[103742.470632] isolated(anon):0kB isolated(file):53412kB
6[103747.661019] isolated(anon):0kB isolated(file):53284kB
6[103752.973978] isolated(anon):0kB isolated(file):53412kB柳暗花明又一村对 NR_ISOLATED_FILE/NR_ISOLATED_ANON 的统计增减主要分布在 vmscan.c migrate.c和 PPR (高通进程内存回收)模块。理论上内核 vmscan.c成双成对 migrate.c 都不会有问题高通 PPR 模块插入在 vmscan. c 和 task_mmu.c 里而我们 IMS 没有直接使用高通 PPR嫌疑最大。于是在上游确实找到了个相关的 patch。https://source.codeaurora.org/quic/la/kernel/msm-4.14/commit/fs/proc/task_mmu.c?hmsm-4.14idc800548eac0350391c6d379a89f2e5d4c31366bf这个 patch 正是修复了 isolated count mismatch 的问题导致一直让 isolated file 增大。 MADV_FREE clears pte dirty bit and then marks the page lazyfree (clear SwapBacked). PPR increments ISOLATE_FILES count, then isolates page and invokes a reclaim. Inbetween if this lazyfreed page is touched by user then it becomes dirty. PPR in shrink_page_list in try_to_unmap finds the page dirty, marks it back as PageSwapBacked and skips reclaim. As PageSwapBacked set, PPR identifies the page as anon and decrements ISOLATED_ANON, thus creating isolated count mismatch. This results in too_many_isolated() check causing delay in reclaim. Skip reclaiming lazyfreed pages in PPR path. MADV_FREE (since Linux 4.5) The application no longer requires the pages in the range specified by addr and len. The kernel can thus free these pages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. After a successful MADV_FREE operation, any stale data (i.e., dirty, unwritten pages) will be lost when the kernel frees the pages. However, subsequent writes to pages in the range will succeed and then kernel cannot free those dirtied pages, so that the caller can always see just written data. If there is no subsequent write, the kernel can free the pages at any time. Once pages in the range have been freed, the caller will see zero-fill-on-demand pages upon subsequent page references. The MADV_FREE operation can be applied only to private anonymous pages (see mmap(2)). In Linux before version 4.12, when freeing pages on a swapless system, the pages in the given range are freed instantly, regardless of memory pressure.madvise(2) is a system call used by processes to tell the kernel how they are going to use their memory, allowing the kernel to optimize the memory management according to these hints to achieve better overall performance. When an application wants to signal the kernel that it isnt going to use a range of memory in the near future, it can use the MADV_DONTNEED flag, so the kernel can free resources associated with it. Subsequent accesses in the range will succeed, but will result either in reloading of the memory contents from the underlying mapped file or zero-fill-on-demand pages for mappings without an underlying file. But there are some kind of apps (notably, memory allocators) that can reuse that memory range after a short time, and MADV_DONTNEED forces them to incur in page fault, page allocation, page zeroing, etc. For avoiding that overhead, other OS like BSDs have supported MADV_FREE, which just mark pages as available to free if needed, but it doesnt free them immediately, making possible to reuse the memory range without incurring in the costs of faulting the pages again. This release adds Linux support for this flag. Recommended LWN article: Volatile ranges and MADV_FREEmadvise[1] 系统调用会建议内核在从 addr 指定的地址开始长度等于 len 参数值的范围内该区域的用户虚拟内存应遵循特定的使用模式使内核可以选择适当的预读和缓存技术。如果使用 madvise() 函数的程序明确了解其内存访问模式则使用此函数可以提高系统性能。自 4.5 开始引入 MADV_FREE 参数「这是为什么 4.9 内核才出现该问题这需要上层和底层同时支持才会出现本问题」。简单来说MADV_FREE 就是让上层设置一段内存可以释放内存的标志但是底层并不会立即释放以便让上层可以在短时间内重复访问以免增加缺页异常等性能开销。也叫 lazy free它只能用于匿名页面。根据描述触发 isolated file 统计增大的路径是「代码省略不贴」•上层调用 madvise 系统调用使用 MADV_FREE 时清除 dirty bit 和 SwapBacked bit把 lazyfree page 加入 inactive file list。•PPR 增加 ISOLATE_FILES 计数SwapBacked0隔离页面并触发回收•上层访问 lazyfreed 页面dirty1•PPR 执行 reclaim_pte_range - reclaim_pages_from_list - shrink_page_list -try_to_unmap - try_to_unmap_one 设置 SwapBacked1, 并跳出回收•PPR 继续执行 reclaim_pte_range - reclaim_pages_from_listputback_lru_page 的时候因为 SwapBacked1减少了 NR_ISOLATED_ANON 计数而不是减少当初增加的 NR_ISOLATED_FILE 计数。•导致 NR_ISOLATED_FILE 一直被增加所以需要在 PPR 中过滤 lazyfree 页面避免这个 NR_ISOLATED_FILE 计数异常导致的卡 too_many_isolated。匿名页面一开始就会设置 SwapBacked1, 并且只有在上层设置 lazyfree 页面时才会清除 ClearPageSwapBacked(page) 没别的地方了。所以PageAnon(page) !PageSwapBacked(page) 能指示这是 lazyfree 页面。ok已经理清了前因后果。再退一步试想下假如上游没有修复这个 patch。我们能不能想出来我觉得很难因为我们缺乏 madvise 的相关认识并且它经过了 dirty, SwapBacked 标志的变化好像几乎没办法做这么微观的页面标志追踪才导致 NR_ISOLATED_ANON/FLIE 的变化。请作者吃根辣条References[1] madvise: http://www.man7.org/linux/man-pages/man2/madvise.2.html扫码或长按关注回复「加群 」进入技术群聊
