🇰🇷 한글 설명
Linux 커널의 Btrfs 파일 시스템에서 btree inode의 dirty metadata 임계값이 cgroup의 dirty limit보다 높게 설정되어 발생하는 데드락 취약점입니다. 이로 인해 프로세스들이 writeback을 기다리며 대기 상태에 빠져 시스템이 멈추고 커널 코어덤프가 발생할 수 있습니다.
💡 쉬운 설명
파일 시스템이 데이터를 저장하려는 기준치와 메모리 관리 시스템이 데이터를 저장하라고 명령하는 기준치가 서로 맞지 않아 발생하는 문제입니다. 이 충돌로 인해 컴퓨터가 아무 작업도 못 하고 멈춰버리는 현상이 발생합니다.
🇺🇸 원문 (English)
In the Linux kernel, the following vulnerability has been resolved:
btrfs: do not strictly require dirty metadata threshold for metadata writepages
[BUG]
There is an internal report that over 1000 processes are
waiting at the io_schedule_timeout() of balance_dirty_pages(), causing
a system hang and trigger a kernel coredump.
The kernel is v6.4 kernel based, but the root problem still applies to
any upstream kernel before v6.18.
[CAUSE]
From Jan Kara for his wisdom on the dirty page balance behavior first.
This cgroup dirty limit was what was actually playing the role here
because the cgroup had only a small amount of memory and so the dirty
limit for it was something like 16MB.
Dirty throttling is responsible for enforcing that nobody can dirty
(significantly) more dirty memory than there's dirty limit. Thus when
a task is dirtying pages it periodically enters into balance_dirty_pages()
and we let it sleep there to slow down the dirtying.
When the system is over dirty limit already (either globally or within
a cgroup of the running task), we will not let the task exit from
balance_dirty_pages() until the number of dirty pages drops below the
limit.
So in this particular case, as I already mentioned, there was a cgroup
with relatively small amount of memory and as a result with dirty limit
set at 16MB. A task from that cgroup has dirtied about 28MB worth of
pages in btrfs btree inode and these were practically the only dirty
pages in that cgroup.
So that means the only way to reduce the dirty pages of that cgroup is
to writeback the dirty pages of btrfs btree inode, and only after that
those processes can exit balance_dirty_pages().
Now back to the btrfs part, btree_writepages() is responsible for
writing back dirty btree inode pages.
The problem here is, there is a btrfs internal threshold that if the
btree inode's dirty bytes are below the 32M threshold, it will not
do any writeback.
This behavior is to batch as much metadata as possible so we won't write
back those tree blocks and then later re-COW them again for another
modification.
This internal 32MiB is higher than the existing dirty page size (28MiB),
meaning no writeback will happen, causing a deadlock between btrfs and
cgroup:
- Btrfs doesn't want to write back btree inode until more dirty pages
- Cgroup/MM doesn't want more dirty pages for btrfs btree inode
Thus any process touching that btree inode is put into sleep until
the number of dirty pages is reduced.
Thanks Jan Kara a lot for the analysis of the root cause.
[ENHANCEMENT]
Since kernel commit b55102826d7d ("btrfs: set AS_KERNEL_FILE on the
btree_inode"), btrfs btree inode pages will only be charged to the root
cgroup which should have a much larger limit than btrfs' 32MiB
threshold.
So it should not affect newer kernels.
But for all current LTS kernels, they are all affected by this problem,
and backporting the whole AS_KERNEL_FILE may not be a good idea.
Even for newer kernels I still think it's a good idea to get
rid of the internal threshold at btree_writepages(), since for most cases
cgroup/MM has a better view of full system memory usage than btrfs' fixed
threshold.
For internal callers using btrfs_btree_balance_dirty() since that
function is already doing internal threshold check, we don't need to
bother them.
But for external callers of btree_writepages(), just respect their
requests and write back whatever they want, ignoring the internal
btrfs threshold to avoid such deadlock on btree inode dirty page
balancing.
📍 영향받는 시스템
Linux kernel v6.4 기반 및 v6.18 미만의 모든 업스트림 커널 (LTS 커널 포함)
⚔️ 공격 벡터
로컬 사용자가 제한된 메모리를 가진 cgroup 내에서 Btrfs btree inode에 대량의 메타데이터를 생성하여 시스템 중단을 유도할 수 있음
🛡️ 대응 방법
커널을 최신 버전으로 업데이트하거나, btree_writepages()의 내부 임계값 체크를 제거한 패치를 적용해야 함
🔧 검증 코드
출력된 커널 버전이 v6.18 미만인지 확인하여 취약한 버전 여부를 판단합니다.