🇰🇷 한글 설명
Linux 커널의 netfs 구성 요소에서 참조 카운트 누수(Reference Leak) 취약점이 발견되었습니다. I/O 작업이 제출되기 전에 요청이 해제될 경우 참조 카운터가 제대로 감소하지 않아, netfs_wait_for_outstanding_io() 함수에서 작업이 차단되고 Ceph와 같은 시스템에서 데드락(Deadlock)을 유발할 수 있습니다.
💡 쉬운 설명
컴퓨터가 데이터를 읽고 쓰는 과정에서 사용한 메모리 자원을 제대로 반환하지 않고 계속 붙잡고 있는 문제입니다. 이로 인해 시스템이 멈추거나 서버 전체가 응답하지 않는 먹통 상태가 될 수 있습니다.
🇺🇸 원문 (English)
In the Linux kernel, the following vulnerability has been resolved:
netfs: fix reference leak
Commit 20d72b00ca81 ("netfs: Fix the request's work item to not
require a ref") modified netfs_alloc_request() to initialize the
reference counter to 2 instead of 1. The rationale was that the
requet's "work" would release the second reference after completion
(via netfs_{read,write}_collection_worker()). That works most of the
time if all goes well.
However, it leaks this additional reference if the request is released
before the I/O operation has been submitted: the error code path only
decrements the reference counter once and the work item will never be
queued because there will never be a completion.
This has caused outages of our whole server cluster today because
tasks were blocked in netfs_wait_for_outstanding_io(), leading to
deadlocks in Ceph (another bug that I will address soon in another
patch). This was caused by a netfs_pgpriv2_begin_copy_to_cache() call
which failed in fscache_begin_write_operation(). The leaked
netfs_io_request was never completed, leaving `netfs_inode.io_count`
with a positive value forever.
All of this is super-fragile code. Finding out which code paths will
lead to an eventual completion and which do not is hard to see:
- Some functions like netfs_create_write_req() allocate a request, but
will never submit any I/O.
- netfs_unbuffered_read_iter_locked() calls netfs_unbuffered_read()
and then netfs_put_request(); however, netfs_unbuffered_read() can
also fail early before submitting the I/O request, therefore another
netfs_put_request() call must be added there.
A rule of thumb is that functions that return a `netfs_io_request` do
not submit I/O, and all of their callers must be checked.
For my taste, the whole netfs code needs an overhaul to make reference
counting easier to understand and less fragile & obscure. But to fix
this bug here and now and produce a patch that is adequate for a
stable backport, I tried a minimal approach that quickly frees the
request object upon early failure.
I decided against adding a second netfs_put_request() each time
because that would cause code duplication which obscures the code
further. Instead, I added the function netfs_put_failed_request()
which frees such a failed request synchronously under the assumption
that the reference count is exactly 2 (as initially set by
netfs_alloc_request() and never touched), verified by a
WARN_ON_ONCE(). It then deinitializes the request object (without
going through the "cleanup_work" indirection) and frees the allocation
(with RCU protection to protect against concurrent access by
netfs_requests_seq_start()).
All code paths that fail early have been changed to call
netfs_put_failed_request() instead of netfs_put_request().
Additionally, I have added a netfs_put_request() call to
netfs_unbuffered_read() as explained above because the
netfs_put_failed_request() approach does not work there.
📍 영향받는 시스템
Linux Kernel (netfs 구성 요소가 포함된 버전)
⚔️ 공격 벡터
직접적인 외부 공격보다는 특정 I/O 오류 상황을 유도하여 시스템 자원을 고갈시키고 데드락을 유발하는 방식
🛡️ 대응 방법
커널 패치(Commit 20d72b00ca81)를 적용하여 netfs_put_failed_request() 함수를 통해 실패한 요청을 즉시 해제하도록 업데이트해야 합니다.
🔧 검증 코드
현재 실행 중인 커널 버전을 확인하여 해당 취약점이 수정된 패치 버전이 적용되었는지 확인합니다.