[Devel] [PATCH vz10 12/24] oracle/mm: enter COW seqcount/mmu-notifier section in copy_page_range_mt()

[Konstantin Khorenko]

copy_page_range_mt() is the padata-multithreaded copy used by vma_dup()
to preserve a VM_EXEC_KEEP (MADV_DOEXEC) VMA across exec. A MADV_DOEXEC
VMA is private anon, so is_cow_mapping() is true and the copy write
protects the source PTEs - a permission downgrade. Unlike the normal
fork path copy_page_range(), this helper jumped straight into
padata_do_multithreaded() without the COW prologue/epilogue, so it:

  - never entered src_mm->write_protect_seq (left it even), which trips
    the DEBUG_VM invariant in folio_needs_cow_for_dma() reached from
    folio_try_dup_anon_rmap_pte() during the copy:

      kernel BUG at include/linux/mm.h:2054!
      copy_present_ptes <- copy_pte_range <- copy_page_range_chunk
        <- padata_do_multithreaded <- copy_page_range_mt <- vma_dup
        <- exec_mmap <- begin_new_exec <- load_elf_binary

  - never called mmu_notifier_invalidate_range_start/end(), a real latent
    bug: a secondary MMU (KVM, IOMMU SVA, GPU) registered on the old mm
    outlives de_thread() and could keep writable mappings of pages whose
    PTEs the exec copy just wrprotected and shared into the new mm.

Mirror copy_page_range(): for a COW mapping, invalidate secondary MMUs
and take the write_protect_seq write side around the padata job. The
exec'ing task is single-threaded by now and is the only user of src_mm,
so there is no concurrent seqcount writer (fork needs mmap_write_lock)
or GUP-fast reader; taking the raw write side while holding only
mmap_read_lock is safe, which is why vma_assert_write_locked() from
copy_page_range() is intentionally not carried over here.

Fixes: 11e4b1dc50e3 ("oracle/mm: use padata for copying page ranges in vma_dup()")
Feature: oracle/mm: MADV_DOEXEC madvise() flag
https://virtuozzo.atlassian.net/browse/VSTOR-137234
Signed-off-by: Konstantin Khorenko <khorenko at virtuozzo.com>
---
 mm/memory.c | 35 ++++++++++++++++++++++++++++++++++-
 1 file changed, 34 insertions(+), 1 deletion(-)

diff --git a/mm/memory.c b/mm/memory.c
index 837194e87664..4d04591f8f40 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1478,6 +1478,10 @@ static int copy_page_range_chunk(unsigned long addr,
 int copy_page_range_mt(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
 {
 	struct copy_page_range_args args = { dst_vma, src_vma };
+	struct mm_struct *src_mm = src_vma->vm_mm;
+	struct mmu_notifier_range range;
+	bool is_cow;
+	int ret;
 	struct padata_mt_job job = {
 		.thread_fn   = copy_page_range_chunk,
 		.fn_arg      = &args,
@@ -1490,7 +1494,36 @@ int copy_page_range_mt(struct vm_area_struct *dst_vma, struct vm_area_struct *sr
 
 	BUG_ON(!(src_vma->vm_flags & VM_EXEC_KEEP));
 
-	return padata_do_multithreaded(&job);
+	/*
+	 * Mirror copy_page_range(): for a COW mapping the copy write
+	 * protects the source PTEs, which is a permission downgrade, so
+	 * secondary MMUs must be invalidated, and write_protect_seq must be
+	 * entered (odd) so GUP-fast cannot race with the wrprotect
+	 * (folio_needs_cow_for_dma() asserts this under CONFIG_DEBUG_VM).
+	 *
+	 * The exec'ing task is single-threaded by now and the only user of
+	 * src_mm, so no concurrent write_protect_seq writer (fork, which
+	 * needs mmap_write_lock) or GUP-fast reader can exist; taking the
+	 * raw write side while holding only mmap_read_lock is safe here, and
+	 * the padata helper threads only read the seqcount.
+	 */
+	is_cow = is_cow_mapping(src_vma->vm_flags);
+	if (is_cow) {
+		mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE,
+					0, src_mm, src_vma->vm_start,
+					src_vma->vm_end);
+		mmu_notifier_invalidate_range_start(&range);
+		raw_write_seqcount_begin(&src_mm->write_protect_seq);
+	}
+
+	ret = padata_do_multithreaded(&job);
+
+	if (is_cow) {
+		raw_write_seqcount_end(&src_mm->write_protect_seq);
+		mmu_notifier_invalidate_range_end(&range);
+	}
+
+	return ret;
 }
 #endif /* CONFIG_PADATA */
 
-- 
2.47.1