[Devel] [PATCH vz10 12/24] oracle/mm: enter COW seqcount/mmu-notifier section in copy_page_range_mt()

[Pavel Tikhomirov]

Reviewed-by: Pavel Tikhomirov <ptikhomirov at virtuozzo.com>

This is really non-obvious, the logic behind "mirroring" is sound, also
we should report this to oracle, I don't see full miroring there, they have
raw_write_seqcount_begin(), but not mmu_notifier_...

On 7/6/26 12:59, Konstantin Khorenko wrote:
> copy_page_range_mt() is the padata-multithreaded copy used by vma_dup()
> to preserve a VM_EXEC_KEEP (MADV_DOEXEC) VMA across exec. A MADV_DOEXEC
> VMA is private anon, so is_cow_mapping() is true and the copy write
> protects the source PTEs - a permission downgrade. Unlike the normal
> fork path copy_page_range(), this helper jumped straight into
> padata_do_multithreaded() without the COW prologue/epilogue, so it:
> 
>   - never entered src_mm->write_protect_seq (left it even), which trips
>     the DEBUG_VM invariant in folio_needs_cow_for_dma() reached from
>     folio_try_dup_anon_rmap_pte() during the copy:
> 
>       kernel BUG at include/linux/mm.h:2054!
>       copy_present_ptes <- copy_pte_range <- copy_page_range_chunk
>         <- padata_do_multithreaded <- copy_page_range_mt <- vma_dup
>         <- exec_mmap <- begin_new_exec <- load_elf_binary
> 
>   - never called mmu_notifier_invalidate_range_start/end(), a real latent
>     bug: a secondary MMU (KVM, IOMMU SVA, GPU) registered on the old mm
>     outlives de_thread() and could keep writable mappings of pages whose
>     PTEs the exec copy just wrprotected and shared into the new mm.
> 
> Mirror copy_page_range(): for a COW mapping, invalidate secondary MMUs
> and take the write_protect_seq write side around the padata job. The
> exec'ing task is single-threaded by now and is the only user of src_mm,
> so there is no concurrent seqcount writer (fork needs mmap_write_lock)
> or GUP-fast reader; taking the raw write side while holding only
> mmap_read_lock is safe, which is why vma_assert_write_locked() from
> copy_page_range() is intentionally not carried over here.
> 
> Fixes: 11e4b1dc50e3 ("oracle/mm: use padata for copying page ranges in vma_dup()")
> Feature: oracle/mm: MADV_DOEXEC madvise() flag
> https://virtuozzo.atlassian.net/browse/VSTOR-137234
> Signed-off-by: Konstantin Khorenko <khorenko at virtuozzo.com>
> ---
>  mm/memory.c | 35 ++++++++++++++++++++++++++++++++++-
>  1 file changed, 34 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/memory.c b/mm/memory.c
> index 837194e87664..4d04591f8f40 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -1478,6 +1478,10 @@ static int copy_page_range_chunk(unsigned long addr,
>  int copy_page_range_mt(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)
>  {
>  	struct copy_page_range_args args = { dst_vma, src_vma };
> +	struct mm_struct *src_mm = src_vma->vm_mm;
> +	struct mmu_notifier_range range;
> +	bool is_cow;
> +	int ret;
>  	struct padata_mt_job job = {
>  		.thread_fn   = copy_page_range_chunk,
>  		.fn_arg      = &args,
> @@ -1490,7 +1494,36 @@ int copy_page_range_mt(struct vm_area_struct *dst_vma, struct vm_area_struct *sr
>  
>  	BUG_ON(!(src_vma->vm_flags & VM_EXEC_KEEP));
>  
> -	return padata_do_multithreaded(&job);
> +	/*
> +	 * Mirror copy_page_range(): for a COW mapping the copy write
> +	 * protects the source PTEs, which is a permission downgrade, so
> +	 * secondary MMUs must be invalidated, and write_protect_seq must be
> +	 * entered (odd) so GUP-fast cannot race with the wrprotect
> +	 * (folio_needs_cow_for_dma() asserts this under CONFIG_DEBUG_VM).
> +	 *
> +	 * The exec'ing task is single-threaded by now and the only user of
> +	 * src_mm, so no concurrent write_protect_seq writer (fork, which
> +	 * needs mmap_write_lock) or GUP-fast reader can exist; taking the
> +	 * raw write side while holding only mmap_read_lock is safe here, and
> +	 * the padata helper threads only read the seqcount.
> +	 */
> +	is_cow = is_cow_mapping(src_vma->vm_flags);
> +	if (is_cow) {
> +		mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE,
> +					0, src_mm, src_vma->vm_start,
> +					src_vma->vm_end);
> +		mmu_notifier_invalidate_range_start(&range);
> +		raw_write_seqcount_begin(&src_mm->write_protect_seq);
> +	}
> +
> +	ret = padata_do_multithreaded(&job);
> +
> +	if (is_cow) {
> +		raw_write_seqcount_end(&src_mm->write_protect_seq);
> +		mmu_notifier_invalidate_range_end(&range);
> +	}
> +
> +	return ret;
>  }
>  #endif /* CONFIG_PADATA */
>  

-- 
Best regards, Pavel Tikhomirov
Senior Software Developer, Virtuozzo.