[CRIU] dlopen() and inodes

[Dmitry Safonov]

Hi Nicolas,

On 12/11/19 8:04 PM, Nicolas Viennot wrote:
> Hello,
> 
> When loading a shared library via a call to dlopen(libpath), glibc and musl compare
> the pair device+inode of the file opened at libpath with the one of previously opened libraries.
> If the newly opened library is a duplicate of another already loaded library from this comparison,
> the old library is instead used. This ensures that a library accessible via multiple paths does
> not get loaded twice. Typically the paths are soft links, and rarely hard links.
> 
> We checkpoint and restore using CRIU on Kubernetes. Even though we use the same
> container image when restoring applications, we find different device+inodes for the
> same files compared to the checkpoint environment, leading to issues with dlopen().

Ugh, that sucks.

> We thought of patching glibc to either follow library soft links to canonicalize
> paths,

That sounds like the most easy solution. Though, `git grep ino_t` in
Glibc suggests that there are quite a few places where the logic is
based on inode numbers. Also, patches to musl & ulibc would be needed
too, to make C/R work with them..

> or use the DT_SONAME field in the ELF header to uniquely identify a library.
> We also thought of interposing fstat() to virtualize devices and inodes, but this
> inevitably adds some execution overhead.

I don't think any logic that gets inodes is on the fast-path, so it
should be fine. What is very hard in this approach is to convince kernel
folks that inodes virtualisation is needed.

> Has anyone encountered this issue? Any suggestions? 

I don't think anyone has worked on this issue yet.
You could look into eBPF to "patch" the returned inode number - that
might be the way to go. AFAIR, there isn't any syscall that takes inode
as a parameter, so lying to userspace after C/R may just work..

Thanks,
          Dmitry