Skip to content

Typeid information not synchronized across shared library boundaries #912

Closed

Description

@wjakob

Dear all,

I've been running into strange inconsistencies where type information is not synchronized across shared library boundaries. This happens when using Clang & libc++ on Linux/x86_64 machines.

Before escalating to Clang/LLVM, I wanted to gauge if anyone here has previously run into this. To reproduce:

issue.py

import mod1, mod2

mod2.test(mod1.Test())

# Expected output
# ================
# $ python issue.py
# $


# Actual output:
# ================
# $ python issue.py
# Traceback (most recent call last):
#   File "issue.py", line 3, in <module>
#     mod2.test(mod1.Test())
# TypeError: test(): incompatible function arguments. The following argument types are supported:
#     1. (arg0: Test) -> None
#
# Invoked with: <mod1.Test object at 0x7f12861f5df8>

CMakeLists.txt:

cmake_minimum_required(VERSION 2.8.12)
project(example)

# Issue is triggered in a release build
set(CMAKE_BUILD_TYPE Release CACHE STRING "" FORCE)

# Issue is triggered when using libc++
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -stdlib=libc++")

add_subdirectory(pybind11)

add_library(shared SHARED shared.cpp)

pybind11_add_module(mod1 mod1.cpp)
pybind11_add_module(mod2 mod2.cpp)

target_link_libraries(mod1 shared)
target_link_libraries(mod2 shared)

shared.h

#pragma once

#define EXPORT __attribute__ ((visibility("default")))

class EXPORT Test {
public:
    Test();
};

shared.cpp

#include "shared.h"

Test::Test() { }

mod1.cpp

#include <pybind11/pybind11.h>
#include "shared.h"

namespace py = pybind11;

PYBIND11_MODULE(mod1, m) {
    py::class_<Test>(m, "Test")
        .def(py::init<>());
}

mod2.cpp

#include <pybind11/pybind11.h>
#include "shared.h"

namespace py = pybind11;

void test(Test b) { }

PYBIND11_MODULE(mod2, m) {
    py::module::import("mod1");
    m.def("test", &test);
}

Best,
Wenzel

Activity

  1. aldanor commented on Jun 21, 2017

    @aldanor
    Member

    Hm, is typeid(Test) not the same in mod1/mod2, can you check?

  2. wjakob commented on Jun 21, 2017

    @wjakob
    MemberAuthor

    Yes, indeed -- the typeids differ in these two modules despite the __attribute__ ((visibility("default"))) annotation. Interestingly, the problem disappears when the -stdlib=libc++ flag is removed from the compilation flags.

  3. jagerman commented on Jun 21, 2017

    @jagerman
    Member

    We do have a few places that do a pointer equality comparison, which seems slightly wrong, but fixing it (PR #914) doesn't fix the issue.

    I think, however, given the response in https://bugs.llvm.org/show_bug.cgi?id=33542:

    I think our stance has been that if you dlopen with RTLD_LOCAL (the default),
    then the "Test" type in mod1.so and mod2.so are different types. They live in
    different symbol namespaces, all the symbols in mod1.so are not visible from
    mod2.so, etc.

    libstdc++ has taken a different stance by moving to string equality for
    std::type_info::operator==.


    Unfortunately, the C++ standard gives us very few tools to reason about how C++
    should behave in the presence of dynamic libraries. The best mental model we
    have is either a DSO is part of the "program" (a collection of translation
    units) or they represent separate "programs" with some implementation-defined
    semantics for interoperating.

    we might have to switch to using the type.name() field to identify types, at least under libc++.

    I'm curious, though, why this doesn't seem to affect macOS.

  4. jagerman commented on Jun 21, 2017

    @jagerman
    Member

    As a workaround, you can call this before loading the pybind modules:

    import os, sys
    sys.setdlopenflags(os.RTLD_NOW | os.RTLD_GLOBAL)
  5. added a commit that references this issue on Jun 21, 2017
    74f8acb
  6. jagerman commented on Jun 21, 2017

    @jagerman
    Member

    @wjakob - give #915 a try in your project: it switches to name-based type lookups outside libstdc++ (which already does name-based equality/hashing). It does fixes the code you posted in the initial report under clang/libc++ for me (tested with clang/libc++ 4.0).

  7. wjakob commented on Jun 22, 2017

    @wjakob
    MemberAuthor

    I just did, and it works with that patch -- thank you! Regarding the OSX inconsistency: it's likely that the dlopen semantics are a bit different on Mach-O binaries.

  8. added a commit that references this issue on Jun 24, 2017
    2196696
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions