[C++-sig] Memory management in Boost.Python

[Ernie Lee]

Hello,


I am working on adding Python binding to C++ project and run into the
following problem: For some reason Python tries to delete objects that are
still held by smart pointer instances in our C++ code.


We are using std::shared_ptr for memory management and classes exposed
through Boost::Python have HeldType set to std::shared_ptr<
>. This problem
manifests itself in the following scenarios:


- in Python i subclass C++ class, which acts as ‘callback’, and create an
instance of it and pass it to C++ code, where it is held in std::shared_ptr
instance. Python script itself does not use this callback object after that
point. When Python script terminates i receive memory error, which seems to
be triggered by attempting to double-free this callback object.


- in Python i subclass C++ class, create an instance of it (call it ‘A')
and store it in container-like object (call it ‘B') (B is defined in C++
and exposed to Python). B stores shared_ptr to A. As soon as Python
execution point leaves current scope my object A gets deleted while B still
holds shared_ptr to A.


It looks to me that ether i have a bug in my wrapping code (any idea what
this could be?) or in some cases Python disregards HeldType and tries to
determine object lifespan by its own means (in both cases from Python point
of view created object is no longer in scope so i am guessing thats why it
tries to delete it). If the latter is correct, any idea how to force Python
to use smart pointer class specified in HeldType?


Thank you,

Ernie.

[Stefan Seefeld]

Ernie,

it appears what you are seeing is the fact that boost.python only
supports boost::shared_ptr, but not std::shared_ptr. (See
https://svn.boost.org/trac/boost/ticket/6545). Is it possible for you to
switch to Boost's shared_ptr, at least for the Python bindings ? Yes, we
need to fix this urgently... :-(

Stefan

[Ernie Lee]

Thank you for looking this up Stefan! It turned out that we actually
using boost::shared_ptr but in rather non-straight-forward way: it got pull
in to our own namespace so in the end Boost::Python::class_ is 'called'
with held-type set to a formally different type name. I will see if setting
held-type to boost::shared_ptr fix this.

Best,

Ernie.

Post by Stefan Seefeld
Ernie,
it appears what you are seeing is the fact that boost.python only
supports boost::shared_ptr, but not std::shared_ptr. (See
https://svn.boost.org/trac/boost/ticket/6545). Is it possible for you to
switch to Boost's shared_ptr, at least for the Python bindings ? Yes, we
need to fix this urgently... :-(
Stefan
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig

[Ernie Lee]

Hi Stefan,

I updated my code so it now use 'boost::shared_ptr' class while
specifying the held-type and i got exactly the same errors (i guess boost
did recognize class even in different namespace).

Any other theories of what could be wrong? Could it be that Python in
some cases tries to manage memory directly, disregarding SP layer?

Thanks,

Ernie.

Post by Ernie Lee
Thank you for looking this up Stefan! It turned out that we actually
using boost::shared_ptr but in rather non-straight-forward way: it got pull
in to our own namespace so in the end Boost::Python::class_ is 'called'
with held-type set to a formally different type name. I will see if setting
held-type to boost::shared_ptr fix this.
Best,
Ernie.

Post by Stefan Seefeld
Ernie,
it appears what you are seeing is the fact that boost.python only
supports boost::shared_ptr, but not std::shared_ptr. (See
https://svn.boost.org/trac/boost/ticket/6545). Is it possible for you to
switch to Boost's shared_ptr, at least for the Python bindings ? Yes, we
need to fix this urgently... :-(
Stefan
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig

[Stefan Seefeld]

Post by Ernie Lee
Hi Stefan,
I updated my code so it now use 'boost::shared_ptr' class while
specifying the held-type and i got exactly the same errors (i guess
boost did recognize class even in different namespace).
Any other theories of what could be wrong? Could it be that Python
in some cases tries to manage memory directly, disregarding SP layer?

In that case I suggest you narrow down the failure into a minimal test
case and send that to the list. Otherwise this would be highly
speculative and thus inefficient.

Regards,
Stefan

[Ernie Lee]

Hi Stefan,

I tried to create minimal tests and it turned out that there might be two
separate issues here: one related to memory management and other to
subclassing C++ classes in Python (i will post the subclassing example in
separate mail).

Running code below as-is works fine and i can see printed message
generated by ‘A’ class destructor. However commenting out line
‘reset_callback()’ in Python code will lead to a segfault and also no call
to ‘A’ class destructor occur. Thoughts?

Thanks,


Ernie.


Python code: ————————
from callback2 import *

a1 = A()
a1.info()
set_callback(a1)
test_callback()
reset_callback()

C++ code: ——————————
include <iostream>
#include <boost/shared_ptr.hpp>
#include <boost/python.hpp>

class A
{
public:
virtual ~A() { std::cout << "A destructor for object: " << this <<
std::endl; }

virtual void info() { std::cout << "C++ A info for object" << this <<
std::endl; }
};

static boost::shared_ptr<A> current_callback;

void set_callback(boost::shared_ptr<A> a) {current_callback = a; }

void reset_callback() { current_callback = boost::shared_ptr<A>(); }

void test_callback()
{
std::cout << "test_callback: ";
if(current_callback) current_callback->info();
else std::cout << "Callback is NULL!" << std::endl;
}


BOOST_PYTHON_MODULE(callback2)
{
boost::python::class_<A, boost::shared_ptr< A >, boost::noncopyable>("A")
.def("info", &A::info)
;

boost::python::def("set_callback", set_callback);
boost::python::def("reset_callback", reset_callback);
boost::python::def("test_callback", test_callback);
}
————————————————

Post by Stefan Seefeld

Post by Ernie Lee
Hi Stefan,
I updated my code so it now use 'boost::shared_ptr' class while
specifying the held-type and i got exactly the same errors (i guess
boost did recognize class even in different namespace).
Any other theories of what could be wrong? Could it be that Python
in some cases tries to manage memory directly, disregarding SP layer?

In that case I suggest you narrow down the failure into a minimal test
case and send that to the list. Otherwise this would be highly
speculative and thus inefficient.
Regards,
Stefan
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig

[Ernie Lee]

Also, i found that i manually call 'Py_XINCREF' on callback object (ie on
'a1' in this example) then script is terminating without errors and do not
see destructor call for a1 object (but i guess later one is expected).
Thoughts?

Post by Ernie Lee
Hi Stefan,
I tried to create minimal tests and it turned out that there might be
two separate issues here: one related to memory management and other to
subclassing C++ classes in Python (i will post the subclassing example in
separate mail).
Running code below as-is works fine and i can see printed message
generated by ‘A’ class destructor. However commenting out line
‘reset_callback()’ in Python code will lead to a segfault and also no call
to ‘A’ class destructor occur. Thoughts?
Thanks,
Ernie.
Python code: ————————
from callback2 import *
a1 = A()
a1.info()
set_callback(a1)
test_callback()
reset_callback()
C++ code: ——————————
include <iostream>
#include <boost/shared_ptr.hpp>
#include <boost/python.hpp>
class A
{
virtual ~A() { std::cout << "A destructor for object: " << this <<
std::endl; }
virtual void info() { std::cout << "C++ A info for object" << this <<
std::endl; }
};
static boost::shared_ptr<A> current_callback;
void set_callback(boost::shared_ptr<A> a) {current_callback = a; }
void reset_callback() { current_callback = boost::shared_ptr<A>(); }
void test_callback()
{
std::cout << "test_callback: ";
if(current_callback) current_callback->info();
else std::cout << "Callback is NULL!" << std::endl;
}
BOOST_PYTHON_MODULE(callback2)
{
boost::python::class_<A, boost::shared_ptr< A >, boost::noncopyable>("A")
.def("info", &A::info)
;
boost::python::def("set_callback", set_callback);
boost::python::def("reset_callback", reset_callback);
boost::python::def("test_callback", test_callback);
}
————————————————

Post by Stefan Seefeld

Post by Ernie Lee
Hi Stefan,
I updated my code so it now use 'boost::shared_ptr' class while
specifying the held-type and i got exactly the same errors (i guess
boost did recognize class even in different namespace).
Any other theories of what could be wrong? Could it be that Python
in some cases tries to manage memory directly, disregarding SP layer?

In that case I suggest you narrow down the failure into a minimal test
case and send that to the list. Otherwise this would be highly
speculative and thus inefficient.
Regards,
Stefan
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig