Discussion:
[C++-sig] pybind11 -- alternative to Boost.Python
Wenzel Jakob
2015-10-18 12:56:42 UTC
Permalink
Hello all,

after being a long-time Boost.Python user, I’ve been working on an alternative that makes more effective use of recent C++11-capable compilers. The overall syntax and ideology are very similar to Boost.Python, but the implementation only requires a few header files with a a vastly smaller amount of code thanks to C++11 lambda functions, tuples and variadic templates. There is also dedicated support for Python’s buffer protocol and NumPy arrays, which is useful for scientific computing applications.

So far it’s only used by a few projects, but I think it could be useful to this audience.

Code: https://github.com/wjakob/pybind11 <https://github.com/wjakob/pybind11>
Documentation: http://pybind11.readthedocs.org/en/latest/ <http://pybind11.readthedocs.org/en/latest/>

Best,
Wenzel
Axel Huebl
2015-10-19 06:53:31 UTC
Permalink
Wow the docs and examples look great!
Thank you for the tremendous amount of work you put in!

I am eager to test this with CUDA7+/C++11 programs.

Best,
Axel
Post by Wenzel Jakob
Hello all,
after being a long-time Boost.Python user, I’ve been working on an
alternative that makes more effective use of recent C++11-capable
compilers. The overall syntax and ideology are very similar to
Boost.Python, but the implementation only requires a few header files
with a a vastly smaller amount of code thanks to C++11 lambda
functions, tuples and variadic templates. There is also dedicated
support for Python’s buffer protocol and NumPy arrays, which is useful
for scientific computing applications.
So far it’s only used by a few projects, but I think it could be useful
to this audience.
Code: https://github.com/wjakob/pybind11
<https://github.com/wjakob/pybind11>
Documentation: http://pybind11.readthedocs.org/en/latest/
<http://pybind11.readthedocs.org/en/latest/>
Best,
Wenzel
------------------------------------------------------------------------
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig
Wenzel Jakob
2015-10-19 10:24:03 UTC
Permalink
I would be open to it but have my doubts about the feasibility of a merge. Consider the difference in code size alone: Boost.Python (without dependencies like MPL etc.) uses 26K lines of code, compared to about 2K for pybind11 (3K with all extensions). Apart from that, the libraries take very different internal design decisions, which would likely break existing software that ventures beyond the basic .def() syntax.

Cheers,
Wenzel
Post by Wenzel Jakob
________________________________
Sent: Sunday, October 18, 2015 2:56 PM
Subject: [C++-sig] pybind11 -- alternative to Boost.Python
It looks good.
Have you tried contacting the actual boost.python mantainer and maybe propose merge with the boost.python or make a boost.python3 from it? It would be shame not incorporate the useful stuff in boost.python.
Stefan Seefeld
2015-10-19 12:03:19 UTC
Permalink
Post by Wenzel Jakob
I would be open to it but have my doubts about the feasibility of a
merge. Consider the difference in code size alone: Boost.Python
(without dependencies like MPL etc.) uses 26K lines of code, compared
to about 2K for pybind11 (3K with all extensions). Apart from that,
the libraries take very different internal design decisions, which
would likely break existing software that ventures beyond the basic
.def() syntax.
Hi Wenzel,

Indeed, I would be very interested in a detailed comparison. Modernizing
Boost.Python by using C++11 features (for example) and stripping out
obsolete compiler support is one way to move forward. That in itself
will help a lot, and may even allow Boost.Python to strip off
dependencies to other Boost libraries.

I'd be curious to learn about those design decisions that you are
alluding to that lead to incompatibilities. Such a document may
ultimately also be important for potential users when they consider the
alternatives.

Regards,
Stefan
--
...ich hab' noch einen Koffer in Berlin...
Jim Bosch
2015-10-19 12:15:34 UTC
Permalink
At first glance, this looks great. I've been tinkering with something like
this for a while now, but it's never amounted to anything more than a C++11
learning project, and this looks quite solid in comparison.

It may be a long time before I get a chance to evaluate pybind11 for use in
my own projects, but as a prelude to that I was wondering if you could say
anything about support for custom converters and cross-module type
conversion, which I didn't see mentioned in the docs (though I just skimmed
them). Are you using the same sort of global registry Boost.Python used?
If so, I'm curious how that works with a header-only library.

Jim
Post by Wenzel Jakob
Hello all,
after being a long-time Boost.Python user, I’ve been working on an
alternative that makes more effective use of recent C++11-capable
compilers. The overall syntax and ideology are very similar to
Boost.Python, but the implementation only requires a few header files with
a a vastly smaller amount of code thanks to C++11 lambda functions, tuples
and variadic templates. There is also dedicated support for Python’s buffer
protocol and NumPy arrays, which is useful for scientific computing
applications.
So far it’s only used by a few projects, but I think it could be useful to
this audience.
Code: https://github.com/wjakob/pybind11
Documentation: http://pybind11.readthedocs.org/en/latest/
Best,
Wenzel
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig
Wenzel Jakob
2015-10-19 16:10:42 UTC
Permalink
Hi,

it would take a long time to discuss all differences, but I can give some examples. There are basically three ways of interfacing with Python objects in pybind11.

1. using wrapper classes like pybind11::object (analogous to boost::python::object)
2. by creating bindings that map a C++ type to Python — this is done using pybind11::class_ (analogous to boost::python::class_)
3. by declaring a partial template overload that does transparent conversions between different types.

Boost.Python’s approach for communicating type information (item 2. in the above list) between modules entails linking against a shared library with a few containers storing the relevant data. In comparison, pybind11 installs a __pybind11__ capsule object in the global scope for this purpose, which avoids the library dependency. Any extra binding library that is loaded just registers its types there.

In terms of the underlying implementation, 1. and 2. are pretty basic, and 3. is where a lot of the interesting things happen. This is basically a big list of partial template overloads of a class named type_caster which try to match various common types recursively. I’ll show just one example of how C++11 can considerably simplify implementation details here.

For instance, consider the converter which enables transparent conversions between std::tuple<
> and Python’s ‘tuple’ class. Among other things, pybind11 uses this to convert function arguments to Python objects. The top-level signature matches an arbitrary tuple (that could even be nested, or other kinds of type concoctions 
 :)) I’ll expand the snippet literal programming-style, adding code to the <
> part.

template <typename... Tuple> class type_caster<std::tuple<Tuple...>> {
typedef std::tuple<Tuple...> type;
enum { size = sizeof...(Tuple) };

<
>
};

The first thing we’ll do is to declare sub-converters to deal with the individual tuple element types. The decay template simplifies the base type as much as possible by stripping type modifiers like pointers, references, const, etc. (those are handled separately)

<
> +=
std::tuple<type_caster<typename decay<Tuple>::type>...> value;

The following function takes a tuple from Python and converts it into the corresponding C++ object, returning false if the conversion wasn’t possible. It expects a special type index_sequence<0,1,2,3,
., N-1> as an argument, where N is the length of the tuple. This is a pretty common workaround to enable something resembling a loop over variadic template arguments rather than writing a messy recursive function.

<
> +=
protected:
template <size_t ... Indices> bool load(PyObject *src, index_sequence<Indices...>) {
if (!PyTuple_Check(src))
return false;
if (PyTuple_Size(src) != size)
return false;
std::array<bool, size> results {{
(PyTuple_GET_ITEM(src, Indices) != nullptr ? std::get<Indices>(value).load(PyTuple_GET_ITEM(src, Indices)) : false)...
}};
for (bool r : results)
if (!r)
return false;
return true;
}

The following function function calls the above protected function with the needed index_sequence

<
> +=
public:
bool load(PyObject *src) {
return load(src, typename make_index_sequence<sizeof...(Tuple)>::type());
}

which is constructed using a much shorter recursive implementation that runs at compile time:

template<size_t ...> struct index_sequence { };
template<size_t N, size_t ...S> struct make_index_sequence : make_index_sequence <N - 1, N - 1, S...> { };
template<size_t ...S> struct make_index_sequence <0, S...> { typedef index_sequence<S...> type; };

Here is another very short example that I like: this converts a Python function into a std::function<> using a stateful lambda closure that invokes the function object’s call() function.
With this partial template overload, we can easily call functions that take std::function<>s as argument using Python functions. Something similar is also possible for the reverse direction.


template <typename Return, typename... Args> struct type_caster<std::function<Return(Args...)>> {
typedef std::function<Return(Args...)> type;
public:

bool load(PyObject *src_) {
if (!PyFunction_Check(src_))
return false;
object src(src_, true);
value = [src](Args... args) -> Return {
object retval(handle(src).call(std::move(args)...));
return retval.template cast<Return>();
};
return true;
}


<
>
protected:
type value;
}.

The codebase contains many other examples. For instance, the optional auto-vectorization support over NumPy array arguments is something that would have been very painful to do with C++03.

Best,
Wenzel
At first glance, this looks great. I've been tinkering with something like this for a while now, but it's never amounted to anything more than a C++11 learning project, and this looks quite solid in comparison.
It may be a long time before I get a chance to evaluate pybind11 for use in my own projects, but as a prelude to that I was wondering if you could say anything about support for custom converters and cross-module type conversion, which I didn't see mentioned in the docs (though I just skimmed them). Are you using the same sort of global registry Boost.Python used? If so, I'm curious how that works with a header-only library.
Jim
Hello all,
after being a long-time Boost.Python user, I’ve been working on an alternative that makes more effective use of recent C++11-capable compilers. The overall syntax and ideology are very similar to Boost.Python, but the implementation only requires a few header files with a a vastly smaller amount of code thanks to C++11 lambda functions, tuples and variadic templates. There is also dedicated support for Python’s buffer protocol and NumPy arrays, which is useful for scientific computing applications.
So far it’s only used by a few projects, but I think it could be useful to this audience.
Code: https://github.com/wjakob/pybind11 <https://github.com/wjakob/pybind11>
Documentation: http://pybind11.readthedocs.org/en/latest/ <http://pybind11.readthedocs.org/en/latest/>
Best,
Wenzel
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig <https://mail.python.org/mailman/listinfo/cplusplus-sig>
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig
Francesco Biscani
2015-10-19 19:58:05 UTC
Permalink
This looks really neat!

Do you have any measure on the memory/cpu performance wrt Boost.Python when
compiling large bindings? I would expect that variadic templates and all
the other C++11 goodies are more efficient than the preprocessor magic used
in Boost Python, from this point of view.

Cheers,

Francesco.
Post by Wenzel Jakob
Hello all,
after being a long-time Boost.Python user, I’ve been working on an
alternative that makes more effective use of recent C++11-capable
compilers. The overall syntax and ideology are very similar to
Boost.Python, but the implementation only requires a few header files with
a a vastly smaller amount of code thanks to C++11 lambda functions, tuples
and variadic templates. There is also dedicated support for Python’s buffer
protocol and NumPy arrays, which is useful for scientific computing
applications.
So far it’s only used by a few projects, but I think it could be useful to
this audience.
Code: https://github.com/wjakob/pybind11
Documentation: http://pybind11.readthedocs.org/en/latest/
Best,
Wenzel
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig
Wenzel Jakob
2015-10-19 23:13:23 UTC
Permalink
Hi,

I became curious about this myself and ran a simple benchmark for automatically generated binding code of increasing size.

The compilation times for Boost.Python and pybind11 turn out to be fairly similar. However, there is a significant difference in terms of the size of the compilation result, which is almost twice as large for Boost.Python.

See the details here: http://pybind11.readthedocs.org/en/latest/benchmark.html <http://pybind11.readthedocs.org/en/latest/benchmark.html>

Best,
Wenzel
Post by Francesco Biscani
This looks really neat!
Do you have any measure on the memory/cpu performance wrt Boost.Python when compiling large bindings? I would expect that variadic templates and all the other C++11 goodies are more efficient than the preprocessor magic used in Boost Python, from this point of view.
Cheers,
Francesco.
Hello all,
after being a long-time Boost.Python user, I’ve been working on an alternative that makes more effective use of recent C++11-capable compilers. The overall syntax and ideology are very similar to Boost.Python, but the implementation only requires a few header files with a a vastly smaller amount of code thanks to C++11 lambda functions, tuples and variadic templates. There is also dedicated support for Python’s buffer protocol and NumPy arrays, which is useful for scientific computing applications.
So far it’s only used by a few projects, but I think it could be useful to this audience.
Code: https://github.com/wjakob/pybind11 <https://github.com/wjakob/pybind11>
Documentation: http://pybind11.readthedocs.org/en/latest/ <http://pybind11.readthedocs.org/en/latest/>
Best,
Wenzel
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig <https://mail.python.org/mailman/listinfo/cplusplus-sig>
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig
Neal Becker
2015-10-20 11:33:15 UTC
Permalink
I noticed you used -flto when building the shared libs. Do you find this
makes a difference?
Wenzel Jakob
2015-10-20 12:02:23 UTC
Permalink
I use that by default for compiling Python bindings. It should not make any difference for just a single file (including this testcase), but I found that it yields consistently smaller shared libraries when dealing with lots of compilation units.

Wenzel
Post by Neal Becker
I noticed you used -flto when building the shared libs. Do you find this
makes a difference?
_______________________________________________
Cplusplus-sig mailing list
https://mail.python.org/mailman/listinfo/cplusplus-sig
Václav Šmilauer
2015-10-19 20:31:18 UTC
Permalink
Post by Wenzel Jakob
Code: https://github.com/wjakob/pybind11
Documentation: http://pybind11.readthedocs.org/en/latest/
Hi Wenzel, it looks excelent.

There are a few points in boost::python which have been unresolved for a
long time: straightforward support for aligned types (Eigen comes to
mind) and some subtle bugs in shared_ptr/weak_ptr (e.g. not holding GIL
when deleting some python objects and crashing) which are not being
fixed upstream. Could you comment on those?

I would suggest that the documents include some migration hints from
boost::python, if I if ever attempt it (something like: use
reference_internal instead of with_custodian_and_ward, this is how you
write an equivalent of raw_function/raw_constructor).

Cheers, Václav
Loading...