1 [chapter Frequently Asked Questions (FAQs)
6 [section How can I wrap a function which takes a function pointer as an argument?]
8 If what you're trying to do is something like this:
10 typedef boost::function<void (string s) > funcptr;
17 BOOST_PYTHON_MODULE(test)
32 The short answer is: "you can't". This is not a
33 Boost.Python limitation so much as a limitation of C++. The
34 problem is that a Python function is actually data, and the only
35 way of associating data with a C++ function pointer is to store it
36 in a static variable of the function. The problem with that is
37 that you can only associate one piece of data with every C++
38 function, and we have no way of compiling a new C++ function
39 on-the-fly for every Python function you decide to pass
40 to `foo`. In other words, this could work if the C++
41 function is always going to invoke the /same/ Python
42 function, but you probably don't want that.
44 If you have the luxury of changing the C++ code you're
45 wrapping, pass it an `object` instead and call that;
46 the overloaded function call operator will invoke the Python
47 function you pass it behind the `object`.
50 [section I'm getting the "attempt to return dangling reference" error.
51 What am I doing wrong?]
53 That exception is protecting you from causing a nasty crash. It usually
54 happens in response to some code like this:
56 period const &get_floating_frequency() const
58 return boost::python::call_method<period const &>(
59 m_self,"get_floating_frequency");
64 ReferenceError: Attempt to return dangling reference to object of type:
68 In this case, the Python method invoked by `call_method`
69 constructs a new Python object. You're trying to return a reference to a
70 C++ object (an instance of `class period`) contained within
71 and owned by that Python object. Because the called method handed back a
72 brand new object, the only reference to it is held for the duration of
73 `get_floating_frequency()` above. When the function returns,
74 the Python object will be destroyed, destroying the instance of
75 `class period`, and leaving the returned reference dangling.
76 That's already undefined behavior, and if you try to do anything with
77 that reference you're likely to cause a crash. Boost.Python detects this
78 situation at runtime and helpfully throws an exception instead of letting
82 [section Is `return_internal_reference` efficient?]
84 [*Q:] /I have an object composed of 12 doubles. A `const&` to
85 this object is returned by a member function of another class. From the
86 viewpoint of using the returned object in Python I do not care if I get
87 a copy or a reference to the returned object. In Boost.Python I have the
88 choice of using `copy_const_reference` or `return_internal_reference`.
89 Are there considerations that would lead me to prefer one over the other,
90 such as size of generated code or memory overhead?/
92 [*A:] `copy_const_reference` will make an instance with storage
93 for one of your objects, `size = base_size + 12 * sizeof(double)`.
94 `return_internal_reference` will make an instance with storage for a
95 pointer to one of your objects, `size = base_size + sizeof(void*)`.
96 However, it will also create a weak reference object which goes in the
97 source object's weakreflist and a special callback object to manage the
98 lifetime of the internally-referenced object. My guess?
99 `copy_const_reference` is your friend here, resulting in less overall
100 memory use and less fragmentation, also probably fewer total
104 [section How can I wrap functions which take C++ containers as arguments?]
106 Ralf W. Grosse-Kunstleve provides these notes:
108 # Using the regular `class_<>` wrapper:
110 class_<std::vector<double> >("std_vector_double")
115 This can be moved to a template so that several types (`double`, `int`,
116 `long`, etc.) can be wrapped with the same code. This technique is used
117 in the file `scitbx/include/scitbx/array_family/boost_python/flex_wrapper.h`
118 in the "scitbx" package. The file could easily be modified for
119 wrapping `std::vector<>` instantiations.
120 This type of C++/Python binding is most suitable for containers
121 that may contain a large number of elements (>10000).
123 # Using custom rvalue converters. Boost.Python "rvalue converters"
124 match function signatures such as:
126 void foo(std::vector<double> const &array); // pass by const-reference
127 void foo(std::vector<double> array); // pass by value
129 Some custom rvalue converters are implemented in the file
130 `scitbx/include/scitbx/boost_python/container_conversions.h`
131 This code can be used to convert from C++ container types such as
132 `std::vector<>` or `std::list<>` to Python tuples and vice
133 versa. A few simple examples can be found in the file
134 `scitbx/array_family/boost_python/regression_test_module.cpp`
135 Automatic C++ container <-> Python tuple conversions are most
136 suitable for containers of moderate size. These converters generate
137 significantly less object code compared to alternative 1 above.
139 A disadvantage of using alternative 2 is that operators such as
140 arithmetic +,-,*,/,% are not available. It would be useful to have custom
141 rvalue converters that convert to a "math_array" type instead of tuples.
142 This is currently not implemented but is possible within the framework of
143 Boost.Python V2 as it will be released in the next couple of weeks. [ed.:
144 this was posted on 2002/03/10]
146 It would also be useful to also have "custom lvalue converters" such
147 as `std::vector<>` <-> Python list. These converters would
148 support the modification of the Python list from C++. For example:
152 void foo(std::vector<double> &array)
154 for(std::size_t i=0;i<array.size();i++) {
166 Custom lvalue converters require changes to the Boost.Python core library
167 and are currently not available.
171 The "scitbx" files referenced above are available via anonymous
174 cvs -d:pserver:anonymous@cvs.cctbx.sourceforge.net:/cvsroot/cctbx login
175 cvs -d:pserver:anonymous@cvs.cctbx.sourceforge.net:/cvsroot/cctbx co scitbx
179 [section fatal error C1204:Compiler limit:internal structure overflow]
181 [*Q:] /I get this error message when compiling a large source file. What can I do?/
183 [*A:] You have two choices:
185 # Upgrade your compiler (preferred)
187 # Break your source file up into multiple translation units.
189 `my_module.cpp`: [c++]
193 void more_of_my_module();
194 BOOST_PYTHON_MODULE(my_module)
202 `more_of_my_module.cpp`:
204 void more_of_my_module()
210 If you find that a `class_<...>` declaration
211 can't fit in a single source file without triggering the error, you
212 can always pass a reference to the `class_` object to a
213 function in another source file, and call some of its member
214 functions (e.g. `.def(...)`) in the auxilliary source
217 `more_of_my_class.cpp`:
219 void more_of_my_class(class<my_class>& x)
223 .add_property("xx", &my_class::get_xx, &my_class::set_xx)
230 [section How do I debug my Python extensions?]
232 Greg Burley gives the following answer for Unix GCC users:
234 [:Once you have created a boost python extension for your c++ library or
235 class, you may need to debug the code. Afterall this is one of the
236 reasons for wrapping the library in python. An expected side-effect or
237 benefit of using BPL is that debugging should be isolated to the c++
238 library that is under test, given that python code is minimal and
239 boost::python either works or it doesn't. (ie. While errors can occur
240 when the wrapping method is invalid, most errors are caught by the
243 The basic steps required to initiate a gdb session to debug a c++
244 library via python are shown here. Note, however that you should start
245 the gdb session in the directory that contains your BPL my_ext.so
249 (gdb) target exec python
251 >>> from my_ext import *
253 (gdb) break MyClass::MyBuggyFunction
255 >>> pyobj = MyClass()
256 >>> pyobj.MyBuggyFunction()
257 Breakpoint 1, MyClass::MyBuggyFunction ...
258 Current language: auto; currently c++
259 (gdb) do debugging stuff
263 Greg's approach works even better using Emacs' "gdb"
264 command, since it will show you each line of source as you step through it.
266 On *Windows*, my favorite debugging solution is the debugger that
267 comes with Microsoft Visual C++ 7. This debugger seems to work with code
268 generated by all versions of Microsoft and Metrowerks toolsets; it's rock
269 solid and "just works" without requiring any special tricks from the
272 Raoul Gough has provided the following for gdb on Windows:
274 [:gdb support for Windows DLLs has improved lately, so it is
275 now possible to debug Python extensions using a few
276 tricks. Firstly, you will need an up-to-date gdb with support
277 for minimal symbol extraction from a DLL. Any gdb from version 6
278 onwards, or Cygwin gdb-20030214-1 and onwards should do. A
279 suitable release will have a section in the gdb.info file under
280 Configuration - Native - Cygwin Native -
281 Non-debug DLL symbols. Refer to that info section for more
282 details of the procedures outlined here.
284 Secondly, it seems necessary to set a breakpoint in the
285 Python interpreter, rather than using ^C to break execution. A
286 good place to set this breakpoint is PyOS_Readline, which will
287 stop execution immediately before reading each interactive
288 Python command. You have to let Python start once under the
289 debugger, so that it loads its own DLL, before you can set the
294 GNU gdb 2003-09-02-cvs (cygwin-special)
298 Starting program: /cygdrive/c/Python22/python.exe
299 Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
300 Type "help", "copyright", "credits" or "license" for more information.
304 Program exited normally.
305 (gdb) break *&PyOS_Readline
306 Breakpoint 1 at 0x1e04eff0
308 Starting program: /cygdrive/c/Python22/python.exe
309 Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32
310 Type "help", "copyright", "credits" or "license" for more information.
312 Breakpoint 1, 0x1e04eff0 in python22!PyOS_Readline ()
313 from /cygdrive/c/WINNT/system32/python22.dll
316 >>> from my_ext import *
318 Breakpoint 1, 0x1e04eff0 in python22!PyOS_Readline ()
319 from /cygdrive/c/WINNT/system32/python22.dll
320 (gdb) # my_ext now loaded (with any debugging symbols it contains)
324 [h2 Debugging extensions through Boost.Build]
326 If you are launching your extension module tests with _bb_ using the
327 `boost-python-runtest` rule, you can ask it to launch your
328 debugger for you by adding "--debugger=/debugger/" to your bjam
331 bjam -sTOOLS=vc7.1 "--debugger=devenv /debugexe" test
332 bjam -sTOOLS=gcc -sPYTHON_LAUNCH=gdb test
334 It can also be extremely useful to add the `-d+2` option when
335 you run your test, because Boost.Build will then show you the exact
336 commands it uses to invoke it. This will invariably involve setting up
337 PYTHONPATH and other important environment variables such as
338 LD_LIBRARY_PATH which may be needed by your debugger in order to get
339 things to work right.
342 [section Why doesn't my `*=` operator work?]
344 [*Q:] ['I have exported my class to python, with many overloaded
345 operators. it works fine for me except the `*=`
346 operator. It always tells me "can't multiply sequence with non int
347 type". If I use `p1.__imul__(p2)` instead of
348 `p1 *= p2`, it successfully executes my code. What's
351 [*A:] There's nothing wrong with you. This is a bug in Python
352 2.2. You can see the same effect in Pure Python (you can learn a lot
353 about what's happening in Boost.Python by playing with new-style
354 classes in Pure Python).
357 ... def __imul__(self, x):
363 To cure this problem, all you need to do is upgrade your Python to
364 version 2.2.1 or later.
367 [section Does Boost.Python work with Mac OS X?]
369 It is known to work under 10.2.8 and 10.3 using
370 Apple's gcc 3.3 compiler:
371 ``gcc (GCC) 3.3 20030304 (Apple Computer, Inc. build 1493)``
372 Under 10.2.8 get the August 2003 gcc update (free at [@http://connect.apple.com]).
373 Under 10.3 get the Xcode Tools v1.0 (also free).
375 Python 2.3 is required. The Python that ships with 10.3 is
376 fine. Under 10.2.8 use these commands to install Python
378 ``./configure --enable-framework
380 make frameworkinstall``
382 The last command requires root privileges because the target
383 directory is `/Library/Frameworks/Python.framework/Versions/2.3`.
384 However, the installation does not interfere with the Python
385 version that ships with 10.2.8.
387 It is also crucial to increase the `stacksize` before
388 starting compilations, e.g.:
389 ``limit stacksize 8192k``
390 If the `stacksize` is too small the build might crash with
391 internal compiler errors.
393 Sometimes Apple's compiler exhibits a bug by printing an error
394 like the following while compiling a
395 `boost::python::class_<your_type>`
396 template instantiation:
398 .../inheritance.hpp:44: error: cannot
399 dynamic_cast `p' (of type `struct cctbx::boost_python::<unnamed>::add_pair*
400 ') to type `void*' (source type is not polymorphic)
403 We do not know a general workaround, but if the definition of
404 `your_type` can be modified the following was found
405 to work in all cases encountered so far:
409 // before defining any member data
410 #if defined(__MACH__) && defined(__APPLE_CC__) && __APPLE_CC__ == 1493
413 // now your member data, e.g.
420 [section How can I find the existing PyObject that holds a C++ object?]
422 [: "I am wrapping a function that always returns a pointer to an
423 already-held C++ object."]
425 One way to do that is to hijack the mechanisms used for wrapping a class
426 with virtual functions. If you make a wrapper class with an initial
427 PyObject* constructor argument and store that PyObject* as "self", you
428 can get back to it by casting down to that wrapper type in a thin wrapper
429 function. For example:
431 class X { X(int); virtual ~X(); ... };
432 X* f(); // known to return Xs that are managed by Python objects
439 X_wrap(PyObject* self, int v) : self(self), X(v) {}
445 X_wrap* xw = dynamic_cast<X_wrap*>(f());
447 return handle<>(borrowed(xw->self));
453 class_<X,X_wrap,boost::noncopyable>("X", init<int>())
458 Of course, if X has no virtual functions you'll have to use
459 `static_cast` instead of `dynamic_cast` with no
460 runtime check that it's valid. This approach also only works if the
461 `X` object was constructed from Python, because
462 `X`\ s constructed from C++ are of course never
465 Another approach to this requires you to change your C++ code a bit;
466 if that's an option for you it might be a better way to go. work we've
467 been meaning to get to anyway. When a `shared_ptr<X>` is
468 converted from Python, the shared_ptr actually manages a reference to the
469 containing Python object. When a shared_ptr<X> is converted back to
470 Python, the library checks to see if it's one of those "Python object
471 managers" and if so just returns the original Python object. So you could
472 just write `object(p)` to get the Python object back. To
473 exploit this you'd have to be able to change the C++ code you're wrapping
474 so that it deals with shared_ptr instead of raw pointers.
476 There are other approaches too. The functions that receive the Python
477 object that you eventually want to return could be wrapped with a thin
478 wrapper that records the correspondence between the object address and
479 its containing Python object, and you could have your f_wrap function
480 look in that mapping to get the Python object out.
483 [section How can I wrap a function which needs to take ownership of a raw pointer?]
485 [*Q:] Part of an API that I'm wrapping goes something like this:
488 struct A {}; struct B { void add( A* ); }
489 where B::add() takes ownership of the pointer passed to it.
500 # python interpreter crashes
501 # later due to memory corruption.
504 Even binding the lifetime of a to b via `with_custodian_and_ward` doesn't prevent
505 the python object a from ultimately trying to delete the object it's pointing to.
506 Is there a way to accomplish a 'transfer-of-ownership' of a wrapped C++ object?
510 Yes: Make sure the C++ object is held by auto_ptr:
512 class_<A, std::auto_ptr<A> >("A")
516 Then make a thin wrapper function which takes an auto_ptr parameter:
518 void b_insert(B &b, std::auto_ptr<A> a)
524 Wrap that as B.add. Note that pointers returned via `manage_new_object`
525 will also be held by `auto_ptr`, so this transfer-of-ownership
526 will also work correctly.
529 [section Compilation takes too much time and eats too much memory!
530 What can I do to make it faster?]
532 Please refer to the `Reducing Compiling Time` section in the _tutorial_.
535 [section How do I create sub-packages using Boost.Python?]
537 Please refer to the `Creating Packages` section in the _tutorial_.
540 [section error C2064: term does not evaluate to a function taking 2 arguments]
542 /Niall Douglas provides these notes:/
544 If you see Microsoft Visual C++ 7.1 (MS Visual Studio .NET 2003) issue
545 an error message like the following it is most likely due to a bug
548 boost\boost\python\detail\invoke.hpp(76):
549 error C2064: term does not evaluate to a function taking 2 arguments"
551 This message is triggered by code like the following:
553 #include <boost/python.hpp>
555 using namespace boost::python;
560 bool setAutoDelete(bool doso) throw();
563 void Export_FXThread()
565 class_< FXThread >("FXThread")
566 .def("setAutoDelete", &FXThread::setAutoDelete)
570 The bug is related to the `throw()` modifier.
571 As a workaround cast off the modifier. E.g.:
573 .def("setAutoDelete", (bool (FXThread::*)(bool)) &FXThread::setAutoDelete)
575 (The bug has been reported to Microsoft.)
578 [section How can I automatically convert my custom string type to and from a Python string?]
580 /Ralf W. Grosse-Kunstleve provides these notes:/
582 Below is a small, self-contained demo extension module that shows
583 how to do this. Here is the corresponding trivial test:
586 assert custom_string.hello() == "Hello world."
587 assert custom_string.size("california") == 10
589 If you look at the code you will find:
591 * A custom `to_python` converter (easy):
592 `custom_string_to_python_str`
594 *A custom lvalue converter (needs more code):
595 `custom_string_from_python_str`
597 The custom converters are registered in the global Boost.Python
598 registry near the top of the module initialization function. Once
599 flow control has passed through the registration code the automatic
600 conversions from and to Python strings will work in any module
601 imported in the same process.
604 #include <boost/python/module.hpp>
605 #include <boost/python/def.hpp>
606 #include <boost/python/to_python_converter.hpp>
608 namespace sandbox { namespace {
614 custom_string(std::string const &value) : value_(value) {}
615 std::string const &value() const { return value_; }
620 struct custom_string_to_python_str
622 static PyObject* convert(custom_string const &s)
624 return boost::python::incref(boost::python::object(s.value()).ptr());
628 struct custom_string_from_python_str
630 custom_string_from_python_str()
632 boost::python::converter::registry::push_back(
635 boost::python::type_id<custom_string>());
638 static void* convertible(PyObject* obj_ptr)
640 if (!PyString_Check(obj_ptr)) return 0;
644 static void construct(
646 boost::python::converter::rvalue_from_python_stage1_data* data)
648 const char* value = PyString_AsString(obj_ptr);
649 if (value == 0) boost::python::throw_error_already_set();
651 (boost::python::converter::rvalue_from_python_storage<custom_string>*)
652 data)->storage.bytes;
653 new (storage) custom_string(value);
654 data->convertible = storage;
658 custom_string hello() { return custom_string("Hello world."); }
660 std::size_t size(custom_string const &s) { return s.value().size(); }
664 using namespace boost::python;
666 boost::python::to_python_converter<
668 custom_string_to_python_str>();
670 custom_string_from_python_str();
676 }} // namespace sandbox::<anonymous>
678 BOOST_PYTHON_MODULE(custom_string)
680 sandbox::init_module();
684 [section Why is my automatic to-python conversion not being found?]
686 /Niall Douglas provides these notes:/
688 If you define custom converters similar to the ones
689 shown above the `def_readonly()` and `def_readwrite()`
690 member functions provided by `boost::python::class_` for
691 direct access to your member data will not work as expected.
692 This is because `def_readonly("bar",&foo::bar)` is
696 .add_property("bar", make_getter(&foo::bar, return_internal_reference()))
698 Similarly, `def_readwrite("bar",&foo::bar)` is
702 .add_property("bar", make_getter(&foo::bar, return_internal_reference()),
703 make_setter(&foo::bar, return_internal_reference())
705 In order to define return value policies compatible with the
706 custom conversions replace `def_readonly()` and
707 `def_readwrite()` by `add_property()`. E.g.:
710 .add_property("bar", make_getter(&foo::bar, return_value_policy<return_by_value>()),
711 make_setter(&foo::bar, return_value_policy<return_by_value>()))
715 [section Is Boost.Python thread-aware/compatible with multiple interpreters?]
717 /Niall Douglas provides these notes:/
719 The quick answer to this is: no.
721 The longer answer is that it can be patched to be so, but it's
722 complex. You will need to add custom lock/unlock wrapping of every
723 time your code enters Boost.Python (particularly every virtual
724 function override) plus heavily modify
725 `boost/python/detail/invoke.hpp` with custom unlock/lock
726 wrapping of every time Boost.Python enters your code. You must
727 furthermore take care to /not/ unlock/lock when Boost.Python
728 is invoking iterator changes via `invoke.hpp`.
730 There is a patched `invoke.hpp` posted on the C++-SIG
731 mailing list archives and you can find a real implementation of all
732 the machinery necessary to fully implement this in the TnFOX
733 project at [@http://sourceforge.net/projects/tnfox/ this]
734 SourceForge project location.