Debugging Hung Python Processes with GDB

When things go wrong in production, it can be necessary to troubleshoot problems where they occur, instead of in a development environment. In those situations having a working knowledge of GDB, GDB Python Extensions, and strace is very helpful. You will see some  simple techniques to get insight into those situations. This talk  outlines several techniques for connecting to an already running, "stuck", or deadlocked Python process using GDB for debugging.
 
During the talk, we will:
 
* inspect the current state of threads with
* use and demo the GDB macros for Python
* inspect a locally running process and a core dump collected from a remote machine
* use strace to gather system call information about a process
* discuss the SIGTRAP handler as a proactive way to make rpdb available in production.
 
I have had to debug several hard-to-find bugs that were very infrequent deadlocks using Python. Furthermore it was happening on remote machines I could not have network access to. This technique was invaluable in those situations.