How to debug your Python code effectively(IPython)

2020-07-08 Update: Add notes for flask debug

As a Python programmer, you may have used pdb or Pycharm to debug your Python code.

But IMHO, those tools are a misunderstanding of Python language concepts. As Python is an interpreted language, it means we should abandon breakpoint and just tell the interpreter what we really want to do.

IPython

IPython is a powerful interactive Python shell, which has powerful tab completion and gives you access to talk with your program wherever you want to have a breakpoint.

This tool has too many dependencies and will pollute other libraries, so I recommend you use it with virtualenv to split the environment.

You can use ipython in your shell environment or just add this to your program.

import IPython
IPython.embed()

This is an example for embeding.

# hello.py

def main():
    print('Hello world')
    a = 10
    print(a, a * 5)
    import IPython; IPython.embed()


if __name__ == '__main__':
    main()

When you run it:

You can get the value of a and when the debugging is over, exit IPython with Ctrl-D.

When should you use IPython?

Here are some scenarios I have encountered at work. IPython could save you time if you’re bothered by these scenarios.

1. Development phase: When you don’t know the return value of a function, or have questions about the data structure in the documentation, you can use IPython to look at the exact place in the code.

2. When you use try-except, or if-else, you could insert IPython.embed() to watch what happend. Yes, the IPython.embed() could be used in the except:

try:
    raise Exception("xxx")
except Exception as e:
    import IPython; IPython.embed()

3. Debug phase: IPython could be used as breakpoint, which is more powerful than a breakpoint. You can execute some statements manually, including:
   1. statements that can be executed next
   2. unrelated statements, just for testing

A few notes when using IPython:

1. Always remember to delete the embed statements before you git add. It should not be used in the production environment, especially in tornado or uwsgi

2. Don’t use IPython in the for loop. It may go into interactive shell mode several times, and you may need kill to stop the program. If you really want this, please add a break after you embed

3. Don’t use IPython in Windows. It is very slow. You could use WSL or virtual linux machine on it

How to use in uwsgi

If your program use uwsgi to start, IPython.embed() could quit immediately. So I’ll give you two solution:

uwsgi settings

The first solution is to allow uwsgi work with stdin, and you could add this setting.

honour-stdin: 1

or just add this argument when you start

uwsgi --honour-stdin -y app.yaml

Use tornado to debug uwsgi

I have wrote another blog: Using Tornado to debug all types of Python servers, You can use Tornado to run servers that use the WSGI protocol.

How to use in tornado asyncio mode

After Tornado 6.0 version, use IPython will cause RuntimeError: This event loop is already running. There is a simple solution for this, use nest_asyncio

import os

# enable nest_asyncio only in DEBUG mode:
# > DEBUG=true python main.py
if os.environ.get("DEBUG") == "true":
    print("In nest asyncio mode")
    import nest_asyncio
    nest_asyncio.apply()

Actually, I use uwsgi in production mode, so I’d like to recommend you just keep use tornado==5.

How to use in flask

I want to run a local server with beautiful log and autoreload support. You may only need a flask runner. so when in debug mode, you could use this:

# use tornado log support
from tornado.log import enable_pretty_logging, app_log
enable_pretty_logging()

log_mock = mock.patch('logging.getLogger', return_value=app_log)
log_mock.start()
start_server()
log_mock.end()


# Allow jinja auto reload
application.jinja_env.auto_reload = True
application.config['TEMPLATES_AUTO_RELOAD'] = True

# multiprocess
application.run(debug=True, port=options.port, threaded=False, processes=3)

# single process
application.run(debug=True, port=options.port)