Millet Porridge

English version of https://corvo.myseu.cn

0%

On Using IPython in asyncio Mode

Posted on Edited on In Disqus:
Tornado 6.0 has fully switched to asyncio, but my IPython embed debugging didn't go smoothly. This blog mainly introduces how to keep using IPython in the new Tornado.

Tornado 6.0 has fully switched to asyncio, but my IPython embed debugging didn’t go smoothly. This blog mainly introduces how to keep using IPython in the new Tornado.

The Problem

Take the code below as an example: requesting 127.0.0.1:8888 produces RuntimeError: This event loop is already running

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import tornado.ioloop
import tornado.web


class MainHandler(tornado.web.RequestHandler):
    def get(self):
        # ---------- XXX: Can't GIT add [START] ---------- #
        import IPython
        IPython.embed(using=False)
        # ---------- XXX: Can't GIT add  [END]  ---------- #
        self.write("Hello, world")


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])


if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

The detailed stack trace:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Python 3.8.1 (default, Jan 22 2020, 06:38:00) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

ERROR:tornado.application:Uncaught exception GET / (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/tornado/web.py", line 1697, in _execute
    result = method(*self.path_args, **self.path_kwargs)
  File "main.py", line 11, in get
    IPython.embed(using=False)
  File "/usr/lib/python3.8/site-packages/IPython/terminal/embed.py", line 388, in embed
    shell(header=header, stack_depth=2, compile_flags=compile_flags,
  File "/usr/lib/python3.8/site-packages/IPython/terminal/embed.py", line 228, in __call__
    self.mainloop(local_ns, module, stack_depth=stack_depth,
  File "/usr/lib/python3.8/site-packages/IPython/terminal/embed.py", line 324, in mainloop
    self.interact()
  File "/usr/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 541, in interact
    code = self.prompt_for_code()
  File "/usr/lib/python3.8/site-packages/IPython/terminal/interactiveshell.py", line 467, in prompt_for_code
    text = self.pt_app.prompt(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", line 997, in prompt
    return self.app.run()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 810, in run
    return loop.run_until_complete(self.run_async(pre_run=pre_run))
  File "/usr/lib/python3.8/asyncio/base_events.py", line 599, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.8/asyncio/base_events.py", line 554, in run_forever
    raise RuntimeError('This event loop is already running')
RuntimeError: This event loop is already running
ERROR:tornado.access:500 GET / (127.0.0.1) 718.57ms

The Solution

I found the related problem on StackOverflow: RuntimeError: This event loop is already running in python

The error occurs because asyncio doesn’t support nesting; a workable solution is using nest_asyncio to patch it, allowing asyncio nesting.

1
2
3
# pip install nest_asyncio
import nest_asyncio
nest_asyncio.apply()

The corresponding code changes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import os

# the patch must go before all other statements, to prevent other libraries from importing the unpatched library
# for safety, only enable it in DEBUG mode; usage: DEBUG=true python main.py
if os.environ.get("DEBUG") == "true":
    print("In nest asyncio mode")
    import nest_asyncio
    nest_asyncio.apply()

import tornado.ioloop
import tornado.web


class MainHandler(tornado.web.RequestHandler):
    def get(self):
        # ---------- XXX: Can't GIT add [START] ---------- #
        import IPython
        IPython.embed(using=False)
        # ---------- XXX: Can't GIT add  [END]  ---------- #
        self.write("Hello, world")


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])


if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

Afterwards you can happily use IPython.

A Little Reading of nest_asyncio’s Source

Those interested can look at the nest_asyncio code — not much code, just one file. The main patched part is the original function’s error for multiple tasks, and it preserves the execution context information.

For example, in patch_loop:

the original RuntimeError is removed, and the context environment is restored.

Summary

Honestly, I don’t want to apply a strategy like nest_asyncio to production environments; I only want to use it for program debugging.

I didn’t dig into whether recreating a new event loop is better or making the original event loop nestable is better. After all, my problem is solved — I can still happily debug with IPython, and that’s enough.

Related Posts