Millet Porridge

English version of https://corvo.myseu.cn

0%

JSON Serialization of Custom Objects in Python

Posted on Edited on In Disqus:
Recently while writing Python, built-in objects can be formatted directly with json.dumps, but for custom classes json.dumps falls short. Here I explore several solutions, and in the end I picked a suitable one — sharing it with everyone.

First, I Hope Everyone Understands

json has never been the first choice for serialization in Python; json is used only to communicate with other applications. For internal Python communication and object serialization, use pickle — just as with C/C++ you basically wouldn’t consider json; for internal communication only, you should use ProtoBuf.

A Problem I Encountered

In Python you inevitably define some classes, and serializing custom classes becomes a troublesome matter. For the class below, suppose we want to format an object.

1
2
3
4
5
6
7
8
9
10
class Inner():
    def __init__(self):
        self.m_a = 'a'
        self.m_b = 'b'

In [7]: i = Inner()

In [8]: json.dumps(i)   # and then you hit a TypeError, because it can't serialize
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

So we consider adding a function to this Inner class, making it look like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class Inner():
    """ A class """
    def __init__(self):
        self.m_a = 'a'
        self.m_b = 'b'

    def to_json(self):
        return json.dumps({
            'm_a': self.m_a,
            'm_b': self.m_b,
        })


In [19]: i.to_json()
Out[19]: '{"m_a": "a", "m_b": "b"}'

Then you think the world has become normal again, everything satisfied. Or is it?

The wish is excellent, but… normal life is not composed of a single class. The class above is named Inner — might there be a class named Outer containing it? Seems quite likely.

1
2
3
4
5
6
7
8
9
10
11

class Outer():
    def __init__(self):
        self.m_i = Inner()
        self.m_c = 'c'

In [21]: o = Outer()

In [22]: json.dumps(o)   # the same problem seems to happen again
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)

So you decide to add a function to Outer too; something like the following seems to achieve the effect.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class Outer():
    """ A class """
    def __init__(self):
        self.m_i = Inner()
        self.m_c = 'c'

    def to_json(self):
        return json.dumps({
            'm_i': {
                'm_a': self.m_i.m_a,    # a new function could be defined here;
                'm_b': self.m_i.m_b,    # for ease of understanding I wrote it directly
                },
            'm_c': self.m_c
            })

In [26]: o.to_json()
Out[26]: '{"m_i": {"m_a": "a", "m_b": "b"}, "m_c": "c"}'

The World Surely Has More Than Two Classes

Now we face a problem: when similar situations appear again, we have to wrap them layer by layer, purely by hand. Can you imagine how special this process is? And every object needs to call to_json-style functions — truly hard.

At this point you should consider Google and StackOverflow; maybe someone has encountered this problem before. I searched with keywords like python json dumps custome object.

How to make a class JSON serializable was the first result; it introduces some methods, for example:

1
2
3
4
5
6
7
8
class CustomJsonEncoder(json.JSONEncoder):  # define a class inheriting from json.JSONEncoder
    def default(self, obj):
        if isinstance(obj, Outer):
            return {'m_c': obj.m_c}         # for simplicity I didn't write inner
        return json.JSONEncoder.default(self, obj)

In [32]: json.dumps(o, cls=CustomJsonEncoder)
Out[32]: '{"m_c": "c"}'

With this approach we can update the original Outer and Inner classes and combine them with CustomJsonEncoder:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
class Inner():
    """ A class """
    def __init__(self):
        self.m_a = 'a'
        self.m_b = 'b'

    def to_json(self):
        return {
            'm_a': self.m_a,
            'm_b': self.m_b,
        }

class Outer():
    """ A class """
    def __init__(self):
        self.m_i = Inner()
        self.m_c = 'c'

    def to_json(self):
        return {
            'm_i': self.m_i,
            'm_c': self.m_c
        }

class CustomJsonEncoder(json.JSONEncoder):  # define a class inheriting from json.JSONEncoder
    def default(self, obj):
        if getattr(obj, "to_json", None): # if there's a to_json method, call it
            return obj.to_json()
        return json.JSONEncoder.default(self, obj)

In [35]: o = Outer()

In [36]: json.dumps(o, cls=CustomJsonEncoder)
Out[36]: '{"m_i": {"m_a": "a", "m_b": "b"}, "m_c": "c"}'

Believe it or not, it’s quite successful. This way, classes you define yourself can also implement the to_json method. The downside is that every json.dumps now needs the trailing cls=CustomJsonEncoder.

Can It Be More Elegant

json.dumps(o, cls=CustomJsonEncoder) is really quite long; it would be nice to simplify it a little — how good it was originally: json.dumps(o).

I was fairly lucky and found How to make a class JSON serializable again. It provides an idea: since during json.dumps the default JSONEncoder can be found by us, replacing it is also possible. This is the code from the original article (the article has other methods worth reading too).

1
2
3
4
5
6
7
from json import JSONEncoder

def _default(self, obj):
    return getattr(obj.__class__, "to_json", _default.default)(obj)

_default.default = JSONEncoder().default
JSONEncoder.default = _default

In my own project I switched to another form: I want the original default to be called first, and only then our custom to_json. Also, for Decimal objects in the project the original function doesn’t seem to work, so I had to pull that out separately.

1
2
3
4
5
6
7
8
9
10
11
12
13
def _default(self, obj):
    """ Rewrites json.JSONEncoder.default in patch form,
        avoiding modifications at every json.dumps call
    """
    if isinstance(obj, Decimal):
        return float(obj)

    try:
        orig_default(self, obj)
    except:
        return getattr(obj.__class__, "to_json")(obj)

json.JSONEncoder.default = _default

** << Dive Into Python >> ** There are no constants in Python; everything can be changed if you try hard enough. This satisfies one of Python’s core principles: bad behavior should be overcome, not outlawed.

What Do You Think of This Method?

The benefit is that you can drop the cls=CustomJsonEncoder.

The accompanying downside is that your program has modified the original default in JSONEncoder. You see, in my program above I did my best to let the program call the original default first — even so, Decimal objects still had problems serializing and had to be pulled out separately.

Another problem: every class you define means you need to implement a to_json method. This kind of implicit convention is always a place where problems can arise.

Fortunately, what we should normally use is pickle, not json.

json should only be used in this one situation: when you want to communicate with programs in other languages.

Related Posts