marshmallow: skip_missing is gone, how to skip None elements ?

Sorry for the delayed response on this.

How about using a post-dump method to remove any null values? Something like:

from marshmallow import Schema, fields, post_dump

class BaseSchema(Schema):
    SKIP_VALUES = set([None])

    @post_dump
    def remove_skip_values(self, data):
        return {
            key: value for key, value in data.items()
            if value not in self.SKIP_VALUES
        }


class MySchema(BaseSchema):
    foo = fields.Field()
    bar = fields.Field()


sch = MySchema()
sch.dump({'foo': 42, 'bar': None}).data  # {'foo': 42}

Howdy! Thanks @sloria for this. Though you will run into problems if you have dicts or anything not hashable. Luckily I just needed to filter Nones out so I could change it to

if value is not None

class BaseSchema(Schema):
    SKIP_VALUES = set([None])

    @post_dump
    def remove_skip_values(self, data):
        return {
            key: value for key, value in data.items()
            if value not in self.SKIP_VALUES
        }

This also removes None values when the field has allow_none=True. To avoid this, we’d need to iterate over the Schema fields. But then it’s not that easy because the name of the fields may not match (due to attribute/dump_to/prefix…).

I didn’t try it but I think it could be done clean and easy in Marshaller.serialize() by adding a skip_none parameter:

https://github.com/marshmallow-code/marshmallow/blob/dev/marshmallow/marshalling.py#

if value is missing or (skip_none and value is None and getattr(field_obj, 'allow_none', False)):
    continue

OK, reading https://github.com/marshmallow-code/marshmallow/commit/a3908d36967ac30764ceb66c52c99357425fbdec, I realize this looks like the skip_missing you removed…

When serializing, marshmallow ignores missing attributes (attributes that raise an AttributeError). I’ve been happy with this when using @touilleMan’s umongo (MongoDB ODM) because when a value is missing, object.attribute raises AttributeError. Now, I’m trying to work with SQLAlchemy and missing fields are expressed as None so my API spits a lot of null values. I’m a total SQLAlchemy beginner so I may be missing something. But I had the same issue when dealing with simple objects. Looks like umongo’s handy attribute management is the exception rather than the rule and the solution should be on Marshmallow’s side.

This also removes None values when the field has allow_none=True. To avoid this, we’d need to iterate over the Schema fields. But then it’s not that easy because the name of the fields may not match (due to attribute/dump_to/prefix…).

What I finally did to fix this is to implement a custom object that will not be considered as None and that I can handle however I want:

NotProvided = object()

class BaseSchema(Schema):
    SKIP_VALUES = set([NotProvided])

    @post_dump
    def remove_skip_values(self, data):
        return {
            key: value for key, value in data.items()
            if value not in self.SKIP_VALUES
        }



class MySchema(BaseSchema):
    foo = fields.Field()
    bar = fields.Field()


sch = MySchema()
sch.dump({'foo': 42, 'bar': NotProvided}).data  # {'foo': 42}

I solve this trouble in my library (marshmallow intregration with mongoengine) by creating a mixin class which overload the _serialize method

class SkipEmptyClass(object):
    def __init__(self, *args, **kwargs):
        skip_empty = kwargs.pop('skip_empty', True)
        super(SkipEmptyClass, self).__init__(*args, **kwargs)
        self.skip_empty = skip_empty

    def _serialize(self, value, attr, obj):
        value = super(SkipEmptyClass, self)._serialize(value, attr, obj)
        if (self.skip_empty and
                (value is None or isinstance(value, (list, tuple)) and not value)):
            return missing
        return value

# Overload your marshmallow field class here
class String(SkipEmptyClass, fields.String):  # watch out inheritance order with MRO
    pass

see: https://github.com/touilleMan/marshmallow-mongoengine/blob/master/marshmallow_mongoengine/fields.py#L7

@Bachmann1234

Though you will run into problems if you have dicts or anything not hashable.

True; in these cases, you can just make SKIP_VALUES a list.

@makmanalp

I get AttributeError: “foo” is not a valid field for <obj> and such if a field is missing.

Would you mind posting code that produces this error?

I think before this hits 2.0 final, it might be a decent idea to enumerate all the possibilities

Are you planning on updating the docs? Making code changes?

I volunteer myself for the task unless anyone objects.

Thank you!

What if i need all my fields with None value have been ignored in schema? I need to create some Meta class? Here is my case: In frontend i have a backbone model and it fetch data from backend, backend returns this None values and when i save model, i get errors - “Field may not be null”, because i sent the data that i got from backend, i think it’s a weird behaviour

I think we need to have skip_if argument:

    name = fields.Str(skip_if=None)

@sloria If it’s ok i can create PR