PynamoDB: Querying GlobalSecondaryIndex doesn't return all attributes

I just got a few new gray hairs trying to debug why the heck LSI queries were not projecting non-key attributes. In desperation, I started browsing all issues returned for the keyword “projection,” and I ended up here.

@DNakevski3Shape, nearly a year later, is there now a recommended approach for single-table designs with a single base class that defines keys, indexes, and common attributes and methods?

Given the following example code

class LSI1Index(LocalSecondaryIndex):
    class Meta:
        index_name = f"{settings.dynamodb_table_name}_LSI1"
        projection = AllProjection()

    PK = UnicodeAttribute(hash_key=True)
    LSI1SK = UnicodeAttribute(range_key=True)


class BaseDynamoModel(Model):
    class Meta:
        table_name = settings.dynamodb_table_name
        region = settings.aws_region

    PK = UnicodeAttribute(hash_key=True)
    SK = UnicodeAttribute(range_key=True, default_for_new=SK_ROOT)

    LSI1SK = UnicodeAttribute()
    LSI1_index = LSI1Index()

class DocumentInDB(BaseDynamoModel):
    data = JSONAttribute()

How can I “instruct” the index to use the correct model and project all attributes when querying DocumentInDB.LSI1_index?

The subclassing approach I have taken seems to work because the base model “owns” the index and so all the attributes of the sub classes seem to be accessible.

I think that this should be the go-to approach. When you have many models using the same table and indexes, the other one quickly becomes inconvenient and hard to maintain due to code duplication.

But the Index owner is set at the attribute assignment so I am not sure how to alter that from a child class.

Trying to do a naïve reproduction, it seems to work correctly:

from pynamodb.attributes import UnicodeAttribute
from pynamodb.indexes import GlobalSecondaryIndex, AllProjection
from pynamodb.models import Model


class FooValueIndex(GlobalSecondaryIndex):
    class Meta:
        index_name = 'value_index'
        read_capacity_units = 1
        write_capacity_units = 1
        projection = AllProjection()

    value = UnicodeAttribute(hash_key=True)


class FooModel(Model):
    class Meta:
        table_name = 'same_name'
        host = 'http://localhost:8000'
        aws_access_key_id = 'dummy'
        aws_secret_access_key = 'dummy'
        aws_session_token = 'dummy'
        read_capacity_units = 1
        write_capacity_units = 1
    hk = UnicodeAttribute(hash_key=True)
    value = UnicodeAttribute()

    value_index = FooValueIndex()


# now same, but Bar


class BarValueIndex(GlobalSecondaryIndex):
    class Meta:
        index_name = 'value_index'
        read_capacity_units = 1
        write_capacity_units = 1
        projection = AllProjection()

    value = UnicodeAttribute(hash_key=True)


class BarModel(Model):
    class Meta:
        table_name = 'same_name'
        host = 'http://localhost:8000'
        aws_access_key_id = 'dummy'
        aws_secret_access_key = 'dummy'
        aws_session_token = 'dummy'
        read_capacity_units = 1
        write_capacity_units = 1
    hk = UnicodeAttribute(hash_key=True)
    value = UnicodeAttribute()

    value_index = BarValueIndex()

# and now let's test

FooModel.create_table()
BarModel.create_table()  # pointless, but ...

foo = FooModel()
foo.hk = 'foo'
foo.value = '42'
foo.save()

result = FooModel.get('foo')
print(type(result), isinstance(result, FooModel))
result = BarModel.get('foo')
print(type(result), isinstance(result, BarModel))

(result, *_) = FooModel.value_index.query('42')
print(type(result), isinstance(result, FooModel))
(result, *_) = BarModel.value_index.query('42')
print(type(result), isinstance(result, BarModel))

However, the behavior observed by @DNakevski3Shape can be seen by changing:

class BarModel(Model):
     ...

-   value_index = BarValueIndex()
+   value_index = FooValueIndex() 

This is indeed what happens – you’re reusing GlobalIndex1 for two models. However, when you use an index class in a model, the model takes “ownership” of that index class. In the short term, we should produce an error when an index class is already owned by a different model. In the long term, perhaps we can store the association with the instance of the index and not by modifying the index class. This would mean, however, that the following wouldn’t be possible anymore (I’m surprised it’s possible now… 🤷 ):

(result, *_) = GlobalIndex1.query(...)
# Shall 'result' be Stimulus or Result? Apparently it'll be the last of them to be defined.