aerich: Add ManyToManyField will break migrate

I took a bit of a closer look at this and it seems that there’s a whole type of action missing in the migration methods. When a key is change (i.e not add or remove) the script will always break since that case isn’t treated at all. If you skip all “changes” from the diff, it doesn’t crash, but also misses the additions or removes from m2m related entries of the model.

I didn’t have time to test more in depth, but it seemed to me the issue isn’t with the differ but with the serialization of the models since the differ isn’t able to discriminate the difference between a change in some m2m field and an addition of a new m2m field.

But anyway, bumping this thread, it is a big breaking issue.

Related code as below:

class Sys_Role(MyAbstractBaseModel):
    """角色表"""
    role_code = CharField(max_length=128, description="角色代码", null=False, unique=True)
    role_name = CharField(max_length=128, description="角色名称", null=False, unique=True)
    status = SmallIntField(description="角色状态", null=False, default=0)
    users: ManyToManyRelation[Sys_User]
    apis: ManyToManyRelation[Sys_Api]
    ## menus: ManyToManyRelation["Sys_Menu"]

    class Meta:
        table = "sys_role"
        table_description = "角色表"

    def __str__(self):
        return f'{self.name}({self.code})'


class Sys_Menu(MyAbstractBaseModel):
    """菜单组件表"""
    path = CharField(max_length=128, description="路由路径", null=False, unique=True)
    name = CharField(max_length=128, description="路由名称", null=False, unique=True)
    parent_id = BigIntField(description="父级菜单ID", null=False, default=0)
    full_path = CharField(max_length=256, description="路由全路径", null=False)
    sort = SmallIntField(description="排序", null=False, default=0)
    menu_type = SmallIntField(description="菜单类型", null=False, default=0)
    hidden = BooleanField(description="是否隐藏", null=False, default=False)
    permission = CharField(max_length=128, description="路由权限", null=False, default="", unique=True)
    icon = CharField(max_length=128, description="菜单图标", null=False, default="")
    title = CharField(max_length=128, description="菜单标题", null=False)
    ## roles: ManyToManyRelation[Sys_Role] = ManyToManyField('system.Sys_Role', related_name="menus", through="sys_role_menu")

    class Meta:
        table = "sys_menu"
        table_description = "菜单组件表"

    def __str__(self):
        return f'{self.full_path}[{self.name}]'

After I added new ManyToManyField (sys_role_menu, uncomment those lines)，and run aerich migrate，it throws an AttributeError Exception:

Traceback (most recent call last):
  File "/data/py-nwci/venv/bin/aerich", line 8, in <module>
    sys.exit(main())
  File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/cli.py", line 298, in main
    cli()
  File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/cli.py", line 41, in wrapper
    loop.run_until_complete(f(*args, **kwargs))
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/cli.py", line 95, in migrate
    ret = await Migrate.migrate(name)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/migrate.py", line 130, in migrate
    cls.diff_models(cls._last_version_content, new_version_content)
  File "/data/py-nwci/venv/lib/python3.7/site-packages/aerich/migrate.py", line 208, in diff_models
    table = change[0][1].get("through")
AttributeError: 'str' object has no attribute 'get'

Just came across this now…

Had to hack my way around it, by manually changing

The following lines in aerich/migrate.py @ diff_models, to: Line 232: Original: if change[0][0] == "db_constraint": New: if isinstance(change[0], bool) or change[0][0] == "db_constraint":

Line 235: Original: table = change[0][1].get("through") New:

if isinstance(change[0][1], str):
   for new_m2m_field in new_m2m_fields:
          if new_m2m_field['name'] == change[0][1]:
              table = new_m2m_field.get('through')
              break
else:
    table = change[0][1].get("through")

This then allows you to actually migrate M2M fields…

We shouldn’t really have to hack this together, though, for it to work.

Another small issue I’ve found, is if you’re using the through kwarg for a M2M, Aerich correctly uses the right table name when doing initial migrations (the one specified in through), but then seems to ignore the table specified in through later on when you try create new migrations.

@long2ice - just checking you’ve seen this issue?

Still not fixed. aerich==0.6.2.

If someone looks for more production ready workaround here it is:

   # overload in Tortoise model, note: aerich may blow up at model which has M2M field declared
   # OR at destination model(!!)

    @classmethod
    def describe(cls, serializable: bool = True) -> dict:
        result = super().describe(serializable)
        m2m_order = ('location', 'slaves', 'devicegroups')  # << here put your M2M fields names
        assert set(m2m_order) == set(cls._meta.m2m_fields)
        result['m2m_fields'] = [
            cls._meta.fields_map[name].describe(serializable)
            for name in m2m_order
        ]
        return result

Workaround explanation

As @dstlny said, the issue is in m2m fields compare. Depending on many things the order of M2M fields may change between migrations (but not have to, I hit this issue after adding third M2M relation to the same model). So to make it working without hacking aerich code or migrations data in aerich database table, I tried to make sure that order of M2M fields will be always the same and newer fields always appears after existing ones.

After some time of debugging I found how to achieve this. Aerich is taking model state using describe() method from Tortoise model, so to make sure that differ is always getting same order this method must return M2M fields in the same order. It can be done manually by hardcoding the order. I’ve add an assert line to make sure that nothing wrong happens when someone adds new relation to my model (by adding M2M field in the model or relation in another model to the model).

Generic fix proposal

At the aerich level I think there should be some code which orders new_m2m_fields list basing on the order in old_m2m_fields (somewhere in diff_models method in migrate.py). Matching by field names should solve most of issues. It would be great if as a fallback aerich try to match missing old list fields by relation destination and if it’s found on new list consider this as relation field name change (like Django do). Reordered new_m2m_fields list can be passed to differ along with current old_m2m_fields list.

Change 244 line in migrate.py

table = change[0][1].get("through")

to

if isinstance(change[0][1], str):
    for new_m2m_field in new_m2m_fields:
        if new_m2m_field['name'] == change[0][1]:
            table = new_m2m_field.get('through')
            break
else:
    table = change[0][1].get("through")

Hi, could you make a PR?

Look like permission denied((

Why? just fork and make pull request

0.7.1 has bug too. Using dictdiffer is a problem.