data-migration-tool: Integrity Check failed (FK_CATALOG_*

@craigcarnell

Please see this gist I have started. I have labeled for you the individual queries that will fix your Foreign key constraints failures.

For example for this error:

[2017-01-05 12:39:50][ERROR]: Foreign key (FK_CATALOG_EAV_ATTRIBUTE_ATTRIBUTE_ID_EAV_ATTRIBUTE_ATTRIBUTE_ID) constraint fails. Orphan records id: 112,154,155,337,338,339,340,341,342,343,345,346,347,348,349,350 from `catalog_eav_attribute`.`attribute_id` has no referenced records in `eav_attribute`

Search in the file for the comment FK_CATALOG_EAV_ATTRIBUTE_ATTRIBUTE_ID_EAV_ATTRIBUTE_ATTRIBUTE_ID and you’ll find this query on the following line:

DELETE FROM `catalog_eav_attribute` WHERE `attribute_id` NOT IN (SELECT `attribute_id` FROM `eav_attribute`);

@noindie

Integrity constraint violation: 1062 Duplicate entry '1-Migration_Default' for key 'EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME'

This error will occur when there is already the attribute set “Migration_Default” in your Magento 2 database, and you are attempting to run the Eav Step in Data mode over again. Most likely you have already ran this step, and your EAV attributes are already migrated.

If you are happy with them, you could just remove the “EAV Step” from your configuration file and move onto the next step in the migration process. If you see that there are issues with your EAV attributes or attribute sets, then restore your Magento 2 database from backup prior to migration, revise the configuration, and try again.

    <bulk_size>0</bulk_size>

Sorry I am still unsure what this configuration does for now.

@noindie

regarding <bulk_size>16M</bulk_size> it’s the number of records that are processed at a time, if not defined the default is 100 and the maximum allowed is 50000. it’s used here

Check if this can solve your issue https://github.com/magento/data-migration-tool/issues/114#issuecomment-231965717

I am in the very same process of my own data migration. A small recommendation that I came up with just this morning is to run your configuration with just the data integrity configurations until you are able to clean up all errors that are output.

In my config.xml file I am currently running the data migration tool with these settings for my data mode:

<steps mode="data">
        <step title="Data Integrity Step">
            <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
        </step>
        <step title="EAV Step">
            <integrity>Migration\Step\Eav\Integrity</integrity>
        </step>
        <step title="Custom Customer Attributes Step">
            <integrity>Migration\Step\CustomCustomerAttributes\Integrity</integrity>
        </step>
        <step title="Customer Attributes Step">
            <integrity>Migration\Step\Customer\Integrity</integrity>
        </step>
        <step title="Map Step">
            <integrity>Migration\Step\Map\Integrity</integrity>
        </step>
        <step title="Url Rewrite Step">
            <integrity>Migration\Step\UrlRewrite\Version11410to2000</integrity>
        </step>
        <step title="Log Step">
            <integrity>Migration\Step\Log\Integrity</integrity>
        </step>
        <step title="Ratings Step">
            <integrity>Migration\Step\Ratings\Integrity</integrity>
        </step>
        <step title="SalesOrder Step">
            <integrity>Migration\Step\SalesOrder\Integrity</integrity>
        </step>
        <step title="ConfigurablePrices step">
            <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
        </step>
        <step title="OrderGrids Step">
            <integrity>Migration\Step\OrderGridsEE\Integrity</integrity>
        </step>
        <step title="Tier Price Step">
            <integrity>Migration\Step\TierPrice\Integrity</integrity>
        </step>
        <step title="SalesIncrement Step">
            <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
        </step>
        <step title="VisualMerchandiser Step">
            <integrity>Migration\Step\VisualMerchandiser\Integrity</integrity>
        </step>
    </steps>

Until I have all errors cleaned up, I will just continue to rerun the integrity checks on my database and no actual migration of data will be attempted yet.

Yes all orphaned data should (probably) just be deleted. For example, if there was a name value for customer 1234, but customer 1234 no longer exists, in most common cases it’s rather useless information at this point and is just cluttering up your database.

Whenever the data migration tool complains about “Source fields not mapped”, it is telling you that “Hey there is a column on the original database, that isn’t on the new database, and you haven’t told me what to do with it”. If you see this very same message, but instead “Destination fields not mapped”, it would be vice versa (an unknown column on your Magento 2 database).

If it is from a Magento 1 extension that you do not plan to have on your Magento 2 site, then it can most likely be ignored. To ignore a column you want to find in your configuration file this line: <map_file>etc/ee-to-ee/1.14.2.0/map.xml.dist</map_file>. Copy paste this map.xml.dist to map.xml, and update your configuration file to reflect this change.

Inside this file, you’ll find all table (they call it a document) and table column (they call it a field) configurations for your Magento 1 database (they call it the Source) and Magento 2 database (they call it the destination).

For example, to ignore the entire msp_cashondelivery_foreign table which you have mentioned, you would add this XML configuration to map.xml:

<map>
    <source>
        <document_rules>
            <ignore>
                <document>msp_cashondelivery_foreign</document>
            </ignore>
        </document_rules>
    <source>
</map>

Every single table, and column, from both your Magento 1 database, and Magento 2 database, must be mapped out.

Yes, this kinda sucks, but it forces you to make sure you are damn sure you know what you’re doing with your data from one database to the next. The Data Migration Tool has provided for us, all the configuration necessary for native Magento 1 and Magento 2 database structure. If you have made any changes to your Magento 2 database already, those changes will have to be mapped out in the destination configuration section of this file as well.