data-migration-tool: Undefined Offset: in Migration/Step/Eav/Data.php

Before I continue here:

For reference, I created this topic around the same time I added this issue last year: Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812

Edit 2021:

Please read the link above for the concern I raised:
Summary: Looking at the current commits (2021) for Step/Eav/Data.php it still has less code (alot old code removed… and hard-coded quantity $this->progress->start(7); vs OLDER commits: $this->progress->start($this->getIterationsCount()); ) as previous commits I mentioned in that topic; I wondered then, and still do: is this not the reason why Step/Eav/Data.php processing is giving us so much issues? Why were so much attribute processing removed, why are the dynamic getIterationsCount() not used anymore?

Original topic:

Preconditions

Magento 1.7.0.2 database
Magento 2.3.5 database, PHP 7.3

Steps to reproduce

Setup and install a clean new version of Magento 2.3.5
Install Migration Data tool v 2.3.5 composer require magento/data-migration-tool:2.3.5
Run setup:upgrade 4.1 Make sure that var/migration* is deleted so that you have a new migration from start. (in particular the file migration-tool-progress.lock). I would have assumed --reset does the same, but seems not.
Run bin/magento migrate:data path/to/config.xml that has configuration info from your source Magento 1 database and Magento 2 destination.

Expected result

A proper clean migration with no exceptions/errors/halts.

Actual result

Halts immediately on the following error 1.

Notice: Undefined offset: 4 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

Inside src/Migration/Step/Eav/Data.php on line 492: It seems that it is attempting to update the record attribute_group_id with the populated entry inside mapAttributeGroupIdsSourceDest[id], but it is not available inside my data…

$record['attribute_group_id'] = $this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']];

attribute_group_id for $record is 4, but there seems to be a missing mapAttributeGroupIdsSourceDest[4] corresponding map source/destination

In fact there seems to be more missing indexes when I debugged

[Screenshot, logs]

_{[2020-05-11 07:11:45] main.ERROR: Notice: Undefined offset: 4 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492
#0 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php(494): Magento\Framework\App\ErrorHandler->handler(8, ‘Undefined offse…’, ‘/var/www/html/m…’, 492, Array)
#1 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php(183): Migration\Step\Eav\Data->migrateCustomEntityAttributes()
#2 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/AbstractMode.php(82): Migration\Step\Eav\Data->perform()
#3 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/Data.php(124): Migration\Mode\AbstractMode->runStage(Object(Migration\Step\Eav\Data), ‘EAV Step’, ‘data migration’)
#4 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/Data.php(69): Migration\Mode\Data->runData(Array, ‘EAV Step’)
#5 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Console/MigrateDataCommand.php(59): Migration\Mode\Data->run()
#6 /var/www/html/mag_new/vendor/symfony/console/Command/Command.php(255): Migration\Console\MigrateDataCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(138): Migration\Console\MigrateDataCommand\Interceptor->___callParent(‘run’, Array)
#9 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(153): Migration\Console\MigrateDataCommand\Interceptor->Magento\Framework\Interception{closure}(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /var/www/html/mag_new/generated/code/Migration/Console/MigrateDataCommand/Interceptor.php(104): Migration\Console\MigrateDataCommand\Interceptor->___callPlugins(‘run’, Array, Array)
#11 /var/www/html/mag_new/vendor/symfony/console/Application.php(1001): Migration\Console\MigrateDataCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /var/www/html/mag_new/vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(Migration\Console\MigrateDataCommand\Interceptor), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /var/www/html/mag_new/vendor/magento/framework/Console/Cli.php(115): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/html/mag_new/vendor/symfony/console/Application.php(147): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/html/mag_new/bin/magento(23): Symfony\Component\Console\Application->run()
#16 {main} [] []}

Additional notes

** This is a clean installation of v2.3.5 Magento as described above, with no pending migration-tool-progress.lock inside var/
My possible related query [ - regarding a much smaller Eav/Data.php file](https://github.com/magento/data-migration-tool/issues/812)

Questions

Is the map code from the last few commits that should append array mapAttributeGroupIdsSourceDest with entries not missing some processing that used to be there? (See my related query above) (I have done previous migrations pre v.2.3 successfully using the same source Magento 1 database)
Is my issue that I need to map groups as well? Where do I do this?

About this issue

Original URL
State: open
Created 4 years ago
Reactions: 3
Comments: 18 (1 by maintainers)

Most upvoted comments

‘Undefined offset’ error was fixed in ea9cc9f21d3634dd3695fd88f9c869d9535cb6a6 and merged into 2.3-develop branch

victor-v-rad on May 13, 2020

When you do not have that fix applied, the data migration immediately fails, so that is not an option. At the point that fix is applied in Step/Eav/Data/migrateCustomEntityAttributes() the current implementation was created as follows by maintainer:

`if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) {
            continue;
        }`

The processing of the mapAttributeGroupIdsSourceDest beforehand in code seems to not have any group Ids mapped - no idea why.

There is this new updated commit 4 days ago (I assume this is the one you have) : #af40def29438fb that added another new condition:

if (empty($customAttributeIds)) { return; }

I seem to recall having this issue as well at one point where $customAttributeIds are somehow empty. See the history of the commits of Migration/Step/Eav/Data.php here : commits history.

The original issue discussed here was fixed in #ea9cc9f21d3634dd3695fd88f9c869d9535cb6a6.

Maybe you should revert one step commit back from the newest one and see if it changes anything?

My concern with this whole Step/Eav/Data.php file content was discussed here where I noticed some strange things: Why is the newest Eav Data file so much smaller.

Go read there, I do not even know if the current Data.php file we have are accurate - alot of processing code got removed in the commits I mention there. Also – why is this there $this->progress->start(7); when previous older commits had $this->progress->start($this->getIterationsCount()); - hardcoded quantity vs dynamic.

Finally I just want to say good luck with Eav migration - it is a nightmare when you later discover all your missing attribute data. Hints: (Attribute sets vs attribute data), catalog_product_entity_int, catalog_product_super_link etc.

ChameleonDevil on Jun 26, 2020

I had the same issue with a migration from magento 1.9.4.5 to 2.3.5 today.

This comment gave me the idea to change the name of the default product attribute set to “Default” in the magento1 database: https://github.com/magento/data-migration-tool/issues/112#issuecomment-242317826

So far it seems to fix my problem.

jerry-shopcommerce on Jun 15, 2020

Hi @yogeshkhasturi

Sorry just saw your message now.

Glad to hear that it fixed your first issue (of many 😛).

This eav_attribute_group issue that you mentioned as far as I remember only occurs when you try to run bin/magento migrate:data --reset again once a previous migration already migrated all the attributes. I seem to recall the migration tool confused with the IDs of the groups, so just restart from the beginning.

What I did for myself to test and restore migrations (I have alot of other issues still not working 😦 ) was sort of the following:

Install Magento 2.3.5 completely, setup modules etc, setup backend settings that you want. Once you have a working database before even migrating M1 data, make a backup of this M2 database state… because you will most likely restart alot of times because of migration issues. So create a full m2_full_install_before_migration.sql backup.
Create the same for Magento 1 database - until you are at the point where you can migrate without failures, you can restore this as well. NOTE: Delta migration is useless until you have successful migrate:data.
DROP M2 migrated database and restore from SQL in step 1.
VERY IMPORTANT: *Delete the var/migration ** entries in Magento 2 folder. These files are used by the tool it seems to keep track of where you stopped previously. So since the migration does not work 100% yet, just restart from the beginning. (this might also be why you have group issues). I always delete these files.
So a complete DROP/RECREATE, RESTORE M2 DB, setup:upgrade, migrate data from M1 copy (locally) takes around an hour in my case because I have done it so many times.
I would ignore migrate:delta for now if I were you, you will have to restart so many times. Get data working, make notes of issues that are specific to your instance, so that you could do things step by step. My current list before even getting to the point where I am able to run migrate:delta is roughly 15 things that I have to change manually in the database beforehand!

Good luck!

ChameleonDevil on May 27, 2020

@ChameleonDevil Thank you for the guidance this step was really new and seems to be helpful.

*VERY IMPORTANT: Delete the var/migration ** entries in Magento 2 folder. These files are used by the tool it seems to keep track of where you stopped previously. So since the migration does not work 100% yet, just restart from the beginning. (this might also be why you have group issues). I always delete these files.

yogeshkhasturi on May 27, 2020