magento2: Product images are being duplicated on import
When importing product data the product images are being duplicated for each import run.
Preconditions
- Magento 2.4-develop
- PHP 7.3
- MySQL 5.7.25
Steps to reproduce
- composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition
<install dir> - cd
<install dir> - ./bin/magento setup:install --backend-frontname=? --db-host=? --db-name=? --db-user=? --db-password=? --base-url=? --admin-user=? --admin-password=? --admin-email=? --admin-firstname=? --admin-lastname=?
- mkdir var/import-images
- Add images
test-1.jpg,test-2.jpg,test-3.jpgandtest-4.jpginside the path<install dir>/var/import-images. Any valid jpeg will do. - Import the product CSV data below twice
"sku","product_online","website_id","store_view_code","attribute_set_code","product_type","categories","name","description","short_description","visibility","price","special_price","special_price_from_date","special_price_to_date","url_key","meta_title","meta_keywords","meta_description","display_product_options_in","msrp_display_actual_price_type","additional_attributes","qty","out_of_stock_qty","use_config_min_qty","is_qty_decimal","allow_backorders","use_config_backorders","min_cart_qty","use_config_min_sale_qty","max_cart_qty","use_config_max_sale_qty","is_in_stock","use_config_notify_stock_qty","manage_stock","use_config_manage_stock","use_config_qty_increments","qty_increments","use_config_enable_qty_inc","enable_qty_increments","is_decimal_divided","deferred_stock_update","use_config_deferred_stock_update","related_skus","crosssell_skus","upsell_skus","custom_options","bundle_price_type","bundle_sku_type","bundle_price_view","bundle_weight_type","bundle_values","associated_skus","base_image","base_image_label","small_image","small_image_label","thumbnail_image","thumbnail_image_label","swatch_image","swatch_image_label","additional_images","additional_image_labels","configurable_variations"
"some-test-product",1,1,,"Default","simple","Default Category/Some Category/Some Test Product","Some test product","This is just some test product",,"Catalog, Search",2999,,,,"some-test-product","Some test product",,,"Block after Info Column","Use config",,100,0,1,0,0,1,1,0,0,1,1,1,0,1,1,0,1,0,0,0,1,,,,,,,,,,,"test-1.jpg",,"test-1.jpg",,"test-1.jpg",,"test-1.jpg",,"test-3.jpg",,
"another-test-product",1,1,,"Default","simple","Default Category/Some Category/Some Test Product","Some test product","This is just some test product",,"Catalog, Search",2999,,,,"another-test-product","Another test product",,,"Block after Info Column","Use config",,100,0,1,0,0,1,1,0,0,1,1,1,0,1,1,0,1,0,0,0,1,,,,,,,,,,,"test-2.jpg",,"test-2.jpg",,"test-2.jpg",,"test-2.jpg",,"test-4.jpg",,
Import settings
- Entity Type: Products
- Import Behavior: Add/Update
- Stop on Error (default)
- Allowed Errors Count: 10 (default)
- Field separator: , (default)
- Multiple value separator: , (default)
- Empty attribute value constant:
__EMPTY__VALUE__(default) - Fields Enclosure: Not checked (default)
- Images File Directory: var/import/images/import-images
Expected result
Products end up with each product image only attached once.
Actual result
Products end up with all product images duplicated.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Reactions: 8
- Comments: 78 (26 by maintainers)
Commits related to this issue
- product images while exporting does not override previous image having same name in other folder — committed to magento/magento2 by deleted user 6 years ago
Temporary solution to this issue:
1. Edit this file:
/vendor/magento/module-catalog-import-export/Model/Import/Uploader.php:1342. Modify init() function: Change line 134 in the init() function from
to
Result: This will replace the product images instead of just renaming and duplicating them.
Keep in mind: This is just a temporary solution and with the next magento-update you have to do this again. To make the changes permanent you have to create a module that overrides this file.
This is my first comment on github. So please be kind with your feedback - thanks!
For the Magento dev team:
/vendor/magento/module-catalog-import-export/Model/Import/Uploader.phpgetNewFileName()and just renameing it if the filesize has changed.Hi, @hostep this issue still exists in Magento 2.3.5-p1 — I was able to replicate it.
edit: all that those changed did was break all the PRs and patches 😉
edit2: I’m using this PR with success now: https://github.com/magento/magento2/pull/26713
@erfanimani Yes, we indeed used a combination of the solutions (with some minor edits). Really odd that Magento doesn’t look at the fixes because the importer is a vital part of most webshops.
@dmanners Both actually — this issue is a duplicate of #14398
But then there are two open PRs (I referenced them both in case someone wants to create a patch for it):
Hi @simonworkhouse. Thank you for your report. The issue has been fixed in magento/magento2#26713 by @PieterCappelle in 2.4-develop branch Related commit(s):
The fix will be available with the upcoming 2.4.2 release.
Get the patch corresponding to the pull request : https://patch-diff.githubusercontent.com/raw/magento/magento2/pull/26713.patch And apply it to
magento/catalog-import-exportcomposer dependency using https://github.com/cweagans/composer-patches@erfanimani OK, I can confirm that the images are now no more duplicated on import. That is very good… BUT… The Add/Update does not delete images either. Example: I import a product (sample_1) with 8 images => fine Add/Import the product sample_1 with 2 images … the product is not correctly updated, still 8 images in the db.
Thank you @erfanimani
With a combination of the solutions, the import works very well.
Actually, there’s a better PR here, based on mine but it also deals with image deletion: https://github.com/magento/magento2/pull/21855
I would like to share some point about the issue. It happens in Magento\CatalogImportExport\Model\Import\Product.php line 1772. If add a variable for exists images in mapper like _media_is_disabled, it should be fix it but the code try to match it with path. That why it doesn’t work.
MDVA-31021 patch available in Quality Patches Tool
I can confirm #26713 is working fine on 2.3.5-p1. Since this seems to be an endless issue, here the link to the correct application of this patch via composer: https://devdocs.magento.com/guides/v2.4/comp-mgr/patching.html
@PsmIndia @jalung Check PR https://github.com/magento/magento2/pull/26713 The PR comes with image hash compare logic, which is working fine (in my workflow) for the community version.
It does not work for most of us because we have in our import CSV the base name of the image, for example;
base_image_78989.jpgbut magento’s existing images are named/b/a/base_image_78989_10.jpg. Unless there is a comparison between these two names on updating the product, this issue will persist.2.3.5-p1 duplicate images - issue not fixed
I’ve seen some change in the codebase of Magento 2.3.5 which might fix this issue. Would be great if somebody could verify this after 2.3.5 is officially released on 28 April.
Product-revised-version.txt Fixed case where CSV row has new and existing images.
the commit that @simonworkhouse referred to, was the suicide bomber in our case as well. Patching that to the previous implementation fixed it. In my opinion this really should be reverted.
Product-revised-version.txt
version 2.3.3 Atttached is the patch that is used to fix the issue in our use case. Magento import is expecting images to have the dynamic paths it created on export. However, the CSV is hand crafted and the image names are plain; “blablabla.jpg” as opposed to “/b/l/a/blablabla_60.jpg”. This patch works repeatedly because the CSV file is from the same feed. The patch basically strips off the “/b/l/a/” and “_60” and makes a match to the existing image(s) for that product.