openverse: Remove duplicated tags
Problem
Here’s an example on the frontend of an image with duplicated tags: https://search-staging.openverse.engineering/image/9fc28cae-ec9b-437a-a960-98f48db2cac8 (coffee, lol, lol).
Description
We should remove duplicated tags as part of the cleaning steps when loading ingested data.
For backfilling: Non-dated DAGs consume all the data on each run and ought to update old records on the next DAG run. For dated DAGs, we may need to write a backfill query.
Implementation
- 🙋 I would be interested in implementing this feature.
About this issue
- Original URL
- State: closed
- Created 5 years ago
- Comments: 15 (9 by maintainers)
It’s definitely nice to run the update through a DAG so we have a little more visibility, although not strictly necessary. I don’t see the branch you mentioned so I haven’t looked at your updates to the
batched_update, but FWIW I believe the existing DAG can be made to work without changes using thisselect_query:It’s a bit more nested than I’d like but that worked for me locally to update the 362 images with duplicate tags in about 14 seconds. When I ran just the SELECT in production for
imageit took a little over 2 hours, so I hope that shouldn’t be too terrible either 🤞Either way we’ll want to wait to run this until after confirming with @sarayourfriend that the catalog deployments are complete and DAGs can be unpaused.
This is done ✅
@stacimc That’s awesome! Better if we don’t require modifications to the DAG, glad that you found a query that works! I’ll mark this as blocked until the other work is finished.
I don’t know that I have the full context on
raw_tagsand why that took the structure it did, though I suspect it might be a dictionary because we occasionally had other values in there, like source and accuracy. Might be relevant for #431!That said though, if we could 1) ensure that the
MediaStoreclass is removing duplicate tags from the provider before saving and 2) remove the existing duplicates using the batched update DAG, I think that’s a great path forward!After some fiddling around, it looks like we are able to query for this!!
(Note that a duplicate
{'name': 'Female', 'provider': 'animaldiversity'},was removed in theunique_tags🎉)I’m going to run a count for this to see how many we have 😄