openverse: Decode and deduplicate tags in the catalog with a TargetedReingestionDAG
Problem
In #4143, @obulat proposed to add new cleaning steps to the fix tags in the Catalog, but the option of including them in the Ingestion Server was declined in favor of using the batched_update DAG.
It is important to fix the encoding quickly because it can cause a gray Nuxt error screen for pages that contain tags with character sequences that cannot be URI-encoded (such as udadd from “ciudaddelassiencias”).
Description
We want to take the functions that were planned to include in said PR and translate them into parameters for this DAG. Given the complexity of the decoding transformation it might require some advanced functions of PostgreSQL, like a combination of pattern matching and PL/Python Functions.
In #1566, duplicated tags were previously removed, so we will apply the same solution given the decoding may cause new duplicates.
Additional context
About this issue
- Original URL
- State: open
- Created 2 years ago
- Comments: 20 (2 by maintainers)
@stacimc I’ve assigned you to this issue based on your expressed interest! There is no urgency to begin the work until you’re able 😄 I’ve moved it to the backlog as well until you’re ready to prioritize it.
@sarayourfriend I wouldn’t mind working on this, myself, given that I’ve already been a big part of the discussion! I’m wrapping up the last big (non-test) ingestion server removal PR this week, so as long as it’s not critical that work continue on this immediately I’m actually already invested/very interested in this issue. Happy to answer questions if @krysal really wants to take it though!
Oh, I would also add, it might be a good idea to split this into a couple PRs, and maybe use a simpler provider than Flickr to proof-of-concept the idea, before getting too bogged down with Flickr details, if necessary. On the other hand, the complexity of Flickr might be the best way to prove the concept in the first place, but multiple PRs will probably be easier to understand and review nonetheless!
@sarayourfriend is my understanding correct that you’re suggesting we use the approach I outlined in this comment, but also update the
upsertquery for all of our provider DAGs to:?
No objections from me if so, particularly because our load steps are not a performance bottleneck so I’m not particularly concerned about additional complexity there. This is the best minimal solution that keeps the tag history, IMO.
Although I agree we should retain data even if we are initially skeptical of its usefulness (deleted/modified tags were likely low quality tags, as you point out), the issue to me remains privacy/ethical concerns with deliberately retaining this historical data that is intentionally changed at source. I can’t think of a use case for this data that would avoid those issues. That being said I have no blocking objection to storing it in s3, and this solution is even a huge step forward, since it will bring tags in our catalog/API up to date. 👍
These are all really good points, and I think asking the question of what do we hope to get from this data is always a good place to start before we invest time into building out a way to store/track/record it.
I’m on board with the approach of making a minimally viable, storage-optimized approach for unblocking this specific issue. That’s similar to the plan we’ve had with #4610 and the data normalization efforts thus-far. As long as we’re preserving the old values somewhere, I think that’s best here!
I’m not opposed, although that raises some questions for me (for example, should
deletedtags be ignored in search? I’d argue that might be a good idea, since it’s very likely that if a tag was deleted from the record at source, it wasn’t a “good quality” tag. Similarly, should they be displayed on the frontend?).I also feel like this is a little different from, for example, the machine-generated tags below a certain accuracy threshold. In that case those are all perfectly valid tags, and the issue is that we’re making decisions about which ones we want to use/display. This case could be thought of as correcting data that has been invalidated upstream (in the way that when the title or description of a record changes at the source, we simply update that in the catalog). But… that is certainly a nuanced distinction! I certainly have no objection to keeping the data (beyond noting that it’ll make the upsert a little more complicated, which is totally fine). Those questions about deleted tags in display/search can also be answered later. There’s no regression from current behavior 😃 And the great thing about keeping the data is you can always change your mind later, which is obviously not the case if we delete it.
As far as a
tag_historytable or something more complicated, I think we should take a step back and do a full project proposal/etc. For example, immediately I’m questioning if we’re going to do that, whether we should also be keeping history for other fields that can change upstream. And if not, why?@sarayourfriend if the only way to safely determine what the tags should be is to reingest, then I think that’s fine.
The Flickr reingestion DAG is absolutely not reliable for fixing this for a variety of reasons, but I do think we could implement something like you’ve described using a lot of existing code from the ProviderDataIngester and the loader tasks.
Actually I would look into extending the existing FlickDataIngester. You can add
create_preingestion_tasksandcreate_postingestion_taskshooks (see iNaturalist for an example) to the FlickrDataIngester for creating a temp table where you select theforeign_identifiers of the rows you want to update, in the style of thebatched_updateDAG, and eventually drop that table. Then modify the new ingester class to select batches of ids from this table and query the individual record endpoint from Flickr. You might need to tinker with the base class a little but I think this should all be doable pretty easily. The advantage is that records all get committed through the MediaStore, we’d get TSVs in S3 just like a regular provider DAG, etc. You’d just register a new DAG inprovider_workflows.pywith the new ingester class.The only thing that would be an issue is that the
upsertstep of our provider DAGs currently merges tags rather than overwriting them 🤔 I brought this up on the linked PR too… I’m interested in changing the upsert strategy for tags to overwrite tags from the provider. Obviously we’d have to be careful to ensure we’re still merging tags with a differentproviderfield, so that we don’t overwrite machine-generated tags for example, but this honestly feels like a bug as implemented. If we could make that upsert a little more sophisticated then I think this would work pretty simply. @AetherUnbound @WordPress/openverse-catalog any thoughts/objections to that in particular?I tried to run the DAG again today, and it turns out RDS does not support the PL/Pythonu extension 😞
RDS does support plperl, plrust, plpgsql, and plv8. Of those, plv8 (v8 being the same JS engine as Chrome), might be the most proximate for this use case, but I’ll see. It might be that the reingestion of these records is the easiest, most reliable way to do it.