vectara-ingest: `pos_regex` does not behave as expected for indexing Salesforce knowledge base pages

You give the following example:

urls: ["https://www.hbs.edu"]
pos_regex: [".*hbs.edu.*"]

However, when I try the same as below, I get a grand total of 2 URLs:

urls: ["https://care.qumulo.com"]
pos_regex: [".*care.qumulo.com.*"]

About this issue

  • Original URL
  • State: closed
  • Created 3 years ago
  • Comments: 15 (8 by maintainers)

Most upvoted comments

Hey @mig281

  • Regarding docker version, you are using 19.03 and --build-arg should have been supported since much earlier so I am not sure why this is occurring. Perhaps we can find time for a quick live session to review? Or if you have the ability to just upgrade docker itself to a latest version and try again that might help?
  • Regarding the crawling, the use of “extraction: playwright” is aimed to address this issue. Do you know if the original crawl used “extraction: pdf” instead? (or maybe that was missing altogether)? If we reindex properly with the playwright option - that’s exactly what it’s aimed to do - wait until the JS renders the page and then extracts the content from there. However if you are confident playwright was used all the time, please send me a specific URL and I’m happy to test locally and debug as well.

Thanks for reporting. Investigating and will revert back.