gitea: Indexer returns no results for some terms

Gitea version (or commit ref): 1.11.0-rc1
Git version: 2.24.1
Operating system: Debian testing
Database (use [x]):
- PostgreSQL
- MySQL
- MSSQL
- SQLite
Can you reproduce the bug at https://try.gitea.io:
- Yes (provide example URL)
- No
- [X ] Not relevant

Description

I enabled the indexer. It has been running for couple days since then. I am able to search and get some results but some results return no results by the Code search page meanwhile I can get 10s on results for with grep

For the term “tool_set” in the Code search page I get No source code matching your search term found.

Grepping the same code base (eve after deleting the comment lines)

find -type f -name "*.py" -exec grep -i 'tool_set' {} \; |sed '/#/d' |wc -l 44

ini

[indexer]
REPO_INDEXER_ENABLED = true
ISSUE_INDEXER_PATH: indexers/issues.bleve
REPO_INDEXER_PATH: indexers/repos.bleve
UPDATE_BUFFER_LEN: 20
MAX_FILE_SIZE: 1048576

About this issue

Original URL
State: closed
Created 4 years ago
Comments: 27 (14 by maintainers)

Most upvoted comments

Re-checked https://github.com/go-gitea/gitea/issues/9190#issuecomment-571563226 behavior with latest upstream version 1.12.0+dev-174-g5b17bb8f3 Seems working now! Repo index was updated after git push

vvrein on Jan 25, 2020

I think I’ve found an important bug! But it should only manifest itself as repos not being updated (creation of indexes from scratch should not be affected).

As for the error message in my instance:

2020/01/22 21:04:05 ...ndexer/code/queue.go:39:processRepoIndexerOperationQueue() [E] indexer.Index: exit status 1

I’ve been debugging and it turns out this error is expected as I have one corrupt repo, so git show-ref -s returns… a silent exit status of 1. I believe this should not affect the indexing of other repos, because the error is logged and the indexer just continues processing its queue.

About the bug I’ve mentioned, I’ll post a PR momentarily.

guillep2k on Jan 24, 2020

Btw is there a way to force the indexer while gitea is running?

If by force you mean rebuild all, no, there isn’t. But files are re-indexed with each commit (only the affected files, the whole file is re-indexed, not just the diff).

guillep2k on Jan 23, 2020

My log configuration in app.ini:

[log]
MODE             = file
MAX_DAYS         = 15
LEVEL = Info
ROUTER           = file
ROUTER_LOG_LEVEL = Trace
STACKTRACE_LEVEL = Error
XORM = file
REDIRECT_MACARON_LOG = true

[log.file.xorm]
FILE_NAME = xorm.log

(It’s a little redacted, so maybe not all options make sense)

This separates the SQL (XORM) log from the other logs, making everything cleaner. I’ve also set up a trace to every error, so I know exactly where every log is produced.

To get a meaningful log I stopped Gitea and deleted the repos.bleve directory to force the system to rebuild them when restarted. You’ll know it finished when it stops growing (which is not necessarily when the log says it does… in fact my log was not useful about that).

Then I’ve edited a file using the web UI, and when the indexer attempted to do its thing, it crashed.

(NOTE: your paste doesn’t say much, unfortunately)

guillep2k on Jan 23, 2020