harbor: Error in Tag Retention Log "Object is not found"

Expected behavior and actual behavior: I have set up a Tag Retention Policy for housekeeping the images. The Dry Run was working fine and as soon as it was done, I got the information from the Policy log, which images will be kept or deleted. When executing the policy, the images were deleted based on the rule, but in the policy log I see the following error message:

{"errors":[{"code":"NOT_FOUND","message":"{\"code\":10010,\"message\":\"object is not found\",\"details\":\"a83ac447704ac5c98c720eef\"}"}]}

I expected a log, where I get the information about the deleted image.

Steps to reproduce the problem: Step 1: Create a Tag Retention policy with the following configuration.

Step 2: Run the policy

Step 3: After successful run, check the policy log.

Versions: Please specify the versions of following systems.

harbor version: 2.4

Additional context:

I also checked our log files, if we can see any errors but everything was successful based on the log information.

2022-03-25 08:03:29	
jobservice.log
Mar 25 08:03:29 <host> jobservice[25099]:  Repository: <project>/<image> 

2022-03-25 08:03:31	
jobservice.log
Mar 25 08:03:31 <host> jobservice[25099]: 2022-03-25T07:03:31Z [INFO] [/pkg/retention/job.go:98]: Load 13 candidates from repository <project>/<image>

2022-03-25 08:03:31	
core.log
Mar 25 08:03:31 <host> core[25099]: 2022-03-25T07:03:31Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'RetentionWebhook' on topic 'TAG_RETENTION': TaskID-1325630 Status-SUCCESS Deleted-[<project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[]] OccurAt-2022-03-25 07:03:31

2022-03-25 08:04:33	
jobservice.log
Mar 25 08:04:33 <host> jobservice[25099]:  Repository: <project>/<image> 

2022-03-25 08:04:35	
jobservice.log
Mar 25 08:04:35 <host> jobservice[25099]: 2022-03-25T07:04:35Z [INFO] [/pkg/retention/job.go:98]: Load 13 candidates from repository <project>/<image>

2022-03-25 08:04:35	
core.log
Mar 25 08:04:35 <host> core[25099]: 2022-03-25T07:04:35Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'RetentionWebhook' on topic 'TAG_RETENTION': TaskID-1325631 Status-SUCCESS Deleted-[<project>:<image>:[]] OccurAt-2022-03-25 07:04:35

2022-03-25 08:04:49	
jobservice.log
Mar 25 08:04:49 <host> jobservice[18398]:  Repository: <project>/<image> 

2022-03-25 08:04:51	
jobservice.log
Mar 25 08:04:51 <host> jobservice[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/retention/job.go:98]: Load 13 candidates from repository <project>/<image>

2022-03-25 08:04:51	
core.log
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:51	
core.log
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'Replication' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:51	
core.log
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:51	
core.log 
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'DeleteArtifactWebhook' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:52	
core.log
Mar 25 08:04:52 <host> core[18398]: 2022-03-25T07:04:52Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'RetentionWebhook' on topic 'TAG_RETENTION': TaskID-1325632 Status-SUCCESS Deleted-[<project>:<image>:[]] OccurAt-2022-03-25 07:04:52

Are you aware of this error message? If yes, how can we solve it?

Thank you in advance!

Alexander Barth (alexander.barth@mercedes-benz.com) on behalf of Daimler TSS, Provider Information

About this issue

Original URL
State: open
Created 2 years ago
Comments: 20 (7 by maintainers)

Most upvoted comments

I have a similar issue. Entries older than 2 days give me the aforementioned error. The other two show log content, as expected. Either I am doing something wrong or have missed a configuration or something else is wrong. In my case, harbor runs on a separate system, singel instance. But I think the UI should only show, what is available. If the log is not available (e.g. because of log rotation), then it should also not be present in the UI. There’s not much value int it anyway, if one cannot access the logs.

Jeansen on Jan 29, 2023

This issue can be closed because we found a solution to fix the above mentioned error message. Our solution is, that we created an NFS share to store the job logs on this share instead on a local VM.

Alexander Barth (alexander.barth@mercedes-benz.com) on behalf of Mercedes-Benz Tech Innovation GmbH, Provider Information

AlexBarth13 on Jul 5, 2022