parca: "failed to fetch any source profiles"

Sometimes I click on a blob in the timeline and it opens up a page saying “failed to fetch any source profiles”.

I have seen three different log messages that all seem to be associated with (different variants of?) this issue:

2019/11/28 18:03:48 : decompressing profile: gzip: invalid header
2019/11/28 18:04:46 : decompressing profile: unexpected EOF
2019/11/28 18:07:49 : parsing profile: unrecognized profile format

I guess it’s always possible that the scrape fails, but it would be better if the UI didn’t show a blob at that time.

About this issue

Original URL
State: closed
Created 5 years ago
Reactions: 1
Comments: 18 (12 by maintainers)

Commits related to this issue

Merge pull request #36 from parca-dev/storage-stripeSeries pkg/storage: Create stripeSeries to improve BenchmarkHeadQuerier_Select — committed to parca-dev/parca by brancz 3 years ago

Most upvoted comments

I finally managed to find that last bug, and all new e2e tests are passing with this patch: https://github.com/conprof/conprof/pull/113

Thank you so much everyone for bearing with me!

brancz on Dec 1, 2020

Quick update: It appears I have found part of the problem, there was an unsafe loading of data in the WAL. So the good news is that there is no corruption of the data on disk. Bad news is that for some reason new appends still yield this error, but when restarted those appends can be viewed without issues. I’m continuing to investigate.

I’ll open a PR as soon as I fix the remaining issue.

brancz on Nov 27, 2020

My apologies for taking so long to get back to this. Re-reading your https://github.com/conprof/conprof/issues/36#issuecomment-729810363 @inge4pres , are you saying you are trying to view trace type profiles? If so I think this is our mistake, as we don’t actually support viewing trace profiles, as we never ended up finishing https://github.com/conprof/conprof/pull/56 / #61 .

I think we should probably disable scraping trace profiles by default until we have this support implemented, to avoid confusing situations like this (if this is the case).

Regarding your last comment, do you have a sequence of steps to run to reproduce this? I can’t by just running a conprof instance from scratch, however, if I run conprof, collect some data, shut it down, and then start it back up, I can indeed reproduce some failed to fetch any source profiles errors. I’ll investigate why this is happening.

brancz on Nov 26, 2020