airbyte-connectors: Getting 404 with azure-repos-source
I am using the azure-repos-source connector locally on my machine. I have set the values for project, organization and access_token in the config.json. At the beginning everything works fine and data gets transferred but after a while I am getting:
{"log":{"level":"ERROR","message":"Encountered an error while reading stream repositories: Request failed with status code 404","stack_trace":"Error: Request failed with status code 404\n at createError (/Users/c.haumann/airbyte-connectors/node_modules/axios/lib/core/createError.js:16:15)\n at settle (/Users/c.haumann/airbyte-connectors/node_modules/axios/lib/core/settle.js:17:12)\n at IncomingMessage.handleStreamEnd (/Users/c.haumann/airbyte-connectors/node_modules/axios/lib/adapters/http.js:322:11)\n at IncomingMessage.emit (node:events:525:35)\n at endReadableNT (node:internal/streams/readable:1359:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"},"type":"LOG"} {"log":{"level":"ERROR","message":"Encountered an error while reading from source: VError: API responded with status 404 - {\"jse_shortmsg\":\"API responded with status 404\",\"jse_info\":{\"req\":{\"baseUrl\":\"https://dev.azure.com/vwfs/GisAtWeb/_apis\",\"url\":\"git/repositories/c9526643-4ee7-498a-97e8-92b9a50f5229/annotatedtags/0fd17aca2a0e800979e0c4f83bc71350479ac23e\",\"method\":\"GET\",\"params\":{\"api-version\":\"6.0\"}},\"res\":{\"status\":404,\"headers\":{\"cache-control\":\"no-cache, no-store, must-revalidate\",\"pragma\":\"no-cache\",\"content-length\":\"329\",\"content-type\":\"application/json; charset=utf-8; api-version=6.0\",\"expires\":\"-1\",\"p3p\":\"CP=\\\"CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT\\\"\",\"set-cookie\":[\"VstsSession=%7B%22PersistentSessionId%22%3A%224239e114-7c70-49f8-9571-3009149e3b7a%22%2C%22PendingAuthenticationSessionId%22%3A%2200000000-0000-0000-0000-000000000000%22%2C%22CurrentAuthenticationSessionId%22%3A%2200000000-0000-0000-0000-000000000000%22%2C%22SignInState%22%3A%7B%7D%7D;SameSite=None; domain=.dev.azure.com; expires=Sat, 25-Nov-2023 10:23:53 GMT; path=/; secure; HttpOnly\"],\"x-tfs-processid\":\"79f3b842-50a5-4ca5-b665-35d6c8413863\",\"strict-transport-security\":\"max-age=31536000; includeSubDomains\",\"activityid\":\"3ec05a58-40fa-41eb-af64-46c5d4c4f355\",\"x-tfs-session\":\"3ec05a58-40fa-41eb-af64-46c5d4c4f355\",\"x-vss-e2eid\":\"3ec05a58-40fa-41eb-af64-46c5d4c4f355\",\"x-vss-senderdeploymentid\":\"fa85298d-d7df-eba2-6cd2-9d3c9247a546\",\"x-vss-userdata\":\"6384316d-b38d-609f-acc4-aa37c06225d5:t.bettmann@reply.de\",\"x-frame-options\":\"SAMEORIGIN\",\"request-context\":\"appId=cid-v1:a32646d4-6542-4ff0-b4b1-416ef6b2d466\",\"access-control-expose-headers\":\"Request-Context\",\"x-content-type-options\":\"nosniff\",\"x-cache\":\"CONFIG_NOCACHE\",\"x-msedge-ref\":\"Ref A: 26960DD965FB4D109B6DC1FDFE798E62 Ref B: DUS30EDGE0708 Ref C: 2022-11-25T10:23:53Z\",\"date\":\"Fri, 25 Nov 2022 10:23:52 GMT\",\"connection\":\"close\"},\"data\":{\"$id\":\"1\",\"innerException\":null,\"message\":\"Expected a Tag, but objectId 0fd17aca2a0e800979e0c4f83bc71350479ac23e resolved to a Commit\",\"typeName\":\"Microsoft.TeamFoundation.Git.Server.GitUnexpectedObjectTypeException, Microsoft.TeamFoundation.Git.Server\",\"typeKey\":\"GitUnexpectedObjectTypeException\",\"errorCode\":0,\"eventId\":3000}}},\"message\":\"API responded with status 404\"}","stack_trace":"VError: API responded with status 404\n at wrapApiError (/Users/c.haumann/airbyte-connectors/faros-airbyte-cdk/lib/errors.js:58:12)\n at Command.<anonymous> (/Users/c.haumann/airbyte-connectors/faros-airbyte-cdk/lib/sources/source-runner.js:91:53)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Command.parseAsync (/Users/c.haumann/airbyte-connectors/node_modules/commander/lib/command.js:917:5)"},"type":"LOG"} {"trace":{"type":"ERROR","emitted_at":1669371833295,"error":{"message":"API responded with status 404","stack_trace":"VError: API responded with status 404\n at wrapApiError (/Users/c.haumann/airbyte-connectors/faros-airbyte-cdk/lib/errors.js:58:12)\n at AirbyteTrace.make (/Users/c.haumann/airbyte-connectors/faros-airbyte-cdk/lib/protocol.js:151:51)\n at AirbyteLogger.traceError (/Users/c.haumann/airbyte-connectors/faros-airbyte-cdk/lib/logger.js:37:44)\n at process.<anonymous> (/Users/c.haumann/airbyte-connectors/faros-airbyte-cdk/lib/runner.js:12:20)\n at process.emit (node:events:513:28)\n at process._fatalException (node:internal/process/execution:149:25)","internal_message":"VError","failure_type":"system_error","jse_shortmsg":"API responded with status 404","jse_info":{"req":{"baseUrl":"https://dev.azure.com/vwfs/GisAtWeb/_apis","url":"git/repositories/c9526643-4ee7-498a-97e8-92b9a50f5229/annotatedtags/0fd17aca2a0e800979e0c4f83bc71350479ac23e","method":"GET","params":{"api-version":"6.0"}},"res":{"status":404,"headers":{"cache-control":"no-cache, no-store, must-revalidate","pragma":"no-cache","content-length":"329","content-type":"application/json; charset=utf-8; api-version=6.0","expires":"-1","p3p":"CP=\"CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT\"","set-cookie":["VstsSession=%7B%22PersistentSessionId%22%3A%224239e114-7c70-49f8-9571-3009149e3b7a%22%2C%22PendingAuthenticationSessionId%22%3A%2200000000-0000-0000-0000-000000000000%22%2C%22CurrentAuthenticationSessionId%22%3A%2200000000-0000-0000-0000-000000000000%22%2C%22SignInState%22%3A%7B%7D%7D;SameSite=None; domain=.dev.azure.com; expires=Sat, 25-Nov-2023 10:23:53 GMT; path=/; secure; HttpOnly"],"x-tfs-processid":"79f3b842-50a5-4ca5-b665-35d6c8413863","strict-transport-security":"max-age=31536000; includeSubDomains","activityid":"3ec05a58-40fa-41eb-af64-46c5d4c4f355","x-tfs-session":"3ec05a58-40fa-41eb-af64-46c5d4c4f355","x-vss-e2eid":"3ec05a58-40fa-41eb-af64-46c5d4c4f355","x-vss-senderdeploymentid":"fa85298d-d7df-eba2-6cd2-9d3c9247a546","x-vss-userdata":"6384316d-b38d-609f-acc4-aa37c06225d5:t.bettmann@reply.de","x-frame-options":"SAMEORIGIN","request-context":"appId=cid-v1:a32646d4-6542-4ff0-b4b1-416ef6b2d466","access-control-expose-headers":"Request-Context","x-content-type-options":"nosniff","x-cache":"CONFIG_NOCACHE","x-msedge-ref":"Ref A: 26960DD965FB4D109B6DC1FDFE798E62 Ref B: DUS30EDGE0708 Ref C: 2022-11-25T10:23:53Z","date":"Fri, 25 Nov 2022 10:23:52 GMT","connection":"close"},"data":{"$id":"1","innerException":null,"message":"Expected a Tag, but objectId 0fd17aca2a0e800979e0c4f83bc71350479ac23e resolved to a Commit","typeName":"Microsoft.TeamFoundation.Git.Server.GitUnexpectedObjectTypeException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitUnexpectedObjectTypeException","errorCode":0,"eventId":3000}}}}},"type":"TRACE"}
What is the source of this problem and how can I fix it?
About this issue
- Original URL
- State: closed
- Created 2 years ago
- Comments: 17 (16 by maintainers)
Now it works. I think it was about the accesstoken. You can type anything for the accesstoken and you get a successmessage. I edit “Test” for the accesstoken and there were the same problems as before. With the not already base64 encoded accesstoken I had a success and data was read and wrote. The problem I still have is that in “VCS Commit” the “CreatedAt” Column is empty so some metrics like “Release Lead Time” in “DORA - Releases and Bugs” can not be calculated. The “Message”, “URL” and “Author” Column in “VCS Commit” is also empty. That was also with the old converter the case. Here are the logs of my last sync https://github.com/Christopher-Haumann/azure-repo-log/blob/main/logs-86.txt
Hey @Christopher-Haumann, I just published a new version (v0.4.79) of the
farosai/airbyte-azure-repos-sourcewith a fix that may be related to the issue that you are seeing. Please pull that image and try running with it, thanks!Also, is there a specific reason you are explicitly setting the
API versionandGraph API Version? The default that I recommend using are:Another thing to check if you still are having the same issue: I added logic within the source to correctly handle the raw Azure Access Token. You use to have to prefix it with
:and base64 encode it yourself. Now, the source does that for you. So, please make sure that you are providing the token to the source exactly as Azure mints it.Thank you now my “latest” is up-to-date. But no data is read or wrote. In Airbyte in Source Settings I have: Name: Azure Repo Page Size: 100 API Version: 6.0 Cutoff Days: 90 Max Number of Retries: 3 Azure Access Token: **************** (my Access Token) Azure Organization: vwfs Graph API Version: 4.1-preview.1 Branch Name Pattern: ^main$ Request Timeout: 60000
Why I get a successmessage but no data is read or wrote? With the former source setting where was no “Branch Name Pattern” and so on I had no problems.