crawl4ai: [Bug]: Crawl returning 'str' object has no attribute 'choices

crawl4ai version

0.5.0

Expected Behavior

Return a normal crawl matching my schema.

Current Behavior

I am crawling a set of about 600 links. For some links, the crawl works perfectly, but for others, the crawler returns: [ { “index”: 0, “error”: true, “tags”: [ “error” ], “content”: “‘str’ object has no attribute ‘choices’” } ]

there is no pattern for which links the crawler returns this for which makes me wonder whether this is an API issue. Has anyone seen or encountered this bug?

Is this reproducible?

Yes

Inputs Causing the Bug


Steps to Reproduce


Code snippets


OS

mac)S

Python version

3.13

Browser

Chrome

Browser version

No response

Error logs & Screenshots (if applicable)

No response

About this issue

  • Original URL
  • State: closed
  • Created 2 years ago
  • Reactions: 4
  • Comments: 22

Commits related to this issue

Most upvoted comments

Looks like the the response variable get’s redefined and that causes an issue with this try-except block

I was able to resolve this issue for myself by changing the try-except block in .venv\Lib\site-packages\crawl4ai\extraction_strategy.py on line 657

Just commented the re-definition and refer to response.choices[0].message.content where the redefined response variable was referenced

            try:
                # response = response.choices[0].message.content
                blocks = None

                if self.force_json_response:
                    blocks = json.loads(response.choices[0].message.content)
                    if isinstance(blocks, dict):
                        # If it has only one key which calue is list then assign that to blocks, exampled: {"news": [..]}
                        if len(blocks) == 1 and isinstance(list(blocks.values())[0], list):
                            blocks = list(blocks.values())[0]
                        else:
                            # If it has only one key which value is not list then assign that to blocks, exampled: { "article_id": "1234", ... }
                            blocks = [blocks]
                    elif isinstance(blocks, list):
                        # If it is a list then assign that to blocks
                        blocks = blocks
                else: 
                    blocks = extract_xml_data(["blocks"], response.choices[0].message.content)["blocks"]
                    # blocks = extract_xml_data(["blocks"], response)["blocks"]
                    blocks = json.loads(blocks)

                for block in blocks:
                    block["error"] = False
            except Exception:
                parsed, unparsed = split_and_parse_json_objects(
                    response.choices[0].message.content
                )
                blocks = parsed
                if unparsed:
                    blocks.append(
                        {"index": 0, "error": True, "tags": ["error"], "content": unparsed}
                    )

thanks @quangvinh2080 for the lead

I’ve resolved the issue. The fix is now available in the 2025-MAY-2 branch.

Thank you all for your assistance in identifying the root cause of this issue. I will work on a fix, which will be included in our next release. I will most likely add the fix to the May bug-fix branch. I’ll keep everyone updated on the progress here.

cc @aravindkarnam

The error occurs because some models return reasoning_content instead of content in crawl4ai/extraction_strategy.py (response = response.choices[0].message.content). Try using a different model to resolve this

Chinese discussion forum implementing fix for same problem: https://linux.do/t/topic/561221

Same here. Using deepseek chat API