anyio: Task group is swallowing `CancelledError`.
If any of the tasks within the task group raise a CancelledError, then that does not propagate out of the task group. In my case, this happens because the code within the task group is cancelled from somewhere else.
from anyio import create_task_group
import asyncio
async def main():
async def in_task_group():
raise asyncio.CancelledError
async with create_task_group() as tg:
tg.start_soon(in_task_group)
asyncio.run(main())
Would this be a bug? Or is this expected behavior?
About this issue
- Original URL
- State: closed
- Created 3 years ago
- Reactions: 5
- Comments: 36 (23 by maintainers)
Happy to report that I’m making progress again, and hopefully I can make a release candidate of v4.0 this week, unless I encounter unexpectedly difficult corner cases.
Sorry for the trouble. It may not be much of a consolation, but I’ve been hit by this bug too occasionally, and it’s my highest priority in this project right now (that is, when I get back to working on it).
I’ve restarted my efforts based on current
master. I have a test in place that reproduces the issue. I think I might start writing a fix that works on Python 3.11+ (where thanks touncancel()we can actually track the cancellations without too many hacks). Then I could work on said hacks for earlier Pythons, as I was trying to do before going on a hiatus.The first order of business I have with AnyIO right now is getting the v3.7.0 release out. It contains a ton of other fixes that have waited a long time to be released. One of these fixes will even make tackling this issue a bit easier by reducing the amount of new code that would have to be written. I’m just about ready to make that release. After that, I can fully focus on fixing this particular issue, but I will have to scrap my previous attempt and start from scratch, as the code got pretty complicated back then and I was overwhelmed with the complexity. As for an ETA, I hope to get this done in 2-3 weeks.
I already have, in #496. The PR includes some heavy handed overall changes to cancellation semantics, so it requires extra care and review. I myself am not done with it yet.
Fixing this is a high priority for v4.0. My initial attempts have not gone very well, as I’m seeing recursion errors from exception groups.
The merged PR should solve the problem as described. There is still the issue of cancel scopes not distinguishing between native cancellations and AnyIO cancellations on Python < 3.11. I have a PR for that in progress, but I’m running into some very subtle issues and am not yet confident about it, so that fix might not materialize in v4.0.0.
@jonathanslenders would you mind testing the PR to make sure that it solves your original problem? Do note that the exception raising behavior has changed in a backwards incompatible fashion when you test.
The expected release date for v4.0.0rc1 would be later this week.
I think I found a way around this, but on Python 3.9 and above only: we can set a special cancellation message in the exception to detect cancel scope based cancellations.
Well, imagine we have a library that uses structured concurrency everywhere within that library. But this library is called by a different codebase that does not follow structured concurrency. The above issue means that it’s impossible to use libraries written with anyio in a code base that doesn’t use anyio by itself. Is that correct?