Implement `anext` and `aiter` in pure Python

[kumaraditya303]

Feature or enhancement

Feature or enhancement

I propose to implement anext() and aiter() in Python instead of C, compiled at interpreter startup and copied into builtins. This removes the private anext_awaitable, async_callable_iterator and async_callable_iterator_awaitable types.

anext(it, default) becomes a plain coroutine that awaits type(it).__anext__(it) and converts StopAsyncIteration into the default. The callable form of aiter() (gh-64862) becomes a small class with async def __anext__ that calls and awaits the callable and turns the sentinel or stop_exception into StopAsyncIteration. Semantics stay the same.

The motivation is #157032 and #157044: the C awaitables break the asyncio await graph. capture_call_graph walks cr_await/ag_await, and these types have neither, so the graph stops at the frame calling anext(it, default) or iterating aiter(callable, sentinel) and hides where the task is actually suspended. This would fix #157032, #157044 without any special casing.

Repro with a class-based async iterator:

import asyncio

async def fetch():
    await asyncio.sleep(10)

class AIter:
    def __aiter__(self): return self
    async def __anext__(self):
        await fetch()
        return 1

async def consumer():
    await anext(AIter(), None)

async def main():
    t = asyncio.create_task(consumer(), name="consumer")
    for _ in range(3):
        await asyncio.sleep(0)
    asyncio.print_call_graph(t)
    t.cancel()

asyncio.run(main())

Before:

* Task(name='consumer', id=0x10a42c7d0)
  + Call stack:
  |   File 'demo.py', line 13, in async consumer()

After:

* Task(name='consumer', id=0x108959b80)
  + Call stack:
  |   File 'Lib/asyncio/tasks.py', line 707, in async sleep()
  |   File 'demo.py', line 4, in async fetch()
  |   File 'demo.py', line 9, in async AIter.__anext__()
  |   File '<builtins>', line 26, in async _anext_with_default()
  |   File 'demo.py', line 13, in async consumer()

Linked PRs

• gh-157361: Implement anext() in Python instead of C #157362

[picnixz]

I propose to implement anext() and aiter() in Python instead of C, compiled at interpreter startup and copied into builtins

OOC, how would you do it? do you mean to add some _builtins.py file somewhere or do you mean that we do it differently? I never had to deal with this specific problem but it would be good to know whether we have the architecture in place to avoid future issues.

[kumaraditya303]

OOC, how would you do it? do you mean to add some _builtins.py file somewhere or do you mean that we do it differently? I never had to deal with this specific problem but it would be good to know whether we have the architecture in place to avoid future issues.

See #157362, I am adding the source for anext as a C string and compiling it at startup and adding it to builtins dict, doing this in C is easier as I can compile function manually with filename as <builtins>.

[kumaraditya303]

We do have module freezing infrastructure inplace so we can use that as well. I'll try that as well and if it works for builtins as well then might as well reuse that for this.