Importing `bz2` will succeed under `lazy_imports=all` even if bz2 is not supported

[DavidCEllis]

Bug report

Edit: Originally this bug was expressed through shutil's support detection. shutil now directly imports from _bz2 which works around this issue.

Bug description:

Under lazy_imports=all, modules imported in a try/except block are still imported eagerly. However, imports that occur within that module are still imported lazily.

This may be intentional, but this bug is the result of that behaviour.

On CPython built without bz2 support:

$ PYTHON_LAZY_IMPORTS=normal ./python -c "import shutil; print(shutil._BZ2_SUPPORTED)"
False
$ PYTHON_LAZY_IMPORTS=all ./python -c "import shutil; print(shutil._BZ2_SUPPORTED)"
True

The try/except check is on bz2, which succeeds because the import to _bz2 which would fail is converted to a lazy import and is not used at top level so the import which would fail isn't triggered.

cpython/Lib/shutil.py

Lines 21 to 26 in 87a879f

	try:
	import bz2
	del bz2
	_BZ2_SUPPORTED = True
	except ImportError:
	_BZ2_SUPPORTED = False

shutil could directly check _bz2 in this particular case, but I expect this pattern isn't uncommon. I looked into this after I ran into a similar issue with pip and rich with an OS-specific import that should have failed in a similar manner.

Related: #149640 as this also causes test_shutil to fail under lazy_imports=all if _bz2 isn't present.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-150167: Make shutil compression probes eager #150279
• gh-150167: Force eager import of _bz2 in bz2.py to fix supported format detection #154771
• gh-150167: Prevent successful import of bz2 under lazy_imports=all if _bz2 is unavailable #157321

[vstinner]

I can reproduce the issue on Linux using:

$ find -name "_bz2*.so" -delete
$ ./python -c "import shutil; print(shutil._BZ2_SUPPORTED)"
False
$ PYTHON_LAZY_IMPORTS=all ./python -c "import shutil; print(shutil._BZ2_SUPPORTED)"
True

_ZLIB_SUPPORTED, _LZMA_SUPPORTED and _ZSTD_SUPPORTED are correct since import zlib, import lzma and from compression import zstd fail immediately with ModuleNotFound even with PYTHON_LAZY_IMPORTS=all.

_BZ2_SUPPORTED is incorrect because import bz2 completes successfully even if import _bz2 will fail. It behaves differently since _bz2 symbols are only accessed when bz2 functions are called, not at import time.

[DavidCEllis]

lzma is working somewhat by luck as it uses a * import on the _lzma module and * imports can't be made lazy. zstd uses _zstd at module level so similarly escapes this issue. I say by luck as these aren't done with the intention of preventing the lazy import, it's just a side-effect.

I believe zlib itself is the module that would fail so its check isn't affected but I haven't confirmed this.

The simplest solution for _bz2 specifically is probably to add import _bz2; _bz2 to bz2.py to force the resolution. lzma and zstd could also do the same, but I'm not sure it's necessary as long as there are tests. (I'm not actually sure how you'd write the test for this case given it requires missing modules and the lazy import flag.)

We could instead check the internal modules directly in shutil but that makes shutil dependant on internals of the compression libraries which is probably undesirable?

[DavidCEllis]

Perhaps compression should have some kind of function or attribute that provides a list or set of available compression types?

I think there it would be fine to only check the internal modules and shutil could then use this directly rather than checking all of the modules itself.

[DavidCEllis]

I was having a look at writing a test for this and noticed that the test.support.requires_<format> functions have the same logic and probably the same issue.

cpython/Lib/test/support/__init__.py

Lines 548 to 581 in 993a0c6

	def requires_zlib(reason='requires zlib'):
	try:
	import zlib
	except ImportError:
	zlib = None
	return unittest.skipUnless(zlib, reason)
	def requires_gzip(reason='requires gzip'):
	try:
	import gzip
	except ImportError:
	gzip = None
	return unittest.skipUnless(gzip, reason)
	def requires_bz2(reason='requires bz2'):
	try:
	import bz2
	except ImportError:
	bz2 = None
	return unittest.skipUnless(bz2, reason)
	def requires_lzma(reason='requires lzma'):
	try:
	import lzma
	except ImportError:
	lzma = None
	return unittest.skipUnless(lzma, reason)
	def requires_zstd(reason='requires zstd'):
	try:
	from compression import zstd
	except ImportError:
	zstd = None
	return unittest.skipUnless(zstd, reason)

[DavidCEllis]

So I believe the specific case of failure in shutil has been resolved as a side effect of #154908 but the logical issue still exists in other places that do check the modules directly like test.support. Not sure what to do here.