C API coverage

[wshanks]

Background: One of the packages I maintain for conda-forge (Qiskit) had been building a PyPy variant without issue until its most recent release (1.0.0). In the new release, it added a new Rust crate bound using PyO3. In previous releases, it had already been binding to two other crates without error. The new crate failed to build to because it uses PyModule_GetFilename which PyPy does not support. I reported the build issue to Qiskit in Qiskit/qiskit#11822, but I don't know if anyone uses the PyPy build. So the question was if it was worth the effort to keep the project compatible with PyPy. The PyModule_GetFilename use could be avoided but there was a concern that new issues could keep coming in new releases.

So what I was wondering about was why does PyPy not implement PyModule_GetFilename? It seems like a fairly simple function (return the __file__ of a module) but maybe I am missing subtleties. I noticed the FAQ says that support for extensions that use the C-API is "almost complete." From https://github.com/orgs/pypy/discussions/4884#discussioncomment-8309845, I take that it is difficult to support the C-API and the hope is that HPy will solve this. Qiskit uses the abi3 mode of PyO3 which I think means it only uses the limited stable C-API of Python. I would have hoped that PyPy would have support for the limited stabl C-API for compatibility with CPython extensions, but I don't know much about how hard this is. I was mainly interested in how much of a problem compatibility would be with a project targeting the limited stable C-API with PyO3 but didn't have the resources to test against PyPy.

[mattip]

There are hundreds if not thousands of c-api functions, and they must be implemented by hand. See issues #3327 and #3397 for discussions of implementing a limited API mode in PyPy and tools to compare PyPy's headers to CPython's headers.

The c-api functions that return a char * are particularily tricky to implement. PyPy's object model is not based on C, so we need to carefully consider how to handle the lifetime of char * arrays. See #4902 for a similar problem with PyModule_GetName, where the solution is to use PyModule_GetNameObject (which also is not implemented, but I am working on this). Looking at the documentation for PyModule_GetFilename, it actually is deprecated in favor of using PyModule_GetFilenameObject (which is also not implemented). I opened #4906 for the missing PyModule_Get*Object functions.

[wshanks]

Thanks, for the information and for adding PyModule_GetFilename! I had noticed that PyModule_GetFilename was deprecated while looking into this and was going to point that out to the Qiskit maintainers any way even if it would not have helped with the PyPy case before.

So for the Qiskit case now, the steps are:

• Ask Qiskit to replace PyModule_GetFilename with PyModule_GetFilenameObject
• Wait for a new PyPy release with add PyModule_GetFilenameObject and PyModule_GetNameObject #4907.
• Ask PyO3 to remove the not(PyPy) guard on PyModule_GetFilenameObject.
• Try to build the Qiskit conda recipe again once new PyO3, Qiskit, and PyPy releases are available (I could test locally by patching the three packages before the releases but I need the official releases of PyO3 and PyPy before doing the real Qiskit package).

Do you happen to know how to specify a lower bound on a PyPy version in a conda recipe? My guess would be that I need to add pypy >=7.3.16 # [ python_impl == "pypy" ] to the build and run dependencies (and not set something linked to the Python version like pypy3.9).

[mattip]

My guess would be that I need to add pypy >=7.3.16 # [ python_impl == "pypy" ]

I think so