C API¶
Use of a C API is out of scope for this array API, as mentioned in Scope (includes out-of-scope / non-goals). There are a lot of libraries that do use such an API - in particular via Cython code or via direct usage of the NumPy C API. When the maintainers of such libraries want to use this array API standard to support multiple types of arrays, they need a way to deal with that issue. This section aims to provide some guidance.
The assumption in the rest of this section is that performance matters for the library,
and hence the goal is to make other array types work without converting to a
numpy.ndarray or another particular array type. If that’s not the case (e.g. for a
visualization package), then other array types can simply be handled by converting
to the supported array type.
Note
Often a zero-copy conversion to numpy.ndarray is possible, at least for CPU arrays.
If that’s the case, this may be a good way to support other array types.
The main difficulty in that case will be getting the return array type right - however,
this standard does provide a Python-level API for array construction that should allow
doing this. A relevant question is if it’s possible to know with
certainty that a conversion will be zero-copy. This may indeed be
possible, see Data interchange mechanisms.
Example situations for C/Cython usage¶
Situation 1: a Python package that is mostly pure Python, with a limited number of Cython extensions¶
Note
Projects in this situation include Statsmodels, scikit-bio and QuTiP
Main strategy: documentation. The functionality using Cython code will not support other array types (or only with conversion to/from numpy.ndarray), which can be documented per function.
Situation 2: a Python package that contains a lot of Cython code¶
Note
Projects in this situation include scikit-learn and scikit-image
Main strategy: add support for other array types per submodule. This keeps it manageable to explain to the user which functionality does and doesn’t have support.
Longer term: specific support for particular array types (e.g. cupy.ndarray can be supported with Python-only code via cupy.ElementwiseKernel).
Situation 3: a Python package that uses the NumPy or Python C API directly¶
Note
Projects in this situation include SciPy and Astropy
Strategy: similar to situation 2, but the number of submodules that can support all array types may be limited.
Device support¶
Supporting non-CPU array types in code using the C API or Cython seems problematic, this almost inevitably will require custom device-specific code (e.g., CUDA, ROCm) or something like JIT compilation with Numba.
Other longer-term approaches¶
Further Python API standardization¶
There may be cases where it makes sense to standardize additional sets of
functions, because they’re important enough that array libraries tend to
reimplement them. An example of this may be special functions, as provided
by scipy.special. Bessel and gamma functions for example are commonly
reimplemented by array libraries. This may avoid having to drop into a
particular implementation that does use a C API (e.g., one can then rely on
arraylib.special.gamma rather than having to use scipy.special.gamma).
HPy¶
HPy is a new project that will provide a higher-level C API and ABI than CPython offers. A Cython backend targeting HPy will be provided as well.
Better PyPy support
Universal ABI - single binary for all supported Python versions
Cython backend generating HPy rather than CPython code
HPy isn’t quite ready for mainstream usage today, but once it does it may help make supporting multiple array libraries or adding non-CPU device support to Cython more feasible.