Fix Series construction from numpy array with non-native byte order #18151
Conversation
| @@ -2719,8 +2719,16 @@ def as_column( | |||
| return as_column(arbitrary, dtype=dtype, nan_as_null=nan_as_null) | |||
| elif arbitrary.dtype.kind in "biuf": | |||
| from_pandas = nan_as_null is None or nan_as_null | |||
| try: | |||
| pa_array = pa.array(arbitrary, from_pandas=from_pandas) | |||
| except pa.ArrowNotImplementedError: | |||
There was a problem hiding this comment.
Can we check if an array is byte swapped? Instead of relying on generic not implemented error?
There was a problem hiding this comment.
Sure, switch to check if the byte order is non native.
There was a problem hiding this comment.
Thanks for starting on this so quickly! Left two small comments, feel free to ignore.
Note that this doesn't fully resolve #18149 since cupy can also have byteswapped arrays and those still error (though in a different location, as documented in the original issue).
| @@ -2719,8 +2731,15 @@ def as_column( | |||
| return as_column(arbitrary, dtype=dtype, nan_as_null=nan_as_null) | |||
| elif arbitrary.dtype.kind in "biuf": | |||
| from_pandas = nan_as_null is None or nan_as_null | |||
| return as_column( | |||
| if not is_np_native_byteorder(arbitrary.dtype): | |||
There was a problem hiding this comment.
You can just use dtype.isnative here I believe, no need for a helper function: https://numpy.org/doc/2.2/reference/generated/numpy.dtype.isnative.html
| if not is_np_native_byteorder(arbitrary.dtype): | |
| if not arbitrary.dtype.isnative: |
There was a problem hiding this comment.
Ah great, didn't know about this API. Thanks!
| return as_column( | ||
| if not is_np_native_byteorder(arbitrary.dtype): | ||
| # Not supported by pyarrow | ||
| arbitrary = arbitrary.astype(arbitrary.dtype.newbyteorder()) |
There was a problem hiding this comment.
If it's not native byteorder then swapping byteorder will get you native byteorder - I'd find specifying native byteorder explicitly would be a bit more explicit though:
| arbitrary = arbitrary.astype(arbitrary.dtype.newbyteorder()) | |
| arbitrary = arbitrary.astype(arbitrary.dtype.newbyteorder("=")) |
|
/merge |
|
Can someone reopen #18149 (I lack permissions to reopen it)? This PR didn't fully resolve the issue (it fixed it for |
Done. |
|
Oops sorry, followed up with #18164 |
Description
closes #18149
Checklist