GH-35740: Add documentation for list arrays' values property

[spenczar]

Just docs.

I'm a little shaky on my understanding of exactly what's going on with FixedSizeListArray.values, and its behavior with nulls, so that wording might deserve a careful read.

• Closes: [Python][Docs] 'values' attribute is missing documentation for ListArray and its friends #35740

[github-actions]

⚠️ GitHub issue #35740 has been automatically assigned in GitHub to PR creator.

[AlenkaF]

Thank you for the contribution @spenczar !

I think the main difference between values and flatten is when the array is a slice into another array (see description in https://arrow.apache.org/docs/dev/cpp/api/array.html#_CPPv4N5arrow18FixedSizeListArrayE). You could play around with looking at the code for array = pa.array([[1, 2], None, [3, None]],type=pa.large_list(pa.int32()))and array.slice(0, 1).

You are correct in the understanding that the behaviour of FixedSizeListArray.values is a bit different. I am now aware of the reason for this, but the difference can be seen in the link to the docs from before (https://arrow.apache.org/docs/dev/cpp/api/array.html#_CPPv4N5arrow9ListArrayE).

Some nit for finish: there are doctest errors that need to be fixed with adding ...:

=================================== FAILURES ===================================
___________ [doctest] pyarrow.lib.FixedSizeListArray.values.__get__ ____________
2378         values : Array
2379 
2380         Examples
2381         --------
2382         >>> import pyarrow as pa
2383         >>> array = pa.array(
2384         ...     [[1, 2], None, [3, None]],
2385         ...     type=pa.list_(pa.int32(), 2)
2386         ... )
2387         >>> array.values
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,3 @@
    -<pyarrow.lib.Int32Array object at 0x12d1743a0>
    +<pyarrow.lib.Int32Array object at 0x7f3828fa5440>
     [
       1,

/opt/conda/envs/arrow/lib/python3.9/site-packages/pyarrow/lib.cpython-39-x86_64-linux-gnu.so:2387: DocTestFailure
_____________ [doctest] pyarrow.lib.LargeListArray.values.__get__ ______________
2189         values : Array
2190 
2191         Examples
2192         --------
2193         >>> import pyarrow as pa
2194         >>> array = pa.array(
2195         ...     [[1, 2], None, [3, 4, None, 6]],
2196         ...     type=pa.large_list(pa.int32()),
2197         ... )
2198         >>> array.values
Differences (unified diff with -expected +actual):
    @@ -1,3 +1,3 @@
    -<pyarrow.lib.Int32Array object at 0x12d174100>
    +<pyarrow.lib.Int32Array object at 0x7f3828faeb40>
     [
       1,

[spenczar]

Thanks for the explanation @AlenkaF, that makes a lot of sense, and I learned a bunch about Arrow internals from playing with slice. Do you think the docs I wrote need any edits?

I think I've fixed the doctest issue, too, we'll see if it passes CI now.

[AlenkaF]

Thanks for the corrections!
Some minor nits ...

[jorisvandenbossche]

Thanks for the PR @spenczar!
I added a few more comments. I think if we expand it even more (eg additional example), we can further clarify the difference between values and flatten(). But note that what you have here is already a clear improvement, so it's certainly fine as well to just focus on finalizing the edits you have now in this PR.

[spenczar]

Hello! Anything else to be done here? This is affecting some intersphinx stuff I'm doing in a mild way.

[spenczar]

Is there anything I can do to get this merged?

[AlenkaF]

Sorry for the delay @spenczar!

I would like us to address the unresolved comment before merging. Otherwise the PR looks good 👍

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 6837d05.

There were 108 benchmark results with an error:

• Commit Run on ursa-i9-9960x at 2023-08-24 09:10:15Z
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-19, scale_factor=10
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-04, scale_factor=1
• and 106 more (see the report linked below)

There was 1 benchmark result indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2023-08-24 09:10:15Z
  • dataset-selectivity (Python) with dataset=nyctaxi_multi_ipc_s3, selectivity=10%

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.