GH-36845: [C++][Python] Allow type promotion on pa.concat_tables

[Fokko]

Revival of #12000

Rationale for this change

It would be great to be able to do promotions when concat'ing a table, such as:

def test_concat_tables_with_promotion_int():
    import pyarrow as pa
    t1 = pa.Table.from_arrays(
        [pa.array([1, 2], type=pa.int64())], ["int"])
    t2 = pa.Table.from_arrays(
        [pa.array([3, 4], type=pa.int32())], ["int"])

    result = pa.concat_tables([t1, t2], promote=True)

    assert result.equals(pa.Table.from_arrays([
        pa.array([1, 2, 3, 4], type=pa.int64())
    ], ["int"]))

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

• Closes: [Python] Allow promotion from int32 to int64 #36845

[github-actions]

⚠️ GitHub issue #36845 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

Thanks a lot for the updates!

Added a few small comments (I can also push if needed to ensure we get this in).
It might be good to add a small test to ensure promote keyword still works.

[jorisvandenbossche]

Suggested change

	If promote_options=="none", a zero-copy concatenation will be performed. The schemas
	If promote_options="none", a zero-copy concatenation will be performed. The schemas

[jorisvandenbossche]

Suggested change

	If promote_options=="default", any null type arrays will be casted to the type of other
	If promote_options="default", any null type arrays will be casted to the type of other

[jorisvandenbossche]

Suggested change

	"promote has been superseded by mode='default'.", FutureWarning)
	"promote has been superseded by mode='default'.", FutureWarning, stacklevel=2)

[jorisvandenbossche]

Suggested change

	Default; null and only null can be unified with another type.
	Permissive; promotes types to the greater common denominator.
	Default: null and only null can be unified with another type.
	Permissive: types are promoted to the greater common denominator.

[jorisvandenbossche]

Suggested change

	promote_options = "permissive"
	promote_options = "default"

Isn't that needed to preserve the current default behaviour?

[Fokko]

I thought we agreed on promoting when promote=True, but this also works for me.

[jorisvandenbossche]

Yeah, I mentioned that I would find a promote=True a better default, but now that we deprecated it in favor of promote_options, we don't need to change the current behaviour, as it will be removed anyway. And users can now replace it with promote_options="permissive" if they prefer that

[jorisvandenbossche]

Suggested change

	result = pa.concat_tables([t1, t2], promote=True)
	with pytest.warns(FutureWarning):
	result = pa.concat_tables([t1, t2], promote=True)

This asserts the warning is raised and at the same time also ensures we don't unnecessarily see the warning in the pytest logs

[jorisvandenbossche]

Thanks @Fokko! (and @lidavidm for the initial PR)

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 5f57219.

There were 2 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2023-10-11 12:25:39Z
  • file-read (Python) with compression=uncompressed, dataset=fanniemae_2016Q4, file_type=feather, output_type=table
  • file-read (Python) with compression=uncompressed, dataset=fanniemae_2016Q4, file_type=feather, output_type=dataframe

The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them.

[jorisvandenbossche]

FYI, those reported performance regressions were just flakes. The timings are still stable at the same level for later commits.