Improvements to list_files_cache table function

[alamb]

Which issue does this PR close?

• Follow on to Make default ListingFilesCache table scoped #19616

Rationale for this change

I had a few minor comments / suggestions while reviewing #19616 from @jizezhang but they weren't needed to do the initial merge, so I would like to propose them in a follow up PR

What changes are included in this PR?

1. Improve documentation
2. Improve handling of table_ref in ListingTableURL
3. use Null rather than "NULL" in list_files_cache table function

I can break this into separate PRs if that would help

Are these changes tested?

Yes by CI

Are there any user-facing changes?

The list_files_cache function now might return null

[alamb]

This uses NULL (rather than "NULL") in list_files_cache

[alamb]

There is no need to call the constructor again as self.path hasn't changed and the constructor doesn't do anything with glob

datafusion/datafusion/datasource/src/url.rs

Lines 149 to 157 in 8a9bf43

	pub fn try_new(url: Url, glob: Option<Pattern>) -> Result<Self> {
	let prefix = Path::from_url_path(url.path())?;
	Ok(Self {
	url,
	prefix,
	glob,
	table_ref: None,
	})
	}

This then simplifies the code in datafusion/core/src/datasource/listing_table_factory.rs

[alamb]

This can be simplified due to the change in with_glob

[comphead]

Thanks @alamb it is lgtm!

[jizezhang]

thanks a lot!