Method download_and_prepare poorly documented (+Tedlium broken)

[knightcode]

Description of issue

Using this bit of python:

  dl_config = tfds.download.DownloadConfig(
    beam_options=beam.options.pipeline_options.PipelineOptions(flags=[]),
    compute_stats=tfds.download.ComputeStatsMode.SKIP)
  builder = tfds.builder("tedlium/release3", data_dir="./datasets")
  logger.info(f"info: {builder.info}")
  builder.download_and_prepare(
    download_dir="./downloads",
    download_config=dl_config)

Initially threw a KeyError on some file in the tarball, which I fixed with this work around in core/download/extractor.py:

      try:
        extract_file = tar.extractfile(member)
      except KeyError:
        print("Failed extracting: {}".format(member))
        continue

The process has been running for 6 days straight now, and I have no idea what it's currently doing. The last output from like 5.5 days ago was this:

2020-10-16T08:26:59-0400 INFO Generating split validation
2020-10-16T08:26:59-0400 INFO Generating split test
2020-10-16T08:26:59-0400 INFO Generating split train
2020-10-16T08:27:00-0400 INFO ==================== <function annotate_downstream_side_inputs at 0x165f74430> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function fix_side_input_pcoll_coders at 0x165f74550> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function lift_combiners at 0x165f745e0> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function expand_sdf at 0x165f74670> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function expand_gbk at 0x165f74700> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function sink_flattens at 0x165f74820> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function greedily_fuse at 0x165f748b0> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function read_to_impulse at 0x165f74940> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function impulse_to_input at 0x165f749d0> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function sort_stages at 0x165f74c10> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function setup_timer_mapping at 0x165f74b80> ====================
2020-10-16T08:27:00-0400 INFO ==================== <function populate_data_channel_coders at 0x165f74ca0> ====================
2020-10-16T08:27:00-0400 INFO Creating state cache with size 100
2020-10-16T08:27:00-0400 INFO Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x1661f1940> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
2020-10-16T08:27:00-0400 INFO Running ((((ref_AppliedPTransform_test/Create/Impulse_47)+(ref_AppliedPTransform_test/Create/FlatMap(<lambda at core.py:2826>)_48))+(ref_AppliedPTransform_test/Create/MaybeReshuffle/Reshuffle/AddRandomKeys_51))+(ref_AppliedPTransform_test/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)_53))+(test/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write)
2020-10-16T08:27:00-0400 INFO Running (((((((test/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read)+(ref_AppliedPTransform_test/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)_55))+(ref_AppliedPTransform_test/Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys_56))+(ref_AppliedPTransform_test/Create/Map(decode)_57))+(ref_AppliedPTransform_test/FlatMap(_generate_examples_from_stm_file)_58))+(ref_AppliedPTransform_test/Encode_59))+(ref_AppliedPTransform_test/SerializeBucketize_60))+(test/GroupByBucket/Write)
2020-10-16T09:01:11-0400 INFO Running ((((ref_AppliedPTransform_train/Create/Impulse_90)+(ref_AppliedPTransform_train/Create/FlatMap(<lambda at core.py:2826>)_91))+(ref_AppliedPTransform_train/Create/MaybeReshuffle/Reshuffle/AddRandomKeys_94))+(ref_AppliedPTransform_train/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)_96))+(train/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write)
2020-10-16T09:01:11-0400 INFO Running (((((((train/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read)+(ref_AppliedPTransform_train/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)_98))+(ref_AppliedPTransform_train/Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys_99))+(ref_AppliedPTransform_train/Create/Map(decode)_100))+(ref_AppliedPTransform_train/FlatMap(_generate_examples_from_stm_file)_101))+(ref_AppliedPTransform_train/Encode_102))+(ref_AppliedPTransform_train/SerializeBucketize_103))+(train/GroupByBucket/Write)

To me, download_and_prepare sounds like downloading a file, extracting it, and maybe moving some files around. I don't know why that would take this long. The 54GB file is downloaded. It's extracted. And the working directory has been at a constant 217GB. Yet, somehow, the disk is still filling up with... something. It doesn't seem to be using anything in /tmp. My home directory's storage has been constant. So I have no idea where this crap is getting saved.

The code is difficult to trace, and I can't figure out what earthly, reasonable function it could be doing that I actually want it to be doing. As such, it seems like the documentation should reflect what the bulk of the work actually entails, so that I know whether or not this is normal processing.

[Conchylicultor]

After downloading the files, download_and_prepare will read process each example into serialized format (tfrecord) into ~/tensorflow_datasets/dataset_name/version/train-00001-of-01024.tfrecord.

Here it seems you're using a Apache Beam dataset, https://www.tensorflow.org/datasets/beam_datasets, which might indicates that the dataset require too much processing to be generated on a single computed.
Apache Beam is likely the reason temporary files are created.
The code which gets executed is the Tedlium dataset preprocessing code: https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/audio/tedlium.py#L203

[knightcode]

It hasn't output any tfrecord files at all. Is it suppose to output everything to a single file? Why isn't the file present, open, and being appended to?

[Conchylicultor]

I'm not sure how Apache Beam is implemented, it might be that the pipeline is still currently executed, so is only save on temporary files. Only the final datasets files would be recorded in the ~/tensorflow_datasets. However, if the code is running since 6 days, there might an issue with the implementation as the dataset doesn't look that big.

[junyaoshi]

Hi, I also ran into a similar issue when trying to run fitvid. The issue occurs when the fitvid code is downloading the robonet dataset (specifically robonet/robonet_64) from TensorFlow datasets. The process has also been running for a day or so, and it is stuck at the following output:

Generating splits...:   0%|                          | 0/1 [00:00<?, ? splits/s]W1010 22:50:55.549950 139939710800256 split_builder.py:207] **************************** WARNING *********************************
Warning: The dataset you're trying to generate is using Apache Beam,
yet no beam_runner nor beam_options was explicitly provided.
Some Beam datasets take weeks to generate, so are usually not suited
for single machine generation. Please have a look at the instructions
to setup distributed generation:
https://www.tensorflow.org/datasets/beam_datasets#generating_a_beam_dataset
**********************************************************************
I1010 22:50:55.552827 139939710800256 pipeline.py:184] Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
W1010 22:51:05.238499 139939710800256 pipeline_options.py:309] Discarding unparseable args: ['--output_dir', 'outputs/debug', '--n_future', '2', '--dataset', 'robonet', '--batch_size', '1', '--log_every', '100']
W1010 22:51:05.239382 139939710800256 environments.py:371] Make sure that locally built Python SDK docker image has Python 3.8 interpreter.
I1010 22:51:05.239497 139939710800256 environments.py:380] Default Python SDK image for environment is apache/beam_python3.8_sdk:2.33.0
I1010 22:51:19.020636 139939710800256 translations.py:650] ==================== <function annotate_downstream_side_inputs at 0x7f409034f0d0> ====================
I1010 22:51:19.022668 139939710800256 translations.py:650] ==================== <function fix_side_input_pcoll_coders at 0x7f409034f1f0> ====================
I1010 22:51:19.023658 139939710800256 translations.py:650] ==================== <function pack_combiners at 0x7f409034f670> ====================
I1010 22:51:19.027122 139939710800256 translations.py:650] ==================== <function lift_combiners at 0x7f409034f700> ====================
I1010 22:51:19.029040 139939710800256 translations.py:650] ==================== <function expand_sdf at 0x7f409034f8b0> ====================
I1010 22:51:19.030132 139939710800256 translations.py:650] ==================== <function expand_gbk at 0x7f409034f940> ====================
I1010 22:51:19.032133 139939710800256 translations.py:650] ==================== <function sink_flattens at 0x7f409034fa60> ====================
I1010 22:51:19.033673 139939710800256 translations.py:650] ==================== <function greedily_fuse at 0x7f409034faf0> ====================
I1010 22:51:19.039504 139939710800256 translations.py:650] ==================== <function read_to_impulse at 0x7f409034fb80> ====================
I1010 22:51:19.039874 139939710800256 translations.py:650] ==================== <function impulse_to_input at 0x7f409034fc10> ====================
I1010 22:51:19.040270 139939710800256 translations.py:650] ==================== <function sort_stages at 0x7f409034fe50> ====================
I1010 22:51:19.042296 139939710800256 translations.py:650] ==================== <function setup_timer_mapping at 0x7f409034fdc0> ====================
I1010 22:51:19.042983 139939710800256 translations.py:650] ==================== <function populate_data_channel_coders at 0x7f409034fee0> ====================
I1010 22:51:19.054950 139939710800256 statecache.py:172] Creating state cache with size 100
I1010 22:51:19.056337 139939710800256 worker_handlers.py:890] Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7f4058503e20> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
I1010 22:51:19.056640 139939710800256 fn_runner.py:607] Running ((((ref_AppliedPTransform_train-Create-Impulse_4)+(ref_AppliedPTransform_train-Create-FlatMap-lambda-at-core-py-2968-_5))+(ref_AppliedPTransform_train-Create-MaybeReshuffle-Reshuffle-AddRandomKeys_8))+(ref_AppliedPTransform_train-Create-MaybeReshuffle-Reshuffle-ReshufflePerKey-Map-reify_timestamps-_10))+(train/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write)
I1010 22:51:31.251776 139939710800256 fn_runner.py:607] Running (((((((train/Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read)+(ref_AppliedPTransform_train-Create-MaybeReshuffle-Reshuffle-ReshufflePerKey-FlatMap-restore_timestam_12))+(ref_AppliedPTransform_train-Create-MaybeReshuffle-Reshuffle-RemoveRandomKeys_13))+(ref_AppliedPTransform_train-Create-Map-decode-_14))+(ref_AppliedPTransform_train-Map-_process_example-_15))+(ref_AppliedPTransform_train_write-Encode_17))+(ref_AppliedPTransform_train_write-SerializeBucketize_18))+(train_write/GroupByBucket/Write)

Because the process is stuck, I have no way to tell how much more time it will take or how much has already been downloaded. I see some files in ~/tensorflow_datasets/download/extracted but the size of this folder is not changing. I also can't tell if there is an issue with the implementation. Any help or advice would be appreciated. Thanks!