Submodule: Level-2 ImproPhe for Harmonization

[cshatto]

Hi all,

I am seeking to increase the spatial resolution of Landsat-8 (and potentially Landsat-9) to that of Sentinel-2. I originally sought to harmonize all Landsat satellites with Sentinel-2, creating a long harmonized time series, but it seems that this isn't possible with FORCE since harmonization must be done with active sensors that have overlapping time frames. Simply using RESOLUTION = 10 to perform analysis at 10m is useful but since LND08 pixels are duplicated 9x, I think harmonization (via ImproPhe) would provide a more accurate dataset that also aligns the LND08 grid with the S2 one.

Using the submodule Level-2 ImproPhe, I have attempted to harmonized LND08 & S2 imagery for year 2020. As it stands, the ETA is 01m 04d 21h 35m 13s (running already for roughly 16.5 hrs; nthread - 8/22/4). This seems like a rather excessive amount of time. Attached is my L2IMP .prm file (attempted using force-higher-level, not force-improphe). The ./level2 folder has a single datacube-definition.prj for the entire datacube as all images were downloaded and pre-processed via the same force-level1-csd and force-higher-level programs.

L2IMP-skeleton.txt

Chris

[davidfrantz]

Hi Chris,

the module is very compute-heavy. There might be some ways of (slightly) improving performance, but first (as you are using the default values for threading):

how many CPUs do you have on this system?

Best,
David

[cshatto]

Hi David,

Thanks for the quick response. I am running 40 CPUs with 128 RAM. I also tried the process using 4/7/2 as done in another improphe discussion (#255) but it struggled to get off the ground after about 30 minutes.

Best,
Chris

[ernstste]

Just on a side note: force-level1-csd is the interface for Google Cloud Storage, which only hosts Landsat Collection 1 data. You may want to consider switching over to force-level1-landsat for Landsat downloads (l1-csd is still fine for Sentinel-2).

[cshatto]

Thanks @ernstste for this suggestion. I use force-level1 csd because when using force-level1-landsat, it returns this message:

This tool is deprecated and will only receive minimal support in the future.
Please consider using force-level1-csd instead

Usage: force-level1-landsat from to queue cp/mv [dry]

I haven't been able to see anything that looks like the stuff on the "Lower Level" page for force-level1-landsat. I've double checked all the dependencies are installed, particularly 'aria2'.

[ernstste]

We repurposed the old force-level1-landsat program name for new functionality, so if you're running an older version of FORCE (earlier than 3.7.9), the message you posted will pop up. If you do need to stay with your current version of FORCE for some reason, you can use a newer FORCE docker image just for downloading or alternatively install landsatlinks, which is what FORCE uses under the hood.

[cshatto]

I see. I'll update FORCE and give it a shot. But I do already have landsatlinks installed. I'll double check this is installed correctly/up-to-date.

Thanks!

[davidfrantz]

OK, this means you are wasting a lot of compute resources. The data fusion is solely compute-bound. Better use 1/40/1. This slightly overtasks, but it will ensure that all CPUs are doing something.

This will probably cut the processing time into half.

---

Other ways of speeding things up are by subsetting as much as possible.
As an example, if you do not want to work with January imagery, leave them out.
If you do not need wall-to-wall coverage, work with masks (DIR_MASK), either a subset of the area (e.g. some district), or even some specific land-cover (e.g. only agriculture).

But in the end, it will definitely take quite some time...

[cshatto]

I see. Thanks for that.

I redistributed my CPU usage as you suggested but the processing time is still well over one month. Could this have anything to do with having used a WKT for my Level-2 ARD process without having changed the default ORIGIN LAT/LONG?

Here is that Level-2 .prm:
LEVEL2-skeleton.txt

[davidfrantz]

no, this is unrelated.

Increasing CPUs should normally increase speed. Do you have 40 physical CPUs? Or rather something like 20 CPUs with 2 hyperthreads each?

[cshatto]

It is 40 physical CPUs. I suppose it could be the latter but Force shows that I have 40 available.

In other news, I changed my study area to a slightly smaller tile in Germany and though it has an ETA of 3 days for the remaining ~150 images, I can confirm the output of LND08 is 10m resolution. There are (only) 6 bands with values but they're now labeled NA.

There are 2 issues with my previous study area that I'm pondering.

1. Its an island and the tiles overlap some areas without DEMs (as they're not available). This was no problem for the Level-2 ARD but could it now be an issue?
2. I am attempting to process 2013 LND08 images ... I only considered that LND08 & S2 have an overlapping period but failed to consider whether is it only possible to harmonize the overlapping time periods between satellites. Is this the case?

Thanks again for all the help, I know this is a rather particular case.

Best,
Chris

[davidfrantz]

most likely, you will have 20 physical cores, with 2 threads each. (lscpu)

In other news, I changed my study area to a slightly smaller tile in Germany and though it has an ETA of 3 days for the remaining ~150 images, I can confirm the output of LND08 is 10m resolution. There are (only) 6 bands with values but they're now labeled NA.

I am not sure if I understand you correctly. Did it work?

1. Can you elaborate please, I am not sure what you mean..
2. Yes, you can only harmonize during the overlapping period (you could rename the 2013 files to another year, though, as a dirty workaround)
   Best,
   David

[cshatto]

Yes, its currently running and I've confirmed by checking the output files. They are 10m resolution but with only 6 bands that have values and are labeled 'NA'. Its still has an ETA of roughly 4 days which is much better than it was before. However, the ETA is slowly increasing (it was ~3 days yesterday).

To elaborate on the issue with my other study area. I was speculating that my srtm.vrt file was generating the error since I don't have DEMs covering the entire tile. I don't have DEMs for those parts because they are over oceans. This wasn't an issue when using force-level2 but I thought it could be the issue when attempting the improphe with force-higher-level. But because I was trying to harmonize 2013 LND08 data, would this explain the excessive processing time?

[cshatto]

Hi @davidfrantz,

I want to follow-up with this.

Even though the processing time was excessive, it worked! However, my projection was off (attempted to use WKT string) and it sent my fully processed German tiles into the Atlantic (I can see vegetation in the images). I switched back to the GLANCE7 projection and it reduced my processing time immensely, but now I have the issue that it doesn't process the entire LND08 scene. It processes one strip and blacks out the rest of the scene. I attached my parameter files and the L8_IMP product.

Chris

L2IMP-skeleton.txt
LEVEL2-skeleton.txt
20200820_LEVEL2_LND08_IMP.zip

[davidfrantz]

Dear Chris,
I have to admit that I don't fully understand what you are trying to describe...
I will try to have a look at your files soon, though.
Best,
David

[cshatto]

Hi David,
Sorry for the confusion! I was elaborating on my experience fixing one of the previous issues I was having.

Here is my current challenge:
When attempting to run the improphe submodule (force-higher-level /L2IMP-skeleton.prm), it yields only a partially processed tile.
I'm running everything out of the latest docker container.

Chris