[force-lut-modis] Buffer Overflow in assembling app key

[thielfab]

When running force-lut-modis I get a Buffer Overflow in assembling app key error message.
Any idea why that is?
I have generated a token at Earthdata, which is located in ~/.laads.

[davidfrantz]

hmm, does your token happen to have more than 256 characters - incl. this prefix?

Authorization: Bearer %s

[thielfab]

It already has 578 characters without the prefix.

[davidfrantz]

seems like they boosted up security...

[davidfrantz]

I increased the allowed char length to 4096 in the dev branch. Can you please check if this does the trick for you?

[thielfab]

It seems to solve the issue regarding the buffer overflow. Now it says it is downloading, but no file is actually downloaded after 10 tries:

[~/data/wvdb]$ force-lut-modis wrs-2-land.coo . geometa/ hdf/                                                                                                    2000/02/24. do. Download TERRA geometa. Try # 10

  0.0% of useful frames.
2000/02/25. do. Download TERRA geometa. Try # 10

  0.0% of useful frames.
2000/02/26. do. Download TERRA geometa. Try # 10

  0.0% of useful frames.
2000/02/27. do. Download TERRA geometa. Try # 10

  0.0% of useful frames.
2000/02/28. do. Download TERRA geometa. Try # 10

[...]

[davidfrantz]

Hi @thielfab, have you tried again? To make sure that the interruption was not on LAADS' end?

[davidfrantz]

btw, we have a ready-to-go global dataset here; https://zenodo.org/record/4468701#.YyQm2HZBy5g

This is not up-to-date, but you can avoid re-processing data from < 2021.

[thielfab]

It still does not seem to work. Is there a way to "debug" the download or get more information on whats the issue?
So far I used the mentioned database (v2) for the processing. Also for 2021 data (in cases without matching daily tables the climatology tables are used, right?). Maybe I'll reprocess those once I have an updated database.

[davidfrantz]

To be sure, I just checked on my end.
It is working here, thus it is definitely an issue on your end.

As it is already failing to download the geometa tables, I assume that something is off with your app key...

What happens if you use curl directly?

curl -H "Authorization: Bearer YOURKEY" https://ladsweb.modaps.eosdis.nasa.gov/archive/geoMeta/61/TERRA/2005/MOD03_2005-01-01.txt -o MOD03_2005-01-01.txt

[thielfab]

Yes, I think so. I also copied the key from the .laads file to run your curl snippet.

[davidfrantz]

Hey, thanks for sharing the key in private. I could not make it work with your key though (neither with curl nor with FORCE). Are you sure that the download worked on your end? If it fails, an empty file will be generated.

[thielfab]

You are right, I did not check the file. It is empty...

[davidfrantz]

Alright, then I assume your app key is the culprit...

[thielfab]

Seems so. I generated a new token, but no luck so far, I tried the download with aria2c and it clearly gives me an errorCode=24 Authorization failed. error.
Just to check, there is only one place to generate a token, right?

[davidfrantz]

There is a list of authorized apps:

I don't remember exactly, but it seems like you need to authorize services separately?

[davidfrantz]

This?

[thielfab]

I also accepted those EULAs, yes.

[davidfrantz]

I am running out of ideas unfortunately...

The only other thing to test that I can think of: try to insert the provided URL in your browser and see what happens... In my case, it downloads the file after being logged in

[thielfab]

Thanks for your help so far David! Browser download works just fine.
I'm also wondering why there is no mention of the Bearer Token Authorization in the documentation https://urs.earthdata.nasa.gov/documentation/for_users/data_access/curl_and_wget
I think I will stop trying now. Maybe some day it magically works. Until then, I ssh to Berlin and steal their processed tables ;-)

[davidfrantz]

Stealing from HU is a very good idea 🤪

See this documentation for Bearer authorization: https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/

But in any case, MODIS is a living dead satellite... We will need some other workaround anyways in the foreseeable future

[thielfab]

So I revisited this issue, and so far no luck using force-lut-modis. But: I managed to download the file mentioned in your curl snippet above using wget (as described here), as well as aria2c (aria2c --header "Authorization: Bearer TOKEN" https://ladsweb.modaps.eosdis.nasa.gov/archive/geoMeta/61/TERRA/2005/MOD03_2005-01-01.txt)

So my token seems to be working at least. But not with curl and hence not inside FORCE.

[davidfrantz]

Moin moin @thielfab,

I am puzzled... it works for me, thus I believe that this is not the fault of FORCE or curl...
It might be newer key generations, but honestly I am out of ideas...

I am also not sure if we should invest time in fixing this. It would actually be nicer to find a more sustainable solution:
#260

frantz@ahsoka:~/temp/wvdb$ dforce force-lut-modis -d 20100801,20100805 wrs-2-land_sub.coo wvdb/ geo/ hdf/
2010/08/01. do. Download TERRA geometa. Try #  1
 1Download AQUA geometa. Try #  1
 1 50.0% of useful frames.
2010/08/02. do. Download TERRA geometa. Try #  1
 1Download AQUA geometa. Try #  1
 1 60.0% of useful frames.
2010/08/03. do. Download TERRA geometa. Try #  1
 1Download AQUA geometa. Try #  1
 1 70.0% of useful frames.
2010/08/04. do. Download TERRA geometa. Try #  1
 1Download AQUA geometa. Try #  1
 1 60.0% of useful frames.
2010/08/05. do. Download TERRA geometa. Try #  1
 1Download AQUA geometa. Try #  1
 1 20.0% of useful frames.
build climatology!