prx: evaluate.py handle cases when NAV files are not perfect or missing data #82

Raviu56 · 2024-05-30T17:32:24Z

Rinex NAV files from NASA hourly for example can usually miss data like some satellites or so even when merged from many stations. This commit fixes cases for missing some data or where data is not perfect, so as computation can be done on available satellites. Caller should handle cases where a subset of query is not fulfilled.

…ng data Rinex NAV files from NASA hourly for example can usually miss data like some satellites or so even when merged from many stations. This commit fixes cases for missing some data or where data is not perfect, so as computation can be done on available satellites. Caller should handle cases where a subset of query is not fulfilled.

Solve this issue https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

plutonheaven · 2024-06-01T06:59:27Z

Hi @Raviu56, and thanks for the contribution.
Could you hint the RINEX NAV file on which there was an issue. Do not add it to the repo, just mention the name in the PR description. That'll allow us to reproduce the problem.
Thanks!

Raviu56 · 2024-06-01T11:53:25Z

@plutonheaven
Actually many if not most of files from NASA hourly Mixed Nav, even after repair with gfzrnx - different errors after repair -
Hard to specify a specific one as each shows different error;

Pick some of MN.rnx.gz files from here or any other recent hour and test yourself please.

https://cddis.nasa.gov/archive/gnss/data/hourly/2024/146/06/

Raviu56 · 2024-06-01T12:07:59Z

You can use some bash script like this - does need an account - and test them in a for loop;

cd $(mktemp -d)
wget --ftp-user=anonymous --ftp-password='anonymous@' "ftps://gdc.cddis.eosdis.nasa.gov/gnss/data/hourly/2024/146/06/*_MN.rnx.gz"
gunzip  *_MN.rnx.gz

for R in *_MN.rnx; do
echo "from pathlib import Path; from prx.rinex_nav import evaluate as prx_evaluate; prx_evaluate.parse_rinex_nav_file(Path('$R'))" | python ;
done

You can repeat the test running this before python code;

for R in *_MN.rnx; do gfzrnx -finp $R -fout $R -chk -vo 3 -vosc 3 -f ; done

Still many errors.

Hope this fulfills your request
@plutonheaven

You can see the huge number of files missing data (despite mixed) and having errors.

Raviu56 · 2024-06-01T13:37:39Z

For Galileo error - as it's harder to find -
check those files:

LICC00GBR_R_20241460600_01H_MN.rnx
WTZZ00DEU_R_20241460600_01H_MN.rnx

It's found almost in each hour in ~3 randoms files, after patching other errors; you'd get:

AssertionError: Unexpected Galileo navigation message type: {0.0}
AssertionError: Unexpected Galileo navigation message type: {0.0, 1.0, 2.0}

And so.

Even after fixing them with latest gfzrnx 2.1.9 those 2 files still shows the errors.

By the way, I'm not a programmer by any definition, I just did my best to solve the errors I'd, so you may find a better solution now, as all test cases are in your hands.

Raviu56 · 2024-06-01T13:51:07Z

For error:

ValueError: System I NAV data is not the same length as the number of fields.

I could not/did not solve them; I simply remove all instances of IRNSS with by adding -satsys CEGJRS to gfzrnx
line

plutonheaven · 2024-06-03T22:02:50Z

src/prx/rinex_nav/evaluate.py

+                    helpers.get_gpst_utc_leap_seconds_from_rinex_header(rinex_file)
+            )
+        except:
+            ds.attrs["utc_gpst_leap_seconds"] = 18


This solution is not future proof. I have just coded a leap seconds function, that I will push to your branch.

plutonheaven · 2024-06-03T22:19:21Z

I added a function that compute leap seconds from the astropy library. It should be future-proof.
I removed your hard-coded try: except: block that was setting leap seconds to 18 in case of exceptions.

cd $(mktemp -d)
wget --ftp-user=anonymous --ftp-password='anonymous@' "ftps://gdc.cddis.eosdis.nasa.gov/gnss/data/hourly/2024/146/06/*_MN.rnx.gz"
[...]

The bash commands did not work in my terminal... As I am not familiar with bash, I just downloaded manually the NASA hourly files and created a new test with some of the files you mentioned. @Raviu56, it would be great if you added some other files that were causing the different issues you corrected.

Thanks for your weekend contributions!!

tn-helmen · 2024-06-05T10:14:59Z

dear all,
i checked the mrged file of the NASA source and to me the outputs for IRNSS look formally fine.

gfzrnx -finp *.rnx -fout xxxx -f
gfzrnx -finp *.rnx -fout xxxx -f -vo 4

can you figure out more clear the problem on a single broadcast epoch ?

gfzrnx developer

jtec · 2024-06-06T06:59:16Z

src/prx/helpers.py

-    assert "LEAP SECONDS" in header, "LEAP SECONDS not found in RINEX header"
-    ls_before = header["LEAP SECONDS"][0:6].strip()
+
+    if "LEAP SECONDS" not in header:


Was wondering - is there a reason to not always use the astropy leap second? It would remove a small piece of complexity, and I see now drawback.

As an alternative, why not compare the header leap second (when present) to the astropy leap second, as an additional sanity check?

Modified in #81, 9b52d25

src/prx/helpers.py

jtec · 2024-06-06T07:05:10Z

src/prx/rinex_nav/evaluate.py

@@ -25,7 +25,7 @@ def cached_parse(rinex_file: Path, file_hash: str):
    def cached_load(rinex_file: Path, file_hash: str):
        ds = cached_parse(rinex_file, file_hash)
        ds.attrs["utc_gpst_leap_seconds"] = (
-            helpers.get_gpst_utc_leap_seconds_from_rinex_header(rinex_file)
+                helpers.get_gpst_utc_leap_seconds_from_rinex_header(rinex_file)


we should pick one order of utc and gpst here and apply it to both the ds.attrs field and the function names.

jtec · 2024-06-06T07:06:37Z

src/prx/rinex_nav/evaluate.py

@@ -496,11 +497,13 @@ def compute_ephemeris_and_clock_offset_reference_times(group):
        }
    )
    df = df.reset_index(drop=True)
-    df = compute_gal_inav_fnav_indicators(df)
+    if (df.sv.str[0] == "E").any():


Nice, thanks for adding those checks.

jtec · 2024-06-06T07:09:35Z

src/prx/helpers.py

@@ -91,6 +94,9 @@ def timestamp_2_timedelta(timestamp: pd.Timestamp, time_scale):


 def timedelta_2_weeks_and_seconds(time_delta: pd.Timedelta):
+    if pd.isnull(time_delta):


Thanks for adding the check!

jtec · 2024-06-06T07:10:14Z

src/prx/rinex_nav/evaluate.py

@@ -591,14 +598,16 @@ def find_ephemeris_index(row, df):
    df = df.reset_index(drop=True)
    df["ephemeris_index"] = df.index

-    query[query.ephemeris_index.isna()]["ephemeris_index"] = df.index[len(df) - 1]
+    # Use .loc to set the values in the query DataFrame to solve SettingWithCopyWarning


jtec · 2024-06-06T07:11:55Z

src/prx/rinex_nav/evaluate.py

    # Copy ephemerides into query dataframe
    # We are doing it this way around because the same satellite might show up multiple times in the query dataframe,
    # e.g. with different query times
    query = query.merge(df.drop(columns=["sv"]), on="ephemeris_index")
    # For Galileo satellites we can have both F/NAV and I/NAV ephemerides for the same satellite and time, keep
    # only one
    # Compute times w.r.t. orbit and clock reference times used by downstream computations
+    query = query[query["ephemeris_reference_time_isagpst"].notna() & (query["ephemeris_reference_time_isagpst"] != '')]


It looks like the ephemeris_reference_time_isagpst column's dtype is string here, it should be float.

jtec · 2024-06-06T07:15:46Z

@Raviu56 It looks like github actions are not running on your fork. We could merge your branch into a branch in https://github.com/jtec/prx/ so that we get CI checks back - what do you think?

Raviu56 · 2024-06-06T09:56:37Z

@Raviu56 It looks like github actions are not running on your fork. We could merge your branch into a branch in https://github.com/jtec/prx/ so that we get CI checks back - what do you think?

I did not use github actions before, so do what you see suitable.

Raviu56 · 2024-06-06T10:27:40Z

@tn-helmen
Have you check before or after the merge with gfzrnx ?
For this error, it occurs before gfzrnx usage;

ValueError: System I NAV data is not the same length as the number of fields.

Here are list of files at day 155 that could be downloaded from here ftps://gdc.cddis.eosdis.nasa.gov/gnss/data/hourly/2024/155/10/*_MN.rnx.gz showing the error;

ALIC00AUS_R_20241551000_01H_MN.rnx.gz
DARW00AUS_R_20241551000_01H_MN.rnx.gz
GAMG00KOR_R_20241551000_01H_MN.rnx.gz
IITK00IND_R_20241551000_01H_MN.rnx.gz
JDPR00IND_R_20241551000_01H_30S_MN.rnx.gz
KARR00AUS_R_20241551000_01H_MN.rnx.gz
KIRI00KIR_R_20241551000_01H_MN.rnx.gz
KITG00UZB_R_20241551000_01H_MN.rnx.gz
MAJU00MHL_R_20241551000_01H_MN.rnx.gz
MOBS00AUS_R_20241551000_01H_MN.rnx.gz
NAUR00NRU_R_20241551000_01H_MN.rnx.gz
PARK00AUS_R_20241551000_01H_MN.rnx.gz
PBR400IND_R_20241551000_01H_30S_MN.rnx.gz
PTGG00PHL_R_20241551000_01H_MN.rnx.gz
PTVL00VUT_R_20241551000_01H_MN.rnx.gz
SHLG00IND_R_20241551000_01H_30S_MN.rnx.gz
SOLO00SLB_R_20241551000_01H_MN.rnx.gz
TID100AUS_R_20241551000_01H_MN.rnx.gz
YAR300AUS_R_20241551000_01H_MN.rnx.gz
YARR00AUS_R_20241551000_01H_MN.rnx.gz

==========================

Have you checked the Galilio issue mentioned above here #82 (comment) ? It still shows errors after gfzrnx repair/merge?

plutonheaven · 2024-06-12T11:16:46Z

@Raviu56, I have added 2 of the files you mention to the test, and the problem is from the georinex package. For the moment, we cannot parse those NAV file with Indian constellation data.

Let's maybe raise an issue on the georinex repo?

plutonheaven · 2024-06-12T11:19:37Z

@tn-helmen, so nothing wrong on the side of gfzrnx. I take the opportunity to thank you for this solid tool, and also for coming here to discuss this issue.

Raviu56 · 2024-06-12T11:22:13Z

@Raviu56, I have added 2 of the files you mention to the test, and the problem is from the georinex package. For the moment, we cannot parse those NAV file with Indian constellation data.

Let's maybe raise an issue on the georinex repo?

Why not? Can you open it please and mention this?

@tn-helmen, so nothing wrong on the side of gfzrnx. I take the opportunity to thank you for this solid tool, and also for coming here to discuss this issue

Only Galileo issue is confirmed on gfzrnx side

Raviu56 added 2 commits May 30, 2024 20:31

rinx_nav evaluate.py: solve SettingWithCopyWarning

8cf9106

Solve this issue https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy

plutonheaven reviewed Jun 3, 2024

View reviewed changes

plutonheaven added 2 commits June 4, 2024 00:08

cherry picking commit with leap second computation with astropy

a1b020d

added test for incomplete RNX NAV file parsing

6689f39

jtec reviewed Jun 6, 2024

View reviewed changes

src/prx/helpers.py Show resolved Hide resolved

jtec reviewed Jun 6, 2024

View reviewed changes

src/prx/helpers.py Show resolved Hide resolved

jtec reviewed Jun 6, 2024

View reviewed changes

Added test files to raise error on Indian constellation

c93c654

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prx: evaluate.py handle cases when NAV files are not perfect or missing data #82

prx: evaluate.py handle cases when NAV files are not perfect or missing data #82

Raviu56 commented May 30, 2024

plutonheaven commented Jun 1, 2024

Raviu56 commented Jun 1, 2024

Raviu56 commented Jun 1, 2024

Raviu56 commented Jun 1, 2024 •

edited

Loading

Raviu56 commented Jun 1, 2024

plutonheaven Jun 3, 2024

plutonheaven commented Jun 3, 2024

tn-helmen commented Jun 5, 2024

jtec Jun 6, 2024 •

edited

Loading

plutonheaven Jun 6, 2024

jtec Jun 6, 2024

jtec Jun 6, 2024

jtec Jun 6, 2024

jtec Jun 6, 2024

jtec Jun 6, 2024

jtec commented Jun 6, 2024

Raviu56 commented Jun 6, 2024

Raviu56 commented Jun 6, 2024

plutonheaven commented Jun 12, 2024 •

edited

Loading

plutonheaven commented Jun 12, 2024

Raviu56 commented Jun 12, 2024

		@@ -91,6 +94,9 @@ def timestamp_2_timedelta(timestamp: pd.Timestamp, time_scale):


		def timedelta_2_weeks_and_seconds(time_delta: pd.Timedelta):
		if pd.isnull(time_delta):

prx: evaluate.py handle cases when NAV files are not perfect or missing data #82

Are you sure you want to change the base?

prx: evaluate.py handle cases when NAV files are not perfect or missing data #82

Conversation

Raviu56 commented May 30, 2024

plutonheaven commented Jun 1, 2024

Raviu56 commented Jun 1, 2024

Raviu56 commented Jun 1, 2024

Raviu56 commented Jun 1, 2024 • edited Loading

Raviu56 commented Jun 1, 2024

plutonheaven Jun 3, 2024

Choose a reason for hiding this comment

plutonheaven commented Jun 3, 2024

tn-helmen commented Jun 5, 2024

jtec Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

plutonheaven Jun 6, 2024

Choose a reason for hiding this comment

jtec Jun 6, 2024

Choose a reason for hiding this comment

jtec Jun 6, 2024

Choose a reason for hiding this comment

jtec Jun 6, 2024

Choose a reason for hiding this comment

jtec Jun 6, 2024

Choose a reason for hiding this comment

jtec Jun 6, 2024

Choose a reason for hiding this comment

jtec commented Jun 6, 2024

Raviu56 commented Jun 6, 2024

Raviu56 commented Jun 6, 2024

plutonheaven commented Jun 12, 2024 • edited Loading

plutonheaven commented Jun 12, 2024

Raviu56 commented Jun 12, 2024

Raviu56 commented Jun 1, 2024 •

edited

Loading

jtec Jun 6, 2024 •

edited

Loading

plutonheaven commented Jun 12, 2024 •

edited

Loading