E3SM v3 ocean and sea-ice meshes

[xylar]

It's time to make new E3SM v3 meshes. These include (please add or modify if needed):

Meshes with ice-shelf cavities:

• ECwISC30to60E3r2
  • New Mesh: ECwISC30to60E3r2 #689
  • Add new ECwISC30to60E3r2 mesh E3SM-Ocean-Discussion/E3SM#63
• IcoswISC30E3r2
  • New Mesh: IcoswISC30E3r2 #691
  • Add new IcoswISC30E3r2 mesh E3SM-Ocean-Discussion/E3SM#64
• SOwISC12to60E3r1
  • New Mesh: SOwISC12to60E3r1 #669
  • Add new SOwISC12to60E3r1 mesh E3SM-Ocean-Discussion/E3SM#61
• RRSwISC6to18E3r1
  • New Mesh: RRSwISC6to18E3r1 #668
  • Add new RRSwISC6to18E3r1 mesh E3SM-Ocean-Discussion/E3SM#59
• WCwISC14to60E3r1
• IcoswISC480E3r1
• IcoswISC240E3r1
• IcoswISC120E3r1
• ARRMwISC10to60E3r1

Consensus is that we also need meshes without ice-shelf cavities for LIGHT support and BGC:

• SO12to60E3r2
• E30to60E3r2
• Icos30E3r2
• RRS6to18E3r2
• WC14to60E3r2
• Icos480E3r2
• Icos240E3r2
• Icos120E3r2
• ARRM10to60E3r2

There will be PRs for each of these meshes to discuss details. This space is for discussing common decisions for all (or many) of them.

Progress so far:

• [MPAS-Ocean standalone] Allow land-ice topo to come from MPAS variables  E3SM-Project/E3SM#5553
• Add bathymetry remapping #566
• Add WCwISC14 and ARRMwISC12to60 meshes #575
• Switch to 64 vertical levels for 3 meshes #581
• Add a test case for extrapolating WOA 2023 data #604
• Add WOA23 initial condition #610
• Get global ocean resources and time steps based on mesh size #631
• Add general QU and Icos meshes #632
• Add data ice-shelf melt fluxes (DISMF) configuration E3SM-Project/E3SM#5698
• Add a global ocean test case for computing and running with data (prescribed) melt rates #580
• Increase the max Haney number to 20 #655
• Don't run remap_ice_shelf_melt without ice-shelf cavities #661
• Add NetCDF C version of mesh conversion tools MPAS-Tools#514
• Fix widen transect function MPAS-Tools#517
• Fix MOC edge sign and mask MPAS-Tools#518
• Add bandwidth-minimising mesh sorting routine MPAS-Tools#515
• Update to v0.22.0 MPAS-Tools#519
• Update to v0.22.0 conda-forge/mpas_tools-feedstock#82
• Update to MPAS-Tools v0.22.0 #662
• Sort mesh indices before saving culled global ocean meshes #657
• Clean up global ocean test group for large meshes #663
• Add RRS6to18 and RRSwISC6to18 meshes  #576
• Add a step for remapping sea surface salinity restoring #676
• Add a step for remapping iceberg climatologies  #677

Redi fixes would be nice to have but won't be available in time:

• A quasi-monotone flux limiter for isopycnal diffusion (Redi) E3SM-Ocean-Discussion/E3SM#53
• Still seeing temperature extrema (likely from Redi) in mesh spin-up #331

[xylar]

Bathymetry

Compass is currently using GEBCO 2022 combined with Bedmachine Antarctica v2.

Bathymetry is currently sampled at cell centers without any filtering or quadrature. This can lead to noisy results that are sensitive to the details of cell locations and poor representation of small-scale features. Do we want to improve this? @dengwirda, you have a quadrature tool. Is it ready for incorporation into the compass workflow?

[xylar]

@xylar, aren't you glad you opened all of this up for discussion?

Yes, absolutely! Totally serious! I'm thriving on us having an animated discussion of these issues. It's great to have a sense that we'll hopefully have everyone's feedback whose interested before we make v3 meshes. I'd much rather feel like we have something of a consensus than that I'm just dictating what we do because I'm the one implementing it in compass.

[philipwjones]

300-500 cores....you've just inadvertently thrown a gauntlet down...hate to see that kind of waste. Can you point me to your merged topography file? I won't spend a huge amount of time or take time away from other stuff, but I think a can spend a little time on this...

[xylar]

Sure:
https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/mpas-ocean/bathymetry_database/BedMachineAntarctica_v3_and_GEBCO_2022_0.0125_degree_20230315.nc

[xylar]

If you want to play the the full res GEBCO, it's here on LCRC:

/lcrc/group/e3sm/ac.xylar/bathymetry/GEBCO_2022_sub_ice_topo.nc

(Don't tell @rljacob that I have a 7 GB file on LCRC that could be deleted!)

[xylar]

The approach for adding ocean bathymetry and ice-shelf topography is being updated in #566. Anyone interested is invited to comment there.

[xylar]

Ocean Initial conditions

We use the PHC (Polar science center Hydrographic Climatology) data set for our ocean initial conditions. Are we still happy with that?

[milenaveneziani]

@xylar: I seem to understand that this is a partial release before the full WOA23 product will be available at the end of the year. I couldn't find anything about 23 vs 18 version.

[xylar]

this is a partial release before the full WOA23 product will be available at the end of the year.

Would you prefer to stick with WOA 2018 then, since it is the "release" version? I do recall that it caused some confusion 5 years ago with ISMIP6 when we used a preliminary WOA 2018 dataset that then was gone when folks wanted to reproduce our workflow.

[milenaveneziani]

I found what I was looking for, just needed some more digging. Here is what they say regarding this initial release (it is still an official release, just not 'full'):

The WOA23, initially released in October 2022, included the 1991–2020 climate normals of global ocean temperature and salinity at selected standard depths. These fields were, in part, generated to complement the recently released U.S. Climate Normals.

The full WOA23 release is planned for September 30, 2023 and will include oxygen, nutrients, density, conductivity, mixed layer depth, and bottom temperature, in addition to temperature and salinity climatological fields. WOA23 includes approximately 2.4 million new oceanographic casts added to the WOD since WOA18’s release, as well as renewed and updated quality control.

[milenaveneziani]

So I think we are good for the purposes of making the initial condition. We may consider switching to the full WOA23 product for MPAS-Analysis, especially for the inclusion of MLD.

[xylar]

For those following along, please take a look at #610, which points to some initial conditions on the EC30to60 mesh with the WOA23 initial condition. Please feel free to comment there. We don't have to do a thorough review just yet but it would be nice to know if clear problems with WOA23 jump out to anyone.

[xylar]

Ocean vertical coordinate

We have been sticking with the 60-layer PHC vertical coordinate (10 to 250 m resolution) for most meshes. Are we happy with that?

[xylar]

Yes, we could make 2 versions of each RRM mesh. We need to decide on 64 vs. 80 levels. We can discuss that on a specific mesh PR.

[philipwjones]

Thanks @vanroekel - yes, friendly factors on this axis will help, esp multiples of 16/32/64. We won't hit a perfect value for all machines but 64/80/96/128 are better than arbitrary decades (60,70, etc).

[xylar]

@vanroekel, how do you feel about switching from 60 to 64 layers but keeping the surface layers about the same resolution as before?

[vanroekel]

I would certainly be in support of that.

[xylar]

The ocean vertical coordinate is being updated in #581. Anyone interested is invited to comment there.

[xylar]

Ocean critical passages and land blockages

We have a fairly long list of critical passages and land blockages in geometric features. Are there any to be added?
https://github.com/MPAS-Dev/geometric_features/blob/main/geometric_features/features_and_tags.json#L1597-L1829

[xylar]

The Southern ocean doesn't need very many of these. Mostly just the Peninsula. I think we're in good shape.

[maltrud]

how do people feel about creating somewhat non-realistic passages? I dug out the Florida Strait between Florida and the Bahamas (mostly widened) in the EC30to60 and got ~10 more Sv of transport (didn't affect AMOC at all). this is what CESM did with the gx1 in order to get more transport--I used that bathymetry as a guide for my changes.

[proteanplanet]

@maltrud That's already done for some critical passages in the Arctic for the same reason, so there is precedent.

[milenaveneziani]

From what I understand, there is a difference in how we do it in the Arctic though (at least the way I did it for the critical passages that I added). I only modified the mesh, not the bathymetry.
I don't have a problem with artificially modifying key passages, but I wonder if we should be careful with the big channels, maybe looking at the real geometry (say width * depth) and try to stay within a certain range of that? But maybe you already do that, Mat.

[maltrud]

@milenaveneziani I agree we need to be very careful with this kind of thing. I didn't pay any quantitative attention to channel geometry, but I also didn't go too crazy with it. I just wanted to see what would happen to AMOC (which, again was essentially no change--verification of Sverdrup balance?). the flow just reorganized a bit by reducing the Antilles Current and increasing the Florida Current (both toward more realistic values).

[xylar]

Haney number or z-star coordinate in ice-shelf cavities

Idealized modeling has indicated that the Haney-number coordinate (a coordinate constrained so the Haney number is below 5) is not needed in ice-shelf cavities. It has some pretty undesirable properties (jagged bathymetry, very thick ocean layers) so it would be good ro move to z-star or to allow much larger Haney numbers in the new meshes.

[xylar]

@cbegeman, do you have thoughts on this?

[cbegeman]

@xylar Am I correct in assuming that we haven't tested this out recently in realistic meshes?

We could probably move to z-star and wait and see if that causes any problems. I haven't seen any issues related to z-star in my idealized tests. The more conservative option would be using Haney number constraint at around 20.

[xylar]

Yep, I'm inclined to use z-star in our first attempts and back off to Haney number of 20 if need be, as you suggest.

And you're right that this hasn't been tested rigorously in a global, realistic configuration so there's no guarantee that we can use z-star but there are no indications of trouble that we've seen in idealized modeling.

[xylar]

It's looking like z-star is going to be one bridge too far for this round of meshes.

MPAS-Ocean init mode doesn't support z-star with ice-shelf cavities. I thought this would be a good opportunity to switch to making global ocean initial conditions in python but that also hasn't been too successful yet, and also seems like a bridge too far.

So I'll go with the backup option of increasing the Haney Number to 20 for now and see how that goes.

[vanroekel]

Ice shelf cavities in all meshes?

One separate question I have about meshes -- Is v3 a good time to pursue use of ice shelf cavities across all groups (so having the coupled group use the wISC meshes for simulations)? If the polar group has a configuration they are happy with I think we could explore use of ice shelves in the coupled campaign. I could see advantages to pursuing that path, but this may not be the right time.

[xylar]

@cbegeman, as we discussed, that's a really great idea. I've been working on that:
https://github.com/xylar/compass/tree/add_prescribed_melt_fluxes
but I haven't had time to iron out the kinks yet. In principle it should be pretty easy so I'll have to remind myself where I was having trouble, or if it was just a question of finishing up the work.

[maltrud]

I am strongly opposed to only having meshes with ISC. don't you want to have the capability to compare runs with and without? and though it isn't the strongest reason, LIGHT doesn't work with ISC. I'm using it more and more for BGC (using multiple meshes), and it makes no sense to modify it since it's being completely rewritten in a scidac project. I appreciate that supporting more grids is extra burdensome, but I think it way too premature to jettison non-ISC meshes.

[xylar]

@maltrud, continuing to have meshes without ice shelf cavities. Do you need all 3 meshes (EC, WC and SO) without isc? That's fine if so.

[maltrud]

@xylar yes, please. and ARRM10 if it gets updated.

[vanroekel]

I'm perfectly happy to keep with and without meshes, the comparisons will be good of course. But I would like to keep on our radars a move toward just ice cavity meshes further down the road. Once we have the rewritten LIGHT and have done sufficient comparisons on with/without cavities it seems there will come a time where supporting both configurations is not necessary. But I agree with @maltrud's points that the time for that is not now.

[xylar]

ARRM

@milenaveneziani, should we include ARRM in this discussion or not? If so, do you want to consider including Antarctic ice-shelf cavities in that mesh?

[vanroekel]

I agree with @xylar that we should be moving toward ice-shelf cavities as the default.

[proteanplanet]

@xylar Great! if everyone is happy to switch off thermodynamics as a default, then I don't see any downside from including it in all meshes.

[milenaveneziani]

That is a good point about just using the geometry of the ice cavities without turning on the thermodynamics. I don't have anything against it.

[milenaveneziani]

ARRM is behaving well with the atm ArcticRRM mesh (arcticx4v1pg2), so that is another reason for keeping it.

[alicebarthel]

Just chiming in: I support the idea of having the ARRM mesh part of our standard meshes.

[xylar]

Variable resolution

From @dengwirda:

A more fundamental question re: model config.'s though --- is this mix of resolutions working well wrt. simulation quality?

[milenaveneziani]

Agreed on the salinity bias. SORRM has a drop of ~0.7mm/year, so about 0.4 m in 600 years.

[maltrud]

thanks for clarifying the SORRM, @milenaveneziani . I thought the SL drop was 10x bigger than that.

[xylar]

@maltrud, we could make new versions of the HR and RRS meshes but that wasn't my current plan.

[maltrud]

I was just curious if there was a plan to update given the changes in bathymetry, but I would say they're a low priority.

[vanroekel]

I'd very much prefer not trying to push for a bunch of new meshes or tinkering with resolution minimums for v3. V3 has to be finalized in only 7 months from now. I would suggest we keep trying to understand why SORRM or fully coupled ARRM have stable/robust AMOC and others don't and then push toward new meshes for v4.

[xylar]

RRS6to18

In discussions with @vanroekel this morning, it became clear that it is not presently a priority to make a new RRS mesh, though we will add support for building on to compass.

[xylar]

The priority for a new RRS mesh has been increased. We will try to have an E3SMv3 version, thought it will be lower priority than the other meshes

[xylar]

Adjacency of cells, edges and vertices

As discussed on this confluence page
https://acme-climate.atlassian.net/wiki/spaces/ED/pages/496174213/MPAS+ID+Numbering
(requires E3SM confluence access), we would like to renumber cells, edges and vertices to be adjacent to one another for better I/O and coupler performance.

[xylar]

This is taken care of.

[xylar]

Priority of the meshes

Please discuss here which meshes need to change in order of priority. For reference, here is the priority for meshes with ice-shelf cavities now:

• ECwISC30to60E3r1
• IcoswISC30E3r1
• SOwISC12to60E3r1
• WCwISC14to60E3r1
• RRSwISC6to18E3r1
• IcoswISC480E3r1
• IcoswISC240E3r1
• IcoswISC120E3r1
• ARRMwISC10to60E3r1

And for those without ice-shelf cavities:

• SO12to60E3r2
• E30to60E3r2
• Icos30E3r2
• RRS6to18E3r2
• WC14to60E3r2
• Icos480E3r2
• Icos240E3r2
• Icos120E3r2
• ARRM10to60E3r2

[xylar]

I'm not quite sure when I will stop making the wISC meshes and start making the ones without but we can discuss that as things progress.

[vanroekel]

for the wISC ones, I'd like to put the RRSwISC6to18 higher up the priority list (above SO and WC). Coupled group has expressed a lot of interest in that mesh.

[xylar]

Done.