Adding SqlServer DistributedCache breaks perceived tenant features, endpoints, and type definitions.

[willnationsdev]

We have an app that we've deployed to dev, test, and prod environments where prod has multiple servers. After noticing seemingly in-memory cache-related bugs exclusively in prod, we realized that we need to add an IDistributedCache implementation to fix the issue.

It's currently using the 1.3 codebase though we plan to upgrade as soon as the distributed cache issue is resolved. We decided to use Microsoft.Extensions.Caching.SqlServer for our distributed cache, like we've done before.

I first tried adding it before the AddOrchardCms() method and realized that doesn't work because OC's AddCaching() method overrides whatever prior IDistributedCache implementation there may have been with the in-memory one.

I then tried the following, which does indeed print both info logs successfully & register the SqlServerCache successfully:

public class Startup
{
    private readonly IConfiguration _configuration;

    public Startup(IConfiguration configuration)
    {
        _configuration = configuration;
    }

    public void ConfigureServices(IServiceCollection services)
    {
        var logger = LogManager.GetCurrentClassLogger();
        var connectionString = _configuration.GetConnectionString("DistCache");
        logger.info("Registering SqlServer for DistributedCache");
        services.AddDistributedSqlServerCache(options =>
        {
            options.ConnectionString = connectionString;
            options.SchemaName = "dbo";
            options.TableName = "DistCache";
        });
        
        services.AddOrchardCms()
            .AddDatabaseShellsConfiguration()
            .ConfigureServices((tenantServices, provider) =>
            {
                var distributedCaches = provider.GetServices<IDistributedCache>();
                if (distributedCaches.FirstOrDefault(c => c is SqlServerCache) is SqlServerCache sqlServerCache)
                {
                    logger.Info($"Registering {nameof(SqlServerCache)} to tenant services");
                    tenantServices.AddSingleton<IDistributedCache>(sqlServerCache);
                }
                else
                {
                    logger.Error($"No {nameof(SqlServerCache)} was found in the tenant service collection");
                }
            });
    }
}

However, when the app starts, all of the content items stop displaying in the OC admin as if all the data had disappeared. I've confirmed the data is still in the database, and undoing the change makes everything reappear in the GUI.

Why is this happening? And how do I resolve it?

I hope it's just that I'm registering the service incorrectly or that I have to override some sort of default configuration to make OC resume seeing all the existing content.

Edit: it looks like everything has disappeared because the ContentTypeDefinitions have all reverted to a default installation. The custom endpoints from our modules do not appear to be registered properly either (yielding a 404 response) even though the Features section of the tenant shows that the corresponding features are enabled.

[Piedone]

While I won't be able to help with this directly, unfortunately, since several versions ago, OC supports multi-node hosting, including distributed caching, out of the box. So, upgrading would fix this as well.

[willnationsdev]

Oh, interesting. I thought the version I had already supported it since I saw references to the IDistributedCache in other parts of the 1.3 codebase. I was gonna table this on the off chance that upgrading would fix it (though I wasn't hopeful).

If that'll do the job, then I'll report back if I continue to have issues after the upgrade is complete. Thanks!

[Piedone]

Wait, you're right, 1.3 was actually the version that got Redis support, which added full multi-node support I misremembered, sorry (but yeah, IDistributedCache should somewhat work even before that). However, later versions did fix things related to that, but I don't know how much of that was only Redis or generally multi-node.

You can probably take some things from the Redis implementation, but I think just registering an IDistributedCache how you did should just work (this is how the Redis module does it).

Are you using the DB content definition store? (If not, you really need to.) That indirectly uses IDistributedCache via DocumentManager). In its current implementation, I don't see why it would cause the issue you're seeing, however, since 1.3 on Mar 10, 2022 it has seen multiple bug fixes. So, while an unsatisfying answer, probably an upgrade would be the best first step.

[willnationsdev]

You can probably take some things from the Redis implementation, but I think just registering an IDistributedCache how you did should just work

Due to other requirements of our application, I needed to add the distributed cache at the application level before making the call to AddOrchardCms() which in turn means that OC forcibly replaces ours since it doesn't use a TryAddSingleton call. But as the code sample in the original post shows, we found a workaround for that.

Are you using the DB content definition store?

We are not, but I agree, we will need to make that adjustment too (would explain some previously inexplicable definition inconsistencies that have crept into a few of our tenants over time). Thanks for the tip/reminder. Will make sure to include this in our Distributed cache changes.

So, while an unsatisfying answer, probably an upgrade would be the best first step.

No worries. If upgrading is a potential out-of-the-box solution, then I'll be plenty satisfied. We won't do another prod release until all of it is upgraded AND the distributed cache issue is resolved anyway, so it's not like we need an immediate solution.

As mentioned, I'll comment here again after our upgrade is completed to let you/others know if we still experience any issues.

[Piedone]

Great then! Yes, 1.8.3 with Redis is an out-of-the-box solution for this.

[willnationsdev]

Finally was able to try deploying this code. I have a custom module that enables the OrchardCore.Tenants.Distributed feature in the Default tenant and which, in all tenants, registers the SqlServerCache as an IDistributedCache service, mimicking the Redis module's cache implementation. It's running OC 1.8.3 and all the migrations run successfully. Unfortunately, while things appear to work fine in the local and dev environments, any attempt to run the application in our test environment results in a stack overflow exception that fatally crashes the application.

Trace logs indicate the application successfully gets through all backend processing (e.g. loading tenants, running migrations, authenticating against AzureAD and returning, even processing the ocauth-ticket from cache); however, as soon as it actually tries to render a Razor MVC view, there's one final log message from RazorViewEngine before the app seemingly crashes: "View lookup cache miss for view ..."

I can't reproduce the problem in local/dev environments, and the trace logs don't illuminate any further. Do you devs have any thoughts as to what the cause may be? Do razor views handle cache differently in non-Development environments or something?

Edit: @Piedone Curiously, this crash only appears to happen if I have the environment name set to something other than Development.

Edit 2: Nevermind I guess: we tried using the SQL Server Distributed Cache in a fresh install with a fresh database, activating the distributed tenants module in the Default tenant, and running it all in a "Staging" environment, and everything worked totally fine. So all the above has something to do with various customizations we introduced into our own system (somehow). Nothing you guys could really help with after all.