Shared metadata cache

[AlexPeshkoff]

Tasks to be solved.

• Have a single metadata cache that is used by all attachments in both jrd and dsql.
• Provide the ability to mix DLL/DML in a single transaction, including working with new/modified elements.

The old metadata cache is essentially single-threaded. For me, the problem I faced with attempts to introduce synchronization into this structure and get off with relatively little blood was taking into account the use of procedures and functions using a second reference counter, and in order to decide whether to delete an object, a global recalculation of these references is required throughout the cache. dsqlcache is also extremely unsympathetic, which puts objects created and still unmixed in a single transaction for all transactions of the same attach. An additional consideration is that objects stored in the cache are used in a huge number of places in the code, and they can be modified from other places (and threads) during MET_scan_xxx. This could still be dealt with using CopyOnWrite, but how to approach the first two points was unclear.

The new metadata cache is versioned. Each object is divided into 2 parts - a constant and a variable. For relation, for example, the type (table, view, external table, etc.), name, all locks (GCLock, partners lock, rescan lock) will be constant, and format, list of fields, triggers will be variable. The list of versions is linked to a permanent part of the object. As versions become obsolete, the list is cleared (although I won't say that it's very active - only as the cache is accessed, there is no special garbage collector).

In order to access the desired object for reading without locks, it was possible to allocate memory for the entire cache at once. In principle, there are not so many - 64K pointers to each type of cached objects. But I kept in mind the possible increase in the number of objects in the database and therefore made a 2-level array - each lower-level block of N objects is allocated once and for all (until the database is closed), and the upper-level block has the right to grow when the required number of lower-level blocks does not fit there. Therefore, the upper level is implemented using a read-safe array. For reading, we would take a slice of the current state of the array and this slice remains unchanged while we work with it. An external mutex is used to modify the array. This array (SharedReadVector) turned out to be very convenient, I also use it for requests in a statement and a list of formats in a relation. The 2-level arrays (CacheVector) for all objects are the same (template), and all this ultimately lives in DBB.

Ideally, I would like to remove dsqlcache altogether. So far, it has been removed for relations, procedures, and functions. At the same time, I did not abandon the dsql_rel format and the like - they are needed in order to be able to work with it when creating/changing an object (dsql_rel), although there is no such object in the metadata cache yet / or the format is not the same. Temporary objects of this format are created in dsqlScratch and die with it (by pool).

libcds is used to implement secure pointers (HazardPtr), they are needed for version headers in the list of object versions and for a slice in the SharedReadVector.

In all requests, instead of a pointer directly to objects, the cache stores a structure of 2 elements - a pointer to the permanent part of the object and a index in an array of pointers to versions of objects. These arrays may differ in different query clones. To reduce duplication, a RefCounted array is used. rpb_relation's are registered in rpb when creating a clone. The decision on whether to update a set of versions in a clone of the request is made in GetRequest, that is, when a clone of the request is received.

At the moment, TCS is fully working. QA makes mistakes - but I think it's time to pour metacache into the master so as not to prevent merge of other branches. And it's probably time to decide with the indexes of the system tables, which ones are needed and which ones are not very necessary. For example, I need to add indexes by ID for relations and other cached tables.