Skip to content

Conversation

ssddanbrown
Copy link
Member

@ssddanbrown ssddanbrown commented Sep 15, 2025

This PR aims to merge the core entity types (books, chapters, pages, shelves) into one core table, with sub-tables for added data.

This moves things into three new tables:

  • entities - All items, containing most shared metadata.
  • entity_container_data - The added data for container (non-page) items.
  • entity_page_data - The page specific data.

This is something I've wanted to do for a while.
These core structural items are often queried and used in the same locations, and joint with many of the same relations.
This alignment allows querying & fetching & joining across all items via one indexed table, instead of needing to run multiple queries.

This should bring a lot of query efficiency, while allowing things like properly pageable and ordered search results (compared to the sketchy non-predictable-length handling we have now due to separate query results being combined together).

This also takes the opportunity to perform some data cleaning, like changing 0 relation Ids to nulls, switching to unsigned bigint for entity ids.

Code logic approach

  • We'll handle joins automatically at query level, so usage in almost all read-based cases can stay the same.

Considerations

  • As part of this, the use of numeric ids will be changing. Existing item ids should remain the same, but for new items, the id values will be incrementing across all entity items (although is not assured to be unique across all, to remain back compatible).
  • This new table relies strongly on multi-column primary keys, which have before caused issues in some environments, but I think should be fine to have here since they're already required for the joint_permissions table.
  • Migrations will take a while, will need to note this to ensure it's left. May cause trouble with watchtower instances. As a reference, my dev instance took 30s for the change in relation column types.

Todo

  • Complete data conversion migration.
    • Core migration of main data
    • Fix of created_by, updated_by, owned_by & chater_id columns to convert 0 to null, and remove where no relation exists
  • Complete entity relation column conversion migration.
  • Ensure we're handling deleting of books/shelves relations on destroy since they are no longer foreign key constrained.
  • Add query handling logic to app.
  • Update logical use in app.
    • Create new multi-column-key entity object for entities.
      • Really need this for working with base methods like refresh, find, delete etc...
      • Delete filtering at global scope level if possible? (Check single, query, and relation usage since scopes are used for all).
        • Done this at a more direct way instead by overriding the base queries to set the global scope.
    • Update entity model hydration to move extra data loaded (via joins) into relation rather than stuck in model.
    • Update entity model relations.
    • Update usage in repos.
    • Update usage in related services.
    • Update usage in views.
  • Address remaining code todo notes.
  • Check API responses against samples to ensure no changes.

Made some other tweaks/fixes while testing.
Updated base entity class, and worked through BaseRepo.
Need to go through other repos next.

Removed a couple of redundant interfaces as part of this since we can
move the logic onto the shared ContainerData model as needed.
Changes to core entity models are now done on clones to ensure clean
state before save, and those clones are returned back if changes are
needed after that action.
Added smart loading from builder instances which should hydrate with
"contents()" loaded via join, while keeping the core model original.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

1 participant