diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 000000000..b5e104416 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,258 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Build & Test Commands + +```bash +# Build all modules +./gradlew build + +# Run all tests +./gradlew test + +# Build and test (includes all checks) +./gradlew check + +# Run a single test class +./gradlew ::test --tests "com.contentgrid.appserver.ClassName" + +# Run a single test method +./gradlew ::test --tests "com.contentgrid.appserver.ClassName.methodName" + +# Generate test coverage reports (JaCoCo) +./gradlew jacocoTestReport + +# Run SonarCloud analysis +./gradlew sonar +``` + +## Project Structure + +Multi-module Gradle project using Java 21, Spring Boot 3.5.10, and JOOQ for database access. + +**Key Modules:** + +- **contentgrid-appserver-application-model**: Core domain model defining entities, attributes, relationships, and constraints +- **contentgrid-appserver-domain**: Business logic APIs for entities, relations, and content (DatamodelApi, ContentApi) +- **contentgrid-appserver-rest**: REST controllers providing dynamic HATEOAS endpoints based on application model +- **contentgrid-appserver-query-engine-api**: Database abstraction interfaces (QueryEngine) +- **contentgrid-appserver-query-engine-impl-jooq**: JOOQ-based implementation of query engine +- **contentgrid-appserver-contentstore-api**: Content storage interfaces (ContentStore) +- **contentgrid-appserver-contentstore-impl-fs**: Filesystem-based content storage +- **contentgrid-appserver-contentstore-impl-s3**: S3-compatible content storage +- **contentgrid-appserver-contentstore-impl-encryption**: Encryption wrapper for content stores +- **contentgrid-appserver-autoconfigure**: Spring Boot auto-configuration +- **contentgrid-appserver-integration-test**: Integration tests with Testcontainers + +## Architecture Overview + +### Schema-Driven Dynamic API + +The application is **schema-driven**: an `Application` model defines entities, attributes, relationships, and constraints. The REST layer dynamically generates endpoints based on this model. + +**Application Model Loading:** +``` +JSON Schema File → ApplicationSchemaConverter → Application Domain Object → ApplicationResolver → Injected into Controllers +``` + +**Core Domain Model:** +- `Application`: Top-level aggregate containing entities and relations with validation +- `Entity`: Maps to database table, contains attributes and search filters +- `Attribute` (sealed interface): `SimpleAttribute`, `CompositeAttribute`, `ContentAttribute` +- `Relation` (sealed abstract class): `ManyToOneRelation`, `OneToManyRelation`, `OneToOneRelation`, `ManyToManyRelation` +- `Constraint`: `RequiredConstraint`, `UniqueConstraint`, `AllowedValuesConstraint` + +The model uses **immutable value objects** and **sealed interfaces** for type safety. + +### Layered Architecture + +``` +REST Layer (EntityRestController, XToOneRelationRestController, XToManyRelationRestController, ContentRestController) + ↓ +Domain Layer (DatamodelApi, ContentApi) + ↓ +Query Engine API (QueryEngine interface) + ↓ +Query Engine Implementation (JOOQQueryEngine) + ↓ +Database / Content Store +``` + +Each layer depends on abstractions above, not implementations below. + +### REST Controllers & Dynamic Routing + +Controllers use `@SpecializedOnEntity(entityPathVariable = "entityName")` to dynamically resolve entities from URL paths. + +**Request Processing Pipeline:** +1. `ApplicationArgumentResolver`: Injects Application from ApplicationResolver +2. `AuthorizationContextArgumentResolver`: Extracts authorization context +3. `EncodedCursorPaginationHandlerMethodArgumentResolver`: Parses pagination parameters +4. `LinkProviderArgumentResolver`: Injects HAL link builders +5. Controller method invokes `DatamodelApi` +6. `EntityDataRepresentationModelAssembler`: Converts response to HAL+JSON + +**Endpoint Mapping:** +- `/{entityName}` → List/Create entities (EntityRestController) +- `/{entityName}/{id}` → Get/Update/Delete entity (EntityRestController) +- `/{entityName}/{id}/{relationName}` → Manage relations (XToOneRelationRestController, XToManyRelationRestController) +- `/{entityName}/{id}/{contentName}` → Upload/Download content (ContentRestController) + +### Query Engine Abstraction + +The `QueryEngine` interface provides: +- `findAll()`: Query with predicates, sorting, pagination +- `findById()`: Fetch single entity +- `create()`, `update()`, `delete()`: CRUD operations +- `linkRelation()`, `unlinkRelation()`: Relationship management + +**JOOQ Implementation Flow:** +``` +Search Filters → ThunkExpression → JOOQThunkExpressionVisitor → JOOQ Condition → SQL Query → EntityData +``` + +Key components: +- `JOOQThunkExpressionVisitor`: Converts filter expressions to SQL WHERE clauses +- `EntityDataMapper`: Maps JOOQ Record to EntityData DTOs +- `JOOQRelationStrategyFactory`: Polymorphic strategies for relation types +- `TransactionalQueryEngine`: Wraps JOOQQueryEngine with Spring transactions + +### Content Storage + +Content storage uses a **strategy pattern** with pluggable implementations: + +**`ContentStore` Interface:** +- `getReader()`: Stream content for download +- `writeContent()`: Upload content +- `remove()`: Delete content + +**Implementations:** +- `FilesystemContentStore`: File-based storage +- `S3ContentStore`: S3-compatible cloud storage +- `EncryptedContentStore`: Decorator for encryption + +### Event-Driven Callbacks + +Query operations accept consumer callbacks for domain events: +- `CreateEventConsumer`, `UpdateEventConsumer`, `DeleteEventConsumer` +- `LinkEventConsumer`, `UnlinkEventConsumer` + +This decouples persistence logic from event handling. + +### Key Architectural Patterns + +1. **Type Safety**: Sealed interfaces (`Constraint`, `Attribute`, `Relation`) ensure exhaustive pattern matching +2. **Value Objects**: `EntityName`, `AttributeName`, `EntityId`, `PropertyPath`, etc. provide type-safe identifiers +3. **Immutability**: `@Value` classes and unmodifiable collections prevent state mutations +4. **Builder Pattern**: Lombok `@Builder` with `@Singular` for complex object construction +5. **Delegation**: `Entity` delegates `Translatable` interface to reduce boilerplate +6. **Decorator Pattern**: `EncryptedContentStore` wraps any `ContentStore` implementation +7. **Strategy Pattern**: Content storage and relation handling use pluggable strategies +8. **Visitor Pattern**: `ThunkExpression` uses visitor for predicate evaluation + +## Code Style & Conventions + +### Language Features +- Java 21 with preview features enabled +- Use `var` for local variable declarations +- Maximum line length: ~120 characters +- 4-space indentation, no tabs + +### Import Ordering +1. `java.*` imports +2. `lombok.*` imports +3. Third-party libraries (Spring, etc.) +4. Project imports (`com.contentgrid.appserver.*`) + +### Naming Conventions +- Classes: PascalCase (e.g., `EntityRestController`) +- Methods/fields: camelCase (e.g., `getEntityOrThrow`) +- Constants: UPPER_SNAKE_CASE (e.g., `PRIMARY_KEY`) +- Test classes: `*Test.java` (not `*Tests`) +- Test methods: descriptive camelCase (e.g., `createEntity_withValidData_returns201`) +- Packages: lowercase with `com.contentgrid.appserver` prefix + +### Lombok Usage +- `@Getter` / `@Setter` for accessors +- `@RequiredArgsConstructor` for constructor injection +- `@Builder` for complex object construction +- `@NonNull` for null-safety +- `@Data` for data containers +- `@Value` for immutable classes +- Use `lombok.val` for immutable local variables + +### Exception Handling +- Create domain-specific exceptions in `exception` or `exceptions` packages +- Extend appropriate base exceptions: + - `QueryEngineException` for database errors + - `ApplicationModelException` for model validation + - `InvalidDataException` for input validation +- Exception names: `Exception` (e.g., `EntityIdNotFoundException`) + +### Testing +- JUnit 5 (Jupiter) with `@Test`, `@ParameterizedTest`, `@Nested` +- Spring Boot tests: `@SpringBootTest` + `@AutoConfigureMockMvc` +- AssertJ for assertions: `assertThat()` +- Hamcrest for matchers: `is()`, `containsString()` +- Integration tests use Testcontainers (PostgreSQL, RabbitMQ, MinIO) + +### REST API Guidelines +- Use Spring HATEOAS for hypermedia responses +- Return `Optional` for queries that may not find results +- Use `ResponseEntity` to control HTTP status codes and headers +- ProblemDetails (RFC 7807) for error responses +- ETags for optimistic locking + +### Documentation +- Javadoc for public APIs with `@param`, `@return`, and `@throws` tags +- Focus on "why" rather than "what" in comments + +## Key Files Reference + +**Core Model:** +- `contentgrid-appserver-application-model/src/main/java/com/contentgrid/appserver/application/model/Application.java` +- `contentgrid-appserver-application-model/src/main/java/com/contentgrid/appserver/application/model/Entity.java` +- `contentgrid-appserver-application-model/src/main/java/com/contentgrid/appserver/application/model/Attribute.java` +- `contentgrid-appserver-application-model/src/main/java/com/contentgrid/appserver/application/model/Relation.java` + +**REST Layer:** +- `contentgrid-appserver-rest/src/main/java/com/contentgrid/appserver/rest/EntityRestController.java` +- `contentgrid-appserver-rest/src/main/java/com/contentgrid/appserver/rest/ContentRestController.java` + +**Query Engine:** +- `contentgrid-appserver-query-engine-api/src/main/java/com/contentgrid/appserver/query/engine/api/QueryEngine.java` +- `contentgrid-appserver-query-engine-impl-jooq/src/main/java/com/contentgrid/appserver/query/engine/jooq/JOOQQueryEngine.java` + +**Domain Layer:** +- `contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/DatamodelApi.java` +- `contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/ContentApi.java` + +**Auto-Configuration:** +- `contentgrid-appserver-autoconfigure/src/main/java/com/contentgrid/appserver/autoconfigure/` + +## Request Flow Example + +**GET /entities/persons?first_name=John&page=1** + +1. Request arrives at `EntityRestController.listEntity()` +2. `ApplicationArgumentResolver` resolves Application from registry +3. `AuthorizationContextArgumentResolver` extracts auth context +4. `EncodedCursorPaginationHandlerMethodArgumentResolver` parses pagination +5. Controller calls `datamodelApi.findAll()` +6. `DatamodelApiImpl` converts query params to `ThunkExpression` predicates +7. `QueryEngine.findAll()` called with predicates +8. `JOOQThunkExpressionVisitor` converts to JOOQ Condition +9. Query builder creates SQL SELECT with WHERE, ORDER BY, LIMIT OFFSET +10. Results mapped via `EntityDataMapper` +11. `EntityDataRepresentationModelAssembler` converts to HAL+JSON +12. Response includes `_links` (self, next, prev, relation links) + +**PUT /entities/persons/{id}** + +1. Request body deserialized via `RequestInputDataJacksonModule` +2. `ConversionService` converts strings to proper types +3. `DatamodelApi.update()` called +4. `QueryEngine.update()` performs version check, authorization, validation, and database UPDATE +5. `UpdateEventConsumer` called with old/new EntityInstance +6. HAL response with updated entity and links diff --git a/agents/content-deletion-design.md b/agents/content-deletion-design.md new file mode 100644 index 000000000..01c839a43 --- /dev/null +++ b/agents/content-deletion-design.md @@ -0,0 +1,219 @@ +# Content Deletion Design + +## Overview + +ContentGrid previously did not delete content from the ContentStore when: +- Entity was deleted +- Content property was set to null + +This design implements a **verified reference counting** approach to safely delete orphaned content. + +## Design Decisions + +| Aspect | Decision | Rationale | +|--------|----------|-----------| +| Tracking | Reference counting table `_content_references` | Incremental tracking, supports multi-entity sharing | +| Safety check | Query each entity table before deletion | Catches count drift, prevents premature deletion | +| Drift handling | Alert only (log + metrics) | Manual intervention required | +| Deletion trigger | K8s CronJob (one-shot) | Full app context, scales independently | +| Grace period | Configurable (default 7 days) | Allows recovery, ensures backup coverage | + +## Architecture + +``` +┌─────────────────────────────────────────────────────────────────────┐ +│ contentgrid-appserver-app │ +│ ┌──────────────────────────────────────────────────────────────┐ │ +│ │ HTTP API Layer (Controllers) │ │ +│ └──────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ┌───────────────────────────┼───────────────────────────────────┐ │ +│ │ domain module │ │ +│ │ ┌─────────────────────┐ │ ┌─────────────────────────────┐ │ │ +│ │ │ContentUploadMapper │ │ │ContentModificationValidator│ │ │ +│ │ │(upload → increment) │ │ │(null → decrement) │ │ │ +│ │ └─────────────────────┘ │ └─────────────────────────────┘ │ │ +│ └───────────────────────────┼───────────────────────────────────┘ │ +│ │ │ +│ ┌───────────────────────────┼───────────────────────────────────┐ │ +│ │ content-lifecycle module │ │ +│ │ │ │ │ +│ │ ┌───────────────────────┴───────────────────────────────┐ │ │ +│ │ │ ContentReferenceTracker │ │ │ +│ │ │ - incrementReference() │ │ │ +│ │ │ - decrementReference() (afterCommit) │ │ │ +│ │ └───────────────────────────────────────────────────────┘ │ │ +│ │ │ │ │ +│ │ ┌───────────────────────┴───────────────────────────────┐ │ │ +│ │ │ _content_references table │ │ │ +│ │ │ content_id | ref_count | last_dereferenced | marked │ │ │ +│ │ └───────────────────────────────────────────────────────┘ │ │ +│ └───────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ + + K8s CronJob (configurable schedule) + │ +┌──────────────────────────┼──────────────────────────────────────────┐ +│ content-lifecycle module (ContentDeletionJob) │ +│ ┌───────────────────────┴───────────────────────────────┐ │ +│ │ 1. Find candidates: marked_for_deletion_at <= now │ │ +│ │ 2. Safety check: query all entity tables │ │ +│ │ 3a. If referenced: clear mark, log drift alert │ │ +│ │ 3b. If orphaned: delete from ContentStore, DEKs, │ │ +│ │ _content_references │ │ +│ └───────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────┘ +``` + +## Database Schema + +```sql +CREATE TABLE _content_references ( + content_id VARCHAR(36) PRIMARY KEY, + reference_count INTEGER NOT NULL DEFAULT 1, + first_referenced_at TIMESTAMP NOT NULL DEFAULT NOW(), + last_dereferenced_at TIMESTAMP, + marked_for_deletion_at TIMESTAMP +); +``` + +## Components + +### ContentReferenceTracker (interface) +Location: `contentgrid-appserver-content-lifecycle` + +```java +public interface ContentReferenceTracker { + void incrementReference(ContentReference ref); + void decrementReference(ContentReference ref); +} +``` + +### JooqContentReferenceTracker +- Implements reference counting with JOOQ +- On increment: INSERT or UPDATE (upsert), clear deletion marker +- On decrement: Reduce count, mark for deletion if count reaches 0 + +### DeferredContentReferenceTracker +- Wraps `ContentReferenceTracker` with after-commit callback +- Uses `TransactionSynchronizationManager` to ensure decrements only happen after successful commit +- Prevents premature deletion if transaction rolls back + +### ContentReferenceVerificationQuery +- Safety check before deletion +- Queries each entity table with content attributes +- Returns true if content_id is still referenced anywhere + +### ContentDeletionJob +- One-shot job for K8s CronJob +- Finds content past grace period +- Verifies not referenced, then deletes +- Metrics: `content.deletion.success`, `content.deletion.failure`, `content.deletion.drift` + +## Integration Points + +### 1. Content Upload (`ContentUploadAttributeMapper`) +After `contentStore.writeContent()`: +```java +if (contentReferenceTracker != null) { + contentReferenceTracker.incrementReference(contentAccessor.getReference()); +} +``` + +### 2. Content Dereference (`ContentAttributeModificationValidator`) +When content is set to null: +```java +// In validate() method - track dereferenced content IDs +if (hasContent && dataEntry instanceof NullDataEntry) { + // Extract content_id and add to dereferencedContentIds list +} +``` + +Then in `DatamodelApiImpl.update()` and `updatePartial()`: +```java +for (String contentId : contentModificationValidator.getDereferencedContentIds()) { + if (contentReferenceTracker != null) { + contentReferenceTracker.decrementReference(ContentReference.of(contentId)); + } +} +``` + +### 3. Entity Delete (`DatamodelApiImpl.deleteEntity()`) +After entity deletion: +```java +if (contentReferenceTracker != null) { + for (var contentAttr : entity.getContentAttributes()) { + // Extract content_id from deleted entity + // Call decrementReference() + } +} +``` + +## Configuration + +```yaml +contentgrid: + content: + lifecycle: + enabled: true + deletion: + enabled: true + grace-period: P7D # ISO-8601 Duration + batch-size: 100 +``` + +## Metrics + +| Metric | Description | +|--------|-------------| +| `content.deletion.success` | Successfully deleted content | +| `content.deletion.failure` | Failed to delete content | +| `content.deletion.drift` | Count drift detected (marked but still referenced) | + +## Edge Cases + +| Scenario | Behavior | +|----------|----------| +| Content uploaded then immediately deleted | Count → 1 → 0, marked for deletion after grace period | +| Transaction rollback after upload | No increment (never tracked) | +| Transaction rollback after delete | Decrement in afterCommit, so no decrement | +| Content shared across entities | Each entity increments count | +| Content resurrected (deleted, then re-uploaded with same ID) | Increment clears `marked_for_deletion_at` | +| Count drift detected | Clear deletion marker, log warning, increment drift counter | +| Content without entry in `_content_references` | Safety check finds reference, no deletion | + +## Migration + +Before enabling the deletion job, populate `_content_references` from existing data: + +```sql +INSERT INTO _content_references (content_id, reference_count, first_referenced_at) +SELECT content_id, 1, NOW() +FROM ( + SELECT content_id FROM invoices WHERE content_id IS NOT NULL + UNION ALL + SELECT attachment_id FROM documents WHERE attachment_id IS NOT NULL + -- ... all content columns +) AS all_content_ids +GROUP BY content_id; +``` + +## Future Considerations + +1. **Content sharing API**: Currently content sharing is database-only. API support could allow explicit content reuse. + +2. **Bulk operations**: Batch deletion for efficiency with large content volumes. + +3. **Soft delete**: Move to trash location before final deletion for additional safety. + +4. **Deduplication**: If same content uploaded multiple times, could share content_id to save storage. + +## Files Changed + +- `settings.gradle` - Added new module +- `contentgrid-appserver-content-lifecycle/` - New module +- `contentgrid-appserver-domain/build.gradle` - Added dependency +- `contentgrid-appserver-domain/.../DatamodelApiImpl.java` - Integration +- `contentgrid-appserver-domain/.../ContentUploadAttributeMapper.java` - Integration +- `contentgrid-appserver-domain/.../ContentAttributeModificationValidator.java` - Integration +- `contentgrid-appserver-autoconfigure/.../AutoConfiguration.imports` - Registered auto-config diff --git a/contentgrid-appserver-autoconfigure/build.gradle b/contentgrid-appserver-autoconfigure/build.gradle index 99c015bef..fb6ef4d39 100644 --- a/contentgrid-appserver-autoconfigure/build.gradle +++ b/contentgrid-appserver-autoconfigure/build.gradle @@ -14,6 +14,7 @@ dependencies { compileOnly project(':contentgrid-appserver-contentstore-impl-encryption') compileOnly project(':contentgrid-appserver-contentstore-impl-fs') compileOnly project(':contentgrid-appserver-contentstore-impl-s3') + compileOnly project(':contentgrid-appserver-content-lifecycle') compileOnly project(':contentgrid-appserver-domain') compileOnly project(':contentgrid-appserver-events') compileOnly project(':contentgrid-appserver-json-schema') @@ -51,6 +52,7 @@ dependencies { testImplementation project(':contentgrid-appserver-contentstore-impl-encryption') testImplementation project(':contentgrid-appserver-contentstore-impl-fs') testImplementation project(':contentgrid-appserver-contentstore-impl-s3') + testImplementation project(':contentgrid-appserver-content-lifecycle') testImplementation project(':contentgrid-appserver-domain') testImplementation project(':contentgrid-appserver-events') testImplementation project(':contentgrid-appserver-json-schema') diff --git a/contentgrid-appserver-autoconfigure/src/main/java/com/contentgrid/appserver/autoconfigure/domain/ContentGridDomainAutoConfiguration.java b/contentgrid-appserver-autoconfigure/src/main/java/com/contentgrid/appserver/autoconfigure/domain/ContentGridDomainAutoConfiguration.java index d1a965dd8..047791e48 100644 --- a/contentgrid-appserver-autoconfigure/src/main/java/com/contentgrid/appserver/autoconfigure/domain/ContentGridDomainAutoConfiguration.java +++ b/contentgrid-appserver-autoconfigure/src/main/java/com/contentgrid/appserver/autoconfigure/domain/ContentGridDomainAutoConfiguration.java @@ -2,6 +2,7 @@ import com.contentgrid.appserver.application.model.Application; import com.contentgrid.appserver.autoconfigure.events.ContentGridEventsAutoConfiguration; +import com.contentgrid.appserver.content.lifecycle.ContentReferenceTracker; import com.contentgrid.appserver.contentstore.api.ContentStore; import com.contentgrid.appserver.domain.ContentApi; import com.contentgrid.appserver.domain.ContentApiImpl; @@ -13,6 +14,7 @@ import com.contentgrid.appserver.domain.paging.cursor.SimplePageBasedCursorCodec; import com.contentgrid.appserver.query.engine.api.QueryEngine; import java.time.Clock; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.autoconfigure.AutoConfiguration; import org.springframework.boot.autoconfigure.condition.ConditionalOnClass; import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; @@ -45,8 +47,8 @@ public void dispatchDelete(Application application, EntityInstance instance) {} @Bean DatamodelApiImpl datamodelApi(QueryEngine queryEngine, ContentStore contentStore, DomainEventDispatcher dispatcher, - CursorCodec cursorCodec, Clock clock) { - return new DatamodelApiImpl(queryEngine, contentStore, dispatcher, cursorCodec, clock); + CursorCodec cursorCodec, Clock clock, @Autowired(required = false) ContentReferenceTracker contentReferenceTracker) { + return new DatamodelApiImpl(queryEngine, contentStore, dispatcher, cursorCodec, clock, contentReferenceTracker); } @Bean diff --git a/contentgrid-appserver-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports b/contentgrid-appserver-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports index 8634e8320..250af6ee9 100644 --- a/contentgrid-appserver-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports +++ b/contentgrid-appserver-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports @@ -11,3 +11,4 @@ com.contentgrid.appserver.autoconfigure.rest.ContentGridRestFormatterAutoConfigu com.contentgrid.appserver.autoconfigure.actuator.ContentgridActuatorAutoConfiguration com.contentgrid.appserver.autoconfigure.security.DefaultSecurityAutoConfiguration com.contentgrid.appserver.autoconfigure.webjars.WebjarsRestAutoConfiguration +com.contentgrid.appserver.content.lifecycle.autoconfigure.ContentLifecycleAutoConfiguration diff --git a/contentgrid-appserver-content-lifecycle/build.gradle b/contentgrid-appserver-content-lifecycle/build.gradle new file mode 100644 index 000000000..771a1de92 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/build.gradle @@ -0,0 +1,24 @@ +plugins { + id 'java-library' + id 'maven-publish' + id 'io.freefair.lombok' +} + +dependencies { + api project(':contentgrid-appserver-contentstore-api') + api project(':contentgrid-appserver-application-model') + + implementation 'org.jooq:jooq' + implementation 'org.slf4j:slf4j-api' + implementation 'org.springframework:spring-tx' + implementation 'org.springframework.boot:spring-boot' + implementation 'org.springframework.boot:spring-boot-autoconfigure' + implementation 'io.micrometer:micrometer-core' + + testImplementation 'org.mockito:mockito-junit-jupiter' + testImplementation 'org.assertj:assertj-core' +} + +test { + useJUnitPlatform() +} diff --git a/contentgrid-appserver-content-lifecycle/gradle.properties b/contentgrid-appserver-content-lifecycle/gradle.properties new file mode 100644 index 000000000..a384c7eb3 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/gradle.properties @@ -0,0 +1 @@ +description=Content lifecycle management for ContentGrid Appserver \ No newline at end of file diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentDeletionJob.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentDeletionJob.java new file mode 100644 index 000000000..33c109652 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentDeletionJob.java @@ -0,0 +1,118 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.jooq.impl.DSL.currentTimestamp; +import static org.jooq.impl.DSL.field; +import static org.jooq.impl.DSL.name; +import static org.jooq.impl.DSL.table; + +import com.contentgrid.appserver.contentstore.api.ContentReference; +import com.contentgrid.appserver.contentstore.api.ContentStore; +import com.contentgrid.appserver.contentstore.api.UnwritableContentException; +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.MeterRegistry; +import java.util.List; +import lombok.extern.slf4j.Slf4j; +import org.jooq.DSLContext; +import org.jooq.Record1; +import org.jooq.Select; + +@Slf4j +public class ContentDeletionJob { + private static final String TABLE_NAME = "_content_references"; + private static final org.jooq.Table CONTENT_REFERENCES = table(name(TABLE_NAME)); + private static final org.jooq.Field CONTENT_ID = field(name(TABLE_NAME, "content_id"), org.jooq.impl.SQLDataType.VARCHAR); + private static final org.jooq.Field MARKED_FOR_DELETION_AT = field(name(TABLE_NAME, "marked_for_deletion_at"), org.jooq.impl.SQLDataType.TIMESTAMP); + + private final DSLContext dslContext; + private final ContentStore contentStore; + private final ContentReferenceVerificationQuery verificationQuery; + private final ContentLifecycleProperties properties; + private final Counter successCounter; + private final Counter failureCounter; + private final Counter driftCounter; + + public ContentDeletionJob( + DSLContext dslContext, + ContentStore contentStore, + ContentReferenceVerificationQuery verificationQuery, + ContentLifecycleProperties properties, + MeterRegistry meterRegistry + ) { + this.dslContext = dslContext; + this.contentStore = contentStore; + this.verificationQuery = verificationQuery; + this.properties = properties; + this.successCounter = Counter.builder("content.deletion") + .tag("result", "success") + .register(meterRegistry); + this.failureCounter = Counter.builder("content.deletion") + .tag("result", "failure") + .register(meterRegistry); + this.driftCounter = Counter.builder("content.deletion.drift") + .register(meterRegistry); + } + + public void run() { + log.info("Starting content deletion job"); + int processed = 0; + int deleted = 0; + int skipped = 0; + + List candidates = findDeletionCandidates(); + + for (String contentId : candidates) { + processed++; + + if (verificationQuery.isStillReferenced(contentId)) { + log.warn("Count drift detected: content_id {} marked for deletion but still referenced", contentId); + driftCounter.increment(); + clearDeletionMarker(contentId); + skipped++; + continue; + } + + if (deleteContent(contentId)) { + deleted++; + } + } + + log.info("Content deletion job completed: processed={}, deleted={}, skipped={}", processed, deleted, skipped); + } + + private List findDeletionCandidates() { + Select> query = dslContext + .select(CONTENT_ID) + .from(CONTENT_REFERENCES) + .where(MARKED_FOR_DELETION_AT.isNotNull()) + .and(MARKED_FOR_DELETION_AT.le(currentTimestamp())) + .orderBy(MARKED_FOR_DELETION_AT) + .limit(properties.getDeletion().getBatchSize()); + + return query.fetch(CONTENT_ID); + } + + private void clearDeletionMarker(String contentId) { + dslContext.update(CONTENT_REFERENCES) + .setNull(MARKED_FOR_DELETION_AT) + .where(CONTENT_ID.eq(contentId)) + .execute(); + } + + private boolean deleteContent(String contentId) { + try { + contentStore.remove(ContentReference.of(contentId)); + + dslContext.deleteFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .execute(); + + successCounter.increment(); + log.debug("Deleted content_id {}", contentId); + return true; + } catch (UnwritableContentException e) { + log.error("Failed to delete content_id {}", contentId, e); + failureCounter.increment(); + return false; + } + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentLifecycleProperties.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentLifecycleProperties.java new file mode 100644 index 000000000..6878ca546 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentLifecycleProperties.java @@ -0,0 +1,20 @@ +package com.contentgrid.appserver.content.lifecycle; + +import java.time.Duration; +import lombok.Data; +import org.springframework.boot.context.properties.ConfigurationProperties; + +@Data +@ConfigurationProperties(prefix = "contentgrid.content.lifecycle") +public class ContentLifecycleProperties { + private boolean enabled = true; + + private Deletion deletion = new Deletion(); + + @Data + public static class Deletion { + private boolean enabled = true; + private Duration gracePeriod = Duration.ofDays(7); + private int batchSize = 100; + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTracker.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTracker.java new file mode 100644 index 000000000..47ba4c5c7 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTracker.java @@ -0,0 +1,8 @@ +package com.contentgrid.appserver.content.lifecycle; + +import com.contentgrid.appserver.contentstore.api.ContentReference; + +public interface ContentReferenceTracker { + void incrementReference(ContentReference ref); + void decrementReference(ContentReference ref); +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceVerificationQuery.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceVerificationQuery.java new file mode 100644 index 000000000..c8e283518 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceVerificationQuery.java @@ -0,0 +1,45 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.jooq.impl.DSL.field; +import static org.jooq.impl.DSL.name; +import static org.jooq.impl.DSL.selectOne; +import static org.jooq.impl.DSL.table; + +import com.contentgrid.appserver.application.model.Application; +import com.contentgrid.appserver.application.model.Entity; +import com.contentgrid.appserver.application.model.attributes.ContentAttribute; +import java.util.function.Supplier; +import lombok.RequiredArgsConstructor; +import org.jooq.DSLContext; +import org.jooq.Select; + +@RequiredArgsConstructor +public class ContentReferenceVerificationQuery { + private final DSLContext dslContext; + private final Supplier applicationSupplier; + + public boolean isStillReferenced(String contentId) { + Application app = applicationSupplier.get(); + + for (Entity entity : app.getEntities()) { + for (ContentAttribute contentAttr : entity.getContentAttributes()) { + if (isReferencedIn(entity, contentAttr, contentId)) { + return true; + } + } + } + + return false; + } + + private boolean isReferencedIn(Entity entity, ContentAttribute contentAttr, String contentId) { + String tableName = entity.getTable().getValue(); + String contentIdColumn = contentAttr.getId().getColumn().getValue(); + + return dslContext.fetchExists( + dslContext.selectOne() + .from(table(name(tableName))) + .where(field(name(contentIdColumn)).eq(contentId)) + ); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/DeferredContentReferenceTracker.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/DeferredContentReferenceTracker.java new file mode 100644 index 000000000..1d24569f0 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/DeferredContentReferenceTracker.java @@ -0,0 +1,30 @@ +package com.contentgrid.appserver.content.lifecycle; + +import com.contentgrid.appserver.contentstore.api.ContentReference; +import lombok.RequiredArgsConstructor; +import org.springframework.transaction.support.TransactionSynchronization; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +@RequiredArgsConstructor +public class DeferredContentReferenceTracker implements ContentReferenceTracker { + private final ContentReferenceTracker delegate; + + @Override + public void incrementReference(ContentReference ref) { + delegate.incrementReference(ref); + } + + @Override + public void decrementReference(ContentReference ref) { + if (TransactionSynchronizationManager.isSynchronizationActive()) { + TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() { + @Override + public void afterCommit() { + delegate.decrementReference(ref); + } + }); + } else { + delegate.decrementReference(ref); + } + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/DereferencedContentCollector.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/DereferencedContentCollector.java new file mode 100644 index 000000000..ed5d51ebf --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/DereferencedContentCollector.java @@ -0,0 +1,27 @@ +package com.contentgrid.appserver.content.lifecycle; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +public class DereferencedContentCollector { + private final List contentIds = new ArrayList<>(); + + public void add(String contentId) { + if (contentId != null && !contentId.isEmpty()) { + contentIds.add(contentId); + } + } + + public List getContentIds() { + return Collections.unmodifiableList(contentIds); + } + + public boolean isEmpty() { + return contentIds.isEmpty(); + } + + public void clear() { + contentIds.clear(); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/JooqContentReferenceTracker.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/JooqContentReferenceTracker.java new file mode 100644 index 000000000..258bce550 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/JooqContentReferenceTracker.java @@ -0,0 +1,76 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.jooq.impl.DSL.currentTimestamp; +import static org.jooq.impl.DSL.field; +import static org.jooq.impl.DSL.name; +import static org.jooq.impl.DSL.table; +import static org.jooq.impl.DSL.val; + +import com.contentgrid.appserver.contentstore.api.ContentReference; +import java.time.Duration; +import lombok.RequiredArgsConstructor; +import org.jooq.DSLContext; +import org.jooq.Field; +import org.jooq.Record; +import org.jooq.impl.SQLDataType; + +@RequiredArgsConstructor +public class JooqContentReferenceTracker implements ContentReferenceTracker { + private final DSLContext dslContext; + private final Duration gracePeriod; + + private static final String TABLE_NAME = "_content_references"; + private static final org.jooq.Table CONTENT_REFERENCES = table(name(TABLE_NAME)); + private static final Field CONTENT_ID = field(name(TABLE_NAME, "content_id"), SQLDataType.VARCHAR); + private static final Field REFERENCE_COUNT = field(name(TABLE_NAME, "reference_count"), SQLDataType.INTEGER); + private static final Field FIRST_REFERENCED_AT = field(name(TABLE_NAME, "first_referenced_at"), SQLDataType.TIMESTAMP); + private static final Field LAST_DEREFERENCED_AT = field(name(TABLE_NAME, "last_dereferenced_at"), SQLDataType.TIMESTAMP); + private static final Field MARKED_FOR_DELETION_AT = field(name(TABLE_NAME, "marked_for_deletion_at"), SQLDataType.TIMESTAMP); + + public void setupTables() { + dslContext.createTableIfNotExists(TABLE_NAME) + .column(CONTENT_ID) + .column(REFERENCE_COUNT) + .column(FIRST_REFERENCED_AT) + .column(LAST_DEREFERENCED_AT) + .column(MARKED_FOR_DELETION_AT) + .primaryKey(CONTENT_ID) + .execute(); + } + + public void dropTables() { + dslContext.dropTableIfExists(TABLE_NAME).execute(); + } + + @Override + public void incrementReference(ContentReference ref) { + dslContext.insertInto(CONTENT_REFERENCES) + .columns(CONTENT_ID, REFERENCE_COUNT, FIRST_REFERENCED_AT) + .values(val(ref.getValue()), val(1), currentTimestamp()) + .onConflict(CONTENT_ID) + .doUpdate() + .set(REFERENCE_COUNT, REFERENCE_COUNT.plus(1)) + .setNull(MARKED_FOR_DELETION_AT) + .execute(); + } + + @Override + public void decrementReference(ContentReference ref) { + dslContext.transaction(configuration -> { + var txDsl = configuration.dsl(); + + txDsl.update(CONTENT_REFERENCES) + .set(REFERENCE_COUNT, REFERENCE_COUNT.minus(1)) + .set(LAST_DEREFERENCED_AT, currentTimestamp()) + .where(CONTENT_ID.eq(ref.getValue())) + .and(REFERENCE_COUNT.gt(0)) + .execute(); + + txDsl.update(CONTENT_REFERENCES) + .set(MARKED_FOR_DELETION_AT, currentTimestamp().plus((int) gracePeriod.toSeconds())) + .where(CONTENT_ID.eq(ref.getValue())) + .and(REFERENCE_COUNT.le(0)) + .execute(); + }); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/autoconfigure/ContentLifecycleAutoConfiguration.java b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/autoconfigure/ContentLifecycleAutoConfiguration.java new file mode 100644 index 000000000..608552940 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/autoconfigure/ContentLifecycleAutoConfiguration.java @@ -0,0 +1,63 @@ +package com.contentgrid.appserver.content.lifecycle.autoconfigure; + +import com.contentgrid.appserver.application.model.Application; +import com.contentgrid.appserver.content.lifecycle.ContentDeletionJob; +import com.contentgrid.appserver.content.lifecycle.ContentLifecycleProperties; +import com.contentgrid.appserver.content.lifecycle.ContentReferenceTracker; +import com.contentgrid.appserver.content.lifecycle.ContentReferenceVerificationQuery; +import com.contentgrid.appserver.content.lifecycle.DeferredContentReferenceTracker; +import com.contentgrid.appserver.content.lifecycle.JooqContentReferenceTracker; +import com.contentgrid.appserver.contentstore.api.ContentStore; +import io.micrometer.core.instrument.MeterRegistry; +import java.util.function.Supplier; +import org.jooq.DSLContext; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.autoconfigure.AutoConfiguration; +import org.springframework.boot.autoconfigure.condition.ConditionalOnBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean; +import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.context.annotation.Bean; + +@AutoConfiguration +@ConditionalOnBean({DSLContext.class, ContentStore.class}) +@ConditionalOnProperty(prefix = "contentgrid.content.lifecycle", name = "enabled", havingValue = "true", matchIfMissing = true) +@EnableConfigurationProperties(ContentLifecycleProperties.class) +public class ContentLifecycleAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public JooqContentReferenceTracker jooqContentReferenceTracker(DSLContext dslContext, ContentLifecycleProperties properties) { + var tracker = new JooqContentReferenceTracker(dslContext, properties.getDeletion().getGracePeriod()); + tracker.setupTables(); + return tracker; + } + + @Bean + @ConditionalOnMissingBean(ContentReferenceTracker.class) + public DeferredContentReferenceTracker contentReferenceTracker(JooqContentReferenceTracker delegate) { + return new DeferredContentReferenceTracker(delegate); + } + + @Bean + @ConditionalOnMissingBean + public ContentReferenceVerificationQuery contentReferenceVerificationQuery( + DSLContext dslContext, + @Autowired Supplier applicationSupplier + ) { + return new ContentReferenceVerificationQuery(dslContext, applicationSupplier); + } + + @Bean + @ConditionalOnMissingBean + @ConditionalOnProperty(prefix = "contentgrid.content.lifecycle.deletion", name = "enabled", havingValue = "true", matchIfMissing = true) + public ContentDeletionJob contentDeletionJob( + DSLContext dslContext, + ContentStore contentStore, + ContentReferenceVerificationQuery verificationQuery, + ContentLifecycleProperties properties, + MeterRegistry meterRegistry + ) { + return new ContentDeletionJob(dslContext, contentStore, verificationQuery, properties, meterRegistry); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentLifecyclePropertiesTest.java b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentLifecyclePropertiesTest.java new file mode 100644 index 000000000..4f5a4174d --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentLifecyclePropertiesTest.java @@ -0,0 +1,19 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Test; + +class ContentLifecyclePropertiesTest { + + @Test + void defaultValues_areCorrect() { + var properties = new ContentLifecycleProperties(); + + assertThat(properties.isEnabled()).isTrue(); + assertThat(properties.getDeletion()).isNotNull(); + assertThat(properties.getDeletion().isEnabled()).isTrue(); + assertThat(properties.getDeletion().getGracePeriod().toDays()).isEqualTo(7); + assertThat(properties.getDeletion().getBatchSize()).isEqualTo(100); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTrackerBehaviorTest.java b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTrackerBehaviorTest.java new file mode 100644 index 000000000..dfc3e70b0 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTrackerBehaviorTest.java @@ -0,0 +1,148 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.contentgrid.appserver.contentstore.api.ContentReference; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.atomic.AtomicInteger; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +class InMemoryContentReferenceTracker implements ContentReferenceTracker { + private final AtomicInteger count = new AtomicInteger(0); + private final List decrementedRefs = new ArrayList<>(); + + @Override + public void incrementReference(ContentReference ref) { + count.incrementAndGet(); + } + + @Override + public void decrementReference(ContentReference ref) { + count.decrementAndGet(); + decrementedRefs.add(ref.getValue()); + } + + public int getCount() { + return count.get(); + } + + public List getDecrementedRefs() { + return List.copyOf(decrementedRefs); + } +} + +class InMemoryDeferredContentReferenceTracker implements ContentReferenceTracker { + private final InMemoryContentReferenceTracker delegate; + private boolean transactionActive = false; + private final List pendingDecrements = new java.util.ArrayList<>(); + + public InMemoryDeferredContentReferenceTracker(InMemoryContentReferenceTracker delegate) { + this.delegate = delegate; + } + + @Override + public void incrementReference(ContentReference ref) { + delegate.incrementReference(ref); + } + + @Override + public void decrementReference(ContentReference ref) { + if (transactionActive) { + pendingDecrements.add(ref.getValue()); + } else { + delegate.decrementReference(ref); + } + } + + public void setTransactionActive(boolean active) { + this.transactionActive = active; + } + + public InMemoryContentReferenceTracker getDelegate() { + return delegate; + } + + public void flushPendingDecrements() { + for (var ref : pendingDecrements) { + delegate.decrementReference(ContentReference.of(ref)); + } + pendingDecrements.clear(); + } +} + +class ContentReferenceTrackerBehaviorTest { + + @Nested + class InMemoryTracker { + @Test + void incrementIncreasesCount() { + var tracker = new InMemoryContentReferenceTracker(); + + tracker.incrementReference(ContentReference.of("content-1")); + + assertThat(tracker.getCount()).isEqualTo(1); + } + + @Test + void decrementDecreasesCount() { + var tracker = new InMemoryContentReferenceTracker(); + tracker.incrementReference(ContentReference.of("content-1")); + + tracker.decrementReference(ContentReference.of("content-1")); + + assertThat(tracker.getCount()).isEqualTo(0); + } + + @Test + void multipleIncrementsAndDecrements() { + var tracker = new InMemoryContentReferenceTracker(); + + tracker.incrementReference(ContentReference.of("content-1")); + tracker.incrementReference(ContentReference.of("content-2")); + tracker.incrementReference(ContentReference.of("content-3")); + + tracker.decrementReference(ContentReference.of("content-2")); + + assertThat(tracker.getCount()).isEqualTo(2); + } + + @Test + void decrementTracksReferences() { + var tracker = new InMemoryContentReferenceTracker(); + + tracker.incrementReference(ContentReference.of("content-1")); + tracker.decrementReference(ContentReference.of("content-1")); + + assertThat(tracker.getDecrementedRefs()).containsExactly("content-1"); + } + } + +@Nested + class DeferredTracker { + @Test + void delegatesWhenNoTransaction() { + var inner = new InMemoryContentReferenceTracker(); + var deferred = new InMemoryDeferredContentReferenceTracker(inner); + + deferred.incrementReference(ContentReference.of("content-1")); + deferred.decrementReference(ContentReference.of("content-1")); + + assertThat(inner.getCount()).isEqualTo(0); + } + + @Test + void doesNotDecrementWhenTransactionActive() { + var inner = new InMemoryContentReferenceTracker(); + var deferred = new InMemoryDeferredContentReferenceTracker(inner); + + deferred.incrementReference(ContentReference.of("content-1")); + deferred.setTransactionActive(true); + deferred.decrementReference(ContentReference.of("content-1")); + + assertThat(inner.getCount()).isEqualTo(1); + } + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceVerificationQueryTest.java b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceVerificationQueryTest.java new file mode 100644 index 000000000..c664c3896 --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceVerificationQueryTest.java @@ -0,0 +1,55 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +import com.contentgrid.appserver.application.model.Application; +import com.contentgrid.appserver.application.model.Entity; +import com.contentgrid.appserver.application.model.attributes.ContentAttribute; +import com.contentgrid.appserver.application.model.values.AttributeName; +import com.contentgrid.appserver.application.model.values.EntityName; +import java.util.Collections; +import java.util.List; +import java.util.function.Supplier; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.jooq.DSLContext; + +class ContentReferenceVerificationQueryTest { + + private DSLContext dslContext; + private Supplier applicationSupplier; + private ContentReferenceVerificationQuery query; + + @BeforeEach + void setUp() { + dslContext = mock(DSLContext.class); + applicationSupplier = mock(Supplier.class); + query = new ContentReferenceVerificationQuery(dslContext, applicationSupplier); + } + + @Test + void isStillReferenced_returnsFalse_whenNoEntities() { + Application app = mock(Application.class); + when(app.getEntities()).thenReturn(Collections.emptyList()); + when(applicationSupplier.get()).thenReturn(app); + + boolean result = query.isStillReferenced("content-123"); + + assertThat(result).isFalse(); + } + + @Test + void isStillReferenced_returnsFalse_whenNoContentAttributes() { + Application app = mock(Application.class); + Entity entity = mock(Entity.class); + when(app.getEntities()).thenReturn(List.of(entity)); + when(entity.getContentAttributes()).thenReturn(Collections.emptyList()); + when(applicationSupplier.get()).thenReturn(app); + + boolean result = query.isStillReferenced("content-123"); + + assertThat(result).isFalse(); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/DeferredContentReferenceTrackerTransactionTest.java b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/DeferredContentReferenceTrackerTransactionTest.java new file mode 100644 index 000000000..885bf794c --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/DeferredContentReferenceTrackerTransactionTest.java @@ -0,0 +1,69 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; + +import com.contentgrid.appserver.contentstore.api.ContentReference; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.transaction.support.TransactionSynchronizationManager; + +class DeferredContentReferenceTrackerTransactionTest { + + private ContentReferenceTracker delegate; + private DeferredContentReferenceTracker tracker; + + @BeforeEach + void setUp() { + TransactionSynchronizationManager.initSynchronization(); + delegate = new ContentReferenceTracker() { + @Override + public void incrementReference(ContentReference ref) {} + @Override + public void decrementReference(ContentReference ref) {} + }; + tracker = new DeferredContentReferenceTracker(delegate); + } + + @AfterEach + void tearDown() { + TransactionSynchronizationManager.clear(); + } + + @Test + void decrementReference_registersSynchronization_whenTransactionActive() { + assertThat(TransactionSynchronizationManager.isSynchronizationActive()).isTrue(); + + tracker.decrementReference(ContentReference.of("test-id")); + + assertThat(TransactionSynchronizationManager.getSynchronizations()).hasSize(1); + } + + @Test + void decrementReference_doesNotCallDelegateImmediately() { + var delegateSpy = new ContentReferenceTracker() { + private boolean called = false; + @Override + public void incrementReference(ContentReference ref) {} + @Override + public void decrementReference(ContentReference ref) { + called = true; + } + public boolean wasCalled() { return called; } + }; + var spyTracker = new DeferredContentReferenceTracker(delegateSpy); + + spyTracker.decrementReference(ContentReference.of("test-id")); + + assertThat(delegateSpy.wasCalled()).isFalse(); + } + + @Test + void incrementReference_doesNotRegisterSynchronization() { + tracker.incrementReference(ContentReference.of("test-id")); + + assertThat(TransactionSynchronizationManager.getSynchronizations()).isEmpty(); + } +} diff --git a/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/DereferencedContentCollectorTest.java b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/DereferencedContentCollectorTest.java new file mode 100644 index 000000000..ca5cf8c2a --- /dev/null +++ b/contentgrid-appserver-content-lifecycle/src/test/java/com/contentgrid/appserver/content/lifecycle/DereferencedContentCollectorTest.java @@ -0,0 +1,101 @@ +package com.contentgrid.appserver.content.lifecycle; + +import static org.assertj.core.api.Assertions.assertThat; + +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +class DereferencedContentCollectorTest { + + @Nested + class AddContent { + @Test + void addsNonNullContentId() { + var collector = new DereferencedContentCollector(); + + collector.add("content-123"); + + assertThat(collector.getContentIds()).containsExactly("content-123"); + } + + @Test + void addsMultipleContentIds() { + var collector = new DereferencedContentCollector(); + + collector.add("content-1"); + collector.add("content-2"); + collector.add("content-3"); + + assertThat(collector.getContentIds()).containsExactly("content-1", "content-2", "content-3"); + } + + @Test + void ignoresNullContentId() { + var collector = new DereferencedContentCollector(); + + collector.add(null); + + assertThat(collector.getContentIds()).isEmpty(); + } + + @Test + void ignoresEmptyContentId() { + var collector = new DereferencedContentCollector(); + + collector.add(""); + + assertThat(collector.getContentIds()).isEmpty(); + } + } + + @Nested + class GetContentIds { + @Test + void returnsUnmodifiableList() { + var collector = new DereferencedContentCollector(); + collector.add("content-1"); + + var list = collector.getContentIds(); + + var thrown = false; + try { + list.add("should-fail"); + } catch (UnsupportedOperationException e) { + thrown = true; + } + assertThat(thrown).isTrue(); + } + } + + @Nested + class Clear { + @Test + void clearsAllContentIds() { + var collector = new DereferencedContentCollector(); + collector.add("content-1"); + collector.add("content-2"); + + collector.clear(); + + assertThat(collector.getContentIds()).isEmpty(); + } + } + + @Nested + class IsEmpty { + @Test + void returnsTrueWhenEmpty() { + var collector = new DereferencedContentCollector(); + + assertThat(collector.isEmpty()).isTrue(); + } + + @Test + void returnsFalseWhenNotEmpty() { + var collector = new DereferencedContentCollector(); + collector.add("content-1"); + + assertThat(collector.isEmpty()).isFalse(); + } + } +} diff --git a/contentgrid-appserver-domain/build.gradle b/contentgrid-appserver-domain/build.gradle index 47ca17e69..dc3054e73 100644 --- a/contentgrid-appserver-domain/build.gradle +++ b/contentgrid-appserver-domain/build.gradle @@ -12,6 +12,7 @@ dependencies { implementation 'org.slf4j:slf4j-api' implementation project(':contentgrid-appserver-contentstore-api') + implementation project(':contentgrid-appserver-content-lifecycle') implementation 'org.springframework:spring-core' implementation 'com.contentgrid.hateoas:contentgrid-pagination-offset:0.0.4' diff --git a/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/DatamodelApiImpl.java b/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/DatamodelApiImpl.java index 21553b8a2..b4d2e0c8b 100644 --- a/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/DatamodelApiImpl.java +++ b/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/DatamodelApiImpl.java @@ -4,6 +4,7 @@ import com.contentgrid.appserver.application.model.Entity; import com.contentgrid.appserver.application.model.attributes.Attribute; import com.contentgrid.appserver.application.model.values.EntityName; +import com.contentgrid.appserver.content.lifecycle.ContentReferenceTracker; import com.contentgrid.appserver.contentstore.api.ContentStore; import com.contentgrid.appserver.domain.authorization.AuthorizationContext; import com.contentgrid.appserver.domain.data.DataEntry; @@ -49,9 +50,11 @@ import com.contentgrid.appserver.query.engine.api.UnlinkEventConsumer; import com.contentgrid.appserver.query.engine.api.UpdateEventConsumer; import com.contentgrid.appserver.query.engine.api.data.AttributeData; +import com.contentgrid.appserver.query.engine.api.data.CompositeAttributeData; import com.contentgrid.appserver.query.engine.api.data.EntityCreateData; import com.contentgrid.appserver.query.engine.api.data.EntityData; import com.contentgrid.appserver.query.engine.api.data.OffsetData; +import com.contentgrid.appserver.query.engine.api.data.SimpleAttributeData; import com.contentgrid.appserver.query.engine.api.data.SortData; import com.contentgrid.appserver.query.engine.api.data.SortData.FieldSort; import com.contentgrid.appserver.query.engine.api.exception.EntityIdNotFoundException; @@ -81,6 +84,7 @@ public class DatamodelApiImpl implements DatamodelApi { private final DomainEventDispatcher domainEventDispatcher; private final CursorCodec cursorCodec; private final Clock clock; + private final ContentReferenceTracker contentReferenceTracker; private RequestInputDataMapper createInputDataMapper( @NonNull Application application, @@ -109,7 +113,7 @@ private RequestInputDataMapper createInputDataMapper( )) .andThen(new OptionalFlatMapAdaptingMapper<>( AttributeAndRelationMapper.from( - new ContentUploadAttributeMapper(contentStore), + new ContentUploadAttributeMapper(contentStore, contentReferenceTracker), (rel, value) -> Optional.of(value) ) )) @@ -273,15 +277,14 @@ public InternalEntityInstance update(@NonNull Application application, @NonNull AuthorizationContext authorizationContext ) throws QueryEngineException, InvalidPropertyDataException { + var contentModificationValidator = new ContentAttributeModificationValidator(existingEntity); var inputMapper = createInputDataMapper( application, existingEntity.getIdentity().getEntityName(), - // All missing fields are regarded as null FilterDataEntryMapper.missingAsNull() - // Validate that content attribute is not partially set .andThen(new OptionalFlatMapAdaptingMapper<>( AttributeAndRelationMapper.from( - new AttributeValidationDataMapper(new ContentAttributeModificationValidator(existingEntity)), + new AttributeValidationDataMapper(contentModificationValidator), (rel, d) -> Optional.of(d) ) )), @@ -303,6 +306,12 @@ public InternalEntityInstance update(@NonNull Application application, UpdateEventConsumer onUpdate = new EventConsumerImpl(outputMapper); var updateData = queryEngine.update(application, entityData, authorizationContext.predicate(), onUpdate); + for (String contentId : contentModificationValidator.getDereferencedContentIds()) { + if (contentReferenceTracker != null) { + contentReferenceTracker.decrementReference(com.contentgrid.appserver.contentstore.api.ContentReference.of(contentId)); + } + } + return outputMapper.mapAttributes(updateData.getUpdated()); } @@ -312,15 +321,14 @@ public InternalEntityInstance updatePartial(@NonNull Application application, @NonNull RequestInputData data, @NonNull AuthorizationContext authorizationContext ) throws QueryEngineException, InvalidPropertyDataException { + var contentModificationValidator = new ContentAttributeModificationValidator(existingEntity); var inputMapper = createInputDataMapper( application, existingEntity.getIdentity().getEntityName(), - // Missing fields are omitted, so they are not updated FilterDataEntryMapper.omitMissing() - // Validate that content attribute is not partially set .andThen(new OptionalFlatMapAdaptingMapper<>( AttributeAndRelationMapper.from( - new AttributeValidationDataMapper(new ContentAttributeModificationValidator(existingEntity)), + new AttributeValidationDataMapper(contentModificationValidator), (rel, d) -> Optional.of(d) ) )), @@ -342,6 +350,12 @@ public InternalEntityInstance updatePartial(@NonNull Application application, UpdateEventConsumer onUpdate = new EventConsumerImpl(outputMapper); var updateData = queryEngine.update(application, entityData, authorizationContext.predicate(), onUpdate); + for (String contentId : contentModificationValidator.getDereferencedContentIds()) { + if (contentReferenceTracker != null) { + contentReferenceTracker.decrementReference(com.contentgrid.appserver.contentstore.api.ContentReference.of(contentId)); + } + } + return outputMapper.mapAttributes(updateData.getUpdated()); } @@ -349,12 +363,34 @@ public InternalEntityInstance updatePartial(@NonNull Application application, public InternalEntityInstance deleteEntity(@NonNull Application application, @NonNull EntityRequest entityRequest, @NonNull AuthorizationContext authorizationContext) throws EntityIdNotFoundException { var outputMapper = createOutputDataMapper(application, entityRequest.getEntityName()); + var entity = application.getRequiredEntityByName(entityRequest.getEntityName()); DeleteEventConsumer onDelete = new EventConsumerImpl(outputMapper); var deleted = queryEngine.delete(application, entityRequest, authorizationContext.predicate(), onDelete) .orElseThrow(() -> new EntityIdNotFoundException(entityRequest)); - return outputMapper.mapAttributes(deleted); + var deletedInstance = outputMapper.mapAttributes(deleted); + + if (contentReferenceTracker != null) { + for (var contentAttr : entity.getContentAttributes()) { + deletedInstance.getByAttributeName(contentAttr.getName(), CompositeAttributeData.class) + .ifPresent(compositeData -> { + compositeData.getAttributeByName(contentAttr.getId().getName()) + .filter(SimpleAttributeData.class::isInstance) + .map(attr -> (SimpleAttributeData) attr) + .map(attr -> attr.getValue()) + .filter(String.class::isInstance) + .map(String.class::cast) + .ifPresent(contentId -> { + contentReferenceTracker.decrementReference( + com.contentgrid.appserver.contentstore.api.ContentReference.of(contentId) + ); + }); + }); + } + } + + return deletedInstance; } @Override diff --git a/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/mapper/ContentUploadAttributeMapper.java b/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/mapper/ContentUploadAttributeMapper.java index 5d36c86f8..9983b0458 100644 --- a/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/mapper/ContentUploadAttributeMapper.java +++ b/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/mapper/ContentUploadAttributeMapper.java @@ -4,6 +4,7 @@ import com.contentgrid.appserver.application.model.attributes.ContentAttribute; import com.contentgrid.appserver.application.model.attributes.SimpleAttribute; import com.contentgrid.appserver.application.model.values.AttributePath; +import com.contentgrid.appserver.content.lifecycle.ContentReferenceTracker; import com.contentgrid.appserver.contentstore.api.ContentStore; import com.contentgrid.appserver.contentstore.api.UnwritableContentException; import com.contentgrid.appserver.domain.data.DataEntry; @@ -26,6 +27,11 @@ public class ContentUploadAttributeMapper extends AbstractDescendingAttributeMapper { private final ContentStore contentStore; + private final ContentReferenceTracker contentReferenceTracker; + + public ContentUploadAttributeMapper(ContentStore contentStore) { + this(contentStore, null); + } @Override protected Optional mapSimpleAttribute(AttributePath path, SimpleAttribute simpleAttribute, DataEntry inputData) { @@ -36,7 +42,6 @@ protected Optional mapSimpleAttribute(AttributePath path, SimpleAttri protected Optional mapCompositeAttribute(AttributePath path, CompositeAttribute compositeAttribute, DataEntry inputData) throws InvalidDataException { var result = super.mapCompositeAttribute(path, compositeAttribute, inputData); if(compositeAttribute instanceof ContentAttribute contentAttribute && inputData instanceof MapDataEntry mapDataEntry) { - // Remove file id and size from attributes that can be set var blockedAttributes = Set.of( contentAttribute.getId().getName().getValue(), contentAttribute.getLength().getName().getValue() @@ -68,6 +73,10 @@ protected Optional mapCompositeAttributeUnsupportedDatatype(Attribute var inputStream = new CountingInputStream(fileDataEntry.getInputStream()); var contentAccessor = contentStore.writeContent(inputStream); + if (contentReferenceTracker != null) { + contentReferenceTracker.incrementReference(contentAccessor.getReference()); + } + var builder = MapDataEntry.builder(); builder.item(contentAttribute.getId().getName().getValue(), new StringDataEntry(contentAccessor.getReference().getValue())) .item(contentAttribute.getLength().getName().getValue(), new LongDataEntry(inputStream.getSize())) diff --git a/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/validation/ContentAttributeModificationValidator.java b/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/validation/ContentAttributeModificationValidator.java index af3e7031e..a36699e69 100644 --- a/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/validation/ContentAttributeModificationValidator.java +++ b/contentgrid-appserver-domain/src/main/java/com/contentgrid/appserver/domain/data/validation/ContentAttributeModificationValidator.java @@ -8,10 +8,14 @@ import com.contentgrid.appserver.domain.data.DataEntry.MissingDataEntry; import com.contentgrid.appserver.domain.data.DataEntry.NullDataEntry; import com.contentgrid.appserver.domain.data.DataEntry.PlainDataEntry; +import com.contentgrid.appserver.domain.data.DataEntry.StringDataEntry; import com.contentgrid.appserver.domain.data.EntityInstance; import com.contentgrid.appserver.domain.data.InvalidDataException; import com.contentgrid.appserver.domain.data.validation.AttributeValidationDataMapper.Validator; +import java.util.ArrayList; +import java.util.List; import java.util.Optional; +import lombok.Getter; import lombok.RequiredArgsConstructor; import lombok.extern.slf4j.Slf4j; @@ -20,6 +24,9 @@ public class ContentAttributeModificationValidator implements Validator { private final EntityInstance entityData; + @Getter + private final List dereferencedContentIds = new ArrayList<>(); + @Override public void validate(AttributePath attributePath, Attribute attribute, DataEntry dataEntry) throws InvalidDataException { @@ -35,18 +42,19 @@ public void validate(AttributePath attributePath, Attribute attribute, DataEntry } } } else { - // When a content attribute is present, mimetype is required to be filled in var mimeType = contentAttribute.getMimetype().getName(); - // At the point that this validator runs, MissingDataEntry is already converted to null for create/update - // And has been set to missing only for partialUpdate. We don't have to require a mimetype input for partial update, - // as the mimetype will just not be set at all if ((mapDataEntry.get(mimeType.getValue()) instanceof NullDataEntry)) { throw new RequiredConstraintViolationInvalidDataException().withinProperty(mimeType); } } } else if (hasContent && dataEntry instanceof NullDataEntry) { - // TODO: mark content for deletion, it can only be deleted in ContentStore - // after database transaction has completed + var currentData = resolveData(attributePath); + if (currentData.isPresent() && currentData.get() instanceof MapDataEntry currentMap) { + var contentIdEntry = currentMap.get(contentAttribute.getId().getName().getValue()); + if (contentIdEntry instanceof StringDataEntry stringEntry) { + dereferencedContentIds.add(stringEntry.getValue()); + } + } } } diff --git a/contentgrid-appserver-domain/src/test/java/com/contentgrid/appserver/domain/DatamodelApiImplTest.java b/contentgrid-appserver-domain/src/test/java/com/contentgrid/appserver/domain/DatamodelApiImplTest.java index 30cca285b..33c6a3a90 100644 --- a/contentgrid-appserver-domain/src/test/java/com/contentgrid/appserver/domain/DatamodelApiImplTest.java +++ b/contentgrid-appserver-domain/src/test/java/com/contentgrid/appserver/domain/DatamodelApiImplTest.java @@ -142,7 +142,8 @@ void setup() { contentStore, domainEventDispatcher, codec, - clock + clock, + null ); } diff --git a/contentgrid-appserver-integration-test/build.gradle b/contentgrid-appserver-integration-test/build.gradle index 3f2bc0515..0c2335bb7 100644 --- a/contentgrid-appserver-integration-test/build.gradle +++ b/contentgrid-appserver-integration-test/build.gradle @@ -17,6 +17,7 @@ dependencies { testImplementation project(':contentgrid-appserver-domain') + testImplementation project(':contentgrid-appserver-content-lifecycle') // should not be used for most integration tests, but relevant for encryption compatibility testImplementation project(':contentgrid-appserver-contentstore-impl-encryption') testImplementation testFixtures(project(':contentgrid-appserver-rest')) @@ -32,4 +33,5 @@ dependencies { testImplementation 'org.testcontainers:junit-jupiter' testImplementation 'org.testcontainers:minio' testImplementation 'io.minio:minio' + testImplementation 'org.jooq:jooq' } diff --git a/contentgrid-appserver-integration-test/src/test/java/com/contentgrid/appserver/integration/test/contentlifecycle/ContentReferenceTrackingIntegrationTest.java b/contentgrid-appserver-integration-test/src/test/java/com/contentgrid/appserver/integration/test/contentlifecycle/ContentReferenceTrackingIntegrationTest.java new file mode 100644 index 000000000..996bbdfbd --- /dev/null +++ b/contentgrid-appserver-integration-test/src/test/java/com/contentgrid/appserver/integration/test/contentlifecycle/ContentReferenceTrackingIntegrationTest.java @@ -0,0 +1,161 @@ +package com.contentgrid.appserver.integration.test.contentlifecycle; + +import static org.assertj.core.api.Assertions.assertThat; + +import com.contentgrid.appserver.content.lifecycle.ContentReferenceTracker; +import com.contentgrid.appserver.content.lifecycle.JooqContentReferenceTracker; +import com.contentgrid.appserver.contentstore.api.ContentReference; +import com.contentgrid.appserver.contentstore.api.ContentStore; +import java.time.Duration; +import java.util.UUID; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.context.annotation.Bean; +import org.springframework.test.context.TestPropertySource; +import org.jooq.DSLContext; +import org.jooq.Record; +import org.jooq.Table; + +@SpringBootTest( + webEnvironment = WebEnvironment.NONE, + properties = { + "contentgrid.security.unauthenticated.allow = true", + "contentgrid.events.rabbitmq.enabled=false", + "contentgrid.appserver.content-store.type = ephemeral", + "contentgrid.thunx.abac.source = none", + "spring.datasource.url=jdbc:tc:postgresql:15:///", + "contentgrid.content.lifecycle.enabled=true", + "contentgrid.content.lifecycle.deletion.grace-period=PT1H" + }) +@TestPropertySource(properties = { + "contentgrid.content.lifecycle.enabled=true", + "contentgrid.content.lifecycle.deletion.grace-period=PT1H" +}) +class ContentReferenceTrackingIntegrationTest { + + @Autowired + DSLContext dslContext; + + @Autowired + ContentStore contentStore; + + private ContentReferenceTracker tracker; + + private static final Table CONTENT_REFERENCES = + org.jooq.impl.DSL.table("_content_references"); + private static final org.jooq.Field CONTENT_ID = + org.jooq.impl.DSL.field("content_id", org.jooq.impl.SQLDataType.VARCHAR); + private static final org.jooq.Field REFERENCE_COUNT = + org.jooq.impl.DSL.field("reference_count", org.jooq.impl.SQLDataType.INTEGER); + + @BeforeEach + void setUp() { + dslContext.execute(""" + CREATE TABLE IF NOT EXISTS _content_references ( + content_id VARCHAR(255) PRIMARY KEY, + reference_count INTEGER NOT NULL DEFAULT 0, + first_referenced_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(), + last_dereferenced_at TIMESTAMP WITH TIME ZONE, + marked_for_deletion_at TIMESTAMP WITH TIME ZONE + ) + """); + dslContext.deleteFrom(CONTENT_REFERENCES).execute(); + tracker = new JooqContentReferenceTracker(dslContext, Duration.ofHours(1)); + } + + @Test + void incrementReference_createsNewRecord() { + String contentId = UUID.randomUUID().toString(); + + tracker.incrementReference(ContentReference.of(contentId)); + + var record = dslContext.selectFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .fetchOne(); + + assertThat(record).isNotNull(); + assertThat(record.get(CONTENT_ID)).isEqualTo(contentId); + assertThat(record.get(REFERENCE_COUNT)).isEqualTo(1); + } + + @Test + void incrementReference_incrementsExistingCount() { + String contentId = UUID.randomUUID().toString(); + + tracker.incrementReference(ContentReference.of(contentId)); + tracker.incrementReference(ContentReference.of(contentId)); + tracker.incrementReference(ContentReference.of(contentId)); + + var record = dslContext.selectFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .fetchOne(); + + assertThat(record).isNotNull(); + assertThat(record.get(REFERENCE_COUNT)).isEqualTo(3); + } + + @Test + void decrementReference_decrementsCount() { + String contentId = UUID.randomUUID().toString(); + + tracker.incrementReference(ContentReference.of(contentId)); + tracker.incrementReference(ContentReference.of(contentId)); + tracker.decrementReference(ContentReference.of(contentId)); + + var record = dslContext.selectFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .fetchOne(); + + assertThat(record).isNotNull(); + assertThat(record.get(REFERENCE_COUNT)).isEqualTo(1); + } + + @Test + void decrementReference_setsMarkedForDeletion_whenCountReachesZero() { + String contentId = UUID.randomUUID().toString(); + + tracker.incrementReference(ContentReference.of(contentId)); + tracker.decrementReference(ContentReference.of(contentId)); + + var record = dslContext.selectFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .fetchOne(); + + assertThat(record).isNotNull(); + assertThat(record.get(REFERENCE_COUNT)).isEqualTo(0); + assertThat(record.get("marked_for_deletion_at", java.sql.Timestamp.class)).isNotNull(); + } + + @Test + void incrementReference_clearsMarkedForDeletion() { + String contentId = UUID.randomUUID().toString(); + + tracker.incrementReference(ContentReference.of(contentId)); + tracker.decrementReference(ContentReference.of(contentId)); + + var recordBefore = dslContext.selectFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .fetchOne(); + assertThat(recordBefore.get("marked_for_deletion_at", java.sql.Timestamp.class)).isNotNull(); + + tracker.incrementReference(ContentReference.of(contentId)); + + var recordAfter = dslContext.selectFrom(CONTENT_REFERENCES) + .where(CONTENT_ID.eq(contentId)) + .fetchOne(); + assertThat(recordAfter.get("marked_for_deletion_at", java.sql.Timestamp.class)).isNull(); + assertThat(recordAfter.get(REFERENCE_COUNT)).isEqualTo(1); + } + + @SpringBootApplication + static class TestApplication { + @Bean + public ContentReferenceTracker contentReferenceTracker(DSLContext dslContext) { + return new JooqContentReferenceTracker(dslContext, Duration.ofHours(1)); + } + } +} \ No newline at end of file diff --git a/settings.gradle b/settings.gradle index 0ccbb4a03..a5d6d9638 100644 --- a/settings.gradle +++ b/settings.gradle @@ -42,6 +42,7 @@ include 'contentgrid-appserver-contentstore-impl-utils' include 'contentgrid-appserver-contentstore-impl-fs' include 'contentgrid-appserver-contentstore-impl-s3' include 'contentgrid-appserver-contentstore-impl-encryption' +include 'contentgrid-appserver-content-lifecycle' include 'contentgrid-appserver-webjars' include 'contentgrid-appserver-spring-boot-starter' include 'contentgrid-appserver-integration-test'