Skip to content

Commit e4d7233

Browse files
thijslemmensthijslemmens
committed
Agentic implementation of deletion policy
1 parent f288752 commit e4d7233

16 files changed

Lines changed: 694 additions & 15 deletions

File tree

‎agents/content-deletion-design.md‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Content Deletion Design
2+
3+
## Overview
4+
5+
ContentGrid previously did not delete content from the ContentStore when:
6+
- Entity was deleted
7+
- Content property was set to null
8+
9+
This design implements a **verified reference counting** approach to safely delete orphaned content.
10+
11+
## Design Decisions
12+
13+
| Aspect | Decision | Rationale |
14+
|--------|----------|-----------|
15+
| Tracking | Reference counting table `_content_references` | Incremental tracking, supports multi-entity sharing |
16+
| Safety check | Query each entity table before deletion | Catches count drift, prevents premature deletion |
17+
| Drift handling | Alert only (log + metrics) | Manual intervention required |
18+
| Deletion trigger | K8s CronJob (one-shot) | Full app context, scales independently |
19+
| Grace period | Configurable (default 7 days) | Allows recovery, ensures backup coverage |
20+
21+
## Architecture
22+
23+
```
24+
┌─────────────────────────────────────────────────────────────────────┐
25+
│ contentgrid-appserver-app │
26+
│ ┌──────────────────────────────────────────────────────────────┐ │
27+
│ │ HTTP API Layer (Controllers) │ │
28+
│ └──────────────────────────────────────────────────────────────┘ │
29+
│ │ │
30+
│ ┌───────────────────────────┼───────────────────────────────────┐ │
31+
│ │ domain module │ │
32+
│ │ ┌─────────────────────┐ │ ┌─────────────────────────────┐ │ │
33+
│ │ │ContentUploadMapper │ │ │ContentModificationValidator│ │ │
34+
│ │ │(upload → increment) │ │ │(null → decrement) │ │ │
35+
│ │ └─────────────────────┘ │ └─────────────────────────────┘ │ │
36+
│ └───────────────────────────┼───────────────────────────────────┘ │
37+
│ │ │
38+
│ ┌───────────────────────────┼───────────────────────────────────┐ │
39+
│ │ content-lifecycle module │ │
40+
│ │ │ │ │
41+
│ │ ┌───────────────────────┴───────────────────────────────┐ │ │
42+
│ │ │ ContentReferenceTracker │ │ │
43+
│ │ │ - incrementReference() │ │ │
44+
│ │ │ - decrementReference() (afterCommit) │ │ │
45+
│ │ └───────────────────────────────────────────────────────┘ │ │
46+
│ │ │ │ │
47+
│ │ ┌───────────────────────┴───────────────────────────────┐ │ │
48+
│ │ │ _content_references table │ │ │
49+
│ │ │ content_id | ref_count | last_dereferenced | marked │ │ │
50+
│ │ └───────────────────────────────────────────────────────┘ │ │
51+
│ └───────────────────────────────────────────────────────────────┘ │
52+
└─────────────────────────────────────────────────────────────────────┘
53+
54+
K8s CronJob (configurable schedule)
55+
│
56+
┌──────────────────────────┼──────────────────────────────────────────┐
57+
│ content-lifecycle module (ContentDeletionJob) │
58+
│ ┌───────────────────────┴───────────────────────────────┐ │
59+
│ │ 1. Find candidates: marked_for_deletion_at <= now │ │
60+
│ │ 2. Safety check: query all entity tables │ │
61+
│ │ 3a. If referenced: clear mark, log drift alert │ │
62+
│ │ 3b. If orphaned: delete from ContentStore, DEKs, │ │
63+
│ │ _content_references │ │
64+
│ └───────────────────────────────────────────────────────┘ │
65+
└─────────────────────────────────────────────────────────────────────┘
66+
```
67+
68+
## Database Schema
69+
70+
```sql
71+
CREATE TABLE _content_references (
72+
content_id VARCHAR(36) PRIMARY KEY,
73+
reference_count INTEGER NOT NULL DEFAULT 1,
74+
first_referenced_at TIMESTAMP NOT NULL DEFAULT NOW(),
75+
last_dereferenced_at TIMESTAMP,
76+
marked_for_deletion_at TIMESTAMP
77+
);
78+
```
79+
80+
## Components
81+
82+
### ContentReferenceTracker (interface)
83+
Location: `contentgrid-appserver-content-lifecycle`
84+
85+
```java
86+
public interface ContentReferenceTracker {
87+
void incrementReference(ContentReference ref);
88+
void decrementReference(ContentReference ref);
89+
}
90+
```
91+
92+
### JooqContentReferenceTracker
93+
- Implements reference counting with JOOQ
94+
- On increment: INSERT or UPDATE (upsert), clear deletion marker
95+
- On decrement: Reduce count, mark for deletion if count reaches 0
96+
97+
### DeferredContentReferenceTracker
98+
- Wraps `ContentReferenceTracker` with after-commit callback
99+
- Uses `TransactionSynchronizationManager` to ensure decrements only happen after successful commit
100+
- Prevents premature deletion if transaction rolls back
101+
102+
### ContentReferenceVerificationQuery
103+
- Safety check before deletion
104+
- Queries each entity table with content attributes
105+
- Returns true if content_id is still referenced anywhere
106+
107+
### ContentDeletionJob
108+
- One-shot job for K8s CronJob
109+
- Finds content past grace period
110+
- Verifies not referenced, then deletes
111+
- Metrics: `content.deletion.success`, `content.deletion.failure`, `content.deletion.drift`
112+
113+
## Integration Points
114+
115+
### 1. Content Upload (`ContentUploadAttributeMapper`)
116+
After `contentStore.writeContent()`:
117+
```java
118+
if (contentReferenceTracker != null) {
119+
contentReferenceTracker.incrementReference(contentAccessor.getReference());
120+
}
121+
```
122+
123+
### 2. Content Dereference (`ContentAttributeModificationValidator`)
124+
When content is set to null:
125+
```java
126+
// In validate() method - track dereferenced content IDs
127+
if (hasContent && dataEntry instanceof NullDataEntry) {
128+
// Extract content_id and add to dereferencedContentIds list
129+
}
130+
```
131+
132+
Then in `DatamodelApiImpl.update()` and `updatePartial()`:
133+
```java
134+
for (String contentId : contentModificationValidator.getDereferencedContentIds()) {
135+
if (contentReferenceTracker != null) {
136+
contentReferenceTracker.decrementReference(ContentReference.of(contentId));
137+
}
138+
}
139+
```
140+
141+
### 3. Entity Delete (`DatamodelApiImpl.deleteEntity()`)
142+
After entity deletion:
143+
```java
144+
if (contentReferenceTracker != null) {
145+
for (var contentAttr : entity.getContentAttributes()) {
146+
// Extract content_id from deleted entity
147+
// Call decrementReference()
148+
}
149+
}
150+
```
151+
152+
## Configuration
153+
154+
```yaml
155+
contentgrid:
156+
content:
157+
lifecycle:
158+
enabled: true
159+
deletion:
160+
enabled: true
161+
grace-period: P7D # ISO-8601 Duration
162+
batch-size: 100
163+
```
164+
165+
## Metrics
166+
167+
| Metric | Description |
168+
|--------|-------------|
169+
| `content.deletion.success` | Successfully deleted content |
170+
| `content.deletion.failure` | Failed to delete content |
171+
| `content.deletion.drift` | Count drift detected (marked but still referenced) |
172+
173+
## Edge Cases
174+
175+
| Scenario | Behavior |
176+
|----------|----------|
177+
| Content uploaded then immediately deleted | Count → 1 → 0, marked for deletion after grace period |
178+
| Transaction rollback after upload | No increment (never tracked) |
179+
| Transaction rollback after delete | Decrement in afterCommit, so no decrement |
180+
| Content shared across entities | Each entity increments count |
181+
| Content resurrected (deleted, then re-uploaded with same ID) | Increment clears `marked_for_deletion_at` |
182+
| Count drift detected | Clear deletion marker, log warning, increment drift counter |
183+
| Content without entry in `_content_references` | Safety check finds reference, no deletion |
184+
185+
## Migration
186+
187+
Before enabling the deletion job, populate `_content_references` from existing data:
188+
189+
```sql
190+
INSERT INTO _content_references (content_id, reference_count, first_referenced_at)
191+
SELECT content_id, 1, NOW()
192+
FROM (
193+
SELECT content_id FROM invoices WHERE content_id IS NOT NULL
194+
UNION ALL
195+
SELECT attachment_id FROM documents WHERE attachment_id IS NOT NULL
196+
-- ... all content columns
197+
) AS all_content_ids
198+
GROUP BY content_id;
199+
```
200+
201+
## Future Considerations
202+
203+
1. **Content sharing API**: Currently content sharing is database-only. API support could allow explicit content reuse.
204+
205+
2. **Bulk operations**: Batch deletion for efficiency with large content volumes.
206+
207+
3. **Soft delete**: Move to trash location before final deletion for additional safety.
208+
209+
4. **Deduplication**: If same content uploaded multiple times, could share content_id to save storage.
210+
211+
## Files Changed
212+
213+
- `settings.gradle` - Added new module
214+
- `contentgrid-appserver-content-lifecycle/` - New module
215+
- `contentgrid-appserver-domain/build.gradle` - Added dependency
216+
- `contentgrid-appserver-domain/.../DatamodelApiImpl.java` - Integration
217+
- `contentgrid-appserver-domain/.../ContentUploadAttributeMapper.java` - Integration
218+
- `contentgrid-appserver-domain/.../ContentAttributeModificationValidator.java` - Integration
219+
- `contentgrid-appserver-autoconfigure/.../AutoConfiguration.imports` - Registered auto-config

‎contentgrid-appserver-autoconfigure/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,4 @@ com.contentgrid.appserver.autoconfigure.rest.ContentGridRestFormatterAutoConfigu
1111
com.contentgrid.appserver.autoconfigure.actuator.ContentgridActuatorAutoConfiguration
1212
com.contentgrid.appserver.autoconfigure.security.DefaultSecurityAutoConfiguration
1313
com.contentgrid.appserver.autoconfigure.webjars.WebjarsRestAutoConfiguration
14+
com.contentgrid.appserver.content.lifecycle.autoconfigure.ContentLifecycleAutoConfiguration

‎contentgrid-appserver-content-lifecycle/build.gradle‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
plugins {
2+
id 'java-library'
3+
id 'maven-publish'
4+
id 'io.freefair.lombok'
5+
}
6+
7+
dependencies {
8+
api project(':contentgrid-appserver-contentstore-api')
9+
api project(':contentgrid-appserver-application-model')
10+
11+
implementation 'org.jooq:jooq'
12+
implementation 'org.slf4j:slf4j-api'
13+
implementation 'org.springframework:spring-tx'
14+
implementation 'org.springframework.boot:spring-boot'
15+
implementation 'org.springframework.boot:spring-boot-autoconfigure'
16+
implementation 'io.micrometer:micrometer-core'
17+
}

‎contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentDeletionJob.java‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
package com.contentgrid.appserver.content.lifecycle;
2+
3+
import static org.jooq.impl.DSL.currentTimestamp;
4+
import static org.jooq.impl.DSL.field;
5+
import static org.jooq.impl.DSL.name;
6+
import static org.jooq.impl.DSL.table;
7+
8+
import com.contentgrid.appserver.contentstore.api.ContentReference;
9+
import com.contentgrid.appserver.contentstore.api.ContentStore;
10+
import com.contentgrid.appserver.contentstore.api.UnwritableContentException;
11+
import io.micrometer.core.instrument.Counter;
12+
import io.micrometer.core.instrument.MeterRegistry;
13+
import java.util.List;
14+
import lombok.extern.slf4j.Slf4j;
15+
import org.jooq.DSLContext;
16+
import org.jooq.Record1;
17+
import org.jooq.Select;
18+
19+
@Slf4j
20+
public class ContentDeletionJob {
21+
private static final String TABLE_NAME = "_content_references";
22+
private static final org.jooq.Table<org.jooq.Record> CONTENT_REFERENCES = table(name(TABLE_NAME));
23+
private static final org.jooq.Field<String> CONTENT_ID = field(name(TABLE_NAME, "content_id"), org.jooq.impl.SQLDataType.VARCHAR);
24+
private static final org.jooq.Field<java.sql.Timestamp> MARKED_FOR_DELETION_AT = field(name(TABLE_NAME, "marked_for_deletion_at"), org.jooq.impl.SQLDataType.TIMESTAMP);
25+
26+
private final DSLContext dslContext;
27+
private final ContentStore contentStore;
28+
private final ContentReferenceVerificationQuery verificationQuery;
29+
private final ContentLifecycleProperties properties;
30+
private final Counter successCounter;
31+
private final Counter failureCounter;
32+
private final Counter driftCounter;
33+
34+
public ContentDeletionJob(
35+
DSLContext dslContext,
36+
ContentStore contentStore,
37+
ContentReferenceVerificationQuery verificationQuery,
38+
ContentLifecycleProperties properties,
39+
MeterRegistry meterRegistry
40+
) {
41+
this.dslContext = dslContext;
42+
this.contentStore = contentStore;
43+
this.verificationQuery = verificationQuery;
44+
this.properties = properties;
45+
this.successCounter = Counter.builder("content.deletion")
46+
.tag("result", "success")
47+
.register(meterRegistry);
48+
this.failureCounter = Counter.builder("content.deletion")
49+
.tag("result", "failure")
50+
.register(meterRegistry);
51+
this.driftCounter = Counter.builder("content.deletion.drift")
52+
.register(meterRegistry);
53+
}
54+
55+
public void run() {
56+
log.info("Starting content deletion job");
57+
int processed = 0;
58+
int deleted = 0;
59+
int skipped = 0;
60+
61+
List<String> candidates = findDeletionCandidates();
62+
63+
for (String contentId : candidates) {
64+
processed++;
65+
66+
if (verificationQuery.isStillReferenced(contentId)) {
67+
log.warn("Count drift detected: content_id {} marked for deletion but still referenced", contentId);
68+
driftCounter.increment();
69+
clearDeletionMarker(contentId);
70+
skipped++;
71+
continue;
72+
}
73+
74+
if (deleteContent(contentId)) {
75+
deleted++;
76+
}
77+
}
78+
79+
log.info("Content deletion job completed: processed={}, deleted={}, skipped={}", processed, deleted, skipped);
80+
}
81+
82+
private List<String> findDeletionCandidates() {
83+
Select<Record1<String>> query = dslContext
84+
.select(CONTENT_ID)
85+
.from(CONTENT_REFERENCES)
86+
.where(MARKED_FOR_DELETION_AT.isNotNull())
87+
.and(MARKED_FOR_DELETION_AT.le(currentTimestamp()))
88+
.orderBy(MARKED_FOR_DELETION_AT)
89+
.limit(properties.getDeletion().getBatchSize());
90+
91+
return query.fetch(CONTENT_ID);
92+
}
93+
94+
private void clearDeletionMarker(String contentId) {
95+
dslContext.update(CONTENT_REFERENCES)
96+
.setNull(MARKED_FOR_DELETION_AT)
97+
.where(CONTENT_ID.eq(contentId))
98+
.execute();
99+
}
100+
101+
private boolean deleteContent(String contentId) {
102+
try {
103+
contentStore.remove(ContentReference.of(contentId));
104+
105+
dslContext.deleteFrom(CONTENT_REFERENCES)
106+
.where(CONTENT_ID.eq(contentId))
107+
.execute();
108+
109+
successCounter.increment();
110+
log.debug("Deleted content_id {}", contentId);
111+
return true;
112+
} catch (UnwritableContentException e) {
113+
log.error("Failed to delete content_id {}", contentId, e);
114+
failureCounter.increment();
115+
return false;
116+
}
117+
}
118+
}

‎contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentLifecycleProperties.java‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
package com.contentgrid.appserver.content.lifecycle;
2+
3+
import java.time.Duration;
4+
import lombok.Data;
5+
import org.springframework.boot.context.properties.ConfigurationProperties;
6+
7+
@Data
8+
@ConfigurationProperties(prefix = "contentgrid.content.lifecycle")
9+
public class ContentLifecycleProperties {
10+
private boolean enabled = true;
11+
12+
private Deletion deletion = new Deletion();
13+
14+
@Data
15+
public static class Deletion {
16+
private boolean enabled = true;
17+
private Duration gracePeriod = Duration.ofDays(7);
18+
private int batchSize = 100;
19+
}
20+
}

‎contentgrid-appserver-content-lifecycle/src/main/java/com/contentgrid/appserver/content/lifecycle/ContentReferenceTracker.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
package com.contentgrid.appserver.content.lifecycle;
2+
3+
import com.contentgrid.appserver.contentstore.api.ContentReference;
4+
5+
public interface ContentReferenceTracker {
6+
void incrementReference(ContentReference ref);
7+
void decrementReference(ContentReference ref);
8+
}

0 commit comments

Comments
 (0)