feat: ✨ add JSON and INT data types and update FFI to v0.17.1

.changeset/great-experts-repair.md

@@ -0,0 +1,17 @@
+---
+"@cipherstash/protect-dynamodb": major
+"@cipherstash/protect": major
+"@cipherstash/schema": major
+---
+
+Added JSON and INT data type support and update FFI to v0.17.1 with x86_64 musl environment platform support.
+
+* Update @cipherstash/protect-ffi from 0.16.0 to 0.17.1 with support for x86_64 musl platforms.
+* Add searchableJson() method to schema for JSON field indexing (the search operations still don't work but this interface exists)
+* Refactor type system: EncryptedPayload → Encrypted, add JsPlaintext
+* Add comprehensive test suites for JSON, integer, and basic encryption
+* Update encryption format to use 'k' property for searchable JSON
+* Remove deprecated search terms tests for JSON fields
+* Simplify schema data types to text, int, json only
+* Update model helpers to handle new encryption format
+* Fix type safety issues in bulk operations and model encryption

README.md

@@ -990,6 +990,10 @@ Other data types like booleans, dates, ints, floats, and JSON are well-supported
 
 Until support for other data types are available, you can express interest in this feature by adding a :+1: on this [GitHub Issue](https://github.com/cipherstash/protectjs/issues/48).
 
+| Type | Support operations | Available |
+|--|--|--|
+| String | `=`, `LIKE`, `ORDER BY` | ✅ | 
+
 ## Searchable encryption
 
 Read more about [searching encrypted data](./docs/concepts/searchable-encryption.md) in the docs.

docs/prompts/init-protect.md

@@ -0,0 +1,111 @@
+# Implementing Protect.js into a Node.js application
+
+Your task is to introduce Protect.js into a Node.js application. Protect.js requires the Node.js runtime so it will still work with framweworks like Next.js and Tanstack Start, since they support running code that only executes on the server.
+
+---
+
+## Installing Protect.js
+
+Determine what package manager the application is using. This will either be `pnpm`, `npm`, or `bun` and then use the appropriate package manager to add Protect.js to the application:
+
+```bash
+npm install @cipehrstash/protect
+# or
+pnpm add @cipehrstash/protect
+# or
+bun add @cipehrstash/protect
+```
+
+If you detect a mono repo, you need to ask the user which application they want Protect.js installed in.
+
+---
+
+## Adding scafolding for the Protect.js client, schemas, and example code
+
+In the rool of the application (if the application is configred to use something like `src` then this is where these operations will occur), you need to add a `protect` directory with the following files/content. If the application uses TypeScript use the `.ts` extension, else use the `.js` extension.
+
+`protect/schemas.(ts/js)`
+```js
+import { csTable, csColumn } from '@cipherstash/protect`
+
+export const protectedExample = csTable('example_table', {
+  sensitiveData: csColumn('sensitiveData'),
+}
+```
+
+`protect/index.(ts/js)`
+```js
+import { protect } from '@cipehrstash/protect'
+import { * as protectSchemas } from './schemas'
+
+export const protectClient = protect({
+  schemas: [...protectSchemas]
+})
+```
+
+`protect/example.(ts/js)`
+```js
+import { protectClient } from './index'
+import { * as protectSchemas } from './schemas'
+
+const sensitiveData = "Let's encrypt some data."
+
+/** 
+ * There is no need to wrap any protectClient method in a try/catch as it will always return a Result pattern.
+ * ---
+ * The Result will either contain a `failure` OR a `data` key. You should ALWAYS check for the `failure` key first.
+ * If the `failure` key is present, you should handle the error accordingly.
+ * If the `data` key is present, the operation was successful.
+ */
+//
+const encryptResult = protectClient.encrypt(sensitiveData, {
+  table: protectSchemas.protectedExample
+  column: protectSchemas.protectedExample.sensitiveData
+})
+
+if (encryptResult.failure) {
+  // You as the developer can determine exactly how you want to handle the failure scenario.
+}
+
+const encryptedData = encryptResult.data
+
+/**
+ * The encryptedData is a JSON object which is unique to CipherStash.
+ * You can store this data in any data store that supports JSON, however, only PostgreSQL supports searchable encryption operations.
+ * ---
+ * When storing in PostgreSQL, the database instance needs to be configured to Encrypt Query Language (EQL).
+ * More information can be found in the [Encrypt Query Language (EQL) documentation](https://github.com/cipherstash/encrypt-query-language).
+ * After EQL is installed, you can use the `eql_v2_encrypted` data type to store the encrypted data.
+ * ```sql
+ * CREATE TABLE protected_example (
+ *   id SERIAL PRIMARY KEY,
+ *   sensitive_data eql_v2_encrypted
+ * )
+ * ```
+ **/
+console.log('encryptedData:', encryptedData)
+
+const decryptResult = protectClient.decrypt(encryptedData)
+
+if (decryptResult.failure) {
+  // Again, you as the developer can determine exactly how you want to handle the failure scenario.
+}
+
+const decryptedData = decryptResult.data
+
+/**
+ * The decryptedData is the plaintext data.
+ */
+console.log('decryptedData:', decryptedData)
+```
+
+---
+
+After the initialization process, you need to also analyze the application code base to determine the following:
+
+1. What database and general data store (if any) does the application connect to and store data?
+2. Does the application use a specific ORM (like Drizzle ORM, Prisma, or Sequalize)?
+3. What types of data inputs does this application ingest that could be considered sensitive data?
+4. If the application is ingesting sensitive data, where does the application store the data?
+
+Use these answer in combonation with the answers provided a CIPHERSTASH_GETTING_STARTED.md file which will be used to help the user determine the next steps based on the application specific details.

packages/protect-dynamodb/__tests__/dynamodb.test.ts

@@ -8,10 +8,16 @@ const schema = csTable('dynamo_cipherstash_test', {
   firstName: csColumn('firstName').equality(),
   lastName: csColumn('lastName').equality(),
   phoneNumber: csColumn('phoneNumber'),
+  json: csColumn('json').dataType('json'),
+  jsonSearchable: csColumn('jsonSearchable').dataType('json'),
+  //.searchableJson('users/jsonSearchable'),
   example: {
     protected: csValue('example.protected'),
     deep: {
       protected: csValue('example.deep.protected'),
+      protectNestedJson: csValue('example.deep.protectNestedJson').dataType(
+        'json',
+      ),
     },
   },
 })
@@ -30,7 +36,7 @@ describe('protect dynamodb helpers', () => {
     })
   })
 
-  it('should encrypt columns', async () => {
+  it('should encrypt and decrypt a model', async () => {
     const testData = {
       id: '01ABCDEFGHIJKLMNOPQRSTUVWX',
       email: '[email protected]',
@@ -39,6 +45,22 @@ describe('protect dynamodb helpers', () => {
       firstName: 'John',
       lastName: 'Smith',
       phoneNumber: '555-555-5555',
+      json: {
+        name: 'John Doe',
+        age: 30,
+        preferences: {
+          theme: 'dark',
+          notifications: true,
+        },
+      },
+      jsonSearchable: {
+        name: 'John Doe',
+        age: 30,
+        preferences: {
+          theme: 'dark',
+          notifications: true,
+        },
+      },
       companyName: 'Acme Corp',
       batteryBrands: ['Brand1', 'Brand2'],
       metadata: { role: 'admin' },
@@ -48,6 +70,9 @@ describe('protect dynamodb helpers', () => {
         deep: {
           protected: 'deep protected',
           notProtected: 'deep not protected',
+          protectNestedJson: {
+            hello: 'world',
+          },
         },
       },
     }
@@ -80,6 +105,16 @@ describe('protect dynamodb helpers', () => {
     expect(encryptedData.example.notProtected).toBe('I am not protected')
     expect(encryptedData.example.deep.notProtected).toBe('deep not protected')
     expect(encryptedData.metadata).toEqual({ role: 'admin' })
+
+    const decryptResult = await protectDynamo.decryptModel(
+      encryptedData,
+      schema,
+    )
+    if (decryptResult.failure) {
+      throw new Error(`Decryption failed: ${decryptResult.failure.message}`)
+    }
+
+    expect(decryptResult.data).toEqual(testData)
   })
 
   it('should handle null and undefined values', async () => {

packages/protect-dynamodb/src/helpers.ts

@@ -1,4 +1,8 @@
-import type { EncryptedPayload } from '@cipherstash/protect'
+import type {
+  Encrypted,
+  ProtectTable,
+  ProtectTableColumn,
+} from '@cipherstash/protect'
 import type { ProtectDynamoDBError } from './types'
 export const ciphertextAttrSuffix = '__source'
 export const searchTermAttrSuffix = '__hmac'
@@ -83,7 +87,7 @@ export function toEncryptedDynamoItem(
         typeof attrValue === 'object' &&
         'c' in (attrValue as object))
     ) {
-      const encryptPayload = attrValue as EncryptedPayload
+      const encryptPayload = attrValue as Encrypted
       if (encryptPayload?.c) {
         const result: Record<string, unknown> = {}
         if (encryptPayload.hm) {
@@ -92,6 +96,13 @@ export function toEncryptedDynamoItem(
         result[`${attrName}${ciphertextAttrSuffix}`] = encryptPayload.c
         return result
       }
+
+      // TODO: Need to implement the new Encrypt payload type when FFI is updated
+      if (encryptPayload?.sv) {
+        const result: Record<string, unknown> = {}
+        result[`${attrName}${ciphertextAttrSuffix}`] = encryptPayload.sv
+        return result
+      }
     }
 
     // Handle nested objects recursively
@@ -122,8 +133,8 @@ export function toEncryptedDynamoItem(
 }
 
 export function toItemWithEqlPayloads(
-  decrypted: Record<string, EncryptedPayload | unknown>,
-  encryptedAttrs: string[],
+  decrypted: Record<string, Encrypted | unknown>,
+  encryptSchemas: ProtectTable<ProtectTableColumn>,
 ): Record<string, unknown> {
   function processValue(
     attrName: string,
@@ -139,24 +150,42 @@ export function toItemWithEqlPayloads(
       return {}
     }
 
+    const encryptConfig = encryptSchemas.build()
+    const encryptedAttrs = Object.keys(encryptConfig.columns)
+    const columnName = attrName.slice(0, -ciphertextAttrSuffix.length)
+
     // Handle encrypted payload
     if (
       attrName.endsWith(ciphertextAttrSuffix) &&
-      (encryptedAttrs.includes(
-        attrName.slice(0, -ciphertextAttrSuffix.length),
-      ) ||
-        isNested)
+      (encryptedAttrs.includes(columnName) || isNested)
     ) {
-      const baseName = attrName.slice(0, -ciphertextAttrSuffix.length)
+      // TODO: Implement the standardized typing for Encrypted Payloads through the FFI interface
+      const i = { c: columnName, t: encryptConfig.tableName }
+      const v = 2
+
+      // Nested values are not searchable, so we can just return the standard EQL payload.
+      // Worth noting, that encryptConfig.columns[columnName] will be undefined if isNested is true.
+      if (
+        !isNested &&
+        encryptConfig.columns[columnName].cast_as === 'jsonb' &&
+        encryptConfig.columns[columnName].indexes.ste_vec
+      ) {
+        return {
+          [columnName]: {
+            i,
+            v,
+            k: 'sv',
+            sv: attrValue,
+          },
+        }
+      }
+
       return {
-        [baseName]: {
+        [columnName]: {
+          i,
+          v,
+          k: 'ct',
           c: attrValue,
-          bf: null,
-          hm: null,
-          i: { c: 'notUsed', t: 'notUsed' },
-          k: 'notUsed',
-          ob: null,
-          v: 2,
         },
       }
     }

packages/protect-dynamodb/src/index.ts

@@ -1,5 +1,5 @@
 import type {
-  EncryptedPayload,
+  Encrypted,
   ProtectTable,
   ProtectTableColumn,
   SearchTerm,
@@ -42,7 +42,7 @@ export function protectDynamoDB(
     },
 
     decryptModel<T extends Record<string, unknown>>(
-      item: Record<string, EncryptedPayload | unknown>,
+      item: Record<string, Encrypted | unknown>,
       protectTable: ProtectTable<ProtectTableColumn>,
     ) {
       return new DecryptModelOperation<T>(
@@ -54,7 +54,7 @@ export function protectDynamoDB(
     },
 
     bulkDecryptModels<T extends Record<string, unknown>>(
-      items: Record<string, EncryptedPayload | unknown>[],
+      items: Record<string, Encrypted | unknown>[],
       protectTable: ProtectTable<ProtectTableColumn>,
     ) {
       return new BulkDecryptModelsOperation<T>(

packages/protect-dynamodb/src/operations/bulk-decrypt-models.ts

@@ -1,7 +1,7 @@
 import { type Result, withResult } from '@byteslice/result'
 import type {
   Decrypted,
-  EncryptedPayload,
+  Encrypted,
   ProtectClient,
   ProtectTable,
   ProtectTableColumn,
@@ -17,12 +17,12 @@ export class BulkDecryptModelsOperation<
   T extends Record<string, unknown>,
 > extends DynamoDBOperation<Decrypted<T>[]> {
   private protectClient: ProtectClient
-  private items: Record<string, EncryptedPayload | unknown>[]
+  private items: Record<string, Encrypted | unknown>[]
   private protectTable: ProtectTable<ProtectTableColumn>
 
   constructor(
     protectClient: ProtectClient,
-    items: Record<string, EncryptedPayload | unknown>[],
+    items: Record<string, Encrypted | unknown>[],
     protectTable: ProtectTable<ProtectTableColumn>,
     options?: DynamoDBOperationOptions,
   ) {
@@ -37,9 +37,8 @@ export class BulkDecryptModelsOperation<
   > {
     return await withResult(
       async () => {
-        const encryptedAttrs = Object.keys(this.protectTable.build().columns)
         const itemsWithEqlPayloads = this.items.map((item) =>
-          toItemWithEqlPayloads(item, encryptedAttrs),
+          toItemWithEqlPayloads(item, this.protectTable),
         )
 
         const decryptResult = await this.protectClient