diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 0000000..a306457 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,277 @@ +# CBElasticsearch Module Development Instructions + +CBElasticsearch is a CFML (ColdFusion/Lucee) module that provides a fluent API for Elasticsearch integration with the ColdBox Framework. It includes CacheBox cache providers and LogBox appenders for Elasticsearch. + +Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here. + +## Working Effectively + +### Prerequisites and Dependencies +- Install Java 17+ (OpenJDK Temurin recommended) +- Install Docker and Docker Compose +- Install CommandBox CLI for CFML development + +### Bootstrap and Setup Commands +Execute these commands in exact order to set up the development environment: + +```bash +# 1. Install CommandBox (if not available, use Docker approach below) +curl -fsSL https://downloads.ortussolutions.com/debs/gpg | sudo apt-key add - +echo "deb https://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a /etc/apt/sources.list.d/commandbox.list +sudo apt-get update && sudo apt-get install -y commandbox + +# 2. Install project dependencies (takes 2-5 minutes depending on network) +box install +cd test-harness && box install && cd ../ + +# 3. Start Elasticsearch with Docker Compose (takes 30-45 seconds) +# NEVER CANCEL: Elasticsearch startup takes 30-45 seconds. Set timeout to 90+ seconds. +docker compose up -d elasticsearch + +# 4. Wait for Elasticsearch to be ready (critical - wait for green status) +sleep 30 +curl -s http://localhost:9200/_cluster/health | jq '.status' # Should return "green" +``` + +### Alternative Setup (when CommandBox installation fails) +If CommandBox installation fails due to network restrictions: + +```bash +# Use Docker for CommandBox operations (NOTE: ForgeBox access may still fail) +docker run --rm -v $(pwd):/app ortussolutions/commandbox:lucee5 box install +cd test-harness && docker run --rm -v $(pwd):/app ortussolutions/commandbox:lucee5 box install && cd ../ + +# If ForgeBox access fails completely, document this limitation: +# "box install -- fails due to network restrictions preventing ForgeBox access" +``` + +### Build and Test Commands +- **NEVER CANCEL**: Build operations take 5-15 minutes. Set timeout to 30+ minutes. +- **NEVER CANCEL**: Test suite takes 3-8 minutes per engine. Set timeout to 20+ minutes. + +```bash +# Format code (ALWAYS run before committing and pushing any files) +# CRITICAL: This command MUST be run before any commit to ensure code formatting compliance +# NOTE: Requires cfformat module - may fail if dependencies unavailable +box run-script format + +# Check code formatting +# NOTE: Requires cfformat module - may fail if dependencies unavailable +box run-script format:check + +# Lint code (if cflint is available) +cflint + +# Start test server (takes 60-90 seconds to fully boot) +# NEVER CANCEL: Server startup takes 60-90 seconds. Set timeout to 180+ seconds. +box start + +# Run complete test suite (takes 3-8 minutes) +# NEVER CANCEL: Tests take 3-8 minutes to complete. Set timeout to 15+ minutes. +box testbox run + +# Build module for distribution (takes 5-15 minutes) +# NEVER CANCEL: Build takes 5-15 minutes. Set timeout to 30+ minutes. +box run-script build:module + +# Build documentation only +box run-script build:docs +``` + +### Development Workflow Commands +```bash +# Start development environment +docker compose up -d # Starts Elasticsearch + app containers +box start # Starts local CommandBox server + +# Run tests during development +box testbox run --verbose # Run all tests with verbose output +box testbox run --directory=tests/specs/unit # Run only unit tests + +# Stop everything +box stop # Stop CommandBox server +docker compose down # Stop all Docker services +``` + +## Validation + +### Required Validation Steps +ALWAYS run these validation steps after making code changes: + +1. **Elasticsearch Health Validation**: + ```bash + # Verify Elasticsearch is running and healthy (takes 5-10 seconds) + curl -s http://localhost:9200/_cluster/health | jq '.status' # Should return "green" + ``` + +2. **Code Quality Validation** (if dependencies available): + ```bash + box run-script format:check # Verify code formatting + box run-script format # Auto-fix formatting issues + # CRITICAL: Always run format command before committing any files + ``` + +3. **Build Validation** (if dependencies available): + ```bash + # NEVER CANCEL: Build takes 5-15 minutes + box run-script build:module + ``` + +4. **Test Validation** (if dependencies available): + ```bash + # Ensure Elasticsearch is running first + curl -s http://localhost:9200/_cluster/health | jq '.status' + + # NEVER CANCEL: Tests take 3-8 minutes per engine + box testbox run --verbose + ``` + +4. **End-to-End Scenario Testing**: + After making changes, always test these core scenarios: + - **Search Operations**: Create index, add documents, search, retrieve results + - **Cache Operations**: Store/retrieve data via CacheBox provider + - **Logging**: Verify LogBox appender logs to Elasticsearch + - **Index Management**: Create/update/delete indices and mappings + +### CI/CD Validation +The GitHub Actions CI will fail if: +- Code formatting is incorrect (ALWAYS run `box run-script format` before committing) +- Tests fail on any supported engine (Lucee 5/6, Adobe CF 2021/2023/2025, BoxLang) +- Build process fails +- Security scans detect high-severity issues + +## Environment Configuration + +### Elasticsearch Configuration +The module supports both Elasticsearch 7.x and 8.x. Default configuration: +```ini +ELASTICSEARCH_PROTOCOL=http +ELASTICSEARCH_HOST=127.0.0.1 +ELASTICSEARCH_PORT=9200 +``` + +### Multi-Version Testing +The CI tests against multiple configurations: +- **Elasticsearch**: 7.17.10, 8.14.1, 8.17.1 +- **CFML Engines**: Lucee 5/6, Adobe CF 2021/2023/2025, BoxLang 1.0 +- **ColdBox Versions**: 6.x, 7.x + +### Docker Compose Services +Available services via `docker compose up`: +- `elasticsearch`: Elasticsearch 8.17.1 on port 9200 +- `elasticsearch7`: Elasticsearch 7.17.6 on port 9201 +- `app`: Adobe CF 2021 with module on port 8080 +- `app-lucee5`: Lucee 5 with module on port 8081 + +## Critical Timing Information + +### Build Times and Timeouts +- **Elasticsearch startup**: 5-10 seconds (timeout: 30+ seconds) +- **CommandBox server startup**: 60-90 seconds (timeout: 180+ seconds) +- **Dependency installation**: 2-5 minutes (timeout: 10+ minutes) *[may fail due to network restrictions]* +- **Test suite execution**: 3-8 minutes per engine (timeout: 15+ minutes) +- **Module build**: 5-15 minutes (timeout: 30+ minutes) +- **Documentation build**: 2-5 minutes (timeout: 10+ minutes) +- **Docker Compose full startup**: 25-30 seconds (timeout: 60+ seconds) +- **Docker Compose shutdown**: 10-15 seconds (timeout: 30+ seconds) + +### **CRITICAL: NEVER CANCEL LONG-RUNNING OPERATIONS** +All build and test operations may take significant time. Canceling prematurely will result in incomplete builds and failed tests. + +## Key Project Structure + +### Core Module Files +- `ModuleConfig.cfc` - Module configuration and dependencies +- `models/` - Core module classes (SearchBuilder, IndexBuilder, etc.) +- `models/cache/` - CacheBox provider implementation +- `models/logging/` - LogBox appender implementation + +### Testing Infrastructure +- `test-harness/` - Complete test application +- `test-harness/tests/specs/unit/` - Unit tests +- `test-harness/config/Coldbox.cfc` - Test app configuration + +### Build System +- `box.json` - Project dependencies and scripts +- `build/Build.cfc` - Build automation scripts +- `build/release.boxr` - Release automation recipe +- `.github/workflows/` - CI/CD pipeline definitions + +### Configuration Files +- `docker-compose.yml` - Development environment setup +- `.cfformat.json` - Code formatting rules +- `.cflintrc` - Code linting configuration +- `.env.template` - Environment variable template + +## Common Tasks + +### Creating New Features +1. Write unit tests first in `test-harness/tests/specs/unit/` +2. Implement feature in appropriate `models/` subdirectory +3. Update `ModuleConfig.cfc` if adding new mappings +4. Run validation steps (ALWAYS run `box run-script format` before committing, then build, test) +5. Update documentation in `docs/` if needed + +### Debugging Test Failures +1. Check Elasticsearch status: `curl http://localhost:9200/_cluster/health` +2. Review server logs: `box server log` +3. Check Docker logs: `docker compose logs elasticsearch` +4. Run specific test suites: `box testbox run --directory=tests/specs/unit/SpecificTest.cfc` + +### Adding Dependencies +1. Update `box.json` dependencies section +2. Run `box install` to install +3. Update test-harness: `cd test-harness && box install` +4. Test build process: `box run-script build:module` + +### Release Process +Releases are automated via GitHub Actions, but manual releases use: +```bash +# Update changelog.md with new version info +# Update box.json version number +box recipe build/release.boxr +``` + +## Network and Access Limitations + +If you encounter network restrictions preventing access to ForgeBox or other external resources: +- Use Docker-based CommandBox operations when possible +- Document any commands that fail due to network restrictions +- Focus on testing local functionality that doesn't require external dependencies +- The CI environment has full network access and will validate complete builds + +### Commands That May Fail Due to Network Restrictions +```bash +# These commands require ForgeBox/external access: +box install # "forgebox ran into an issue" +box run-script format # Requires cfformat module +box run-script format:check # Requires cfformat module +``` + +### Working Commands in Restricted Environments +```bash +# These commands work without external access: +docker compose up -d elasticsearch # Uses local/cached Docker images +docker compose logs elasticsearch # Local Docker logs +curl http://localhost:9200/_cluster/health # Local Elasticsearch API +docker run --rm -v $(pwd):/app ortussolutions/commandbox:lucee5 box version # Basic CommandBox +``` + +## Error Recovery + +### Common Issues and Solutions +1. **"CommandBox not found"** - Use Docker approach or install from GitHub releases +2. **"Elasticsearch connection failed"** - Verify `docker compose up -d elasticsearch` succeeded +3. **"ForgeBox connection failed"** - Network restrictions, use cached dependencies or Docker +4. **"Tests hanging"** - Increase timeout values, Elasticsearch may still be starting +5. **"Build fails"** - Check code formatting first: `box run-script format` (ALWAYS run before committing) + +### Recovery Commands +```bash +# Reset development environment +box stop +docker compose down +docker compose up -d elasticsearch +sleep 30 +box start +``` \ No newline at end of file diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index a5cd3b9..ba855ec 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -18,31 +18,43 @@ jobs: continue-on-error: ${{ matrix.experimental }} strategy: matrix: - cfengine: [ "lucee@5", "adobe@2018", "adobe@2021", "adobe@2023" ] - coldboxVersion: [ "^6", "^7" ] - javaVersion: [ "11" ] - ELASTICSEARCH_VERSION: [ "7.17.10", "8.14.1" ] - experimental: [ false ] - commandboxVersion: [ "6.1.0" ] + cfengine: ["lucee@5", "adobe@2023", "adobe@2025"] + coldboxVersion: ["^6", "^7"] + javaVersion: ["17"] + ELASTICSEARCH_VERSION: ["7.17.10", "8.14.1"] + experimental: [false] + commandboxVersion: ["6.2.1"] include: - - cfengine: "lucee@6" - coldboxVersion: "^7" + - cfengine: "boxlang@1" + coldboxVersion: "be" javaVersion: "21" ELASTICSEARCH_VERSION: "8.14.1" - experimental: true - commandboxVersion: "6.1.0" + experimental: false + commandboxVersion: "6.2.1" - cfengine: "boxlang@1" coldboxVersion: "be" javaVersion: "21" + ELASTICSEARCH_VERSION: "7.17.10" + experimental: false + commandboxVersion: "6.2.1" + - cfengine: "adobe@2021" + coldboxVersion: "^7" + javaVersion: "11" + ELASTICSEARCH_VERSION: "8.14.1" + experimental: false + commandboxVersion: "6.2.1" + - cfengine: "lucee@6" + coldboxVersion: "^7" + javaVersion: "17" ELASTICSEARCH_VERSION: "8.14.1" experimental: true - commandboxVersion: "6.1.0" - - cfengine: "boxlang@1" + commandboxVersion: "6.2.1" + - cfengine: "boxlang@be" coldboxVersion: "be" javaVersion: "21" - ELASTICSEARCH_VERSION: "7.17.10" + ELASTICSEARCH_VERSION: "8.14.1" experimental: true - commandboxVersion: "6.1.0" + commandboxVersion: "6.2.1" steps: - name: Checkout Repository uses: actions/checkout@v4 @@ -66,7 +78,7 @@ jobs: - name: Install Dependencies run: | box run-script install:dependencies - + - name: Set ColdBox Version working-directory: ./test-harness run: | @@ -139,7 +151,7 @@ jobs: SLACK_CHANNEL: coding SLACK_COLOR: ${{ job.status }} # or a specific color like 'green' or '#ff00ff' SLACK_ICON_EMOJI: ":bell:" - SLACK_MESSAGE: '${{ github.repository }} tests failed :cry:' + SLACK_MESSAGE: "${{ github.repository }} tests failed :cry:" SLACK_TITLE: ${{ github.repository }} Tests For ${{ matrix.cfengine }} failed SLACK_USERNAME: CI SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }} diff --git a/ModuleConfig.cfc b/ModuleConfig.cfc index 944e4ab..8dbbeac 100644 --- a/ModuleConfig.cfc +++ b/ModuleConfig.cfc @@ -68,7 +68,9 @@ component { // Read timeout - the read timeout in milliseconds readTimeout : getSystemSetting( "ELASTICSEARCH_READ_TIMEOUT", 3000 ), // Connection timeout - timeout attempts to connect to elasticsearch after this timeout - connectionTimeout : getSystemSetting( "ELASTICSEARCH_CONNECT_TIMEOUT", 3000 ) + connectionTimeout : getSystemSetting( "ELASTICSEARCH_CONNECT_TIMEOUT", 3000 ), + // used by util:getRealIp, trustUpstream If true, we check the forwarded headers first, else we don't + trustUpstream : false }; // Custom Declared Points diff --git a/box.json b/box.json index 8212cb9..c7444f7 100644 --- a/box.json +++ b/box.json @@ -2,7 +2,7 @@ "name":"Elasticsearch for the Coldbox Framework", "author":"Ortus Solutions ", "location":"https://downloads.ortussolutions.com/ortussolutions/coldbox-modules/cbelasticsearch/@build.version@/cbelasticsearch-@build.version@+@build.number@.zip", - "version":"3.5.0", + "version":"3.5.2", "slug":"cbelasticsearch", "type":"modules", "homepage":"https://cbelasticsearch.ortusbooks.com", @@ -29,7 +29,7 @@ "Luis Majano " ], "dependencies":{ - "hyper":"^8.0.0" + "hyper":"^7.5.3" }, "ignore":[ "**/.*", diff --git a/changelog.md b/changelog.md index 087eb2a..279b985 100644 --- a/changelog.md +++ b/changelog.md @@ -8,10 +8,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 * * * ## [Unreleased] + +## [3.5.0] - 2025-10-01 + +## [3.5.0] - 2025-03-26 + ### Added + - Boxlang runtime certification ### Removed + - `ensureNativeStruct` and `newHashMap` removed from `models.util.Util` - Removed custom routine for async logging as it conflicted with the logger implementation diff --git a/models/Task.cfc b/models/Task.cfc index 697e6a9..d592278 100644 --- a/models/Task.cfc +++ b/models/Task.cfc @@ -54,13 +54,13 @@ component accessors="true" { variables.append( taskProperties ); // calculate our time - if ( taskProperties.keyExists( "start_time_in_millis" ) ){ - var epochDate = parseDateTime( "1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:nn:ss.SSSX" ); + if ( taskProperties.keyExists( "start_time_in_millis" ) ) { + var epochDate = parseDateTime( "1970-01-01T00:00:00.000Z", "yyyy-MM-dd'T'HH:nn:ss.SSSX" ); variables.startTime = dateAdd( "s", round( taskProperties.start_time_in_millis / 1000 ), epochDate - ); + ); } else { variables.StartTime = now(); } diff --git a/models/cache/ColdboxProvider.cfc b/models/cache/ColdboxProvider.cfc index ae7b589..806ef42 100644 --- a/models/cache/ColdboxProvider.cfc +++ b/models/cache/ColdboxProvider.cfc @@ -6,12 +6,10 @@ * @author Jon Clausen * @license Apache v2.0 * + * * Note: We cannot implement the interface coldbox.system.cache.IColdboxApplicationCache on the component declaration until we discontinue CF6 support + * **/ -component - serializable="false" - extends ="Provider" - implements ="coldbox.system.cache.IColdboxApplicationCache" -{ +component serializable="false" extends="Provider" { ColdboxProvider function init() output=false{ super.init(); @@ -38,7 +36,7 @@ component } // set the coldbox controller - void function setColdbox( required any coldbox ) output=false{ + function setColdbox( required any coldbox ) output=false{ variables.coldbox = arguments.coldbox; } @@ -55,7 +53,7 @@ component /** * Clear all events */ - void function clearAllEvents( async = false ) output=false{ + function clearAllEvents( async = false ) output=false{ var threadName = "clearAllEvents_#replace( instance.uuidHelper.randomUUID(), "-", @@ -76,7 +74,7 @@ component /** * Clear all views */ - void function clearAllViews( async = false ) output=false{ + function clearAllViews( async = false ) output=false{ var threadName = "clearAllViews_#replace( instance.uuidHelper.randomUUID(), "-", @@ -97,28 +95,28 @@ component /** * Clear event */ - void function clearEvent( required eventsnippet, queryString = "" ) output=false{ + function clearEvent( required eventsnippet, queryString = "" ) output=false{ instance.elementCleaner.clearEvent( arguments.eventsnippet, arguments.queryString ); } /** * Clear multiple events */ - void function clearEventMulti( required eventsnippets, queryString = "" ) output=false{ + function clearEventMulti( required eventsnippets, queryString = "" ) output=false{ instance.elementCleaner.clearEventMulti( arguments.eventsnippets, arguments.queryString ); } /** * Clear view */ - void function clearView( required viewSnippet ) output=false{ + function clearView( required viewSnippet ) output=false{ instance.elementCleaner.clearView( arguments.viewSnippet ); } /** * Clear multiple view */ - void function clearViewMulti( required viewsnippets ) output=false{ + function clearViewMulti( required viewsnippets ) output=false{ instance.elementCleaner.clearView( arguments.viewsnippets ); } diff --git a/models/cache/Provider.cfc b/models/cache/Provider.cfc index 4638e9f..74ea0e1 100644 --- a/models/cache/Provider.cfc +++ b/models/cache/Provider.cfc @@ -6,11 +6,12 @@ * @author Jon Clausen * @license Apache v2.0 * + * Note: We cannot implement the interface coldbox.system.cache.providers.ICacheProvider on the component declaration until we discontinue CF6 support + * **/ component name ="ElasticsearchProvider" serializable="false" - implements ="coldbox.system.cache.ICacheProvider" accessors =true { @@ -104,70 +105,70 @@ component /** * get the cache name */ - any function getName() output="false"{ + any function getName(){ return this.name; } /** * get the cache provider version */ - any function getVersion() output="false"{ + any function getVersion(){ return this.version; } /** * set the cache name */ - void function setName( required name ) output="false"{ + function setName( required name ){ this.name = arguments.name; } /** * set the event manager */ - void function setEventManager( required any EventManager ) output="false"{ + function setEventManager( required any EventManager ){ this.eventManager = arguments.eventManager; } /** * get the event manager */ - any function getEventManager() output="false"{ + any function getEventManager(){ return this.eventManager; } /** * get the cache configuration structure */ - any function getConfiguration() output="false"{ + struct function getConfiguration(){ return this.config; } /** * set the cache configuration structure */ - void function setConfiguration( required any configuration ) output="false"{ + function setConfiguration( required struct configuration ){ this.config = arguments.configuration; } /** * get the associated cache factory */ - any function getCacheFactory() output="false"{ + coldbox.system.cache.CacheFactory function getCacheFactory(){ return this.cacheFactory; } /** * set the associated cache factory */ - void function setCacheFactory( required any cacheFactory ) output="false"{ + function setCacheFactory( required any cacheFactory ){ this.cacheFactory = arguments.cacheFactory; } /** * configure the cache for operation */ - void function configure() output="false"{ + function configure(){ var config = getConfiguration(); var props = []; var URIs = []; @@ -188,21 +189,21 @@ component /** * shutdown the cache */ - void function shutdown() output="false"{ + function shutdown(){ Logger.info( "Provider Cache: #getName()# has been shutdown." ); } /* * Indicates if cache is ready for operation */ - any function isEnabled() output="false"{ + boolean function isEnabled(){ return this.enabled; } /* * Indicates if cache is ready for reporting */ - any function isReportingEnabled() output="false"{ + boolean function isReportingEnabled(){ return this.reportingEnabled; } @@ -210,21 +211,21 @@ component * Get the cache statistics object as coldbox.system.cache.util.ICacheStats * @colddoc:generic coldbox.system.cache.util.ICacheStats */ - any function getStats() output="false"{ + any function getStats(){ // Not yet implmented } /** * clear the cache stats: */ - void function clearStatistics() output="false"{ + function clearStatistics(){ // Not yet implemented } /** * Returns the underlying cache engine represented by the module ElasticSearch client */ - any function getObjectStore() output="false"{ + any function getObjectStore(){ // This provider uses an external object store return getClient(); } @@ -233,7 +234,7 @@ component * get the cache's metadata report * @tested */ - any function getStoreMetadataReport() output="false"{ + struct function getStoreMetadataReport(){ var md = {}; var keys = getKeys(); var item = ""; @@ -248,7 +249,7 @@ component * Get a key lookup structure where cachebox can build the report on. Ex: [timeout=timeout,lastAccessTimeout=idleTimeout]. It is a way for the visualizer to construct the columns correctly on the reports * @tested */ - any function getStoreMetadataKeyMap() output="false"{ + struct function getStoreMetadataKeyMap(){ var keyMap = { LastAccessed : "LastAccessed", isExpired : "isExpired", @@ -264,7 +265,7 @@ component * get all the keys in this provider * @tested */ - any function getKeys() output="false"{ + array function getKeys(){ local.allView = get( this.designDocumentName ); if ( isNull( local.allView ) ) { @@ -279,7 +280,7 @@ component return local.allView; } - void function appendCacheKey( objectKey ){ + function appendCacheKey( objectKey ){ var result = get( this.designDocumentName ); if ( !isNull( result ) && isArray( result ) ) { @@ -298,7 +299,7 @@ component * get an object's cached metadata * @tested */ - any function getCachedObjectMetadata( required any objectKey ) output="false"{ + struct function getCachedObjectMetadata( required any objectKey ){ // lower case the keys for case insensitivity if ( !getConfiguration().caseSensitiveKeys ) arguments.objectKey = lCase( arguments.objectKey ); @@ -337,7 +338,7 @@ component local.keyStats.timeExpires = dateAdd( "s", local.stats[ "key_exptime" ], - dateConvert( "utc2Local", "January 1 1970 00:00" ) + dateConvert( "utc2Local", "1970-01-01T00:00:00Z" ) ); } // key_last_modification_time @@ -349,7 +350,7 @@ component local.keyStats.lastAccessed = dateAdd( "s", local.stats[ "key_last_modification_time" ], - dateConvert( "utc2Local", "January 1 1970 00:00" ) + dateConvert( "utc2Local", "1970-01-01T00:00:00Z" ) ); } // state @@ -400,7 +401,7 @@ component * get an item from cache, returns null if not found. * @tested */ - any function get( required any objectKey ) output="false"{ + any function get( required any objectKey ){ return getQuiet( argumentCollection = arguments ); } @@ -408,7 +409,7 @@ component * get an item silently from cache, no stats advised: Stats not available on Elasticsearch * @tested */ - any function getQuiet( required any objectKey ) output="false"{ + any function getQuiet( required any objectKey ){ // lower case the keys for case insensitivity if ( !getConfiguration().caseSensitiveKeys ) arguments.objectKey = lCase( arguments.objectKey ); @@ -505,7 +506,7 @@ component /** * Checks if a value has expired */ - any function isExpired( required any objectKey ) output="false"{ + boolean function isExpired( required any objectKey ){ return isNull( getClient().get( arguments.objectKey ) ); } @@ -513,7 +514,7 @@ component * check if object in cache * @tested */ - any function lookup( required any objectKey ) output="false"{ + boolean function lookup( required any objectKey ){ return ( isNull( get( objectKey ) ) ? false : true ); } @@ -521,7 +522,7 @@ component * check if object in cache with no stats: Stats not available on Elasticsearch * @tested */ - any function lookupQuiet( required any objectKey ) output="false"{ + boolean function lookupQuiet( required any objectKey ){ return lookup( arguments.objectKey ); } @@ -535,8 +536,8 @@ component required any object, any timeout = this.config.objectDefaultTimeout, any lastAccessTimeout = 0, // Not in use for this provider - any extra = {} - ) output="false"{ + struct extra = {} + ){ var ts = getTickCount(); var future = setQuiet( argumentCollection = arguments ); @@ -572,13 +573,13 @@ component * lastAccessTimeout.hint Not used in this provider * @tested */ - any function setQuiet( + function setQuiet( required any objectKey, required any object, any timeout = this.config.objectDefaultTimeout, any lastAccessTimeout = 0, // Not in use for this provider - any extra = {} - ) output="false"{ + struct extra = {} + ){ return persistToCache( arguments.objectKey, formatCacheObject( argumentCollection = arguments ) ); } @@ -593,7 +594,7 @@ component any timeout = this.config.objectDefaultTimeout, any lastAccessTimeout = 0, // Not in use for this provider any extra = {} - ) output="false"{ + ){ var ts = getTickCount(); var documents = []; @@ -627,7 +628,7 @@ component any timeout = this.config.objectDefaultTimeout, any lastAccessTimeout = 0, // Not in use for this provider any extra = {} - ) output="false"{ + ){ // create storage element var sElement = { "createdDate" : dateFormat( now(), "mm/dd/yyyy" ) & " " & timeFormat( now(), "full" ), @@ -653,7 +654,7 @@ component required any cacheObject, boolean replaceItem = false any extra - ) output="false"{ + ){ if ( !getConfiguration().caseSensitiveKeys ) arguments.objectKey = lCase( arguments.objectKey ); @@ -683,7 +684,7 @@ component return future; } - void function updateObjectStats( required any objectKey, required any cacheObject ){ + function updateObjectStats( required any objectKey, required any cacheObject ){ if ( !getConfiguration().caseSensitiveKeys ) arguments.objectKey = lCase( arguments.objectKey ); if ( !structKeyExists( cacheObject, "hits" ) ) cacheObject[ "hits" ] = 0; @@ -704,15 +705,15 @@ component * get cache size * @tested */ - any function getSize() output="false"{ - // Not implemented + numeric function getSize(){ + return getKeys().len(); } /** * Not implemented by this cache * @tested */ - void function reap() output="false"{ + function reap(){ // Not implemented by this provider } @@ -720,7 +721,7 @@ component * clear all elements from cache * @tested */ - void function clearAll() output="false"{ + function clearAll(){ // If flush is not enabled for this bucket, no error will be thrown. The call will simply return and nothing will happen. // Be very careful calling this. It is an intensive asynch operation and the cache won't receive any new items until the flush // is finished which might take a few minutes. @@ -738,7 +739,7 @@ component * clear an element from cache and returns the Elasticsearch java future * @tested */ - any function clear( required any objectKey ) output="false"{ + boolean function clear( required any objectKey ){ // lower case the keys for case insensitivity if ( !getConfiguration().caseSensitiveKeys ) arguments.objectKey = lCase( arguments.objectKey ); @@ -751,13 +752,13 @@ component var document = newDocument().new( getConfiguration().index, getConfiguration().type ); document.setId( arguments.objectKey ); - var future = ElasticsearchClient.delete( document, true, { "refresh" : "wait_for" } ); + var deleteresult = ElasticsearchClient.delete( document, true, { "refresh" : "wait_for" } ); // ColdBox events var iData = { - cache : this, - cacheObjectKey : arguments.objectKey, - ElasticsearchFuture : future + cache : this, + cacheObjectKey : arguments.objectKey, + deleteResult : deleteresult }; getEventManager().processState( @@ -766,14 +767,14 @@ component async = true ); - return future; + return deleteresult; } /** * Clear with no advising to events and returns with the Elasticsearch java future * @tested */ - any function clearQuiet( required any objectKey ) output="false"{ + boolean function clearQuiet( required any objectKey ){ // normal clear, not implemented by Elasticsearch return clear( arguments.objectKey ); } @@ -781,11 +782,11 @@ component /** * Clear by key snippet */ - void function clearByKeySnippet( + function clearByKeySnippet( required keySnippet, regex = false, async = false - ) output="false"{ + ){ var threadName = "clearByKeySnippet_#replace( this.uuidHelper.randomUUID(), "-", "", "all" )#"; // Async? IF so, do checks @@ -802,15 +803,16 @@ component * Expiration not implemented by Elasticsearch so clears are issued * @tested */ - void function expireAll() output="false"{ + function expireAll(){ clearAll(); } /** * Expiration not implemented by Elasticsearch so clear is issued * @tested + * */ - void function expireObject( required any objectKey ) output="false"{ + function expireObject( required any objectKey ){ clear( arguments.objectKey ); } @@ -819,7 +821,7 @@ component /** * Validate the incoming configuration and make necessary defaults **/ - private void function validateConfiguration() output="false"{ + private function validateConfiguration(){ var cacheConfig = getConfiguration(); var key = ""; diff --git a/models/logging/AppenderService.cfc b/models/logging/AppenderService.cfc index 13b0d6e..d141189 100644 --- a/models/logging/AppenderService.cfc +++ b/models/logging/AppenderService.cfc @@ -147,10 +147,12 @@ component accessors="true" singleton { } ); elasticsearchClient.processBulkOperation( - inserts.map( ( doc ) => [ - "operation": { "create" : createOptions }, - "source" : doc - ] ), + inserts.map( ( doc ) => { + return [ + "operation": { "create" : createOptions }, + "source" : doc + ]; + } ), { "refresh" : refresh } ); } else { @@ -180,7 +182,7 @@ component accessors="true" singleton { }, "http" : { "request" : { "referer" : CGI.HTTP_REFERER } }, "host" : { "name" : CGI.HTTP_HOST, "hostname" : CGI.SERVER_NAME }, - "client" : { "ip" : CGI.REMOTE_ADDR }, + "client" : { "ip" : util.getRealIp() }, "user" : {}, "user_agent" : { "original" : CGI.HTTP_USER_AGENT } }; diff --git a/models/logging/LogstashAppender.cfc b/models/logging/LogstashAppender.cfc index 57611b9..19cfa71 100644 --- a/models/logging/LogstashAppender.cfc +++ b/models/logging/LogstashAppender.cfc @@ -219,7 +219,7 @@ component "path" : expandPath( "/" ) }, "host" : { "name" : CGI.HTTP_HOST, "hostname" : CGI.SERVER_NAME }, - "client" : { "ip" : CGI.REMOTE_ADDR }, + "client" : { "ip" : util.getRealIp() }, "user" : {}, "user_agent" : { "original" : CGI.HTTP_USER_AGENT } }; diff --git a/models/util/Util.cfc b/models/util/Util.cfc index cd7dc51..67bc003 100644 --- a/models/util/Util.cfc +++ b/models/util/Util.cfc @@ -2,6 +2,7 @@ component accessors="true" singleton { property name="appEnvironment" inject="box:setting:environment"; property name="interceptorService" inject="coldbox:InterceptorService"; + property name="configStruct" inject="box:modulesettings:cbelasticsearch"; /** * Workaround for Adobe 2018 metadata mutation bug with GSON: https://tracker.adobe.com/#/view/CF-4206423 @@ -202,4 +203,25 @@ component accessors="true" singleton { return reReplace( arguments.script, "\n|\r|\t", "", "ALL" ); } + /** + * Get Real IP, by looking at clustered, proxy headers and locally. + * borrowed from cbSecurity + * + * @trustUpstream If true, we check the forwarded headers first, else we don't + */ + string function getRealIP( boolean trustUpstream = configStruct.trustUpstream ){ + // When going through a proxy, the IP can be a delimtied list, thus we take the last one in the list + if ( arguments.trustUpstream ) { + var headers = getHTTPRequestData( false ).headers; + if ( structKeyExists( headers, "x-cluster-client-ip" ) ) { + return trim( listLast( headers[ "x-cluster-client-ip" ] ) ); + } + if ( structKeyExists( headers, "X-Forwarded-For" ) ) { + return trim( listFirst( headers[ "X-Forwarded-For" ] ) ); + } + } + + return len( cgi.remote_addr ) ? trim( listFirst( cgi.remote_addr ) ) : "127.0.0.1"; + } + } diff --git a/server-adobe@2025.json b/server-adobe@2025.json new file mode 100644 index 0000000..ab6c4cd --- /dev/null +++ b/server-adobe@2025.json @@ -0,0 +1,24 @@ +{ + "name":"cbelasticsearch-adobe@2025", + "app":{ + "serverHomeDirectory":".engine/adobe2025", + "cfengine":"adobe@2025" + }, + "web":{ + "http":{ + "port":"60299" + }, + "rewrites":{ + "enable":"true" + }, + "webroot":"test-harness", + "aliases":{ + "/moduleroot/cbelasticsearch":"../", + "/root":"./test-harness" + } + }, + "openBrowser":"false", + "scripts":{ + "onServerInstall":"cfpm install zip,debugger" + } +} diff --git a/server-boxlang@1.json b/server-boxlang@1.json index a6d684d..f5bf583 100644 --- a/server-boxlang@1.json +++ b/server-boxlang@1.json @@ -1,34 +1,34 @@ { - "app":{ - "cfengine":"boxlang@1.0.0-snapshot", - "serverHomeDirectory":".engine/boxlang" + "app": { + "cfengine": "boxlang@1", + "serverHomeDirectory": ".engine/boxlang" + }, + "name": "cbelasticsearch-boxlang@1", + "force": true, + "openBrowser": false, + "web": { + "directoryBrowsing": true, + "http": { + "port": "60299" }, - "name":"cbelasticsearch-boxlang@1", - "force":true, - "openBrowser":false, - "web":{ - "directoryBrowsing":true, - "http":{ - "port":"60299" - }, - "rewrites":{ - "enable":"true" - }, - "webroot":"test-harness", - "aliases":{ - "/moduleroot/cbelasticsearch":"./" - } + "rewrites": { + "enable": "true" }, - "fusionreactor":{ - "enable":"${FR_ENABLE:false}", - "port":"8088", - "licenseKey":"${FR_LICENSE_KEY}" - }, - "JVM":{ - "javaVersion":"openjdk21_jdk_x64", - "args":"-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9999 -Dboxlang.debugMode=true" - }, - "scripts":{ - "onServerInitialInstall":"install bx-compat-cfml@be,bx-esapi --noSave" + "webroot": "test-harness", + "aliases": { + "/moduleroot/cbelasticsearch": "./" } -} \ No newline at end of file + }, + "fusionreactor": { + "enable": "${FR_ENABLE:false}", + "port": "8088", + "licenseKey": "${FR_LICENSE_KEY}" + }, + "JVM": { + "javaVersion": "openjdk21_jdk_x64", + "args": "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9999 -Dboxlang.debugMode=true" + }, + "scripts": { + "onServerInitialInstall": "install bx-compat-cfml,bx-esapi --noSave" + } +} diff --git a/server-boxlang@be.json b/server-boxlang@be.json new file mode 100644 index 0000000..c9221c8 --- /dev/null +++ b/server-boxlang@be.json @@ -0,0 +1,34 @@ +{ + "app": { + "cfengine": "boxlang@be", + "serverHomeDirectory": ".engine/boxlang" + }, + "name": "cbelasticsearch-boxlang@1", + "force": true, + "openBrowser": false, + "web": { + "directoryBrowsing": true, + "http": { + "port": "60299" + }, + "rewrites": { + "enable": "true" + }, + "webroot": "test-harness", + "aliases": { + "/moduleroot/cbelasticsearch": "./" + } + }, + "fusionreactor": { + "enable": "${FR_ENABLE:false}", + "port": "8088", + "licenseKey": "${FR_LICENSE_KEY}" + }, + "JVM": { + "javaVersion": "openjdk21_jdk_x64", + "args": "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9999 -Dboxlang.debugMode=true" + }, + "scripts": { + "onServerInitialInstall": "install bx-compat-cfml,bx-esapi --noSave" + } +} diff --git a/server-lucee@6.json b/server-lucee@6.json index ae894c8..03b14e5 100644 --- a/server-lucee@6.json +++ b/server-lucee@6.json @@ -2,7 +2,7 @@ "name":"cbelasticsearch-lucee@6", "app":{ "serverHomeDirectory":".engine/lucee6", - "cfengine":"lucee@be" + "cfengine":"lucee@6" }, "web":{ "http":{