Error when prompt is too large: The user message is too long. Send a short message or decrease property min_new_tokens

[jwmatthews]

Below is an example of sending a prompt that is too large.
Running against the IBM BAM served meta-llama3

We identified 15 incidents and are including a number of solved incidents as diffs.
I believe that we had a bug in solved incidents that caused it to not render in past, hence this was not discovered until recently.

File: 'src/main/java/com/redhat/coolstore/service/OrderServiceMDB.java'
The full prompt is here (~800+ lines):
https://gist.github.com/jwmatthews/5c9292014520fd6ac58b2dcc035b0627

The logs display:
kai.routes.get_incident_solutions_for_file - [get_incident_solutions_for_file.py:38 - post_get_incident_solutions_for_file()] - START - App: 'coolstore', File: 'src/main/java/com/redhat/coolstore/service/OrderServiceMDB.java' with 15 incidents'

{
  "error": "Bad Request",
  "extensions": {
    "code": "INVALID_INPUT",
    "state": null
  },
  "message": "The user message is too long. Send a short message or decrease property min_new_tokens",
  "status_code": 400
}

cat ../kai/config.toml
# Default configuration file for Kai. For a better understanding of the
# configuration options, please refer to `build/example_config.toml`

log_level = "info"
file_log_level = "debug"
log_dir = "$pwd/logs"
demo_mode = false
trace_enabled = true

solution_consumers = ["diff_only", "llm_summary"]

[incident_store]
solution_detectors = "naive"
solution_producers = "text_only"

[incident_store.args]
provider = "postgresql"
host = "127.0.0.1"
database = "kai"
user = "kai"
password = "dog8code"

[models]
provider = "ChatIBMGenAI"

[models.args]
model_id = "meta-llama/llama-3-70b-instruct"
parameters.max_new_tokens = 2048