feat: add progress monitoring and logging #132
Conversation
kushalthaman
force-pushed
the
feat/progress-monitor-and-logging
branch
from
da76445 to
484ab9a
Compare
| self._bars[model_id] = (req_bar, None, None) | ||
| self._next_position += 1 | ||
|
|
||
| async def update_openai_usage( |
There was a problem hiding this comment.
its used by other providers too so maybe don't call openai_usage?
| self.model_wait_times.setdefault(response.model_id, []).append(response.duration - response.api_duration) | ||
|
|
||
| # Update progress monitor with usage info | ||
| if hasattr(self, "progress_monitor") and self.progress_monitor is not None: |
There was a problem hiding this comment.
what is the reason more of this can't go in the class and happen when you run self.progress_monitor.update_openai_usage in each model class?
There was a problem hiding this comment.
Good catch, It’s centralized now, and the progress update happens only in InferenceAPI.call
safetytooling/apis/inference/api.py
Outdated
| if isinstance(model_class, AnthropicChatModel) or isinstance(model_class, HuggingFaceModel) or isinstance(model_class, GeminiModel) or isinstance(model_class, GeminiVertexAIModel): | ||
| request_increment = num_candidates | ||
|
|
||
| await self.progress_monitor.update_openai_usage( |
There was a problem hiding this comment.
yeah I'm confused why this is called again when you call it in each model class
|
This is awesome work! Tysm for implementing this |
|
Can you post a screenshot of what it looks like in the terminal? |
let me know if you have any cosmetic suggestions!
we x-ratelimit-limit-tokens (TPM) and x-ratelimit-limit-requests (RPM) on an initial header probe, and (intentionally) run under the cap by openai_fraction_rate_limit (defaults to 0.8). Also we seed “consumed” from x-ratelimit-remaining-* so the bars reflect the live window state. |
|
@jplhughes bumping this |
2 participants
No description provided.