This issue collects ideas and research regarding what we can/should add to cuml with respect to user facing agent skills.
There are at least two categories of skills: (1) skills that encode human expert knowledge and walk the user through a task (think tutorial) and (2) a skill that aims to improve an agent's performance at a task (the agent might be able to perform the task without the skill, but with it it uses less tokens, completes faster, goes wrong less often).
An example for (1) would be a skill that helps a user use cuml.accel and shows them that you can use the logs to see what was/wasn't accelerated. This is more about helping a user discover features than making the agent better at a task.
An example of (2) would be a skill that takes an existing data science pipeline and translates it to use cuml. A agent can probably already do this without a skill, so the goal is to make the agent better. An important ingredient for these kinds of skills is having a way to evaluate if the skill really improves the performance of the agent.
An important aspect for both categories is to evaluate a given agent/model with and without the skill. It doesn't make sense to create and maintain a skill that doesn't lead to an improvement.
To get started with figuring out what is what I asked a free Claude.ai account (with no history of working on cuml or scikit-learn etc) "Help me run this script on the GPU" (original script). The idea behind this is that I wanted to see what a "low power" model (sonnet 4.6) on a free account can do with a fairly vague prompt. The result is here. It looks like it did a good job translating it to use cuml, gave some instructions on how to install cuml (could be improved) and generally I am not sure how a skill of type (2) would make this much better. I think a skill of type (1) would be able to give the user some more information/walk through.
This issue collects ideas and research regarding what we can/should add to cuml with respect to user facing agent skills.
There are at least two categories of skills: (1) skills that encode human expert knowledge and walk the user through a task (think tutorial) and (2) a skill that aims to improve an agent's performance at a task (the agent might be able to perform the task without the skill, but with it it uses less tokens, completes faster, goes wrong less often).
An example for (1) would be a skill that helps a user use
cuml.acceland shows them that you can use the logs to see what was/wasn't accelerated. This is more about helping a user discover features than making the agent better at a task.An example of (2) would be a skill that takes an existing data science pipeline and translates it to use cuml. A agent can probably already do this without a skill, so the goal is to make the agent better. An important ingredient for these kinds of skills is having a way to evaluate if the skill really improves the performance of the agent.
An important aspect for both categories is to evaluate a given agent/model with and without the skill. It doesn't make sense to create and maintain a skill that doesn't lead to an improvement.
To get started with figuring out what is what I asked a free Claude.ai account (with no history of working on cuml or scikit-learn etc) "Help me run this script on the GPU" (original script). The idea behind this is that I wanted to see what a "low power" model (sonnet 4.6) on a free account can do with a fairly vague prompt. The result is here. It looks like it did a good job translating it to use cuml, gave some instructions on how to install cuml (could be improved) and generally I am not sure how a skill of type (2) would make this much better. I think a skill of type (1) would be able to give the user some more information/walk through.