This quarter I hope to try and get our lab identity organized around rigorous, impactful AI research. To that end, I’d like to meet weekly, as a group, to work on a joint set of experiments involving fine-tuning llama2 for particular tasks of interest. The goal is to build some institutional expertise on working with advanced models that we can actually control, as opposed to being limited to just observing the behavior of opaque, ever-changing industry LLMs.
tl;dr: Please try to make it Friday at 11 am in the Allen conference room for a kickoff and more info! See calendar invite incoming!
About a year ago, my belief, like a lot of people’s, was that we were in trouble. big, for-profit LLMs were so much better than everything else that the only way to be relevant was to study few shot learning and prompt engineering against the major deployments — there was no hope of doing anything in the lab that could compete.
But in the last 6-8 months, there is some evidence that smaller models, trained with some tricks (like LoRa, quantization, carefully curated training data, etc.) could compete with big, opaque LLMs.
We want to test this premise ourselves, on problems that we care about, making contributions with both technical and social maturity.
My hope is to run this as a lab meeting with more concrete goals — a kind of unsanctioned “Directed Research Group.”
By the end of the quarter, we hope to have:
- a better understanding of what is and is not possible at “lab * scale” with these models, and how they compete with industry LLMs.
- publishable/shareable results on specific tasks of interest
- ideas for our own individual research agendas in this space (“Hmmm, given these results, maybe we can train a llama model to do X”)
- a better sense of a lab identity — our shared interests and shared expertise, along with more obvious opportunities for collaboration.
- Meet Friday for a kickoff and to set an agenda
- Develop a GitHub repo with our code and results and data for experiments
- Meet weekly to share findings discuss
- Produce some writing about our findings and research questions going forward.
- Develop an online identity for our lab and shared interests (i.e., a lab website)
- Selfishly, I hope to also produce some material for assignments for courses.