"We Have No Idea How Models will Behave in Production until ...

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning.

How ML engineers operationalize machine learning, their workflow stages (data preparation, experimentation, evaluation and deployment, monitoring and response), and the challenges they face in each stage.

arxiv.org

RelatedHighlights

What We Learned From a Year of Building With LLMs

Bryan Bischof oreilly.com

How Humans Outshine AI in Adapting to Change

Rachel Layne hbswk.hbs.edu

Scott Belsky Talk at South Park Commons

Often designs from frustration
Right now, greater skill is being brought by compute and developing a democratization of many things (code, design, etc.). Because of this, taste will probably be the most important skill
Taste is derived from culture and overlap of industries
Because of that

One interesting thing about LLMs is that they can actually recover (and without error loops). You can have a step that doesn't work right, and a later step can use its common-sense knowledge to ignore some of the missing results, conflicting information, etc. One of the problems with developing with LLMs is that the machine will often cover up bugs... See more

What We Learned From a Year of Building With LLMs

How Humans Outshine AI in Adapting to Change

Ask HN: What are some actual use cases of AI Agents right now? | Hacker News