"We Have No Idea How Models will Behave in Production until ...

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning.

How ML engineers operationalize machine learning, their workflow stages (data preparation, experimentation, evaluation and deployment, monitoring and response), and the challenges they face in each stage.

arxiv.org

RelatedHighlights

the problem seems to come…

Some highlights have been hidden or truncated due to export limits.

Art Kleiner • The Fifth Discipline Fieldbook: Strategies for Building a Learning Organization

An O’Reilly survey found that those with mature AI practices (as measured by how long they’ve had models in production) cited a “lack of data or data quality issues” as the main bottleneck holding back further ML adoption.

Astasia Myers • 4 Data Trends to Watch in 2020

The problem I've found with all of them is it is difficult to get consistent solid results that you would want to build around. Even listening to podcasts this seems to be a theme with the people at the forefront of MAS LLMs.

r/AI_Agents - Reddit

Scott Belsky Talk at South Park Commons

Often designs from frustration
Right now, greater skill is being brought by compute and developing a democratization of many things (code, design, etc.). Because of this, taste will probably be the most important skill
Taste is derived from culture and overlap of industries
Because of that