What Builders Talk About When They Talk About AI

What Builders Talk About When They Talk About AI | Andreessen Horowitz

RelatedHighlights

Getting models into the hands of users will help us discover new use cases

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

If you made a thousand versions of an LLM, that’s good at a thousand different things, and you have to load each of those into the GPUs and serve them, it becomes very expensive. The big holy grail right now that everybody’s looking for is: are there techniques, where you can just do small modifications where you can get really good results? There

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

PEFT in a nutshell.

It will make things way better if you can just dump in massive amounts of information. It should be able to know a billion things about you. The HBM bandwidth is there.

—Noam Shazeer, Character.AI

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

I actually disagree with this. I think a bunch of further improvement will come by figuring out better ways to finding important information in databases (instead of just simarlity search) and figuring out how to extract relevant information from conversations, documents, etc. so we can provide only relevant information.

Powerful AI systems can help us interpret the neurons of weaker AI systems. And those interpretability insights often tell us a bit about how models work. And when they tell us how models work, they often suggest ways that those models could be better or more efficient. —Dario Amodei, Anthropic

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

It's a little bit of a conundrum. A model we do not understand explains another model we do not understand.

If you have to be correct and you’ve got a very long and fat tail of use cases, either you do all of the work technically or you hire people. Often, we hire people. That’s a variable cost. Second, because the tails of the solutions tend to be so long—think something like self-driving where there are so many exceptions that could possibly happen—the... See more

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

My basic view is that inference will not get that much more expensive. The basic logic of the scaling laws is that if you increase compute by a factor of n, you need to increase data by a factor of the square root of n and the size of the model by a factor of square root of n. That square root basically means that the model itself does not get that... See more

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

I have a Turing test question for AI: if we took AI in 1633 and trained on all the available information at that time, would it predict that the Earth or the sun is the center of the solar system—even though 99.9% of the information is saying the Earth is the center of the solar system? I think 5 years is right at the fringe, but if we were to run ... See more

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

There was a study out there that did this "time-line" training.

Sometimes they just need the model that actually fits for their specific use case and it’s far more economical. We want people to build on top of our models and we want to give them tools to make that easy. We want to give them more and more access and control, so you can bring your data and customize these models. And you can really focus on the l... See more

Sarah Wang • What Builders Talk About When They Talk About AI | Andreessen Horowitz

I personally believe when it comes to any creative asset or work automation, the demand is elastic. The more that we make, the more people consume. We’re very much looking forward to a massive expansion in productivity, a lot of new jobs, a lot of new things, just like we saw with the microchip and the internet.

—Martin Casado, a16z