Building and running AI models for the enterprise is a completely different game than building them and running them for everyone else. These customers have tons and tons of data, and transformer architectures struggle to scale with massive context windows. Plus, they have airtight requirements for security, reliability, and consistency. It’s…hard.
Nobody knows more about AI in the enterprise than Ori Goshen. He’s the Co-Founder and Co-CEO of AI21, a team of researchers and engineers building AI models for the enterprise. They’ve quickly scaled up, and most recently closed a $208M Series C. I sat down with Ori to talk about what AI in the enterprise is like, from new model requirements to how to hire the right team to build them.
“There’s something we internally call “product algorithm fit.” There are a lot of great algorithmic capabilities out there today, but now you need to find the right way to manifest them and make a product that utilizes them in a way that’s useful for customers.”
“Back then, there was basically no talent. AI was a new field. Today you need a combination of people with algorithmic understanding with a theoretical background, but who also are good engineers. A lot of the problems with these very large training workloads are around system engineering: distributed systems and computing, running large clusters, things tend to break, etc. A typical researcher out of academia doesn’t spend time on this stuff.”
“Most LLM architectures out there are based on transformers, which are quadratic complexity at inference time: the more you scale the context window, the more computationally expensive it is. This isn’t very scalable as you get up to hundreds of thousands of input tokens. In the paper about our Mamba architecture, our researchers found that there are certain types of tasks where you can get similar levels of quality, but much more efficiently.”
“Things move super fast. If I’d need to describe it, I’d say 2023 (post-ChatGPT) was the year of sporadic experimentation, 2024 was massive experimentation, and 2025 is when we’re going to production, roughly speaking. Enterprises haven’t really scratched the surface of adoption of these models and systems in production. A typical enterprise might have tens of use cases in prod, but there are literally thousands on the backlog.”
.svg)