Why local AI’s the way forward, and the best way period

As generative AI for development expands and becomes more commodified, it's also looking more and more like local models, not cloud hosted ones are the way forward.

Now this could mean anything from hosting a model one's own server for one's own developers, to running them on your own development notebook or desktop.

The point is, you host it, you control it, you know exactly what it is, and there's a whole host of reasons why this is shaping up to be the path, and in fact, I'm willing to bet this is inevitable.

First, we've seen a huge sea change just over the last year alone in how much easier it is to host and run models locally.

Multiple, open source and free to use projects like LM Studio and Lemonade have streamlined the process of downloading models, standing them up and tweaking their settings so they run well on consumer grade hardware.

You still have to understand how these things work and know what at least some of the knobs do when they're tweaked.

But the majority of the hard work can be done automatically, and once you give people more ways to put the power into their own hands, the more they're going to want to do that. Second, smaller and more efficient models are coming out all the time.

I recently took a Look at Google's Gemma 4 model, which comes in a variety of quantizations, different sizes that sacrifice some capability for compactness.

One of them was small enough to run on my desktop's GPU with only 8 gigabytes of VRAM, and it delivered pretty good results for local tasks like chatting about code or generating inline suggestions.

Developer Vicky Boikus talked in a blog post recently about the good results they've had on a 2022 M2 Mac with 64 gigabytes of RAM. So my feeling is we've only begun to explore how compact and efficient local models can become.

There's a lot of Runway here, and we've barely gotten started running down it. Third, we're going to want to leverage local existing resources as much as possible to keep costs down, because hardware prices, especially memory, are exploding and they're likely to stay inflated for years to come.

And this is because of the rush to mob up memory and storage production, not even existing inventory, but future production, for the sake of building data centers to run AI models.

By the time hardware prices return to something like normal, if they ever do, we could be in a place where local open models have become the default rather than the exception, and where cloud based models are seen as a predatory cost trap.

I would argue the blinders are already off here. We've seen explosive jumps in token costs and we've seen the way models can be taken offline arbitrarily.

And these are all forcing a hard rethink of how dependent we want to be on this black box and service based model for these tools.

Over the next couple of years we're going to see multiple conditions accelerate that will make local models the more favorable way to use AI based tooling.

The costs of everything, the growing skepticism about having these tools offered to us as black boxes, and the bigger debates about what they're really useful for and why all of these things are going to become incredibly important.

And the more power that we can take for ourselves on our own hardware with this tooling, the better off we're all going to be. And at the rate we're going, we might not have a choice. So that's it for this episode.

If you liked it, leave a comment below. And don't forget to follow DEVwithSirdar and InfoWorld on Facebook, YouTube and InfoWorld.com.