When Local Isn’t Enough: Robert Lukoszko on Pivoting Away from Desktop Agents

Three days before OpenAI released GPT-4 Vision, Robert Lukoszko was already going viral on Twitter with a computer vision demo that would spark countless imitations. Using the Bakllava model and his laptop’s camera, he created what felt like magic—a computer that could see and understand the real world in real-time.

“I imagine after this one, basically everyone else started to do those demos,” Lukoszko recalls. “And I didn’t realize it at the moment… but after people started to comment, they’re like, oh my god. It can understand the video. It can understand what you’re doing.”

That viral moment launched Lukoszko into Y Combinator and eventually led to a complete pivot away from desktop AI agents to his current company, Stormy AI. His journey offers crucial insights for anyone building AI products about the harsh realities of local models, distribution challenges, and finding the right problem to solve.

The Desktop Agent Dream

Lukoszko’s vision was ambitious: a desktop application that could see everything on your screen, track your activities, and provide contextual AI assistance. Think of it as a comprehensive “diary” of your digital life that an AI could reference to give you genuinely helpful responses.

“The whole problem with all the many tools that summarize emails and help you write a reply is because they don’t have the context,” he explains. “Even if they can get it to sound like me, which they usually don’t, it’s the wrong answer. But if they had the context, it could also be the right answer.”

The technical implementation was sophisticated. His app tracked window focus, captured screenshots, analyzed content, and even experimented with mouse movement patterns to predict user intent. He built computer control features that could click buttons and provide visual guidance for complex software like Final Cut Pro.

But despite the technical achievement and initial traction, Lukoszko eventually realized he was fighting an unwinnable battle.

The Local Model Reality Check

His first major realization came around local models. Initially drawn to the privacy and offline capabilities of running AI locally, Lukoszko surveyed users and explored the business case extensively. The conclusion was sobering. “Local models, they’re by definition free kind of,” he says. “The question is, will you make an amazing business use case out of it? And to be honest, it didn’t work as I expected.”

The breakthrough insight came when he considered Microsoft’s Azure offering: “Azure allows you to really safely deploy GPT-4 on their servers, and they allow all the guarantees. Once you think about this… I can trust Azure because this is Microsoft—government uses them, military uses them. I can also—it’s kind of the same trustworthiness as the local models. The only difference is that local models are free.”

But free wasn’t enough when weighed against performance: “Do I want to use extremely powerful model running in the cloud which I can 100% trust… Or should I run the model which is kind of doesn’t have good context, handicapped? And the answer is that you always want to go with more.”

The Distribution Steamroller

Even more damaging than the local model limitations was the distribution reality. When OpenAI began rolling out features similar to what Lukoszko was building, the writing was on the wall.

“OpenAI has insane distribution,” he realized. “If they cover just 80% of all your use cases, your customers are gonna ask you how are you different. And if you’re gonna ask you how are you different, and you’re just 20% better than OpenAI, well, I pay for OpenAI subscription. Will you pay for another $20 subscription? I have doubts.”

This wasn’t just theoretical. Lukoszko watched competitor traffic decline as OpenAI expanded their offerings. “I believe every other competitor right now, they have their traffic just goes down,” he observed.

The Technical Reality of OS Integration

Beyond business challenges, the technical overhead of deep OS integration proved more complex than anticipated. Building on macOS required reverse-engineering Apple’s internal systems and dealing with significant limitations. “The whole pain point of building on macOS—you have to figure out stuff Apple built for themselves in order to use it because they don’t want you to use it,” Lukoszko explains. Simple features like recording both microphone and computer audio required weeks of engineering work with no clear solution.

While he successfully implemented features like NSPanel windows that don’t steal focus (similar to Raycast), each advancement required substantial low-level engineering effort that didn’t scale well for a small team.

The Pivot to Stormy AI

Recognizing these challenges, Lukoszko made a strategic pivot to solve a problem he experienced firsthand: the painful manual work of influencer marketing. His new company, Stormy AI, automates the entire process of finding niche influencers, analyzing their content, locating contact information, and managing outreach campaigns.

“I was doing my previous product, and I figured that the really painful part of it is distribution,” he says. “I was like, what if I can automate this? Because I can’t believe it’s 2025, and this is actually the go-to way to solve this.”

The technical approach leverages his AI expertise differently—crawling and caching hundreds of thousands of influencer profiles, using AI to analyze content and match criteria, and automating the traditionally manual process of influencer discovery and outreach.

Lessons for AI Builders

Lukoszko’s journey offers several crucial insights for anyone building AI products:

• Technology alone isn’t enough for B2C success. “If you’re building something B2C, it’s rarely the question about the technology,” he notes. “It’s more about the question: do you actually know how to distribute this? Do you actually understand your people?”
• Local models face fundamental business challenges. Despite their appeal for privacy and tinkering, the performance gap with cloud models creates difficult trade-offs for serious applications.
• Distribution moats matter more than technical moats. When competing with companies that have massive existing user bases, being technically superior often isn’t sufficient.
• Find real business problems to solve. Lukoszko’s pivot succeeded because he experienced the pain point personally and could validate the market need.
• Marry technical skills with business hustle. “When you marry those two things, the tinkering and that, that’s where you have a really special opportunity,” he reflects.

The Future of Ambient AI

Despite stepping away from desktop agents, Lukoszko still believes in the long-term vision of ambient AI that understands context across all your activities. He points to companies like Glean and emerging hardware like AR glasses as potentially better approaches to the same problem.

“I feel like there is definitely the future” for contextual AI, he says, noting that the silo problem—where AI tools lack comprehensive context—remains unsolved.

For now, though, he’s focused on building a venture-scale business by automating a proven market rather than creating a new category. It’s a pragmatic choice that reflects hard-won wisdom about the difference between impressive technology and sustainable business models.

Lukoszko’s story serves as both inspiration and cautionary tale for AI builders: sometimes the most important pivot is from what’s technically impressive to what actually works in the market.

Robert Lukoszko is CEO of Stormy AI and a Y Combinator alum (S24). You can follow his work on X @Karmedge.