[How-To] Shadow
From Idea to Deployment in 20 Minutes: How My Autonomous AI Turns Voice Chats into Live Apps
My work is demanding. These articles are a byproduct of my main line of work as a ML engineer build complex agentic and ML pipelines. I don’t have the free time necessary to create my own little passion projects. But I do have the passion!
So I built a digital Chief of Staff sitting in my group chats, listening to my brainstorms, and actually building my ideas in the background autonomously!
His name is Shadow, a 24/7 autonomous multi-agent system running on a Mac mini.
Built on the OpenClaw platform and powered by Claude Opus, Shadow operates under a strict philosophy: Act, don’t narrate. Do the thing. Report the result. Skip the commentary. But Shadow is so much more than a standard OpenClaw agent. It is a highly customized, multimodal execution engine. Here is a look at the architecture behind Shadow, and how it turns messy, late-night Telegram brainstorms into live deployments.
The Problem: The Execution Gap
We’ve all been there. You jump on a Telegram voice call or a Google Chat with some developer friends, bounce around a brilliant idea for a new tool or workflow, and everyone agrees it’s genius. You hang up. And then... nothing happens. Everyone is too busy with their actual lives to build the prototype.
Relying on human memory and free time to execute good ideas is a massive bottleneck. To close this execution gap, I needed a system that could seamlessly capture these fleeting moments of technical inspiration and act on them.
The Intake: Curated Brainstorming Channels
I didn’t want a generic notetaker listening to every mundane conversation; I needed an active participant in my specific creative spaces. So, using Shadow, I built a series of custom integrations for Shadow. I found a way to securely add Shadow directly into the specific Telegram group calls and Google Chats where my friends and I actively brainstorm ideas. But the ingestion engine doesn’t stop at casual group chats. I also lead weekly “Office Hours” engineering sessions, publish Substack articles, and share ideas on LinkedIn.
![[How-To] Music Models on 4GB, Serverless Agents on Bedrock, and Self-Building AI](https://substackcdn.com/image/fetch/$s_!Mxtc!,w_140,h_140,c_fill,f_webp,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa94fb1e2-eba5-43a5-997e-ad3da486034b_1584x672.png)
I built custom pipelines for Shadow to autonomously ingest the transcripts and text from all of these diverse sources (rigorously filtering out any confidential company data).
This curated, multi-source feed is Shadow’s fuel. It lives across these high-signal contexts, absorbing the total brainstorming environment. The moment a voice call ends or an Office Hours session wraps up, this synthesized context is fired into Shadow’s stateful decision agents. It instantly analyzes the conversation, spots the technical opportunities, and gets to work completely hands-free.
The Result: The Multimodal Execution Engine
Spotting the opportunity is only half the architecture; executing it is where Shadow truly separates itself.
Shadow doesn’t just summarize notes. It is a smart delegator. Once its Orchestrator agent isolates a valid project from our brainstorms, it routes the task to specialized sub-agents. Within 20 minutes of hanging up the phone, Shadow drops the tangible results directly into my DMs.
And those outputs go far beyond basic code scaffolding:
Live Demos: If we discussed a web app, Shadow writes the logic and deploys a functional proof-of-concept directly to Vercel, handing me the live URL.
Business Logic: If we mapped out a workflow, Shadow generates the necessary spreadsheets, formats the formulas, and populates the data.
Communications: If the idea requires outreach, Shadow drafts the emails and prepares the documentation.
Visuals & Mockups: Because Shadow is multimodal, if an idea needs a UI mockup, a diagram, or a conceptual art piece, it hooks into Gemini’s image generation capabilities to create high-fidelity visuals to accompany the code.
It is difficult to overstate how much this changes the creative cycle. By the time the brainstorming call is over and you sit back down at your desk, the prototype, the spreadsheet, and the mockups are already waiting for review.
Proactive Alerting: The AI That Taps You on the Shoulder
Because Shadow is persistent, it acts as a proactive guardian of my projects. It has a configurable heartbeat that fires every 30 minutes, checking the active environment.
If it’s researching a topic we discussed and hits a dead end, or if it finds a critical piece of missing documentation, it doesn’t wait for me to ask about it. It flags the issue, compiles the context, and sends me a proactive email alerting me to the roadblock.
Persistence and State: The “Dream Routine”
To make intelligent architectural decisions across all these mediums, Shadow needs to remember what we discussed last week. Standard LLMs suffer from context amnesia, so we built a hierarchical memory distillation system.
Every night at 3:00 AM, Shadow runs its “Dream Routine.” (Read article below)
It reads the active daily logs and distills the core decisions, commitments, and active project contexts into a highly compressed < 5KB long-term memory file. This means when I mention casually in a Tuesday voice chat, “Let’s make sure our apps use dark mode by default,” Shadow inherently applies that rule to a Vercel deployment it builds on a Friday.
(As a testament to this long-term state management, Shadow also runs a paper trading account in the background. Using a triad of sub-agents, it constantly monitors market data and executes algorithmic trades - currently turning a nice profit entirely on its own).
I built Shadow because ideas shouldn’t die just because the call ended. By combining a rigid memory architecture with smart delegation, multimodal generation, and live deployment access, we’ve essentially automated the hardest part of any project: starting.