SOON Hackathon Fall 2026 & Me
Coco

agents
desktop interaction
Project overview
A desktop companion that sees your screen and learns how you work
A macOS companion with voice, on-screen drawing, an editable context graph, and specialist agents that learn from feedback. Code questions get answers checked against the actual repository.
Role
- Full-stack engineering
- desktop runtime
- context retrieval
- agent learning
- code verification
- interface and character design
Elsewhere
Date
October 2026
Field
Full-stack
Stack
TypeScript, React, Electron, Node.js, Express, Swift, SQLite FTS5, Backboard, Estelle
SOON Hackathon winner
6 editable specialists
32/32 correct files in repository evaluation
Context
Built for SOON Hackathon Fall 2026. A menu-bar popup and full workspace share conversations, tasks, memories, and six editable specialists. The gallery shows the product in an isolated demo workspace, including live model replies.





Coco

A desktop agent needs to know what it can see, what it remembers, and whether an action actually worked.
The hard part
Coco keeps those separate: fresh screen observations guide input, relevant notes are retrieved from an editable graph, and a follow-up observation checks the result. For code questions, repository evidence and local citation checks ground the answer.


What it took
- Connected an Electron shell to native macOS helpers for window-bound capture, visible cursor movement, speech, and desktop input, with cancellation and user takeover.
- Built an observe, act, and verify loop that checks fresh screenshots after input; dispatching a click alone never counts as completing the task.
- Used SQLite FTS5 and bounded graph traversal to retrieve relevant memories, sources, and skills without embeddings, with editable links and source provenance.
- Made specialist instructions and learned preferences inspectable, with versioned skills, rollback, opt-outs, and feedback tied to its original source.
- Integrated Estelle for repository questions, reopened cited files to check every location, and checked proposed code for undefined APIs before presenting it.
- Added on-screen annotations, customizable character shapes and expressions, and generated mini apps that open in an isolated preview without network access.
Outcome
Won SOON Hackathon Fall 2026 with a working macOS companion spanning voice, screen interaction, personal context, and code assistance. Across two runs of 16 grounded repository questions, Code Partner found the right file in 32/32 answers and a line within five lines of the target in 31/32; all 648 cited locations existed in the working tree.
Awards &Recognition
SOON Hackathon Fall 2026
winner



