/work/coco
AmanShah
Brief
amanashishshah@gmail.com

© 2026 Aman Shah

Recruiter mode

SOON Hackathon Fall 2026 & Me

Coco, at a glance

agents

desktop interaction

Project overview

A desktop companion that sees your screen and learns how you work

A macOS companion with voice, on-screen drawing, an editable context graph, and specialist agents that learn from feedback. Code questions get answers checked against the actual repository.

Role

  • Full-stack engineering
  • desktop runtime
  • context retrieval
  • agent learning
  • code verification
  • interface and character design

Elsewhere

Date

October 2026

Field

Full-stack

Stack

TypeScript, React, Electron, Node.js, Express, Swift, SQLite FTS5, Backboard, Estelle

SOON Hackathon winner

6 editable specialists

32/32 correct files in repository evaluation

Context

Built for SOON Hackathon Fall 2026. A menu-bar popup and full workspace share conversations, tasks, memories, and six editable specialists. The gallery shows the product in an isolated demo workspace, including live model replies.

Coco, image 1
Coco, image 2
Coco, image 3
Coco, image 4
Coco, image 5

Coco

Coco, image 1

A desktop agent needs to know what it can see, what it remembers, and whether an action actually worked.

The hard part

Coco keeps those separate: fresh screen observations guide input, relevant notes are retrieved from an editable graph, and a follow-up observation checks the result. For code questions, repository evidence and local citation checks ground the answer.

Coco, image 3
Coco, image 3

What it took

  • Connected an Electron shell to native macOS helpers for window-bound capture, visible cursor movement, speech, and desktop input, with cancellation and user takeover.
  • Built an observe, act, and verify loop that checks fresh screenshots after input; dispatching a click alone never counts as completing the task.
  • Used SQLite FTS5 and bounded graph traversal to retrieve relevant memories, sources, and skills without embeddings, with editable links and source provenance.
  • Made specialist instructions and learned preferences inspectable, with versioned skills, rollback, opt-outs, and feedback tied to its original source.
  • Integrated Estelle for repository questions, reopened cited files to check every location, and checked proposed code for undefined APIs before presenting it.
  • Added on-screen annotations, customizable character shapes and expressions, and generated mini apps that open in an isolated preview without network access.

Outcome

Won SOON Hackathon Fall 2026 with a working macOS companion spanning voice, screen interaction, personal context, and code assistance. Across two runs of 16 grounded repository questions, Code Partner found the right file in 32/32 answers and a line within five lines of the target in 31/32; all 648 cited locations existed in the working tree.

Awards &Recognition

SOON Hackathon Fall 2026

winner