Show HN: Umbra – An AI agent that uses Android apps while you use your phone

Most AI agent today treats Android phone as something to reach the service than somewhere agent can work. However, Android itself is a perfect environment for consistent agent to run on.

Firstly, I use Android's 24/7 connectivity and low energy cost NPU to run a 24/7 proactive agent which will check all the information and determine whether it should reach the user or not by LAYA.

Secondly, I use Android's emulation ability to run a full Debian environment for the coding agent to run command and use GUI desktop app for development.

Thirdly, I use Android's virtual display and accessibility api to control apps in the background for testing, working, or even ordering food while user is working or chilling in other app without disturbing them. Also, I use accessibility to create a capsule around the camera hole to make sure user can reach the agent and check the agent's working status while using other app. Also the capsule can expend when the agent needs input, approval, announcement etc.

Lastly, the user can tap the capsule and select something on the screen and send to the agent just like Google's Circle to Search and can record screen and accessibility tree at the same time to teach agent how to perform a task.

I'd like to hear your opinion on the architecture, feature, limitations, or anything you want to share.