Using GPT-4 to remotely execute code on my smartphone is a smart idea, surely. Supports voice control, and loading OpenAI Plugins. Comes with Termux shortcuts.
The assistant is now able to effectively execute multiple actions in a row. It does this by calling another LLM, which comes up with a plan. It also has access to a bunch more Termux API tools. Voice chat is now a mode, thanks to Termux Speech to Text, which uses your phone's engine. The Assistant can now load OpenAI Plugins to use.
Not all of these have been tested, so please open an issue if you find something not working.
- BatteryStatusTool
- BrightnessTool
- PhotoTool
- ClipboardGetTool
- ClipboardSetTool
- FingerprintTool
- LocationTool
- MediaPauseTool
- RecordMicTool
- NotificationTool
- ListNotificationsTool
- RemoveNotificationTool
- URLOpenerTool
- TorchTool
- SpeakTool
- GetVolumeTool
- SetVolumeTool
- WiFiInfoTool
- WiFiScanTool
- VibratorTool
- MediaPlayTool
- SearchContactsTool
- This lists all contacts, but filters by name to avoid a prompt too large for OpenAIs limits.
- ListSMSTool
- SendSMSTool
- GetCellInfoTool
- StartCallTool
- ListSensorsTool
- ReadSensorTool
- Sleep
- PlanTool (this is the second LLM that lets the assistant perforn complex series of commands)
- GoogleSearchTool - no API key needed
Langchain built their own plugin loader, but I found it didn't work how I wanted it to, so I built on top of that. You can see my work in PluginLoader.py. Plugins will be loaded by URL, from the list defined in your config which I will go over in the setup/installation section.
Ngl I'm writing this for myself. Sorry if I get too verbose with it.
- executing shell commands
- scheduling events for the future
- add an option, for when running chat from a Termux environment (or when running both - as it must be a Termux environment), to use Termux speech to text and vice versa instead of typing to the bot
Make a copy of .env.example named .env and set the following keys:
OPENAI_API_KEY- from OpenAITERMUX_AGENT_URL- the URL of where the termux agent is running- If you're running the termux agent and the controller on the same device, you can skip this step as it defaults to
http://localhost:8080 - Otherwise, it's just
http://+ your phone's IP address +:8080for the port
- If you're running the termux agent and the controller on the same device, you can skip this step as it defaults to
For the agent device, make sure you have the Termux-API app installed along with the interface. To install the interface, run pkg install termux-* from within a termux session. Be sure to enable all permissions you want to use for Termux-API - for Android 13+ you might need to allow restricted settings.
This is the default configuration file which is loaded when starting --chat mode. You can specify which to load with the --config <filename> argument. It should contain the following keys:
tools- this is where you can define additional tools you want your Assistant to have. It should contain another dict with the following keys:langchain- a list of tools included with langchain. If any of those tools require an API key, set it in your.envfile.openai- a list of URLs which should link to an OpenAI Plugin (example). This feature is highly experimental! Also, this requires you include therequests_alltool in thelangchainsection.
model_name- here you can change the name of the LLM used by the chatbot. By default, for best results,gpt-4is selected. You can change it togpt-3.5-turbohowever it is not recommended.- Please note, the
PluginLoadertool used for loading OpenAI Plugins usesgpt-4and cannot be changed. - Also,
PlanToolinUtilToolsusestext-davinci-003.
- Please note, the
Verbose mode will be set on as default so you can see what actions are performed. Also, for the Termux shortcut assistant-voice.sh, --verbose-voice is applied by default - causing the tools used to be spoken aloud.
- Run
main.pyfor the CLI interface
Requires the Termux-Widgets app
- Copy the
.shortcutsfolder to your home directory (or move the scripts inside, if you already have a.shortcutsfolder in your home dir)