OpenAdapt

Overview

OpenAdapt is an open-source AI-first process automation system that learns to automate tasks in desktop and web applications by observing human demonstrations. It captures user interactions, including mouse, keyboard, and UI element data, and uses Visual Language Models to automate workflows, emphasizing privacy-sensitive data handling. Its goal is to reduce repetitive tasks, offering a simpler user interface for knowledge workers.

Use Cases

• Freight/logistics
• Insurance underwriting + loss adjusting
• Affiliate Marketing
• Paralegal

Key Features

• Records screenshots and associated user input.
• Aggregates and visualizes user input and recordings.
• Converts screenshots and user input into tokenized formats.
• Generates and replays synthetic input via transformer models.
• Creates process graphs from recording logs.
• Supports all desktop GUIs, including virtualized and web.
• Provides state-of-the-art GUI understanding with Segment Anything.