A Self-Learning Cyber-Physical Robotic System Powered by LLM
We will initiate the development of an affordable teleoperated humanoid robot by utilizing a 6-DoF robotic arm, focusing on achieving high dexterity and precision while keeping costs low. An accessible control system will be created to facilitate seamless teleoperation over standard internet connections, featuring an intuitive user interface that supports various input methods.
To enhance operator control, we will incorporate advanced feedback mechanisms, including real-time sensory data from the robot’s articulated wrists and fingers. The robot will be equipped with multiple cameras: an end-effector camera for close-up views, a top camera for an overhead perspective, a bottom camera to monitor ground-level interactions, and an orbital camera mounted on the head to simulate the operator’s viewpoint through adjustable pitch, yaw, and roll. These cameras will support transformer-based learning models, allowing the robot to refine its autonomous capabilities and improve task execution. The orbital camera, in particular, will be crucial for providing a dynamic perspective that aligns with the operator’s viewpoint, enhancing control precision and reducing task execution errors. The project will include comprehensive design, prototyping, and testing phases to ensure the robot’s effectiveness in precise tasks such as object manipulation.
Training a transformer encoder algorithm to convert camera and kinematics inputs into a sequence of actions involves a structured approach that integrates both data preparation and model training phases. Initially, the process begins with the collection of episodic data that includes camera inputs, such as images or video frames, and kinematics data, which encompasses the positional and movement information of the robot’s arms, wrists, and fingers. Each episode is meticulously annotated with the sequence of actions required to complete the task. This raw data undergoes preprocessing where camera inputs are processed through feature extraction methods—such as convolutional neural networks (CNNs) or vision transformers—to derive embeddings that encapsulate essential visual features.
Objectives
Our definition of success is centered on creating a self-learning infrastructure supported by an affordable humanoid robot, specifically one with an orbital head and two 6-DoF arms, each equipped with a wrist and three fingers.
Teleoperated-Robot - Success in this project will be defined by the teleoperated robot’s ability to perform fine maneuvering of self-learnable dexterous tasks with exceptional precision and accuracy. This includes the robot’s capability to handle delicate operations such as object manipulation and complex assembly with minimal deviation, facilitated by advanced real-time feedback mechanisms. The robot’s manipulator must exhibit high dexterity and adaptability, demonstrating flexibility across various tasks and environments. Safety and reliability will be measured by the implementation of robust fail-safe mechanisms and consistent performance under diverse conditions. Additionally, success will be evaluated based on high task completion rates and positive operator feedback, confirming that the system effectively meets practical needs and excels in fine manipulation tasks in real-world applications.
Immersive-Feedback - Success in using an orbital camera for teleoperated robots will be defined by the operator’s ability to achieve enhanced situational awareness and precise control through a dynamic, 3D perspective of the robot’s environment. The orbital camera must allow the operator to seamlessly rotate around the robot, adjust viewing angles, and zoom in or out, providing a comprehensive understanding of the surroundings. This perspective should enable the operator to effectively navigate and manipulate the robot in relation to obstacles, target objects, and other environmental features, even in complex or hazardous settings where direct line-of-sight is not possible. Success is ultimately measured by the operator’s ability to use the orbital camera interface to execute tasks with precision and confidence, ensuring safe and efficient robot operations.
Self-Learning - Robotic learning will be defined by the seamless integration of advanced computational re- sources, efficient data pipelines, and continuous model optimization. Data management success is marked by the real-time collection, preprocessing, and versioning of data, enabling models to train on the most relevant and up-to-date information. Transformer models, customized for specific robotic tasks, are successfully optimized through distributed training and hyperparameter tuning within simulation environments. A key indicator of success is the smooth operations of continuous integration and deployment pipelines, allowing for timely updates of models in the field. Additionally, success is measured by the effective implementation of feedback loops and online learning mechanisms, where robots continuously refine their models based on real-world in- teractions. Monitoring and logging systems should successfully enable performance evaluation, drift detection, and model improvement, with centralized logging offering actionable insights. Ultimately, success is achieved when the infrastructure securely, compliantly, and scalably supports autonomous adaptation, allowing robots to deploy effectively and improve continuously in dynamic environments, thereby advancing the potential of autonomous robot
Motivations
The resurgence of electronics testing and manufacturing in the United States can be significantly bolstered by the integration of remote robotics. This technology offers enhanced efficiency, cost-effectiveness, and quality control by automating complex tasks and enabling real-time monitoring. Remote operation capabilities reduce labor costs and facilitate flexible manufacturing, making domestic production more competitive. Additionally, robotics supports reshoring efforts by strengthening supply chain resilience and enabling localized, customizable production. The adoption of robotics also fosters workforce development through upskilling and remote collaboration, while promoting sustainability through energy efficiency and waste reduction. By harnessing the power of remote robotics, the U.S. can revitalize its electronics manufacturing sector, driving innovation and economic growth.
Qualifications
Minimum Qualifications:
- Python and PyTorch proficiency
- ML level course
- Prompt Engineering (team 1,2)
- Transformers experience for time series learning (team 1,2)
- Robotics Experience (team 1,2)
- ML-OPS (team 3)
- CI/CD (team 3)
- Cloud backend development experience (team 3)
- Mechanical CAD (like Fusion, etc.) - optional
Details
Project Partner:
Rahul Khanna
NDA/IPA:No Agreement Required
Number Groups:3
Project Status:Accepting Applicants
Keywords:PythonOperations / CloudData ScienceMachine Learning (ML)RoboticsConsultancy
