cv

Statler: State-Maintaining Language Models for Embodied Reasoning

Takuma Yoneda*, Jiading Fang*, Peng Li*, Huanyu Zhang*, Tianchong Jiang, Shengjie Lin, Ben Picker, David Yunis, Hongyuan Mei, Matthew R. Walter

IEEE International Conference on Robotics and Automation

Transcrib3D: 3D Referring Expression Resolution through Large Language Models

Jiading Fang*, Xiangshan Tan*, Shengjie Lin*, Hongyuan Mei, Matthew R. Walter

CoRL Workshop on Language and Robot Learning

NeRFuser: Scalable Scene Representation by NeRF Registration and Blending

Jiading Fang*, Shengjie Lin*, Igor Vasiljevic, Vitor Guizilini, Rares Ambrus, Adrien Gaidon, Gregory Shakhnarovich, Matthew R. Walter

ICLR Workshop on Neural Fields across Fields

Baxter Pose Following

Empower the Baxter robot to follow a human's pose. The project gained great popularity upon its debute at the Museum of Science and Industry in Chicago and has since been featured at multiple public events.

• Use RTMPose-m for SOTA real-time pose detection.
• Implement a robust tracking algorithm to focus on a highlighted pose over time.
• Control the Baxter robot using joint angles computed according to its kinematics.
• Employ threading-based parallel execution to achieve responsive performance.

Code as Policies on UR5

Implementation of Code as Policies on the UR5 robot. The project was presented during the 2023 National Robotics Week at the Museum of Science and Industry in Chicago.

• Robust speech-to-text as input aided by wake-up-word mechanism and dynamic ambient sound adaptation.
• Flexible robot action as output via on-the-fly GPT-powered code generation.
• Open-vocabulary pick & place enabled by MDETR for visual perception.
• Pertinent grasping operation based on object point-cloud analysis.

Infant

Artistic demonstration of a digital infant's reaction to external stimuli. The installation was showcased in the 2020 SAIC Shows.

• The digital infant interacts with the audience/environment via multi-modal perception, including vision, audio, touch and pulse sensing.
• Reflection of external stimuli on the infant is designed with philosophical and realistic considerations.
• Powered by neural style transfer, the infant's state is visualized via the gradation of art style in its skin texture.

Baxter Rubik's Cube

Solving a Rubik's Cube with the Baxter robot. The project was presented during the 2019 National Robotics Week at the Museum of Science and Industry in Chicago.

• The robot picks up and scans the 6 faces of an arbitrarily initialized Rubik's Cube following programmed routine.
• A clustering-based technique is used to robustly determine the grid of colors on each face.
• Generated by Kociemba's Algorithm, the cube solution gets accurately executed via visual servoing.
• The current solution step and the cube's real-time state is visualized in 3D graphics.

Ph.D. Candidate

Toyota Technological Institute at Chicago

Computer Science

M.Sc.

Toyota Technological Institute at Chicago

Computer Science

Graduate coursework

• TTIC 31010: Algorithms
• TTIC 31070: Convex Optimization
• TTIC 31230: Fundamentals of Deep Learning
• TTIC 31040: Introduction to Computer Vision
• CMSC 30600: Introduction to Robotics
• TTIC 31020: Introduction to Statistical Machine Learning
• TTIC 31150: Mathematical Toolkit
• TTIC 31170: Planning, Learning and Estimation for Robotics and Artificial Intelligence
• TTIC 31180: Probabilistic Graphical Models
• TTIC 31240: Self-driving Vehicles: Models and Algorithms for Autonomy

B.Sc.

Tsinghua University

Electronic Engineering

Awards

• Outstanding Undergraduate at EE Dept.
• Outstanding Thesis at EE Dept.
• 2014--2016 Scholarship for Overall Excellence
• 2015 Scholarship for Technological Innovation

Teaching Assistant

TTIC 31170: Planning, Learning and Estimation for Robotics and Artificial Intelligence