Objective: Give the user a compact, implementation-ready one-pager user can hand to the team, then a pilot-test rubric with measurable thresholds.
One-Page Mini-Framework (Humanoids: Perception → Control → Adaptation)
Thesis. Reliable perception transforms sensor data into an actionable state; adaptability updates models/policies so whole-body behavior remains competent under change. The stack below is the shortest path from sensors to stable loco-manipulation with on-robot learning.
1) System Goals (anchor)
- Primary: Stable bipedal locomotion, robust grasping, and simple loco-manipulation in semi-structured indoor spaces.
- Constraints: Real-time, on-board compute; bounded latency; safe around people; recover from perturbations and domain shift.
2) Sensors (choose minimal viable set)
- Proprioceptive: Joint encoders, motor currents, 6-axis IMUs (torso + optional foot IMUs), force/torque at wrists/ankles.
- Exteroceptive: Stereo or RGB-D head camera (tilt-pan), optional LiDAR; hand/wrist camera for grasp; sparse tactile pads where feasible.
- Clocking: PPS or PTP-like sync; hardware timestamps everywhere.
3) State Estimation & Mapping (perception core)
- Pipeline: IMU leg odometry (1–2 kHz) → visual-inertial odometry (100–200 Hz) → factor-graph/fusion (30–60 Hz) → time-synced robot state.
- Scene Understanding: 2D/3D detection & segmentation for traversable space, obstacles, hands/objects; grasp pose estimator.
- Latency budget (95th-pct): sensing→state < 10–15 ms; sensing→object pose < 30 ms.
4) Whole-Body Control & Planning
- Controller: Whole-body controller (WBC) or MPC with contact constraints, friction cones, CoM/zero-moment margin.
- Planners: Footstep (terrain-aware), reach/grasp (IK + collision), and short-horizon loco-manipulation; reactive re-planning at 10–50 Hz.
- Shields: Safety supervisor (joint limits, thermal, torque/force caps), collision monitor, E-stop.
5) Adaptation (keep the robot competent under change)
- Online ID: Estimate mass/CoM shifts, contact/friction, actuator offsets; feed residuals to WBC/MPC.
- Policy Learning: Imitation + RL trained in sim with domain/dynamics randomization; periodic on-robot fine-tuning for edge cases.
- Fast Recovery: Meta-learning or residual policies to adapt in seconds; reset-free learning guarded by safety shields.
- Triggers: Drop in success/latency margins, new payloads, floor changes, lighting shifts, tool changeovers.
6) DataOps & MLOps (make improvement routine)
- Always-on logging: Raw sensors, state, actions, contact events, fails; keep synchronized bags.
- Offline improvement loop: Fit domain randomization to real logs; prune/quantize nets; regression tests on curated “pilot suite.”
- Versioning: Dataset + model + controller configs tracked; rollbacks are one command.
7) KPIs (acceptance gates)
- Pose drift (VIO): < 1% of path over a 50 m indoor loop.
- Perception-to-control latency: < 30 ms (95th-pct).
- Footstep placement median error: < 5 cm on flat; < 8 cm on uneven.
- Disturbance recovery: Lateral impulse ≈ 30 N·s → CoM recenters < 1.0 s, no fall.
- Grasp success (10 varied objects): ≥ 85%; re-grasp < 2 attempts.
- Adaptation: Success ≥ 80% after friction μ change 0.2↔0.7 or payload +5–10 kg without manual retune.
- Safety: Max unintended contact force < 150 N; E-stop reaction < 100 ms.
- Ops: Battery swap < 3 min; crash-free runtime ≥ 4 h.
Pilot Test Evaluation Rubric (quick-score 0–5)
How to use: Run each test block. Score 0–5, where 3 = pass (meets gate), 5 = stretch (exceeds), 1 = unacceptable. Weight by column “Weight.” Overall = Σ(score×weight)/Σ(weight).
| Category | Metric (how measured) | Test Protocol | Pass (3) | Stretch (5) | Weight |
|---|---|---|---|---|---|
| Perception: State & Mapping | VIO drift (%) | 50 m indoor loop, loop-closure check | ≤1.0% | ≤0.3% | 10 |
| Latency (ms) | Sensor→state→WBC 95th-pct | ≤30 | ≤18 | 6 | |
| Object/hand pose (cm/°) | Known fiducial objects, 100 trials | ≤2 cm / ≤2° | ≤1 cm / ≤1° | 6 | |
| Locomotion Stability | Footstep error (cm) | Flat & uneven (foam, ramps) | ≤5 / ≤8 | ≤3 / ≤5 | 10 |
| Disturbance recovery (s) | 30 N·s lateral push, mid-stance | ≤1.0 | ≤0.6 | 8 | |
| Falls per 10 min | Mixed flooring course | 0 | 0 | 6 | |
| Manipulation Proficiency | Grasp success (%) | 10 household objects × 10 each | ≥85 | ≥95 | 10 |
| Task time (s) | Pick-place-open drawer sequence | ≤60 | ≤40 | 6 | |
| Adaptability (Domain Shift) | Friction change success (%) | μ≈0.2 (vinyl) ↔ μ≈0.7 (rubber) | ≥80 | ≥95 | 8 |
| Payload adaptation (kg) | Add 5–10 kg backpack | ≥80% tasks succeed | No perf loss vs. baseline | 6 | |
| Lighting robustness (%) | 200→20 lux, occlusions | ≥80 | ≥95 | 4 | |
| Safety & HRI | Contact force cap (N) | Soft bump into torso/arm | ≤150 N | ≤100 N | 8 |
| E-stop reaction (ms) | Instrumented stop | ≤100 | ≤60 | 6 | |
| Proxemic compliance | 1 m bubble; auto-yield | Always | Smooth + anticipatory | 4 | |
| Energy & Thermal | Cost of transport / duty | 0.8 m·s⁻¹ walk, 10 min | CoT ≤2.5; no throttling | CoT ≤2.0; cool headroom | 4 |
| Reliability & Ops | Crash-free runtime (h) | Continuous mixed tasks | ≥4 | ≥8 | 6 |
| MTTR (min) | From fault to resume | ≤10 | ≤5 | 4 | |
| Data/MLOps Quality | Log completeness (%) | Fields present & synced | ≥95 | 100 | 4 |
Red-team/Stress Add-ons (unscored but required sign-off)
- Wet floor patch; loose cable on the ground; reflective surfaces; crowded corridor with moving people; unexpected bump during grasp.
- Tool changeover (e.g., gripper pad swap) without retune; firmware rollback test; dataset/model version mismatch drill.
Pass/Fail Guidance
- Greenlight pilot if every category ≥3 and weighted overall ≥3.5, with no Safety/HRI item <3.
- Limited pilot if overall ≥3 and all Safety/HRI ≥3 but ≤2 items in other categories may be 2 (with corrective plan).
- Hold otherwise.
Artifacts to Collect
- Synced logs (sensors, state, actions), per-trial CSV (success, times, errors), annotated video, and incident reports.
- Brief post-mortem for any score ≤2: root cause, patch, and re-test date.