Stop Teaching Robots to See: What Physical AI ACTUALLY Needs to Survive

From Pixels to State

A robot can correctly recognize every visible object in a scene and still fail at the physical task in front of it.

That sounds paradoxical only if perception is treated as image recognition.

For an embodied system, recognizing what something is and estimating what the system is doing are different scientific problems. A humanoid may identify a floor, a table and a person correctly while still having an inadequate estimate of its own velocity, contact condition, momentum, distance, sensor latency or uncertainty.

Week 14 of Frontiers of STEAM Intelligence treated humanoid behavior as the result of a coupled body, controller, and environment. The next question is unavoidable: what information must reach the controller before useful action is possible?

Measurements are not states

Sensors produce observations.

A camera produces pixels. An inertial measurement unit produces acceleration and angular-rate measurements. Encoders report joint configurations. Force sensors, tactile arrays, LiDAR and event cameras each reveal different aspects of a robot and its environment.

But the quantities required for control are often not measured directly.

A useful mathematical abstraction is

xk+1=f(xk,uk)+wkx_{k+1}=f(x_k,u_k)+w_k

and

zk=h(xk)+vkz_k=h(x_k)+v_k

The state \(x_k\) evolves according to dynamics and actions. The sensor returns \(z_k\), a noisy and usually incomplete consequence of that state.

The estimator’s task is therefore to reason backward from observations toward quantities required for prediction or action.

Kalman’s classical filtering framework formalized this distinction more than six decades ago: measurements and dynamics are combined recursively to estimate an underlying state and its uncertainty under explicitly defined assumptions.¹ (bishtref.com)

This remains central to modern robotics.

The same image can demand different actions

Imagine a robot whose torso is tilted five degrees forward.

A photograph captures the angle.

But it does not, by itself, tell us whether the robot is moving farther forward, stationary, or already moving back toward equilibrium.

Take the deliberately simplified feedback law

τ=−kpθ−kdθ˙τ=-k_pθ-k_dθ̇

With the same 5∘5^\circ angle, a change in angular velocity from -0.4 to +0.4rad/s+0.4\;\mathrm{rad/s} changes the illustrative torque from approximately +0.26+0.26 to −3.75Nm-3.75\;\mathrm{N\,m} for the stated gains.

Nothing in the static posture changed.

The state changed.

This is one reason robotic perception cannot be reduced to assigning semantic labels to images. Motion history, proprioception and dynamics can contain information that a single frame does not.

Perception is often fusion

Real robots rarely possess one perfect sensor.

Legged-robot state-estimation research illustrates why. Bloesch and colleagues combined inertial and kinematic information to estimate robot motion while preserving the observability properties of the underlying system.² Hartley and colleagues later developed contact-aided invariant filtering that incorporates inertial measurements, kinematics, and changing contacts to estimate biped pose and velocity.³ (roboticsproceedings.org)

Other systems fuse still more modalities. Nobili et al. combined inertial measurements, leg kinematics, stereo vision and LiDAR, partly because each sensing source carries different failure modes, latencies and environmental dependencies.⁴ (roboticsproceedings.org)

The lesson is not “add as many sensors as possible.”

It is more disciplined:

Choose observations that constrain the states required by the decision, then model how trustworthy those observations are.

Adding redundant, biased or badly synchronized sensors can make an estimator more complicated without making its conclusions more reliable.

Uncertainty is part of perception

An estimate without its uncertainty can be dangerously persuasive.

In a probabilistic estimator, uncertainty may be represented through a covariance matrix or a probability distribution. That does not make the uncertainty automatically correct. Noise assumptions, contact models, and sensor correlations can all be wrong.

This problem becomes particularly visible in contact-rich locomotion. Recent CoCo-InEKF research replaces simple binary contact assumptions with continuous contact covariance estimates intended to represent partial contact and directional slippage.⁵ Another 2026 study, PRIME, treats motion estimation as a physically constrained inference problem and jointly reasons about contact forces and inertial parameters rather than assuming kinematic reconstruction tells the complete physical story.⁶ (roboticsproceedings.org)

These studies differ technically, but they reinforce a common point: the quality of perception depends partly on whether the system knows which information is uncertain and why.

Some things cannot be inferred

A sufficiently clever algorithm cannot recover information that the measurements and dynamics do not contain.

This is the problem of observability.

For particular legged-robot sensing configurations, some global position or orientation quantities may remain unobservable and therefore drift. State-estimation research treats that limitation explicitly rather than pretending an estimator can reconstruct everything from inadequate measurements. (journals.sagepub.com)

New sensing technologies do not eliminate this principle. Event-camera research, for example, has shown that the observability of quantities can depend on the motion itself; Lu et al. consequently designed their visual–inertial system around instantaneous velocity, a quantity compatible with the sensing modality and task.⁷ (roboticsproceedings.org)

The important question is therefore not simply, “How accurate is the sensor?”

It is:

What can this sensor configuration actually tell us about the variables that matter?

Must the state be explicit?

Not always.

This is an important boundary on the argument.

In 2024, Geles and colleagues demonstrated a vision-based quadrotor that flew racing gates at high speed without a conventional explicit state-estimation pipeline. Their learned policy transformed visual information directly into control commands.⁸ (roboticsproceedings.org)

That result does not prove that state is irrelevant. It shows that task-relevant information can sometimes be encoded implicitly within a learned policy rather than represented as a named vector such as position, velocity, and orientation.

This distinction matters increasingly as robotics adopts large learned policies.

The scientific question changes from:

“Where is the state estimator?”

to:

“What information has the system actually extracted, under what conditions, with what uncertainty, and how does failure in that representation propagate into action?”

That is a harder question—and often the more useful one.

Perception closes the loop

The classical idea of active perception pushes the argument one step further. Perception need not be passive. A robot can move its camera, touch an object, change viewpoint, or perform a probing action because the action itself can reduce uncertainty.⁹ (rankless.org)

Perception and action are therefore not independent modules connected only at the end of a pipeline.

They can form a cycle:

sense → estimate → evaluate uncertainty → act → generate new evidence

For embodied intelligence, this is a more useful picture than:

“camera → recognition → intelligence.”

The practical test

When assessing a robot perception system, I would therefore ask five questions:

  1. What decision must the robot make?
  2. Which state variables or latent information affect that decision?
  3. Which observations constrain those quantities?
  4. What remains uncertain or unobservable?
  5. What happens to the physical task when the estimate is wrong, delayed or overconfident?

That final question turns perception from an image-processing benchmark into a systems problem.

Image recognition remains extraordinarily valuable. It can identify objects, people, affordances, landmarks, and semantic context.

But for a physical system, successful recognition is often only the beginning.

The robot must still determine what is happening, what can be inferred reliably, what remains uncertain—and what action the evidence actually supports.

That brings us naturally to Week 16.

Knowing the state does not guarantee recovery.

The next problem is balancing control and the physics of stability.

References

  1. Kalman, R. E. (1960). A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 82(1), 35–45. doi:10.1115/1.3662552. Publisher record
  2. Bajcsy, R. (1988). Active perception. Proceedings of the IEEE, 76(8), 966–1005. doi:10.1109/5.5968. Paper record
  3. Bloesch, M., Hutter, M., Hoepflinger, M., Leutenegger, S., Gehring, C., et al. (2012). State estimation for legged robots: Consistent fusion of leg kinematics and IMU. Proceedings of Robotics: Science and Systems VIII. doi:10.15607/RSS.2012.VIII.003. RSS paper
  4. Nobili, S., Camurri, M., Barasuol, V., Focchi, M., Caldwell, D., et al. (2017). Heterogeneous sensor fusion for accurate state estimation of dynamic legged robots. Proceedings of Robotics: Science and Systems XIII. doi:10.15607/RSS.2017.XIII.007. RSS paper
  5. Hartley, R., Ghaffari, M., Eustice, R. M., & Grizzle, J. W. (2020). Contact-aided invariant extended Kalman filtering for robot state estimation. The International Journal of Robotics Research, 39(4), 402–430. doi:10.1177/0278364919894385. Journal article
  6. Geles, I., Bauersfeld, L., Romero, A., Xing, J., & Scaramuzza, D. (2024). Demonstrating agile flight from pixels without state estimation. Proceedings of Robotics: Science and Systems XX. doi:10.15607/RSS.2024.XX.082. RSS paper
  7. Lu, X., Zhou, Y., Niu, J., Zhong, S., & Shen, S. (2024). Event-based visual inertial velometer. Proceedings of Robotics: Science and Systems XX. doi:10.15607/RSS.2024.XX.088. RSS paper
  8. Ha, S., Lee, J., van de Panne, M., Xie, Z., Yu, W., et al. (2025). Learning-based legged locomotion: State of the art and future perspectives. The International Journal of Robotics Research. doi:10.1177/02783649241312698. Journal article
  9. Baumgartner, M., Müller, D., Serifi, A., Grandia, R., Knoop, E., et al. (2026). CoCo-InEKF: State estimation with learned contact covariances in dynamic, contact-rich scenarios. Proceedings of Robotics: Science and Systems XXII. doi:10.15607/RSS.2026.XXII.178. RSS paper
  10. Kang, J., Ren, K., Pang, T., & Xiong, X. (2026). PRIME: Physically-consistent robotic inertial and motion estimation for legged and humanoid robots. Proceedings of Robotics: Science and Systems XXII. doi:10.15607/RSS.2026.XXII.029. RSS paper