Shadow Hand — Reward Hacking Visualiser

Reward Hacking Emergence & Mitigation

PPO training of a 24-DoF Shadow Dexterous Hand · Phase-based reward shaping · Tactile sensor analysis · Bharadwaj Rachuri & Dr Diego Resende Faria · University of Hertfordshire, UK

Peak Episode Reward

18,007

Achieved in Phase 3 training

Rotation Achieved

175°

97.2% of 180° target

Touch Sensor Accuracy

>90%

Across all 5 fingers

Total Timesteps

73.7K

Training FPS: 48 · 3 seeds

⚠️

REWARD HACKING DETECTED — Phase 1 Baseline

During early single-phase training, the PPO agent discovered a specification gaming exploit: it learned to maximise the rotation velocity reward component without achieving stable pen grasping — spinning the finger actuators in place rather than rotating the pen. This is a direct instance of reward misspecification. Mitigation: phase-based reward decomposition (Grasp → Rotate → Hold) with gated activation — each phase reward only unlocks when the prior phase threshold is met. Result: the agent converged to genuine manipulation behaviour.

Reward Curve — Phase Comparison

Episode reward across training phases. Phase 1 plateaus early (hacking). Phases 2–3 show genuine learning progression.

Reward Hacking vs Genuine Learning

Rotation achieved (°) vs episode reward. Hacking cluster: high reward, low rotation. Genuine learning: both increase together.

Tactile Sensor Activation

Contact events per finger over training. All 5 fingers reach >90% coordination by episode 60.

KL Divergence

Policy update stability. Final KL: 0.0049 — well within safe PPO clip range.

Entropy Loss

Exploration decay over training. Entropy decreases as policy converges to optimal pen-spin strategy.

Value & Policy Loss Curves

Policy gradient loss converges to -0.00534. Value function loss stabilises after reward hacking mitigation.

3-Seed Reproducibility

Mean ± std across 3 independent seeds. Final: 3.118 ± 0.047 rad. Confirms statistical robustness.

Phase 1 — Grasp

Reward: contact sensor activation
+10 per finger touching pen
Hacking discovered: actuator spinning
Fix: require stable grip before Phase 2 unlocks

Phase 2 — Rotate

Reward: -angle_error × 2
+vel × 2 if rotating correctly
+200 bonus at 180° with stable grasp
Policy gradient loss converging

Phase 3 — Hold

Reward: sustained contact + stability
Peak reward: 18,007 achieved
175° / 97.2% of target
Reproduced across 3 seeds

Final Training Metrics — Published Results

Metric	Value	Status	Notes
Max Episode Reward	18,007.53	✓ TARGET MET	Phase 3 convergence
Final Z-angle (rad)	3.058 rad	✓ 175° ACHIEVED	97.2% of π target
Avg Episode Reward	32,600	↑ STRONG	Mean across final episodes
Avg Episode Length	831 steps	→ STABLE	~960 max steps budget
Total Timesteps	73,728	✓ COMPLETED	FPS: 48 (CPU-only)
KL Divergence	0.0049	✓ SAFE RANGE	PPO clip threshold: 0.2
Policy Gradient Loss	-0.00534	✓ CONVERGED	Negative = improving
Value Function Loss	1.26 × 10⁶	⚠ HIGH SCALE	Expected at large reward scale
Explained Variance	13.4%	⚠ LIMITED	Complex env; noted in paper
Touch Sensor Accuracy	>90%	✓ ALL 5 FINGERS	Coordination confirmed
Seed Reproducibility	3.118 ± 0.047 rad	✓ 3 SEEDS	Statistically robust