docs: metrics endpoints & smoothing env vars; feat: OneEuro env initialization

README.md

@@ -150,6 +150,11 @@ MIT License - Feel free to use and modify for your projects!
 ## Metrics Endpoints
 - `GET /metrics` – JSON with audio/video counters, EMAs (loop interval, inference), rolling FPS, frame interval EMA.
 - `GET /gpu` – GPU availability & memory (torch or `nvidia-smi` fallback).
+- `GET /metrics/async` – Async worker stats (frames submitted/processed, queue depth, last latency ms).
+- `GET /metrics/stage_histogram` – Histogram buckets of recent inference stage latencies (snapshot window).
+- `GET /metrics/motion` – Recent motion magnitudes (normalized) plus tail statistics.
+- `GET /metrics/pacing` – Latency EMA and pacing hint multiplier ( >1.0 suggests you can raise FPS, <1.0 suggests throttling ).
+- `POST /smoothing/update` – Runtime update of One Euro keypoint smoothing params. JSON body keys: `min_cutoff`, `beta`, `d_cutoff` (all optional floats).
 
 Example:
 ```bash
@@ -198,6 +203,56 @@ If the Space shows a perpetual "Restarting" badge:
 If problems persist, capture the Container log stack trace and open an issue.
 
 ## Enable ONNX Model Downloads (Safe LivePortrait)
+## Advanced Real-time Metrics & Control
+
+New runtime observability & control surfaces were added to tune real-time performance:
+
+### Endpoints Recap
+See Metrics Endpoints section above. Typical usage examples:
+
+```bash
+curl -s http://localhost:7860/metrics/async | jq
+curl -s http://localhost:7860/metrics/pacing | jq '.latency_ema_ms, .pacing_hint'
+curl -s http://localhost:7860/metrics/motion | jq '.recent_motion[-5:]'
+```
+
+### Pacing Hint Logic
+`pacing_hint` is derived from a latency exponential moving average vs target frame time:
+- ~1.0: Balanced.
+- <0.85: System overloaded – consider lowering capture FPS or resolution.
+- >1.15: Headroom available – you may increase FPS modestly.
+
+### Motion Magnitude
+Aggregated from per-frame keypoint motion vectors; higher values trigger more frequent face detection to avoid drift. Low motion stretches automatically reduce detection frequency to save compute.
+
+### One Euro Smoothing Parameters
+You can initialize or override smoothing parameters via environment variables:
+
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `MIRAGE_ONEEURO_MIN_CUTOFF` | 1.0 | Base cutoff frequency controlling overall smoothing strength |
+| `MIRAGE_ONEEURO_BETA` | 0.05 | Speed coefficient (higher reduces lag during fast motion) |
+| `MIRAGE_ONEEURO_D_CUTOFF` | 1.0 | Derivative cutoff for velocity filtering |
+
+Runtime adjustments:
+```bash
+curl -X POST http://localhost:7860/smoothing/update \
+	-H 'Content-Type: application/json' \
+	-d '{"min_cutoff":0.8, "beta":0.07}'
+```
+Missing keys leave existing values unchanged. The response echoes the active parameters.
+
+### Latency Histogram Snapshots
+`/metrics/stage_histogram` exposes periodic snapshots (e.g. every N frames) of stage latency distribution to help identify tail regressions. Use to tune pacing thresholds or decide on model quantization.
+
+## Environment Variables Summary (New Additions)
+
+| Name | Purpose | Default |
+|------|---------|---------|
+| `MIRAGE_ONEEURO_MIN_CUTOFF` | One Euro base cutoff | 1.0 |
+| `MIRAGE_ONEEURO_BETA` | One Euro speed coefficient | 0.05 |
+| `MIRAGE_ONEEURO_D_CUTOFF` | One Euro derivative cutoff | 1.0 |
+
 
 To pull LivePortrait ONNX files into the container at runtime and enable the safe animation path:
 

avatar_pipeline.py

@@ -378,21 +378,27 @@ class RealTimeAvatarPipeline:
         self._frame_submit_count = 0
         self._frame_process_count = 0
         # Keypoint smoothing filter
-        self._kp_filter = KeypointOneEuro(K=21, C=3, min_cutoff=1.0, beta=0.05, d_cutoff=1.0)
+        try:
+            min_cut = float(os.getenv('MIRAGE_ONEEURO_MIN_CUTOFF', '1.0'))
+            beta = float(os.getenv('MIRAGE_ONEEURO_BETA', '0.05'))
+            d_cut = float(os.getenv('MIRAGE_ONEEURO_D_CUTOFF', '1.0'))
+        except Exception:
+            min_cut, beta, d_cut = 1.0, 0.05, 1.0
+        self._kp_filter = KeypointOneEuro(K=21, C=3, min_cutoff=min_cut, beta=beta, d_cutoff=d_cut)
         self._prev_motion_raw = None
         # Adaptive detection interval tracking
         self._dynamic_detect_interval = self.config.detect_interval
         self._recent_motion_magnitudes = deque(maxlen=30)
         self._consecutive_detect_fail = 0
-    # Extended motion history for metrics
-    self._motion_history = deque(maxlen=300)
-    # Latency histogram snapshots (long window)
-    self._latency_history = deque(maxlen=500)
-    self._latency_hist_snapshots = []  # list of {timestamp, buckets}
-    # Frame pacing
-    self._pacing_hint = 1.0  # multiplier suggestion (1.0 = normal)
-    self._target_frame_time = 1.0 / max(self.config.target_fps, 1)
-    self._latency_ema = None
+        # Extended motion history for metrics
+        self._motion_history = deque(maxlen=300)
+        # Latency histogram snapshots (long window)
+        self._latency_history = deque(maxlen=500)
+        self._latency_hist_snapshots = []  # list of {timestamp, buckets}
+        # Frame pacing
+        self._pacing_hint = 1.0  # multiplier suggestion (1.0 = normal)
+        self._target_frame_time = 1.0 / max(self.config.target_fps, 1)
+        self._latency_ema = None
         
     async def initialize(self):
         """Initialize all models"""

liveportrait_engine.py

@@ -63,7 +63,8 @@ class LivePortraitONNX:
         
         # Performance tracking
         self.inference_times = []
-    self._did_warmup = False
+        # Warmup sentinel
+        self._did_warmup = False
         
     def _get_onnx_providers(self) -> List[str]:
         """Get optimal ONNX execution providers. Enforce GPU if required."""
@@ -378,10 +379,30 @@ class LivePortraitONNX:
             return None
     
     def extract_motion_parameters(self, driving_image: np.ndarray) -> Optional[np.ndarray]:
-        """Extract motion parameters from driving image"""
+        """Extract motion parameters from driving image.
+        Returns keypoints/pose tensor shaped [1,K,3] if possible.
+        """
         if self.motion_session is None:
             logger.error("Motion model not loaded")
             return None
+        try:
+            motion = self._run_motion_for_image(driving_image)
+            # Prefer explicit kp_driving from motion model if available
+            if isinstance(motion, dict):
+                kp_drive = (motion.get('kp_driving') or motion.get('driving') or
+                            motion.get('kp_drive') or motion.get('drive'))
+                if kp_drive is None:
+                    # Fallback to first ndarray value
+                    kp_drive = next((v for v in motion.values() if isinstance(v, np.ndarray)), None)
+                if kp_drive is None:
+                    logger.error("Motion model outputs did not include a usable keypoint array")
+                    return None
+                return kp_drive
+            else:
+                return motion
+        except Exception as e:
+            logger.error(f"Motion parameter extraction failed: {e}")
+            return None
 
     def warmup(self) -> bool:
         """Run a minimal generator inference with zero motion to prime kernels and caches.
@@ -414,22 +435,6 @@ class LivePortraitONNX:
         except Exception as e:
             logger.debug(f"Warmup skipped: {e}")
             return False
-        
-        try:
-            motion = self._run_motion_for_image(driving_image)
-            # Prefer explicit kp_driving from motion model if available
-            if isinstance(motion, dict):
-                kp_drive = motion.get('kp_driving') or motion.get('driving')
-                if kp_drive is None:
-                    # Fallback to first array value
-                    kp_drive = next((v for v in motion.values() if isinstance(v, np.ndarray)), None)
-                return kp_drive
-            else:
-                return motion
-            
-        except Exception as e:
-            logger.error(f"Motion parameter extraction failed: {e}")
-            return None
 
     def _run_motion_for_image(self, img: np.ndarray):
         """Helper: run motion/keypoint extractor on an image and return structured outputs."""