Meet the Team
press → to reveal team
Problem Outline
Real-time custom advertising for event livestreams, to integrate with personalized recommendation systems.
- Develop a real-time ROI tracking system for tennis matches.
- Detect advertisement boards and replace them with dynamic ads.
press → to highlight target region
Original broadcast footage sample
Key Challenges
Core technical hurdles we need to solve for reliable, real-time ad replacement.
Existing Approaches & Landscape
COMMERCIAL SYSTEMS
Supponor (NHL DED)
IR strips in dasherboards + AI keying. $1.28B ad revenue (2023-24). Requires proprietary hardware at every venue.
uniqFEED (AdApt)
Software-based, deployed at major tennis events. Requires trained operators and custom-trained CV models per sport.
Vizrt / Viz Arena
Camera tracking hardware (encoder heads). Real-time but hardware-dependent and operator-intensive.
ACADEMIC WORK
Homography estimation — focused on soccer field registration (Nie et al. WACV 2021, Homayounfar CVPR 2017). Not applied to ad replacement.
SAM 2 in sports — used for ball tracking and player tracking. Not applied to advertisement board segmentation.
Virtual ad insertion research — soccer-only, predates foundation models, uses hand-crafted features.
KNOWN CHALLENGES
Jittering
Overlay instability across frames — tracking drift causes visible shaking in replaced banners.
Occlusion
Players walking in front of banners cause artifacts — overlays render on top of or through players.
Speed
Processing speed below real-time thresholds, limiting viability for live broadcast scenarios.
No prior work combines foundation model segmentation with classical CV for end-to-end sports ad replacement.
Each component exists individually — the pipeline does not.
Why This Combination is Novel
Foundation models (SAM 2) eliminate the need for hardware, custom training, and continuous operators for the segmentation problem. The remaining challenges — perspective geometry, compositing, occlusion — are solved with specific classical CV techniques.
Stable Tracking
Homography fitting combined with optical flow produces consistent overlays with minimal jitter across frames.
Occlusion Handling
Dedicated player segmentation pass generates per-pixel masks, allowing overlays to render correctly behind players.
Targeting 30 fps
Early results suggest real-time performance is achievable with GPU acceleration. Benchmarks in progress.
No prior work assembles this specific pipeline — zero-shot segmentation + classical CV geometry + per-pixel occlusion handling.
How We Compare
| System | Hardware | Training | Operators | Occlusion | Stability | Latency |
|---|---|---|---|---|---|---|
| Supponor (NHL) | IR strips | Proprietary | Central hub | No (clipping) | High | Real-time |
| uniqFEED | None | Custom CV | Trained ops | Partial (bbox) | High | Near real-time |
| Vizrt / Viz Arena | Camera HW | Required | Required | Partial (bbox) | High | Real-time |
| MEIL (Prior) | None | Custom | Manual | Limited | Jittering | Below real-time |
| This Pipeline | None | None (SAM 2) | 1 click | Per-pixel masks | Stable | ~30 fps* |
Fully software-based · Zero training · Minimal operator input · Per-pixel occlusion · Stable tracking
* Preliminary estimate — benchmarks in progress
Full Pipeline Overview
Segment Anything
Model 2
We use Meta's SAM 2 as the backbone for banner segmentation. Given a single click prompt on the first frame, SAM 2 tracks and segments across all subsequent frames.
- Pre-trained on 11M images, 1B+ masks
- Prompt with points, boxes, or masks
- Memory bank for temporal consistency
SAM 2 architecture — Ravi et al., 2024
Player
Segmentation
Detect and segment players on the court to ensure overlaid banners render behind them, preserving a natural viewing experience.
Banner Segmentation
& Tracking
Detect advertising banners across different regions of the court — ground-level boards, back wall panels, net-mounted banners, umpire stand signage, and more.
Banner Segmentation
- Banners Stable Camera
- Banners Moving Camera
- Logos Stable Camera
- Logos Moving Camera
- Camera Cutsexperiment
Homography &
Perspective Geometry
Understand the geometry and perspective of the banners from the camera's viewpoint so that overlays and modifications appear realistic to the viewer in the final livestream.
Vanishing
Point
The vanishing point constrains how banner edges converge — it's not a true parallelogram but a perspective quadrilateral.
Quadrilateral
Fitting
- Original frame
- SAM mask
- Binary mask
- Min-area rectangle
- Split along axes
- Fit edge lines
- Intersect corners
- Rectified view
New Logo
Overlay
- InpaintRemove the original logo from the banner surface
- Shadow / Tint MatchingMatch luminosity and color tone of the surface
- Border BlendingSmooth alpha edges for seamless compositing
Demo
Future Improvements
Visual Improvements
- Ball tracking
- Occlusion handling
- Shadows & lighting correction
Camera Cuts
- Detect and handle camera angle transitions gracefully
Real-Time Performance
- Model fine-tuning for speed
- Inference benchmarking at 30 fps target
Automatic Region Selection
- Remove manual click-to-select step
- Auto-detect ad regions from frame content
Other Sports
- Generalize pipeline beyond tennis to football, basketball, etc.
Asks From
Mitsubishi
To move to the next phase, we need a few things from the Mitsubishi team.
Sample Video
Sample footage from existing Mitsubishi models or internal datasets to benchmark and validate our pipeline against real-world broadcast conditions.
Resources
Access to GPU compute for training and real-time inference. Our pipeline targets 30 fps — knowing available hardware helps us tune model complexity.
Pipeline Requirements
Expected livestream format, resolution, and codec. Desired input/output interface — should we consume an RTMP stream and produce one, or work frame-by-frame?