Feature Request: Pluggable AI Media Processing Pipeline

[Bin-hy]

Many real-time AI applications (avatar generation, video understanding, gesture recognition, face swapping, OCR, background replacement, etc.) require access to media frames inside the SFU pipeline.

It would be valuable if Live777 could provide a pluggable media processing interface that allows developers to:

• intercept incoming RTP streams
• optionally decode video frames
• process frames with external AI services
• return modified frames back into the forwarding pipeline

This feature could be implemented as an optional plugin architecture so that normal SFU forwarding performance remains unaffected.

Possible use cases:

• Real-time AI avatars
• Face replacement
• Video analytics
• OCR
• Object detection
• Background removal
• AI moderation

This would make Live777 a strong foundation for AI-native real-time media applications.

[Bin-hy]

Proposed Architecture

To keep Live777 lightweight and high-performance, AI processing should be implemented as an optional extension layer rather than being tightly integrated into the core SFU forwarding path.

Layer 1: RTP Interceptor

Provide a hook interface for observing RTP packets.

pub trait RtpInterceptor {
    async fn on_packet(
        &self,
        packet: &RtpPacket,
        track: &TrackInfo,
    );
}

This layer would allow:

• Recording
• Analytics
• AI stream subscription
• RTP forwarding to external services

without affecting the normal forwarding pipeline.

---

Layer 2: Media Transform Pipeline (Optional)

For advanced use cases requiring frame-level access:

pub trait VideoFrameProcessor {
    async fn process(
        &self,
        frame: VideoFrame,
    ) -> VideoFrame;
}

Pipeline:

RTP
 ↓
Decode
 ↓
VideoFrame
 ↓
AI Processor
 ↓
VideoFrame
 ↓
Encode
 ↓
RTP

This stage should remain optional because it introduces additional latency and compute requirements.

---

Layer 3: External AI Worker Integration

Instead of embedding AI models inside Live777, allow external workers to subscribe to media streams.

Architecture:

Publisher
    │
    ▼
 Live777
    │
 ┌──┴─────────────┐
 ▼                ▼

Viewer         AI Worker
                    │
                    ▼

          OCR / ASR / YOLO
          Face Swap
          Avatar Generation
          Moderation

                    │
                    ▼

          Publish Processed Stream

This approach keeps Live777 focused on media transport while enabling AI-native real-time applications.

---

Benefits

• No performance impact for normal SFU users
• AI features remain fully optional
• Easy integration with GPU inference clusters
• Compatible with OCR, ASR, object detection, avatars, moderation, and video analytics
• Maintains Live777's lightweight architecture