# Gemini Multimodal Live API Integration: Technical Brief for Google Support

---

## 1. Environment & SDK Details
*   **API Interface**: Gemini Multimodal Live API (`bidiGenerateContent` over WebSockets)
*   **Python SDK**: `google-genai` (Version `2.10.0`)
*   **Python Runtime**: `Python 3.10.20`
*   **Container Environment**: Docker (Alpine/Debian-based FastAPI server)
*   **Host OS**: Linux
*   **Endpoint Protocol**: `generativelanguage.googleapis.com` (using standard API Key authentication)

---

## 2. Session Initialization & Configuration
The session is established using the asynchronous Live API namespace:
```python
from google import genai
from google.genai import types

client = genai.Client(api_key="YOUR_API_KEY")

async with client.aio.live.connect(
    model="gemini-2.5-flash-native-audio-latest", # Also tested with gemini-3.1-flash-live-preview
    config=types.LiveConnectConfig(
        response_modalities=[types.Modality.AUDIO],
        system_instruction=types.Content(parts=[types.Part.from_text(text="Your system instructions...")]),
        input_audio_transcription=types.AudioTranscriptionConfig(), # Enabled to receive user text transcripts
    )
) as session:
    # Session started successfully
```

---

## 3. Audio Streaming Protocol

### A. Client-to-Server (Browser to FastAPI Backend)
*   **Web Audio Capture**: Browser captures audio from microphone via `navigator.mediaDevices.getUserMedia`.
*   **Resampling**: Downsampled on the fly in JavaScript to `16,000Hz` (16kHz).
*   **Format**: Signed 16-bit Linear PCM, Mono, Little-Endian.
*   **Streaming**: Streamed over a local WebSocket in raw binary array buffers.
*   **Noise Gate**: Filters out baseline room static (sending nothing during silence).

### B. Server-to-Gemini (FastAPI to Gemini Live API)
*   **Audio Chunks**: Raw 16-bit PCM bytes are forwarded to Gemini as `types.Blob`:
    ```python
    await session.send_realtime_input(
        audio=types.Blob(
            data=pcm_data_bytes, 
            mime_type="audio/pcm;rate=16000"
        )
    )
    ```
*   **Turn Completion (VAD)**:
    When a silence timeout (1.0 second) is detected on the server (meaning the user has stopped speaking and the client-side noise gate is closed), the backend explicitly sends:
    ```python
    await session.send_realtime_input(audio_stream_end=True)
    ```
    This is intended to force Gemini's Voice Activity Detection (VAD) to finalize the user turn and begin generating a response.

---

## 4. Observed Technical Issues

### Issue A: No Response to Streamed Audio
*   **Symptom**: When we initiate the call, we programmatically force a greeting turn via text:
    ```python
    await session.send_client_content(
        turns=[types.Content(role="user", parts=[types.Part.from_text(text="Welcome the client.")])],
        turn_complete=True
    )
    ```
    The Gemini model **correctly responds** to this initial text-based prompt, streaming audio bytes back which the frontend plays back cleanly.
*   However, once the user speaks and the backend streams the raw audio PCM bytes (confirmed arriving with high amplitude e.g. `26579` max amplitude) followed by the `audio_stream_end=True` message, Gemini **never** triggers a transcription callback or returns any model output.
*   The connection eventually times out or disconnects without any transcription text or audio response.

### Issue B: Keepalive Ping Timeouts under Continuous Streaming
*   If we disable the client-side noise gate (streaming continuous background room static to Gemini), the connection closes abruptly with:
    ```
    websockets.exceptions.ConnectionClosedError: sent 1011 (internal error) keepalive ping timeout; no close frame received
    ```
    This suggests the Gemini server WebSocket gets congested or stops responding to keepalive pings when receiving continuous binary audio payloads.

---

## 5. Questions for Google Support
1.  Is the format `audio/pcm;rate=16000` (16-bit Linear PCM, Mono, Little-Endian) fully supported for real-time input under `session.send_realtime_input(audio=...)`?
2.  Is there a specific payload structure or wrapper required for raw audio data chunks that differs from `types.Blob`?
3.  Why does the model fail to trigger turn completion or transcription when receiving `audio_stream_end=True` after audio blobs are sent?
4.  Are there any known VAD or bidi streaming issues with the models `gemini-2.5-flash-native-audio-latest` or `gemini-3.1-flash-live-preview` when authenticated via standard developer API keys?