Streaming Control

After a session is created, the control WebSocket is the place where your application changes what happens next. The video remains on the HLS playback URL; the WebSocket carries commands and responses.

Interaction overview

You can interact with a running story in four ways:

Control	Use it when	Response
`instruction.create`	You want to decide what happens next.	The instruction is received, scheduled, started, and completed or failed.
`state.update` / `state.get`	Your product needs to change or read health, score, location, inventory, or another structured value.	The full state is returned in `state.updated` or `state.snapshot`.
`assets.add`	A new character, object, place, image, or sound becomes relevant.	`assets.added` confirms that the reference can be used by later instructions.
`session.stop`	The user leaves or the story should reach a conclusion.	`session.ending`, then `session.ended`.

A common interaction looks like this: A client does not need to use every control. Most experiences can start with instruction.create, add state when the product has numbers or rules to display, and add references only when a new character or object appears.

Connect once and start playback

Create a session first, then open the returned control URL with the session-specific secret:

const ws = new WebSocket(session.control.url, [
  "streaming-world.v1",
  session.control.client_secret
]);

let lastEventId = 0;

ws.onmessage = ({ data }) => {
  const event = JSON.parse(data);
  if (Number.isInteger(event.id)) lastEventId = event.id;

  if (event.type === "session.ready") {
    player.load(event.playback.url);
    player.play();
  }
};

session.ready means the opening media is ready to play. Its playback.url is an HLS address; pass it to an HLS-capable player instead of constructing a URL yourself. The second WebSocket subprotocol value is the session secret. Do not send your general API key over this channel.

Send the next instruction

instruction.create is the primary interaction. Use it whenever the user chooses an action, asks a question that should become part of the story, or wants to change the direction of the next scene.

Request

Describe one coherent piece of what should happen next. Include the important action, characters, tone, and visual result in the text. You do not need to split it into shots or dialogue turns.

{
  "type": "instruction.create",
  "client_request_id": "choice_001",
  "text": "The detective opens the hidden drawer, finds a torn photograph, and asks the witness who is missing from it. Keep the camera close to the photograph before returning to their faces.",
  "duration_seconds": 10
}

Field	What to put there
`type`	`instruction.create`.
`client_request_id`	Your own correlation id. Keep it for the response events.
`text`	The next story action in natural language.
`duration_seconds`	Requested video time. Omit it to use the service default.
`state_ops`	Optional state changes that belong to this instruction; see state management.

Response events

The normal lifecycle is:

{
  "id": 10,
  "type": "instruction.received",
  "time": "2026-10-28T08:00:30.000Z",
  "instruction_id": "ins_001",
  "client_request_id": "choice_001",
  "text": "The detective opens the hidden drawer, finds a torn photograph, and asks the witness who is missing from it. Keep the camera close to the photograph before returning to their faces.",
  "duration_seconds": 10,
  "estimated_start_seconds": 18.0
}

{
  "id": 11,
  "type": "instruction.scheduled",
  "time": "2026-10-28T08:00:31.000Z",
  "instruction_id": "ins_001",
  "start_seconds": 18.0,
  "end_seconds": 28.0
}

{
  "id": 18,
  "type": "instruction.started",
  "time": "2026-10-28T08:00:50.000Z",
  "instruction_id": "ins_001",
  "start_seconds": 18.0
}

{
  "id": 26,
  "type": "instruction.completed",
  "time": "2026-10-28T08:01:00.000Z",
  "instruction_id": "ins_001",
  "start_seconds": 18.0,
  "end_seconds": 28.0
}

Treat instruction.scheduled as the current timeline estimate. A later scheduled event can revise its position. Treat instruction.started and instruction.completed as playback-time milestones, not as a guarantee that every viewer has rendered the same frame at the same wall-clock instant. If the instruction cannot be prepared or the session is ending, the server may emit:

{
  "id": 29,
  "type": "instruction.failed",
  "time": "2026-10-28T08:01:04.000Z",
  "instruction_id": "ins_002",
  "reason": "preparation_failed"
}

An instruction that arrives after ending has started may instead receive instruction.cancelled. Use a new client_request_id for a genuinely new instruction. Do not resend the same text with a new id just because the acknowledgement is delayed.

Manage world state

State is for structured values your product needs to show or enforce: health points, score, location, inventory, clue count, weather, or quest progress. It is separate from the natural-language instruction, but an instruction can carry state changes that should happen with that scene. A battle might start with:

{
  "characters": {
    "detective": {"hp": 100, "location": "warehouse"},
    "attacker": {"hp": 100, "location": "warehouse"}
  },
  "round": 1,
  "clue_count": 0
}

Change a value immediately

Use state.update when the application has already decided that a value should change now:

{
  "type": "state.update",
  "client_request_id": "state_001",
  "ops": [
    {
      "op": "set",
      "path": ["round"],
      "value": 2,
      "when": {"type": "immediate"}
    },
    {
      "op": "set",
      "path": ["characters", "detective", "hp"],
      "value": 85,
      "when": {"type": "immediate"}
    }
  ]
}

The response is a state.updated event containing the complete state, not just the two changed fields:

{
  "id": 32,
  "type": "state.updated",
  "time": "2026-10-28T08:01:10.000Z",
  "state": {
    "characters": {
      "detective": {"hp": 85, "location": "warehouse"},
      "attacker": {"hp": 100, "location": "warehouse"}
    },
    "round": 2,
    "clue_count": 0
  },
  "changes": [
    {"op":"set","path":["round"],"value":2,"client_request_id":"state_001"},
    {"op":"set","path":["characters","detective","hp"],"value":85,"client_request_id":"state_001"}
  ],
  "media_seconds": 18.0
}

Tie a value to a scene

Use state_ops on instruction.create when the value should become effective with that instruction:

{
  "type": "instruction.create",
  "client_request_id": "fight_002",
  "text": "The detective lands a clean hit and the attacker falls against the wall.",
  "duration_seconds": 8,
  "state_ops": [
    {
      "op": "set",
      "path": ["characters", "attacker", "hp"],
      "value": 40,
      "when": {"type": "instruction_end"}
    }
  ]
}

The state change is applied when this instruction finishes playing. Use instruction_start for a fact that becomes true when the scene begins, and instruction_end for a change that takes effect when the scene ends. For a previously accepted instruction, state.update can target it explicitly:

{
  "type": "state.update",
  "client_request_id": "state_002",
  "ops": [
    {
      "op": "set",
      "path": ["characters", "detective", "location"],
      "value": "warehouse",
      "when": {
        "type": "instruction_start",
        "instruction_id": "ins_001"
      }
    }
  ]
}

Supported operations are set, delete, append, and remove. The service accepts or rejects the whole batch; it does not partially apply an invalid batch.

Read the current state

Use state.get when the UI needs a fresh snapshot:

{"type":"state.get","client_request_id":"state_read_001"}

The response goes only to the connection that asked:

{
  "type": "state.snapshot",
  "time": "2026-10-28T08:01:22.000Z",
  "client_request_id": "state_read_001",
  "state": {
    "characters": {
      "detective": {"hp": 85, "location": "warehouse"},
      "attacker": {"hp": 100, "location": "warehouse"}
    },
    "round": 2,
    "clue_count": 0
  },
  "media_seconds": 18.0,
  "last_event_id": 32
}

Replace the local state with the full state object from state.updated or state.snapshot. Do not infer a separate hidden state schema.

Add a reference when the story needs one

Add a reference when a new character, object, location, or voice becomes relevant. Register the file with the public upload API first, then send its file_id over the control channel.

curl -X POST https://api.vivix.ai/v1/files/images \\
  -H "Authorization: Bearer $VIVIX_API_KEY" \\
  -F "file=@attacker.jpg"

Then add it to the session:

{
  "type": "assets.add",
  "client_request_id": "asset_001",
  "assets": [
    {
      "name": "attacker",
      "type": "image",
      "file_id": "file_img_attacker",
      "description": "Tall attacker wearing a dark jacket"
    }
  ]
}

Response events

Success:

{
  "id": 34,
  "type": "assets.added",
  "time": "2026-10-28T08:01:30.000Z",
  "client_request_id": "asset_001",
  "assets": [
    {"asset_id":"asset_attacker","name":"attacker","type":"image"}
  ]
}

Failure:

{
  "id": 35,
  "type": "assets.failed",
  "time": "2026-10-28T08:01:31.000Z",
  "client_request_id": "asset_002",
  "error": {
    "code": "asset_preparation_failed",
    "message": "could not prepare the requested assets"
  }
}

Wait for assets.added before sending an instruction that depends on the new reference:

{
  "type": "instruction.create",
  "client_request_id": "fight_003",
  "text": "The attacker enters from the left and challenges the detective. Use the reference named attacker for the attacker's appearance.",
  "duration_seconds": 8
}

Adding a reference does not force it into the current frame. It only makes the material available to later story instructions. A session can contain more references than one generated segment uses; the service chooses the references relevant to each segment.

Stop the session

Stop immediately when the user leaves or the application must end the experience now:

{"type":"session.stop","immediate":true}

Request a closing scene when the user wants a conclusion:

{
  "type": "session.stop",
  "closing": {
    "text": "The detective secures the evidence, looks toward the rain-covered window, and leaves the warehouse.",
    "duration_seconds": 8
  }
}

The server sends session.ending while the closing is being prepared and session.ended after the session reaches its terminal state. Once session.ending arrives, stop sending new story instructions.

Reconnect and event ordering

Every historical server event has a monotonically increasing id. Save the latest id after processing each event. If the connection drops, reconnect with:

wss://sw.vivix.ai/v1/streaming-world/sessions/sw_example/control?last_event_id=42

The server sends events after that cursor and then resumes live delivery. state.snapshot.last_event_id is the cursor included in that state read. Process events by id. A later instruction.scheduled can revise an earlier timeline estimate. If a session has already ended, use GET /v1/streaming-world/sessions/{session_id} for the final status and timeline.

Handle control errors

Use the public error.code to decide what the client should do:

Code	Client action
`invalid_request`	Fix the message or value and retry with a new request id.
`unauthorized`	Refresh the session credential.
`session_ending` / `session_ended`	Stop sending new commands and show the final state.
`capacity_unavailable`	Retry later with backoff.
`asset_preparation_failed`	Check the uploaded file and register it again.
`upstream_failed` / `internal_error`	Show a generic retry message and keep the session id for support.

Use Client messages for the complete request fields and Server events for every response schema.

Overview

A1: Streaming Avatar

W1: Streaming World

Blazing Fast Video Generation

Interaction overview

Connect once and start playback

Send the next instruction

Request

Response events

Manage world state

Change a value immediately

Tie a value to a scene

Read the current state

Add a reference when the story needs one

Response events

Stop the session

Reconnect and event ordering

Handle control errors