Voice API

Join voice channels, receive and send audio, record and play 
dc = DiscordClient()
gid = '1493461895615873044'
gld = await dc.guild(gid)
chs = await gld.channels()
intents = Intent.GUILDS | Intent.GUILD_VOICE_STATES | Intent.GUILD_MESSAGES | Intent.MESSAGE_CONTENT
gc = GatewayClient(intents, dc)

await gc.start(debug=True)
vch = next(c for c in chs if c.id == '1493461896139903029')
vch
Channel(id=1493461896139903029, name='General', type=2)

Voice uses three connections at once. The main gateway websocket (gc) requests the join and receives the voice server details. A second websocket to the voice server coordinates the session. A UDP connection carries the audio. Audio goes over UDP because low latency matters more than guaranteed delivery. VoiceClient owns the voice websocket and the UDP connection for one channel. The main GatewayClient delivers the join events to it.

source

VoiceClient

def VoiceClient(
    gc, gid, ch
):
vc = VoiceClient(gc, gid, vch)
vc
VoiceClient(self.ch=Channel(id=1493461896139903029, name='General', type=2))

The voice gateway has its own opcodes. voice_state asks the main gateway to join a channel. voice_identify authenticates with the voice server. select_protocol picks the encryption mode. speaking marks the start and end of audio. voice_heartbeat keeps the voice websocket alive.

The voice heartbeat differs from the main gateway’s. Protocol v8 requires seq_ack, the sequence number of the last message from the server. The heartbeat must start as soon as Hello arrives, before UDP setup. Otherwise the voice server closes the websocket with code 4006.

source

Op.voice_resume

def voice_resume(
    server_id, session_id, token, seq_ack:int=-1
):

source

Op.voice_heartbeat

def voice_heartbeat(
    seq_ack:int=-1
):

source

Op.speaking

def speaking(
    ssrc, speaking:int=0
):

source

Op.select_protocol

def select_protocol(
    ip, port, mode:str='aead_xchacha20_poly1305_rtpsize'
):

source

Op.voice_identify

def voice_identify(
    server_id, user_id, session_id, token
):

source

Op.voice_state

def voice_state(
    guild_id, channel_id
):

Discord needs the bot’s external IP and port, the address the internet sees after NAT. The bot sends a 74-byte UDP request holding its SSRC. Discord returns the same packet with the external address filled in:

Field Size Description
Type 2 bytes 1=request, 2=response
Length 2 bytes 70 (size of remaining fields)
SSRC 4 bytes Our audio stream identifier
Address 64 bytes Blank in request; our external IP in response
Port 2 bytes Blank in request; our external port in response

Discord sends no audio packets until the bot sends a Speaking event, even one with speaking=0.

source

VoiceUDP

def VoiceUDP():

Interface for datagram protocol.

A listen-only bot never sends a packet after IP discovery. NAT and conntrack mappings then expire within minutes. Inbound audio stops while the websocket stays healthy. An 8-byte counter ping every 5 seconds keeps the mapping alive. discord.js does the same. The voice server ignores unknown packets.

source

VoiceClient.execute_transition

def execute_transition(
    tid
):

Op 13 (CLIENT_DISCONNECT) reports that a user left the channel. The client drops that user’s decoder and SSRC mapping. Without this, a user who rejoins inherits stale jitter state. The maps also grow for the length of the session.

vc.decoders,vc.ssrc_to_user,vc.last_seq = {5:'dec'},{111:5, 222:9},{111:1, 222:2}
await vc._handle_trans(13, dict(user_id='5'))
test_eq(vc.decoders, {})
test_eq(vc.ssrc_to_user, {222:9})
test_eq(vc.last_seq, {222:2})
vc.ssrc_to_user,vc.last_seq = {},{}

The voice gateway runs two loops. _hb sends heartbeats at the interval Discord specifies and watches for ACKs. A missed ACK closes the socket with code 4000. That code preserves the session for a resume. _reconnect handles both resume and fresh reconnect. A resume keeps session_id, token, endpoint, the UDP socket, and the DAVE state. A fresh reconnect resets the audio session and re-runs _join for new server details. _listen schedules _reconnect as its own task. _reconnect therefore cancels _listen_task only when called from elsewhere. A reconnect that cancelled the task it runs in would kill itself. Failed attempts retry with exponential backoff, capped at 30 seconds. The counter resets once the client establishes or resumes a session. Recording state survives both paths. Per-user files keep accumulating across a reconnect, with a silence gap during the swap.

source

VoiceClient.reset_audio

def reset_audio():

_listen runs the voice gateway state machine. On HELLO (op 8) it sends voice_resume if self.resuming holds and voice_identify otherwise. Op 9 confirms the resume at the gateway level. Media encryption is a separate handshake. DAVE must catch up before outgoing audio can be encrypted again. The client waits on _wait_dave_ready before setting resumed.

When the socket closes, the close code decides the recovery (docs). Codes 4003 to 4006, 4009, and 4011 mean the session’s credentials are dead. The main gateway issues voice tokens per session. The client rejoins from scratch to get fresh ones. A resume with dead credentials loops. Live testing showed that a bad resume gets 4003 Not authenticated, not 4006. Code 4014 means an admin kicked or moved the bot, or Discord is swapping the voice server. The client must not reconnect to that endpoint. It waits for gateway events instead. Any other code, such as 4015 (voice server crashed), is transient. The client resumes. Recovery runs as a separate task. The dying listener never cancels its own reconnect. The listener logs handler errors and skips them. They do not stop it.

source

VoiceClient.join

async def join(
    debug:bool=False
):

source

VoiceClient.leave

async def leave():

Notify the main gateway that we are leaving the voice channel, then tear down the connection

source

VoiceClient.disconnect

async def disconnect():

Tear down voice ws/UDP without notifying the main gateway (used when Discord already removed us)

leave is the opposite of join: it tells the main gateway we are gone, then tears down. disconnect is the teardown alone, for when Discord has already removed us (kick, channel delete).

await vc.join(debug=True)
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0

With the session up, the cell watches the UDP keepalive on the real socket. It wraps sendto without replacing it and records the 8-byte pings.

await asyncio.wait_for(vc.sess.wait(), 15)
assert vc._ka_task and not vc._ka_task.done()
pings = []
tr = vc.trans
_orig_sendto = tr.sendto
tr.sendto = lambda d, addr=None: (pings.append(d), _orig_sendto(d))[1]
await asyncio.sleep(5.5)
del tr.sendto
test_ne(pings, [])
test_eq(set(map(len, pings)), {8})
voice json 15 {'any': 100}

Audio arrives as RTP packets over UDP. Each packet has a 12-byte or longer RTP header, an encrypted payload, and a 4-byte nonce at the end. The header carries the sequence number, the timestamp, and the SSRC that identifies the speaker. The payload is Opus audio wrapped by DAVE, then encrypted with Discord’s transport key secret_key.

Decryption has two stages. The transport layer uses aead_xchacha20_poly1305_rtpsize. The cipher takes a 24-byte nonce. Only 4 bytes travel in the packet. The client pads them with 20 zero bytes. The plain RTP header is the AAD: authenticated, not encrypted. In rtpsize mode the AAD covers the 12-byte base header, any CSRC entries, and the 4-byte extension preamble only. The extension data itself is inside the encrypted payload. Its length is in the preamble, as a count of 32-bit words. Packets without the extension bit have no extension data. The frame that remains is DAVE-protected. self.dave.decrypt unwraps it using the user id from the SSRC map. Voice SPEAKING events fill that map. The result is the Opus payload.

A packet whose second byte is 0x78 is RTP voice data. Other values, such as 0xC9, are RTCP control packets for quality reporting. The client skips them.

source

VoiceClient.decrypt

def decrypt(
    pkt, uid
):

An earlier version skipped a fixed 8 bytes of extension. It corrupted audio whenever the extension was not two words. On packets with no extension it discarded real Opus data. Discord offers no way to request a particular extension size. The cell below builds RTP packets by hand and runs them through the real cipher: extensions of 0, 2, and 3 words, no extension, and no DAVE session. Without DAVE the transport plaintext is the Opus frame.

_key = bytes(range(32))
def _mk_pkt(payload, ext_words=2, has_ext=True):
    hdr = struct.pack('>BBHII', 0x80 | (0x10 if has_ext else 0), 0x78, 1, 960, 1234)
    if has_ext: hdr += struct.pack('>HH', 0xBEDE, ext_words)
    n4 = (7).to_bytes(4, 'big')
    enc = xchacha_encrypt((bytes(4 * ext_words) if has_ext else b'') + payload, hdr, n4 + b'\0'*20, _key)
    return hdr + enc + n4
fvc = object.__new__(VoiceClient)
fvc.secret_key = _key
cases = [dict(ext_words=0), dict(ext_words=2), dict(ext_words=3), dict(has_ext=False)]
test_eq([fvc.decrypt(_mk_pkt(b'opus!', **kw), 0) for kw in cases], [b'opus!']*len(cases))

decode turns RTP voice packets into PCM and keeps each speaker’s stream in order without a jitter buffer. It compares RTP sequence numbers modulo 16 bits. It accepts wraparound from 65535 to 0. It drops duplicates and packets behind the last accepted sequence. A forward gap means packet loss. Small gaps use Opus packet-loss concealment. Larger gaps use silence.

decode decrypts and decodes a packet before its sequence and timestamp become accepted state. A corrupt packet cannot move the stream backwards or poison the next gap calculation. decode discards a reordered packet rather than buffering it. Recording stays simple at the cost of an occasional 20 ms loss.

source

VoiceClient.decode

def decode(
    pkt
):

source

silence

def silence(
    n_smpls:int, # 2 bytes per sample (s16le)
):

Recording opens one ffmpeg process per speaker and writes that speaker’s decoded PCM to its stdin. Each file covers the whole session. _written counts the samples on disk per user. Before each write the track is padded with silence up to the current wall-clock position. Audio therefore sits at the moment the user spoke it, whether the user joined late, paused, or left and rejoined. Padding happens only when a track is more than 200 ms behind. While someone is speaking, each packet appends its 20 ms and the track keeps pace. Arrival jitter never inserts silence mid-sentence. A packet whose SSRC has had no Speaking event yet has no owner, and is dropped rather than recorded under the SSRC.

start_recording drains the UDP queue first. Stale packets from before the recording never reach the new files.

stop_recording pads every track with silence out to the stop time. All files are then session length and aligned. It closes each ffmpeg stdin and waits for the process. That finalizes the containers. Without it the files can look valid but lack the buffered audio at the end.

source

VoiceClient.stop_recording

async def stop_recording(
    mix:bool=True, mix_path:NoneType=None, timeout:int=10
):

source

VoiceClient.mix_recording

async def mix_recording(
    mix_path:NoneType=None, timeout:NoneType=None, **out_kw
):

Mix the per-speaker files (all session-length, so already aligned) into one recording at mix_path; out_kw passes ffmpeg output options

source

VoiceClient.start_recording

def start_recording(
    path:str='/tmp/recording.mp3'
):

The mix can be slow. mix_recording is separate from stop_recording to let a bot leave the channel first. Call stop_recording(mix=False), then leave(), then mix_recording(). Output options pass through to ffmpeg, for example ar=16000, ac=1 for speech-to-text.

vc.start_recording()
'/tmp/recording.mp3'
out = await vc.stop_recording()
out
{'speakers': {}, 'mixed': None}

Outgoing audio is the receive path in reverse. The client encodes PCM to Opus. It wraps the frame with DAVE when the E2EE session is ready. It encrypts the frame with Discord’s transport key and sends it as an RTP packet. A bot alone in a channel negotiates a DAVE protocol version but has no MLS group until the epoch starts. Until then clients send plain Opus. davey raises NotReady if asked to encrypt earlier. The same applies briefly during epoch transitions.

The RTP sequence, timestamp, and nonce advance by hand for each 20 ms frame. If they drift or repeat, Discord hears broken audio or nothing.

source

VoiceClient.send_pkt

def send_pkt(
    payload
):

source

VoiceClient.encode

def encode(
    fr
):

DAVE-wrapped frames end with the marker 0xFAFA. Whether the encoder output carries it must agree with dave.ready. A bot alone in the channel has a negotiated version but no epoch. It sends plain Opus.

test_eq(vc.encode(silence(spf)).endswith(b'\xfa\xfa'), vc.dave.ready)

The bot must tell Discord when it starts and stops speaking. When a transmission ends, the docs require five frames of Opus silence (0xF8 0xFF 0xFE). Without them receivers interpolate the tail of the audio into the next utterance. The result is an audible garbled blip at the end of playback. Audio that arrives in chunks, such as a realtime TTS feed, should pass end=False for all but the last chunk. The transmission then stays continuous. The finally clears the speaking flag if playback fails halfway through.

source

VoiceClient.send_frames

async def send_frames(
    frames, end:bool=True
):

Send 20ms Opus frames; unless end=False, finish with 5 silence frames and clear the speaking flag.

source

VoiceClient.send_frame

def send_frame(
    fr
):

source

VoiceClient.speaking

async def speaking(
    on:bool=True
):

Discord voice expects 20 ms Opus frames. Padding the final partial frame preserves the tail of short clips instead of silently dropping it.

source

VoiceClient.send_pcm

async def send_pcm(
    pcm, end:bool=True
):

source

pcm2frames

def pcm2frames(
    pcm
):
t = np.arange(sr) / sr
mono = (0.08 * np.sin(2*np.pi*440*t) * 32767).astype(np.int16)
pcm = np.column_stack([mono, mono]).ravel().tobytes()
len(pcm)
192000
await vc.send_pcm(pcm)

Wrapping send_pkt records each packet while it still goes out. Every send_pcm ends with exactly five extra packets, the silence frames. With end=False the transmission stays open and has no tail.

n_frames = len(list(pcm2frames(pcm)))
sent = []
_orig_send_pkt = vc.send_pkt
vc.send_pkt = lambda p: (sent.append(p), _orig_send_pkt(p))[1]
await vc.send_pcm(pcm)
test_eq(len(sent), n_frames + 5)
sent.clear()
await vc.send_pcm(pcm, end=False)
del vc.send_pkt
test_eq(len(sent), n_frames)
await vc.speaking(False)
DEBUG: Received Opcode: 11
await vc._reconnect(resume=True)
await asyncio.wait_for(vc.resumed.wait(), 5)
await vc.send_pcm(pcm)

The cell above called _reconnect directly. A real drop arrives as ConnectionClosed in the listener. The listener classifies the close code (docs) and schedules recovery as a separate task. A listener that ran the reconnect inline could cancel its own recovery. That bug existed once. Closing the live socket with a transient code exercises the resume path end to end.

vc.resumed.clear()
await vc.ws.close(code=4000)
await asyncio.wait_for(vc.resumed.wait(), 20)
await vc.send_pcm(pcm)
voice gateway closed: 4000

Close codes 4003 to 4006, 4009, and 4011 mean the session’s credentials are dead. Resuming with them loops forever. The client must rejoin through the main gateway. The main gateway issues fresh credentials on rejoin. Resuming with a corrupted token produces this for real. Discord answers 4003 Not authenticated. A fake test would have guessed 4006 and encoded the wrong policy. The client should come back with a fresh token and a working session with no help.

vc.token = 'not-a-real-token'
vc.resumed.clear()
await vc.ws.close(code=4000)
for _ in range(60):
    if vc.sess.is_set() and vc.token != 'not-a-real-token': break
    await asyncio.sleep(0.5)
test_ne(vc.token, 'not-a-real-token')
assert vc.sess.is_set()
await vc.send_pcm(pcm)
voice gateway closed: 4000 
voice gateway closed: 4003 Not authenticated.
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
voice json 15 {'any': 100}

Playback from a file goes through ffmpeg:

source

VoiceClient.play_file

async def play_file(
    path
):

source

file2pcm

def file2pcm(
    path
):
await vc.play_file("/tmp/recording_mixed.mp3")

Discord never sends a bot its own audio back. A solo send gets only RTCP in return. The receive test therefore uses a second bot in the channel, DISCORD_BOT_TOKEN_B (see .env.example).

dcb = DiscordClient(os.environ['DISCORD_BOT_TOKEN_B'])
gcb = GatewayClient(intents, dcb)
await gcb.start()
while not getattr(gcb, 'user_id', None): await asyncio.sleep(0.05)
vcb = VoiceClient(gcb, gid, vch)
await vcb.join()
await asyncio.wait_for(vcb.sess.wait(), 15)
test_ne(gcb.user_id, gc.user_id)
vcb
DEBUG: Received Opcode: 0
voice json 11 {'user_ids': ['1361823507679543306']}
voice json 18 {'user_id': '1361823507679543306', 'flags': None}
voice json 20 {'user_id': '1361823507679543306', 'platform': None}
VoiceClient(self.ch=Channel(id=1493461896139903029, name='General', type=2))

With two members the MLS group forms. The recorder here decodes DAVE-wrapped frames. A solo bot never reaches that path.

vc.start_recording('/tmp/twobot.mp3')
await asyncio.sleep(1)
t_send = time.time()
await vcb.send_pcm(pcm)
res = await vc.stop_recording()
sess_len = time.time() - vc._rec_start
uid = int(gcb.user_id)
assert vc.dave.ready and vcb.dave.ready
test_eq(list(res['speakers']), [uid])
res
{'speakers': {1361823507679543306: '/tmp/twobot_1361823507679543306.mp3'},
 'mixed': '/tmp/twobot_mixed.mp3'}

200 ms is the padding threshold. A track can never legitimately lag the clock by more.

track = np.frombuffer(file2pcm(res['speakers'][uid]), np.int16)
mixed = np.frombuffer(file2pcm(res['mixed']), np.int16)
onset = np.flatnonzero(np.abs(track) > 500)[0] / n_chs / sr
dur, mixed_dur = len(track) / n_chs / sr, len(mixed) / n_chs / sr
test_close(onset, t_send - vc._rec_start, 0.2)
test_close(dur, sess_len, 0.2)
test_close(mixed_dur, dur, 0.05)
{k: round(float(v), 3) for k,v in dict(onset=onset, sent_at=t_send - vc._rec_start, dur=dur, sess_len=sess_len).items()}
{'onset': 1.049, 'sent_at': 1.001, 'dur': 2.127, 'sess_len': 2.162}

The second bot leaves before the disconnect scenarios below, which concern this bot’s own connection.

await vcb.leave()
await gcb.stop()
DEBUG: Received Opcode: 0

The last case is an admin disconnecting the bot. Discord sends the bot’s own VOICE_STATE_UPDATE with a null channel and closes the voice socket with 4014. The client must tear down, finalize any recording in progress, and not rejoin. The cell does this to the bot over REST.

vc.start_recording('/tmp/kick_test.mp3')
await asyncio.sleep(1)
await vch('PATCH', f'/guilds/{gld.id}/members/{gc.user_id}', channel_id=None)
for _ in range(30):
    if not vc.running: break
    await asyncio.sleep(0.5)
assert not vc.running and not vc._recording
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
DEBUG: Received Opcode: 0
removed from voice channel; disconnecting

Recording can continue across reconnect types.

vc.start_recording()
await asyncio.sleep(2)
procs_before, start_before = dict(vc._rec_procs), vc._rec_start
await vc._reconnect(resume=False)
await asyncio.sleep(3)
await vc.stop_recording()
{'speakers': {}, 'mixed': None}
await vc.play_file("/tmp/recording_mixed.mp3")
await vc.leave()