•detach — Detach
•detach x gives the value of x as a constant. Under ∂, no derivative flows through the result, at any level of nested ∂. Outside ∂, •detach x gives x. Nested arrays and records are detached throughout.
{+/ω×ω}∂ 1 2 3 ⍝ 2 4 6
{+/ωוdetach ω}∂ 1 2 3 ⍝ 1 2 3
{ωוdetach ω}∂∂ 3 ⍝ 0A value computed before a ∂ call is already a constant inside it. •detach is needed only inside one ∂ call, for a term that comes from the argument but must carry no gradient, such as a target built from the same parameters, or statistics treated as constants.
Apply •detach to a computation’s inputs rather than to its result. net •detach θ runs net on plain values, which costs the same as running it without ∂, as PyTorch’s torch.no_grad() does. •detach net θ runs net under ∂ first and then discards its derivative, as .detach() does. Only net •detach θ saves the work of differentiating net.
A common use is to detach a shift that keeps a result finite. Logsumexp subtracts the largest item before * and adds it back after ⊛. The shift drops out of the gradient. Detaching the input of ⌈/, as here, gives the same gradient and keeps ∂ from differentiating ⌈/:
lse←{m←⌈/•detach ω ◇ m+⊛+/*ω-m}
lse 1000 1001 1002 ⍝ 1002.407606
lse∂ 1 2 3 ⍝ 0.09003057317 0.2447284711 0.6652409558∂ differentiates a function.