<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="https://willagio.com/feed.xml" rel="self" type="application/atom+xml"/><link href="https://willagio.com/" rel="alternate" type="text/html" hreflang="en"/><updated>2026-11-17T15:26:57+00:00</updated><id>https://willagio.com/feed.xml</id><title type="html">blank</title><subtitle>I play piano, percussion, and computers. One has keys, one you hit — somehow the third needs both. </subtitle><entry><title type="html">Backpropagation Through Time (BPTT) 이해하기</title><link href="https://willagio.com/blog/2026/bptt-in-rnn/" rel="alternate" type="text/html" title="Backpropagation Through Time (BPTT) 이해하기"/><published>2026-11-09T00:00:00+00:00</published><updated>2026-11-09T00:00:00+00:00</updated><id>https://willagio.com/blog/2026/bptt-in-rnn</id><content type="html" xml:base="https://willagio.com/blog/2026/bptt-in-rnn/"><![CDATA[<p>RNN은 이전 시점의 Hidden State를 이용해 현재 Hidden State를 계산한다.</p> \[a_t=Ux_t+Vh_{t-1}+b\] \[h_t=\tanh(a_t)\] <p>여기서 $U,V,b$는 학습 가능한 파라미터이며, 모든 시점에서 동일하게 사용된다.</p> <p>BPTT(Backpropagation Through Time)는 RNN을 시간 방향으로 펼친 뒤, 각 시점에 걸쳐 Gradient를 역전파하는 과정이다.</p> <h2 id="1-rnn을-시간-방향으로-펼치기">1. RNN을 시간 방향으로 펼치기</h2> <p>세 시점에 걸친 RNN을 생각해 보자.</p> <div class="tikz-diagram"> <span class="tikz-loading" role="status">도형을 불러오는 중…</span> <div class="tikz-frame" style="width: 745px; height: 157px"> <script type="text/tikz">
\begin{tikzpicture}[
  x=3cm, y=1.8cm,
  >=stealth,
  line width=0.6pt,
  state/.style={
    draw, rounded corners=3pt,
    minimum width=16mm,
    minimum height=11mm
  },
  every node/.style={font=\large}
]
\node[state] (h0) at (-1,0) {$h_0$};

\foreach \t/\x in {1/0,2/2,3/4} {
  \node[state] (a\t) at (\x,0) {$a_\t$};
  \node[state] (h\t) at (\x+1,0) {$h_\t$};
  \node (x\t) at (\x,-1) {$x_\t$};
  \node (l\t) at (\x+1,1) {$L_\t$};

  \draw[->] (x\t) -- (a\t);
  \draw[->] (a\t) -- node[above=3pt] {$\tanh$} (h\t);
  \draw[->] (h\t) -- (l\t);
}

\draw[->] (h0) -- node[above=3pt] {$V$} (a1);
\draw[->] (h1) -- node[above=3pt] {$V$} (a2);
\draw[->] (h2) -- node[above=3pt] {$V$} (a3);
\end{tikzpicture}
</script> </div> </div> <p>각 Hidden State는 다음 시점의 계산에 사용된다.</p> \[h_0\rightarrow h_1\rightarrow h_2\rightarrow h_3\] <p>Language Modeling처럼 모든 시점에서 예측을 수행한다면 전체 Loss는 다음과 같이 정의할 수 있다.</p> \[L=L_1+L_2+L_3\] <p>여기서 중요한 것은 <strong>$V$가 세 시점에 모두 등장한다는 점</strong>이다.</p> <p>따라서 $V$에 대한 전체 Gradient를 구하려면 각 시점에서 $V$가 Loss에 미친 영향을 모두 합해야 한다.</p> <h2 id="2-우리가-구하려는-gradient">2. 우리가 구하려는 Gradient</h2> <p>목표는 다음과 같다.</p> \[\frac{\partial L}{\partial V}\] <p>하지만 $V$는 $a_1,a_2,a_3$를 계산하는 데 반복해서 사용된다.</p> <p>따라서 각 시점에서의 기여를 합하면</p> \[\frac{\partial L}{\partial V} = \left.\frac{\partial L}{\partial V}\right|_{t=1} + \left.\frac{\partial L}{\partial V}\right|_{t=2} + \left.\frac{\partial L}{\partial V}\right|_{t=3}\] <p>가 된다.</p> <p>여기서 각 항은 <strong>해당 시점에서 사용된 $V$가 전체 Loss에 미치는 기여</strong>를 의미한다.</p> <p>이를 계산하려면 먼저 각 $a_t$에 대한 전체 Loss의 Gradient가 필요하다.</p> \[\frac{\partial L}{\partial a_1}, \qquad \frac{\partial L}{\partial a_2}, \qquad \frac{\partial L}{\partial a_3}\] <p>왜 $a_t$일까?</p> <p>$V$가 직접 사용되는 연산이</p> \[a_t=Ux_t+Vh_{t-1}+b\] <p>이기 때문이다.</p> <p>따라서 $a_t$까지 역전파된 Gradient를 알면, Chain Rule을 이용해 해당 시점의 $V$에 대한 기여를 구할 수 있다.</p> <p>이제 이 Gradient들을 마지막 시점부터 계산해 보자.</p> <h2 id="3-마지막-시점부터-역전파하기">3. 마지막 시점부터 역전파하기</h2> <p>먼저 $a_3$에 대한 Gradient를 구한다.</p> \[\frac{\partial L}{\partial a_3}\] <p>$a_3$는 $h_3$를 통해 $L_3$에 영향을 준다. 반면 이전 시점의 $L_1,L_2$에는 영향을 주지 않는다.</p> <p>따라서 Chain Rule에 의해</p> \[\frac{\partial L}{\partial a_3} = \frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3}\] <p>이다.</p> <p>$h_3=\tanh(a_3)$이므로</p> \[\frac{\partial h_3}{\partial a_3}=1-h_3^2\] <p>이다. 위 식은 이해를 위해 Scalar RNN으로 쓴 것이며, 벡터 RNN에서는 원소별 미분을 적용한다.</p> <p>결국</p> \[\frac{\partial L}{\partial a_3} = \frac{\partial L_3}{\partial h_3}(1-h_3^2)\] <p>를 얻는다.</p> <p>여기까지는 일반적인 Backpropagation과 다르지 않다.</p> <h3 id="이번에는-a_2를-살펴보자">이번에는 $a_2$를 살펴보자</h3> <p>$a_2$를 선택하는 이유는 <strong>현재 Loss와 미래 Loss에 동시에 영향을 주는 중간 시점</strong>이기 때문이다.</p> <p>$a_2$는 $h_2$를 거쳐 두 경로로 Loss에 영향을 준다.</p> <div class="tikz-diagram"> <span class="tikz-loading" role="status">도형을 불러오는 중…</span> <div class="tikz-frame" style="width: 574px; height: 111px"> <script type="text/tikz">
\begin{tikzpicture}[
  x=4.5cm, y=1.8cm,
  >=stealth,
  line width=0.6pt,
  state/.style={
    draw, rounded corners=3pt,
    minimum width=16mm,
    minimum height=11mm
  },
  every node/.style={font=\large}
]
\node[state] (a2) at (0,0) {$a_2$};
\node[state] (h2) at (1,0) {$h_2$};
\node[state] (a3) at (2,0) {$a_3$};
\node[state] (h3) at (3,0) {$h_3$};

\node[state] (l2) at (1,1) {$L_2$};
\node[state] (l3) at (3,1) {$L_3$};

\draw[->] (a2) -- node[above=3pt] {$\tanh$} (h2);
\draw[->] (h2) -- node[above=3pt] {$V$} (a3);
\draw[->] (a3) -- node[above=3pt] {$\tanh$} (h3);

\draw[->,thick,blue] (h2) -- (l2);
\draw[->,thick,red] (h3) -- (l3);
\end{tikzpicture}
</script> </div> </div> <p>첫 번째는 현재 시점의 Loss로 이어지는 경로다.</p> \[a_2\rightarrow h_2\rightarrow L_2\] <p>두 번째는 미래 시점의 Loss로 이어지는 경로다.</p> \[a_2\rightarrow h_2\rightarrow a_3\rightarrow h_3\rightarrow L_3\] <p>따라서 두 경로의 Gradient를 합산해야 한다.</p> \[\begin{aligned} \frac{\partial L}{\partial a_2} ={}&amp; \frac{\partial L_2}{\partial h_2} \frac{\partial h_2}{\partial a_2}\\ &amp;+ \frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3} \frac{\partial a_3}{\partial h_2} \frac{\partial h_2}{\partial a_2} \end{aligned}\] <p>두 항에서 공통으로 등장하는 $\partial h_2/\partial a_2$를 묶으면</p> \[\frac{\partial L}{\partial a_2} = \left( \frac{\partial L_2}{\partial h_2} + \frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3} \frac{\partial a_3}{\partial h_2} \right) \frac{\partial h_2}{\partial a_2}\] <p>가 된다.</p> <p>여기서 이미 계산한 결과를 활용할 수 있다.</p> \[\frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3} = \frac{\partial L}{\partial a_3}\] <p>또한 Scalar RNN에서는</p> \[\frac{\partial a_3}{\partial h_2}=v\] <p>이고,</p> \[\frac{\partial h_2}{\partial a_2}=1-h_2^2\] <p>이다.</p> <p>따라서</p> \[\boxed{ \frac{\partial L}{\partial a_2} = \left( \frac{\partial L_2}{\partial h_2} + v\frac{\partial L}{\partial a_3} \right)(1-h_2^2) }\] <p>를 얻는다.</p> <p>이 식에서 중요한 점은 <strong>다음 시점의 Gradient를 재사용한다</strong>는 것이다.</p> <h2 id="4-delta의-등장">4. Delta의 등장</h2> <p>앞에서 반복적으로 등장한 미분을 다음처럼 정의하자.</p> \[\delta_t=\frac{\partial L}{\partial a_t}\] <p>그러면 방금 유도한 식을 간단하게 표현할 수 있다.</p> \[\delta_2= \left( \frac{\partial L_2}{\partial h_2} +v\delta_3 \right)(1-h_2^2)\] <p>같은 방식으로 $t=1$에서도</p> \[\delta_1= \left( \frac{\partial L_1}{\partial h_1} +v\delta_2 \right)(1-h_1^2)\] <p>가 된다.</p> <p>이를 벡터 RNN으로 일반화하면 다음과 같다.</p> \[\boxed{ \delta_t= \left( \frac{\partial L_t}{\partial h_t} + V^\top\delta_{t+1} \right) \odot(1-h_t^2) }\] <p>여기서 각 항의 의미는 다음과 같다.</p> <ul> <li>$\partial L_t/\partial h_t$: 현재 시점의 Loss가 전달하는 Gradient</li> <li>$V^\top\delta_{t+1}$: 미래 시점에서 전달되는 Gradient</li> <li>$1-h_t^2$: Tanh의 Derivative</li> </ul> <p>$\delta_{t+1}$에는 이미 그 이후의 Loss에 대한 영향이 포함되어 있다.</p> <p>따라서 역전파는 마지막 시점부터 시작해, 이전 시점으로 이동하며 계산한 Gradient를 재사용할 수 있다.</p> <h2 id="5-공유-가중치의-gradient-계산">5. 공유 가중치의 Gradient 계산</h2> <p>이제 처음의 목표였던 $\partial L/\partial V$로 돌아가자.</p> <p>각 시점에서는</p> \[a_t=Ux_t+Vh_{t-1}+b\] <p>를 계산한다.</p> <p>해당 시점에서 $V$에 대한 Gradient 기여는</p> \[\delta_t h_{t-1}^\top\] <p>이다.</p> <p>따라서 전체 Gradient는</p> \[\boxed{ \frac{\partial L}{\partial V} = \sum_{t=1}^{T}\delta_t h_{t-1}^\top }\] <p>가 된다.</p> <p>여기서 두 가지 계산이 이루어진다.</p> <p>첫째, <strong>$\delta_t$를 역순으로 계산하며 미래 Loss의 영향을 반영한다.</strong></p> <p>둘째, <strong>동일한 $V$를 사용한 모든 시점의 Gradient 기여를 합산한다.</strong></p> <p>이것이 BPTT의 핵심이다.</p> <h2 id="6-lstm에도-적용할-수-있을까">6. LSTM에도 적용할 수 있을까?</h2> <p>LSTM도 이전 State를 다음 시점으로 전달하고, 동일한 파라미터를 여러 시점에서 사용한다.</p> <p>따라서 BPTT가 그대로 적용된다.</p> <p>다만 Vanilla RNN은 Hidden State $h_t$를 중심으로 계산하지만, LSTM은 Hidden State $h_t$와 Cell State $c_t$를 함께 사용한다.</p> \[c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t\] \[h_t=o_t\odot\tanh(c_t)\] <p>따라서 LSTM에서는 두 State를 통한 Gradient 경로와 Gate의 Gradient까지 계산해야 한다.</p> <p>BPTT의 원리는 동일하지만, 각 시점의 계산 그래프가 더 복잡하다.</p> <h2 id="7-vanishing-gradient">7. Vanishing Gradient</h2> <p>BPTT에서는 Gradient가 여러 시점을 지나며 Recurrent Weight와 Activation Derivative를 반복적으로 곱한다.</p> <p>이 과정에서 Gradient가 작아지면 Vanishing Gradient, 커지면 Exploding Gradient 문제가 발생할 수 있다.</p> <p>LSTM은 Cell State와 Gate를 사용해 장기적인 Gradient 전달을 개선하도록 설계되었다.</p> <h2 id="정리">정리</h2> <p>BPTT를 이해하는 데 중요한 것은 세 가지다.</p> <ol> <li>각 Hidden State는 현재 Loss와 미래 Loss에 모두 영향을 줄 수 있다.</li> <li>Chain Rule을 이용해 Gradient를 마지막 시점부터 역전파한다.</li> <li>동일한 파라미터를 여러 시점에서 사용하므로 Gradient 기여를 합산한다.</li> </ol> <p>Vanilla RNN과 LSTM 모두 같은 원리로 학습한다. 차이는 시간에 따라 역전파해야 하는 내부 계산 구조에 있다.</p>]]></content><author><name></name></author><category term="deep-learning"/><category term="rnn"/><category term="lstm"/><category term="backpropagation"/><category term="deep-learning"/><category term="bptt"/><category term="time-series"/><summary type="html"><![CDATA[BPTT는 RNN에서 시간적 의존성을 따라 Gradient를 역전파하는 알고리즘이다. 이 글에서는 BPTT의 원리와 계산 방법을 설명한다.]]></summary></entry><entry><title type="html">Understanding Backpropagation Through Time (BPTT)</title><link href="https://willagio.com/en/blog/2026/bptt-in-rnn/" rel="alternate" type="text/html" title="Understanding Backpropagation Through Time (BPTT)"/><published>2026-11-09T00:00:00+00:00</published><updated>2026-11-09T00:00:00+00:00</updated><id>https://willagio.com/en/blog/2026/bptt-in-rnn</id><content type="html" xml:base="https://willagio.com/en/blog/2026/bptt-in-rnn/"><![CDATA[<p>An RNN uses the hidden state from the previous time step to compute the current hidden state.</p> \[a_t=Ux_t+Vh_{t-1}+b\] \[h_t=\tanh(a_t)\] <p>Here, $U,V,b$ are learnable parameters shared across all time steps.</p> <p>BPTT (Backpropagation Through Time) unrolls an RNN through time and backpropagates gradients across its time steps.</p> <h2 id="1-unrolling-an-rnn-through-time">1. Unrolling an RNN through time</h2> <p>Consider an RNN over three time steps.</p> <div class="tikz-diagram"> <span class="tikz-loading" role="status">Loading diagram…</span> <div class="tikz-frame" style="width: 745px; height: 157px"> <script type="text/tikz">
\begin{tikzpicture}[
  x=3cm, y=1.8cm,
  >=stealth,
  line width=0.6pt,
  state/.style={
    draw, rounded corners=3pt,
    minimum width=16mm,
    minimum height=11mm
  },
  every node/.style={font=\large}
]
\node[state] (h0) at (-1,0) {$h_0$};

\foreach \t/\x in {1/0,2/2,3/4} {
  \node[state] (a\t) at (\x,0) {$a_\t$};
  \node[state] (h\t) at (\x+1,0) {$h_\t$};
  \node (x\t) at (\x,-1) {$x_\t$};
  \node (l\t) at (\x+1,1) {$L_\t$};

  \draw[->] (x\t) -- (a\t);
  \draw[->] (a\t) -- node[above=3pt] {$\tanh$} (h\t);
  \draw[->] (h\t) -- (l\t);
}

\draw[->] (h0) -- node[above=3pt] {$V$} (a1);
\draw[->] (h1) -- node[above=3pt] {$V$} (a2);
\draw[->] (h2) -- node[above=3pt] {$V$} (a3);
\end{tikzpicture}
</script> </div> </div> <p>Each hidden state is used in the computation at the next time step.</p> \[h_0\rightarrow h_1\rightarrow h_2\rightarrow h_3\] <p>In tasks such as language modeling, where the model makes a prediction at every time step, we can define the total loss as</p> \[L=L_1+L_2+L_3\] <p>The important point is that <strong>$V$ appears at all three time steps</strong>.</p> <p>To compute the total gradient with respect to $V$, we must sum its contributions to the loss across all time steps.</p> <h2 id="2-the-gradient-we-want-to-compute">2. The gradient we want to compute</h2> <p>Our goal is to compute</p> \[\frac{\partial L}{\partial V}\] <p>$V$ is reused in the computations of $a_1,a_2,a_3$.</p> <p>Adding the contributions from each time step gives</p> \[\frac{\partial L}{\partial V} = \left.\frac{\partial L}{\partial V}\right|_{t=1} + \left.\frac{\partial L}{\partial V}\right|_{t=2} + \left.\frac{\partial L}{\partial V}\right|_{t=3}\] <p>Each term represents <strong>the contribution to the total loss from $V$ as used at that time step</strong>.</p> <p>To compute these contributions, we first need the gradient of the total loss with respect to each $a_t$.</p> \[\frac{\partial L}{\partial a_1}, \qquad \frac{\partial L}{\partial a_2}, \qquad \frac{\partial L}{\partial a_3}\] <p>Why $a_t$?</p> <p>Because $V$ is used directly in the operation</p> \[a_t=Ux_t+Vh_{t-1}+b\] <p>Once we know the gradient backpropagated to $a_t$, we can use the chain rule to compute the contribution to the gradient of $V$ at that time step.</p> <p>Let’s compute these gradients, starting at the final time step.</p> <h2 id="3-backpropagating-from-the-final-time-step">3. Backpropagating from the final time step</h2> <p>Start with the gradient with respect to $a_3$.</p> \[\frac{\partial L}{\partial a_3}\] <p>$a_3$ affects $L_3$ through $h_3$. It has no effect on the earlier losses $L_1,L_2$.</p> <p>By the chain rule,</p> \[\frac{\partial L}{\partial a_3} = \frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3}\] <p>Since $h_3=\tanh(a_3)$,</p> \[\frac{\partial h_3}{\partial a_3}=1-h_3^2\] <p>For clarity, these equations describe a scalar RNN. For a vector RNN, we apply the derivatives elementwise.</p> <p>We therefore obtain</p> \[\frac{\partial L}{\partial a_3} = \frac{\partial L_3}{\partial h_3}(1-h_3^2)\] <p>This is the same process as ordinary backpropagation.</p> <h3 id="now-consider-a_2">Now consider $a_2$</h3> <p>We choose $a_2$ because it is <strong>an intermediate time step that affects both the current loss and a future loss</strong>.</p> <p>Through $h_2$, $a_2$ affects the loss along two paths.</p> <div class="tikz-diagram"> <span class="tikz-loading" role="status">Loading diagram…</span> <div class="tikz-frame" style="width: 574px; height: 111px"> <script type="text/tikz">
\begin{tikzpicture}[
  x=4.5cm, y=1.8cm,
  >=stealth,
  line width=0.6pt,
  state/.style={
    draw, rounded corners=3pt,
    minimum width=16mm,
    minimum height=11mm
  },
  every node/.style={font=\large}
]
\node[state] (a2) at (0,0) {$a_2$};
\node[state] (h2) at (1,0) {$h_2$};
\node[state] (a3) at (2,0) {$a_3$};
\node[state] (h3) at (3,0) {$h_3$};

\node[state] (l2) at (1,1) {$L_2$};
\node[state] (l3) at (3,1) {$L_3$};

\draw[->] (a2) -- node[above=3pt] {$\tanh$} (h2);
\draw[->] (h2) -- node[above=3pt] {$V$} (a3);
\draw[->] (a3) -- node[above=3pt] {$\tanh$} (h3);

\draw[->,thick,blue] (h2) -- (l2);
\draw[->,thick,red] (h3) -- (l3);
\end{tikzpicture}
</script> </div> </div> <p>The first path leads to the loss at the current time step.</p> \[a_2\rightarrow h_2\rightarrow L_2\] <p>The second path leads to the loss at a future time step.</p> \[a_2\rightarrow h_2\rightarrow a_3\rightarrow h_3\rightarrow L_3\] <p>We must add the gradients from these two paths.</p> \[\begin{aligned} \frac{\partial L}{\partial a_2} ={}&amp; \frac{\partial L_2}{\partial h_2} \frac{\partial h_2}{\partial a_2}\\ &amp;+ \frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3} \frac{\partial a_3}{\partial h_2} \frac{\partial h_2}{\partial a_2} \end{aligned}\] <p>Factoring out $\partial h_2/\partial a_2$, which appears in both terms, gives</p> \[\frac{\partial L}{\partial a_2} = \left( \frac{\partial L_2}{\partial h_2} + \frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3} \frac{\partial a_3}{\partial h_2} \right) \frac{\partial h_2}{\partial a_2}\] <p>We can reuse a result we have already computed:</p> \[\frac{\partial L_3}{\partial h_3} \frac{\partial h_3}{\partial a_3} = \frac{\partial L}{\partial a_3}\] <p>For a scalar RNN,</p> \[\frac{\partial a_3}{\partial h_2}=v\] <p>and</p> \[\frac{\partial h_2}{\partial a_2}=1-h_2^2\] <p>Thus,</p> \[\boxed{ \frac{\partial L}{\partial a_2} = \left( \frac{\partial L_2}{\partial h_2} + v\frac{\partial L}{\partial a_3} \right)(1-h_2^2) }\] <p>The important point is that <strong>we reuse the gradient from the next time step</strong>.</p> <h2 id="4-introducing-delta">4. Introducing delta</h2> <p>Define the derivative that has appeared repeatedly as</p> \[\delta_t=\frac{\partial L}{\partial a_t}\] <p>This simplifies the expression we just derived:</p> \[\delta_2= \left( \frac{\partial L_2}{\partial h_2} +v\delta_3 \right)(1-h_2^2)\] <p>Likewise, at $t=1$,</p> \[\delta_1= \left( \frac{\partial L_1}{\partial h_1} +v\delta_2 \right)(1-h_1^2)\] <p>Generalizing to a vector RNN gives</p> \[\boxed{ \delta_t= \left( \frac{\partial L_t}{\partial h_t} + V^\top\delta_{t+1} \right) \odot(1-h_t^2) }\] <p>Each term represents the following:</p> <ul> <li>$\partial L_t/\partial h_t$: the gradient from the loss at the current time step</li> <li>$V^\top\delta_{t+1}$: the gradient propagated from future time steps</li> <li>$1-h_t^2$: the derivative of tanh</li> </ul> <p>$\delta_{t+1}$ already includes the effects of all subsequent losses.</p> <p>We can therefore start backpropagation at the final time step and reuse computed gradients as we move backward.</p> <h2 id="5-the-gradient-of-the-shared-weights">5. The gradient of the shared weights</h2> <p>Return to our original goal: $\partial L/\partial V$.</p> <p>At each time step, we compute</p> \[a_t=Ux_t+Vh_{t-1}+b\] <p>The contribution to the gradient of $V$ at that time step is</p> \[\delta_t h_{t-1}^\top\] <p>Summing these contributions gives the total gradient:</p> \[\boxed{ \frac{\partial L}{\partial V} = \sum_{t=1}^{T}\delta_t h_{t-1}^\top }\] <p>This involves two calculations.</p> <p>First, <strong>compute $\delta_t$ in reverse order to account for the effects of future losses.</strong></p> <p>Second, <strong>sum the gradient contributions from every time step that uses the same $V$.</strong></p> <p>This is the core of BPTT.</p> <h2 id="6-does-this-also-apply-to-lstms">6. Does this also apply to LSTMs?</h2> <p>LSTMs pass the previous state forward to the next time step and reuse the same parameters across time steps.</p> <p>BPTT therefore applies to them as well.</p> <p>Vanilla RNNs compute using the hidden state $h_t$, while LSTMs use both a hidden state $h_t$ and a cell state $c_t$.</p> \[c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t\] \[h_t=o_t\odot\tanh(c_t)\] <p>Backpropagation through an LSTM must account for the gradient paths through both states and compute the gradients of the gates as well.</p> <p>The principle of BPTT remains the same, with a more complex computation graph at each time step.</p> <h2 id="7-vanishing-gradients">7. Vanishing gradients</h2> <p>During BPTT, gradient propagation across time steps involves repeated multiplication by recurrent weights and activation derivatives.</p> <p>If the gradients become very small, this leads to vanishing gradients. If they grow very large, it leads to exploding gradients.</p> <p>LSTMs use the cell state and gates to improve gradient flow over longer time spans.</p> <h2 id="summary">Summary</h2> <p>Three points are central to understanding BPTT.</p> <ol> <li>Each hidden state can affect both the current loss and future losses.</li> <li>Use the chain rule to backpropagate gradients, starting at the final time step.</li> <li>Sum the gradient contributions because the same parameters are used at multiple time steps.</li> </ol> <p>Vanilla RNNs and LSTMs learn according to the same principle. They differ in the internal computations through which gradients must propagate over time.</p>]]></content><author><name></name></author><category term="deep-learning"/><category term="rnn"/><category term="lstm"/><category term="backpropagation"/><category term="deep-learning"/><category term="bptt"/><category term="time-series"/><summary type="html"><![CDATA[BPTT backpropagates gradients through temporal dependencies in an RNN. This post explains how BPTT works and how to compute its gradients.]]></summary></entry><entry><title type="html">Convolution의 역전파: 커널과 입력의 기울기 유도하기</title><link href="https://willagio.com/blog/2026/convolution-backpropagation/" rel="alternate" type="text/html" title="Convolution의 역전파: 커널과 입력의 기울기 유도하기"/><published>2026-10-09T00:00:00+00:00</published><updated>2026-10-09T00:00:00+00:00</updated><id>https://willagio.com/blog/2026/convolution-backpropagation</id><content type="html" xml:base="https://willagio.com/blog/2026/convolution-backpropagation/"><![CDATA[<p>Convolution layer의 역전파에서는 두 가지 기울기를 계산한다. 커널에 대한 기울기는 가중치를 학습하는 데 사용하고, 입력에 대한 기울기는 앞선 layer로 전달한다.</p> <p>두 기울기는 같은 순전파 식에서 출발하지만 서로 다른 연산으로 나타난다. 커널의 기울기는 입력과 출력 기울기의 <strong>cross-correlation</strong>, 입력의 기울기는 출력 기울기와 커널의 <strong>full convolution</strong>으로 계산할 수 있다. 각 결과를 chain rule로 유도하고, 커널을 뒤집는 이유까지 살펴보자.</p> <h2 id="순전파와-표기">순전파와 표기</h2> <p>입력 $x$의 크기를 $H\times W$, 커널 $k$의 크기를 $k_1\times k_2$라고 하자. 여기서는 실수 배열, 단일 입력 채널과 단일 커널을 다룬다. Stride와 dilation은 1, padding은 0으로 가정하고 bias는 생략한다. 커널은 입력보다 크지 않다고 가정한다.</p> <p>출력 $y$의 크기는 다음과 같다.</p> \[H_{\mathrm{out}}=H-k_1+1,\qquad W_{\mathrm{out}}=W-k_2+1.\] <p>출력의 각 원소는 입력의 일부 영역과 커널을 원소별로 곱한 뒤 더해 계산한다.</p> \[y(r,c) = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} x(r+a,c+b)\,k(a,b),\] <p>여기서 출력 좌표의 범위는</p> \[0\le r&lt;H_{\mathrm{out}},\qquad 0\le c&lt;W_{\mathrm{out}}\] <p>이다.</p> <p>이 식은 커널을 그대로 이동시키므로 수학적으로는 <strong>cross-correlation</strong>이다. 딥러닝에서는 이 연산을 관례적으로 convolution이라고 부른다. <a href="https://docs.pytorch.org/docs/main/generated/torch.nn.Conv2d.html">PyTorch의 Conv2d 문서</a>도 순전파 연산을 cross-correlation으로 정의한다.</p> <p>손실 $L$은 출력 $y$를 통해 입력과 커널에 의존한다고 가정한다. 뒤쪽 layer에서 전달받는 값은 출력에 대한 손실의 기울기다.</p> \[\frac{\partial L}{\partial y(r,c)}.\] <p>코드에서 흔히 사용하는 <code class="language-plaintext highlighter-rouge">dout</code>이 바로 이 기울기 배열이다. 역전파의 목표는 이 값으로부터 다음 두 기울기를 계산하는 것이다.</p> \[\frac{\partial L}{\partial k(a,b)}, \qquad \frac{\partial L}{\partial x(r,c)}.\] <h2 id="커널에-대한-기울기">커널에 대한 기울기</h2> <p>커널의 한 원소 $k(a’,b’)$를 고정해 생각해 보자. 이 원소는 커널이 이동하는 모든 출력 위치에서 재사용된다. 따라서 손실에 대한 영향을 계산하려면 모든 출력 위치에서 발생한 기여를 더해야 한다.</p> <p>Chain rule을 적용하면</p> \[\frac{\partial L}{\partial k(a',b')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)} \frac{\partial y(r,c)}{\partial k(a',b')}.\] <p>출력 $y$는 배열이므로, chain rule에서도 각 출력 원소를 거치는 기여를 합산해야 한다.</p> <p>순전파 식에서 $k(a’,b’)$가 포함된 항은</p> \[x(r+a',c+b')\,k(a',b')\] <p>이므로,</p> \[\frac{\partial y(r,c)}{\partial k(a',b')} = x(r+a',c+b').\] <p>이를 대입하면 커널의 기울기를 얻는다.</p> \[\boxed{ \frac{\partial L}{\partial k(a',b')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)}\,x(r+a',c+b') }\] <p>각 출력 위치에서 해당 커널 원소와 곱했던 입력값에 출력 기울기를 곱하고, 그 결과를 모두 더하는 식이다. 같은 가중치를 여러 위치에서 공유하므로 기울기도 여러 위치의 기여를 누적한다.</p> <h3 id="cross-correlation으로-해석하기">Cross-correlation으로 해석하기</h3> <p>실수 배열 $u,v$의 cross-correlation을 다음과 같이 정의하자. 합은 $v$의 모든 원소에 대해 취한다.</p> \[(u\star v)(i,j) = \sum_r\sum_c u(i+r,j+c)\,v(r,c).\] <p>그러면 커널 기울기의 식은 그대로</p> \[\frac{\partial L}{\partial k} =x\star\frac{\partial L}{\partial y}\] <p>가 된다. 입력 $x$ 위에서 출력 기울기 $\frac{\partial L}{\partial y}$를 뒤집지 않고 이동시키며, 겹치는 원소끼리 곱해 더하는 연산이다.</p> <p>여기서는 출력 기울기 배열이 입력 안에 완전히 들어오는 위치만 계산하므로 <strong>valid cross-correlation</strong>을 사용한다. 결과의 크기도 커널 크기와 일치한다.</p> \[\begin{aligned} H-H_{\mathrm{out}}+1 &amp;= k_1,\\ W-W_{\mathrm{out}}+1 &amp;= k_2. \end{aligned}\] <h2 id="입력에-대한-기울기">입력에 대한 기울기</h2> <p>이번에는 입력의 한 원소 $x(r’,c’)$를 고정하자. 이 입력값은 자신을 포함하는 여러 입력 영역에서 사용된다. 따라서 입력 기울기도 관련된 모든 출력 위치의 기여를 더해야 한다.</p> \[\frac{\partial L}{\partial x(r',c')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)} \frac{\partial y(r,c)}{\partial x(r',c')}.\] <p>순전파 식을 다시 보면,</p> \[y(r,c) = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} x(r+a,c+b)\,k(a,b).\] <p>이 합에서 $x(r’,c’)$에 해당하는 항은</p> \[r+a=r',\qquad c+b=c'\] <p>를 만족해야 한다. 즉,</p> \[a=r'-r,\qquad b=c'-c\] <p>인 항이다. 해당 좌표가 커널 범위 안에 있을 때 미분값은 그 위치의 커널 원소가 된다.</p> \[\frac{\partial y(r,c)}{\partial x(r',c')} = \begin{cases} k(r'-r,c'-c), &amp; \text{if }0\le r'-r&lt;k_1\text{ and }0\le c'-c&lt;k_2,\\[4pt] 0,&amp;\text{otherwise}. \end{cases}\] <p>이제 합의 인덱스를 출력 좌표 $(r,c)$에서 커널 좌표 $(a,b)$로 바꾸자.</p> \[r=r'-a,\qquad c=c'-b.\] <p>유효한 출력 위치에 대해서는</p> \[\frac{\partial y(r'-a,c'-b)}{\partial x(r',c')} =k(a,b)\] <p>이다. 출력 좌표를 이렇게 선택하면 $x(r’,c’)$와 곱해지는 커널 원소가 정확히 $k(a,b)$가 된다.</p> <p>다만 입력 가장자리에서는 $r’-a$나 $c’-b$가 출력 범위를 벗어날 수 있다. 이하 식에서는 출력 범위 밖의 $\frac{\partial L}{\partial y(r,c)}$를 계산상 0으로 취급한다. 이는 실제로 존재하지 않는 출력 원소까지 미분한다는 뜻이 아니라, 유효한 출력만 합에 기여하도록 하는 zero extension이다.</p> <p>그러면 모든 입력 좌표에 대해 성립하는 식을 얻는다.</p> \[\boxed{ \frac{\partial L}{\partial x(r',c')} = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} \frac{\partial L}{\partial y(r'-a,c'-b)}\,k(a,b) }\] <p>이 식에서 출력 기울기의 인덱스에는 커널 좌표가 <strong>빼기</strong>로 들어간다. 이 형태가 convolution의 핵심이다.</p> <h3 id="stamp-방식-기여를-입력-위치에-찍어-누적하기">Stamp 방식: 기여를 입력 위치에 찍어 누적하기</h3> <p>입력 기울기는 슬라이딩 연산 없이도 계산할 수 있다. 입력과 같은 크기의 배열을 0으로 초기화하고, 각 출력 위치 $(r,c)$에서 다음 값을 해당 입력 위치에 더한다.</p> \[\frac{\partial L}{\partial x(r+a,c+b)} \mathrel{+}= \frac{\partial L}{\partial y(r,c)}\,k(a,b).\] <p>여기서 $\mathrel{+}=$는 그 위치에 기여를 누적한다는 뜻이다. 입력 기울기 배열을 코드에서 <code class="language-plaintext highlighter-rouge">dx</code>라고 부르면, 이 누적은 <code class="language-plaintext highlighter-rouge">dx[r:r+k1, c:c+k2] += dout[r,c] * k</code>에 해당한다. 즉 <code class="language-plaintext highlighter-rouge">dout</code>의 한 원소를 커널 전체에 곱하고, 그 결과를 <code class="language-plaintext highlighter-rouge">dx</code>의 $(r,c)$부터 <strong>stamp처럼 찍어 더한다</strong>. 모든 출력 위치에서 반복하며 겹치는 부분은 더한다. 덮어쓰면 앞서 전달된 기울기가 사라지므로 반드시 <code class="language-plaintext highlighter-rouge">+=</code>로 누적해야 한다.</p> <p>이때 커널은 순전파와 같은 방향으로 사용한다. Stamp 방식에서는 원래 크기의 <code class="language-plaintext highlighter-rouge">dout</code>과 커널을 그대로 사용하므로 명시적으로 padding을 추가할 필요가 없다.</p> <h2 id="왜-full-convolution인가">왜 full convolution인가?</h2> <p>Convolution은 stamp로 누적하거나 커널을 슬라이딩하며 계산할 수 있다. 여기서 <strong>full은 계산 방법이 아니라 출력 범위를 뜻한다</strong>. 두 배열이 일부만 겹치는 가장자리까지 포함해 모든 겹침 위치의 결과를 계산하는 것이 full convolution이다.</p> <p>두 배열의 convolution은 다음 형태로 정의할 수 있다. 합은 $v$의 모든 원소에 대해 취하고, 각 배열은 정의된 범위 밖에서 0으로 취급한다.</p> \[(u*v)(i,j) = \sum_a\sum_b u(i-a,j-b)\,v(a,b).\] <p>입력 기울기의 식과 비교하면</p> \[\frac{\partial L}{\partial x} =\frac{\partial L}{\partial y}*k\] <p>임을 알 수 있다. 앞의 stamp 방식은 이 convolution을 각 출력 위치의 기여로 나누어 누적한 것이다.</p> <p>이때 출력 기울기와 커널이 일부만 겹치는 위치까지 계산해야 한다. 입력 가장자리의 원소도 순전파에 참여했으므로, 그 위치로 돌아가는 기울기가 필요하기 때문이다. 따라서 연산은 <strong>full convolution</strong>이다.</p> <p>결과의 크기는</p> \[\begin{aligned} H_{\mathrm{out}}+k_1-1 &amp;= H,\\ W_{\mathrm{out}}+k_2-1 &amp;= W \end{aligned}\] <p>으로, 입력 크기와 일치한다.</p> <h3 id="cross-correlation으로-계산하려면-커널을-뒤집는다">Cross-correlation으로 계산하려면 커널을 뒤집는다</h3> <p>앞에서 정의한 순전파 연산은 cross-correlation이다. 이 연산으로 입력 기울기를 계산하려면 커널을 높이와 너비 방향으로 모두 뒤집어야 한다.</p> <p>뒤집은 커널을</p> \[k_{\mathrm{flip}}(a,b) = k(k_1-1-a,k_2-1-b)\] <p>라고 하자. 이는 커널을 $180^\circ$ 회전한 것과 같다.</p> <p>입력 기울기 식에서 합의 인덱스를 뒤집으면</p> \[\frac{\partial L}{\partial x(r',c')} = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} \frac{\partial L}{\partial y\bigl( r'-(k_1-1)+a,\, c'-(k_2-1)+b \bigr)} \,k_{\mathrm{flip}}(a,b).\] <p>이 식은 <code class="language-plaintext highlighter-rouge">dout</code>의 위아래에 각각 $k_1-1$, 좌우에 각각 $k_2-1$만큼 zero padding을 적용하고, 뒤집은 커널로 valid cross-correlation을 수행하는 연산이다.</p> <p>Padding이 필요한 이유는 가장자리의 기울기도 계산해야 하기 때문이다. Padding 없이 커널이 <code class="language-plaintext highlighter-rouge">dout</code> 안에 완전히 들어오는 위치만 계산하면 결과 크기는 $(H_{\mathrm{out}}-k_1+1)\times(W_{\mathrm{out}}-k_2+1)$이 된다. 이 크기로는 원래 입력 전체의 기울기를 얻을 수 없고, <code class="language-plaintext highlighter-rouge">dout</code>이 커널보다 작으면 valid 연산 자체가 불가능하다.</p> <p>위아래에 $k_1-1$만큼 padding을 넣으면 valid cross-correlation 결과의 높이는</p> \[H_{\mathrm{out}}+2(k_1-1)-k_1+1 =H_{\mathrm{out}}+k_1-1 =H\] <p>이 된다. 너비도 같은 계산으로 $W$가 된다. 여기의 padding은 역전파를 계산하기 위한 것으로, 순전파에서 padding을 0으로 두었던 가정과 함께 성립한다.</p> <p>따라서 입력 기울기를 계산하는 두 표현은 동등하다.</p> \[\begin{aligned} \frac{\partial L}{\partial x} &amp;=\operatorname{full\_conv}\left(\frac{\partial L}{\partial y},k\right)\\[6pt] &amp;=\operatorname{valid\_corr}\left( \operatorname{pad}\left(\frac{\partial L}{\partial y}\right),k_{\mathrm{flip}} \right). \end{aligned}\] <p>수학적 convolution 식에서는 원래 커널 $k$를 사용한다. 이를 cross-correlation 연산으로 구현할 때 커널을 뒤집는다. Full convolution을 지원하는 연산은 범위 밖을 0으로 처리하므로 호출 전에 직접 padding을 넣을 필요는 없다. Padding을 명시적으로 넣는 것은 기존의 valid cross-correlation으로 같은 결과를 계산하려는 경우다.</p> <h3 id="슬라이딩-한-번과-stamp-한-번은-어떻게-대응할까">슬라이딩 한 번과 stamp 한 번은 어떻게 대응할까?</h3> <p>Convolution을 슬라이딩으로 계산하면 한 위치에서 내적을 구해 <code class="language-plaintext highlighter-rouge">dx</code>의 원소 하나를 완성한다. 반면 stamp 한 번은 <code class="language-plaintext highlighter-rouge">dout</code>의 한 원소에 커널 전체를 곱해 <code class="language-plaintext highlighter-rouge">dx</code>의 여러 위치에 기여를 더한다. 두 방식이 같다는 것은 <strong>모든 위치를 처리한 뒤 얻는 최종 배열이 같다</strong>는 뜻이다.</p> <p>대응 관계를 보기 위해 잠시 1차원 예제를 사용하자.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dout = [1, 2, 3]
k    = [4, 5]
</code></pre></div></div> <p>Full convolution을 슬라이딩으로 계산하려면 <code class="language-plaintext highlighter-rouge">dout</code>의 양쪽에 0을 한 칸씩 채우고, 뒤집은 커널 <code class="language-plaintext highlighter-rouge">[5, 4]</code>로 각 위치의 내적을 구한다.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>padded dout = [0, 1, 2, 3, 0]

dx[0] = [0, 1] · [5, 4] =  4
dx[1] = [1, 2] · [5, 4] = 13
dx[2] = [2, 3] · [5, 4] = 22
dx[3] = [3, 0] · [5, 4] = 15
</code></pre></div></div> <p>Stamp로 계산할 때는 <code class="language-plaintext highlighter-rouge">dx</code>를 0으로 초기화한 뒤, <code class="language-plaintext highlighter-rouge">dout</code>의 각 값에 원래 커널 <code class="language-plaintext highlighter-rouge">[4, 5]</code>를 곱해 해당 위치부터 <code class="language-plaintext highlighter-rouge">+=</code>로 누적한다. 아래 표의 각 행은 stamp 한 번이 더하는 기여다.</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dout[0] = 1:  [4,  5,  0,  0]
dout[1] = 2:  [0,  8, 10,  0]
dout[2] = 3:  [0,  0, 12, 15]
              --------------
sum dx:       [4, 13, 22, 15]
</code></pre></div></div> <p><strong>슬라이딩 한 위치에서 계산하는 값은 stamp 표의 한 열을 모두 더한 값이다.</strong> 예를 들어 <code class="language-plaintext highlighter-rouge">dx[1]</code>은 슬라이딩에서는 $1\cdot5+2\cdot4=13$이라는 내적 한 번으로 구한다. Stamp에서는 첫 번째 stamp가 5를, 두 번째 stamp가 8을 더해서 같은 13을 만든다.</p> <p>2차원에서도 같은 관계가 성립한다. 슬라이딩은 <code class="language-plaintext highlighter-rouge">dx</code>의 한 좌표에 들어올 기여를 모아 계산하고, stamp는 <code class="language-plaintext highlighter-rouge">dout[r,c] * k</code>를 여러 입력 좌표에 나누어 누적한다.</p> <h2 id="작은-예제로-확인하기">작은 예제로 확인하기</h2> <p>다음 입력과 커널을 사용하자.</p> \[x= \begin{bmatrix} 1&amp;2&amp;3\\ 4&amp;5&amp;6\\ 7&amp;8&amp;9 \end{bmatrix}, \qquad k= \begin{bmatrix} 1&amp;2\\ 3&amp;4 \end{bmatrix}.\] <p>순전파 결과는</p> \[y= \begin{bmatrix} 37&amp;47\\ 67&amp;77 \end{bmatrix}\] <p>이다. 예를 들어 왼쪽 위 출력은</p> \[y(0,0) = 1\cdot1+2\cdot2+4\cdot3+5\cdot4 = 37\] <p>로 계산한다.</p> <p>기울기의 흐름을 확인하기 위해 손실을 다음과 같이 정하자.</p> \[L=y(0,0)+2y(1,1).\] <p>그러면 출력 기울기는</p> \[\frac{\partial L}{\partial y} = \begin{bmatrix} 1&amp;0\\ 0&amp;2 \end{bmatrix}.\] <h3 id="커널-기울기">커널 기울기</h3> <p>출력 기울기가 0이 아닌 위치는 $(0,0)$과 $(1,1)$이다. 커널 기울기는 각 위치에서 사용한 입력 영역에 해당 출력 기울기를 곱해 더하면 된다.</p> \[\begin{aligned} \frac{\partial L}{\partial k} &amp;= 1\cdot \begin{bmatrix} 1&amp;2\\ 4&amp;5 \end{bmatrix} + 2\cdot \begin{bmatrix} 5&amp;6\\ 8&amp;9 \end{bmatrix}\\[6pt] &amp;= \begin{bmatrix} 11&amp;14\\ 20&amp;23 \end{bmatrix}. \end{aligned}\] <p>예를 들어 $k(0,0)$는 $y(0,0)$를 계산할 때 $x(0,0)=1$과, $y(1,1)$를 계산할 때 $x(1,1)=5$와 곱해진다. 따라서</p> \[\frac{\partial L}{\partial k(0,0)} = 1\cdot1+2\cdot5 = 11\] <p>이다.</p> <h3 id="입력-기울기">입력 기울기</h3> <p>각 출력은 자신을 계산할 때 사용한 입력 영역으로 기울기를 돌려보낸다. 이때 각 입력으로 전달하는 값은 출력 기울기에 해당 커널 원소를 곱한 값이다.</p> <p>$y(0,0)$의 기울기 1은 왼쪽 위 입력 영역에 $k$를 더하고, $y(1,1)$의 기울기 2는 오른쪽 아래 입력 영역에 $2k$를 더한다.</p> \[\begin{aligned} \frac{\partial L}{\partial x} &amp;= \begin{bmatrix} 1&amp;2&amp;0\\ 3&amp;4&amp;0\\ 0&amp;0&amp;0 \end{bmatrix} + \begin{bmatrix} 0&amp;0&amp;0\\ 0&amp;2&amp;4\\ 0&amp;6&amp;8 \end{bmatrix}\\[6pt] &amp;= \begin{bmatrix} 1&amp;2&amp;0\\ 3&amp;6&amp;4\\ 0&amp;6&amp;8 \end{bmatrix}. \end{aligned}\] <p>중앙 입력 $x(1,1)$에는 두 출력에서 기울기가 전달된다. $y(0,0)$에서는 $k(1,1)=4$, $y(1,1)$에서는 $k(0,0)=1$과 곱해졌으므로,</p> \[\frac{\partial L}{\partial x(1,1)} = 1\cdot4+2\cdot1 = 6\] <p>이다.</p> <h3 id="같은-결과를-패딩과-슬라이딩으로-계산하기">같은 결과를 패딩과 슬라이딩으로 계산하기</h3> <p>이 예제에서 커널은 $2\times2$이므로 <code class="language-plaintext highlighter-rouge">dout</code>의 네 방향에 각각 한 칸씩 0을 채운다. 슬라이딩에 사용할 커널은 $180^\circ$ 회전한다.</p> \[\operatorname{pad}\left(\frac{\partial L}{\partial y}\right) = \begin{bmatrix} 0&amp;0&amp;0&amp;0\\ 0&amp;1&amp;0&amp;0\\ 0&amp;0&amp;2&amp;0\\ 0&amp;0&amp;0&amp;0 \end{bmatrix}, \qquad k_{\mathrm{flip}} = \begin{bmatrix} 4&amp;3\\ 2&amp;1 \end{bmatrix}.\] <p>이 $4\times4$ 배열 위에서 뒤집은 $2\times2$ 커널로 valid cross-correlation을 하면 입력과 같은 $3\times3$ 크기의 기울기를 얻는다. 예를 들어 왼쪽 위에서는</p> \[\frac{\partial L}{\partial x(0,0)} =0\cdot4+0\cdot3+0\cdot2+1\cdot1 =1\] <p>이고, 중앙에서는</p> \[\frac{\partial L}{\partial x(1,1)} =1\cdot4+0\cdot3+0\cdot2+2\cdot1 =6\] <p>이다. Stamp로 누적했던 값과 같다. Padding 없이 valid cross-correlation만 하면 중앙의 값 6 하나만 얻고, 나머지 입력 위치의 기울기는 계산하지 못한다.</p> <p>Stamp 방식은 각 출력이 입력으로 보내는 기여를 찍어서 더한다. 슬라이딩 방식은 각 입력 위치에서 받을 기여를 모아 더한다. 같은 기여를 서로 다른 순서로 합산하므로 두 방식의 결과는 동일하다.</p>]]></content><author><name></name></author><category term="deep-learning"/><category term="convolution"/><category term="cross-correlation"/><category term="backpropagation"/><category term="deep-learning"/><summary type="html"><![CDATA[Convolution layer의 커널과 입력 기울기를 chain rule로 유도하고, cross-correlation과 full convolution의 관계 및 커널을 뒤집는 이유를 설명한다.]]></summary></entry><entry><title type="html">Backpropagation through convolution: deriving kernel and input gradients</title><link href="https://willagio.com/en/blog/2026/convolution-backpropagation/" rel="alternate" type="text/html" title="Backpropagation through convolution: deriving kernel and input gradients"/><published>2026-10-09T00:00:00+00:00</published><updated>2026-10-09T00:00:00+00:00</updated><id>https://willagio.com/en/blog/2026/convolution-backpropagation</id><content type="html" xml:base="https://willagio.com/en/blog/2026/convolution-backpropagation/"><![CDATA[<p>Backpropagation through a convolutional layer computes two gradients. The kernel gradient is used to update the weights, and the input gradient is passed to the preceding layer.</p> <p>Both start from the same forward equation, but they take different forms. The kernel gradient is a <strong>cross-correlation</strong> of the input with the output gradient. The input gradient is a <strong>full convolution</strong> of the output gradient with the kernel. We will derive both using the chain rule and see why the kernel sometimes needs to be flipped.</p> <h2 id="forward-pass-and-notation">Forward pass and notation</h2> <p>Let the input $x$ have shape $H\times W$ and the kernel $k$ have shape $k_1\times k_2$. We use real-valued arrays, a single input channel, and a single kernel. Stride and dilation are 1, padding is 0, and bias is omitted. Assume that the kernel fits within the input.</p> <p>The output $y$ has shape</p> \[H_{\mathrm{out}}=H-k_1+1,\qquad W_{\mathrm{out}}=W-k_2+1.\] <p>Each output element is the sum of elementwise products between an input patch and the kernel:</p> \[y(r,c) = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} x(r+a,c+b)\,k(a,b),\] <p>where the output coordinates satisfy</p> \[0\le r&lt;H_{\mathrm{out}},\qquad 0\le c&lt;W_{\mathrm{out}}\] <p>This operation slides the kernel without flipping it, so its mathematical name is <strong>cross-correlation</strong>. Deep learning convention calls it convolution. The <a href="https://docs.pytorch.org/docs/main/generated/torch.nn.Conv2d.html">PyTorch Conv2d documentation</a> also defines the forward operation as cross-correlation.</p> <p>Assume that the loss $L$ depends on the input and kernel through the output $y$. The value received from the layers that follow is the gradient of the loss with respect to each output element:</p> \[\frac{\partial L}{\partial y(r,c)}.\] <p>This gradient array is commonly called <code class="language-plaintext highlighter-rouge">dout</code> in code. Backpropagation uses it to compute</p> \[\frac{\partial L}{\partial k(a,b)}, \qquad \frac{\partial L}{\partial x(r,c)}.\] <h2 id="gradient-with-respect-to-the-kernel">Gradient with respect to the kernel</h2> <p>Fix one kernel element $k(a’,b’)$. That element is reused at every output position as the kernel slides across the input. Its effect on the loss therefore includes contributions from all output positions.</p> <p>Applying the chain rule gives</p> \[\frac{\partial L}{\partial k(a',b')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)} \frac{\partial y(r,c)}{\partial k(a',b')}.\] <p>Since $y$ is an array, the chain rule sums the contributions through its individual elements.</p> <p>The term containing $k(a’,b’)$ in the forward equation is</p> \[x(r+a',c+b')\,k(a',b')\] <p>so</p> \[\frac{\partial y(r,c)}{\partial k(a',b')} = x(r+a',c+b').\] <p>Substituting this gives the kernel gradient:</p> \[\boxed{ \frac{\partial L}{\partial k(a',b')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)}\,x(r+a',c+b') }\] <p>At each output position, multiply the output gradient by the input value that was paired with this kernel element, then sum the results. Sharing a weight across positions means accumulating its gradient across those positions too.</p> <h3 id="interpreting-this-as-cross-correlation">Interpreting this as cross-correlation</h3> <p>Define cross-correlation for real-valued arrays $u,v$ as follows, with the sum taken over all elements of $v$:</p> \[(u\star v)(i,j) = \sum_r\sum_c u(i+r,j+c)\,v(r,c).\] <p>The kernel gradient is then</p> \[\frac{\partial L}{\partial k} =x\star\frac{\partial L}{\partial y}\] <p>Slide the output gradient $\frac{\partial L}{\partial y}$ across the input $x$ without flipping it, multiply overlapping elements, and sum the products.</p> <p>We use <strong>valid cross-correlation</strong>, which evaluates only positions where the output gradient array fits entirely inside the input. The result has the same shape as the kernel:</p> \[\begin{aligned} H-H_{\mathrm{out}}+1 &amp;= k_1,\\ W-W_{\mathrm{out}}+1 &amp;= k_2. \end{aligned}\] <h2 id="gradient-with-respect-to-the-input">Gradient with respect to the input</h2> <p>Now fix one input element $x(r’,c’)$. It can appear in several input patches used by the forward pass. Its gradient must therefore sum the contributions from every output position that uses it:</p> \[\frac{\partial L}{\partial x(r',c')} = \sum_{r=0}^{H_{\mathrm{out}}-1} \sum_{c=0}^{W_{\mathrm{out}}-1} \frac{\partial L}{\partial y(r,c)} \frac{\partial y(r,c)}{\partial x(r',c')}.\] <p>Recall the forward equation:</p> \[y(r,c) = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} x(r+a,c+b)\,k(a,b).\] <p>The term containing $x(r’,c’)$ must satisfy</p> \[r+a=r',\qquad c+b=c'\] <p>or equivalently,</p> \[a=r'-r,\qquad b=c'-c\] <p>If these coordinates fall within the kernel, the derivative is the kernel element at that position:</p> \[\frac{\partial y(r,c)}{\partial x(r',c')} = \begin{cases} k(r'-r,c'-c), &amp; \text{if }0\le r'-r&lt;k_1\text{ and }0\le c'-c&lt;k_2,\\[4pt] 0,&amp;\text{otherwise}. \end{cases}\] <p>Change the summation indices from output coordinates $(r,c)$ to kernel coordinates $(a,b)$:</p> \[r=r'-a,\qquad c=c'-b.\] <p>For valid output positions,</p> \[\frac{\partial y(r'-a,c'-b)}{\partial x(r',c')} =k(a,b)\] <p>Choosing the output coordinates this way makes $k(a,b)$ the exact kernel element multiplied by $x(r’,c’)$.</p> <p>Near the input boundary, however, $r’-a$ or $c’-b$ may lie outside the output. In the equations below, we treat $\frac{\partial L}{\partial y(r,c)}$ as 0 outside the output bounds. This zero extension ensures that only valid outputs contribute to the sum; it does not introduce derivatives of nonexistent output elements.</p> <p>The resulting formula holds at every input position:</p> \[\boxed{ \frac{\partial L}{\partial x(r',c')} = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} \frac{\partial L}{\partial y(r'-a,c'-b)}\,k(a,b) }\] <p>The kernel coordinates are <strong>subtracted</strong> in the output gradient indices. This is the characteristic form of convolution.</p> <h3 id="the-stamp-method-accumulating-contributions-at-input-positions">The stamp method: accumulating contributions at input positions</h3> <p>We can also compute the input gradient by stamping contributions into an array. Initialize an array with the input shape to 0. At each output position $(r,c)$, add the following contribution to each corresponding input position:</p> \[\frac{\partial L}{\partial x(r+a,c+b)} \mathrel{+}= \frac{\partial L}{\partial y(r,c)}\,k(a,b).\] <p>Here $\mathrel{+}=$ means accumulating a contribution at that position. Calling the input gradient array <code class="language-plaintext highlighter-rouge">dx</code>, this corresponds to <code class="language-plaintext highlighter-rouge">dx[r:r+k1, c:c+k2] += dout[r,c] * k</code>. Multiply one element of <code class="language-plaintext highlighter-rouge">dout</code> by the entire kernel, then <strong>stamp and add</strong> the result into <code class="language-plaintext highlighter-rouge">dx</code>, starting at $(r,c)$. Repeat for every output position, adding overlapping contributions. Assignment would erase earlier contributions, so accumulation with <code class="language-plaintext highlighter-rouge">+=</code> is essential.</p> <p>The kernel keeps the same orientation as in the forward pass. The stamp method uses <code class="language-plaintext highlighter-rouge">dout</code> and the kernel at their original sizes, with no explicit padding.</p> <h2 id="why-full-convolution">Why full convolution?</h2> <p>Convolution can be computed by accumulating stamps or by sliding a kernel. Here, <strong>full specifies the output range, not the computation method</strong>. Full convolution includes every overlap position, including boundary positions where the two arrays overlap only partially.</p> <p>Define convolution as follows, summing over all elements of $v$ and treating values outside either array as 0:</p> \[(u*v)(i,j) = \sum_a\sum_b u(i-a,j-b)\,v(a,b).\] <p>Comparing this with the input gradient formula gives</p> \[\frac{\partial L}{\partial x} =\frac{\partial L}{\partial y}*k\] <p>The stamp method computes this convolution by accumulating the contributions from each output position.</p> <p>We need to include positions where the output gradient and kernel overlap only partially. Input elements at the boundary participated in the forward pass, so their gradients must be computed too. This requires <strong>full convolution</strong>.</p> <p>The result has shape</p> \[\begin{aligned} H_{\mathrm{out}}+k_1-1 &amp;= H,\\ W_{\mathrm{out}}+k_2-1 &amp;= W \end{aligned}\] <p>which matches the input shape.</p> <h3 id="flipping-the-kernel-to-use-cross-correlation">Flipping the kernel to use cross-correlation</h3> <p>The forward operation defined earlier is cross-correlation. To use that operation to compute the input gradient, we must flip the kernel along both its height and width.</p> <p>Define the flipped kernel as</p> \[k_{\mathrm{flip}}(a,b) = k(k_1-1-a,k_2-1-b)\] <p>This is equivalent to rotating the kernel by $180^\circ$.</p> <p>Reversing the summation indices in the input gradient formula gives</p> \[\frac{\partial L}{\partial x(r',c')} = \sum_{a=0}^{k_1-1} \sum_{b=0}^{k_2-1} \frac{\partial L}{\partial y\bigl( r'-(k_1-1)+a,\, c'-(k_2-1)+b \bigr)} \,k_{\mathrm{flip}}(a,b).\] <p>This is valid cross-correlation with the flipped kernel after padding <code class="language-plaintext highlighter-rouge">dout</code> with $k_1-1$ zero rows on both the top and bottom, and $k_2-1$ zero columns on both the left and right.</p> <p>The padding lets us compute the gradients at the boundary. Without padding, evaluating only positions where the kernel fits entirely inside <code class="language-plaintext highlighter-rouge">dout</code> would produce an array of shape $(H_{\mathrm{out}}-k_1+1)\times(W_{\mathrm{out}}-k_2+1)$. For a kernel larger than $1\times1$, this does not cover the whole input. If <code class="language-plaintext highlighter-rouge">dout</code> is smaller than the kernel in either dimension, there are no valid positions at all.</p> <p>With $k_1-1$ rows of padding on each side, the height of the valid cross-correlation result is</p> \[H_{\mathrm{out}}+2(k_1-1)-k_1+1 =H_{\mathrm{out}}+k_1-1 =H\] <p>The width is likewise $W$. This padding belongs to the backward computation and is consistent with using no padding in the forward pass.</p> <p>The two expressions for the input gradient are therefore equivalent:</p> \[\begin{aligned} \frac{\partial L}{\partial x} &amp;=\operatorname{full\_conv}\left(\frac{\partial L}{\partial y},k\right)\\[6pt] &amp;=\operatorname{valid\_corr}\left( \operatorname{pad}\left(\frac{\partial L}{\partial y}\right),k_{\mathrm{flip}} \right). \end{aligned}\] <p>The mathematical convolution expression uses the original kernel $k$. We flip it when implementing that convolution with cross-correlation. An operation that supports full convolution handles values outside the arrays as 0, so it needs no manual padding beforehand. Explicit padding is needed here because we are using valid cross-correlation to compute the same result.</p> <h3 id="how-does-one-sliding-step-correspond-to-one-stamp">How does one sliding step correspond to one stamp?</h3> <p>A sliding convolution computes a dot product at one position to complete one element of <code class="language-plaintext highlighter-rouge">dx</code>. A single stamp multiplies one element of <code class="language-plaintext highlighter-rouge">dout</code> by the entire kernel and adds contributions to several positions in <code class="language-plaintext highlighter-rouge">dx</code>. The equivalence means that <strong>the final arrays match after all positions have been processed</strong>.</p> <p>Consider a one-dimensional example to see the correspondence:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dout = [1, 2, 3]
k    = [4, 5]
</code></pre></div></div> <p>To compute full convolution by sliding, pad <code class="language-plaintext highlighter-rouge">dout</code> with one zero on each side and take dot products with the flipped kernel <code class="language-plaintext highlighter-rouge">[5, 4]</code>:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>padded dout = [0, 1, 2, 3, 0]

dx[0] = [0, 1] · [5, 4] =  4
dx[1] = [1, 2] · [5, 4] = 13
dx[2] = [2, 3] · [5, 4] = 22
dx[3] = [3, 0] · [5, 4] = 15
</code></pre></div></div> <p>For the stamp method, initialize <code class="language-plaintext highlighter-rouge">dx</code> to 0, multiply each value in <code class="language-plaintext highlighter-rouge">dout</code> by the original kernel <code class="language-plaintext highlighter-rouge">[4, 5]</code>, and accumulate with <code class="language-plaintext highlighter-rouge">+=</code> starting at that position. Each row below shows the contribution from one stamp:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dout[0] = 1:  [4,  5,  0,  0]
dout[1] = 2:  [0,  8, 10,  0]
dout[2] = 3:  [0,  0, 12, 15]
              --------------
sum dx:       [4, 13, 22, 15]
</code></pre></div></div> <p><strong>The value computed at one sliding position is the sum of one column in the stamp table.</strong> For example, sliding computes <code class="language-plaintext highlighter-rouge">dx[1]</code> in one dot product: $1\cdot5+2\cdot4=13$. Stamping adds 5 from the first stamp and 8 from the second to obtain the same 13.</p> <p>The same relationship holds in two dimensions. Sliding collects the contributions for one coordinate of <code class="language-plaintext highlighter-rouge">dx</code>. Stamping distributes <code class="language-plaintext highlighter-rouge">dout[r,c] * k</code> across several input coordinates and accumulates those contributions.</p> <h2 id="checking-with-a-small-example">Checking with a small example</h2> <p>Use the following input and kernel:</p> \[x= \begin{bmatrix} 1&amp;2&amp;3\\ 4&amp;5&amp;6\\ 7&amp;8&amp;9 \end{bmatrix}, \qquad k= \begin{bmatrix} 1&amp;2\\ 3&amp;4 \end{bmatrix}.\] <p>The forward pass gives</p> \[y= \begin{bmatrix} 37&amp;47\\ 67&amp;77 \end{bmatrix}\] <p>For example, the top-left output is</p> \[y(0,0) = 1\cdot1+2\cdot2+4\cdot3+5\cdot4 = 37\] <p>To track the gradient contributions, define the loss as</p> \[L=y(0,0)+2y(1,1).\] <p>The output gradient is then</p> \[\frac{\partial L}{\partial y} = \begin{bmatrix} 1&amp;0\\ 0&amp;2 \end{bmatrix}.\] <h3 id="kernel-gradient">Kernel gradient</h3> <p>The output gradient is nonzero only at $(0,0)$ and $(1,1)$. Multiply the input patch used at each position by its output gradient, then add the results:</p> \[\begin{aligned} \frac{\partial L}{\partial k} &amp;= 1\cdot \begin{bmatrix} 1&amp;2\\ 4&amp;5 \end{bmatrix} + 2\cdot \begin{bmatrix} 5&amp;6\\ 8&amp;9 \end{bmatrix}\\[6pt] &amp;= \begin{bmatrix} 11&amp;14\\ 20&amp;23 \end{bmatrix}. \end{aligned}\] <p>For example, $k(0,0)$ multiplies $x(0,0)=1$ when computing $y(0,0)$, and $x(1,1)=5$ when computing $y(1,1)$. Thus,</p> \[\frac{\partial L}{\partial k(0,0)} = 1\cdot1+2\cdot5 = 11\] <h3 id="input-gradient">Input gradient</h3> <p>Each output sends gradient contributions back to the input patch used to compute it. The contribution to each input element is the output gradient multiplied by the corresponding kernel element.</p> <p>The gradient 1 at $y(0,0)$ adds $k$ to the top-left input patch. The gradient 2 at $y(1,1)$ adds $2k$ to the bottom-right input patch:</p> \[\begin{aligned} \frac{\partial L}{\partial x} &amp;= \begin{bmatrix} 1&amp;2&amp;0\\ 3&amp;4&amp;0\\ 0&amp;0&amp;0 \end{bmatrix} + \begin{bmatrix} 0&amp;0&amp;0\\ 0&amp;2&amp;4\\ 0&amp;6&amp;8 \end{bmatrix}\\[6pt] &amp;= \begin{bmatrix} 1&amp;2&amp;0\\ 3&amp;6&amp;4\\ 0&amp;6&amp;8 \end{bmatrix}. \end{aligned}\] <p>The central input $x(1,1)$ receives nonzero contributions from two outputs. It was multiplied by $k(1,1)=4$ in $y(0,0)$ and by $k(0,0)=1$ in $y(1,1)$, so</p> \[\frac{\partial L}{\partial x(1,1)} = 1\cdot4+2\cdot1 = 6\] <h3 id="computing-the-same-result-with-padding-and-sliding">Computing the same result with padding and sliding</h3> <p>The kernel is $2\times2$, so pad <code class="language-plaintext highlighter-rouge">dout</code> with one zero on every side. Rotate the kernel by $180^\circ$ for the sliding computation:</p> \[\operatorname{pad}\left(\frac{\partial L}{\partial y}\right) = \begin{bmatrix} 0&amp;0&amp;0&amp;0\\ 0&amp;1&amp;0&amp;0\\ 0&amp;0&amp;2&amp;0\\ 0&amp;0&amp;0&amp;0 \end{bmatrix}, \qquad k_{\mathrm{flip}} = \begin{bmatrix} 4&amp;3\\ 2&amp;1 \end{bmatrix}.\] <p>Valid cross-correlation of this $4\times4$ array with the flipped $2\times2$ kernel produces a $3\times3$ gradient array, matching the input shape. At the top-left position,</p> \[\frac{\partial L}{\partial x(0,0)} =0\cdot4+0\cdot3+0\cdot2+1\cdot1 =1\] <p>and at the center,</p> \[\frac{\partial L}{\partial x(1,1)} =1\cdot4+0\cdot3+0\cdot2+2\cdot1 =6\] <p>These match the values accumulated by stamping. Without padding, valid cross-correlation would produce only the central value 6, leaving the other input gradients uncomputed.</p> <p>Stamping adds the contributions each output sends to the input. Sliding collects the contributions each input position receives. Both sum the same contributions in a different order and therefore produce the same result.</p>]]></content><author><name></name></author><category term="deep-learning"/><category term="convolution"/><category term="cross-correlation"/><category term="backpropagation"/><category term="deep-learning"/><summary type="html"><![CDATA[Derive the kernel and input gradients of a convolutional layer using the chain rule, and connect cross-correlation, full convolution, kernel flipping, and stamp accumulation.]]></summary></entry><entry><title type="html">왜 하필 Cross Entropy인가: BCE에서 KL Divergence까지</title><link href="https://willagio.com/blog/2026/cross-entropy-to-kl-divergence/" rel="alternate" type="text/html" title="왜 하필 Cross Entropy인가: BCE에서 KL Divergence까지"/><published>2026-08-30T00:00:00+00:00</published><updated>2026-08-30T00:00:00+00:00</updated><id>https://willagio.com/blog/2026/cross-entropy-to-kl-divergence</id><content type="html" xml:base="https://willagio.com/blog/2026/cross-entropy-to-kl-divergence/"><![CDATA[<p>분류 모델을 짤 때 손실 함수 자리에 <code class="language-plaintext highlighter-rouge">CrossEntropyLoss</code>를 넣는 건 거의 반사적이다. 그런데 왜 하필 그건지, 왜 회귀에서 잘 쓰던 MSE(Mean Squared Error)는 안 되는지, <code class="language-plaintext highlighter-rouge">BCELoss</code>와는 뭐가 다른지, 그리고 논문마다 나오는 KL Divergence와는 무슨 관계인지를 이어서 설명하려면 생각보다 막힌다.</p> <p>이 글은 그 사슬을 처음부터 끝까지 한 번에 잇는 것을 목표로 한다. 정보량에서 시작해서 엔트로피, Cross Entropy, BCE, 최대우도추정, KL Divergence 순서로 간다.</p> <h2 id="시작-질문-mse는-왜-안-되나">시작 질문: MSE는 왜 안 되나</h2> <p>이진 분류에서 모델이 $z$라는 logit을 뱉고 sigmoid를 통과시켜 $\hat y = \sigma(z)$를 만든다고 하자. 정답은 $y \in {0, 1}$이다. MSE를 쓰면 손실은 이렇다.</p> \[L_{\text{MSE}} = \frac{1}{2}(\hat y - y)^2\] <p>$z$에 대한 기울기를 구해보면 문제가 드러난다.</p> \[\frac{\partial L_{\text{MSE}}}{\partial z} = (\hat y - y)\,\sigma'(z) = (\hat y - y)\,\hat y (1 - \hat y)\] <p>$\sigma’(z) = \hat y(1-\hat y)$라는 항이 붙어 있다. 이 값은 $\hat y$가 0이나 1에 가까울 때 0으로 죽는다. 그런데 $\hat y$가 0에 가까운데 정답이 1인 상황은 <strong>모델이 가장 크게 틀린 상황</strong>이다. 즉 MSE + sigmoid 조합은 가장 크게 틀렸을 때 기울기가 가장 작아진다. 학습이 필요한 지점에서 학습이 멈춘다.</p> <p>숫자로 보면 심각성이 분명해진다. $y=1$인데 모델이 $z=-8$을 뱉은, 완전히 틀린 경우다.</p> <table> <thead> <tr> <th> </th> <th>$\hat y$</th> <th>$\partial L / \partial z$</th> </tr> </thead> <tbody> <tr> <td>MSE + sigmoid</td> <td>0.000335</td> <td>$-0.000335$</td> </tr> <tr> <td>BCE + sigmoid</td> <td>0.000335</td> <td>$-0.999665$</td> </tr> </tbody> </table> <p>같은 상황인데 기울기 크기가 약 3000배 차이난다. MSE는 사실상 아무것도 배우지 못한다.</p> <p>기울기 문제를 걷어내더라도 더 근본적인 이유가 남는다. 분류 모델의 출력은 숫자가 아니라 <strong>확률분포</strong>다. 두 확률분포가 얼마나 다른지를 재는 데 유클리드 거리를 쓸 이유가 없다. 확률분포에는 확률분포의 척도가 있고, 그 척도는 정보 이론에서 나온다.</p> <h2 id="정보량-놀람의-크기">정보량: 놀람의 크기</h2> <p>“내일 해가 뜬다”는 정보량이 거의 0이다. “내일 눈이 온다”는 8월이라면 정보량이 크다. 드문 사건일수록 정보량이 크다는 직관을 수식으로 옮기려면 어떤 함수여야 할까. 세 가지 조건을 걸어보자.</p> <ol> <li>확률이 낮을수록 정보량이 크다 (단조 감소)</li> <li>확률이 1인 사건의 정보량은 0이다</li> <li>독립인 두 사건을 함께 관측한 정보량은 각각의 합이다: $I(pq) = I(p) + I(q)$</li> </ol> <p>3번이 결정적이다. 곱을 합으로 바꾸는 연속함수는 로그뿐이다. 1번과 2번까지 맞추려면 부호를 뒤집어야 한다.</p> \[I(x) = -\log p(x)\] <p>로그의 밑이 2면 단위는 <strong>bit</strong>, 자연로그면 <strong>nat</strong>이다. 딥러닝 구현은 거의 항상 자연로그를 쓴다.</p> <h2 id="엔트로피-평균-놀람">엔트로피: 평균 놀람</h2> <p>정보량은 사건 하나에 대한 값이다. 분포 전체에 대해 평균을 내면 엔트로피다.</p> \[H(P) = \mathbb{E}_{x \sim P}\left[-\log P(x)\right] = -\sum_x P(x) \log P(x)\] <p>Shannon의 source coding theorem에 따르면 이 값은 <strong>$P$를 따르는 데이터를 인코딩하는 데 필요한 최소 평균 비트 수</strong>다. 공정한 동전은 $H = 1$ bit, 항상 앞면만 나오는 동전은 $H = 0$ bit다. 후자는 결과를 전송할 필요조차 없으니 당연하다.</p> <p>엔트로피는 분포 하나의 고유한 성질이다. 여기까지는 모델이 등장하지 않는다.</p> <h2 id="cross-entropy-틀린-코드북을-쓸-때의-비용">Cross Entropy: 틀린 코드북을 쓸 때의 비용</h2> <p>이제 두 분포를 등장시킨다. 실제 데이터는 $P$를 따르는데, 우리는 그걸 모르고 $Q$가 맞다고 믿고 코드북을 만들었다. 이때 실제로 드는 평균 비트 수가 Cross Entropy다.</p> \[H(P, Q) = \mathbb{E}_{x \sim P}\left[-\log Q(x)\right] = -\sum_x P(x) \log Q(x)\] <p>기댓값은 <strong>실제 분포 $P$</strong>로 잡고, 로그 안에는 <strong>내 예측 $Q$</strong>가 들어간다. 이 비대칭이 핵심이다.</p> <p>$Q = P$일 때 이 값은 최소가 되고, 그 최솟값이 정확히 $H(P)$다. 따라서 언제나</p> \[H(P, Q) \ge H(P)\] <p>내 믿음이 실제와 어긋난 만큼 추가 비용을 지불한다. 그 초과분이 뒤에 나올 KL Divergence다.</p> <h2 id="딥러닝에서의-cross-entropy">딥러닝에서의 Cross Entropy</h2> <p>$P$를 정답 라벨 분포, $Q$를 모델 예측 분포로 두면 손실 함수가 된다. 라벨이 one-hot이면 $P$는 정답 클래스 $c$에서만 1이고 나머지는 0이므로 합이 한 항만 남는다.</p> \[L = -\sum_k P(k) \log Q(k) = -\log Q(c)\] <p><strong>정답 클래스에 부여한 확률의 음의 로그.</strong> 이게 실무에서 마주치는 Cross Entropy의 전부다. 정답에 1을 주면 손실 0, 정답 확률이 0으로 가면 손실이 무한대로 발산한다.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">torch</span>
<span class="kn">import</span> <span class="n">torch.nn.functional</span> <span class="k">as</span> <span class="n">F</span>

<span class="n">logits</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="nf">tensor</span><span class="p">([[</span><span class="mf">2.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">]])</span>
<span class="n">target</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="nf">tensor</span><span class="p">([</span><span class="mi">0</span><span class="p">])</span>

<span class="c1"># softmax([2.0, 1.0, 0.1]) = [0.6590, 0.2424, 0.0986]
# CE = -log(0.6590) = 0.4170
</span><span class="n">F</span><span class="p">.</span><span class="nf">cross_entropy</span><span class="p">(</span><span class="n">logits</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
</code></pre></div></div> <h2 id="bce-클래스가-둘일-때">BCE: 클래스가 둘일 때</h2> <p>Binary Cross Entropy는 별개의 개념이 아니라 클래스가 2개인 특수 케이스다. $P = (y,\ 1-y)$, $Q = (\hat y,\ 1-\hat y)$를 위 정의에 그대로 대입하면 나온다.</p> \[L_{\text{BCE}} = -\left[\,y \log \hat y + (1-y)\log(1-\hat y)\,\right]\] <p>그럼 왜 이름을 따로 붙였을까. 두 가지 이유가 있다.</p> <p>첫째는 표현의 경제성이다. 클래스가 2개면 $\hat y$ 하나만 알면 나머지는 $1 - \hat y$로 결정된다. 출력 노드 2개 + softmax는 파라미터가 중복된다. 노드 1개 + sigmoid로 충분하다.</p> <p>둘째가 실질적으로 더 중요하다. <strong>multi-label 문제를 다룰 수 있다.</strong> softmax는 출력의 합을 1로 강제하므로 클래스들이 상호배타적이라고 가정한다. 반면 sigmoid + BCE는 각 클래스를 독립된 이진 문제로 취급해서 합에 제약이 없다. 사진 한 장에 “고양이”와 “실내”가 동시에 참일 수 있는 문제라면 BCE를 써야 한다.</p> <table> <thead> <tr> <th> </th> <th>CE + softmax</th> <th>BCE + sigmoid</th> </tr> </thead> <tbody> <tr> <td>출력 노드</td> <td>클래스 수 $K$개</td> <td>클래스당 1개</td> </tr> <tr> <td>출력 합</td> <td>항상 1</td> <td>제약 없음</td> </tr> <tr> <td>클래스 가정</td> <td>상호배타 (multi-class)</td> <td>독립 (multi-label)</td> </tr> <tr> <td>대표 문제</td> <td>“이 사진은 개인가 고양이인가”</td> <td>“이 사진에 뭐뭐가 있는가”</td> </tr> </tbody> </table> <h2 id="왜-이게-옳은-손실인가-최대우도추정">왜 이게 “옳은” 손실인가 (최대우도추정)</h2> <p>여기까지는 정보 이론 쪽 해석이었다. 통계 쪽에서 접근하면 같은 식이 전혀 다른 경로로 도출된다.</p> <p>파라미터 $\theta$인 모델이 데이터 ${(x_i, y_i)}$를 만들어냈을 가능성(우도)을 최대화하고 싶다. 샘플이 독립이라면 우도는 곱이고, 로그를 씌우면 합이 된다.</p> \[\hat\theta = \arg\max_\theta \sum_i \log Q_\theta(y_i \mid x_i) = \arg\min_\theta \sum_i -\log Q_\theta(y_i \mid x_i)\] <p>우변이 정확히 Cross Entropy의 합이다.</p> <blockquote> <p><strong>Cross Entropy 최소화 = 최대우도추정</strong></p> </blockquote> <p>여러 후보 중 그럴듯해 보여서 고른 손실이 아니다. “데이터를 가장 잘 설명하는 파라미터를 찾는다”는 통계학의 표준 원리를 분류 문제에 적용하면 필연적으로 나오는 식이다.</p> <h2 id="kl-divergence-그래서-실제로-줄이고-있는-것">KL Divergence: 그래서 실제로 줄이고 있는 것</h2> <p>두 분포의 차이 자체를 재고 싶다면 Cross Entropy는 조금 불편하다. $Q = P$인 완벽한 경우에도 값이 0이 아니라 $H(P)$이기 때문이다. 그 바닥값을 빼주자.</p> \[D_{KL}(P \parallel Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}\] <p>로그 안의 분수를 쪼개면 정체가 드러난다.</p> \[D_{KL}(P \parallel Q) = \sum_x P(x)\log P(x) - \sum_x P(x) \log Q(x) = -H(P) + H(P, Q)\] <p>정리하면 이 글 전체를 관통하는 한 줄이 나온다.</p> \[\boxed{\,H(P, Q) = H(P) + D_{KL}(P \parallel Q)\,}\] <p><strong>Cross Entropy = 데이터 자체의 엔트로피 + 내 예측이 틀린 만큼의 초과 비용.</strong></p> <p>그리고 여기서 실무적으로 중요한 결론이 따라 나온다. 학습 중에 정답 분포 $P$는 고정되어 있으므로 $H(P)$는 $\theta$에 대한 <strong>상수</strong>다. 상수를 빼도 argmin은 변하지 않는다.</p> \[\arg\min_\theta H(P, Q_\theta) = \arg\min_\theta D_{KL}(P \parallel Q_\theta)\] <p>Cross Entropy를 최소화하는 것과 KL Divergence를 최소화하는 것은 <strong>완전히 동일한 최적화 문제</strong>다. 그래서 굳이 $H(P)$를 계산하지 않고 더 싼 Cross Entropy를 쓴다.</p> <p>라벨이 one-hot이면 한 걸음 더 나간다. 확률 1인 사건 하나뿐이라 $H(P) = 0$이고, 따라서</p> \[H(P, Q) = D_{KL}(P \parallel Q)\] <p>두 값이 같아진다. 앞의 예제에서 CE가 0.4170이었는데, $D_{KL}$을 직접 계산해도 $1 \cdot \log(1/0.6590) = 0.4170$으로 같은 값이다.</p> <h3 id="kl의-성질-두-가지">KL의 성질 두 가지</h3> <p><strong>항상 0 이상이다.</strong> $D_{KL}(P\parallel Q) \ge 0$이고, 등호는 $P = Q$일 때만 성립한다 (Gibbs’ inequality). 앞서 $H(P,Q) \ge H(P)$라고 했던 것의 다른 표현이다.</p> <p><strong>대칭이 아니다.</strong> $D_{KL}(P \parallel Q) \ne D_{KL}(Q \parallel P)$이고 삼각부등식도 성립하지 않는다. 그래서 “KL 거리”라는 말은 엄밀히는 틀렸다. divergence라고 부르는 이유다.</p> <p>이 비대칭은 성가신 결점이 아니라 실제로 쓰이는 성질이다.</p> <ul> <li><strong>Forward KL</strong> $D_{KL}(P \parallel Q)$: $P$가 큰 곳에서 $Q$가 작으면 $\log(P/Q)$가 폭발한다. $Q$는 $P$가 존재하는 영역을 전부 덮으려 한다 (mass-covering). 지도학습이 여기 해당한다.</li> <li><strong>Reverse KL</strong> $D_{KL}(Q \parallel P)$: 기댓값을 $Q$로 잡으므로 $Q$가 0인 영역은 아예 벌점을 받지 않는다. $Q$는 $P$의 봉우리 하나에 안전하게 몰린다 (mode-seeking). 변분추론이나 RLHF의 KL penalty가 여기 해당한다.</li> </ul> <h3 id="kl을-직접-쓰는-경우">KL을 직접 쓰는 경우</h3> <p>Cross Entropy로 대체 가능한데도 KL을 명시적으로 계산하는 상황이 있다. $P$가 one-hot이 아닐 때다.</p> <ul> <li><strong>Knowledge distillation</strong>: teacher의 softmax 분포를 $P$로 쓴다. $H(P) \ne 0$이지만 student 파라미터에 대해서는 여전히 상수라, 사실 CE를 써도 최적해는 같다. 그래도 “두 분포를 맞춘다”는 의도를 드러내려고 관례적으로 KL을 쓴다.</li> <li><strong>VAE</strong>: latent 분포를 prior에 붙들어두는 정규화 항이 KL이다. 이건 손실의 일부라 상수항이 아니고, 실제로 값 자체가 필요하다.</li> <li><strong>PPO 등 RL</strong>: 새 정책이 기존 정책에서 너무 멀어지지 않게 KL로 제약한다.</li> </ul> <h2 id="다시-기울기-cross-entropy가-학습을-살리는-이유">다시 기울기: Cross Entropy가 학습을 살리는 이유</h2> <p>첫 절의 문제로 돌아가자. BCE의 $z$에 대한 기울기를 구해보면 이렇게 된다.</p> \[\frac{\partial L}{\partial \hat y} = \frac{\hat y - y}{\hat y(1-\hat y)}, \qquad \frac{\partial \hat y}{\partial z} = \hat y(1 - \hat y)\] \[\frac{\partial L}{\partial z} = \hat y - y\] <p>분모의 $\hat y(1-\hat y)$가 sigmoid의 미분과 정확히 상쇄된다. MSE를 죽이던 포화 항이 사라지고, 기울기가 <strong>오차 그 자체</strong>가 된다. 크게 틀릴수록 크게 움직인다.</p> <p>softmax + CE에서도 똑같은 일이 벌어진다.</p> \[\frac{\partial L}{\partial z_i} = \hat y_i - y_i\] <p>이건 우연이 아니다. sigmoid와 softmax는 exp로 만들어진 함수이고, Cross Entropy는 log를 취한다. log와 exp가 서로를 지운다. 정보 이론에서 유도된 손실과 확률로 정규화하는 활성 함수가 짝이 맞도록 설계되어 있는 것이다.</p> <h2 id="구현할-때-자주-밟는-지뢰">구현할 때 자주 밟는 지뢰</h2> <p><strong>logit을 넘겨라, 확률을 넘기지 말고.</strong> <code class="language-plaintext highlighter-rouge">F.cross_entropy</code>와 <code class="language-plaintext highlighter-rouge">nn.BCEWithLogitsLoss</code>는 내부에서 log-sum-exp trick으로 수치 안정성을 확보한다. 직접 softmax를 거친 뒤 log를 취하면 확률이 0으로 언더플로했을 때 <code class="language-plaintext highlighter-rouge">-inf</code>가 나오고 그대로 NaN이 된다.</p> <p><strong><code class="language-plaintext highlighter-rouge">nn.CrossEntropyLoss</code>는 softmax를 이미 포함한다.</strong> 모델 마지막에 softmax를 넣고 이 손실을 쓰면 softmax가 두 번 걸린다. 에러 없이 조용히 학습만 안 되는, 가장 찾기 짜증나는 종류의 버그다. <code class="language-plaintext highlighter-rouge">nn.BCEWithLogitsLoss</code>와 <code class="language-plaintext highlighter-rouge">nn.BCELoss</code>의 관계도 동일하다.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># 나쁨: softmax 두 번
</span><span class="n">logits</span> <span class="o">=</span> <span class="nf">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">loss</span> <span class="o">=</span> <span class="n">nn</span><span class="p">.</span><span class="nc">CrossEntropyLoss</span><span class="p">()(</span><span class="n">F</span><span class="p">.</span><span class="nf">softmax</span><span class="p">(</span><span class="n">logits</span><span class="p">,</span> <span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">),</span> <span class="n">target</span><span class="p">)</span>

<span class="c1"># 좋음: raw logit 그대로
</span><span class="n">logits</span> <span class="o">=</span> <span class="nf">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">loss</span> <span class="o">=</span> <span class="n">nn</span><span class="p">.</span><span class="nc">CrossEntropyLoss</span><span class="p">()(</span><span class="n">logits</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
</code></pre></div></div> <p><strong><code class="language-plaintext highlighter-rouge">F.kl_div</code>의 인자 규약은 직관과 다르다.</strong> 첫 인자는 확률이 아니라 <strong>log 확률</strong>이고, 두 번째 인자가 확률이다 (<code class="language-plaintext highlighter-rouge">log_target=False</code> 기준). 게다가 기본 <code class="language-plaintext highlighter-rouge">reduction='mean'</code>은 원소 단위 평균이라 수식의 정의와 맞지 않는다. 분포 단위로 맞추려면 <code class="language-plaintext highlighter-rouge">reduction='batchmean'</code>을 써야 한다.</p> <p><strong>Label smoothing은 $P$를 건드리는 기법이다.</strong> one-hot을 살짝 뭉개서 $P = (1-\epsilon, \epsilon/(K-1), \ldots)$로 만든다. 이러면 $H(P) \ne 0$이 되어 CE와 KL의 값이 더 이상 같지 않다. 다만 여전히 상수 차이라 최적화 대상으로서는 동일하다. 손실 값의 절대 크기를 다른 실험과 비교할 때만 주의하면 된다.</p> <h2 id="정리">정리</h2> <ul> <li>정보량 $-\log p$는 “곱을 합으로” 조건에서 필연적으로 나오는 형태다.</li> <li>엔트로피 $H(P)$는 분포 하나의 고유한 최소 인코딩 비용이다.</li> <li>Cross Entropy $H(P,Q)$는 실제 $P$인데 $Q$를 믿고 인코딩할 때의 비용이고, 항상 $H(P)$ 이상이다.</li> <li>BCE는 클래스가 2개인 CE다. 다만 sigmoid와 짝지어 multi-label을 다룰 수 있다는 실질적 차이가 있다.</li> <li>CE 최소화는 최대우도추정과 같다. 임의로 고른 손실이 아니다.</li> <li>$H(P,Q) = H(P) + D_{KL}(P\parallel Q)$이고, 학습 중 $H(P)$는 상수다. 따라서 CE 최소화와 KL 최소화는 같은 문제이며, one-hot 라벨에서는 값까지 같다.</li> <li>KL은 비대칭이라 거리가 아니다. 그 비대칭이 mass-covering과 mode-seeking을 가른다.</li> <li>log와 exp가 상쇄되어 기울기가 $\hat y - y$로 떨어지는 것이, MSE 대신 CE를 쓰는 실전적 이유다.</li> </ul>]]></content><author><name></name></author><category term="deep-learning"/><category term="cross-entropy"/><category term="kl-divergence"/><category term="information-theory"/><category term="loss-function"/><category term="deep-learning"/><summary type="html"><![CDATA[분류 문제에 MSE 대신 Cross Entropy를 쓰는 이유를 정보량과 엔트로피에서 출발해 BCE, MLE, KL Divergence까지 하나의 흐름으로 정리.]]></summary></entry><entry><title type="html">Why cross entropy, of all things: from BCE to KL divergence</title><link href="https://willagio.com/en/blog/2026/cross-entropy-to-kl-divergence/" rel="alternate" type="text/html" title="Why cross entropy, of all things: from BCE to KL divergence"/><published>2026-08-30T00:00:00+00:00</published><updated>2026-08-30T00:00:00+00:00</updated><id>https://willagio.com/en/blog/2026/cross-entropy-to-kl-divergence</id><content type="html" xml:base="https://willagio.com/en/blog/2026/cross-entropy-to-kl-divergence/"><![CDATA[<p>Reaching for <code class="language-plaintext highlighter-rouge">CrossEntropyLoss</code> when you write a classifier is close to a reflex. But explaining why it is that particular loss, why MSE (mean squared error) — which served us well in regression — will not do, what makes it different from <code class="language-plaintext highlighter-rouge">BCELoss</code>, and how it relates to the KL divergence that shows up in every paper turns out to be surprisingly hard to do in one breath.</p> <p>The goal of this post is to connect that chain from end to end, in one pass. Starting from information content, then entropy, cross entropy, BCE, maximum likelihood estimation, and KL divergence.</p> <h2 id="the-opening-question-why-not-mse">The opening question: why not MSE?</h2> <p>Suppose in binary classification the model emits a logit $z$, passes it through a sigmoid to produce $\hat y = \sigma(z)$, and the label is $y \in {0, 1}$. With MSE the loss is</p> \[L_{\text{MSE}} = \frac{1}{2}(\hat y - y)^2\] <p>Taking the gradient with respect to $z$ exposes the problem.</p> \[\frac{\partial L_{\text{MSE}}}{\partial z} = (\hat y - y)\,\sigma'(z) = (\hat y - y)\,\hat y (1 - \hat y)\] <p>There is a factor $\sigma’(z) = \hat y(1-\hat y)$ attached. It dies to zero when $\hat y$ approaches 0 or 1. But “$\hat y$ near 0 while the label is 1” is <strong>exactly the situation where the model is most badly wrong</strong>. So the MSE + sigmoid combination has its smallest gradient precisely when it is most wrong. Learning stops at the very point where learning is needed.</p> <p>The numbers make the severity obvious. Take $y=1$ while the model emits $z=-8$ — completely wrong.</p> <table> <thead> <tr> <th> </th> <th>$\hat y$</th> <th>$\partial L / \partial z$</th> </tr> </thead> <tbody> <tr> <td>MSE + sigmoid</td> <td>0.000335</td> <td>$-0.000335$</td> </tr> <tr> <td>BCE + sigmoid</td> <td>0.000335</td> <td>$-0.999665$</td> </tr> </tbody> </table> <p>Same situation, gradient magnitudes about 3000x apart. MSE learns essentially nothing.</p> <p>Even setting the gradient issue aside, a deeper reason remains. A classifier’s output is not a number but a <strong>probability distribution</strong>. There is no reason to measure how different two probability distributions are with Euclidean distance. Probability distributions have their own yardstick, and that yardstick comes from information theory.</p> <h2 id="information-content-the-size-of-the-surprise">Information content: the size of the surprise</h2> <p>“The sun rises tomorrow” carries almost zero information. “It snows tomorrow” carries a lot, if it is August. To turn the intuition that rarer events carry more information into a formula, what shape must the function have? Impose three conditions.</p> <ol> <li>Lower probability means more information (monotonically decreasing)</li> <li>An event with probability 1 carries zero information</li> <li>Observing two independent events together carries the sum of their information: $I(pq) = I(p) + I(q)$</li> </ol> <p>Condition 3 is the decisive one. The only continuous function that turns products into sums is the logarithm. Satisfying 1 and 2 as well requires flipping the sign.</p> \[I(x) = -\log p(x)\] <p>With base 2 the unit is a <strong>bit</strong>; with the natural logarithm it is a <strong>nat</strong>. Deep learning implementations almost always use the natural logarithm.</p> <h2 id="entropy-average-surprise">Entropy: average surprise</h2> <p>Information content is a value for a single event. Averaging it over the whole distribution gives entropy.</p> \[H(P) = \mathbb{E}_{x \sim P}\left[-\log P(x)\right] = -\sum_x P(x) \log P(x)\] <p>By Shannon’s source coding theorem, this value is the <strong>minimum average number of bits needed to encode data drawn from $P$</strong>. A fair coin has $H = 1$ bit; a coin that always lands heads has $H = 0$ bits. The latter is obvious — there is no need to transmit the outcome at all.</p> <p>Entropy is an intrinsic property of a single distribution. No model has entered the picture yet.</p> <h2 id="cross-entropy-the-cost-of-using-the-wrong-codebook">Cross entropy: the cost of using the wrong codebook</h2> <p>Now bring in two distributions. The real data follows $P$, but not knowing that, we believed $Q$ was correct and built our codebook from it. The average number of bits this actually costs is the cross entropy.</p> \[H(P, Q) = \mathbb{E}_{x \sim P}\left[-\log Q(x)\right] = -\sum_x P(x) \log Q(x)\] <p>The expectation is taken over the <strong>true distribution $P$</strong>, while <strong>our prediction $Q$</strong> sits inside the logarithm. That asymmetry is the whole point.</p> <p>The value is minimised when $Q = P$, and that minimum is exactly $H(P)$. So always</p> \[H(P, Q) \ge H(P)\] <p>We pay an extra cost in proportion to how far our belief diverges from reality. That excess is the KL divergence, coming up later.</p> <h2 id="cross-entropy-in-deep-learning">Cross entropy in deep learning</h2> <p>Let $P$ be the label distribution and $Q$ the model’s predicted distribution, and it becomes a loss function. With a one-hot label, $P$ is 1 only at the true class $c$ and 0 elsewhere, so only one term of the sum survives.</p> \[L = -\sum_k P(k) \log Q(k) = -\log Q(c)\] <p><strong>The negative log of the probability assigned to the correct class.</strong> That is all of the cross entropy you meet in practice. Assign 1 to the correct class and the loss is 0; drive the correct-class probability toward 0 and the loss diverges to infinity.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">torch</span>
<span class="kn">import</span> <span class="n">torch.nn.functional</span> <span class="k">as</span> <span class="n">F</span>

<span class="n">logits</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="nf">tensor</span><span class="p">([[</span><span class="mf">2.0</span><span class="p">,</span> <span class="mf">1.0</span><span class="p">,</span> <span class="mf">0.1</span><span class="p">]])</span>
<span class="n">target</span> <span class="o">=</span> <span class="n">torch</span><span class="p">.</span><span class="nf">tensor</span><span class="p">([</span><span class="mi">0</span><span class="p">])</span>

<span class="c1"># softmax([2.0, 1.0, 0.1]) = [0.6590, 0.2424, 0.0986]
# CE = -log(0.6590) = 0.4170
</span><span class="n">F</span><span class="p">.</span><span class="nf">cross_entropy</span><span class="p">(</span><span class="n">logits</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
</code></pre></div></div> <h2 id="bce-when-there-are-two-classes">BCE: when there are two classes</h2> <p>Binary cross entropy is not a separate concept but the special case of two classes. Substituting $P = (y,\ 1-y)$ and $Q = (\hat y,\ 1-\hat y)$ straight into the definition above gives it.</p> \[L_{\text{BCE}} = -\left[\,y \log \hat y + (1-y)\log(1-\hat y)\,\right]\] <p>So why give it its own name? Two reasons.</p> <p>The first is economy of representation. With two classes, knowing $\hat y$ alone determines the rest as $1 - \hat y$. Two output nodes plus softmax means duplicated parameters. One node plus a sigmoid is enough.</p> <p>The second matters more in practice: <strong>it can handle multi-label problems.</strong> Softmax forces the outputs to sum to 1, which assumes the classes are mutually exclusive. Sigmoid + BCE, by contrast, treats each class as an independent binary problem with no constraint on the sum. If “cat” and “indoors” can both be true of one photo, you need BCE.</p> <table> <thead> <tr> <th> </th> <th>CE + softmax</th> <th>BCE + sigmoid</th> </tr> </thead> <tbody> <tr> <td>Output nodes</td> <td>$K$, one per class</td> <td>one per class</td> </tr> <tr> <td>Sum of outputs</td> <td>always 1</td> <td>unconstrained</td> </tr> <tr> <td>Class assumption</td> <td>mutually exclusive (multi-class)</td> <td>independent (multi-label)</td> </tr> <tr> <td>Typical problem</td> <td>“is this photo a dog or a cat”</td> <td>“what things are in this photo”</td> </tr> </tbody> </table> <h2 id="why-this-is-the-right-loss-maximum-likelihood-estimation">Why this is the “right” loss (maximum likelihood estimation)</h2> <p>Everything so far was the information-theoretic reading. Approach it from statistics and the same formula falls out along an entirely different route.</p> <p>We want to maximise the likelihood that a model with parameters $\theta$ produced the data ${(x_i, y_i)}$. If the samples are independent the likelihood is a product, and taking the logarithm turns it into a sum.</p> \[\hat\theta = \arg\max_\theta \sum_i \log Q_\theta(y_i \mid x_i) = \arg\min_\theta \sum_i -\log Q_\theta(y_i \mid x_i)\] <p>The right-hand side is exactly a sum of cross entropies.</p> <blockquote> <p><strong>Minimising cross entropy = maximum likelihood estimation</strong></p> </blockquote> <p>This is not a loss picked because it looked plausible among candidates. It is the formula that necessarily appears when you apply the standard statistical principle of “find the parameters that best explain the data” to classification.</p> <h2 id="kl-divergence-what-we-are-actually-reducing">KL divergence: what we are actually reducing</h2> <p>If you want to measure the difference between two distributions in its own right, cross entropy is slightly awkward — even in the perfect case $Q = P$ its value is not 0 but $H(P)$. Let us subtract that floor.</p> \[D_{KL}(P \parallel Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}\] <p>Splitting the fraction inside the logarithm reveals what it is.</p> \[D_{KL}(P \parallel Q) = \sum_x P(x)\log P(x) - \sum_x P(x) \log Q(x) = -H(P) + H(P, Q)\] <p>Rearranged, we get the one line that runs through this whole post.</p> \[\boxed{\,H(P, Q) = H(P) + D_{KL}(P \parallel Q)\,}\] <p><strong>Cross entropy = the entropy inherent in the data + the excess cost of my prediction being wrong.</strong></p> <p>And a practically important conclusion follows. During training the label distribution $P$ is fixed, so $H(P)$ is a <strong>constant</strong> with respect to $\theta$. Subtracting a constant does not change the argmin.</p> \[\arg\min_\theta H(P, Q_\theta) = \arg\min_\theta D_{KL}(P \parallel Q_\theta)\] <p>Minimising cross entropy and minimising KL divergence are <strong>exactly the same optimisation problem</strong>. That is why we use the cheaper cross entropy rather than bothering to compute $H(P)$.</p> <p>With one-hot labels we can go one step further. There is only one event with probability 1, so $H(P) = 0$, and therefore</p> \[H(P, Q) = D_{KL}(P \parallel Q)\] <p>the two values coincide. In the earlier example CE was 0.4170, and computing $D_{KL}$ directly gives $1 \cdot \log(1/0.6590) = 0.4170$ — the same value.</p> <h3 id="two-properties-of-kl">Two properties of KL</h3> <p><strong>It is always non-negative.</strong> $D_{KL}(P\parallel Q) \ge 0$, with equality only when $P = Q$ (Gibbs’ inequality). This is another way of stating what we said earlier, $H(P,Q) \ge H(P)$.</p> <p><strong>It is not symmetric.</strong> $D_{KL}(P \parallel Q) \ne D_{KL}(Q \parallel P)$, and the triangle inequality does not hold either. Strictly speaking, then, “KL distance” is wrong. Hence divergence.</p> <p>That asymmetry is not an annoying flaw but a property that is actually used.</p> <ul> <li><strong>Forward KL</strong> $D_{KL}(P \parallel Q)$: where $P$ is large and $Q$ is small, $\log(P/Q)$ explodes. $Q$ tries to cover the whole region where $P$ has mass (mass-covering). Supervised learning falls here.</li> <li><strong>Reverse KL</strong> $D_{KL}(Q \parallel P)$: the expectation is taken over $Q$, so regions where $Q$ is 0 are not penalised at all. $Q$ safely collapses onto one mode of $P$ (mode-seeking). Variational inference and the KL penalty in RLHF fall here.</li> </ul> <h3 id="when-kl-is-used-directly">When KL is used directly</h3> <p>There are situations where KL is computed explicitly even though cross entropy would do — namely when $P$ is not one-hot.</p> <ul> <li><strong>Knowledge distillation</strong>: the teacher’s softmax distribution is used as $P$. $H(P) \ne 0$, but it is still constant with respect to the student’s parameters, so CE would in fact give the same optimum. KL is used by convention to make the intent — “match the two distributions” — explicit.</li> <li><strong>VAE</strong>: the regularisation term that pins the latent distribution to the prior is a KL. Here it is part of the loss rather than a constant, and the value itself is genuinely needed.</li> <li><strong>PPO and other RL</strong>: KL constrains a new policy from drifting too far from the old one.</li> </ul> <h2 id="back-to-gradients-why-cross-entropy-keeps-learning-alive">Back to gradients: why cross entropy keeps learning alive</h2> <p>Return to the problem from the first section. Taking BCE’s gradient with respect to $z$:</p> \[\frac{\partial L}{\partial \hat y} = \frac{\hat y - y}{\hat y(1-\hat y)}, \qquad \frac{\partial \hat y}{\partial z} = \hat y(1 - \hat y)\] \[\frac{\partial L}{\partial z} = \hat y - y\] <p>The $\hat y(1-\hat y)$ in the denominator cancels the sigmoid’s derivative exactly. The saturating factor that killed MSE disappears, and the gradient becomes <strong>the error itself</strong>. The more wrong you are, the more you move.</p> <p>The same thing happens with softmax + CE.</p> \[\frac{\partial L}{\partial z_i} = \hat y_i - y_i\] <p>This is no accident. Sigmoid and softmax are built from exp, and cross entropy takes a log. The log and the exp erase each other. A loss derived from information theory and an activation that normalises into probabilities were designed to be a matched pair.</p> <h2 id="landmines-people-step-on-when-implementing-this">Landmines people step on when implementing this</h2> <p><strong>Pass logits, not probabilities.</strong> <code class="language-plaintext highlighter-rouge">F.cross_entropy</code> and <code class="language-plaintext highlighter-rouge">nn.BCEWithLogitsLoss</code> secure numerical stability internally with the log-sum-exp trick. Applying softmax yourself and then taking a log gives <code class="language-plaintext highlighter-rouge">-inf</code> the moment a probability underflows to 0, and NaN follows.</p> <p><strong><code class="language-plaintext highlighter-rouge">nn.CrossEntropyLoss</code> already includes the softmax.</strong> Putting a softmax at the end of the model and then using this loss applies softmax twice. No error, it just quietly fails to learn — the most irritating kind of bug to find. The relationship between <code class="language-plaintext highlighter-rouge">nn.BCEWithLogitsLoss</code> and <code class="language-plaintext highlighter-rouge">nn.BCELoss</code> is the same.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># bad: softmax twice
</span><span class="n">logits</span> <span class="o">=</span> <span class="nf">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">loss</span> <span class="o">=</span> <span class="n">nn</span><span class="p">.</span><span class="nc">CrossEntropyLoss</span><span class="p">()(</span><span class="n">F</span><span class="p">.</span><span class="nf">softmax</span><span class="p">(</span><span class="n">logits</span><span class="p">,</span> <span class="n">dim</span><span class="o">=-</span><span class="mi">1</span><span class="p">),</span> <span class="n">target</span><span class="p">)</span>

<span class="c1"># good: raw logits
</span><span class="n">logits</span> <span class="o">=</span> <span class="nf">model</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="n">loss</span> <span class="o">=</span> <span class="n">nn</span><span class="p">.</span><span class="nc">CrossEntropyLoss</span><span class="p">()(</span><span class="n">logits</span><span class="p">,</span> <span class="n">target</span><span class="p">)</span>
</code></pre></div></div> <p><strong><code class="language-plaintext highlighter-rouge">F.kl_div</code>’s argument convention is not what you would guess.</strong> The first argument is not probabilities but <strong>log probabilities</strong>, and the second is probabilities (with <code class="language-plaintext highlighter-rouge">log_target=False</code>). Worse, the default <code class="language-plaintext highlighter-rouge">reduction='mean'</code> averages element-wise, which does not match the definition in the formula. To match it at the distribution level you need <code class="language-plaintext highlighter-rouge">reduction='batchmean'</code>.</p> <p><strong>Label smoothing is a technique that modifies $P$.</strong> It slightly blurs the one-hot into $P = (1-\epsilon, \epsilon/(K-1), \ldots)$. Now $H(P) \ne 0$, so CE and KL no longer have the same value. They still differ by a constant, though, so as optimisation targets they are identical. You only need to be careful when comparing absolute loss values against other experiments.</p> <h2 id="summary">Summary</h2> <ul> <li>Information content $-\log p$ is the form that necessarily emerges from the “products into sums” condition.</li> <li>Entropy $H(P)$ is the intrinsic minimum encoding cost of a single distribution.</li> <li>Cross entropy $H(P,Q)$ is the cost of encoding with $Q$ when reality is $P$, and it is always at least $H(P)$.</li> <li>BCE is CE with two classes. The substantive difference is that, paired with a sigmoid, it can handle multi-label problems.</li> <li>Minimising CE is maximum likelihood estimation. It is not an arbitrarily chosen loss.</li> <li>$H(P,Q) = H(P) + D_{KL}(P\parallel Q)$, and during training $H(P)$ is constant. So minimising CE and minimising KL are the same problem, and with one-hot labels even the values coincide.</li> <li>KL is asymmetric and therefore not a distance. That asymmetry is what separates mass-covering from mode-seeking.</li> <li>The log and exp cancelling so the gradient collapses to $\hat y - y$ is the practical reason to use CE instead of MSE.</li> </ul>]]></content><author><name></name></author><category term="deep-learning"/><category term="cross-entropy"/><category term="kl-divergence"/><category term="information-theory"/><category term="loss-function"/><category term="deep-learning"/><summary type="html"><![CDATA[Why classification uses cross entropy instead of MSE, traced as a single line from information content and entropy through BCE and MLE to KL divergence.]]></summary></entry><entry><title type="html">ROUGE</title><link href="https://willagio.com/blog/2026/rogue/" rel="alternate" type="text/html" title="ROUGE"/><published>2026-04-01T00:00:00+00:00</published><updated>2026-04-01T00:00:00+00:00</updated><id>https://willagio.com/blog/2026/rogue</id><content type="html" xml:base="https://willagio.com/blog/2026/rogue/"><![CDATA[<p>ROUGE는 <strong>R</strong>ecall-<strong>O</strong>riented <strong>U</strong>nderstudy for <strong>G</strong>isting <strong>E</strong>valuation 의 약자로, 모델이 생성한 텍스트가 정답과 얼마나 겹치는지 측정하는 지표다. 원래 요약(summarization) 평가용으로 만들어졌지만, QA나 번역 등 텍스트 생성 전반에서 쓰인다.</p> <p>핵심 아이디어는 단순하다: <strong>정답에 있는 단어가 예측에 얼마나 등장하는가?</strong></p> <h2 id="공통-수식-precision-recall-f1">공통 수식: Precision, Recall, F1</h2> <p>모든 ROUGE 변형은 동일한 프레임워크를 따른다. 겹침(overlap)을 세는 단위만 달라질 뿐이다.</p> \[\text{Recall} = \frac{|\text{overlap}|}{|\text{Reference}|}\] \[\text{Precision} = \frac{|\text{overlap}|}{|\text{Prediction}|}\] \[F_1 = \frac{2 \times P \times R}{P + R}\] <ul> <li><strong>Recall</strong>: 정답 중 얼마나 잡아냈는가 (빠뜨린 게 없나)</li> <li><strong>Precision</strong>: 예측 중 정답에 해당하는 비율 (헛소리가 없나)</li> <li><strong>F1</strong>: 둘의 조화 평균. HuggingFace <code class="language-plaintext highlighter-rouge">evaluate</code>가 기본 리포트하는 값이 이것이다.</li> </ul> <h2 id="예시-설정">예시 설정</h2> <p>아래 예시로 모든 ROUGE를 계산해본다.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>정답(Reference):    "서울의 인구는 약 천만 명이고 대한민국의 수도이다"
예측(Prediction):   "서울은 대한민국의 수도이며 인구는 약 구백만 명이다"
</code></pre></div></div> <p>토큰 분리 (공백 기준):</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>R = [서울의, 인구는, 약, 천만, 명이고, 대한민국의, 수도이다]     → m = 7
P = [서울은, 대한민국의, 수도이며, 인구는, 약, 구백만, 명이다]   → n = 7
</code></pre></div></div> <h2 id="rouge-1-unigram">ROUGE-1 (Unigram)</h2> <h3 id="수식">수식</h3> \[\text{ROUGE-1}_{recall} = \frac{\sum_{w \in R} \min(\text{count}_R(w),\ \text{count}_P(w))}{\sum_{w \in R} \text{count}_R(w)}\] <p>각 unigram $w$에 대해, 정답에서의 출현 횟수와 예측에서의 출현 횟수 중 작은 값을 취해 합산한다.</p> <h3 id="카운팅">카운팅</h3> <table> <thead> <tr> <th>정답 토큰</th> <th>예측에 있나?</th> </tr> </thead> <tbody> <tr> <td>서울의</td> <td>❌ (“서울은”과 다른 토큰)</td> </tr> <tr> <td>인구는</td> <td>✅</td> </tr> <tr> <td>약</td> <td>✅</td> </tr> <tr> <td>천만</td> <td>❌ (“구백만”과 다른 토큰)</td> </tr> <tr> <td>명이고</td> <td>❌ (“명이다”와 다른 토큰)</td> </tr> <tr> <td>대한민국의</td> <td>✅</td> </tr> <tr> <td>수도이다</td> <td>❌ (“수도이며”와 다른 토큰)</td> </tr> </tbody> </table> <p>겹치는 토큰: {인구는, 약, 대한민국의} → $|\text{overlap}| = 3$</p> <h3 id="계산">계산</h3> \[R = \frac{3}{7} \approx 0.4286\] \[P = \frac{3}{7} \approx 0.4286\] \[F_1 = \frac{2 \times 0.4286 \times 0.4286}{0.4286 + 0.4286} \approx 0.4286\] <h2 id="rouge-2-bigram">ROUGE-2 (Bigram)</h2> <h3 id="수식-1">수식</h3> \[\text{ROUGE-2}_{recall} = \frac{\sum_{b \in R_{bigram}} \min(\text{count}_R(b),\ \text{count}_P(b))}{\sum_{b \in R_{bigram}} \text{count}_R(b)}\] <p>unigram 대신 연속 2개 단어 쌍(bigram)으로 겹침을 센다.</p> <h3 id="카운팅-1">카운팅</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>정답 bigrams (6개):
  (서울의, 인구는)       (인구는, 약)         (약, 천만)
  (천만, 명이고)         (명이고, 대한민국의)   (대한민국의, 수도이다)

예측 bigrams (6개):
  (서울은, 대한민국의)   (대한민국의, 수도이며) (수도이며, 인구는)
  (인구는, 약)           (약, 구백만)          (구백만, 명이다)

겹치는 bigram:
  (인구는, 약) ✅
  나머지 전부 ❌
</code></pre></div></div> <p>$|\text{overlap}| = 1$</p> <h3 id="계산-1">계산</h3> \[R = \frac{1}{6} \approx 0.1667\] \[P = \frac{1}{6} \approx 0.1667\] \[F_1 \approx 0.1667\] <p>ROUGE-2가 ROUGE-1보다 낮은 이유: “인구는”과 “약”이 각각 겹쳐도, 이 둘이 <strong>연속으로</strong> 나타나야만 bigram으로 잡힌다. “대한민국의”가 겹치지만 앞뒤 토큰이 달라서 어떤 bigram에도 기여하지 못한다.</p> <h2 id="rouge-l-longest-common-subsequence">ROUGE-L (Longest Common Subsequence)</h2> <h3 id="수식-2">수식</h3> <p>LCS(Longest Common Subsequence)의 길이를 이용한다.</p> \[R_{lcs} = \frac{\text{LCS}(R, P)}{m}\] \[P_{lcs} = \frac{\text{LCS}(R, P)}{n}\] \[F_{lcs} = \frac{(1 + \beta^2) \times R_{lcs} \times P_{lcs}}{R_{lcs} + \beta^2 \times P_{lcs}}\] <p>$m$ = 정답 길이, $n$ = 예측 길이, $\beta = P_{lcs} / R_{lcs}$ (이렇게 설정하면 $F_{lcs} = F_1$과 동일해진다)</p> <p><strong>핵심</strong>: subsequence는 연속일 필요 없다. 원래 순서만 유지하면 건너뛸 수 있다.</p> <h3 id="lcs-구하기-dp-테이블">LCS 구하기 (DP 테이블)</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>R = [서울의, 인구는, 약, 천만, 명이고, 대한민국의, 수도이다]
P = [서울은, 대한민국의, 수도이며, 인구는, 약, 구백만, 명이다]
</code></pre></div></div> <p>DP 테이블 $L[i][j]$ = $R[0..i]$와 $P[0..j]$의 LCS 길이:</p> <table> <thead> <tr> <th> </th> <th>서울은</th> <th>대한민국의</th> <th>수도이며</th> <th>인구는</th> <th>약</th> <th>구백만</th> <th>명이다</th> </tr> </thead> <tbody> <tr> <td>서울의</td> <td>0</td> <td>0</td> <td>0</td> <td>0</td> <td>0</td> <td> </td> <td> </td> </tr> <tr> <td>인구는</td> <td>0</td> <td>0</td> <td>0</td> <td>1</td> <td>1</td> <td> </td> <td> </td> </tr> <tr> <td>약</td> <td>0</td> <td>0</td> <td>0</td> <td>1</td> <td>2</td> <td> </td> <td> </td> </tr> <tr> <td>천만</td> <td>0</td> <td>0</td> <td>0</td> <td>1</td> <td>2</td> <td> </td> <td> </td> </tr> <tr> <td>명이고</td> <td>0</td> <td>0</td> <td>0</td> <td>1</td> <td>2</td> <td> </td> <td> </td> </tr> <tr> <td>대한민국의</td> <td>0</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td> </td> <td> </td> </tr> <tr> <td>수도이다</td> <td>0</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td> </td> <td> </td> </tr> </tbody> </table> <p>$\text{LCS}(R, P) = 2$ → [인구는, 약]</p> <p>“대한민국의”가 양쪽에 존재하지만, 정답에서는 6번째(인구는, 약 뒤), 예측에서는 2번째(인구는, 약 앞)이므로 [인구는, 약]과 동시에 LCS에 포함될 수 없다.</p> <h3 id="계산-2">계산</h3> \[R_{lcs} = \frac{2}{7} \approx 0.2857\] \[P_{lcs} = \frac{2}{7} \approx 0.2857\] \[F_1 \approx 0.2857\] <p>이 예시에서 ROUGE-L &lt; ROUGE-1이다. ROUGE-1은 순서를 무시하고 {인구는, 약, 대한민국의} 3개를 세지만, ROUGE-L은 순서 제약 때문에 2개만 센다.</p> <h3 id="rouge-1--10인데-rouge-l이-낮은-극단-예시">ROUGE-1 = 1.0인데 ROUGE-L이 낮은 극단 예시</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>R = [A, B, C]
P = [C, B, A]

ROUGE-1: overlap = {A, B, C} → 3/3 = 1.0
ROUGE-L: LCS = [A] 또는 [B] 또는 [C] → 1/3 = 0.33
</code></pre></div></div> <p>모든 단어가 있지만 순서가 완전히 뒤집혀서 ROUGE-L은 크게 떨어진다. 이것이 ROUGE-L의 존재 이유다.</p> <h2 id="rouge-lsum">ROUGE-Lsum</h2> <h3 id="수식-3">수식</h3> <p>다중 문장에서 문장별로 LCS를 따로 구한 뒤 합산한다.</p> <p>정답이 문장 $r_1, r_2, \ldots, r_k$로 구성될 때:</p> \[R_{lsum} = \frac{\sum_{i=1}^{k} \text{LCS}(r_i, P)}{m}\] <h3 id="예시">예시</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>정답:
  r1: "서울의 인구는 약 천만 명이다"           (5토큰)
  r2: "서울은 대한민국의 수도이다"              (3토큰)

예측:
  "서울은 대한민국의 수도이며 인구는 약 구백만 명이다"
</code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>LCS(r1, P) = [인구는, 약] → 2
LCS(r2, P) = [서울은, 대한민국의] → 2

Recall = (2 + 2) / (5 + 3) = 4/8 = 0.5
</code></pre></div></div> <p>ROUGE-L을 전체에 한 번 적용하면 문장 경계를 무시해서 한 문장의 매칭이 다른 문장의 점수를 방해할 수 있다. ROUGE-Lsum은 문장별로 독립 계산하므로 요약처럼 여러 문장이 있는 태스크에서 더 공정하다.</p> <p>단일 문장이면 ROUGE-L = ROUGE-Lsum.</p> <h2 id="전체-비교-정리">전체 비교 정리</h2> <table> <thead> <tr> <th>지표</th> <th>수식의 overlap 단위</th> <th>예시 F1</th> <th>특징</th> </tr> </thead> <tbody> <tr> <td>ROUGE-1</td> <td>unigram</td> <td>0.4286</td> <td>가장 관대. 순서 무시</td> </tr> <tr> <td>ROUGE-2</td> <td>bigram</td> <td>0.1667</td> <td>구문 유사도. 연속 매칭 필요</td> </tr> <tr> <td>ROUGE-L</td> <td>LCS</td> <td>0.2857</td> <td>순서 고려. 비연속 허용</td> </tr> <tr> <td>ROUGE-Lsum</td> <td>문장별 LCS 합</td> <td>-</td> <td>다중 문장에서 ROUGE-L 개선</td> </tr> </tbody> </table> <p>일반적으로: <strong>ROUGE-1 &gt;= ROUGE-L &gt;= ROUGE-2</strong></p> <h2 id="rouge의-한계">ROUGE의 한계</h2> <h3 id="동의어를-못-잡는다">동의어를 못 잡는다</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>정답: "기쁘다"
예측: "행복하다"
→ ROUGE = 0 (의미는 같지만 토큰이 다름)
</code></pre></div></div> <h3 id="길게-쓰면-precision이-희석된다">길게 쓰면 Precision이 희석된다</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>정답: "교향곡"               → m = 1
예측: "바그너는 교향곡을 쓰려 했다"  → n = 4

Recall = 1/1 = 1.0  (정답 토큰을 다 잡음)
Precision = 1/4 = 0.25  (예측의 75%가 불필요)
F1 = 2 × (1.0 × 0.25) / (1.0 + 0.25) = 0.4
</code></pre></div></div> <h3 id="의미적-정확도를-못-본다">의미적 정확도를 못 본다</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>정답: "2023년"
예측: "2024년"
→ ROUGE-1 = 0이지만, 사실상 거의 맞춘 것
</code></pre></div></div> <p>이런 한계 때문에 실무에서는 ROUGE 단독이 아니라 BERTScore(임베딩 유사도), Exact Match, Human Evaluation 등을 함께 쓴다.</p> <h2 id="파이썬으로-직접-계산">파이썬으로 직접 계산</h2> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">evaluate</span>

<span class="n">rouge</span> <span class="o">=</span> <span class="n">evaluate</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="sh">"</span><span class="s">rouge</span><span class="sh">"</span><span class="p">)</span>

<span class="n">predictions</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">서울은 대한민국의 수도이며 인구는 약 구백만 명이다</span><span class="sh">"</span><span class="p">]</span>
<span class="n">references</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">서울의 인구는 약 천만 명이고 대한민국의 수도이다</span><span class="sh">"</span><span class="p">]</span>

<span class="n">results</span> <span class="o">=</span> <span class="n">rouge</span><span class="p">.</span><span class="nf">compute</span><span class="p">(</span><span class="n">predictions</span><span class="o">=</span><span class="n">predictions</span><span class="p">,</span> <span class="n">references</span><span class="o">=</span><span class="n">references</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="n">results</span><span class="p">)</span>
<span class="c1"># {'rouge1': 0.4286, 'rouge2': 0.1667, 'rougeL': 0.2857, 'rougeLsum': 0.2857}
</span></code></pre></div></div>]]></content><author><name></name></author><category term="deep-learning"/><category term="rouge"/><category term="nlp"/><category term="evaluation"/><summary type="html"><![CDATA[Recall-Oriented Understudy for Gisting Evaluation]]></summary></entry><entry><title type="html">ROUGE</title><link href="https://willagio.com/en/blog/2026/rogue/" rel="alternate" type="text/html" title="ROUGE"/><published>2026-04-01T00:00:00+00:00</published><updated>2026-04-01T00:00:00+00:00</updated><id>https://willagio.com/en/blog/2026/rogue</id><content type="html" xml:base="https://willagio.com/en/blog/2026/rogue/"><![CDATA[<p>ROUGE stands for <strong>R</strong>ecall-<strong>O</strong>riented <strong>U</strong>nderstudy for <strong>G</strong>isting <strong>E</strong>valuation, a metric that measures how much a model’s generated text overlaps with the reference. It was originally built for evaluating summarisation, but it is used across text generation generally — QA, translation, and so on.</p> <p>The core idea is simple: <strong>how many of the words in the reference show up in the prediction?</strong></p> <h2 id="the-shared-formulas-precision-recall-f1">The shared formulas: precision, recall, F1</h2> <p>Every ROUGE variant follows the same framework. Only the unit in which overlap is counted changes.</p> \[\text{Recall} = \frac{|\text{overlap}|}{|\text{Reference}|}\] \[\text{Precision} = \frac{|\text{overlap}|}{|\text{Prediction}|}\] \[F_1 = \frac{2 \times P \times R}{P + R}\] <ul> <li><strong>Recall</strong>: how much of the reference did we capture (did we miss anything)</li> <li><strong>Precision</strong>: what fraction of the prediction was actually in the reference (is there any nonsense)</li> <li><strong>F1</strong>: the harmonic mean of the two. This is what HuggingFace <code class="language-plaintext highlighter-rouge">evaluate</code> reports by default.</li> </ul> <h2 id="the-running-example">The running example</h2> <p>We will compute every ROUGE variant on the example below.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference:   "Seoul has about ten million residents and is the capital"
Prediction:  "Seoul is the capital and has about nine million residents"
</code></pre></div></div> <p>Tokenised on whitespace:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>R = [Seoul, has, about, ten, million, residents, and, is, the, capital]   -&gt; m = 10
P = [Seoul, is, the, capital, and, has, about, nine, million, residents]  -&gt; n = 10
</code></pre></div></div> <h2 id="rouge-1-unigram">ROUGE-1 (unigram)</h2> <h3 id="formula">Formula</h3> \[\text{ROUGE-1}_{recall} = \frac{\sum_{w \in R} \min(\text{count}_R(w),\ \text{count}_P(w))}{\sum_{w \in R} \text{count}_R(w)}\] <p>For each unigram $w$, take the smaller of its count in the reference and its count in the prediction, and sum.</p> <h3 id="counting">Counting</h3> <table> <thead> <tr> <th>Reference token</th> <th>In the prediction?</th> </tr> </thead> <tbody> <tr> <td>Seoul</td> <td>✅</td> </tr> <tr> <td>has</td> <td>✅</td> </tr> <tr> <td>about</td> <td>✅</td> </tr> <tr> <td>ten</td> <td>❌ (the prediction says “nine”)</td> </tr> <tr> <td>million</td> <td>✅</td> </tr> <tr> <td>residents</td> <td>✅</td> </tr> <tr> <td>and</td> <td>✅</td> </tr> <tr> <td>is</td> <td>✅</td> </tr> <tr> <td>the</td> <td>✅</td> </tr> <tr> <td>capital</td> <td>✅</td> </tr> </tbody> </table> <p>Overlapping tokens: everything but “ten” → $|\text{overlap}| = 9$</p> <h3 id="computation">Computation</h3> \[R = \frac{9}{10} = 0.9\] \[P = \frac{9}{10} = 0.9\] \[F_1 = \frac{2 \times 0.9 \times 0.9}{0.9 + 0.9} = 0.9\] <h2 id="rouge-2-bigram">ROUGE-2 (bigram)</h2> <h3 id="formula-1">Formula</h3> \[\text{ROUGE-2}_{recall} = \frac{\sum_{b \in R_{bigram}} \min(\text{count}_R(b),\ \text{count}_P(b))}{\sum_{b \in R_{bigram}} \text{count}_R(b)}\] <p>Overlap is counted over pairs of adjacent words (bigrams) instead of unigrams.</p> <h3 id="counting-1">Counting</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference bigrams (9):
  (Seoul, has)         (has, about)        (about, ten)
  (ten, million)       (million, residents) (residents, and)
  (and, is)            (is, the)           (the, capital)

Prediction bigrams (9):
  (Seoul, is)          (is, the)           (the, capital)
  (capital, and)       (and, has)          (has, about)
  (about, nine)        (nine, million)     (million, residents)

Overlapping bigrams:
  (has, about) ✅   (million, residents) ✅   (is, the) ✅   (the, capital) ✅
  everything else ❌
</code></pre></div></div> <p>$|\text{overlap}| = 4$</p> <h3 id="computation-1">Computation</h3> \[R = \frac{4}{9} \approx 0.4444\] \[P = \frac{4}{9} \approx 0.4444\] \[F_1 \approx 0.4444\] <p>ROUGE-2 comes out lower than ROUGE-1 because a word only counts once it appears <strong>adjacent</strong> to the right neighbour. “Seoul” and “and” overlap as unigrams, but their surrounding tokens differ, so they contribute to no bigram at all.</p> <h2 id="rouge-l-longest-common-subsequence">ROUGE-L (longest common subsequence)</h2> <h3 id="formula-2">Formula</h3> <p>This one uses the length of the LCS (longest common subsequence).</p> \[R_{lcs} = \frac{\text{LCS}(R, P)}{m}\] \[P_{lcs} = \frac{\text{LCS}(R, P)}{n}\] \[F_{lcs} = \frac{(1 + \beta^2) \times R_{lcs} \times P_{lcs}}{R_{lcs} + \beta^2 \times P_{lcs}}\] <p>$m$ = reference length, $n$ = prediction length, $\beta = P_{lcs} / R_{lcs}$ (setting it this way makes $F_{lcs}$ identical to $F_1$).</p> <p><strong>The key point</strong>: a subsequence does not have to be contiguous. It can skip, as long as the original order is preserved.</p> <h3 id="finding-the-lcs-the-dp-table">Finding the LCS (the DP table)</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>R = [Seoul, has, about, ten, million, residents, and, is, the, capital]
P = [Seoul, is, the, capital, and, has, about, nine, million, residents]
</code></pre></div></div> <p>DP table $L[i][j]$ = the LCS length of $R[0..i]$ and $P[0..j]$:</p> <table> <thead> <tr> <th> </th> <th>Seoul</th> <th>is</th> <th>the</th> <th>capital</th> <th>and</th> <th>has</th> <th>about</th> <th>nine</th> <th>million</th> <th>residents</th> </tr> </thead> <tbody> <tr> <td>Seoul</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> </tr> <tr> <td>has</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td>2</td> <td>2</td> <td>2</td> <td>2</td> </tr> <tr> <td>about</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td>3</td> <td>3</td> <td>3</td> <td>3</td> </tr> <tr> <td>ten</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td>3</td> <td>3</td> <td>3</td> <td>3</td> </tr> <tr> <td>million</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td>3</td> <td>3</td> <td>4</td> <td>4</td> </tr> <tr> <td>residents</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td>3</td> <td>3</td> <td>4</td> <td>5</td> </tr> <tr> <td>and</td> <td>1</td> <td>1</td> <td>1</td> <td>1</td> <td>2</td> <td>2</td> <td>3</td> <td>3</td> <td>4</td> <td>5</td> </tr> <tr> <td>is</td> <td>1</td> <td>2</td> <td>2</td> <td>2</td> <td>2</td> <td>2</td> <td>3</td> <td>3</td> <td>4</td> <td>5</td> </tr> <tr> <td>the</td> <td>1</td> <td>2</td> <td>3</td> <td>3</td> <td>3</td> <td>3</td> <td>3</td> <td>3</td> <td>4</td> <td>5</td> </tr> <tr> <td>capital</td> <td>1</td> <td>2</td> <td>3</td> <td>4</td> <td>4</td> <td>4</td> <td>4</td> <td>4</td> <td>4</td> <td>5</td> </tr> </tbody> </table> <p>$\text{LCS}(R, P) = 5$ → [Seoul, has, about, million, residents]</p> <p>“and”, “is”, “the” and “capital” all exist on both sides, but in the reference they come <em>after</em> “residents” while in the prediction they come <em>before</em> “has”, so they cannot be part of the LCS at the same time as [has, about, million, residents].</p> <h3 id="computation-2">Computation</h3> \[R_{lcs} = \frac{5}{10} = 0.5\] \[P_{lcs} = \frac{5}{10} = 0.5\] \[F_1 = 0.5\] <p>Here ROUGE-L &lt; ROUGE-1. ROUGE-1 ignores order and counts 9 tokens; ROUGE-L counts only 5 because of the ordering constraint.</p> <h3 id="an-extreme-case-where-rouge-1--10-but-rouge-l-is-low">An extreme case where ROUGE-1 = 1.0 but ROUGE-L is low</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>R = [A, B, C]
P = [C, B, A]

ROUGE-1: overlap = {A, B, C} -&gt; 3/3 = 1.0
ROUGE-L: LCS = [A] or [B] or [C] -&gt; 1/3 = 0.33
</code></pre></div></div> <p>Every word is present, but the order is completely reversed, so ROUGE-L falls sharply. This is exactly why ROUGE-L exists.</p> <h2 id="rouge-lsum">ROUGE-Lsum</h2> <h3 id="formula-3">Formula</h3> <p>For multi-sentence text, compute the LCS per reference sentence and sum.</p> <p>When the reference consists of sentences $r_1, r_2, \ldots, r_k$:</p> \[R_{lsum} = \frac{\sum_{i=1}^{k} \text{LCS}(r_i, P)}{m}\] <h3 id="example">Example</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference:
  r1: "Seoul has about ten million residents"    (6 tokens)
  r2: "Seoul is the capital of South Korea"      (7 tokens)

Prediction:
  "Seoul is the capital and has about nine million residents"
</code></pre></div></div> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>LCS(r1, P) = [Seoul, has, about, million, residents] -&gt; 5
LCS(r2, P) = [Seoul, is, the, capital] -&gt; 4

Recall = (5 + 4) / (6 + 7) = 9/13 ~ 0.6923
</code></pre></div></div> <p>Applying ROUGE-L once over the whole text ignores sentence boundaries, so a match in one sentence can get in the way of another sentence’s score. ROUGE-Lsum computes each sentence independently, which is fairer for tasks like summarisation where there are several sentences.</p> <p>For a single sentence, ROUGE-L = ROUGE-Lsum.</p> <h2 id="overall-comparison">Overall comparison</h2> <table> <thead> <tr> <th>Metric</th> <th>Overlap unit in the formula</th> <th>F1 on the example</th> <th>Character</th> </tr> </thead> <tbody> <tr> <td>ROUGE-1</td> <td>unigram</td> <td>0.9</td> <td>most lenient; ignores order</td> </tr> <tr> <td>ROUGE-2</td> <td>bigram</td> <td>0.4444</td> <td>phrase similarity; requires adjacency</td> </tr> <tr> <td>ROUGE-L</td> <td>LCS</td> <td>0.5</td> <td>order-aware; allows gaps</td> </tr> <tr> <td>ROUGE-Lsum</td> <td>sum of per-sentence LCS</td> <td>-</td> <td>improves ROUGE-L for multi-sentence text</td> </tr> </tbody> </table> <p>Generally: <strong>ROUGE-1 &gt;= ROUGE-L &gt;= ROUGE-2</strong></p> <h2 id="the-limits-of-rouge">The limits of ROUGE</h2> <h3 id="it-cannot-catch-synonyms">It cannot catch synonyms</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference:  "glad"
Prediction: "happy"
-&gt; ROUGE = 0 (same meaning, different token)
</code></pre></div></div> <h3 id="writing-at-length-dilutes-precision">Writing at length dilutes precision</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference:  "symphony"                 -&gt; m = 1
Prediction: "Wagner wrote a symphony"  -&gt; n = 4

Recall = 1/1 = 1.0  (caught every reference token)
Precision = 1/4 = 0.25  (75% of the prediction is unnecessary)
F1 = 2 x (1.0 x 0.25) / (1.0 + 0.25) = 0.4
</code></pre></div></div> <h3 id="it-does-not-see-semantic-accuracy">It does not see semantic accuracy</h3> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Reference:  "2023"
Prediction: "2024"
-&gt; ROUGE-1 = 0, even though it is essentially almost right
</code></pre></div></div> <p>Because of limits like these, practitioners use ROUGE alongside BERTScore (embedding similarity), exact match, and human evaluation rather than on its own.</p> <h2 id="computing-it-in-python">Computing it in Python</h2> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="n">evaluate</span>

<span class="n">rouge</span> <span class="o">=</span> <span class="n">evaluate</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="sh">"</span><span class="s">rouge</span><span class="sh">"</span><span class="p">)</span>

<span class="n">predictions</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">Seoul is the capital and has about nine million residents</span><span class="sh">"</span><span class="p">]</span>
<span class="n">references</span> <span class="o">=</span> <span class="p">[</span><span class="sh">"</span><span class="s">Seoul has about ten million residents and is the capital</span><span class="sh">"</span><span class="p">]</span>

<span class="n">results</span> <span class="o">=</span> <span class="n">rouge</span><span class="p">.</span><span class="nf">compute</span><span class="p">(</span><span class="n">predictions</span><span class="o">=</span><span class="n">predictions</span><span class="p">,</span> <span class="n">references</span><span class="o">=</span><span class="n">references</span><span class="p">)</span>
<span class="nf">print</span><span class="p">(</span><span class="n">results</span><span class="p">)</span>
<span class="c1"># {'rouge1': 0.9, 'rouge2': 0.4444, 'rougeL': 0.5, 'rougeLsum': 0.5}
</span></code></pre></div></div>]]></content><author><name></name></author><category term="deep-learning"/><category term="rouge"/><category term="nlp"/><category term="evaluation"/><summary type="html"><![CDATA[Recall-Oriented Understudy for Gisting Evaluation]]></summary></entry><entry><title type="html">덕 타이핑, 그리고 파이썬이 타입을 다루는 방식</title><link href="https://willagio.com/blog/2026/duck-typing/" rel="alternate" type="text/html" title="덕 타이핑, 그리고 파이썬이 타입을 다루는 방식"/><published>2026-03-24T00:00:00+00:00</published><updated>2026-03-24T00:00:00+00:00</updated><id>https://willagio.com/blog/2026/duck-typing</id><content type="html" xml:base="https://willagio.com/blog/2026/duck-typing/"><![CDATA[<blockquote> <p>“오리처럼 걷고, 오리처럼 꽥꽥거리면 그것은 오리다.”</p> </blockquote> <h2 id="덕-타이핑이란">덕 타이핑이란?</h2> <p>덕 타이핑은 <strong>객체의 타입을 클래스 선언이 아니라, 그 객체가 가진 메서드와 속성으로 판단</strong>하는 프로그래밍 철학이다. Python, JavaScript, Ruby 같은 동적 타입 언어에서 자주 볼 수 있는 접근 방식이다.</p> <p>타입을 명시적으로 확인하지 않고, 필요한 동작을 그냥 호출한다. 해당 메서드나 속성이 존재하면 성공, 없으면 런타임 에러가 난다.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Duck</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">꽥꽥!</span><span class="sh">"</span>

<span class="k">class</span> <span class="nc">Dog</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">멍멍!</span><span class="sh">"</span>

<span class="k">class</span> <span class="nc">Robot</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">삐빅!</span><span class="sh">"</span>

<span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">):</span>
    <span class="c1"># 타입 확인 없이 그냥 호출
</span>    <span class="nf">print</span><span class="p">(</span><span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">())</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Duck</span><span class="p">())</span>   <span class="c1"># 꽥꽥!
</span><span class="nf">make_sound</span><span class="p">(</span><span class="nc">Dog</span><span class="p">())</span>    <span class="c1"># 멍멍!
</span><span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># 삐빅!
</span></code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">Duck</code>이든 <code class="language-plaintext highlighter-rouge">Dog</code>든 <code class="language-plaintext highlighter-rouge">Robot</code>이든, <code class="language-plaintext highlighter-rouge">sound()</code> 메서드만 있으면 동일하게 취급된다. <strong>“이 객체가 무엇인가”보다 “이 객체가 무엇을 할 수 있는가”</strong> 에 집중하는 셈이다.</p> <h2 id="abc는-덕-타이핑을-깨뜨리나">ABC는 덕 타이핑을 깨뜨리나?</h2> <p>결론부터 말하면 <strong>꼭 그렇지는 않다.</strong> ABC를 어떻게 사용하느냐에 따라 달라진다.</p> <h3 id="isinstance-강제-체크--덕-타이핑-깨짐">isinstance() 강제 체크 → 덕 타이핑 깨짐</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>

<span class="k">class</span> <span class="nc">Animal</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
    <span class="nd">@abstractmethod</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="k">pass</span>

<span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">):</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">animal</span><span class="p">,</span> <span class="n">Animal</span><span class="p">):</span>  <span class="c1"># 타입 직접 확인!
</span>        <span class="k">raise</span> <span class="nc">TypeError</span><span class="p">(</span><span class="sh">"</span><span class="s">Animal이 아님</span><span class="sh">"</span><span class="p">)</span>
    <span class="nf">print</span><span class="p">(</span><span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">())</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># ❌ sound()가 있어도 거부됨
</span></code></pre></div></div> <h3 id="abc는-명세-역할만-체크-없음--덕-타이핑-유지">ABC는 명세 역할만, 체크 없음 → 덕 타이핑 유지</h3> <p>함수 안에서 타입을 체크하지 않으면 ABC를 써도 덕 타이핑은 그대로 동작한다. ABC가 하는 일은 “이 메서드는 꼭 구현해야 해”라는 개발자 간의 약속일 뿐이다.</p> <h3 id="register로-절충">register()로 절충</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Animal</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">Robot</span><span class="p">)</span>  <span class="c1"># Robot을 Animal로 등록
</span>
<span class="nf">print</span><span class="p">(</span><span class="nf">isinstance</span><span class="p">(</span><span class="nc">Robot</span><span class="p">(),</span> <span class="n">Animal</span><span class="p">))</span>  <span class="c1"># ✅ True (상속 없이도 통과)
</span></code></pre></div></div> <p>덕 타이핑의 유연함을 유지하면서 타입 체계도 일부 활용하는 절충안이다. 단, 개발자가 수동으로 등록해야 한다는 번거로움이 있다.</p> <table> <thead> <tr> <th>상황</th> <th>덕 타이핑</th> </tr> </thead> <tbody> <tr> <td>ABC + <code class="language-plaintext highlighter-rouge">isinstance()</code> 강제 체크</td> <td>❌ 깨짐</td> </tr> <tr> <td>ABC만 정의, 체크 없이 호출</td> <td>✅ 유지</td> </tr> <tr> <td>ABC + <code class="language-plaintext highlighter-rouge">register()</code> 활용</td> <td>〰 절충</td> </tr> </tbody> </table> <h2 id="타입-힌트는-덕-타이핑과-무관하다">타입 힌트는 덕 타이핑과 무관하다</h2> <p>타입 힌트는 런타임에 <strong>완전히 무시</strong>된다. <code class="language-plaintext highlighter-rouge">animal: Duck</code>이라고 써도 파이썬 인터프리터는 실행 중에 이를 전혀 검사하지 않는다.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">:</span> <span class="n">Duck</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
    <span class="k">return</span> <span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">()</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># ✅ 아무 문제 없이 실행됨
</span></code></pre></div></div> <p>타입 힌트가 실제로 하는 일은 다음과 같다.</p> <ul> <li><strong>정적 분석</strong>: mypy, pyright 같은 타입 체커가 코드 실행 전에 경고를 띄워줌</li> <li><strong>IDE 지원</strong>: 자동완성과 경고 표시에 활용됨</li> <li><strong>문서화</strong>: 코드를 읽는 사람에게 의도를 전달하는 역할</li> </ul> <p>mypy를 CI에 붙여 강제하면 실질적으로 덕 타이핑을 제한하는 효과가 생기지만, 이는 어디까지나 개발 단계의 검사다. 런타임 동작은 여전히 덕 타이핑이다.</p> <h2 id="protocol-타입-힌트를-위한-덕-타이핑">Protocol: 타입 힌트를 위한 덕 타이핑</h2> <p>mypy를 사용하면서도 덕 타이핑의 철학을 유지하고 싶을 때 <strong>Protocol</strong>을 쓴다. Protocol의 핵심 존재 이유는 바로 <strong>타입 힌트와 정적 분석</strong>이다.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">typing</span> <span class="kn">import</span> <span class="n">Protocol</span>

<span class="k">class</span> <span class="nc">Soundable</span><span class="p">(</span><span class="n">Protocol</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span> <span class="bp">...</span>

<span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">:</span> <span class="n">Soundable</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
    <span class="c1"># "sound() 메서드만 있으면 OK"
</span>    <span class="k">return</span> <span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">()</span>

<span class="k">class</span> <span class="nc">Robot</span><span class="p">:</span>  <span class="c1"># Soundable을 상속하지 않아도
</span>    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">삐빅!</span><span class="sh">"</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># ✅ mypy 통과, 런타임도 통과
</span></code></pre></div></div> <p>이를 <strong>구조적 서브타이핑(Structural Subtyping)</strong> 이라고 부른다. 상속 관계 없이 “이 메서드를 가지고 있으면 이 타입으로 간주”하는 방식으로, 타입 힌트 세계에서 덕 타이핑의 정신을 그대로 구현한다.</p> <h3 id="runtime_checkable-런타임에서도-사용하기">@runtime_checkable: 런타임에서도 사용하기</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">typing</span> <span class="kn">import</span> <span class="n">Protocol</span><span class="p">,</span> <span class="n">runtime_checkable</span>

<span class="nd">@runtime_checkable</span>
<span class="k">class</span> <span class="nc">Soundable</span><span class="p">(</span><span class="n">Protocol</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span> <span class="bp">...</span>

<span class="nf">print</span><span class="p">(</span><span class="nf">isinstance</span><span class="p">(</span><span class="nc">Robot</span><span class="p">(),</span> <span class="n">Soundable</span><span class="p">))</span>  <span class="c1"># ✅ True
# 메서드 존재 여부를 자동으로 확인
</span></code></pre></div></div> <p>ABC의 <code class="language-plaintext highlighter-rouge">register()</code>와 비슷하지만, 개발자가 수동으로 등록할 필요 없이 메서드의 존재만으로 자동으로 판단한다.</p> <table> <thead> <tr> <th>방식</th> <th>덕 타이핑</th> <th>정적 분석</th> <th>런타임 isinstance</th> </tr> </thead> <tbody> <tr> <td>순수 덕 타이핑</td> <td>✅</td> <td>❌</td> <td>❌</td> </tr> <tr> <td>ABC + <code class="language-plaintext highlighter-rouge">register()</code></td> <td>〰</td> <td>❌</td> <td>✅ (수동)</td> </tr> <tr> <td>Protocol</td> <td>✅</td> <td>✅</td> <td>❌</td> </tr> <tr> <td>Protocol + <code class="language-plaintext highlighter-rouge">runtime_checkable</code></td> <td>✅</td> <td>✅</td> <td>✅ (자동)</td> </tr> </tbody> </table> <h2 id="정리">정리</h2> <ul> <li>덕 타이핑은 “무엇인가”보다 “무엇을 할 수 있는가”로 타입을 판단한다.</li> <li>ABC는 사용 방식에 따라 덕 타이핑을 깨뜨릴 수도, 유지할 수도 있다.</li> <li>타입 힌트는 런타임에 무시되므로 덕 타이핑에 직접 영향을 주지 않는다.</li> <li>Protocol은 타입 힌트 + 정적 분석 환경에서 덕 타이핑 철학을 유지하는 수단이다.</li> </ul>]]></content><author><name></name></author><category term="python"/><category term="python"/><category term="type-system"/><category term="oop"/><summary type="html"><![CDATA[덕 타이핑이란 무엇인지, ABC와 Protocol이 덕 타이핑에 어떤 영향을 미치는지, 그리고 타입 힌트가 덕 타이핑과 어떻게 공존하는지에 대한 설명.]]></summary></entry><entry><title type="html">Duck typing, and how Python handles types</title><link href="https://willagio.com/en/blog/2026/duck-typing/" rel="alternate" type="text/html" title="Duck typing, and how Python handles types"/><published>2026-03-24T00:00:00+00:00</published><updated>2026-03-24T00:00:00+00:00</updated><id>https://willagio.com/en/blog/2026/duck-typing</id><content type="html" xml:base="https://willagio.com/en/blog/2026/duck-typing/"><![CDATA[<blockquote> <p>“If it walks like a duck and quacks like a duck, it’s a duck.”</p> </blockquote> <h2 id="what-is-duck-typing">What is duck typing?</h2> <p>Duck typing is the programming philosophy of <strong>judging an object’s type by the methods and attributes it has, not by its class declaration</strong>. It is a common approach in dynamically typed languages such as Python, JavaScript, and Ruby.</p> <p>Rather than checking the type explicitly, you simply call the behaviour you need. If the method or attribute exists, it works; if not, you get a runtime error.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Duck</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">Quack!</span><span class="sh">"</span>

<span class="k">class</span> <span class="nc">Dog</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">Woof!</span><span class="sh">"</span>

<span class="k">class</span> <span class="nc">Robot</span><span class="p">:</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">Beep!</span><span class="sh">"</span>

<span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">):</span>
    <span class="c1"># just call it, no type check
</span>    <span class="nf">print</span><span class="p">(</span><span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">())</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Duck</span><span class="p">())</span>   <span class="c1"># Quack!
</span><span class="nf">make_sound</span><span class="p">(</span><span class="nc">Dog</span><span class="p">())</span>    <span class="c1"># Woof!
</span><span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># Beep!
</span></code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">Duck</code>, <code class="language-plaintext highlighter-rouge">Dog</code>, <code class="language-plaintext highlighter-rouge">Robot</code> — all are treated identically as long as they have a <code class="language-plaintext highlighter-rouge">sound()</code> method. The focus is on <strong>“what can this object do?” rather than “what is this object?”</strong></p> <h2 id="do-abcs-break-duck-typing">Do ABCs break duck typing?</h2> <p>The short answer is <strong>not necessarily.</strong> It depends on how you use them.</p> <h3 id="forcing-an-isinstance-check--duck-typing-broken">Forcing an isinstance() check → duck typing broken</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>

<span class="k">class</span> <span class="nc">Animal</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
    <span class="nd">@abstractmethod</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span> <span class="k">pass</span>

<span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">):</span>
    <span class="k">if</span> <span class="ow">not</span> <span class="nf">isinstance</span><span class="p">(</span><span class="n">animal</span><span class="p">,</span> <span class="n">Animal</span><span class="p">):</span>  <span class="c1"># checking the type directly!
</span>        <span class="k">raise</span> <span class="nc">TypeError</span><span class="p">(</span><span class="sh">"</span><span class="s">not an Animal</span><span class="sh">"</span><span class="p">)</span>
    <span class="nf">print</span><span class="p">(</span><span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">())</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># ❌ rejected even though it has sound()
</span></code></pre></div></div> <h3 id="abc-as-specification-only-no-check--duck-typing-preserved">ABC as specification only, no check → duck typing preserved</h3> <p>If you do not check the type inside the function, duck typing keeps working even with an ABC in play. All the ABC does then is act as an agreement among developers: “this method must be implemented.”</p> <h3 id="a-compromise-with-register">A compromise with register()</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Animal</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="n">Robot</span><span class="p">)</span>  <span class="c1"># register Robot as an Animal
</span>
<span class="nf">print</span><span class="p">(</span><span class="nf">isinstance</span><span class="p">(</span><span class="nc">Robot</span><span class="p">(),</span> <span class="n">Animal</span><span class="p">))</span>  <span class="c1"># ✅ True (passes without inheritance)
</span></code></pre></div></div> <p>This is a middle ground that keeps duck typing’s flexibility while making partial use of the type system. The drawback is that a developer has to register things manually.</p> <table> <thead> <tr> <th>Situation</th> <th>Duck typing</th> </tr> </thead> <tbody> <tr> <td>ABC + forced <code class="language-plaintext highlighter-rouge">isinstance()</code> check</td> <td>❌ broken</td> </tr> <tr> <td>ABC defined only, called without a check</td> <td>✅ preserved</td> </tr> <tr> <td>ABC + <code class="language-plaintext highlighter-rouge">register()</code></td> <td>〰 compromise</td> </tr> </tbody> </table> <h2 id="type-hints-have-nothing-to-do-with-duck-typing">Type hints have nothing to do with duck typing</h2> <p>Type hints are <strong>completely ignored</strong> at runtime. Writing <code class="language-plaintext highlighter-rouge">animal: Duck</code> does not make the Python interpreter check anything during execution.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">:</span> <span class="n">Duck</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
    <span class="k">return</span> <span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">()</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># ✅ runs without any problem
</span></code></pre></div></div> <p>What type hints actually do is this:</p> <ul> <li><strong>Static analysis</strong>: type checkers like mypy and pyright warn you before the code runs</li> <li><strong>IDE support</strong>: used for autocompletion and warnings</li> <li><strong>Documentation</strong>: conveying intent to whoever reads the code</li> </ul> <p>Wiring mypy into CI effectively constrains duck typing, but that is a check at development time. The runtime behaviour is still duck typing.</p> <h2 id="protocol-duck-typing-for-type-hints">Protocol: duck typing for type hints</h2> <p>When you want to use mypy and still keep the spirit of duck typing, you use <strong>Protocol</strong>. The whole reason Protocol exists is <strong>type hints and static analysis</strong>.</p> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">typing</span> <span class="kn">import</span> <span class="n">Protocol</span>

<span class="k">class</span> <span class="nc">Soundable</span><span class="p">(</span><span class="n">Protocol</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span> <span class="bp">...</span>

<span class="k">def</span> <span class="nf">make_sound</span><span class="p">(</span><span class="n">animal</span><span class="p">:</span> <span class="n">Soundable</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
    <span class="c1"># "having a sound() method is enough"
</span>    <span class="k">return</span> <span class="n">animal</span><span class="p">.</span><span class="nf">sound</span><span class="p">()</span>

<span class="k">class</span> <span class="nc">Robot</span><span class="p">:</span>  <span class="c1"># without inheriting from Soundable
</span>    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="sh">"</span><span class="s">Beep!</span><span class="sh">"</span>

<span class="nf">make_sound</span><span class="p">(</span><span class="nc">Robot</span><span class="p">())</span>  <span class="c1"># ✅ passes mypy, passes at runtime
</span></code></pre></div></div> <p>This is called <strong>structural subtyping</strong>. It treats an object as belonging to a type if it has the required methods, with no inheritance relationship — the spirit of duck typing, implemented inside the world of type hints.</p> <h3 id="runtime_checkable-using-it-at-runtime-too">@runtime_checkable: using it at runtime too</h3> <div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="n">typing</span> <span class="kn">import</span> <span class="n">Protocol</span><span class="p">,</span> <span class="n">runtime_checkable</span>

<span class="nd">@runtime_checkable</span>
<span class="k">class</span> <span class="nc">Soundable</span><span class="p">(</span><span class="n">Protocol</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">sound</span><span class="p">(</span><span class="n">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span> <span class="bp">...</span>

<span class="nf">print</span><span class="p">(</span><span class="nf">isinstance</span><span class="p">(</span><span class="nc">Robot</span><span class="p">(),</span> <span class="n">Soundable</span><span class="p">))</span>  <span class="c1"># ✅ True
# checks for the presence of the methods automatically
</span></code></pre></div></div> <p>This resembles an ABC’s <code class="language-plaintext highlighter-rouge">register()</code>, but there is no manual registration: the decision is made automatically from the presence of the methods.</p> <table> <thead> <tr> <th>Approach</th> <th>Duck typing</th> <th>Static analysis</th> <th>Runtime isinstance</th> </tr> </thead> <tbody> <tr> <td>Pure duck typing</td> <td>✅</td> <td>❌</td> <td>❌</td> </tr> <tr> <td>ABC + <code class="language-plaintext highlighter-rouge">register()</code></td> <td>〰</td> <td>❌</td> <td>✅ (manual)</td> </tr> <tr> <td>Protocol</td> <td>✅</td> <td>✅</td> <td>❌</td> </tr> <tr> <td>Protocol + <code class="language-plaintext highlighter-rouge">runtime_checkable</code></td> <td>✅</td> <td>✅</td> <td>✅ (automatic)</td> </tr> </tbody> </table> <h2 id="summary">Summary</h2> <ul> <li>Duck typing judges a type by “what it can do”, not “what it is”.</li> <li>Depending on how you use them, ABCs can either break duck typing or preserve it.</li> <li>Type hints are ignored at runtime, so they do not directly affect duck typing.</li> <li>Protocol is the way to keep the duck typing philosophy in a type-hinted, statically analysed codebase.</li> </ul>]]></content><author><name></name></author><category term="python"/><category term="python"/><category term="type-system"/><category term="oop"/><summary type="html"><![CDATA[What duck typing is, how ABCs and Protocol affect it, and how type hints coexist with duck typing.]]></summary></entry></feed>