~

AAAI2021/tex/experiments.tex

 \section{Experiments}
 \label{sec:exp}
 
-\subsection{}
-\label{ssec:}
+%\section{Agent Capacity vs. Compositionality}
+%\label{ssec:exp}
 
-We examine the compositionality of symbolic language that emerged in our natural
-referential game with various vocabulary size. For each configuration of
+We examine the relationship between agent capacity and the compositionality of
+symbolic language that emerged in our natural referential game with various
+vocabulary size. 
+For each configuration of
 vocabulary size, we train the speaker-listener agents to emerge symbolic
-language under different agent capacities, i.e., hidden layer size ($h_{size}$).
+language when varying the agent capacities, i.e., hidden layer size
+($h_{size}$), from 6 to 100. 
+
+Figure~\ref{fig:exp1} reports the experimental results. It can be observed that
+the mean value of MIS decreases as the value of $h_{size}$ increases. Taking the
+configuration of vocabulary size $|V|=10$ as an example, the mean value of MIS
+is around 0.8 when $h_{size}\le 20$; MIS significantly decreases to 0.75 when
+$h_{size}$ increases from 20 to 40; MIS further reduces to 0.7 when $h_{size}$
+increases from 40 to 100. For different vocabulary sizes, the MIS shares the
+similar behaviour. In summary, lower agent capacity improves the possibility of
+emerging high compositional symbolic language.
+
 
 \begin{figure}[t]
   \centering
   \includegraphics[width=0.9\columnwidth]{fig/occupy}
-  \caption{Compositionality of symbolic language under different parameters.}
+  \caption{Compositionality of symbolic language under different parameters
+    (mean value with variation $[\mu-\sigma,\mu+\sigma]$).}
   \label{fig:exp1}
 \end{figure}
 
 
+We further breakdown our results to investigate the importance of agent capacity
+to the compositionality of symbolic language. Figure~\ref{fig:exp2} reports the
+ratio of high compositional symbolic language in all emerged languages,
+Figure~\ref{fig:exp2} (a) and (b) for $MIS>0.99$ and $MIS>0.9$, respectively. It
+cam be observed that the ratio of high compositional symbolic languages
+decreases drastically with the increase of $h_{size}$. Especially, when $h_size$
+is large enough (e.g., $>40$), high compositional symbolic language is hard to
+emerge in a natural referential game.
+
+
+\begin{figure}[t]
+  \centering
+  \includegraphics[width=0.9\columnwidth]{fig/occupy}
+  \caption{The ratio of high compositional language. (a) $h_{size}>0.99$. (b) $h_{size}>0.9$}
+  \label{fig:exp2}
+\end{figure}
+
+
+%\subsection{Breakdown}
+%\label{ssec:language}
+
+\textbf{Breakdown into language teaching.}
+We further breakdown the learning process to investigate the language teaching
+scenario, where the Speaker teaches the Listener its fixed symbolic language.
+We define three symbolic languages in different compositionality for Speaker to
+teach, i.e., high (LA, $MIS=1$), mediate (LB, $MIS=0.83$), low (LC, $MIS=0.41$), see
+Figure~\ref{fig:bench}.  
+
+\begin{figure}[t]
+  \centering
+  \includegraphics[width=0.9\columnwidth]{fig/occupy}
+  \caption{Three pre-defined language for teaching. (a) LA: high compositionality
+    ($MIS=1$). (b) LB: mediate compositionality ($MIS=0.83$). (c) LC: low compositionality ($MIS=0.41$).}
+  \label{fig:bench}
+\end{figure}
+
+
+\begin{figure}[t]
+  \centering
+  \includegraphics[width=0.9\columnwidth]{fig/occupy}
+  \caption{}
+  \label{fig:exp3}
+\end{figure}
+
+
+%\begin{figure}[t]
+%  \centering
+%  \includegraphics[width=0.9\columnwidth]{fig/occupy}
+%  \caption{}
+%  \label{fig:exp4}
+%\end{figure}
+
+
+Figure~\ref{fig:exp3} reports the accurcy of Listener, i.e., correctly
+predicting the symbols spoken by Speaker ($t=\hat(t)$), which varies with the
+training iterations under different agent capacities.
+Figure~\ref{fig:exp3} (a) shows that when $h_size$ equals to 1, the agent capacity is
+too low to handle languages. Figure~\ref{fig:exp3} (b) shows that when $h_size$
+equals to 2, agent can only learn $LA$ whose compositionality (i.e. \emph{MIS})
+is highest in all three languages. Combing these two observations, we can infer that
+language with lower compositionality need higher agent capacity to ensure communicating
+successfully (i.e., $h_size$). Figure~\ref{fig:exp3} (c) to (h) show that the
+higher agent capacity cause a faster training process for all three languages, but the
+improvement for different languages is quite different.
+It is obvious that language with lower compostionality also need higher agent
+capacity to training faster.
 
-\subsection{}
-\label{ssec:}
 
+%In conclude, teaching an artificial language with
+%lower compositionality to agent require higher agent capacity both for learning
+%successfully and training faster.