title={Emergence of Communication in an Interactive World with Consistent Speakers},
author={Bogin, Ben and Geva, Mor and Berant, Jonathan},
journal={arXiv},
pages={arXiv--1809},
year={2018}
}
@inproceedings{jaques2019social,
title={Social influence as intrinsic motivation for multi-agent deep reinforcement learning},
author={Jaques, Natasha and Lazaridou, Angeliki and Hughes, Edward and Gulcehre, Caglar and Ortega, Pedro and Strouse, DJ and Leibo, Joel Z and De Freitas, Nando},
booktitle={International Conference on Machine Learning},
pages={3040--3049},
year={2019},
organization={PMLR}
}
@article{mul2019mastering,
title={Mastering emergent language: learning to guide in simulated navigation},
author={Mul, Mathijs and Bouchacourt, Diane and Bruni, Elia},
journal={arXiv preprint arXiv:1908.05135},
year={2019}
}
@inproceedings{kharitonov2019egg,
title={EGG: a toolkit for research on Emergence of lanGuage in Games},
author={Kharitonov, Eugene and Chaabouni, Rahma and Bouchacourt, Diane and Baroni, Marco},
booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations},
pages={55--60},
year={2019}
}
@article{labash2020perspective,
title={Perspective taking in deep reinforcement learning agents},
author={Labash, Aqeel and Aru, Jaan and Matiisen, Tambet and Tampuu, Ardi and Vicente, Raul},
journal={Frontiers in Computational Neuroscience},
The emergence of symbolic language has always been an important issue,
The emergence of language has always been an important issue,
which attracts attention from a broad range of communities,
which attracts attention from a broad range of communities,
including philology~\cite{}, biology~\cite{}, and computer
including philology~\cite{}, biology~\cite{}, and computer
science~\cite{}. Especially in computer science, efforts in recent years trying to explore
science~\cite{}. Especially in computer science, efforts in recent years trying to explore
the emergence of symbolic language in virtual multi-agent environments, where
the emergent language in virtual multi-agent environments, where
agents are trained to communicate with neural-network-based methods such as deep
agents are trained to communicate with neural-network-based methods such as deep
reinforcement learning~\cite{}.
reinforcement learning~\cite{}.
%Such works can be roughly classified into two categories,
%Such works can be roughly classified into two categories,
...
@@ -13,7 +13,7 @@ reinforcement learning~\cite{}.
...
@@ -13,7 +13,7 @@ reinforcement learning~\cite{}.
%the environment setting.
%the environment setting.
The quality of emergent symbolic language is typically measured by its
The quality of emergent language is typically measured by its
\emph{compositionality}.
\emph{compositionality}.
Compositionality is a principle that determines
Compositionality is a principle that determines
whether the meaning of a complex expression (e.g, phrase), which is assembled out of a
whether the meaning of a complex expression (e.g, phrase), which is assembled out of a
...
@@ -72,14 +72,16 @@ vocabulary can express almost infinite concepts.}
...
@@ -72,14 +72,16 @@ vocabulary can express almost infinite concepts.}
\end{table*}
\end{table*}
Prior studies focus on achieving high compositional symbolic language
Prior studies focus on achieving high compositional symbolic language
through \emph{deliberately handcrafted} inductions, e.g., small vocabulary
through \emph{deliberately handcrafted} inductions, e.g., additional rewards~\cite{},
sizes~\cite{}, memoryless~\cite{}, additional rewards~\cite{}, constructed loss functions~\cite{}, and
constructed loss functions~\cite{}, structural input data~\cite{},
ease-of-teaching~\cite{}. \note{Such optimization methodologies are driven by the challenges to generate high compositional symbolic without induction in an existing multi-agent environment.}
memoryless~\cite{}, and ease-of-teaching~\cite{}.
\note{Such optimization methodologies are driven by the challenges to generate high compositional symbolic without induction in an existing multi-agent environment.}
Figure~\ref{fig:induction} reports the compositionality when training two agents
Figure~\ref{fig:induction} reports the compositionality when training two agents
in the widely-used listener-speaker referential game for emerging 100 symbolic
in the widely-used listener-speaker referential game~\cite{} for emerging 100
languages, and it can be observed that \note{the compositionality
languages, and it can be observed that \note{the compositionality
of emerged symbolic language is extremely low without any induction. Moreover, varying
of emergent language is seldom high (e.g., $<5\%$ for compositionality $>0.99$)
without any induction. Moreover, varying
the vocabulary size does not affect the compositionality notably.}
the vocabulary size does not affect the compositionality notably.}
Though such unnatural inductions are useful, they prevent us from better understanding the mystery of
Though such unnatural inductions are useful, they prevent us from better understanding the mystery of
the emergence of language and even intelligence among our pre-human ancestors.
the emergence of language and even intelligence among our pre-human ancestors.
...
@@ -89,14 +91,14 @@ In other words, it is never clear whether \emph{natural}
...
@@ -89,14 +91,14 @@ In other words, it is never clear whether \emph{natural}
environment and agents are sufficient for achieving high compositionality.
environment and agents are sufficient for achieving high compositionality.
This paper is the first one to achieve high compositional
This paper is the first one to achieve high compositional
symbolic language without any deliberately handcrafted induction. The key observation
language without any deliberately handcrafted induction. The key observation
is that the internal \emph{agent capacity} plays a crucial role in the
is that the internal \emph{agent capacity} plays a crucial role in the
compositionality of symbolic language.
compositionality of emergent language.
%by thoroughly
%by thoroughly
%analyzing the compositionality after removing the inductions in
%analyzing the compositionality after removing the inductions in
%the most widely-used listener-speaker referential game framework.
%the most widely-used listener-speaker referential game framework.
Concretely, the relationship between the agent capacity and the compositionality
Concretely, the relationship between the agent capacity and the compositionality
of symbolic language is characterized, with a novel mutual information-based
of emergent language is characterized, with a novel mutual information-based
metric for the compositionality.
metric for the compositionality.
%both theoretically and experimentally.
%both theoretically and experimentally.
%theoretically
%theoretically
...
@@ -112,10 +114,10 @@ capacity and the compositionality of symbolic language that emerged
...
@@ -112,10 +114,10 @@ capacity and the compositionality of symbolic language that emerged
%Regarding the experimental validation, it is conducted on a listener-speaker
%Regarding the experimental validation, it is conducted on a listener-speaker
%referential game framework with eliminated unnatural inductions.
%referential game framework with eliminated unnatural inductions.
Both the theoretical analysis and experimental results lead to a counter-intuitive
Both the theoretical analysis and experimental results lead to a counter-intuitive
conclusion that \emph{lower agent capacity facilitates the emergence of symbolic language
conclusion that \emph{lower agent capacity facilitates the emergence of language
with higher compositionality}. \note{Therefore, by only reducing the agent capacity
with higher compositionality}. \note{Therefore, by only reducing the agent capacity
in such a natural environment, we
in such a natural environment, we
can generate a higher compositional symbolic language with a higher probability.}
can generate a more compositional language with a higher probability.}
%Prior studies focus on investigating how to affect the
%Prior studies focus on investigating how to affect the
Previous works focus on the external environmental factors that impact the
Previous works focus on the external environmental factors that impact the
compositionality of emerged symbolic language.
compositionality of emerged symbolic language.
Some significant works on studying the external environmental factor on the compositionality of emergent language are summarized on Table~\ref{tab:rel}.
Some significant works on studying the external environmental factor on the compositionality of emergent language are summarized in Table~\ref{tab:rel}.
For example, ~\citet{kirby2015compression} explored how the pressures for expressivity and compressibility lead the structured language.
For example, ~\citet{kirby2015compression} explored how the pressures for expressivity and compressibility lead the structured language.
~\citet{kottur-etal-2017-natural} constrained the vocabulary size and whether the listener has memory to coax the compositionality of the emergent language.
~\citet{kottur-etal-2017-natural} constrained the vocabulary size and whether the listener has memory to coax the compositionality of the emergent language.
~\citet{lazaridou2018emergence} showed that the degree of structure found in the input data affects the emergence of the symbolic language.
~\citet{lazaridou2018emergence} showed that the degree of structure found in the input data affects the emergence of the symbolic language.
~\citet{li2019ease} studied how the pressure, ease of teaching, impact on the iterative language of the population regime.
~\citet{li2019ease} studied how the pressure, ease of teaching, impact on the iterative language of the population regime.
~\citet{evtimova2018emergent} designed a novel multi-modal scenarios, which the speaker and the listener should access to different modalities of the input object, to explore the language emergence.
~\citet{evtimova2018emergent} designed novel multi-modal scenarios, which the speaker and the listener should access to different modalities of the input object, to explore the language emergence.
Such factors are deliberately designed, which are too ideal to be true in
Such factors are deliberately designed, which are too ideal to be true in
the real world.
the real world.
In this paper, these handcrafted inductions above are all removed, and the high compostional language is leaded only by the agent capacity.
In this paper, these handcrafted inductions above are all removed, and the high compositional language is leaded only by the agent capacity.
%either speakers or listeners. They can not measure the degree of \emph{bilateral}
%either speakers or listeners. They can not measure the degree of \emph{bilateral}
%understanding between speakers and listeners, i.e., the concept-symbol mapping
%understanding between speakers and listeners, i.e., the concept-symbol mapping
%consistency between speakers and listeners.
%consistency between speakers and listeners.
At the initial stage, many researches only analyzed the language compositionality qualitatively.
At the initial stage, many studies only analyzed the language compositionality qualitatively.
For example, ~\citet{choi2018compositional} printed the agent messages with the letter `abcd' at some training round, and directly analyzed the compositionality on these messages.
For example, ~\citet{choi2018compositional} printed the agent messages with the letter `abcd' at some training round, and directly analyzed the compositionality on these messages.
~\citet{kottur-etal-2017-natural} introduced the dialog tree to show the evolution of language compositionality during the trianing process.
~\citet{kottur-etal-2017-natural} introduced the dialog tree to show the evolution of language compositionality during the training process.
Latter, some quantitative metrics are explored.
Latter, some quantitative metrics are explored.
The topographic similarity\cite{lazaridou2018emergence} is introduced to measure the distances between all the possible pairs of meanings and the corresponding pairs of signals.
The topographic similarity\cite{lazaridou2018emergence} is introduced to measure the distances between all the possible pairs of meanings and the corresponding pairs of signals.
\citet{chaabouni2020compositionality} proposed the positional disentanglement, which measures whether symbols in specific postion clearly relate to the specific attribute of the input object.
\citet{chaabouni2020compositionality} proposed the positional disentanglement, which measures whether symbols in a specific position relate to the specific attribute of the input object.
From Table~\ref{tab:rel}, most metrics are proposed on the sight of the speaker. In our view, human begings developed the language based on both the speakers and the listener. Only one research of \cite{choi2018compositional} in Table~\ref{tab:rel} qualitatively considered from the sight of the speaker and the listener. In this paper, we propose a novel quatitative metric from both the speaker's sight and the listener's sight.
From Table~\ref{tab:rel}, most metrics are proposed on the sight of the speaker. In our view, human beings developed the language based on both the speakers and the listener. Only one research of \cite{choi2018compositional} in Table~\ref{tab:rel} qualitatively considered from the perspective of the speaker and the listener. In this paper, we propose a novel quantitative metric from both the speaker's sight and the listener's sight.
In conclusion, the previous works coaxed the compositional language based on some careful designed handcrafted inductions,
In conclusion, the previous works coaxed the compositional language based on some careful designed handcrafted inductions,
@@ -62,32 +62,15 @@ Before going to the detail of the training algorithms, we first introduce the en
...
@@ -62,32 +62,15 @@ Before going to the detail of the training algorithms, we first introduce the en
\subsection{Environment setup}
\subsection{Environment setup}
\label{ssec:env}
\label{ssec:env}
Figure~\ref{fig:game} shows the entire environment used in this study,
Figure~\ref{fig:game} shows the entire environment used in this study,
i.e., a commonly used referential game. Roughly, the referential game requires
i.e., a commonly used referential game. Roughly, the referential game requires the speaker and listener to work cooperatively to accomplish a certain task.
the speaker and listener working cooperatively to accomplish a certain task.
In this paper, the task is to have the listener agent reconstruct the object
In this paper, the task is to have the listener agent reconstruct the object
what the speaker claims it has seen, only through their emerged communication
what the speaker claims it has seen, only through their emerged communication protocol. The success in this game indicates that symbolic language has emerged between speaker and listener.
protocol. The success in this game indicates that symbolic language has emerged
between speaker and listener.
\textbf{Game rules} In our referential game, agents follow the following rules
\textbf{Game rules} In our referential game, agents follow the following rules to finish the game in a cooperative manner. In each round, once received an input object $t$, Speaker $S$ speaks a symbol sequence $s$ to Listener $L$ ; Listener $L$ reconstruct the predicted result $\hat{t}$ based on the listened sequence $s$; if $t=\hat{t}$, agents win this game and receive positive rewards ($r(t,\hat{t})=1$); otherwise agents fail this game and receive negative rewards ($r(t,\hat{t})=-1$).
to finish the game in a cooperative manner. In each round, once received an
input object $t$, Speaker $S$ speaks a symbol sequence $s$ to Listener $L$ ;
Listener $L$ reconstruct the predicted result $\hat{t}$ based on the listened
sequence $s$; if $t=\hat{t}$, agents win this game and receive positive rewards
($r(t,\hat{t})=1$); otherwise agents fail this game and receive negative rewards
($r(t,\hat{t})=-1$).
Precisely, during the game, Speaker $S$ receives an input object $t$, which is
Precisely, during the game, Speaker $S$ receives an input object $t$, which is an expression with two words from the vocabulary set $V$, i.e., two one-hot vectors representing shape and color, respectively. Based on the $t$, Speaker $S$ speaks a symbol sequence $s$, which similarly contains two words from $V$. The Listener $L$ receives $s$ and output predicted result $\hat{t}$, a single word (one-hot vector) selected from the Cartesian product of set two $V$s ($V\times V$), which represents all the meanings of two combined words from $V$. Please note that since $t$ and $\hat{t}$ have different length, we say $t=\hat{t}$ if $t$ expresses the same meaning as $\hat{t}$, e.g., ``red circle''.
an expression with two words from the vocabulary set $V$, i.e., two
one-hot vector representing shape and color, respectively. Based on the $t$,
Speaker $S$ speaks a symbol sequence $s$, which similarly contains two words
from $V$. The Listener $L$ receives $s$ and output predicted result $\hat{t}$,
a single word (one-hot vector) selected from the Cartesian product of set two $V$s
($V\times V$), which representing all the meanings of two combined words from $V$.
Please note that since $t$ and $\hat{t}$ have different length, we say
$t=\hat{t}$ if $t$ expresses the same meaning as $\hat{t}$, e.g., ``red circle''.
...
@@ -134,8 +117,8 @@ expected reward$ J(\theta_S, \theta_L)$ by fixing the parameter $\theta_S$ and
...
@@ -134,8 +117,8 @@ expected reward$ J(\theta_S, \theta_L)$ by fixing the parameter $\theta_S$ and
adjusting the parameter $\theta_L$.
adjusting the parameter $\theta_L$.
Additionally, to avoid the handcrafted induction on emergent language, we only
Additionally, to avoid the handcrafted induction on emergent language, we only
use the predict result $\hat{t}$ of the listener agent as the
use the predicted result $\hat{t}$ of the listener agent as the
evidence of whether giving the positive rewards. Then, the gradients of the
evidence of whether giving positive rewards. Then, the gradients of the
expected reward $ J(\theta_S, \theta_L)$ can be calculated as follows:
expected reward $ J(\theta_S, \theta_L)$ can be calculated as follows:
Each column of $M$ correspond to the semantic information carried by one symbol. In a perfectly compositional language, each symbol represents one specific concept exclusively. Therefore, the similarity between the columns of $M$ and a one-hot vector is align with the compositionality of the emergent language.
Each column of $M$ corresponds to the semantic information carried by one symbol. In a perfectly compositional language, each symbol represents one specific concept exclusively. Therefore, the similarity between the columns of $M$ and a one-hot vector is aligned with the compositionality of the emergent language.
MIS is a bilateral metric. Unilateral metrics, e.g. \emph{topographic similarity (topo)}\cite{} and \emph{posdis}\cite{}, only take the policy of the speaker into consideration. We provide an example to illustrate the inadequacy of unilateral metrics, shown in Figure~\ref{fig:unilateral}. In this example, the speaker only uses $s_1$ to represent shape. From the perspective of speaker, the language is perfectly compositional (i.e. both topo and posdis are 1). However, the listener cannot distinguish the shape depend only on $s_1$, showing the non-compositionality in this language. The bilateral metric MIS addresses such defect by taking the policy of the listener into account, thus $\mathit{MIS} < 1$.
MIS is a bilateral metric. Unilateral metrics, e.g. \emph{topographic similarity (topo)}\cite{} and \emph{posdis}\cite{}, only take the policy of the speaker into consideration. We provide an example to illustrate the inadequacy of unilateral metrics, shown in Figure~\ref{fig:unilateral}. In this example, the speaker only uses $s_1$ to represent the shape. From the perspective of the speaker, the language is perfectly compositional (i.e. both topo and posdis are 1). However, the listener cannot distinguish the shape depend only on $s_1$, showing the non-compositionality in this language. The bilateral metric MIS addresses such defects by taking the policy of the listener into account, thus $\mathit{MIS} < 1$.
