@@ -33,19 +33,26 @@ agent itself. \rmk{this should be largely emphasized.}
%measure
To measure the compositionality of emerged symbolic language, many metrics are
proposed~\cite{}.
Widely accepted metrics can be classified into two categories, measuring
positive signaling~\cite{} and measuring positive listening~\cite{}. The former
metrics measure the relationship between spoken symbols and received concepts
\rmk{not clear}, from the perspective of \emph{speakers}.
For example,.
The latter metrics measure the relationship between received symbols and
predicted concepts \rmk{not clear}, from the perspective of \emph{listeners}.
For example,.
However, these metrics are not appropriate, for they only measure
compositionality of symbolic language in \emph{unilateral} role\rmk{not sure},
either speakers or listeners. They can not measure the degree of \emph{bilateral}
understanding between speakers and listeners, i.e., the concept-symbol mapping
consistency between speakers and listeners.
%Widely accepted metrics can be classified into two categories, measuring
%positive signaling~\cite{} and measuring positive listening~\cite{}. The former
%metrics measure the relationship between spoken symbols and received concepts
%\rmk{not clear}, from the perspective of \emph{speakers}.
%For example,.
%The latter metrics measure the relationship between received symbols and
%predicted concepts \rmk{not clear}, from the perspective of \emph{listeners}.
%For example,.
%However, these metrics are not appropriate, for they only measure
%compositionality of symbolic language in \emph{unilateral} role\rmk{not sure},
%either speakers or listeners. They can not measure the degree of \emph{bilateral}
%understanding between speakers and listeners, i.e., the concept-symbol mapping
%consistency between speakers and listeners.
At the initial stage, many researches only analyzed the language compositionality qualitatively.
For example, ~\citet{choi2018compositional} printed the agent messages with the letter `abcd' at some training round, and directly analyzed the compositionality on these messages.
~\citet{kottur-etal-2017-natural} introduced the dialog tree to show the evolution of language compositionality during the trianing process.
Latter, some quantitative metrics are explored.
The topographic similarity\cite{lazaridou2018emergence} is introduced to measure the distances between all the possible pairs of meanings and the corresponding pairs of signals.
\citet{chaabouni2020compositionality} proposed the positional disentanglement and the bag-of-symbols disentanglement. The positional disentanglement measures whether symbols in specific postion clearly relate to the specific attribute of the input object. The bag-of-symbols measure the permutation-invariant characteristic of a language.
