\documentclass{article}

% if you need to pass options to natbib, use, e.g.:
     \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2020

% ready for submission
%\usepackage{neurips_2020}

% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
%     \usepackage[preprint]{neurips_2020}

% to compile a camera-ready version, add the [final] option, e.g.:
%     \usepackage[final]{neurips_2020}

% to avoid loading the natbib package, add option nonatbib:
\usepackage[nonatbib]{neurips_2020}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage[pdftex]{graphicx}
\usepackage{enumitem}
\usepackage[sort,nocompress]{cite}

%%Added by Du, Zidong
\usepackage{ifthen}
\usepackage{color}
%\definecolor{alizarin}{rgb}{0.82, 0.1, 0.26}
\ifthenelse{\equal{\submitmode}{true}}
{
  %%submission mode
  \newcommand{\rmk}[1]{}
  \newcommand{\rmkdone}[1]{}
  \newcommand{\del}[1]{}
  \newcommand{\add}[1]{#1}
  \newcommand{\zdel}[1]{}
  \newcommand{\zadd}[1]{#1}
  \newcommand{\note}[1]{#1}
  \newcommand{\lnote}[1]{#1}
  \newcommand{\main}[1]{#1}
  \newcommand{\para}[2]{#1#2}
  \newcommand{\ladd}[1]{#1}
  \newcommand{\lanadd}[1]{#1}
  \newcommand{\uw}[1]{#1}
  \newcommand{\doubt}[1]{#1}
}
{
 %%writing mode

 \newcommand{\rmk}[1]{\textcolor{red}{--[#1]--}}%%communication and remarks
 \newcommand{\del}[1]{\textcolor{gray}{\sout{#1}}}%%delete the text
 \newcommand{\rmkdone}[1]{\textcolor{gray}{--[#1]--}}%%done for remarks
 \newcommand{\add}[1]{\textcolor{PineGreen}{#1}}%%add
 \newcommand{\zdel}[1]{\textcolor{gray}{\sout{#1}}}%%used by Zidong
 \newcommand{\zadd}[1]{\textcolor{teal}{#1}}%%used by Zidong
 % \newcommand{\ladd}[1]{\textcolor{BlueViolet}{#1}}
 \newcommand{\ladd}[1]{#1}
 \newcommand{\note}[1]{\textcolor{red}{#1}}%%in the text, need to be checked or filled

 % added by lanhuiying
 % uncertain word
 \newcommand{\uw}[1]{\textcolor{gray}{#1}}
 \newcommand{\lanadd}[1]{\textcolor{cyan}{#1}}
 \newcommand{\doubt}[1]{\textcolor{teal}{#1}}
}


\title{The Natural Emergence of Symbolic Language through a Language Game among
  Individual Agents}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.

\author{%
%%  David S.~Hippocampus\thanks{Use footnote for providing further information
%%    about author (webpage, alternative address)---\emph{not} for acknowledging
%%    funding agencies.} \\
%%  Department of Computer Science\\
%%  Cranberry-Lemon University\\
%%  Pittsburgh, PA 15213 \\
%%  \texttt{hippo@cs.cranberry-lemon.edu} \\
  % examples of more authors
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \AND
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
  % \And
  % Coauthor \\
  % Affiliation \\
  % Address \\
  % \texttt{email} \\
}

\begin{document}

\maketitle

\begin{abstract}
Symbolic language, a language using symbols and characters to represent concepts
for communication, is widely taken as the key factor for the emergence and
evolution of human intelligence. Its emergence among our pre-human ancestors is
still a mystery accompanied with many controversial theories and
hypothesis. Recently, efforts about the symbolic language have been made by
computer scientists in the neural network based multi-agent systems.
%, trying to understand the emergence of symbolic language.
\note{However, even using the multi-agent systems, previous works at root are
training one brain for communicating among connected sensors, as they combine
the neural networks of all agents in training process.} As a result, such works did not
involve the emergence of symbolic language. 
Here, we introduce a novel mechanism with
\emph{Self-grounding-Introspection-Cooperation} model that enables the natural 
emergence of symbolic language among individual agents. Experimental results
show that \note{xxxx}
To the best of our knowledge, we are the first to \note{xxx}
This work presents a trial to shed some lights on the origin of human language
and the emergence of intelligence in individual human.

\end{abstract}


\section{Introduction}

%Symbolic Language is important
Symbolic language, a language using symbols and characters to represent concepts
for communication, is widely taken as the key factor for the emergence and
evolution of human intelligence. xxxxxxxxxx\rmk{some claimes.} Even with so many studies,
the emergence of symbolic language among our pre-human ancestors is still a
mystery accompanied with many controversial theories and hypothesis. This old
problem had been a long-standing challenge in many fields, including artificial
intelligence in computer science, as it could provide insights into 
the origins of artificial intelligence.


%Recent effort for emergence is not correct
Recent efforts in computer science start to explore the emergence of symbolic
language but in a virtual environment among multiple agents by leveraging the
neural network based methods, i.e., deep reinforcement learning. In these works,
researchers put multiple agents into a scenario that agents have to achieve a
pre-defined goal cooperatively. Researchers hope
that agents can evolve a stable communication protocol (e.g., symbolic language)
among themselves by the compulsory communication for cooperation. Such
works can be roughly classified into two categories, referential game and
multi-agent reinforcement learning (MARL), based on the environment setting.
However, previous works, no matter referential game related or multi-agent reinforcement
learning related, ignore the independence of agents in \note{training or
observation of environments, and thusly can be taken as one huge brain with
multiple connected sensors (agents). In other words, previous works did not
really achieve the emergence of symbolic language among \emph{multiple} agents. 
%difference from group intelligence, population intelligence, community intelligence

Besides, previous works also failed on the \emph{naturally emergence}. In
referential games, agents act in a pre-set role of sender or receiver for speaking or
listening, respectively. In MARL, agents xxxxx\rmk{how}

%In this paper, we proposed a SIC model
In this paper, To achieve the naturally emergence of symbolic language among
individual agents, we propose a novel three-step model---the
Self-grounding-Introspection-Cooperation 
(SIC) model. In the first step of SIC model, i.e., \emph{Self-grounding}, each agent is trained
in the virtual environment to perform tasks successfully, through a
self-playing process. In the second step of SIC model, i.e., \emph{Introspection}, each
agent stops to act with the observed target but self-examine the consistency of spoken words
and listened words through a talking-to-oneself process. In the third step of
SIC model, i.e., \emph{Cooperation}, multiple agents are put in a same
environment and are required to perform a certain task together, where the
symbolic language appears. With the SIC model, we could achieve goal of
naturally emergence of symbolic language among multiple agents.  Experimental
results show that \rmk{xxxx} To the best of our knowledge, we are the first to
\rmk{xxx} This work presents a trial to shed some lights on the origin of human
language and the emergence of intelligence in individual human.

%Our contribution
We made the following contributions.
First, we proposed the SIC model, which is the first work to naturally generate symbolic
language among multiple \emph{individual} agents. 
Second, we confirmed the necessity and importance of environment, \rmk{xxxxxx}.
Third, we observed that the intention for cooperation is the base for
communication, \rmk{xxxxx}.
%\begin{itemize}[topsep=0cm,itemsep=0cm,leftmargin=0.3cm]
%\item 
%\item 
%\item 
%\end{itemize}


\section{Background and Motivation}
In this section, we introduce the \emph{Symbolic Language} and the major efforts
related to the emergence of symbolic language.

\subsection{Symbolic Language}

%Symbolic language,
%How to evaluate whether a language is a symbolic language 
Symbolic languages use symbols and characters to represent concepts for
communication. Before diving into the question of emergence, we first examine
the measurement of symbolic language, i.e., whether a generated communication
protocol among agents can be taken as a symbolic language. \rmk{Add related work,
and our proposals.}




\subsection{Multi-agent Systems}
Recent works are focusing on the emergence of grounded symbolic language in
neuron network based multi-agent systems. The grounded symbolic language, where
each symbol is mapped to its own meaning, is obtained by training the agents in
systems. However, even using the multi-agent systems, previous works at root are
training one brain for communicating among connected sensors (agents), as they combine
the neural networks of all agents in the training process.



\begin{figure}
  \centering
  \fbox{\rule[-.5cm]{0.0cm}{4cm} 
    \includegraphics{fig/occupy.pdf}
    \rule[-.5cm]{0.5cm}{0.0cm}}
  \caption{(a) A referential game example~\cite{??}. (b) Training procedure.}
  \label{fig:rg}
\end{figure}

Roughly, to evolve a symbolic language, previous works force the
agents to finish a set target through cooperation, which requires communication
among the agents.
These works can be classified into two categories based on the
environment settings: \emph{referential games} and \emph{multi-agent
reinforcement learning system} (MARL).
%referential game
In \emph{referential games}, agents are divided into \emph{sender} and
\emph{receiver}, for speaking and listening, respectively. As the
the referential game example shown in Figure~\ref{fig:rg}, one agent (Agent A) sends
description of a target picture to another agent (Agent B), who will identify
the target picture from a set of pictures~\cite{??}. However, in training, \note{xxxxxx}.
\rmk{Some other examples}
Moreover, most referential games do not consider the environments, where agents
only send/receive symbols to finish a target together. \rmk{Some examples.}
%marl
In \emph{MARL}, agents are placed in a virtual environment to cooperate in a
continuous action space. For generating symbolic language, agents share the
model parameters and/or environment information. Therefore, those agents can be
taken as different sensors connected to a huge brain, not separate, individual
brains. For example, \rmk{some examples}


Moreover, despite the individual agents issue, two more common flaws existed in previous works.
%intention
First, intention is not considered in the cooperation among agents. Previous
works always allocate each agent a role, either sender or receiver, for forced
communication without considering their intention.
%the actions from other agents
Second, actions of other agents are not considered in the cooperation among
agents. \note{Agents in previous works mainly focus on the targets and the
  environment, without understanding actions from other agents. Thus, their
  cooperation can hardly be taken as cooperation. }


\textbf{Our Proposal.} To achieve the naturally emergence of symbolic language among
individual agents, we propose a novel Self-grounding-Introspection-Cooperation
(SIM) model. There are four key difference between SIC model and previous
works.
First, to the best of our knowledge, SIC is the first work to generate
symbolic language among \emph{individual} agents.
Second, SIC is the first to have symbolic language emerged naturally.
Third, SIC performs the communication based on the intention of agents, not by
forcing agents to communicate.
Fourth, xxxxx.


\section{The Self-grounding-Introspection-Cooperation Model}

In this section, we introduce the proposed Self-grounding-Introspection-Cooperation
(SIM) model in details.


\subsection{Design principles}
To achieve the goal of emerging symbolic language naturally among individual
agents, we decide to follow three key design principles.
\begin{itemize}[topsep=0cm,itemsep=0cm,leftmargin=0.3cm]
\item \textbf{Individual agents.}
\item \textbf{Complex environments.} As \note{mentioned by
  Tomasello~\cite{??}, outside environment is the key factor in the emergence of
humane language and animal language, which can not ignored.}
\item \textbf{.}
\end{itemize}


\begin{figure}
  \centering
  \fbox{\rule[-.5cm]{0.0cm}{4cm} 
    \includegraphics[width=.99\columnwidth]{fig/overall.png}
    \rule[-.5cm]{0.5cm}{0.0cm}}
  \caption{The processing flow of SIC model.}
  \label{fig:sic}
\end{figure}

\subsection{SIC model}

Figure~\ref{fig:sic} shows the entire processing procedure in details.

\textbf{Self-grounding.}

\textbf{Introspection.}

\textbf{Cooperation.}


\subsection{Environment settings}


\subsection{The Language Game}



\section{Experiments}

\subsection{Methodology}


\subsection{Experimental Results}

\textbf{Grounded Language.}

\textbf{Intention.}

\textbf{Common knowledge.}

\section{Related Work}


\section{Conclusions}


\cite{DBLP:conf/iclr/WuLCS18}


\clearpage
\newpage


%%%%%%%%% -- BIB STYLE AND FILE -- %%%%%%%%
\bibliographystyle{IEEEtranS}
\bibliography{ref}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


\end{document}
