X-Git-Url: https://fleuret.org/cgi-bin/gitweb/gitweb.cgi?a=blobdiff_plain;f=report%2Fculture.tex;h=43aaefe651eed71537f6a48ffa2c16a27024febd;hb=779e3675414e061ad294c6b5599a7843d9e887bc;hp=f8f77671ecfb92fc0974aa4c6bb9c97a72bfd667;hpb=7b07bf3ee13853beac7800327825fcc68ca58dd8;p=culture.git

diff --git a/report/culture.tex b/report/culture.tex
index f8f7767..43aaefe 100644
--- a/report/culture.tex
+++ b/report/culture.tex
@@ -83,7 +83,9 @@ Fran\c cois Fleuret
 
 \vspace*{2ex}
 
-\centerline{\color{red}(work in progress, to be updated)}\\[3ex]
+\centerline{\color{red}(work in progress, to be updated)}
+
+\medskip
 
 \centerline{\url{https://fleuret.org/public/culture/culture.pdf}}
 
@@ -188,9 +190,61 @@ present in the original quizzes:
 \includegraphics[scale=0.35]{pics/occlusions_1.png}
 \end{center}
 
+\section{Various thoughts}
+
+\begin{itemize}
+
+\item The whole process can be envisioned as natural selection of
+  quizzes in the representation landscape of GPTs. There probably is a
+  subtle relation between the temperature (mutation rate) and the
+  number of models used to validate with the ``all but one'' criterion
+  (survival criterion).
+
+\item The ``all but one'' could be ``all but K'', and there may be
+  some information-theoretical thing, where the goal is to maximize
+  mutual information, with $K=N$ being total randomness, so high
+  entropy but no structure, and $K=0$ is total determinism, so no
+  information to share.
+
+\item The setup does not push toward any specific invariance or
+  property in the generated quizzes, their consistency is entirely due
+  to the statistics of the ``world quizzes'' that remain in the
+  training set, and to the GPTs' inductive biased.
+
+\item The GPTs obviously get a sense of objectness and 2d topology
+  early on, since they rapidly increase the number of birds and
+  ``discover'' occlusion even though they never was in the world
+  quizzes.
+
+\item There may not be so many problems that can be cast as pairs of
+  patterns that are each a deterministic function of the other, which
+  is probably critical here.
+
+\item This overall process probably fight the ``simplicity bias'': If
+  a model is lacking a ``cue'' that the others have, there will
+  rapidly be quizzes that require this cue, they will be added to the
+  training data, and that model will catch up.
+
+\item The randomness of the process probably allow to even go beyond
+  just synchronizing the abilities of the models. There may be some
+  additional complexification of quizzes that get accepted by chance.
+
+\item The current process to generate new quizzes, which simply
+  samples them at random is very rudimentary and probably not
+  sufficient in a real-data setup. It can probably be supplemented
+  with a MCTS-type search.
+
+\item There may be already in the generated quizzes some structure
+  that \textemph{we} do not pick up (e.g. certain color or motion
+  patterns).
+
+\end{itemize}
+
 \section*{Appendix}
 
-The code is available at\\[-2ex]
+The code is available at
+
+\medskip
 
 \centerline{\url{https://fleuret.org/git/culture}}