1 Introduction
1.1 Social situations and social robotics
The basic idea of social robots refers to robots that are situated in a human social environment. In this context, we expect social robots to exhibit social awareness, i.e. to appraise and maintain a model of the social situation in which they are embedded. Depending on the role of the robot, this might include understanding who is present, who is interacting with whom, which are the resulting groups, what are the in-group roles, etc.
Social awareness, as a socio-cognitive skill, is essential for the robot to e.g. act in a context-sensitive manner; reason and apply social norms (for instance, do not navigate in the middle of a group, or do not suddenly interrupt a conversation); or create proactive social agents (in order to acknowledge and respond to a human who would like to engage with the robot, the robot must first adequately model and recognise the corresponding social situation).
Social situations have indeed been an important object of study in social psychology for a long time [Argyle et al. (1981)], with, for instance, the following definition by Grabett [Garbett (1970)]: a social situation is a temporally and spatially bounded series of events abstracted by the observer from the on-going flow of social life. When specifically looking at artificial agents like robots, their level of social cognition depends on their ability to identify and interpret the world surrounding them [Szczepanowski et al. (2017)], and in particular, correctly interpret transactions of social signals – specific events in which an agent performs a social action aimed at another agent [Pantic et al. (2011)]. This process is known as situational awareness, of which Endsley [Endsley (1995)] defines three levels:perception, interpretation, and evaluation.
While the perception of social signal has been studied in depth in the HRI community (for instance, [Pantic et al. (2011)]), and the evaluation of the social situation is usually handled as an aspect of the robot’s decision making, the interpretation of social situations is a difficult problem. It requires to build and maintain a task-appropriate model of the situations, and represent it in such a way that a machine can reason about it. Embeddings are such representations.
1.2 Embeddings
In the context of machine learning, we refer to an embedding as a real-valued vector representation of a typically much higher dimensionality input. In other words, a representation that encodes high-dimensionality input (for instance, an image) into a lower-dimensional space. Critically, embeddings are trained to encode the relationships and semantic nuances that might exist in the original input space. For instance, two pictures of the same face transformed with an embedding tuned for facial recognition would yield two vectors that are similar to each other (i.e., close to each other for a given metric, usually the cosine distance). As such, the process of embedding not only condenses high-dimensional information into a more manageable form but also captures latent associations that might otherwise remain obscured [Bengio (2009)].
Unsurprisingly, the training of compact yet semantically-rich embeddings has been a very active research topic over the last two decades, yielding exceptional results in machine learning, where the (otherwise high) dimensionality of real-world percepts might turn common machine learning tasks like classification or prediction intractable.
While research on embeddings initially focused on data that would intuitively lend itself well to mathematical transformations (for instance, reducing the dimensionality of an image, represented as an array of pixel intensities, or processing sound), it has since then been discovered that many constructs – physical or not – can also be embedded in a low-dimension numerical space, while preserving many of their semantics. One of the landmark achievements in that regard is the work published in 2013 by Mikolov et al. – themselves building on previous work spanning another decade. They showed that embeddings can be computed for words, also encoding some of their semantic meaning [Mikolov et al. (2013)], with the famous example of embedding(‘king’) - embedding(‘man’) + embedding(‘woman’) ≈ embedding(‘queen’). Effectively, a conceptual equivalence of terms, involving semantics related to gender and social role could be transformed into simple mathematical additions and subtractions.
This outcome ushered in a decade of intense research on text representation, ultimately resulting in the current Large Language Models (LLM) like GPT or Llama2. Importantly for this work, these very large pre-trained networks can also be used to compute text-level embeddings, representing a short text as a numerical vector [Reimers & Gurevych (2019), Muennighoff (2022)]. The resulting embeddings can be then used to measure text-relatedness for instance, as in the BEIR benchmark [Thakur et al. (2021)].
1.3 Key insight
Combining the above concepts of social situations and text embeddings, we introduce in this paper the idea of social embeddings. A social embedding is a compact, real-valued numerical representation (a vector) of a social situation, as experienced by an agent immersed in that social environment. Following the general idea of embeddings, social embeddings are designed to encode the semantics of the social situation currently experienced by the agent, facilitating the interpretation of the situation. For instance, it could make it straightforward to compare two social situation by simply measuring how similar the two corresponding embeddings are.
The key insight to construct these embeddings is to exploit the social knowledge already encoded in the latent space of large language models. We do so by automatically generating a textual description of the social environment of the robot (using regular perception routines), and by transforming this description into a text embedding via a large language model. By doing so, we effectively construct an embedding, i.e., a projection, of the social space into a machine-friendly numerical space.
This article is a first investigation of this idea. In the following sections, we start the exploration of the design space of social embeddings by presenting a simple algorithm to generate scene descriptions and derive embeddings; we analyse and discuss several key characteristics and parameters of the derived embeddings – like their application as a quantitative social distance; and we discuss several promising directions for follow-up research.
1.4 Motivating example
Figure 1 shows three schematic social situations, described from the egocentric perspective of the yellow character. In Figure 1a, the character is engaged with one person, and another group of two people are chatting; in Figure 1b, no one is interacting directly with the character (but someone is walking towards her/him), and three people seem to be chatting together on the side; similarly in Figure 1c, someone is walking towards the character, with another group of two people chatting together.
Figure 1. Three simple social situations, described from the perspective of the yellow agent (arrows indicate direction and magnitude of motion).
Figure 1d shows the open-source social situation editor used to create the test scenarios.
From the perspective of the yellow person the social situations depicted in 1b and 1c are more similar: in both cases, the person is not actively engaged in an interaction yet, but someone seems about to do so. However, a cursory look at the scene could lead to think that 1a and 1b are closer to each other, since they both involve more people than 1c, and most people seem located at the left of the main character.
Accordingly, social embeddings are to be designed so that the situations from Figures 1b and 1c are closer in the embedding space than situations 1a and 1b: by doing so, a robot could correctly compute similarities between social scenes, and e.g. recognise a situation as similar to one it would have previously experienced.
2. Embedding construction
Our approach to embedding construction consists in generating a textual description of the social environment of a situated agent (e.g. a social robot) and computing a text embedding of that description, relying on a pre-trained large language model.
This process entails the following steps: (1) building a model of the social environment of the agent; (2) synthetising a textual description of that environment; (3) computing the text embedding.
2.1 Social signals and situation perception
Step 1 relies on the availability of social perception capabilities, both hardware (for sensing) and software (e.g. recognition and tracking of people). For this initial exploration of social embeddings, we focus here on perception capabilities commonly found on social robots. Namely, we build social embedding from the following social signals:
relative position of agents;
tracking of agents (the same person is always refered with the same identifier across frames);
velocity and direction of movement;
gaze direction (field of view and field of attention);
(optionally) whether or not agents are currently talking.
These signals are readily available using perception frameworks like the open-source ROS4HRI [Mohamed & Lemaignan (2021)].
Importantly, we adopt a situated and ego-centric perspective: the scene is described as seen by an agent, itself embedded in the social situation. This choice is driven by practical considerations: we are designing social embeddings to be a useful tool for a social robot to analyse and recognise the current situation it is embedded in; as such, the robot is itself part of the social situation, and must describe the world from its own perspective. The three scenarios pictured in Figure 1 are 2D top-down representations of such situated and ego-centric perspectives on three social situations.
To evaluate our approach, in this work we also use a custom-made social situation simulator (Figure 1d) to quickly generate synthetic social data as described in Section 3.1.
2.2 Scene description
Step 2 consists in converting the perceived social situation into a textual description. Many approaches are possible. In this work we choose a relatively direct translation of the social signals into a concatenated list of descriptors separated by semi-colons. Examples of descriptors are: “X is looking at Y”, if Y lies within the field of attention of X; or “X is walking towards me”, if the dot product of the motion and relative position vectors of X is close to −1.
Specifically, we compute the following descriptors:
X is close to Y (< 1.5m)
X is not far from Y (< 4m and facing each other)
X is walking towards me
X is walking away from me
X is passing by
X is looking at Y
X and Y are looking at each other
X is talking (if speech detection is available)
Algorithm 1 details the computation of these descriptors. The name placeholders in the resulting descriptors are replaced by random names (yet keeping the same name for a given agent). By applying this algorithm, the situation pictured in Figure 1a leads to the following description:
“Emily is talking; Emily is close to me; Emily and I are looking at each other; Will is talking; Bob and Will are looking at each other; Bob is not far from me; Bob is looking at me”
Likewise, Figure 1b is described as:
“Bob is talking; Bob is looking at Emily; Bob is not far from me; Bob is looking at me; Will is walking towards me; Will and I are looking at each other; Will is not far from me”
Finally, Figure 1c is described as:
“Violet is walking towards me; Violet is not far from me; Violet is looking at me; Emily is talking; Emily and Will are looking at each other; Will and I are looking at each other; Will is not far from me”.
Algorithm 1: Generation of social situation descriptions
input : (Xi, , Θi, isTalkingi) with i ∈ Agents
input : ego, the index of the agent whose perspective we adopt
output : a set of textual descriptors
CLOSE ← 1.5m; MEDIUM ← 3m; FAR ← 4m
δi,j ← ||Xi − Xj||
FoV ← 100°; FoA ← 30°
descriptors = ∅
/* j sees i? */
func isVisibleBy i, j
return |arctan(Xi − Xj) − Θj | < FoV/2
/* i looks at j and j is the closest to i? */
func isLookingAt i, j
Λ = {k, |arctan(Xk − Xi) − Θi| < FoA/2}
return j == argmink(δi,k, k ∈ Λ)
func isFacing i, j
return isVisibleBy(i, j) ∧ isVisibleBy(j, i)
func relativeDistance i, j
if δi,j < CLOSE then return “{i} is close to {j}”
if (CLOSE < δi,j < FAR) ∧ isFacing(i, j) then
return “{i} is not far from {j}”
func relativeMotion i, ego
if CLOSE < δi,ego < FAR then
angle ←
switch angle do
case angle < −0.7 do
return “{i} is walking towards me”
case angle > 0.7 do
return “{i} is walking away from me”
case |angle| < 0.3 do
return “{i} is passing by”
for Agent i ≠ ego do
if isVisibleBy(i, ego) then
descriptors ← relativeDistance(i, ego) descriptors ← relativeMotion(i, ego) if isTalkingi then
descriptors ← “{i}istalking′′
for Agent j ≠ i do
if isVisibleBy(j, i) ∧ isVisibleBy(j, ego) then
if isLookingAt(i, j) ∧ δi,j < MEDIUM then
if isLookingAt(j, i) then
descriptors ← “{i}and{j}arelookingateachother′′else
descriptors ← “{j}islookingat{i}′′
return descriptors
As the aim of the embedding is to compactly represent the semantic of the social situation, the pragmatics of the description, and in particular, the order of the descriptors in the final description, should have little influence on the resulting vector. Likewise, the randomly drawn names should not impact the semantic proximity of two similar social situations within this context. We evaluate these properties in Section 3.
2.3 Embedding generation
The last step consists in computing a text embedding of the description. We first add some task context by wrapping the description in the following text: “This is the description of a social setting with a few people: [generated descriptions]. We want to generate a good description of the situation.”, before transforming the full text with a text embedder.
We use langchain1 to abstract the call to the text embedder. In the following sections we primarily use Meta’s Llama2 13B text embedder, while comparing it to OpenAI’s ada-002 in Section 3.5.1. Moreover, in Section 3.5.2, we discuss in greater details the use and impact of the added task context. Our results show that, depending on the text embedding model, task context do or do not have significant impact on the representation of social situations.
3 Evaluation on synthetic data
To be useful, social embeddings must exhibit at least the following key properties:
pragmatics invariance: if we want the represent the semantics of a social scene, the pragmatics of the description (exact wording, order in which we describe the scene, etc.) should only have a minimal influence;
semantic similarity: two social situations that are similar should result in embeddings that are close to each others in the embedding space;
continuity: small changes in the social situation should result in correspondingly small variations in the embedding space.
As our methodology does not depend whether the data is synthetic or acquired in real-world conditions, we perform first an in-depth evaluation on simulated social data, and we demonstrate how the same methodology can be applied to real-world data in Section 4.
3.1 Generation of synthetic social data
We have developed an open-source social interactions ‘editor’2, (Figure 1d) that we use to design small interaction situations. The tool allows the manual creation of short animations of 2D characters moving around, forming groups, and talking. Scenarios are generated by creating keyframes at various points in the timeline, with the tool interpolating the position, orientation and velocity of the characters between frames. Characters can also be marked pairwise as ‘interacting’ with each other (this particular information is not used to generate embeddings descriptions; it could however be used in future work to e.g. train a engagement detector).
The editor saves scenarios as JSON files; it can also generate snapshots and videoclips of the interactions, as viewed from any character’s perspective (see examples in Figure 1). These video-clips can then be used for e.g. online studies.
The output JSON files are then post-processed by a set of helper scripts (also open-source) to perform the following steps:
1. generate, for each agent in the scene, and at every time point, situation descriptions (using the algorithm presented in Section 2.2).
The descriptions are normalised such that a similar social situation, viewed by a different agent, will have the exact same description (e.g., same descriptor ordering); the names of agents are represented as slots like {A}, {B},... so that new identical descriptions with different names can later be generated.
2. compute the embeddings, optionally randomising the order of the descriptions, and/or randomising names to create variations of the same situations.
The end result is a CSV file containing a list of descriptions, a group ID associating together the variations of the same description (if they are generated), and the embedding vector (the dimension of the embedding vector depends on the text embedding model; for instance, d = 5120 for Meta’s Llama2 model).
In the following sections we use two short scenarios created with this tool to extract a set of social situations. The two scenarios3 last a total of 48 seconds and involve five people in a variety of changing group situations: people move from one group to the other, sometimes interact, sometimes not. Figures 1a to 1c are snapshots from this dataset.
These scenarios have been created ex-nihilo to generate a broad range of different social situations – they do not attempt to faithfully represent actual human social interactions; their purpose is instead to represent prototypical interactions that should test the representational capabilities of the social embeddings presented in this work.
The scenarios are sampled at 2Hz, and each scene is described from the perspective of each of the five persons, leading to 48 × 2 × 5 = 480 unique situations. Once duplicate descriptions are removed, we obtain 123 unique scene descriptions.
Note that the data analysis scripts used below are attached to the submission as a Python notebook.
3.2 Invariance to pragmatics
We first investigate whether our embeddings actually encode the semantics of the social situation, and not merely the lexical structure of the description.
To this end, and for each situation description, we create one random variant by randomising the order of the descriptors. We then compute the cosine distance between the original description embedding and its variant.
We compare the resulting distribution of distances to the distances of the original description to every other descriptions in the scenario. Figure 2 shows the resulting density distributions, with Table 1 reporting additional descriptive statistics.
Figure 2. Density distribution of the distances in embedding space (cosine distance, left) and textual space (Levenshtein distance, right).
The shaded area correspond to measured distances between each social scene descriptions and one of their variants.
Table 1. Distances between representations, embedding space vs textual space.
The cosine distance is used to compute the distance between embeddings; the Levenshtein distance is used to compute similarity between textual descriptions.
| n | Embedding space | Textual space |
|---|
| | mean | stddev | mean | stddev |
|---|
| → to variant | 90 | 0.0084 | 0.0089 | 46.1 | 22.66 |
| → to other descriptions | 4005 | 0.022 | 0.012 | 59.21 | 23.63 |
For comparison, we perform a similar computation directly on the textual descriptions (hereafter the textual space), using the Levenshtein distance as a string similarity measure.
We see on Figure 2, left, that random variations of descriptions end up very close to their original, canonical descriptions (distances close to zero). In contrast, using string similarity as metric (Figure 2, right), the distances between pairs (canonical description, variants) are spread over, where two descriptions of the same scene are not typically close to each other.
This first result indicate that our embeddings do encode at least some semantics of the scene description, independently of the exact pragmatics of the description.
3.3 Social similarity
The main aim of social embeddings is to provide a reliable metric of social situation similarity. While we intuitively understand it, the ‘similarity’ of two social situations is actually not easily quantified, and to the best of our knowledge, no tool exists to automatically measure how similar two social situations are, beyond manual annotations by experts.
Because it would be difficult for people to rate on an absolute scale whether two situations are similar or not, we approached the problem through pairwise comparisons (akin to A/B testing): ‘are two randomly-chosen situations more similar to each other, than two other randomly-chosen situations?’
To assess whether our embeddings correctly encode the similarity of social scene, we therefore designed an online study where participants were shown a short 2-seconds video clip of a randomly-picked social situation animation (in same visual style as Figure 1). This first situation was referred to as the ‘reference situation’. Two additional clips (video A and video B) where displayed below, and the participants were given the following options: select the clip (A or B) which resembled the most to the reference situation; indicate that both clips where very similar to the reference situations; or that both clips were very different from the reference situation (see Figure 3).
Figure 3. Screenshot of the online study setup.
We collected 893 such pairwise comparisons. Out of these, 92 were double-coded by at least two independent experts, with 74 achieving agreement (inter-rater agreement measured with Kripendorff’s alpha: 0.68). In the following, we only keep these n = 74 comparisons.
We then ran the same comparisons using the cosine distance in the embedding space. We implemented a simple threshold-based predictor over the four classes (A closer to reference; B closer to reference; A and B both similar to reference; A and B both different to reference). We used the 20th and 80th quartile of the full distribution of distances as thresholds for ‘similar’ and ‘different’, respectively.
Using Llama2, the 4-classes accuracy of the predictor is measured at 60.8% (chance, computed by setting random distances between embeddings, is at 33.3%). Figure 4 shows the full confusion matrix of this heuristic-based classifier. Note that the B closer than A row has zero true label, as we swapped videos A and B (and their votes) during the data analysis to always have video A reported as more similar to the reference than video B in order to simplify the interpretation.
Figure 4. Confusion matrix between the human baseline of similarity of social situations vs similarity predicted using distances in embedding space.
Both absolute number of situations, and number of situation normalized with respect to the true labels, i.e. y-axis, are reported; see text for ‘B closer than A’ zero true labels.
These initial results show that, while not perfect, our social embeddings, derived from simple scene descriptions, already perform significantly better than chance when trying to quantitatively assess the similarity of social situations.
While this first study has important limitations (simplified model of social interactions, no precise measurement of how similar or dissimilar situations are, etc.), it does support the general validity of the approach.
3.4 Continuity
The third desirable property of our embedding is continuity: small changes in the social environment should lead to similarly small changes in the embedding space. Formally, if we denotes S the set of social situations, the embedding process f : S → [0; 1]n is continuous iff limx→c f(x) = f(c) for any x and c in S.
This formulation is theoretical, as it assumes a metric of social similarity between x and c that is not available (at least not in a quantitative way). Regardless, our embeddings are by construction not continuous, as the textual descriptions we generate are built from a discrete set of descriptors, and the addition or removal of one descriptor will cause in practice a discontinuity in the embedding space.
Future work might investigate into how this could be mitigated, by e.g. redefining the concept of continuity for social situations. Nevertheless, looking at the sequence of social embeddings during a continuous interaction provides interesting insights. For instance, Figure 5 represents the distance between consecutive embeddings of the social situation experienced by one of the agents in one of our synthetic test scenario. In effect, this graph represents the rate of social change experienced by this agent over the 39 seconds of the scenario.
Figure 5. Distances between two consecutive social situations viewed by one agent in the test scenario 2.
Embeddings are sampled at 2Hz, and distances smoothed over a 2 seconds rolling window.
For example, in this instance, the agent was in a stable social situation between seconds 20 and 28, while it experienced rapid and significant change of social environment around second 10.
3.5 Other hyper-parameters
The design space of social embeddings, as we define them, is large, and many parameters might influence the final quality of the embed representation. In this section, we examine some of those hyper-parameters, namely: the model used for text embedding; the role of task context; the generation of ego-centric versus allo-centric descriptions; and the use of human names versus abstract identifiers.
3.5.1 Comparison between Large Language Models. Different large language models have different text embedding space, which might be more or less favourable to encoding the semantic of social situations. We compare the Meta Llama2 13B model used so far to OpenAI’s text-embedding-ada-002 model.
At the time of writing, Llama2 is the leading open-source large language model, and while a 70B version of this model is available (and might yield even better results), we settled for the 13B with 4 bits quantization applied, so that we could generate the embeddings ourselves, on consumer-grade, CPU-only hardware (on an Intel i7 11th, it took on average 17 seconds per embedding). Indeed, the ability to generate offline embeddings on hardware found on off-the-self robots is critical to ensure future privacy-concious applications.
OpenAI ada-002 performs text embedding in a d = 1536 dimensional space, versus d = 5120 for Llama2. Performing the same computations as in Section 3.3 with the ada-002 embeddings yields an overall classification accuracy of 56.8%, slightly below the Llama2 results (60.8%). While the results are close to each other, Llama2 seems to perform slightly better on this task, which might be partially explained by the higher dimensionality of the embeddings.
3.5.2 Influence of the description context. As explained in Section 2.3, we computed our social embeddings by inserting the generated scene description in a short task context (“This is the description of a social setting with a few people: [descriptors]. We want to generate a good description of the situation.”).
The use of such task context is suggested by research on LLM prompting (for instance [Kojima et al. (2022)]) showing that simple prompt modifications, like adding “Let’s think step by step” can very significantly improve the reasoning capabilities of many LLMs.
To better understand the impact of context-setting on the embedding of social situations, we computed social embeddings with four different contexts:
no context (only the descriptors);
short context (see text above);
short context with “Let’s think step by step” appended at the end;
longer context that include one example of inference, and example of possible follow-up questions (for instance, after the scene description, the context include the text “Am I engaged in an interaction right now? if so, with whom?”).
Running the classification task with the OpenAI model, the accuracy results are respectively 58.1%, 56.8%, 56.8% and 56.8%. It generally appears context setting has little impact on the quality of the representation, and in particular, adding “Let’s think step by step” does not improve classification results at the level of the text embedding.
In the case of Llama2 13B, the classification results were 31.1%, 60.8% and 56.8% (long context omitted due to prohibitively long computation costs). The major drop of accuracy when embedding descriptions in Llama2 without context is unexpected, and would require further investigation in the internals of the Llama2 model.
3.5.3 Egocentric vs allocentric descriptions. In Section 2.2, we generate a description of the social environment from the egocentric perspective of the agent (first-person perspective). For instance, we generate descriptors like Mary is walking towards me. This choice is driven by the intuition that social experiences that would be part on the training corpora of large language models would predominantly be narrated from a first person point of view.
The alternative is to describe the scenes from a ‘god-like’ allocentric perspective, where we do not adopt a particular agent’s point of view. The first person pronoun is replaced by an arbitrary name, and the previous example would for instance become Mary is walking towards John.
Running the same classification task using allo-centric description with the OpenAI ada-002 text embedder yields an overall accuracy of 55.4%, with 63.0% for the Llama2 model.
While the difference is small, it appears that egocentric descriptions yield slightly better social representations.
3.5.4 Use of human names vs abstract identifiers. As presented in Section 2.2, agents present in a given scene are assigned arbitrary human names – only ensuring that the same agent receive the same name in subsequent frames of the same scenario.
The intuition for assigning human names rather than random identifiers (like A, B, C...) is that is might help steering the embeddings towards representing social interactions, by implicitly signaling that the agents are humans.
Re-running the classification task with letter identifiers instead of names however actually slightly improves the classification results (62.2% vs 60.8%). This result might however be an artifact of our classification task. Indeed, participants were asked to compare social situations without any specific information about the identities of the characters drawn on the screen: the characters appearing in the reference video clip might or might not be the same as the ones appearing in the two other videos. However, by using human names, the embed descriptions did encode the identities of the characters across videos. On the contrary, we can hypothesise that, when using letters instead of names, we implicitly signal that the agents should be treated as anonymous entities, resulting in embedding representation more similar to what the human annotators would have experienced.
4 Validation on real-world data
Because the Algorithm 1 is designed to take as input social signals that are commonly found on robots, there is no conceptual difficulty in running the same algorithm on actual hardware. As a proof-of-concept, we recorded a short sequence (1min 42sec) of four people interacting in front of a PAL Robotics ARI robot [Cooper et al. (2020)]. The robot is running out-of-the-box a ROS4HRI-compatible pipeline [Mohamed & Lemaignan (2021)] that exposes the location, gaze direction and velocities of the people around the robot. The robot does not support speaker extraction and identification, so we could not uniquely identify who was speaking. Instead we relied on the available voice activity detector to simply record that ‘someone’ was speaking.
Scene descriptions were generated at 3Hz (maximum permissible rate to generate embeddings in real-time from the OpenAI API endpoint), using the same algorithm as Algorithm 1 (except for the {agent} is talking replaced by Someone is talking).
Using these computed embeddings, and similar to Figure 5, Figure 6 represents the rate of social change experienced by the robot. As illustration, snapshots of the robot’s camera have been added before and after two peaks of social change.
Figure 6. Cosine distances between two consecutive embeddings sampled at 3Hz, and averaged on a 2sec rolling window.
5 Discussion
5.1 Generation of descriptions and ethical considerations
The process to generate textual descriptions of social situations is critical to the computation of our social embeddings. In this initial investigation, we took a simple approach consisting in concatenating descriptors directly computed from commonly available robot’s sensing.
We have not yet investigated alternative approaches – or whether all the descriptors that we use are in fact required. Future work should investigate this design choice, investigating possible richer descriptions (e.g. individuals’ expressions, group formation recognition, etc.), and running ablation studies to determine if particular descriptors are redundant.
The way descriptions are generated also has ethical implications. Because we rely on the use of LLMs to generate embeddings, we might introduce undesirable biases in the obtained representations. Indeed, as reported by OpenAI4 for instance, “the models encode social biases, e.g. via stereotypes or negative sentiment towards certain groups.” Thus, careful attention must be paid when designing the input descriptors to generate the embeddings. In this work we include one potentially biased descriptor: the name identifiers. While they are randomly picked from a gender-balanced list, they are all English names, and do not balance diverse cultural backgrounds. Future work might look into additional sensitive descriptors, like age, gender, task context, that could unwillingly lead to encoding stereotypes. We therefore encourage researchers to evaluate the adequateness and potential impacts of the descriptors introduced in the social embeddings construction based on the context of use.
5.2 Context awareness
The correct interpretation of a social situation is strongly context-based: for example, a person starting to laugh at a robot designed to help visitors in a library likely indicates some inappropriate behaviour, while the same person laughing while the robot is telling a joke would on the contrary indicate a successful interaction.
While we have not explored the representation of context in this first presentation of social embeddings, our approach lends itself well to this kind of extension: assuming one can create a description of the current context of interaction (‘We are in the reception hall of an hospital and my role is to welcome visitors’; or ‘We are in an office, and people do not want to be disturbed without reason’; etc.), the context textual description could simply be prepended to the scene description, to embed as well contextual information. This represents an important area of future research.
5.3 Latent semantics
In this initial investigation, we have not attempted to uncover the latent semantics encoded in the embeddings. For instance, we can expect that social situations like ‘two persons chatting together’; or ‘a group of three people walking together’; or ‘one single person walking towards the robot’; etc. are all semantically distinct, and, consequently, would belong to distinct regions in the embedding space. Identifying such clusters to broadly characterize the semantic topology of the embedding space would enable not only to measure how similar two social situations are, but also characterize key features of the current situations. This idea is related to e.g. the recent investigation by Sun and Nelson [Sun & Nelson (2023)] on latent semantics of sentence embeddings.
5.4 Fine-tuning
The text encoders used to generate the embeddings were not fine-tuned for the specific task presented in the paper. However, contrastive-learning-based fine-tuning [Hadsell et al. (2006)] has shown potential for instruction-based text embeddings [Canal et al. (2022)]. Exploring contrastive-learning-based fine-tuning techniques is expected to improve the results reported here.
A major challenge, in addition to the fine-tuning itself, is however the collection of a dataset expressing differences and similarities between social situations. The collection of this type of data is a non-trivial problem. Intuitively, it will require participants labelling social situation as more or less similar, based on their perception and understanding.
6 Conclusion: Social Embeddings as a Social Metric?
We have introduced in this paper the concept of social embeddings: compact vector representations of arbitrary social situations, computed by first generating a textual description of the social environment, and then leveraging pre-trained large language models to embed the resulting description into a low-dimensional numerical space. Moreover, we have shown evidence that the resulting embeddings encode some of the social semantics of the original situations, and their topology partially reflects the qualitative similarity observed by humans between given social situations.
By providing a algorithmic bridge between qualitative appraisal and quantitative measurement of similarity, we can naturally question whether social embeddings could in fact be a metric over the ‘social space’. Formally, the definition of a metric d over a space implies the following four properties:
1. distance to itself is zero: d(x, x) = 0
2. positivity: If x ≠ y, then d(x, y) > 0
3. symmetry: d(x, y) = d(y, x)
4. triangle inequality: d(x, z) ≤ d(x, y) + d(y, z)
While properties 1–3 are trivially derived from the computation of the cosine distance between embeddings, the triangle inequality property is difficult to formally prove for text embeddings. An empirical evaluation with 1000 randomly-picked triplets of scenes (x, y, z) shows that in 96.1% of the cases, the inequality d(x, z) ≤ d(x, y) + d(y, z) is verified. As such, while they are not formally a metric of social situations, social embeddings appears to behave as a good pseudo-distance for the social space.
While we have made it abundantly clear that much work remains to be done to refine and fully characterize social embeddings, equipping robots with such a metric over the social space would open multiple new venues for research and applications. In order to conclude this initial investigation of the potential of social embeddings, we list hereafter some of these mid-term/long-term ideas, hoping to initiate fruitful new lines of research on data-driven social psychology and HRI:
by clustering a diverse set of social situation embeddings, it should be possible to isolate key prototypical social situations, to which rules, norms, behaviours could be associated. A robot could then recognise when it encounter such situations, and also estimate the dynamics of the current situation, i.e. towards what other situation it is headed;
this ability opens the door to social prediction, letting the robot anticipate future social state (and adjust accordingly its behaviour to avoid undesirable situation, or facilitate desirable ones);
as vector representations, social encodings lend themselves well to many downstream machine learning tasks, for instance socially-appropriate behaviour generation, or situated interactive machine learning;
because social embeddings also lend themselves to encoding task and social context, they open new possibilities for context-aware representation and behaviour selection;
when used to represent changes in the social environment (as in Figure 5 and Figure 6), social embeddings could also become a powerful tool to automatically recognise interaction issues: discontinuities in the embedding space would represent sudden changes in the social environment that might be unexpected in the current context of the robot.
