Keyboard Shortcuts?

×
  • Next step
  • Previous step
  • Skip this slide
  • Previous slide
  • mShow slide thumbnails
  • nShow notes
  • hShow handout latex source
  • NShow talk notes latex source

Click here and press the right key for the next slide (or swipe left)

also ...

Press the left key to go backwards (or swipe right)

Press n to toggle whether notes are shown (or add '?notes' to the url before the #)

Press m or double tap to slide thumbnails (menu)

Press ? at any time to show the keyboard shortcuts

Natural
Inter-
actions

Theoretical & Motor Aspects

Inspiration for my talk is comes from the workshop aims and the manifesto for the jointactionHRI project, which has a strong conceptual component ...

‘how is joint action actually defined?’

--- workshop aims

Let me start by considering how philosophers have approached defining it
So how have philosophers’ attempted to answer these questions?
The key device is the use of examples and contrast cases. To illustrate, suppose a scientist and a philosopher walk into the park.
They see two sisters cycling to school together.
This is a paradigm example of joint action.
The scientist immediately asks, How are their actions coordinated?
The philosopher objects: There’s more to joint action than mere coordination.
We know this because in another park there are two strangers who happen to be cycling the same route side-by-side.
And although the strangers are not engaged in joint action---their’s are a parallel but merely individual actions---, their actions do need to be exquisitely coordinated if they are not to collide.

?

This and other contrasts motivate asking, What distinguishes genuine joint action from parallel but merely individual action?

Each sister intends that they, the sisters, cycle together.

Bratman offers a counterexample to something related to the Simple View \citep[see][]{Bratman:1992mi,bratman:2014_book}. Suppose that you and I each intend that we, you and I, go to New York together. But your plan is to point a gun at me and bundle me into the boot (or trunk) of your car. Then you intend that we go to New York together, but in a way that doesn't depend on my intentions. As you see things, I'm going to New York with you whether I like it or not. Does this provide the basis for an objection to the Simple View?

The Simple View

Two or more agents perform an intentional joint action
exactly when there is an act-type, φ, such that
each agent intends that
they, these agents, φ together
and their intentions are appropriately related to their actions.

Bratman’s ‘mafia case’

Michael Bratman offers a counterexample to something related to the Simple View. Suppose that you and I each intend that we, you and I, go to New York together. But your plan is to point a gun at me and bundle me into the boot (or trunk) of your car. Then you intend that we go to New York together, but in a way that doesn't depend on my intentions. As you see things, I'm going to New York with you whether I like it or not. This doesn't seem like the basis for shared agency. After all, your plan involves me being abducted.
But it is still a case in which we each intend that we go to New York together and we do. So, apparently, the conditions of the Simple View are met (or almost met) and yet there is no shared agency.

1. I intend that we, you and I, go to NYC together.

2. You intend that we, you and I, go to NYC together.

3. You intend that we, you and I, go to NYC together by way of you forcing me into the back of my car.

***BRATMAN's DIAGNOSIS - have to intend to do it by way of the other’s intentions. This is what is wrong in mafia case. Shared agency means connecting with each other as agents, not merely as bodies
Bratman’s brilliant idea for avoiding this sort of problem is to suggest that we don’t just each intend the action but rather we each intend to act by way of the other's intentions.
We can put this by saying that our intentions must interlock: mine specify yours and yours mind.
Now this appeal to interlocking intentions enables Bratman to avoid counterexamples like the Tarantino walkers; if I intend that we walk by way of your intention that we walk, I suppose can't rationally also point a gun at you and coerce you to walk.

‘each agent does not just intend that the group perform the […] joint action.

‘Rather, each agent intends as well that the group perform this joint action in accordance with subplans (of the intentions in favor of the joint action) that mesh’

(Bratman 1992: 332)

`each agent does not just intend that the group perform the […] joint action. Rather, each agent intends as well that the group perform this joint action in accordance with subplans (of the intentions in favor of the joint action) that mesh' \citep[p.\ 332]{Bratman:1992mi}.
Our plans are \emph{interconnected} just if facts about your plans feature in mine and conversely.
‘shared intentional [i.e.\ collective] agency consists, at bottom, in interconnected planning agency of the participants’ \citep{Bratman:2011fk}.
In making this idea more precise, Bratman proposes sufficient conditions for us to have a shared intention that we J ... ... the idea is then that an intentional joint action is an action that is appropriately related to a shared intention.

We have a shared intention that we J if

‘1. (a) I intend that we J and (b) you intend that we J

‘2. I intend that we J in accordance with and because of la, lb, and meshing subplans of la and lb; you intend [likewise] …

‘3. 1 and 2 are common knowledge between us’

(Bratman 1993: View 4)

\begin{minipage}{\columnwidth}
\emph{Bratman’s claim}. For you and I to have a collective/shared intention that we J it is sufficient that:
\begin{enumerate}[label=({\arabic*}),itemsep=0pt,topsep=0pt]
\item `(a) I intend that we J and (b) you intend that we J;
\item `I intend that we J in accordance with and because of la, lb, and meshing subplans of la and lb; you intend that we J in accordance with and because of la, lb, and meshing subplans of la and lb;
\item `1 and 2 are common knowledge between us' \citep[View 4]{Bratman:1993je}
\end{enumerate}
\end{minipage}
One feature of Bratman’s account is that you can be engaging in a joint action even if you merely see the other’s intentions as opportunities to exploit and constraints to work around. As long as you know about, and go along with, the other’s intentions, there is joint action.
Many, like Gilbert find this unacceptable

1. ‘The notion of a we-intention [shared intention]
... implies the notion of cooperation’

\citep[p.~95]{Searle:1990em}

Searle (1990, p. 95)

[explain ‘we-intention’]
Why is this relevant. Because ...

2. Meeting Bratman’s proposed sufficient conditions for shared intention does not imply that your actions will be cooperative.

Bratman says this explicitly.

Therefore:

3. Bratman’s conditions are not in fact sufficient.

There are just two [HERE: one] problems with this argument
First problem: how do we know this is true?
Gilbert has a different, incompatible

Joint commitment is a ‘precondition of the correct ascription’ of acting together, collective belief, shared intention, and more.’

\citep[p.~9]{gilbert:2014_book}

Gilbert (2013, p. 9)

For us to have

a shared intention that we φ

is for us to be jointly committed

to emulate a single body

which

intends to φ.

This is a technical notion; I won’t tell you what it is because I think Gilbert uses the term for two fundamentally incompatible ideas and it takes a while to disentangle all this ...
Brat-manSearleGil-bertBlom-bergPach-erie...

Are all joint actions cooperative?

✗?✓?...

Are commitments necessary for joint action?

...

Does acting jointly entail being aware of doing so?

...
So here are three questions about joint action that it seems we cannot answer if what anchors discussion is merely examples and intuitions.
(Of course someone might reply that careful use of intuitions would be enable us to answer some or all of these questions. There is a hint that this is Bratman’s own view in remarks on tricky cases involving coercion such as, ‘Though what we are doing seems ill-described as a cooperative activity, it may be plausible to describe it as a shared intentional activity’ \citep[p.~38]{bratman:2014_book}.)
Why does this matter? Suppose we are trying to construct a theory of joint action.
[1] Any such theory will, if it is adequate, provide answers to these questions. For example, Bratman’s theory implies that non-coercion, cooperation and commitment are not required for joint action [or what he calls ‘shared intentional activity’] whereas awareness is.
[2] How can we tell whether these answers are correct? We might try appealing to features of the theory itself, such as its internal coherence or the absence of a competing theory.
But there are competing, internally coherent, theories which provide different answers.
For example, as we will see, the Simple View Revised implies that cooperation but not non-coercion is required for joint action, and it imposes a complex condition on awareness.
[2b] Alternatively we might try appealing to metatheoretical considerations like Bratman’s ‘Continuity Thesis’. Again, the obstacle to doing this is that we can construct accounts which have similar metatheoretical properties but give different answer to the four questions.
[We will consider this point more deeply when we return to Bratman vs Gilbert.]
[3] So it seems that to determine whether a theory is correct we need to know whether it correctly answers the four questions about non-coercion, awareness, cooperation and commitment. And to know this, it seems that we cannot rely on internal features of the theory, nor on metatheoretical considerations
[4] Apparently, then, our pre-theoretical fix on the things to be explained should enable us to be able to answer these questions.
Unless we can answer these three questions, it seems to me that we do not have a sufficient grip on joint action--that is, on the thing to be explained.
So in attempting to understand joint action we face a theoretical problem right at the outset ...
On the one hand, the philosophers are right that you can’t get joint action from coordination alone.

1

You can’t get joint action from coordination alone.

On the other hand, we run into two problems. orthodox philosophical accounts are (i) impossible to decide among in any principled way and (ii) involve concepts which are not readily operationalised.
\textbf{One problem} is that there are at least as many different accounts of shared intention as there are philosophers, and no two philosophers agree. This is perhaps because philosophical accounts of joint action are based entirely on informal observation, guesswork and pure reason.
\textbf{Another problem} is that the main philosophical accounts are not readily operationalisable. [This is perhaps because they combine identifying the phenomena to be explained with identifying mechanisms which underpin coordination. This does not appear to leave room for \textbf{discoveries} about mechanisms.]
 

2

Orthodox philosophical accounts are
(i) impossible to decide among and
(ii) inoperationalisable.

So \textbf{how \emph{could} philosophers serve HRI research on joint action?} Let me first offer a diagnosis of what has gone wrong ...
Diagnosis:
It seems to me that, whatever we are characterising, \textbf{it is important to distinguish characterising the thing to be explained from constructing a candidate explanation of it}.

Thing to Be Explained

Candidate Explanation

dimming of a star

conjecture about a planet

object-tracking abilities in infants

conjecture about innate knowledge

joint action

conjecture about shared intention/commitment/team reasoning/motor representation/...

Key issue: how to characterise the abilities to be explained by the theory?
So, as I was asking a moment ago, \textbf{how \emph{could} philosophers serve HRI research on joint action?}

a different approach

How to characterise joint action?

Step 1: identify features associated with things commonly taken to be paradigm joint actions in nonmechanistic terms, e.g.

- collective goals

- coordination

- cooperation

- contralateral commitments

- experience

- ...

Step 2: ... which generate how questions

Step 3: ... leading to discoveries about mechanisms

Important for joint action for HRI : mechanisms are likely to differ between different kinds of agents anyway
This is really more of a circle than a two step proposal since whether the features identified in step 1 should be retained, and whether any structure emerges partly depends on how the ‘how questions’ eventually get answered.
But, you know, this talk is already too long so there’s no way I’m going to discuss Steps 3 onwards here.

common goal

A common goal is is a single goal to which the agents’ actions are directed.
[Can illustrate common vs collective goal with current political situation in which the actions of bitter rivals may have a common goal but no collective goal (unlike the political allies).]
This is still not enough. To see why, suppose that the strangers’ actions are no longer coordinated and they are walking different routes to their gate, but that each stranger is concerned that the Marseille flight should leave on time. As each sees it, the only thing she can do to this end is to walk to the gate. Her actions are therefore directed to the same goal as the other’s: to ensure the Marseille flight leaves on time.
So there is one goal to which each of their actions are directed; that is, a common goal. I suspect we still haven’t captured what talk of a ‘we mode’ aims at.

collective goal

‘The injections saved her life.’ [distributive vs collective]

Consider the statement, ‘The injections saved her life.’ This could be true in virtue of her receiving several injections on different occasions, each of which saved her life. In this case, the injections saving her life is just a matter of each injection individually saving her life; this is the distributive interpretation. But the statement is also true if she was given two injections on a single occasion where each injection was necessary but not sufficient to save her life. In this case the injections saving her life is not, or not just, a matter of each injection individually saving her life; this is the collective interpretation.
The difference between distributive and collective interpretations is clearly substantial, for on the distributive interpretation the statement can only be true if her life has been saved more than once, whereas the truth of the collective interpretation requires only one life-threatening situation.
Just as some injections can be collectively life-saving, so some actions can be collectively directed to a goal. For example, consider this sentence:

‘The goal of their actions is to find a new home.’

This can be interpreted distributively: each of their actions is separately directed to finding a new home. But it can also be interpreted collectively: finding a home is an outcome to which their actions are directed and this is not, or not just, a matter of each of their actions being individually directed to finding a home.
To say that an outcome is a \emph{collective goal} of some actions is just to say that it is an outcome to which the actions are directed and this is not, or not just, a matter of each action being individually directed to that outcome.
No mechanisms! Separate the thing to be explained from the thing which explains it.
Note that collective goals do not plausibly require any kind of intentions or commitments. After all, there is a sense in which some of the actions of swarming bees are directed to finding a nest and this is not, or not just, a matter of each bee’s actions being individually directed to finding a nest. So finding a nest is a collective goal of the bees’ actions.

Step 2: How could some agents’ actions have a collective goal?

Step 2: how could our actions have a collective goal?
a clue: when agents perform joint actions, motor representation concerning a partner’s action can occur.

a clue:

motor representations concerning
another’s actions occur in joint action

An important clue as to how we might be in the ‘we mode’ is provided by some experimental data concerning motor representation in joint action ...

Kourtis et al, 2012

Kourtis et al, 2012

Kourtis et al, 2012

Kourtis et al (2014, figure 1c)

I think we're a long way from having a large body of converging evidence for this conjecture, but there is some that points in this direction. One of the most relevant experiments is this one by \citet{kourtis:2014_attention}.
They contrasted a simple joint action involving two agents clinking glasses.
The CNV is a signal of motor preparation for action which is time-locked to action onset. In previous research, Kourtis et al show (i) that the CNV occurs when joint action partners act, suggesting that when acting together we represent others' actions motorically as well as our own \cite{kourtis:2012_predictive}; and (ii) (roughly) a stronger CNV occurs in relation to actions of others one is engaged in joint action than in relation to actions of others one is merely observing \cite{kourtis:2010_favoritism}.
Kourtis et al hypothesised that in actions like clinking glasses, A single outcome represented is motorically, which triggers planning-like processes concerning all the agents' actions. This leads to the prediction that the CNV in joint action will resemble that occurring in bimanual action more than that occuring in unimanual action.

Kourtis et al (2014, figure 4a)

... and this is exactly what they found.

Ramenzoni et al, 2014 figure 1

Importantly there is converging evidence for the involvement of motor representation concerning a partner’s action in joint action from studies which use behavioural measures ... Joint performance is better when observing joint actors; individual performance when observing individual actors.

Ramenzoni et al, 2014 figure 1

Ramenzoni et al, 2014 figure 1

della Gatta et al, ‘Drawn Together’ Cognition 2017

So once again we are forced to ask,

What are those motor representations doing here?

Conjecture:

Collective goals are represented motorically.

Let me explain what this amounts to. \begin{enumerate} \item There is one outcome which each agent represents motorically, and \item in each agent this representation triggers planning-like processes \item concerning all the agents’ actions, with the result that \item coordination of their actions is facilitated. \end{enumerate}
This is not my conjecture but something which I take several researchers to be moving towards. The conjecture has recently been very neatly formulated by Lucia Sacheli and colleagues:

Prediction 1 (della Gatta et al, 2017):

Framing two agents’ simultaneous unimanual actions as joint can induce effects similar to bimanual coupling.

Prediction 2 (Sacheli et al):

Framing two agents’ sequential actions as a joint action modulates the effects of ‘incongruent’ actions.

Sacheli et al, 2018 figure 2 (part)

Sacheli et al, 2018 figure 5

[Skip -- just in case anyone asks]

Sacheli et al, 2018 figure 3

So I was asking,

What are those motor representations doing here?

Conjecture:

Collective goals are represented motorically.

Prediction 1 (della Gatta et al, 2017):

Framing two agents’ simultaneous unimanual actions as joint can induce effects similar to bimanual coupling.

Prediction 2 (Sacheli et al):

Framing two agents’ sequential actions as a joint action modulates the effects of ‘incongruent’ actions.

But how does the conjecture relate to joint action for HRI?

How to characterise joint action?

Step 1: identify features associated with things commonly taken to be paradigm joint actions in nonmechanistic terms, e.g.

- collective goals

- coordination

- cooperation

- contralateral commitments

- experience

- ...

Step 2: ... which generate how questions

Step 3: ... leading to discoveries about mechanisms

What about coordination?
No evidence but theoretically it should work. (I cut this but put it back in because it’s relevant to the concluding challenge about HRI and motor coordination.)
Let me start by stepping back and consider an individual action.
An agent moves a mug from one place to another, passing in from her left hand to her right hand half way [*demonstrate].
It’s a familiar idea that motor representations of outcomes resemble intentions in that they can trigger processes which are like planning in some respects.
These processes are like planning in that they involve starting with representations of relatively distal outcomes and gradually filling in details, resulting in a structure of motor representations that can be hierarchically arranged by the means-end relation \citep{bekkering:2000_imitation,grafton:2007_evidence}.
Processes triggered by motor representations of outcomes are also planning-like in that they involve selecting means for actions to be performed now in ways that anticipate future actions \citep{jeannerod_motor_2006,zhang:2007_planning,rosenbaum:2012_cognition}.
Now in this action of moving a mug, there is a need, even for the single agent, to coordinate the exchange between her two hands.
(If her action is fluid,
she may proactively adjust her left hand in advance of the mug’s being lifted by her right hand \citep[compare][]{diedrichsen:2003_anticipatory,hugon:1982_anticipatory, lum:1992_feedforward}.)
How could such tight coordination be achieved?
Part of the answer involves the fact that motor representations and processes concerning the actions involving each hand are not entirely independent of each other.
Rather there is a plan-like structure of motor representation for the whole action and motor representations concerning actions involving each hand are components of this larger plan-like structure.
It is in part because they are components of a larger plan-like structure that the movements of one hand constrain and are constrained by the movements of the other hand.
But how is any of this relevant to the case of joint action?
Earlier we considered what is involved in performing an ordinary, individual action, where an agent moves a mug from one place to another passing it between her hands half-way.
Compare this individual action with the same action performed by two agents as a joint action.
One agent takes the mug and passes it to the other, who then places it.
The joint action is like the individual action in several respects.
First, the goal to which the joint action is directed is the same, namely to move the mug from here to there.
Second, there is a similar coordination problem---the agents’ two hands have to meet.
And, third, the evidence we have mentioned suggests that in joint action, motor representations and processes occur in each agent much like those that would occur if this agent were performing the whole action alone.
Why would this be helpful?
 
Suppose the agents' planning-like motor processes are similar enough that, in this context, they will reliably produce approximately the same plan-like structures of motor representations.
Then having a single planning-like motor process for the whole joint action in each agent means that
\begin{enumerate}
\item in each agent there is a plan-like structure of motor representations concerning each of the others’ actions,
\item each agent's plan-like structure concerning another's actions is approximately the same as any other agent's plan-like structure concerning those actions,
\item each agent's plan-like structure concerning her own actions is constrained by her plan-like structures concerning the other’s actions.
\end{enumerate}
So each agent’s plan-like structure of motor representations concerning her own actions is indirectly constrained by the other agents' plan-like structures concerning their own actions
by virtue of being directly constrained by her plan-like structures concerning their actions.
In this way it is possible to use ordinary planning-like motor processes to achieve coordination in joint action.
What enables the two or more agents' plan-like structures of motor representations to mesh is not that they represent each other's plans but that they processes motorically each other's actions and their own as parts of a single action.
 
So how does the joint action differ from the corresponding individual action?
There are at least two differences.
First, we now have two plan-like structures of motor representations because in each agent there is a planning-like motor process concerning the whole action.
These two structures of motor representations have to be identical or similar enough that the differences don’t matter for the coordination of the agents’ actions---let us abbreviate this by saying that they have to \emph{match}.
The need for matching planning-like structures is not specific to joint action;
it is also required where one agent observing another is able to predict her actions thanks to planning-like motor processes concerning the other’s actions (we mentioned evidence that this occurs above).
A second difference between the joint action and the individual action is this.
In joint action there are planning-like motor processes in each agent concerning some actions which she will not eventually perform.
There must therefore be something that prevents part but not all of the planning-like motor process leading all the way to action.
Exactly how this selective prevention works is an open question.
We expect bodily and environmental constraints are often relevant.
There may also be differences in how others’ actions are processed motorically \citep[compare][]{novembre:2012_distinguishing}.
\footnote{\citep[p.\ 2901]{novembre:2012_distinguishing}: 'in the context of a joint action—the motor control system is particularly sensitive to the identity of the agent (self or other) of a represented action and that (social) contextual information is one means for achieving this distinction'}
And inhibition could be involved too \citep[compare][]{sebanz:2006_twin_peaks}.
My proposal, then, is this. In both practical reasoning and motorically, sometimes agents are able to achieve coordination for joint action not by representing each others’ plans but by treating each other's actions and their own as if they were parts of a single action. This is the fundamental idea behind co-representation, as I see it.

How to characterise joint action?

Step 1: identify features associated with things commonly taken to be paradigm joint actions in nonmechanistic terms, e.g.

- collective goals

- coordination

- cooperation

- contralateral commitments

- experience

- ...

Step 2: ... which generate how questions

Step 3: ... leading to discoveries about mechanisms

One mechanism can explain how humans achieve all of these things, at least for the case of very small-scale actions.

conclusion

In conclusion, ...

We need to identify features associated with things commonly taken to be paradigm joint actions in nonmechanistic terms.

This isn’t just about collective goals. Although I haven’t presented the case for it here, I think there’s substantial new theoretical work needed on the notions of cooperation and commitment too.
In conclusion, I suggest that motor representations are among the things which can enable humans to perform very small-scale joint actions:

In very small-scale joint actions,
we each typically represent the same outcome motorically,
which facilitates coordination,
grounds a collective goal,
& enables us to cooperate.

Caveat: this view won’t over large-scale cases of acting together in the ‘we mode’. For example, if in acting in the ‘we mode’ our goal is to organise an international conference, since this goal cannot be represented motorically (I suppose), it can’t be that our acting in the ‘we mode’ is entirely a matter of how we represent things motorically.
The excitement isn’t that motor representation can explain ALL cases but that it can explain ANY cases at all.

‘a human agent should be able to engage in an interaction with a robot in a natural way’

The theory leads to a challenge given JointAction4HRI’s commitment to the idea that ‘a human agent should be able to engage in an interaction with a robot in a natural way’ together with the assumption that some artificial agents will be attractive partners because they have abilities humans lack (and conversely). The challenge is to understand how, despite the role of motor representation in joint action, individuals with different action abilities might interact in a natural way when performing very small-scale joint actions.
In the case of humans, motor representations of collective goals works for coordination and trade-off cooperation to the extent that the two humans are similar enough motorically ...