Experiment Design

Visualizing Identified States

This figure shows a heatmap for states weights and some example critical states for the game BeamRider. The left part shows most states are close to zero, and consecutive states tend to have similar weights. The right part shows examples of consecutive states. The top three corresponds to an example whose weights deviate from zero, in which the agent is close to an incoming missile in the presence of enemies. This is indeed a critical for this game, as once hit by the missile the agent loses a life. The middle part illustrates an example for states with weights close to zero. In this case the agent has launched a missile in an open space, which is irrelevant for conquering the game. The bottom part illustrates a transition from a state with large absolute weight to a state with small absolute weight. This example corresponds to the process of losing a life.

A categorization of 60 identified states demonstrates how the proposed framework may be used in interpreting reward functions. 74.8% of them are indeed critical for the task, which can be categorized into seven cases. The remaining are not critical, and they can be grouped into three cases.

Critical Cases

(a) Incoming enemies

In this case, there are incoming enemies (circled in blue) in the screen. The agent is supposed to destroy enemies in this game.

There are eight of the identified states belongs the this case. The remaining seven are shown below.

(b) Incoming enemies and enemy missiles

In this case, there are both incoming enemies and enemy missiles (circled in green). While destroying enemies, the agent should avoid being destroyed by enemy missiles.

There are two of this case. The other one is shown below.

(c) Off-target missiles

In this case, the agent launches a missile (circled in yellow) but fails to destroy an enemy.

There are 10 of this case. The other nine are shown below.

（d) Destroying an enemy

In this case, the agent destroys an enemy.

In addition to this, there are six identified states belong to this category.

(e) Losing a life

In this case, the agent loses a life (circled in red).

There are 14 other states belonging to this case.

(f) Launching missiles in the presence of enemies and enemy missile

Another state for this case is shown below.

(g) Destroying an enemy in the presence of incoming enemy missile

Another state for this case is shown below.

Non-critical cases

(a) Launching missiles in open space

In this case, the agent launches missiles in open space, which take no effect.

There are two other states for this case.

(b) Flying in open space

In this case, the agent's spacecraft is flying in open space.

Six more states are shown below.

(c) Screen flickering

In this case the screen flickers.

Four more cases are shown below.

Evaluating State Weights

A perturbation analysis is proposed to investigate whether the identified states are indeed critical. The state-action pairs in expert demonstrations are arranged in descending order according to some method, and the top-k states are removed. The methods in comparison are the proposed framework, absolute reward values (AR) and random removal. Behavioral cloning agents are trained on what remains.

In this figure, the x-axis is the ratio of data removed. The y-axis is the task performance of behavioral cloning agents.

Task Performance

Performance is not sacrificed in exchange for interpretability.