actor critic possible error

[hawkeoni]

Hello, Ben!
Thank you for a great tutorial series. I have a question regarding your actor-critic notebook.
In function update_policy

def update_policy(returns, log_prob_actions, values, optimizer):

    returns = returns.detach()
    
    policy_loss = - (returns * log_prob_actions).sum()
    
    value_loss = F.smooth_l1_loss(returns, values).sum()
    
    optimizer.zero_grad()
    
    policy_loss.backward()
    value_loss.backward()
    
    optimizer.step()
    
    return policy_loss.item(), value_loss.item()

in policy loss you calculate the usual policy gradient for agent, in value loss you calculate loss for the critic. They seem to be independent, the critic does not affect the agent at all. Shouldn't returns for policy loss be calculated with values from critic or something like that?

[bentrevett]

@hawkeoni Sorry for the delayed response. You are correct, the two are independent of each other, which is an error. I believe you're right, in that the policy loss should be calculate with the predicted values. I'll look into fixing this asap.

[bentrevett]

@hawkeoni I have been playing around using the following for the policy loss function:

policy_loss = - (values.detach() * log_prob_actions).sum()

I believe this is mathematically correct, however I cannot seem to get a good set of hyperparameters that will make this notebook train. I guess this is why people always use the "A2C" actor critic version. Although technically my notebook isn't A2C at all. I'll have to have a think about how to re-arrange these notebooks for them to make sense.