where Q t+1(s t,a t ) means the update of Q value at time t+1, after executing the action a at time step t. r t+1 means the immediate reward after executing the action a at time t.
In addition to the predictive effects, the sensation of the actual effect (after executing the action) also has been noted as important in action formation research [15, 17, 21, 37, 43].
where r t+1 is the reward after executing the action a. Reward can be calculated by following equation r_{t+1}=frac{r_{t}}{P_{a,t}} (41).
If the execution of an action generates random requests, the algorithm cannot detect the correct action since executing the action generates requests which are different from the requests in input trace.
where Q t+1(s t,a t ) means the update of the Q value at time t+1 after executing the action a at time step t. r t+1 represents the immediate reward after executing the action a at time t, V t represents the value function for node at time t, and V t+1 represents the value function at time t+1.
If the agent is executing the action which is associated to negative feeling (though at the same time a possible positive action also exists) this will affect the agent's mood (the mood of the agent has not been in the scope of this model but will be a future work).
