The value function represents the long-term quality of a state. This is the cumulative reward that is expected in the future if the agent starts from a given state. If the reward measures the immediate performance, the value function measures the performance in the long run. This means that a high reward doesn't imply a high-value function and a low reward doesn't imply a low-value function.
Moreover, the value function can be a function of the state or of the state-action pair. The former case is called a state-value function, while the latter is called an action-value function:
Here, the diagram shows the final state ...
