I think that there might be some lag time or it is a bug.
I have also observed that if I review something the box doesn't always go away; however, I have seen that the items that I reviewed were still there. Specifically I marked something to be Deleted and it didn't go away. When I clicked to Delete it again, I got an error message that I already Deleted it, or something to that effect.
I don't see the items that I reviewed anymore, but it's been a few hours since I looked.
I just noticed today that it keeps the number in the bar, but the page then shows me that I've already reviewed some of them. In the case where I hadn't reviewed all of them it showed the ones that I reviewed as faded. In the case where I already reviewed all of them I got a message with a link to show posts I've already flagged. Note how it still has the 4 in the nav bar.