You can readily look at how many votes a user has cast on SFSE by going to UersUsers then clicking on voters. As someone who's very high on that list, I've always found it interesting to see the distribution of where others stand who have high vs medium and low reputations in our community. The person right below me on that list has about 1/6th of what I do in reputation. I think that says a LOT about his interest in this forum.
There are others who have quite high amounts of reputation that are much lower down in the rankings for voting. There are more than a few with 15k or more of reputation that don't appear until the 3rd and 4th page. What impresses me is the number of users with 1.5k or less that appear on the 1st and 2nd page!
I think in general, your hypothesis is correct that up votes come from the "council of elders", but those elders may not all have high reputations. Many of the names on that voter list I recognize as being long time members of this community for having asked questions, but not necessarily for having answered posts that received up votes. That may be the difference that you've not taken into account in whom you've included in your list of "elders".
I do agree though that over time, we see our answers voted up on a regular basis as they're viewed by users who have sufficient reputation to do so. I know it happens to me on a regular basis, quite often on answers that I don't even remember having written. I sometimes find my own answers when I'm searching for something. Its nice to have them cataloged for my use when needed. :-)
I've corrected the links to go to the main SF.SE Users instead of to Meta. My original comments were based on viewing the meta links. In looking at the main site, the results are remarkably similar which to me says a lot. My apologies for the original error in my post.