Tim says I missed his point
, but I think he's the one who's just off center here. When I say "The Man" I don't mean the advertiser but the engine. Google and Yahoo! are The Man.
They have (or will have, in Google's case) a quarterly number to hit. If they get paid for clicks to advertisers, where's the incentive for them to keep their indices pure? Yahoo! (really Inktomi) plays a particularly sneaky game with their Paid Inclusion program, which puts paying advertisers' links into their main index (along with the rest of their "natural" search).
I'm saying that where the commercial engines will always compromise in favor of cash, nutch-driven search projects have a greater chance aligning with Joe Searchpack's best interests: a pure index, relevant results, free from spammers, SEOs, and hucksters. Relatively speaking, they stand a greater chance of doing so than do Google or Inktomi.
