Google launches “news-specific crawler”

GOOGLE News has launched a “news-specific crawler” that lets online media automatically keep stories, photos or video out of its index.

The announcement today comes a day after the California-based internet giant said it is letting publishers limit the number of online pages people can view after being routed to their websites by Google’s search engine.

Publishers have always been able to block Google from including their website content in the search engine index.

Google senior business product manager Josh Cohen said in a blog post that a new “web crawler” extends that option to Google News.

Web crawlers are automated programs that scour the internet for content and then index it in databases routinely mined for results to online search queries.

News website publishers can fill out online forms telling Google’s crawler which content, if any, can be indexed.

Similar directives can be given to Google’s main search engine, which operates separately from the firm’s online news aggregation service.

“Most people put their content on the web because they want it to be found, so very few choose to exclude their material from Google,” Mr Cohen said.

“But we respect publishers’ wishes. If publishers don’t want their websites to appear in web search results or in Google News, we want to give them easy ways to remove it.”

Google, under fire from Rupert Murdoch, chairman and CEO of News Corporation, and some other newspaper owners, said yesterday it will let publishers set a limit on the number of articles people can read for free through its search engine.

Google’s announcement came as Mr Murdoch, who has threatened to block the internet giant from indexing his newspapers, and other US media heavyweights gathered in Washington to discuss journalism in the internet age.

Mr Murdoch has blasted Google and other news aggregators for “stealing” stories without sharing advertising revenue and has reportedly been holding talks with Microsoft about making News Corp’s content accessible exclusively through the software giant’s new search engine, Bing.

News Corporation is the parent company of the publisher of

Acknowledging that “creating high-quality content is not easy and, in many cases, expensive”, Google said in a blog post it is changing its “First Click Free” program.

First Click Free directs readers from Google or Google News to a story on a newspaper’s website but prevents them from having unrestricted access.

Google said, however, that some readers were “abusing” the program by returning to Google or Google News and clicking through to other stories.

“Previously, each click from a user would be treated as free,” Mr Cohen said. “Now, we’ve updated the program so that publishers can limit users to no more than five pages per day without registering or subscribing.”


thanks to yeni for this story..

this is to do with the move to “pay for news” that murdoch is so keen to get up and going..its one of those situations that if they dont get everyone on board then the ones who do make it pay only will be left out in the cold and lose hits..


~ by seeker401 on December 4, 2009.

15 Responses to “Google launches “news-specific crawler””



  3. wats the cats name

  4. whats the dogs name

  5. so pro

  6. thats nice. you`ll never encounter that kind of news every day

  7. salam arje minkom a3toli facebook dyali achabaka me3lja

  8. ha ha ha

  9. what got a ninja cat to google????

  10. bob dol is rich!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: