Comments on: How to write robots.txt file effectively?

By: elan

elan — Mon, 10 Nov 2008 04:36:53 +0000

Zack, if you want to remove php files from yahoo, then go with this,

User-agent: yahoo-slurp
Disallow: /*.php$

You can use for Live also.

zack — Mon, 10 Nov 2008 03:33:32 +0000

Hey Elen. Thanks for the simple tips!

Before this I don’t know that we can block SE to crawl any php files.

By the way, are this command only for googlebot?

User-agent: Googlebot
Disallow: /*.php$

elan — Wed, 15 Oct 2008 15:58:51 +0000

Hi George,

Yeap you can follow the below instruction,

Disallow: filename.php

tats it.

But for blocking one particular URL u can use meta robot.

George — Fri, 10 Oct 2008 15:17:23 +0000

hi elan,

one doubt. how can i write if i just want to block one url