Register Log In

Contour Enthusiasts Group Archives Forums General Common Interests 1984

Print Thread

1984 #849344 01/21/04 08:49 PM
Joined: Feb 2001 Posts: 750 Nashville, TN T tiv_dup1 OP Veteran CEG\'er
OP tiv_dup1 Veteran CEG\'er T Joined: Feb 2001 Posts: 750 Nashville, TN	www gurus out there, does this story make sense about this robots.txt stuff ? I found it hard to believe even harder to explain if it is true. T.

Re: 1984 #849345 01/21/04 09:09 PM
Joined: Jun 2000 Posts: 1,231 Between the Moon and New York ... S Stevers Hard-core CEG\'er
Stevers Hard-core CEG\'er S Joined: Jun 2000 Posts: 1,231 Between the Moon and New York ...	Makes perfect sense, and it's probably legit. -06 GTO Torrid Red/M6 -98 LS with BPU -05 Honda Odyssey EX-L mv .zig ..\for\great\.justice

Re: 1984 #849346 01/21/04 09:15 PM
Joined: Jun 2001 Posts: 682 Middle of nowhere, IN D daenku32_dup1 Veteran CEG\'er
daenku32_dup1 Veteran CEG\'er D Joined: Jun 2001 Posts: 682 Middle of nowhere, IN	Makes sense for the Bush admin, but how about the rest of the country? 98.5 Contour SVT "Too many OB/GYNs aren't able to practice their love with women all across this country" --US President George W Bush

Re: 1984 #849347 01/21/04 09:59 PM
Joined: Dec 2002 Posts: 2,069 Southwest, Ohio D DimitriPopov Hard-core CEG'er
DimitriPopov Hard-core CEG'er D Joined: Dec 2002 Posts: 2,069 Southwest, Ohio	Very low believability. The listing isnt very official

Re: 1984 #849348 01/21/04 10:05 PM
Joined: Aug 2002 Posts: 68 CT- USA J JDRUID_dup1 CEG\'er
JDRUID_dup1 CEG\'er J Joined: Aug 2002 Posts: 68 CT- USA	yes. that is one way to prevent search engines from indexing certain dir's on your site. AOL AIM: nerdsalot2 05 Mustang GT DropTop 98 ZJ 4.0L Limited

Re: 1984 #849349 01/21/04 10:06 PM
Joined: Jul 2001 Posts: 3,037 Midlothian, VA J Jeb Hoge_dup1 Hard-core CEG\'er
Jeb Hoge_dup1 Hard-core CEG\'er J Joined: Jul 2001 Posts: 3,037 Midlothian, VA	Sounds like x-no-archive for Google Groups/Usenet. "Think of it, if you like, as a librarian with a G-string under the tweed." Clarkson on the Mondeo.

Re: 1984 #849350 01/22/04 02:27 AM
Joined: Dec 2001 Posts: 777 Atlanta, GA C chemguru_dup1 Veteran CEG\'er
chemguru_dup1 Veteran CEG\'er C Joined: Dec 2001 Posts: 777 Atlanta, GA	Sure... Go there yourself and look at the disallow list: http://whitehouse.gov/robots.txt u-238:/home/jamest %wget http://whitehouse.gov/robots.txt . . . u-238:/home/jamest %wc -l robots.txt 1734 robots.txt u-238:/home/jamest %grep -c Disallow robots.txt 1729 You'll see what directives are disallowed to searched by an archiver/search engine. Now, that's not stopping anyone from using wget's* mirror function and grabbing everything from a site, but it keeps Google, et al from archiving and/or indexing the content. *: Of course, the site could still have an IP filter in place to block any request from certain locales. --JamesT >--------------< --Chemguru 99 CSVT Frost /Mid. Blue 00 Suzuki SV650 Red, Naked

Moderated by Lance Kinley_dup1, Shaggy_dup1, Trapps_dup1, unisys12

Link Copied to Clipboard

Powered by UBB.threads™ PHP Forum Software 7.7.5