16 lines
456 B
Text
16 lines
456 B
Text
User-Agent: *
|
|
Disallow: /directorysearch
|
|
Disallow: /latest
|
|
Disallow: /search
|
|
Disallow: /tools/tellafriend
|
|
|
|
#
|
|
# Blocked journals aren't listed here because robots.txt files
|
|
# can't be above 50k or so, depending on the spider.
|
|
#
|
|
# Instead, blocked journals have HTML inserted in them which
|
|
# should prevent behaved spiders from indexing it.
|
|
#
|
|
# Note that https://username.dreamwidth.org journals have an
|
|
# autogenerated robots.txt, since it can be small.
|
|
#
|