an
groups-icon

Robots.txt

Forum Home / Directory Theme
Wait a minute... This content is very very old. We no longer sell or support this theme version. New Topics
  • Version 9
    user
    Richard Corfield
    0 points
    Beginner

    Having a look at robots.txt file

    Does this look right so SE’s can see the site?

    In advance, many thanks…

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /category
    Disallow: /tag
    Disallow: /author
    Disallow: /trackback
    Disallow: /*trackback
    Disallow: /*trackback*
    Disallow: /*/trackback
    Disallow: /*?*
    Disallow: /*.html/$
    Disallow: /*feed*

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    Sitemap: https://site.com/sitemap.xml

    RewriteCond %{HTTP_USER_AGENT} libwww-perl.*
    RewriteRule .* ? [F,L]

    September 21, 2015 at 3:05 am
  • user
    Mark Fail
    11,845 points
    Platinum User
    Bronze User Silver User Gold User Platinum User Helpful Award Fast Responder Thank You Coffee Villian Award Video Testimonial
    Members Only Content

    This reply is for PremiumPress customers only.

    Login Now
    September 21, 2015 at 3:48 am

8,236

Views

1

Replies

Discussion Groups
New Topics