Twisted PageGetter – block or not

I found a Bot identifying itself as Twisted PageGetter in the access log of my site.

After I googled for it I found on Matt’s Blog that he suggests to block this Bot with an htaccess entry.

This is working quite well, I tried it.

But: I am using Spotplex as a counter and wondered why they were not able to update my feed and get my latest posts.

Spotplex about Spotplex: …provides internet users with real-time ranking of blog articles based on actual impression count….

So I removed the 403 Forbidden Error code for Twisted PageGetter from my .htaccess file and within some minutes my latest posts showed up on Spotplex.

For now I decided to let them take my feed as they bring back readers to my site too.

If you feel you have to block this Bot try the code from Matt’s Blog, it will do it.

Am Ende waren es dann rund 55.000 Hits in einem Zeitraum von fünf Stunden. Am "fleißigsten" war der UnwindFetchor mit 33,890 Hits, gefolgt von Twisted PageGetter und Voyager mit jeweils rund 8.000 Hits. Nachdem wir Instance Variables: _pletelyDone: A boolean indicating whether any further requests are necessary after this one pletes in order to provide a result to self.factory.deferred. E.U. Fines Google 1.49 Billion Euros Over Antitrust in Advertisiing; ICANN May Allow .g Domain Renewal Prices to go Sky High; Firefox 66 Released, Autoplay ing By Default Tags: twisted, pagegetter, search, engine, twisted.web.client.HTTPClientFactory : API documentation overrides .internet.protocol.Facty.buildProtocol Create an instance of a subclass of Protocol. The returned instance will handle input on an ining server connection, and an attribute "facty" pointing to the creating facty. Gets a resource via HTTP, then quits. Typically used with HTTPClientFacty. e that this class does , by itself, do anything with the response.

One Comment

  1. Thanks for the link.

    So this bot is for SpotPlex, odd. I actually stopped the blocking awhile back, because I started using FeedBurner and the rule blocking Twisted PageGetter was underneath the rules to redirect feeds to FeedBurner, so the bot just got redirected.

    I have an email in to SpotPlex to confirm this.

Leave a Reply

Required fields are marked *.

This site uses Akismet to reduce spam. Learn how your comment data is processed.