Main Menu

Yesterday and today

Dr. Conspiracy

Unless you’re interested in the technical aspects of WordPress blogging, stop here.

I tried to get on the blog late yesterday afternoon and I kept getting Internal Server Errors and 404 Page Not Found errors. What this means is that the script processes that feed this web site’s content were running out of memory and getting canceled. This can happen during times of very heavy traffic and (I believe) when the shared server where the site is located has very heavy traffic on other sites.

This site uses WordPress caching technology (WP Super Cache, Widget Cache and Plug-in Cache) so that when the pages don’t change, the software will feed you a copy it created earlier, requiring hardly any computer resources. That’s great in theory. However, this is a heavily-commented blog meaning that every time someone leaves a comment on an article, that page has to be built again to include the new comment.

Back to yesterday. Analysis of yesterday’s site statistics shows strong activity, but not remarkably strong, and not the heaviest of the month. The peak load yesterday was about 11.5 pages a minute (which doesn’t seem all that much) from 5 pm to 6 pm. Still, generating a page with a couple hundred comments on it can take a while, and if a few requests collide they can start a jam just like cars on the highway.

This brings me, finally (!) to what this article is about. Last night at 8:20 pm I made a significant change in how the site works. It’s called FastCGI and it’s a trade off between memory and speed. In theory, it should smooth out the memory usage (albeit at a high level) and make the scripts run faster, making it less likely to jam up.

Normally FastCGI isn’t very helpful when caching is enabled, but on this blog with lots of comments, I think it might help, and perhaps help a lot. FastCGI is technically more complex leading the down side of more things potentially to go wrong.

So if you notice performance today different from yesterday (or not), leave a comment.

9 Responses to Yesterday and today

  1. avatar
    G June 16, 2010 at 12:36 am #

    I was having those same errors over the past 2 days. I haven’t had any problems today. It seems faster, but I don’t know if that is just due to lesser traffic at the moment or even my own perception bias, by having read this article before really thinking about it.

  2. avatar
    Jody June 16, 2010 at 2:11 am #

    I’m getting errors when I click on your links from Twitter.

  3. avatar
    Dr. Conspiracy June 16, 2010 at 9:43 am #

    Jody: I’m getting errors when I click on your links from Twitter.

    Sorry for the inconvenience. The URL shortening service used to make short article links suitable for Twitter is broken.

    URL shortening services are ubiquitous these days so we tend to forget how these systems can be exploited if not properly secured. The latest example comes via an announcement that Cli.gs, the 4th most popular URL shortening service on Twitter, has been hacked. According to the Cli.gs blog, sometime late Sunday night a hacker exploited a security hole that allowed the attacker to redirect around 2.2 million cli.gs URLs to a single domain name, freedomblogging.com

    Read more at: http://thenextweb.com/2009/06/16/popular-url-shortener-cligs-hacked/

  4. avatar
    Dr. Conspiracy June 16, 2010 at 8:02 pm #

    This comment came in email:

    With no disrespect intended, I just don’t see the traffic here being any real burden on a server. Comments too. They’re very low in numbers and nothing like the thousands found on HuffPo and other servers.

    So no slam to you or this blog, but I just don’t think that any problems are attributable to the traffic here, the ratings system or any of that.
    A personal computer could handle such a load and servers are considerably faster.
    Perhaps your server itself is having problems, but I just don’t see the traffic or comment numbers here being the problem at all.

    I thank you for this blog and all the effort poured into it, but I get the same feeling I got when Dr. Manning stated 30,000 people as likely attending his “trial” as I do when bandwidth is mentioned here as a problem.

    Thanks for your comment. The Obama Conspiracy Theories blog runs on a shared server that runs many other web sites. I don’t know how many, but I suppose it is as many as the hosting company can put on it while allowing them to run reasonably well.

    In a peak hour, the site processes about one page every 4 seconds (on average). There are over 46,000 comments in the database, though.

    If you view page source on one of the pages from this site, you will see at the very bottom something like this:


    Showing that the page generation takes about .4 seconds (now that FastCGI is on). One hopes that most of the time, however, you receive a cached page, one already generated, which takes very little server resources.

    So I agree with you that site traffic shouldn’t be a problem at the levels this site experiences.

    By the way, I found that one of the caching plug-ins I installed was not turned on. This has been fixed.

  5. avatar
    Northland10 June 16, 2010 at 9:03 pm #

    Actually, HuffPo is running off of multiple servers. I would suspect the most of them are dedicated serves on top of that. But one would expect that of a large online news company with a technical staff. It is not an apples to apples comparison.

  6. avatar
    Sef June 17, 2010 at 12:53 pm #

    Doc, I presume you are aware of this, but it appears that the recent caching changes at your server are preventing updates of the “Recent Comments” section. Previously a page refresh would get the changes, but that’s not working now.

  7. avatar
    Dr. Conspiracy June 17, 2010 at 2:01 pm #

    Sef: Doc, I presume you are aware of this, but it appears that the recent caching changes at your server are preventing updates of the “Recent Comments” section. Previously a page refresh would get the changes, but that’s not working now.

    I was not aware of this. The Recent Comments widget is supposed to be refreshed whenever a comment is posted. I’ll look into it.

  8. avatar
    G June 19, 2010 at 4:28 pm #

    Sef: Doc, I presume you are aware of this, but it appears that the recent caching changes at your server are preventing updates of the “Recent Comments” section.Previously a page refresh would get the changes, but that’s not working now.

    I noticed the same problem late last night, so it is not just Sef experiencing this.

    It seemed that the problem was not consistent either. For instance, I could post a comment on an article and then see my comment appear as the most recent comment while still on that article. However, if I then went to just the main home page and refreshed, I somehow saw an earlier recent comments view, prior to *any* comments I had made.

    So yeah, definitely a confusing problem, as Recent Comments are all I have to go by to try to ascertain where there is new info that I haven’t seen yet.

    Please let us know when you’ve found/fixed the problem.

  9. avatar
    Dr. Conspiracy June 19, 2010 at 8:15 pm #

    @ G and Sef re: Caching issues

    Late this morning I installed WordPress 3.0 and I reset some parameters in the caching. Please let me know if this is still happening. In any case, all the pages should get refreshed at least every 30 minutes.