Google Ignores Whitehouse.gov Attempt to Block Snowden Pardon Petition

Whitehouse.gov may be trying to hide the Snowden pardon petition from search engine results. Why might Google be indexing the content anyway?

I’ve been following the Edward Snowden – NSA saga the past week or so with fascination, as I suspect some of you are as well. Last night over dinner, my wife and I were pondering what might be the final outcome of this, depending what happens between Russia (or now Ecuador) and the US in the coming days. I wondered – might there be any chance of an eventual pardon for Snowden from the White House on Obama’s last day in office? There must be some discussion of whether a pardon could be in the works or not, right? So I consulted the Oracle of Google, searching pardon Edward Snowden.

Below a few news feed results, the number one organic result is a subdomain from Whitehouse.gov. This ‘petitions’ subdomain facilitates citizens to create, manage, and promote petitions to our government. If a petition receives more than 100,000 supporters then the administration has made a commitment to address the petition with a response on the matter in question.

What is immediately curious to any of us with a trained eye in search marketing is that the result from Petitions.whitehouse.gov is ranking highly despite the page being marked disallowed by the subdomain’s robots.txt file.


My first thought leaned on the side of ‘conspiracy theory’: Does Whitehouse.gov want to stifle the discoverability of the petition to pardon Snowden, so they took proactive steps to get the page out of the index? I’m not one that generally subscribes to paranoia, but given what Snowden revealed with PRISM, maybe we would be right to be more skeptical.

Was the page indexed by Google and then Whitehouse.gov later updated its robots.txt file in order to block it? After some quick digging, I uncovered a few things worth noting:

  • The entire directory for Disallow: /petition is blockedso this issue is systematic to pages in the directory, not the individual page.
  • There is not a no-index meta tag at the page level
  • The robots.txt file was last cached on 6/13/13 and as of that date the /petition/ directory was blocked, so at least there was not a (very) recent update to the robots file to block this page or directory.
  • While Google will not crawl a robots blocked page, it will at times determine a page has merits because of other reference factors outside the page (read: link quality and volume, anchor text, social factors) and thereby show the page in its results.

Initial Conclusion: No after-the-fact steps appear to be taken by webmasters at Whitehouse.gov to block the Snowden petition page in particular. The actual content of the page is not showing in the indexed results, but Google did choose to include the page URL in its result set because other signals that indicate that it’s a relevant page (to Google it’s the number one standard search result out of ~571,000).


The Bigger Question: Why is Whitehouse.gov choosing to block search engines from indexing content of their petition pages, when these pages are created by the people and for the people to express and promote concerns to their government leaders? I cannot think of a good rationale for this. Can you? Is anyone out there at Sunlight.org listening?

I’ve created a petition page on Petitions.whitehouse.gov to petition the Obama administration to remove the robots.txt disallow from petitions on their site. This action will promote the transparency and conduit for democracy in action that the web platform was created to serve in the first place.

Find the petition located here and pass this URL to your networks.

People may have trouble finding my new petition via search engines, so that will make it harder to achieve the 100,000 signatures to garner its due attention. Oh, the delicious irony…


29 responses to “Google Ignores Whitehouse.gov Attempt to Block Snowden Pardon Petition”

  1. […] noted the oddity this week, that the page is listed, but without a description – and that this is due to be […]

  2. […] noted the oddity this week, that the page is listed, but without a description – and that this is due to be […]

  3. […] noted the oddity this week, that the page is listed, but without a description – and that this is due to the page […]

  4. […] remarkable a oddity this week, that a page is listed, yet but a outline – and that this is due to a page strictly […]

  5. sshdrone says:

    Your article suffers from a common myopia more likened to a conspiracy theorist-SO it’s nice that you don’t want to sound that way But you do. The connections are irrelevant, and at face value to any rational web developers likely regarded as unrelated. Your article title and “bigger” concern more poignantly suggest feelings of disdain towards your government in light of the actual issues at hand, their proactive response regarding a possible pardon, and what you show no restraint is your interpretation of covering over every Americans’ right to fair and impartial justice. Please let it go-Or let me get paid to help you write such a nonsensical article. At least let ME benefit from the popularity that no doubt has SOME impact on your financial well being. I’m sure that last part might make you shake your head in some fantastical humor towards a stranger who finds your critique of a website that is not your own amazingly biased in your duality of concern with transparency of the greatest government that has ever existed to serve it’s people. More simply just get over it-He embarrassed himself. You don’t need to do the same. -A democrat..PS any application developer or Junior web architect could silently black hole any attempt to crawl a page. Fantastical assertions in this respect, and regarding something so simple as the robots text file, were made by You. And you know that. /Cheers!

    • Teresa Hearne says:

      I interpreted this article to be more about the overall attempt to make access to the content of all Whitehouse.gov petitions difficult rather than targeting any particular petition. I really didn’t read it as political. Why have a petition page at all if you are then going to take steps to try to bury it? I think that’s a legitimate observation because the exercise ends up looking more like theater than any genuine interest in what the people may have to say. As to transparency, if you create something only to hide it or make it difficult to find, that hardly meets the definition. In any case, I’m glad I stumbled onto this article. Thanks for writing it.

      • Kyle Anderson says:

        This change to the robots.txt occurred on January 8, 2013.



        Someone else can make conjectures as to why, but it is probably not related to Snowden.

        • Brian Donovan says:

          Now here’s a man who actually does a little homework as oppossed to pondering and making silly statements!

          • Lamapunk says:

            And it would be how hard to change the archive date? About as hard as it would be to rename a .txt file. In fact that is all they had to do. But there is no proof of that so I will let it go. However; if it is unrelated to Snowden, then why does a search for any other active petitions have a full description allowed by robot.txt? Really!? Snowden’s petition is the only one and we are to think it is not related? Oh yeah! I forgot. We ARE supposed to trust a spy (pointing at Washington).

        • esoberanis says:

          what was the issue at hand?

      • sshdrone says:

        That’s nice. And to summarize my OP just, “no”. Two mis understandings now. Keep going. No one tried to bury any thing. That could be done much easier. Robots text file was at the heart of the issue, if you want to even call it an issue. And inherently by nature is not used to bury anything. Fail.

        • Teresa Hearne says:

          Then I’m curious. If you were the webmaster of the site, why would you make this change? What would be your purpose? That’s a genuine question, SSH. No sarcasm or ironic voice. I’m just curious… why do it?

  6. Gaz Copeland says:

    I’m confused about the headlines of this post, they seem to directly contradict what’s actually happening.

    “Google Ignores Whitehouse.gov Attempt to Block Snowden Pardon Petition” – This isn’t the case, Google behaves in exactly the way you would expect it to when a site page is blocked by robots.txt

    ” Why might Google be indexing the content anyway?”- Google isn’t indexing the content in any way, your screen shots clearly show that as does your post.

    I’m all for making click worthy headlines, but they have to have some grounding in reality!

    • Brian Donovan says:

      Excellent point! Pretty sloppy and glib for an article that purports to be the recitation of such deep ponderings. The author may be a great marketer (???) but he’s no journalist. That much, at least, is clear.

      • esoberanis says:

        what’s he measure of a journalist? Those guys on tv? the ones who write for big name newspapers? if that is the measure of a journalist then this person is way ahead of them.

  7. netdragon says:

    You’re better off using the contact us link. I’ve made suggestions through it that have been accommodated (I probably wasn’t the only one with the same suggestion) such as archiving old petitions that don’t make the cut.

  8. Brian Donovan says:

    Soooo your credentials in national security are nil … At least acording to your bio above. You seem pretty glib about national security matters for a run of the mill marketing guy. Perhaps you’d like to trust Snowden in your own company. You guys sure make a federal crime out of every “industrial espionage” episode that comes along. Pretty hypocritical if you ask me.

    • I don’t think he was commenting on any national security issues. The point seemed to be that restricting access of search engines to its petition pages goes against the purpose of the White House’s petition website. It should be encouraging democracy, and yet they’re taking a step to limit visibility of these petitions.

      • Joe Griffin says:


      • Tom Lahman says:

        White House restricting access to petition websites? I am certain that you are incorrect in this baseless accusation! I distinctly recall Obama’s promise that his was to be the most transparent administration ever! Do you seriously believe an American president would lie to this nation’s citizens?
        More at: “Obama and the Destruction of America” Scribd.com

    • Susan Gagne says:

      Well put………….. hear, hear, what?

    • Lamapunk says:

      And your credentials in Security Exchange is most likely nil. As your credentials in Constitutional Law. Your ignorance is astounding, buddy. If my company was breaking the law I would welcome a whistleblower like Snowden. It’s about what’s right. In a speech following 9/11, Bush said if we shop as usual and go to Disney Land then the “terrorist will have lost”. The truth is if we preserved our guiding principles set forth by our Founding Fathers in the Constitution of The United States then no man lost will be in vain. No family shattered will go unjustified. No wound would heal without honor.
      Our government is becoming a bit too big for it britches. And besides; the article was indeed about the government trying to hide a petition, Not his view on the NSA … But you have mine.

    • pkill says:

      No one asked you, Brian, and if we can be harmed by telling the truth, then something went wrong with the government, not the Bill of Rights.

  9. NOSVEMOS2 says:

    The link you give for your petition doesn’t work. What a surprise. Obama moves from GRACIOUS to JERK

  10. bobw-66554432 says:

    Petition has “expired”. Another convenient way to suppress dissent.

    “Sorry, nobody was interested*” so we flagged it as expired. (*The fact we hid it doesn’t matter.)

    Right in there with “Didn’t pass our editorial review which is designed to ensure quality content*.” (*You disagreed with us so we’re not letting anyone else see what you wrote.)

  11. […] it appears that the petitions were blocked on January 8, 2013: before, […]

  12. Allen Marples says:

    What is funny is how convinced people are that an online petition, any on;ine petition can affect anything..
    The whole online petition thing was an illusion created by the Obama campaign to make people feel empowered when they really are not.

  13. Crampon says:

    Dictators always control the media. They start by hanging giant posters of the dictator around the country, coerce the media, then they will indoctrinate the kids in public school. Obama has done all this and more.

  14. vfibsux says:

    We don’t pardon traitorous pieces of dung.

  15. pkill says:

    Snowden is better qualified than Obama to be President…or anything else, including just a citizen. Snowden read and understands and agrees with the Bill of Rights, unlike the royalist dictatorship of liberals that is being created.

  16. One of the greatest American heroes of the modern age, Edward Snowden, once again forces the US government to show itself as a fascist police state bent on destroying freedom and the US constitution, violating the will of the people.

  17. brp says:

    I’m not sure how to come down on this guy, but remember when we used to execute traitors? A pardon?

  18. JamVee says:

    When the rest of the world decides to play it straight and honest, I will come out in Snowden’s defense, however, while many in this world are trying to murder us in any way they can, Snowden is aiding and abetting our enemies in that quest.