Comment On Best of 2006: The Spider of Doom

And how did that declaration to the users work out?

Or did they just restore a backup copy every night when it got overwritten yet again?

10:30AM (at least here in New York)???? You are going to spoil us!

Happy Holidays To All

BTW: You can grab any image (CodeThulu, ...), upload to zazzle.com and make a custom mug for ~$20 (including shipping)!

Ok, this is great!

Imagine one little change could have fixed a lot of this, instead of looking for IsLoggedOn = False to deny access, simply look for IsLoggedOn = True to allow it.

This is a perfect example of why you always default to deny access rather than grant.

It would be somehow amusing if they subsequently tried to sue Google for unauthorised tampering on their systems. It would be less amusing if they won of course.

 

KattMan:

Ok, this is great!

Imagine one little change could have fixed a lot of this, instead of looking for IsLoggedOn = False to deny access, simply look for IsLoggedOn = True to allow it.

This is a perfect example of why you always default to deny access rather than grant.

Ummm, guys...  Basic security lesson here is that you can't do ANY client-side security and cookies are most definitely client-side.  Anyone with the tiniest bit of hacking skills can go to this site and add a cookie called isLoggedOn and set it to true and poof, they're logged in without entering any credentials.

You need to use a secure, unique (i.e. non-guessable) session cookie, put a token into the HTTP session on the server side indicating that they've logged in and check it on every request. (Except for the login request of course!)
 

Anonymous:

Ummm, guys...  Basic security lesson here is that you can't do ANY client-side security and cookies are most definitely client-side.  Anyone with the tiniest bit of hacking skills can go to this site and add a cookie called isLoggedOn and set it to true and poof, they're logged in without entering any credentials.

You need to use a secure, unique (i.e. non-guessable) session cookie, put a token into the HTTP session on the server side indicating that they've logged in and check it on every request. (Except for the login request of course!)

Sir, you offer a proper, reasonable, well thought out solution to a common problem. If everyone took suggestions like yours, this forum would have nothing to post. We'll have none of that! </smirk>

Anonymous:

Ummm, guys...  Basic security lesson here is that you can't do ANY client-side security and cookies are most definitely client-side.  Anyone with the tiniest bit of hacking skills can go to this site and add a cookie called isLoggedOn and set it to true and poof, they're logged in without entering any credentials.

You need to use a secure, unique (i.e. non-guessable) session cookie, put a token into the HTTP session on the server side indicating that they've logged in and check it on every request. (Except for the login request of course!)
 

I know that, but as for a quick fix with a very low cost my sugegstion works.  Then comesthe major overhaul of the security system afterwards.  You have to buy some time somewhere.

The "real" workaround would be to use a automatically generated robots.txt file. This is real enterprisely!

PH 

Anonymous:

WALL STREET FGT, UR WORK USES MIRRORS AND BUTTER

Wow, buttered mirrors.  Is this to assure that our reflections have a slick appearance.

KattMan:

Ok, this is great!

Imagine one little change could have fixed a lot of this, instead of looking for IsLoggedOn = False to deny access, simply look for IsLoggedOn = True to allow it.

This is a perfect example of why you always default to deny access rather than grant.

You should also probably not implement your entire security layer on the client side and not use GET to perform horribly, horribly non-idempotent operations.

But hey, that's a start :-)
 

The issue lies not with the authentication, but with the fact that the rule that all HTTP GETs should be safe was broken.

See http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.1.1 which states that "... GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval."

Anonymous:

<snip>

Apparently disaster recovery isn't that big of a deal for a government agency.

I work for a place that is militant on having full SDLC - everything thoroughly documented, signed off on by all relevent departments, and a central SDLC-group that makes sure enough people looked at and approved it. One thing in particular is redundancy, failover and backup. They buy active/standby local server pairs in production, duplicate the pair in the disaster recovery location, striped raid hit-it-with-a-bazooka-and-no-data-is-lost disk arrays, fully backed up databases; the works.

Sounds good, right?

Except that many of the folks doing it have become so dependent on following the checklist of signoffs that they only look for the signatures without verifying that what they are approving makes any sense.

Someone recently ''architected' something that included staged incremental backups to be taken every 15 minutes. That essentially means duplicate the DB daily, snag the incremental backups every 15 minutes, and apply offline to backup system periodically to keep it close to, if not, current. The incremental backups are relatively small and quick to make, so in case of disaster, we're covered, right?

Except that the procedure put the incremental backups on the same physical disk platters as the stuff being backed up, under the same physical database servers.

It's like backing up C: to D: in another partition on the same physical disk. Sheesh!

Even when it's a big deal it's not a big deal.

*sighs*

Anonymous:

Isn't the point is that googlebot isn't paying attention to any client side code? You can use a client side check like IsLoggedOn, isLoggedOff or IsCarryingCodFish, but googlebot doesn't care. Once it GETs the page it sees the links regardless and merrily chews right through them while the client code looks on like an ineffective bouncer.

The point is nothing to do with googlebot.

I use Firefox with NoScript and CookieSafe. I could easily surf to that page and just delete everything. Hell, I could also use Lynx and just delete the content in a few minutes. Alternatively, I could change the pages to reflect arbitrary content. I could change safety tips to be dangerous, put ad banners for arbitrary sites, a m@|) $|-|0\/t 0\/+ +0 |\/|y |-|0|\/|13z, or whatever.

But even that's not the point.

This is the point:

Your user is an idiot. If you trust them with any data, whether it is formatting, security, or tracking, you are an idiot too. Idiots using an idiot-designed system = failure. This is because at least one of your users is going to enter something wrong. It will be by accident or by design, but something wrong will be entered. When that happens, your design will let you take one of two actions:

1. Ignore the erroneous data.
2. Take it up your bus.

You should design so that Option 1 is what you always, always, choose. It's harder to implement, since you have to restrict ranges, validate users, and do a lot more work on your end. It's a lot easier to just say, "Hey, just make sure you only enter numbers. Oh, and don't enter too many or you'll overwrite the program." Otherwise, your bus is going to hurt all day, and nobody wants that.

That's the point. The guy who wrote that website didn't have the slightest idea that the Internet has malicious users, and the managers didn't CARE that their website would be arbitrarily destroyed by spiders, spam crawlers, Lynx users, and cookie haters. Even after the site was destroyed, they didn't care enough to change it. Seriously. It's a complete CF.

"Hey, I left my keys in the ignition. Someone drove off with my car. Oh well, I'll get another one." 

"Hey, I left my keys in the ignition, again. Someone drove off with my car, again. Oh well, I'll get another one, again."

"Hey, I left my keys in the ignition, again. Someone drove off with my car, again. Oh well, I'll get another one, again." 

The problem is not car thieves. 

Anonymous:

It would be less amusing if they had purposely set up Google. The fact is that Google caused them material damage. I think this would be a valid suit. Lets take a more mundane example. You remove the rear bumper of truck and replace it with an attractive facia. Someone rear-ends you and does much more damage than if you had a bumper present. They are still responsible for the damage even though you did something dumb.

 I think that there is a key flaw in you anology.  It is illegal in most cases to read-end someone.  Now say that you take out the bulbs of your tail lights and your headlights and decide to drive your car around at night.  You come to a sudden stop in the midedle of the road because you just remembered something that you wanted to write down on your grocery list.  Now the Google truck behind you doesn't realize that you're coming to a sudden stop because like a total dumbass you are driving your car without lights at night and your stop lights are disabled.  Google would actually have a good case to sue you for causing the accident.  But since Google wasn't damaged in the original scenario that analogy fails as well.

 How about this one.  I put a pan of grease on the stove in my house and turn the stove on high.  I then tie a one end of a string on to the pan and take the string out the front door and tie the other end onto a stake end the ground on the other side of the sidewalk from my house.  Google comes walking down the public sidewalk and doesn't notice the hard to see string and catches it on their foot pulling the pan of by now burning grease off the stove and onto my wood floor.  This starts a fire in my house costing several thousands of dollars of damage.  Now I want to sue Google for "causing" this disaster. 

This is one of my favorite wtf's evar.

themagni:

That's the point. The guy who wrote that website didn't have the slightest idea that the Internet has malicious users, and the managers didn't CARE that their website would be arbitrarily destroyed by spiders, spam crawlers, Lynx users, and cookie haters. Even after the site was destroyed, they didn't care enough to change it. Seriously. It's a complete CF.

"Hey, I left my keys in the ignition. Someone drove off with my car. Oh well, I'll get another one." 

"Hey, I left my keys in the ignition, again. Someone drove off with my car, again. Oh well, I'll get another one, again."

"Hey, I left my keys in the ignition, again. Someone drove off with my car, again. Oh well, I'll get another one, again." 

The problem is not car thieves. 

I don't think this is an accurate analogy. I think it's more along these lines:

"Hey, I left my keys in the ignition, my doors unlocked and the car running. Someone drove off with my car. Oh well, I'll get another one." 

"Hey, I left my keys in the ignition, my doors unlocked and the car running, again. Someone drove off with my car, again. Oh well, I'll get another one, again."

"Hey, I left my keys in the ignition, my doors unlocked and the car running, again. Someone drove off with my car, again. Oh well, I'll get another one, again." 

Pretty close though.

--doc0tis 

Gollum: Web-Masssster is falsssse and trickssssey! Filthy javassssscriptsssesss.
Smeagol: No, Maaaahster takes caaaaaare of us, and giiiiives us cooookies.
Gollum: We will remove preciousssss content from Massster's Webssssite. Then we will haves preciousssss content for ourssselvessss.
Smeagol: Noooo, it is tooooo riiiisky.
Gollum: We could let HER do it! She will remove precioussss content from webssssite, and ssssince she doessss not want precioussss, then we will haves our birthday pressssent to oursssselvessss.
Smeagol: Yeeees, feeeed websiiiiite to the spiiiiider!

OneFactor:

Gollum: Web-Masssster is falsssse and trickssssey! Filthy javassssscriptsssesss.
Smeagol: No, Maaaahster takes caaaaaare of us, and giiiiives us cooookies.
Gollum: We will remove preciousssss content from Massster's Webssssite. Then we will haves preciousssss content for ourssselvessss.
Smeagol: Noooo, it is tooooo riiiisky.
Gollum: We could let HER do it! She will remove precioussss content from webssssite, and ssssince she doessss not want precioussss, then we will haves our birthday pressssent to oursssselvessss.
Smeagol: Yeeees, feeeed websiiiiite to the spiiiiider!

Wow. Funniest thing I've seen all week!
 

Anonymous:

It would be less amusing if they had purposely set up Google. The fact is that Google caused them material damage. I think this would be a valid suit. Lets take a more mundane example. You remove the rear bumper of truck and replace it with an attractive facia. Someone rear-ends you and does much more damage than if you had a bumper present. They are still responsible for the damage even though you did something dumb.

Yeah, good luck with that. This is like storing your valuables in your front yard with a luggage cart next to them. Sure, people aren't supposed to steal, but you haven't taken the least precaution to protect your stuff.

oops!

c:\del *.* -r -s

< captcha: why not?? />

themagni:

Your user is an idiot. If you trust them with any data, whether it is formatting, security, or tracking, you are an idiot too. Idiots using an idiot-designed system = failure.

Sounds like Wikipedia. :-P
 

themagni:

Your user is an idiot. If you trust them with any data, whether it is formatting, security, or tracking, you are an idiot too. Idiots using an idiot-designed system = failure. .....

The guy who wrote that website didn't have the slightest idea that the Internet has malicious users, and the managers didn't CARE that their website would be arbitrarily destroyed by spiders, spam crawlers, Lynx users, and cookie haters. Even after the site was destroyed, they didn't care enough to change it. Seriously. It's a complete CF.

Words... I can see you feel strongly about this. I agree. This was not a malicious user attack. This was a technology attack. A backdoor (programmer's short-cut) was left exposed, and a technology found it. Imagine that.

The WTF is that the short-cut was a link or a web page itself. How lazy is that?.  - "let's build a web site with a Start Over button on it". Probably got cached. There was probably no worthy server side challenge, so the obient server did it's job.
 
At first I was thinking that your response was a bit over the top (perhaps you needed a nap), But the more I digest it, the more I'm inclinded to think - the idiot was in fact just that.

Perhaps this is simply technology's method of natural selection.

tiro:

You should also probably not implement your entire security layer on the client side and not use GET to perform horribly, horribly non-idempotent operations

I hope you're not suggesting that GooogleBot can't do POST, since... believe it or not, it can.

Of course, initially you're totally right - the client should never be trusted. Not on the Internet, and in the "real life" - even less so :)

utu:

I hope you're not suggesting that GooogleBot can't do POST, since... believe it or not, it can.

Of course, initially you're totally right - the client should never be trusted. Not on the Internet, and in the "real life" - even less so :)

 IIRC, GoogleBot only follows hyperlinks but does not press buttons, and since POST is generally done with buttons in forms, Google won't delete the internet.

OneFactor:

Gollum: Web-Masssster is falsssse and trickssssey! Filthy javassssscriptsssesss.
Smeagol: No, Maaaahster takes caaaaaare of us, and giiiiives us cooookies.
Gollum: We will remove preciousssss content from Massster's Webssssite. Then we will haves preciousssss content for ourssselvessss.
Smeagol: Noooo, it is tooooo riiiisky.
Gollum: We could let HER do it! She will remove precioussss content from webssssite, and ssssince she doessss not want precioussss, then we will haves our birthday pressssent to oursssselvessss.
Smeagol: Yeeees, feeeed websiiiiite to the spiiiiider!

 ROFLMAO! ROFLMAO! ROFLMAO! ...
 

I think that's what has caused the issue here - with the website implementing change functionality (which included
the option to delete pages altogether unfortunately) as part of GET requests which in turn led to the eventual
disaster when the google master attempted an evening walk ;)

themagni:

Your user is an idiot. If you trust them with any data, whether it is formatting, security, or tracking, you are an idiot too. Idiots using an idiot-designed system = failure. This is because at least one of your users is going to enter something wrong. It will be by accident or by design, but something wrong will be entered. When that happens, your design will let you take one of two actions:

1. Ignore the erroneous data.
2. Take it up your bus.

I suppose Alex won't be amused that my first thought on reading this, is that it applies so well to tdwtf and community server. :p
 

 

I hope you're not suggesting that GooogleBot can't do POST, since... believe it or not, it can.

I'll call it. Examples?

There is no way any non-malicious spider uses POST, precisely to avoid this happening to sites that are coded *properly*. Only spambots use POST. 

Anonymous:

I think that's what has caused the issue here - with the website implementing change functionality (which included
the option to delete pages altogether unfortunately) as part of GET requests which in turn led to the eventual
disaster when the google master attempted an evening walk ;)

No, the huge problem here is that the site by default allows all actions, unless the person has been identified as not allowed, and they ask the untrusted client to remember that fact for them!  That's the WTF.  It's like a bank allowing anyone to walk into the vault, unless they have a name tag that says, "Not an Employee".  Sure, the guard at the front door (initial login page) is diligent about handing out name tags, but robbers are free to throw the tags away, or enter the building via an open window, or back door, etc...  Anyone who doesn't wear name tags due to religious beliefs (cookies disabled) also will not have such a name tag, and is free to roam the bank.

Anonymous:

OneFactor:

Gollum: Web-Masssster is falsssse and trickssssey! Filthy javassssscriptsssesss.
Smeagol: No, Maaaahster takes caaaaaare of us, and giiiiives us cooookies.
Gollum: We will remove preciousssss content from Massster's Webssssite. Then we will haves preciousssss content for ourssselvessss.
Smeagol: Noooo, it is tooooo riiiisky.
Gollum: We could let HER do it! She will remove precioussss content from webssssite, and ssssince she doessss not want precioussss, then we will haves our birthday pressssent to oursssselvessss.
Smeagol: Yeeees, feeeed websiiiiite to the spiiiiider!

Wow. Funniest thing I've seen all week!
 

X2 mate! That was bloody hillarious. I almost spat out my breakfast as I was reading it. 

One more WTF: Why weren't there more current backups?

What kind of retarded moron does MONTHS of work without backing it up, especially before going live?

I have backups of my backups backed up, thank you. It'd take annhilation of four major US cities (or a very, very carefully coordinated "surgical strike") to leave me with backups more than a day or two old...
 

tiro:

You should also probably not implement your entire security layer on the client side and not use GET to perform horribly, horribly non-idempotent operations.

+1! Almost.

What you really mean is that GET shouldn't be used for destructive operations. Deletes are idempotent but unsafe.

AWKScooby:

Anonymous:

I think that's what has caused the issue here - with the website implementing change functionality (which included the option to delete pages altogether unfortunately) as part of GET requests which in turn led to the eventual disaster when the google master attempted an evening walk ;)

No, the huge problem here is that the site by default allows all actions, unless the person has been identified as not allowed, and they ask the untrusted client to remember that fact for them!  That's the WTF.  It's like a bank allowing anyone to walk into the vault, unless they have a name tag that says, "Not an Employee".  Sure, the guard at the front door (initial login page) is diligent about handing out name tags, but robbers are free to throw the tags away, or enter the building via an open window, or back door, etc...  Anyone who doesn't wear name tags due to religious beliefs (cookies disabled) also will not have such a name tag, and is free to roam the bank.

'Jason':

The fact is that Google caused them material damage. I think this would be a valid suit.

You've just given out legal advice, 'Jason'. I do hope you're a lawyer.

Of course, since you're building an argument from an analogy with an obvious flaw, and not supporting your legal advice with case or statutory support, you're clearly not a very good lawyer.

gwenhwyfaer:

'Jason':

The fact is that Google caused them material damage. I think this would be a valid suit.

You've just given out legal advice, 'Jason'. I do hope you're a lawyer.

Of course, since you're building an argument from an analogy with an obvious flaw, and not supporting your legal advice with case or statutory support, you're clearly not a very good lawyer.

You might be a Lawyer? The fact remains. The bot did the damage (as proven).  Where else, but the courts should this be argued? 

If my robotic lawn-mower ( on it's own ) somehow finds it's way into my neighbors fence-less garden, I'm safe... right? It's a robot. Or... am I responsible for it's actions?

gwenhwyfaer:

'Jason':

The fact is that Google caused them material damage. I think this would be a valid suit.

You've just given out legal advice, 'Jason'. I do hope you're a lawyer.

Of course, since you're building an argument from an analogy with an obvious flaw, and not supporting your legal advice with case or statutory support, you're clearly not a very good lawyer.

Of course he's not a lawyer!  Since when does a lawyer give out legal advice for free?