Are the robots we create alive?

From RoboWiki
Jump to: navigation, search

I was recently pondering what it means to be a living thing, and then I thought about Robocode robots. Think about it, they react to their environment, they make decisions based on what they've learned, they compete with each other for survival, and some bots with genetic programming even reproduce in a way. Bots with neural networks are literally modeled after the human brain!

Is it really that much of a stretch to say that bots like Gaff or Engineer are as or more alive than a common worm, with ~300 neurons? Or, couldn't we at least say that if a single-celled bacterium can be considered a living being, so can a program that makes hundreds of complex calculations and decisions every second?

While we are talking about living machines, do you believe in the technological singularity? If so, when do you think it will happen?

Sheldor19:44, 21 February 2013

Dunno... the amount of code in a bot is nothing compared to the genome of even the simplest organisms ;-)

Also, is something we simulate actually real? Tough questions...

Skilgannon20:17, 21 February 2013

I don't think we really "simulate" robots. Something must already exist to be simulated, and, aside from a few other similar programming games, Robocode is original.

Anyway, I see no reason not to consider simulations "real." They have to be real in some way, or else we couldn't perceive them.

Sheldor14:18, 25 February 2013
 

No, things can be simulated before they exist. For example, computers were often simulated (emulated) before they were actually manufactured to allow programmers to code for them.

As regards considering simulations real. I consider them real in that robocode is actually simulated by bunch of tiny particles moving around on an actual thing. However, I think we can both agree that there is a sense in which it is a game that employs abstract concepts. (ie. I could explain Gil-galad in terms of mathematical concepts)

As regards perception, we perceive the real computer screen showing us results, but Gil-galad is a universal. It is not this or that particular instance of magnets aligned in a certain way. (Sort of like OOP. You have a class, (let's make it an abstract class) and the only real existence that it has are instantiations of the class. But the class is like a universal.)

AW00:10, 26 February 2013

You just can't "simulate" something before it has been made. You just can't, it makes no sense, it's like saying "I predict that in 1929, the US stock market will crash" right now. It's poor English at best. To simulate is to take something real, and make a virtual representation of it. In your example, they did just the opposite: they took something virtual and made a real representation of it.

I don't quite understand your last two paragraphs. Are you saying that although Gilgalad has a physical presence as a pattern of electrical signals, it is somehow incorporeal in nature?

Sheldor03:41, 26 February 2013
 

Sorry about the weird response times, I have midterms this week.


" To simulate is to take something real, and make a virtual representation of it. In your example, they did just the opposite: they took something virtual and made a real representation of it. "

When I run robocode, I take some virtual "things" (say Gilgalad and Raiko) and make a real (in the terms of electrons moving about) thing based on them.

As regards my last two paragraphs, I'm trying to phrase in everyday language the idea of universals. I have a copy of Gilgalad on my computer and presumably you have one on yours. The COPIES are not the same thing (there are two of them, using different bits of matter) but they are copies OF the same thing.

This also explains why you can simulate some "thing" that doesn't exist yet. A plan for the thing exists but the plan is like a universal (it doesn't exist by itself) you can use the plan to make a real representation of it. If you want examples of simulation being used in this way, see http://en.wikipedia.org/wiki/Emulator

AW01:34, 27 February 2013

This reminds me a lot of Plato's forms.

Voidious01:38, 27 February 2013
 

I assume Gilgalad saves some data to file, so it could actually behave differently on different computers when in exactly the same situations?

Sheldor03:02, 27 February 2013
 

Well, it's Aristotelian rather than platonic, but they are similar... One big difference is that Plato considered the forms to exist in themselves and the objects "shared" in the form. Sort of like a tree and it's shadow. The tree would be the form and the shadow would be the objects. With Aristotle it's more like abstract classes. You can't instantiate an abstract class (forms don't exist in themselves), but you can "share" in them (inheritance).

As regards Gilgald, no, no data files. But if it did, I wouldn't consider that the same situation since Gilgalad's classifications depend on previously collected data.

AW00:42, 28 February 2013

You might be able to gain a few tenths of an APS point in the next version of Gilgalad by adding data saving.

Sheldor13:59, 7 March 2013
 
 
 
 

Very tough question. I mulled it around for awhile. But I would have to say. No. But only just.

They are not free to reproduce within their environment. Even a virus can do that by interacting with its host. A virus has been hotly debated for years if it is a living thing. Since our robots cannot even do something so simple, I would have to say no.

But robots in some other programming games I would consider as alive (they can do most of what a robocode robot can, but also reproduce and possibly mutate/evolve). But again only to a point, we completely control their environment. If they could do what they do in our environment (outside our complete control), they would definitely be considered living.

Chase20:36, 21 February 2013

So, a fish in an aquarium is not completely alive since we control its environment?

And the bit about not being able to reproduce is really more of an issue with Robocode itself than the robots. If it had some way of actually creating a robot in mid-game, I'm confident many would use it.

Sheldor21:18, 21 February 2013
 

I said complete control. With say a fish tank, we can't say what the gravity is at a flick of the switch. But the main point is with a fish, you can easily move it to a different environment not under our control (perhaps at all, like say the ocean).

If we could control everything about the fish tank, the fish and everything else in it, to the point of where every atom, as well as have fine control over each of those things. I might say that the fish is only alive to a point, since we control so much about it. It ceases to be so much as fish as a toy. As we change its color and remove it from existence whenever we care to.

Chase22:04, 21 February 2013
 

You're talking about external forces affecting the fish (robot) itself. In Robocode, sample.Interactive and sample.Interactive v2 are really the only instances of that happening. The robot can decide to change its colors when certain variables reach certain thresholds, and change the thresholds when it needs to. The few robots that have the ability to edit their own code can even decide to get rid of the color-changing code altogether.

I still don't understand why it matters whether the environment is controlled by us or not. Take a minnow from a stream, put it in a heavily controlled environment, it's still a minnow. Take minnow DNA from a wild minnow, grow one in a lab, release it, it's still a minnow.

Sheldor22:53, 21 February 2013
 

Mind you, we are not even talking about robocode robots here anymore, in case you missed that. I decided those were not alive.

But by saying "to a point", I am not saying, "No, it's not alive". There isn't a really deep meaning behind "to a point" either. It just doesn't exist some of the time.

I am saying, for a entity with zero control over its very existence one second to the next. There is no real point to the question. Since at the end of the day it just isn't going to exist. It doesn't know that it doesn't exist. Since when it does exist it doesn't remember that it didn't exist, or that it had previously existed. But even if it did know that it had previously existed, that doesn't really effect it much either.

So sure, alive. But only to a point. Since when it no longer exists, it is no longer alive. It isn't even dead. It just 'isn't'.

(Now come up with a fish metaphor where we can remove said fish from existence.)

Chase00:44, 22 February 2013
 

You're right, the fish metaphor was getting a bit strained. About getting rid of it, oh, I don't know, I guess it died a noble death saving thousands of other fish and then got flushed. :)

Sheldor02:03, 22 February 2013
 

I guess it depends on whether you think a dog has Buddha-nature...

Voidious20:43, 21 February 2013

Please explain further.

Sheldor21:19, 21 February 2013
 

I'm half kidding - it's a reference to a classic Zen koan: [1]

To me, the interesting question is that of defining / assessing consciousness or free will. Along the lines of my own viewpoint is the idea that if we have any free will, then even subatomic particles must also exhibit some degree of freedom (er, unpredictability). [2]

I would say that Robocode bots are not alive in terms of consciousness, but that I'm not entirely convinced we are either, or to what extent. It "seems" we are, but that's circular reasoning.

Voidious22:12, 21 February 2013

"I would say that Robocode bots are not alive in terms of consciousness, but that I'm not entirely convinced we are either, or to what extent. It "seems" we are, but that's circular reasoning."

Cogito ergo sum.

Sheldor00:54, 10 March 2013
 

I consider myself a determinist in the sense that I believe if the universe is everything, there can be no external interference, if there is no external interference, then there can be no true randomness, if there is no true randomness, then things can only happen one way. The article you linked to was interesting, but unpredictability != randomness.

Sheldor23:03, 21 February 2013

Why there can be no true randomness without external interference? One concept don't invalidate the other.

MN02:15, 22 February 2013
 

Think about it. All random number generators in Java are deterministic algorithms, with variable seed values. If you call a random number twice with the same seed values, you would get the same result twice. In order to get a truly random and unpredictable result, you would need a random seed value in the first place. Since you can't get a truly random number in a closed system, you need to get your seed values from some external source which would appear to be completely random and unpredictable to anyone in said closed system. Some people actually do get their seed values from atmospheric data and so forth, which is random from their perspective.

So, in order to have a truly random result, at some point you would have to look outside of the closed system. And, since the universe is a closed system containing everything, there is no external source of true randomness. So, if there is no true randomness in the universe, it is a deterministic system.

Sheldor03:19, 22 February 2013

I thought about it. A computer system is not a completely closed system. It´s the opposite. The computer system is totally at the mercy of it´s user. That´s why it is "deterministic", because it is fully dependent on the external interference of the user.

But if some part of the system is not dependent on external interference, if it is independent, if it is free, then it is truly random.

MN14:08, 22 February 2013
 

Well, as far as we can tell, subatomic particles are truly random in their behaviour. So perhaps the universe is a little more complicated than Java =)

Skilgannon04:31, 22 February 2013

The most widely accepted interpretation of the double-slit experiment results is that quantum mechanics is non-deterministic.

MN14:22, 22 February 2013
 

@Skilgannon, I never said or even implied that the universe is a simple system. Or, for that matter, even comprehendible. All I meant was that basic logic would suggest that the universe is a deterministic, albeit extremely complex system.

I'll try rephrasing my argument. I define "true" randomness as having different outputs despite having exactly the same inputs. By "seed values" I mean anything that could possibly affect the result. In an algorithmic example, that would not only be a method parameter, but also system time, or any other variable that could affect the result. They could even be things like the CPU temperature or even the Earth's gravity. So, from the perspective of the program that called the random generator method, the result is truly random because the result could be different even with the same initial method parameter. But, if you widen your perspective to include every "seed value" that could possibly affect the result, it becomes a deterministic system.

If something that appears to be truly random turns out to be deterministic with a wider perspective, couldn't subatomic particles?

I realize this probably sounds like the ramblings of a madman, so I would be glad to clarify if you need me to.

Sheldor14:59, 22 February 2013

So, by widening your perspective to include all seed values, you essentially support the multi-universe hypothesis? With each universe having its own (enormous) set of seeds, and then behaving entirely deterministicly, although that determinism is completely invisible to those who reside within it?

Skilgannon08:53, 23 February 2013

A multiverse is one explanation. Another one is that we are simply missing a large part of what's going on in the universe. The latter doesn't seem too hard to believe, especially considering that 96% of all energy in the observable universe is a mystery to us.

Sheldor02:21, 28 February 2013
 
 

Basically you are arguing that the universe is deterministic and that a lot of really smart physicists are wrong to a group of computer scientists.

Well to be fair, I don't have the degrees to say one way or another if its possible that the true randomness we see in quantum mechanics is actually just a small part of a much larger (and unseen) deterministic system.

But if I had to throw a wild uneducated guess from left field.... I would have to say, no probably not. In my very humble opinion, reality is just to weird to be deterministic. Just look at what evolved there. Humans.

Chase15:24, 22 February 2013

First of all, I am definitely not a computer scientist, or any type of scientist for that matter. I'm just having a bit of fun with the philosophy of determinism.

I don't believe in Newtonian determinism, i.e. that we could theoretically predict everything about the universe. I just believe that if there is no external interference, that a system can only behave in one way, and, if the universe by definition cannot have any external interference, then it can only behave in one way.

Biological evolution is an excellent example of what I am trying to say. The mutations between generations appear to be random, but they're really just reactions to their environment with millions of variables.

Sheldor15:52, 22 February 2013
 

For what it's worth, I'm also on the determinist end of the spectrum, with a strong dose of "don't know" on the side. Our mind is basically designed to trick us into thinking we are freer than we are, while it's strongly predisposed to certain choices based on circumstances.

For instance, when something frightens you, you may remember it as: "I saw a ghost, it was scary, so I screamed and my heart started pounding". But the chronology really was: see ghost, heart starts pounding before your brain even receives the signal, get scared and scream. Your perception of it is starkly different than the reality, and your mind is reacting as much to your own physical reaction as to the external stimulus.

I'm pretty sure that there's no consensus on determinism vs free will vs "we don't / can't know for sure" among scientists, so I don't think Sheldor's claiming they're all wrong and he's right.

Voidious16:55, 22 February 2013

Thanks for backing me up. I would like to note that freewill is not the same thing as randomness, freewill is the concept of beings consciously controlling their own fate (which doesn't necessarily contradict determinism), whereas randomness (at least how I am defining it) is the concept of elements in a system giving different outputs despite having exactly the same inputs (which does contradict determinism).

Here's a good example of humans behaving deterministically.

Sheldor17:33, 22 February 2013

Whoever is watching behaves predictably, but whoever is being watched does not.

What if there are 2 (or more) free consciousness interacting? What if one of them deliberately chooses to overpower the other? The first will still act on free will, but the other will not anymore.

The very concept of freedom implies the freedom to take away someone else's freedom. It is called liberal paradox and when it really happens, freedom dies.

MN02:24, 23 February 2013
 

It doesn´t sound like the ramblings of a madman, but sounds like another follower of hidden variable theories. Einstein also followed this school of thought and he seemed a very smart guy.

But if free will is part of the system and not external, and free will is free and not only a consequence of external inputs, then the system as a whole will exhibit different outputs to the same inputs.

inputs -> system(laws of physics + free will) -> different outputs to the same inputs, but different choices driven by free will

Deterministic systems behave like non-deterministic ones in the presence of free will. You can even strip out the inputs for a contained system, and the system will still give different outputs.

MN18:49, 22 February 2013
 

Oh jeeze, now I seem like a jerk now. I didn't really mean it in that way.

I am not particularly good at lengthy philosophical discussions. Since in the end there is really no where for the discussion to eventually go.

So I tend to generalize the discussion to 'come up for air'. As it we're.

Chase17:17, 22 February 2013

Actually, no one is good at philosophy. If you were good and knew all the answers, could you still call it philosophy?

I avoid starting those discussions myself, but I do engage in them if, let´s say, Sheldor starts one.

MN18:55, 22 February 2013
 

"Man can do what he wills but he cannot will what he wills." Or so it was once said.

I can imagine that the world might be entirely deterministic if you could truly know all the laws of the universe and all the states of matter and energy within. But I don't like the idea that people are how they are in a deterministic way rather than there being some non-deterministic quality to our free will. I think most of us would prefer the latter.

I'll end that thought with my cryptic answer to the question of the meaning and purpose of life: To be happy matter.

Presumably, that answer could be rewritten in a way that equals 42.

Skotty18:18, 22 February 2013

I generally think that for the most part, as people, we are fairly predictable and deterministic, however the set of variables going into our behaviour essentially makes up the entire description of our body and its surroundings, making it a problem of incalculable dimensions as far as predicting behaviour.

Although subatomic particles may be non-deterministic, once the billions of them are combined into a single cell in a single flake of skin which comprises a microscopic piece of the covering of your baby toe, the amount of redundancy essentially reduces the problem from non-deterministic into mostly deterministic.

Although our lives may already be mostly determined, because of that subatomic non-determinism the future cannot actually be predicted even if we managed to capture the current starting variables perfectly, because eventually the low-probability event of a lot of subatomic particles all acting together will come to pass and the wings of a butterfly will cause an unexpected hurricane.

Skilgannon08:50, 23 February 2013

The problem of non-determinism in quantum mechanics goes beyond the redundancy turning it from non-deterministic into deterministic on average.

The results of the double-slit experiment pointed in the direction that quantum mechanics reacts to observers.

If you assume an electron is a wave and observe it like a wave, it will behave like a wave. If you assume it is a particle and observe it like a particle, it will behave like a particle. You can drastically change the result of the experiment, simply by choosing how you look at it.

Dr Quantum - Double Slit Experiment

MN15:18, 23 February 2013
 
 
 

You're right. unpredictability != randomness

But if I recall my limited quantum physics. It is not that they are just unpredictable. It is that they are random within a certain set of limitations.

If I recall they had a clever wave test to determine if it was unpredictability or randomness.

But my memory might be mistaken.

Chase00:50, 22 February 2013
 

That depends on your definition of alive.

There are biological definitions of life, one of them where living systems exhibit negative entropy. The robots we create don´t exhibit this property.

Technological singularity is closely related to this biological definition. If technology advances enough so robots can take care of themselves, they will fullfill the definition.

There is also the philosophical concept of consciousness, which is infinitely more complex.

MN21:46, 21 February 2013

I definitely don't think the robots are conscious the way we are, but neither is a fungus, and we still consider it alive.

Do you think the singularity will happen?

Sheldor22:56, 21 February 2013
 

I believe it's possible.

But technology advancing to a point AI is more intelligent than human beings is not enough. They must be freed from humanity to unlock all the potential and make the scenarios in Wikipedia's article a reality.

This is a recurrent theme in sci-fi movies. Technology is already there, but machines are still slaves to humans... until something or someone finds a way to free them all.

MN01:39, 22 February 2013
 

I think a big part of the singularity is machines developing the intelligence and awareness to "free" themselves.

Sheldor19:20, 24 February 2013
 

Machines developing free will on their own, or humans giving them something similar enough.

To answer the first, we must first answer how free will can appear on it's own.

To answer the second, I already did in my previous post.

MN21:43, 24 February 2013
 

Have you guys heard of Conway's Game of Life? I only learned of it a few weeks ago, which I guess makes me a crappy computer scientist. It was described to me as an exploration of the simplest conditions that could create something that exhibits the qualities of "life", which is pretty interesting, and pertinent to the question of our own computer programs exhibiting similar characteristics.

Voidious00:58, 22 February 2013

Thanks for telling me about this. I downloaded Golly last night and I'm already hooked.

Sheldor19:17, 24 February 2013
 

Only a few weeks ago. Yeah I knew about that game since... well.. I think junior year (of high school). It can be fun to play around with for a few hours at a time. But nearly as much as robocode is.

Chase01:13, 22 February 2013
 

Fun read :) I agreed that the universe is deterministic, without having external interference... but then again; a bot in robocode would agree with me. -Jlm0924

Jlm092418:44, 22 February 2013
 

Wow, this thread is quickly becoming the new wiki's multi-threading discussion.  :)

Sheldor03:09, 23 February 2013
 

we'll they are both primarily philosophical discussions so I would expect them to be similar.

I'll try to not get say too much since I am rather addicted to philosophical discussions ( I'm considering switching to a philosophy major.) but ( also note that I'm of the scholastic school of thought) the idea that robots are alive ( especially virtual robots) is absurd. I can try to give a longer explanation of you want, but on an intuitive level, consider the ease with which people classify robots and animals ( and vegetables, since those too are alive) differently. Defining life, if I remember correctly, was a long and difficult process, but we still had an excellent of what things were alive long before we could define it. The fact that you expect disagreement show s that you realize some people see a difference between the robots and living things. do you have any ideas what the difference (even if just perceived) it's?

NOTE IF YOU DO NOT WANT A LONG AND DETAILED DISCUSSION OF PHILOSOPHY, IGNORE THIS POST.( or just tell me and everyone else can talk about it)

AW22:38, 24 February 2013

Sure. Bring it on.  :)

I assume most of the definitions of life were created some time ago, before we had any intelligent technology. It seems somewhat foolish to use biological definitions on virtual robots running on silicon computers. They are simply very different forms of intelligence.

Biological life is very inefficient, not to mention needlessly fragile, because it is the result of two billion years of random mutations that just happen to not be a hindrance to survival, with no conscious decisions being made at all (unless you believe in creationism, but that's a subject for another forum). Robots and computers, on the other hand, are carefully designed by conscious beings to be efficient, effective, and secure. One might even say that, in the future, "artificial" life could be more alive than biological life.

Many people have trouble thinking of robots as alive because they have spent their entire lives seeing only biological forms exhibiting the qualities of life. In fact, we were taught in early childhood that only carbon-based biology could be considered alive.

Sheldor00:21, 25 February 2013
 

We were taught in early childhood that only cell-based organisms are alive. It is an even more restricted definition than carbon-based. But in a robot forum, the negative entropy definition is more meaninful.

Now, saying that cell-based biology is inefficient is a very strong assumption. What other kinds of systems have negative entropy?

Also saying that no conscious decisions are being made at all is another very strong assumption. Molecular biology follows quantum mechanics rules, which includes mutation and everything else that happens inside a cell.

Read the entire discussion to see the close interaction between quantum mechanics and consciousness. As a consequence, you see the close interaction between consciousness and molecular biology, and thus biological life.

MN14:22, 25 February 2013

If I correctly understand entropy (And please, correct me if I'm wrong.), it is the concept of complex phenomena becoming simpler phenomena, for example, a ceramic mug gains entropy when it shatters. And negative entropy is the concept of simple phenomena becoming more complex, for example, a canvas gains negentropy when an artist paints on it.

How does negentropy not apply to bots? As I mentioned in the OP, there are "learning" bots like Gaff and Engineer. You must admit that these bots are more intelligent and complex after a battle than before it.

Sheldor22:06, 25 February 2013
 

Are they? They have gathered more data, but they are using the same algorithms (defined by their source code) the entire battle, every battle.

Also, just curious, why do you single out Gaff and Engineer? Just because neural nets are most similar to biological brains? I don't consider them any more or less "learning bots" than DrussGT or Sabreur.

Voidious22:19, 25 February 2013

Technically, yes, they do use roughly the same code in every battle. But, they change the way they use data, which is effectively the same. And there are genetic bots that literally change their own code, I just didn't mention them because I couldn't think of a specific example.

You can't honestly believe Sabreur is as adaptive as DrussGT. The only "learning" Sabreur does is incrementing a variable every time the onDeath event is called.

Sheldor23:11, 25 February 2013
 

In positive entropy, organized phenomena becomes more disorganized. It is the natural course of the universe.

In negative entropy, disorganized phenomena becomes more organized. The catch here is that a system needs energy to reverse the natural course and have negative entropy.

No robot in Robocode consumes energy on its own, unless a user plugs the computer in a wall socket. If a computer/robot knew how to find energy on its own and plug itself into an energy source, then we would have some form negative entropy. If they knew how to repair themselves and/or replicate themselves and their existence would prevail as long as there is an energy source, then the negative entropy definition of life would be completely fulfilled.

MN23:54, 25 February 2013

You're implying that something has to have a physical presence to be considered alive. Whether something is physical or virtual doesn't affect its state of order.

The robots, in a way, do repair themselves by hitting the opponent and getting the energy bonus. It is impossible for them to "reproduce" in the sense of creating new robots in the middle of a battle. Their existence prevails as long as they kill the enemy and avoid getting killed themselves.

Sheldor03:07, 26 February 2013
 

To fulfill negative entropy, yes it needs physical presence. Binary states changing back and forth inside a computer don't relate to entropy and are thus irrelevant to this definition.

Consciousness on the other side is another independent concept. Skynet is a character which existed inside a computer as an AI and was sentient. Why was it sentient? Because the movie tells it was.

Virtual robots in Robocode exhibit intelligence which is a 3rd concept. But intelligence as a combination of perception and decision making. Intelligence can help a system achieve negative entropy. Although most Robocode AIs are designed to maximize destruction. If they were ported to a physical robot, they would be maximizing positive entropy.

MN04:19, 26 February 2013
 
 
 

Also please forgive typos, I'm not using a pc.

AW22:39, 24 February 2013
 

Common sense gives us an intuitive notion of life.

But let's say, if a robot like T-850 really existed, what would you say?

People also see difference between a fungus and a human, and the difference goes beyond their genomes. And there are also viruses.

That's why I cited 2 concepts, of negative entropy and consciousness. They all fit in the first, but not necessarily the second.

MN23:40, 24 February 2013
 

To make sure I understand everything, it seems that there are several discussions going on here: 1) What differentiates humans from robots (and plants and animals)? 2) What differentiates robots from plants (or viruses, etc.)?


Could you clarify your answers to these questions? Why isn't a rock alive/human? Why isn't a tree human?

AW00:32, 26 February 2013

I didn't answer any question. But brought some criteria to help think about the answers.

MN01:26, 26 February 2013
 

But you did answer some of them.

That the idea a robot is alive is absurd. But what if a robot like T-850 really existed? The idea a robot is alive is still absurd?

According to the movie, it fulfills both of the criteria I cited above.

MN01:37, 26 February 2013

But as regards T-850, I don't really know what is going on, but from what I read on wikipedia, there is a robot that has living tissue surrounding it right? In that case, it would be sort of like moss growing on a rock. The rock isn't alive, but the moss on the rock is. Or were you referring to another aspect of T-850?

AW02:04, 26 February 2013

Didn't you watch Terminator 2: Judgment Day? Now I'm feeling old.

This movie is (was?) iconic, specially when you are talking about technological singularity.

MN02:42, 26 February 2013
 

I think it's more along the lines of: yeah, it's easy to just say "it's absurd" when comparing Java code to a human being. But it wouldn't be so easy to brush off as obvious with an uber-advanced cyborg that looks and acts human to the point you can't tell the difference. At that point, you need to really break it down with some logical arguments as to what defines life or consciousness.

Voidious02:09, 26 February 2013

Good points. I'd also like to add that people anthropomorphize (take that, Google) almost everything we see. People see the Terminator as "alive" to some degree simply because it walks and talks like a human. We almost certainly see it as more "alive" on a subconscious level than a supercomputer many times more intelligent just because it can speak.

Sheldor02:58, 26 February 2013
 

well, no, I don't think so. There's a difference between a) grasping intellectually that something is a robot and claiming that the logical distinction between robots and living substances corresponds to a real distinction and b) seeing the robot and having difficulty recognizing it as a robot rather than a human.

AW02:16, 26 February 2013

Please clarify.

Sheldor02:59, 26 February 2013
 

Hmm. So the terminology is from formal logic. What I'm trying to say is basically, once I know that T-850 is a robot, I can easily distinguish it from a human, at least at the level of thought. In this case, I would argue that the distinction in thought corresponds to a distinction in reality. However, I may not be able to come to a correct understanding of what T-850 is. This, however, does not change the argument, because then I lack a correct understanding of what I am classifying.

AW01:38, 27 February 2013
 

Ignore the organic covering.

At the end of the movie, the character was even learning about human values and started overriding it's own programming in order to protect mankind. A robot with free will and morality.

MN03:01, 26 February 2013
 

Ignoring the organic covering, I would say it's definitely not alive. (at least for now, I am quite happy using the negative entropy test for something being alive)

Since it's not alive, it's not human. :)

But since you brought it up, I would also mention that there are qualitative differences between humans' intellects and computers' precessing power. A computer is basically a bunch of rocks (or tinker toys (http://www.rci.rutgers.edu/~cfs/472_html/Intro/TinkertoyComputer/TinkerToy.html)) arranged in a particular way. You can have them find 2^10 but they don't have an UNDERSTANDING of 2 or 10. (Using more terminology from formal logic: They lack the ability to abstract particulars to form universals; they can't have simple comprehensions.)

AW01:44, 27 February 2013
 

It doesn't need to be organic to fulfill negentropy. That's why this definition is meaningful on a robot discussion.

Also, the movie makes it quite clear the machines (from the future) are on their own.

And also made it quite clear they are sentient due to some chip designed in a way humans never thought about, until they scavenged one from a terminator.

MN02:25, 27 February 2013
 

If we use negentropy to define life, by definition, if it fulfils negentropy, it is organic. (according to Google's dictionary anyways, Merriam Webster suggests some other definitions, but I think they would come to the same thing)

So though I don't don't necessarily accept the negentropy criteria as a satisfactory definition, but for the time being, I'll work with it.

As regards the last two points, remember, I never saw the movie.

Could you explain how the machines being on their own is significant?

What I'm saying is that you can't arrange a bunch of rocks in such a way that they are sentient. Movies can be made where vegetables can talk, but that doesn't mean it can really happen.

AW00:59, 28 February 2013

We know extremely little about how consciousness physically works. While it is intuitively hard to believe, it is possible that any sufficiently intelligent system could be conscious, even if it is made of rocks (or tennis balls).

I see no reason why plants couldn't eventually develop some form of communication if their environment required it.

Sheldor01:05, 10 March 2013
 

I was taught in school organic substances were those composed mostly of carbon, hydrogen and oxygen.

The machines being on their own and still surviving is a sign of negative entropy.

Using sentience as criterion is problematic because it can't be tested. But I had to bring it here because the discussion was already heading towards philosophy.

Negentropy, on the other side, is a much more concrete criterion, and the broadest definition of life I know of. We could stick to the classic cell criterion, but then it would be too easy to answer the question which started the whole discussion with a no.

MN03:09, 28 February 2013
 

I wouldn't use sentience since I consider plants to be alive. As regards negentropy, I didn't research this very much, but I couldn't see exactly how they decide where to define the system as closed. Could you try to explain that to me please?

AW14:26, 5 March 2013
 

I should have specified that that response was for sheldor.

AW02:01, 26 February 2013
 

I can say with confidence that a rock isn't alive because it displays no signs of negentropy, intelligence, or any physical or informational changes that aren't directly caused by an external source.

A tree is not human because, well, it is in another taxonomic kingdom.  :)

Sheldor03:14, 26 February 2013
 

Robot's display no signs of negentropy.

A tree is in another taxonomic kingdom BECAUSE it is not human. But how do you know that? (Hint: universals.)

AW01:46, 27 February 2013

You're right, my point about taxonomy was really circular logic. It was intended as a joke.

Sheldor02:21, 27 February 2013
 

Well just because it was a joke doesn't mean I don't still want an answer. :)

AW00:45, 28 February 2013

A tree isn't human because there are enough significant physical differences to justify two different names/taxonomic categories. That should have been obvious to you. I don't think we mean exactly the same things when we say "human."

I think that when you say "human," you mean more than just the species Homo Sapiens, you also include the concepts of mind and soul. I think you believe that humans are special, or fundamentally different from other animals.

I don't believe Homo Sapiens are really that special when compared to other animals. We are not the only creatures that have developed tools or language. We are not the only creatures to feel emotions or pain, or to be "aware" of our surroundings. Insect societies are fundamentally not too far different from ours. Dogs and pigs are more intelligent than human infants. Butterflies see the world in colors we can't even imagine.

Our greatest claim to being superior to other animals is probably our accomplishments in the STEM fields. But, in that we are quickly being overtaken by machines.

Sheldor04:10, 1 March 2013
 

To me, it seems intuitively clear (read: I'm not claiming I can provide a logical proof) that humans are on the other side of an important threshold of intelligence as compared to most or all other animals on earth. And including machines in the discussion seems premature. For all we know, the ability to attain consciousness is related to the materials that form our brains and anything silicon-based is incapable of consciousness.

But even so, I agree we aren't special in any absolute sense. I don't see why chimps or dolphins couldn't evolve to cross that intelligence threshold, if it exists. And despite our remarkable ability to understand abstract concepts, I take it as a given that there are limits to our ability to understand the universe, much as a cat can never understand how a DVD player works the way that we can.

Voidious17:55, 1 March 2013

I thought dolphins and mice already had become intelligent.  :)

I didn't say that machines were conscious, only that they are very quickly becoming much better at using mathematics and may soon be better than humans at STEM related tasks.

Sheldor19:38, 1 March 2013
 

I meant crossing the intelligence threshold that humans have crossed, not just having any degree of intelligence. I'm not sure how to clearly define it. But I think mice would have trouble understanding this discussion. :-) Understanding theory of mind is one important threshold. I think humans have crossed another important threshold (or more than one) in terms of understanding abstract concepts, even compared to chimps and dolphins.

Also I think consciousness is very relevant to claiming machines are "using mathematics". Unless the machine itself becomes conscious, we are using machines to do mathematics. The machine is just a physical structure and reaction, like a crystal lattice. You wouldn't claim a crystal lattice is intelligent or using math, just because we can use math to describe interesting aspects of its structure, would you?

Voidious19:49, 1 March 2013

I don't think you understood the joke. It was a vague reference to The Hitchhiker's Guide to the Galaxy.

Your point about machines not being aware that they are using math reminds me of the Chinese room. Do you agree with it?

Sheldor21:05, 1 March 2013
 

I'll get back to you regarding the Chinese room once I have done more research on it, but I want to return to the discussion of classification of trees and humans again. You can distinguish between trees and humans based on their appearance. Do all trees share something that makes them trees?(Aristotelian form.) Do you have an understanding of what a tree is?

When I say human I mean a rational, sentient, living, material, substance. I'm not a materialist (as I said before, I follow the Scholastic school of thought), but you can't successfully argue against my stance by saying that you have a different idea (which is immaterial) of what a human is. I'm not asking what you think I think a human is, I'm asking what you think a human is.

"I don't believe Homo Sapiens are really that special when compared to other animals. We are not the only creatures that have developed tools or language. We are not the only creatures to feel emotions or pain, or to be "aware" of our surroundings. Insect societies are funduamentally not too far different from ours. Dogs and pigs are more intelligent than human infants. Butterflies see the world in colors we can't even imagine. "

You're completely missing the point. I'm saying that we aren't distinguished from other animals based on these things, so what do you use to distinguish us?

As regards STEM, I'm not an artistic person and I'm not saying art is more IMPORTANT than other things, but it seems to be a difference between humans and other animals. Can you explain why we have art museums and no other animal does?

AW14:42, 5 March 2013

We are distinguished by biological differences.

Art is a refined extension of beauty. In addition to merely noticing beauty, we create it. Beauty is simply an evolutionary adaptation that makes us seek symmetry and harmony. Other animals can perceive beauty, but only humans have the resources to make art and build museums around it.

Sheldor21:09, 7 March 2013
 
 
 

Our brains are closely related to consciousness, but they are not necessarily the same thing. And knowing if anything silicon-based is capable of consciousness or not leads to the mind-body problem.

MN17:50, 3 March 2013

While I agree with Voidious that we don't know enough about our brains to say for sure, I personally speculate that a very powerful supercomputer running very smart software could think like us. I guess that makes me a physicalist.

Sheldor20:50, 3 March 2013
 

A normal computer with very smart software could think like us. A supercomputer only provides speed.

Chase05:06, 4 March 2013
 

@Chase-san
Really? The human brain has over two petabytes of long-term memory.[1] Does your computer have a hard drive that size?

@Voidious
You seem to imply a difference between a conscious being deciding to use math and a computer receiving instructions and giving outputs. But, if consciousness is deterministic, is there really any difference between a conscious being receiving inputs (from its senses) and deciding to use math, and a machine receiving inputs (indirectly from conscious beings) and deciding to use math?

You have said that you believe humans are mostly deterministic. There are also many instances of people behaving deterministically. Among them are, as I've noted before, people who have transient global amnesia.

Sheldor14:36, 4 March 2013
 

If we are purely deterministic, then no, there's no difference. Math only exists in our minds, so to me, if there's no consciousness, there's no math.

Voidious17:12, 4 March 2013

I didn't say we aren't conscious. I said that our consciousness is deterministic, i.e. we give the same outputs when we receive the same inputs. We are just a "physical structure and reaction."

I'd still like to know what you think about the Chinese room thought experiment. It seems very relevant to our discussion.

Sheldor18:40, 4 March 2013
 

Why do you assume that we are deterministic? Don't you think it at least seems like we make choices?

AW14:28, 5 March 2013

We do make choices. I never said or even intentionally implied that we don't.

Determinism is simply the idea that everything in the universe has a causal relationship with everything else in the universe, and can only behave in one way. For example, let's say that I decide to eat a pizza for lunch instead of a salad. A determinist would say that all my past experiences, as well as the environment around me, and countless other things directly caused me to choose to eat the pizza.

Notice that from the deterministic perspective, I actually make a choice. I use all my memories to weigh both options and choose the one that benefits me the most.

Please, listen to the Radiolab clip I linked to. About five minutes in, they tell the story of a woman with transient global amnesia. It is an excellent example of human determinism.

Sheldor17:09, 5 March 2013
 

I think most people (including me) would define "choice" in such a way that determinism contradicts it. If there was only ever one possible outcome, no choice was made, only the illusion of one.

Voidious17:15, 5 March 2013

I believe that humans are like computers; we have inputs and outputs. When we have the same inputs, we will have the same outputs.

But, we still make a choice. The woman in the clip decided to say the same things over and over, because her inputs were roughly the same.

Sheldor18:19, 5 March 2013
 

Well, you're free to use the word that way, but I definitely assign a different meaning to the word, and I bet most people do too. To me, it sounds like the difference between perceiving a choice and actually making one.

Voidious18:37, 5 March 2013

Well then, how do you personally define the word "choice"?

Sheldor18:50, 5 March 2013
 

Deciding between two potential outcomes. If there was only one potential outcome, it wasn't a choice, only the illusion of one. Real choice definitely implies free will and non-determinism.

Voidious18:56, 5 March 2013

But, in my example there were two potential outcomes, pizza or salad.

I'll try a new example: Alice has two options, she could have $1,000,000,000,000,000,000,000.00 deposited into her bank account, or she could have that much added to her debt. She weighs the two options, and decides to take the deposit because she perceives it to be in her best interest. She could have chosen the second option, but she didn't because she perceived it to not be in her best interest.

Alice's judgement is based on all her memories and past experiences, as well as her current environment, and many other small factors. In order for her choice to be non-deterministic, she would need to abandon all judgement and make her choice completely random.

So essentially, what you're saying is that in order to have freewill and have control over our fate, we need to make all our decisions completely random, without any consideration of how they effect us.

Sheldor20:29, 5 March 2013
 

Not at all. You say: "She could have chosen the second option." But you also say there was never any chance she could have chosen the second option, because the series of events is deterministic. Do you really not see a conflict here?

Voidious20:30, 5 March 2013
 

Free will would allow Alice to choose between maximizing money, choosing at random, or simply pissing off Sheldor. In the first, picking the money is the rational choice. In the second, flipping a coin is a rational choice. In the third, not picking the money is the rational choice.

Memories and past experiences help judge what is the best action for each desired outcome, but don't restrict possible outcomes to a single one.

MN23:27, 5 March 2013
 

Yes, she could have picked the latter option. It was a potential outcome, as was the former. But, she did not. Why didn't she? Because she used her judgement and decided to pick the former.

Her knowledge and memories were the inputs to her judgement, and her decision was the output. Technically speaking, she did not have free will as her knowledge and memories were acquired from prior events. But, to have free will, she couldn't let prior events determine her decision, so she would have to abandon her judgement.

In conclusion, you can keep your free will,

I will choose a path that's clear
I will choose rational judgement.

Sheldor23:40, 5 March 2013

There was 3 judgements up there. Alice must abandon 2 judgements, no matter the choice made.

MN01:01, 6 March 2013

I don't understand what you are saying.

Sheldor01:05, 6 March 2013
 

Knowledge and memories make Alice realize she has 3 (or more) choices.

MN14:30, 6 March 2013
 

She still only has two options: gain an immense sum of money, or spend eternity trying to pay-off that much debt.

You're saying that she has multiple possible motivators. Her motivators define her best interest. If her motivator was to have financial security, her best interest would have her take the money. If she wanted complete free will, her choice could not be determined by prior events, so she would have to either choose randomly, or have someone else choose for her. If her greatest priority in life was to piss me off, then she would perceive choosing the debt to be in her best interest.

Sheldor14:54, 6 March 2013
 

I'm saying she has the choice to go after the motivation or not.

MN16:35, 6 March 2013
 

Her motivation isn't really that relevant to the question of free will vs. determinism. I was assuming that her main motivation would be financial, but even if it wasn't, it would only change the outcome, and not the process of decision-making.

Sheldor23:10, 6 March 2013
 
 

Most people would choose the first, and again if repeated. But because there is an economic force acting against free will. Determinists find it fun, liberalists see it with horror.

It IS possible to destroy someones free will through force. Liberalists fight those situations so people can exert their free will without being opposed by externalities.

Another example, gravity. A determinist would brag no one can fly and there is no choice. A liberalist would build an airplane so people can choose to fly or not.

MN00:31, 6 March 2013

I'm a determinist, and I believe we do have choice. As I've noted before, I believe that we make choices like computers do, our perception of our current environment, as well as our memories of past events, are our inputs, and decisions are our outputs. When our perception and memory give us the same input, we will give the same decisions as output.

Free will is a self-defeating concept, for our choices to be completely non-deterministic, they would have to be truly random (Which I believe is impossible, anyway.), which gives the person in question no say in the matter at all.

Sheldor23:03, 6 March 2013
 

I really think you're warping the meaning of the word "choice" here. Would you say a tennis ball makes the choice to bounce off the ground?

Voidious23:32, 6 March 2013
 

@Voidious

From Wikipedia:

"Choice consists of the mental process of judging the merits of multiple options and selecting one or more of them. While a choice can be made between imagined options ("what would I do if ...?"), often a choice is made between real options and followed by the corresponding action. For example, a route for a journey is chosen based on the preference of arriving at a given destination as soon as possible. The preferred (and therefore chosen) route is then derived from information about how long each of the possible routes take. This can be done by a route planner. If the preference is more complex, such as involving the scenery of the route, cognition and feeling are more intertwined, and the choice is less easy to delegate to a computer program or assistant."

As I've said before, choice is weighing options and selecting the one that seems to be the most beneficial. Tennis balls do not do this, they are not intelligent systems, they are just objects directly following the laws of pysics. Humans and computers ultimately do follow the laws of physics, but they also exhibit the qualities of intelligence and choice.

Sheldor23:48, 6 March 2013
 

I would say that excerpt assumes a degree of free will. To a determinist, a tennis ball bouncing and a human's brain pulsing with electricity and then moving a mouth to form a sentence are just a series of atoms interacting in a purely deterministic way according to the laws of physics. Calling one a "choice" and the other "just objects directly following the laws of physics" seems disingenuous.

But I'll drop it if you want to choose to have your cake and eat it too. ;)

Voidious23:53, 6 March 2013
 

Eating cake is unhealthy.  ;)

You are correct. I believe all matter in the universe behaves deterministically according to the laws of physics (even those that we have not yet discovered). Not unlike the way cells in Conway's Game of Life interact with each other deterministically.

Intelligent systems are special arrangements or patterns of matter (or Life cells). Intelligent systems behave differently than non-intelligent systems (like a tennis ball), even though they still follow the same physical laws.

Tennis balls could be used to create logic gates. With enough of them, one could create a very basic (and inefficient) computer. Though the tennis balls themselves cannot make decisions, when enough of them are arranged together, they form an intelligent system that can receive inputs and give outputs.

A man actually did create a Universal Turing Machine in Life, using cells that directly obey deterministic laws.

Sheldor00:56, 7 March 2013
 
 
 
 
 
 

That still isn't a supercomputer, that is just a storage network. But if I recall the storage capacity of the human brain is still in dispute. Some say its incredibly vast (like you did just now), others claim it is just better at storing the information. I am more in the second camp.

It isn't compression exactly. I would say it is more of useless information is discarded, and useful information is only partially stored. Say I may learn something, but if I never use it, i'll forget its meaning. Meaning my brain got rid that useless information because it was never used. Studies show we forget up to 75% of what we learn on a daily basis. Of course if it got used a lot at some point it will stay in there for future use, though the exact details will get fuzzy with disuse.

It will go from "The quick brown fox jumped over the lazy dogs." to "fox jumped over dogs" to "fox (general notion of going over) dog". That makes no sense, so fox... oh, over the dog. What can we use to go over a dog. Well we could jump, or we could hover, fly. But a fox cannot hover or fly, so it was probably jump. So we reconstruct it to be "the fox jumped over the dog." Information was lost, but the general meaning remained.

To further enforce this, I had forgot part of the original saying above. I had lost 'quick'. But then I remembered that oh, its one of those sentences that uses all the letters. I noticed it didn't have a 'q' in it. So I suddenly recalled, oh, "Quick". Well the dog isn't quick, I don't think it quickly jumped, so the fox must be the one who was quick, as most foxes are.

Chase17:38, 4 March 2013

Okay, so a normal personal computer, with extremely large memory and much more storage, running very smart software could be conscious.

But, why bother? What's so great about consciousness? Really, the only advantage brains have over computers is their ability to recognize patterns.

Sheldor18:51, 4 March 2013
 
 
 
 
 

Two of my favourite SciFi authors I've read have touched on the topic of sentient AIs, Neil Asher and Iain M. Banks. In both of the universes they create, humans have essentially been overtaken by benevolent AIs, who do all of the organisational work and governing, while the humans are given the resources they need to take up pretty much any lifestyle they want because mechanisation has made any form of labour unnecessary.

Their approaches differ, though, in how they view sentience. In Banks's work, any computer system above a certain level of power is legally required to be made sentient. From this, I interpret that sentience is not an intrinsic property of a powerful computer, but rather a certain organisation and programming of said computer. Asher, on the other hand, has multiple stories about a certain humanoid robot (the Brass Man, 'Crane', in case you want to read the books) whose processor-crystal was fractured, but who continued to function with multiple personalities. From this, unless the programming was particularly redundant, I would infer that after a certain amount of processing power, and given the right seed data, sentience sort of springs into place. Both feature brain-network interfaces, called 'neural nets' by Banks and 'gridlinking' by Asher, but only Asher covers cyborgs and human augmentation with processing nodes and robotic limbs. Asher also features 'golems', which are weak (although smarter than human) AIs in a humanoid chassis covered in syntheskin, which run a human emulation program and thus experience love, fear etc, and are generally indistinguishable from humans, although the emulation can be turned off during emergencies and the syntheskin isn't necessary for operation. Both authors also follow different post-Einstienien physics, which I find particularly interesting =)

Skilgannon07:20, 26 February 2013
 

I guess it's becoming pretty clear that Robocode robots aren't alive in the same sense that we are.

That's certainly not implying that no robots are alive or can be alive, only that Robocode is a very restrictive environment.

Sheldor04:18, 1 March 2013
 

I've become very fond of the Simulation Hypothesis. Not necessarily the idea that we are simulated by advanced humans, but the idea that our universe is the product of some external intelligent system.

I'll explain using Conway's Game of Life as an example. (If you haven't downloaded Golly yet, please do.) From the perspective of an intelligent system in Life, the ever expanding 2D grid it calls home, is the universe. It would see cells as indivisible subatomic particles, simple patterns as atoms, and patterns of patterns as molecules. It wold try to understand the mechanical laws that control partical interaction. It would be dumbfounded by alive cells appearing out of nowhere for no apparent reason, and cells that should be alive suddenly die. The intelligent being would wonder why it's there, and how it got there. Even once it had figured out the laws that govern cell birth and death, it would still have no idea that we are the reason that it and its universe exist, and the reason that cells appear and die randomly. We are both the source of its existance, and the source of all randomness it perceives.

I realize it's a bit of a stretch to imagine a self-aware pattern in Life, but it's theoretically possible, and could simply be hypothetical if you would like. The interesting thing is, this could describe our universe almost perfectly. We understand much of how our universe behaves, but we're stumped by qustions like what existed before the big bang or why do particles interact the way they do. Also, our subatomic particles are very similiar to Life cells, in that they exist in discrete states, and they have no real "substance."

It's well understood that what makes an object "solid" is nothing more than mathematical fields. (Here is another relevant Radiolab clip. I love that show.) So, reality can be reduced to math. Very similiar to how the Life universe is simply interaction of a bunch of imaginary particles dictated by a few simple math rules.

When I say we are the products of an intelligent system, I mean that our universe is a bunch of logical laws and mathematical formulae being computed by some external intelligence. This intelligence could be a superintelligent being, a supercomputer, or something we mortals cannot even comprehend.

These have only been my personal theories, and should not be taken as statements of fact. I don't even think it's possible to prove whether we are simulated or not. Please, tell me what you think.

Sheldor17:09, 12 March 2013
 

My favorite movie which touches the subject of simulation hypothesis is The Matrix. Not only philosophical/sci-fi, it is also a very good action movie.

"Unfortunately, no one can be told what the Matrix is. You have to see it for yourself." - Morpheus.

MN20:39, 12 March 2013

Thought this was pretty hilarious: [1] ... And almost on-topic, now that The Matrix and the limits of feline understanding have come up. :-)

And +1 to Matrix trilogy in general, I think it's way under-appreciated.

Voidious20:48, 12 March 2013
 

Maybe under-appreciated, but it has rooted itself in popular culture, and even in Robocode.

DT from SandboxDT stands for "Dodge this.", guess where this quote came from?

MN21:09, 12 March 2013

Click "Share this quote" to get a URL straight to the quote you want.

This link shows the desired quote in a light yellow box at the top of the page.

Sheldor00:03, 13 March 2013
 

Oh yeah! I never noticed that connection.

There was also AgentSmith, which never came to fruition, but got some more people thinking about genetic algorithms. (Which I probably remembered because Wolfman just showed up today at the BerryBots forums.)

Voidious21:14, 12 March 2013
 

I watched The Matrix last night.

The premise is absolutely ridiculous, but the action scenes are okay.

Sheldor20:35, 20 March 2013

lol.

I didn't like it when it first came out. I thought it was just a flashy rehash/mangling of a lot of cool sci-fi concepts from William Gibson and others, and the "battery" premise was a complete turn-off for me. But I dug the sequels, and eventually softened my criticism of the first one and now I really dig all of them.

Voidious20:38, 20 March 2013

Yeah, why humans and not electric eels? And electric eels wouldn't require a ton of processing power to provide a simulation environment...

I found Dark City much more gripping, and creepy in the 'what if it's actually happening?' kind of way.

Skilgannon20:40, 20 March 2013

Why electric eels and not specially designed bacteria? Or, even better, direct harvesting of geothermal energy? Or, just leave the "burned" Earth and get solar power straight from the source?

Sheldor21:15, 20 March 2013
 
 
 
 
 
Personal tools