Micro-transactions, DLC and IAP – a primer.

Recently I saw a facebook post about a friend mentioning that he was seeing people complain about MicroTransactions in Dead Space 3 – people were complaining that this is a full price game, so why should they have to spend more $ on stuff in game?

And it’s a valid point – why should they?

Before we get to that though, I’d like to talk a little about what MicroTransactions, DLC and IAP actually mean from a developer point of view, and how they are used to generate cash for the developer.

So, MicroTransactions is a catchall term for any in game purchase the game offers you – required or not. The idea is that the game offers you a really easy way to spend very little to gain some item or experience, or potentially to just buy a virtual currency (there’s a long post about virtual currencies coming up at another time).

IAP is In App Purchase – generally this is where you buy an item (E.g. Binoculars, a new weapon, extra shields etc) that aids / enhances your game playing experience. The concept here though is that you are buying something that the game already has within it – all you are really doing is just unlocking something that is already there. The game may allow you to earn it through game play, but is offering you a way to purchase it early, so it’s there for your use before you have enough game time to earn it.

DLC is, by definition,extra content. This is content created posted the game shipping – and almost always requires you to actually download something new. New levels, a new skin, new sounds, an new mission etc. Usually games that offer this are designed so that new content can be created later. DLC is purchased in the same way that items are – either by direct money purchase (where you go through the Apple IOS purchasing system / google IAP purchasing system) or by in game currency (e.g. XBox Live Points). If a virtual currency is used, some of that virtual currency may have been earned in game or been awarded – it means that there is a cut out between the item you are getting for the game and the means by which it is paid for.

So the whole concept of MicroTransactions is that the actual cost, to you, the player is minimal and it appeals to the impulse purchase. The idea being “It’s only $.99 and I can build castles almost instantly, instead of waiting an hour. A dollar – my time is worth that”. Now this only works if the purchasing method is what we call frictionless – ie extremely easy and as close to one click as is possible. Both Apple and Google have spent a lot of time getting their purchasing systems down to as little interaction as possible, although to some degree some is necessary just in order to stop people accidentally buying $99.99 worth of items.

Ok, so we’ve defined it. Now we need to talk about how it’s used, because that has practical bearing on the point under contention – ie is Dead Space bad for including this?

Generally, MicroTransactions are used in one of three ways. Friction, blocking and enhancement.

Blocking is the term used when you hit something in the game that requires a purchase in order for you to proceed. The most obvious the case is “You’ve finished levels 1 & 2, now you have to pay to play level 3″ or “You’ve played free for 4 hours, now you have to pay for another 4 hours” or “Now you have to buy a paid account to continue”. Lots of websites take this route – where you get 10 free emails before having to pay for a full account in order to continue. Blocking can be indirect and less obvious though – for example, I am playing a hunting game. I am given 50 bullets up front, but once they are gone, I cannot shoot the gun any more. I may, perhaps, have been awarded another 10 by the game for marksmanship, but effectively it’s cost me 50 free bullets to buy me 10 more. Once those are gone, there is no way to play this game except by buying more bullets. That’s a blocker, because there is simply no way around it. No matter how well you play, you will run into this wall.

Friction is where the game employs either completely artificial or inherent systems that slow down game play and progress. For example, waiting for your crops to grow in Farmville, or putting down Tiny Towers while you wait for the building to earn enough to add another floor. Both of these employ time as a friction – you are barred from progressing until the time goes past. But you can pay to remove that restriction and progress instantly. Time is not the only way that games employ friction – another, more sneaky, way is to limit equipment usage by count – you get to use the scope on the sniper rifle 5 times before it’s useless and you have to buy another one. The whole point here is to impede player progress – while still giving them enough game play that they just pay to continue the experience.

Now I mentioned Artificial or Inherent systems – there is a difference between the two. An inherent system is one that is part of the game world, is understandable and obvious to all. Running out of ammo in a shooting game is an inherent part of game play. Running out of gas in a driving game is another. Its understandable and obvious and a natural part of the game. Even, to a certain extent, some of the time friction is understandable – it does take time to build a building or to grow crops.

Artificial friction is restrictions and restraints that are 100% imposed by the game developer for no other reason that to make people pay. The example of the sniper scope is such – sniper scopes do not only work 5 times then break. You buy one once and it should work forever. But since this is a high value item, the developer chooses to limit usage and effectively put a tax on.

Enhancement is where you offer an item for sale that enhances the experience, but that is strictly speaking not required for game play. It may add purely visual enhancement (“I have gold armor! You do not!) or it may add actual game value (“My gold armor blocks 2x more attacks than yours does”). This is where you are buying a physical usage item that other people do not have. It’s different from friction in that friction is about the act of playing the game; enhancement items are about changing the event of playing. Friction is “I have overcome progress inhibitors” – enhancement is “When I _do_ play, my playing is either more effective than yours, or I am visually more unique than you are.” Friction is “I don’t have to wait for my crops to harvest” – enhancement is “I used a fertilizer that is not generally available, so when my crops harvest, I get 2x the return.”

It’s a definite but subtle difference – it’s possible to play the entire game with purely enhancement but no bypassing friction methods, for example.

So, we’ve defined the terminology. What of it?

Well, this is really where the rubber meets the road. The definition and balancing of MicroTransactions is where most games fail, and why IAP has such a bad rap with so many gamers – and it’s why people are genuinely upset over MicroTransactions in Dead Space 3.

The problem is this – the mobile market has taught many gamers that games should either be free or cost $2.99. The trouble is that this is not economically viable in terms of production costs. Games are not free to make and build. They cost time and effort and skills, and you need to sell a lot of $2.99 games to make up the often tens (if not hundreds) of thousands of dollars sunk into the development (for example, the IOS darling Infinity Blade, from Epic, has over half a million dollars worth of assets in it. Think about that. Half a million dollars. Who has that amount to invest in a game selling for $2.99?).

Plus, this kind of sale is a once in a lifetime sale. You sell that game exactly once – the player gets to give you money once. Even if they think the game should have sold for $10, they can only give you $2.99, and only once. Obviously there needs to be a mechanism whereby the player can continue to contribute.

What makes this even more imperative is the Android Piracy problem. The fact is that, because the platform is so open by design, it’s incredibly easy to pirate apps and games. Unless your game uses an external game server to run the game, chances are you WILL be pirated on Android. The better the game, the more the piracy will occur. It’s just too easy. So again, what is needed here is a way to avoid single point of sale and bring in $ from players over time. Make your game free in the first place and the whole piracy issue suddenly works for you, getting the game into more players hands, instead of against you.

But the problem with all this, as I mentioned, is balance. So many developers are resentful of the fact that they are out on a limb for tens if not hundreds of thousands of dollar, and can’t charge for it. So what they do is make way too many blocking MicroTransaction requirements, or employ too many friction systems, or make those friction systems way too punishing.

Players don’t react well to constantly being asked for money – even if it is in dribs and drabs of $1 and $2 – in order to continue their playing. They also do not appreciate being constantly gouged – as they see it – based on wholly artificial restrictions in the first place.

The best games do it incrementally and have very few, if any blocking systems. Now, friction systems can _become_ blocking systems if the player isn’t skillful enough – the example of ammo replenishment is a case in point. Lets say that the game awards bullets based on marksmanship, and as long as you are over 50% on target, it’s awarding you enough bullets to keep playing. The moment you drop below 50%, it stops awarding you. Too long at a sub 50% level and you’ll run out. Suddenly the friction of being required to be better than 50% has now become a blocking MicroTransactional requirement. This is generally ok, because the player will recognise that it’s is his deficiency as a player that is causing this.However, if it becomes obvious that the game has changed it’s ability for the player to become accurate (suddenly targets are further away, or wind starts moving the bullet trajectory), then all bets are off, since the players ability to make the 50% rate is now affected in ways he cannot compensate for. So while, ostensibly, this is still a friction system, it’s really become a blocking system quietly, behind the scenes.

If this happens too early, players can get disheartened, and they Will Complain. It’s a very tough balancing act.

What’s more, having several friction systems operating at once makes it difficult to ensure that they are balanced. You don’t want 10 of them all hitting at the same time and requiring 10 different IAP buy in’s to continue playing.

Then there’s the aspect of full price games – which is where we get back into the original point under question. Is it ok for a full price game to ask for MicroTransactions? They’ve already taken a full price – $60 – for the game – why am I being asked for more $? Well, in the case of games like SkyRim and the ilk, generally MicroTransactions are for either DLC (in which case this is extra content, so it’s not covered by the original $60) or it’s for enhancements – usually of the purely cosmetic variety. The moment the enhancements make any significant difference to game play – particularly if multiplayer is involved – then it becomes more of a issue to players. I don’t want to buy a game, then get on a Multi-player game and have my ass handed to me by another player with a gun I cannot have in my game because I haven’t spent another $50 buying it.

Personally, as long as the MicroTransactions are for purely cosmetic or DLC, or for small items that aid in single play, I could careless. More power to the developers, frankly.

The moment you introduce something that unbalances the game – particularly in Multi-player modes,- I think someone needs to sit and have a good think.

Destroying an IP’s long term value for a quick buck now is not strategic thinking, but unfortunately, for a lot of hand to mouth developers, this is how they survive. And it sucks.

Anyway, next up is a review of Virtual Currencies – how they work and what the issues with them are, from a developers point of view.

Posted in Game Development, MicroTransactions | Leave a comment

Movie Review – Silver Linings Playbook

I’m a member of Bafta. This means that, around Christmas time, I get movie screeners – for ‘my consideration’.

It’s quite interesting seeing what movies the studios think should be deserving of an award. We get sent things like The Hobbit, but would never see Jack Reacher or This is 40 (which is what I’d actually like to see). Still, we do get quite a lot of great movies to watch, and sometimes stuff I’d never pay to see at a movie theater.

Silver Linings Playbook is one such movie. It stars Bradley Cooper, and I have to admit I’m a bit of a Bradley Cooper fan boi – even as far back as Alias, where he got his start.

And generally, I think he’s chosen projects that are compelling and good vehicles for him – until now at least.

So lets get the review out of the way. Silver Linings Playbook is not a good film. Why anyone would imagine this is deserving of an award is beyond me. Technically, it’s badly made. The editing is terrible – some scenes start literally as someone shouted action, and suddenly every one moves with that terrible artificial purpose. The camera work is beyond understanding – technically it’s ok, but in terms of direction it’s unfocussed and breaks camera positioning rules without understandingwhy those rules exist in the first place. You can only break the rules effectively when you understand under what circumstances you should do so – the director here – David O. Russell – clearly does not. The whole thing looks very like ‘Movie Students First Movie Out of Movie School”. There are elementary mistakes, continuity mistakes, terrible direction, scenes included that do not progress the story, or, indeed, make much sense at all.

And further to this, the combination of writer/director here means that not only we don’t get good camera direction, we get incredibly stilted and unbelievable dialog. The principle characters actually sit down and explain the back ground of the movie to other characters who plainly already know all this stuff, and as a result, it’s a very obvious attempt to give background when there are plainly better ways to do it. I can’t honestly believe that Bradley Cooper wouldn’t have seen that in the script, but that probably gave way to the fact that Robert DeNiro is in this movie. I’ve _got_ to believe that the reason Bradely Cooper attached himself to this was because of Robert DeNiro. Which does beg the question of why Robert DeNiro is in this movie. I can only imagine he owes someone a favor, or they have pictures of him having sex with a horse – one or the other.

DeNiro basically phones in the Angry/Bewildered Dad dialog and has a bunch of fairly normal but relatively pointless exchanges with other characters. It’s a shame because he’s way better than this.

But you know what? I could forgive all of that if the movie were interesting. Unfortunately, it just isn’t that interesting. The characters are definitely ‘real’, but ultimately fairly predictable and just not people that induces emotion on your part, except for repeated glancing at the watch to see how much longer the movie is going to be on for.

The fact is that Bradley Cooper does his best with his mentally-unbalanced-but-holding-on-purely-through-his-own-delusions-about-his-future-with-his-exwife character, but ultimately the character arc just isn’t interesting enough to make you care enough, certainly not to overlook all of the other issues with this movie.

That’s not to say there aren’t some interesting scenes – the whole Hemmingway-book-at-4am thing was quite fun – but it’s just not enough to save what is effectively a pretty tedious and not-well made movie.

Then you put on Lincoln and wow, the flaws in this movie suddenly get magnified 10x. Which isn’t really fair, but as a movie watcher, I judge it my job to watch a movie and get involved. If you haven’t grabbed me 20 mins in, then you may well have made the movie YOU wanted to make, but you certainly haven’t made one _I_ want to see, and I would consider that a failure on your part.

My $0.04.

Posted in Movies | Leave a comment

Video games, Gun control and Mass Murder.

So yeah, this awful awful thing happened. Can’t escape it and nor should you really want to.

It’s an indictment on our civilization in the western hemisphere and it says a lot about us as a culture.

Now it’s happened – Again – there is finger pointing. As there usually is in the media, there is a need to appropriate blame and try and work out some silver bullet fix that will stop something like this ever happening again. We, as a culture, want to blame someone (which is hardly surprising, given the heinousness of the crime, and the natural anger and outrage to which everyone is subject over this).

Now targets like the NRA are easy. They campaign relentlessly for gun ownership and fight ANY legislation aimed at curtailing their ‘hobby’. Assault weapon bans are easy.

There’s also a mental health blog post going round titled “I am Adam Lanza’s Mother” which details the trials or raising a socially maladjusted child, be it from mis matched brain chemistry, social maladjustment or neuroses. There’s a lack of support for mental health that is lamentable in this country, and worse still, a belittlement of that condition.

Then there’s the inevitable Fox News finger pointing at the evils of Video Games, since we are obviously training and desensitizing the psychopaths of the future.

So what’s to blame? Where do we point the finger? What can we change?

Well, unfortunately, it’s not that simple. For each of these issues – and they are all issues – there are problems.

Lets take Mental Health first. The first thing to say is that Mental Health in the US (and in other countries) is shockingly bad. The actual care isn’t so bad, but the events and tracking leading up to being committed are. It wasn’t always this way, so why is it now?

That’s a pretty easy question to answer though. Money. The medical care system in the US is For Profit. Insurance companies are a for profit endeavor. Health care is an on going process an often just as expensive day to day for care as being in regular hospital. And guess what? Lots of mental health issues cannot be ‘cured’ per-se. They can only be managed. That means the costnever goes away. How do imagine the health companies view this? Think that’s something they want to sign up for? Hell no.

During the 1980′s, in his tenure as Californian Governor, Ronald Regan closed lots of the mental healthcare facilities in CA, because they cost so much to run. Think about that the next time you see a homeless person in CA, because most of them used to be housed in these facilities.

Mental Health care isn’t going to get a revamp or look over until we find some other way of paying for it that isn’t For Profit.

Then there’s the fact that the linked article above isn’t as germaine to the actual Connecticut shootings as it might originally seem. Sure, what the lady is complaining about is relevant to our culture, and she does have a point. But most g0-mad-with-an-ak47 shootings are NOT performed by people who are trackable, with warning signs that are easy to see. This blog post details it nicely http://thegirlwhowasthursday.wordpress.com/2012/12/16/you-are-not-adam-lanzas-mother/

The fact is that while there needs to be far greater help for those with mental illness, it wouldn’t stop massacres of this nature – well, it might stop some, but there are those out there who are very good at hiding their own psychoses (like the nutter who shot up the movie theater in Colorado) or whom just suddenly snap. You cannot just lower the filters of what is a warning sign because in that case I guarantee every person who’s reading this blog would be put away at some point or another. We’ve ALL had moments of high stress where we’ve reacted badly and had other people looking sideways at us.

So, while Mental Health needs a good deal of work, it’s only one piece in the jigsaw.

The next is gun control.

Yeah, we need more of this. I don’t think there’s a question about that. There’s NO NEED for ANYONE to need 47 guns at home. There just isn’t. The general public is in no need of semi automatic weapons.

Really, the second amendment needs to be repealed completely – it’s from a bygone era designed to ensure that America could fight back to the UK if they tried to invade. It was designed to create groups that could assume police and army like stances, before there were any. There are both police and armies now. It’s un-necessary and frankly, dangerous. But – there is an entire class of people out there who are both afraid (and we’ll get into that in a bit) and also knowthey don’t matter. And they don’t. 95% of the US population is interchangeable and can be replaced and the world would keep turning and nothing would change. That’s the sad reality (and to be clear, I’m in that 95%). That’s a terrible thing to carry around and the fact is, holding and shooting a weapon is the opposite of that feeling. It makes you feel like the world would pay attention. You have a weapon, and the world would sit up and take notice. And for most people, that feeling is enough – but it’s such a powerful feeling that they will _never_ allow you to remove their guns, because then they trulywould have no self respect.

Now to be clear, that’s not every gun owner – but those who need to own 5 guns at home? I have serious questions about how big they think their penises are, and how afraid they truly are – both of “the gubermint takin’ over”, “The black people down the block coming to rob them” and “I don’t matter and if you take my guns, I really _won’t_ matter”.

And this is why the 2nd amendment isn’t going to go away – because the NRA has their finger very accurately on the pulse of this fear and resentment feeling that quite a lot of western civilization festers.

While we are there, lets also deal with this “I need guns for defence” specious argument. The fact is that no, you don’t need guns for defence. The majority of guns at home will never be shot by those owning it, even if confronted by armed robbers. Most of us do NOT have what it takes to take a life. In fact, stats show that you are 3 times more likely to be shot by your own gun if you have one in your home than not. Here’s a link making that point. http://www.newscientist.com/article/dn17922-carrying-a-gun-increases-risk-of-getting-shot-and-killed.html

And that’s assuming you can actually get the gun out. Most people will simply not react to a shooting situation the way they imagine, based on all the movies they’ve seen and games they’ve played. So yeah, here – go watch this – it makes my point for me. http://www.youtube.com/watch?v=8QjZY3WiO9s

And lastly, lets also deal with this “More people get killed by cars – we should ban cars” BS. Yes, that’s true. The trouble is, cars are NOT designed to kill people, and guns are. There’s no apples to apples comparison here at all. If you took all cars away, the US would grind to a halt. If you took all guns away, it would just become safer. So yeah, that argument is a waste of time.

So ok, if we can’t repeal the 2nd amendment, what can we do? Well, we can do several things to help.

  1. Ban automatic weapons. The public doesn’t need them and nothing good can come of them having them, except certain people feel their penis size is OK.
  2. Ban all clips that contain more than 8 bullets. Why would you need more than 8 bullets anyway?
  3. Improve purchasing restrictions on weapons. I have to take a test (both written and practical) and have my eyes tested before I can get a drivers license. Why is that not the same for gun ownership? I can also have my license taken away for bad decision making – why is that not the same for guns? There needs to be a psych eval for a request to own a gun – that would at least highlight some of the nutters who want a gun for nefarious purposes.
  4. Remove the concealed carry right. NOTHING good comes of this. When the congress woman in Arizona was shot, there happened to be a gentleman nearby who WAS carrying. He came upon the scene of the actual shooter being sat on by a member of the public, who was holding his gun to keep it away from the shooter. That individual who was carrying was a split second away from putting a bullet in the member of the publics head, because he mis-read the situation. Concealed carry is a bad thing – of that there is no question.
  5. Remove the right for home stored weapons. You want to own a gun? Fine. It’s locked up in a gun club. As has already been demonstrated, you having one at home is dangerous and you are more likely to be shot by it than use it.
  6. Pull the NRA’s teeth. Lobbyists need to be refused access to senators and congress people – remove the access and the money goes away.

Now that would help a go some way to removing the availability of guns to those who snap. It won’t stop all of them – but short of repealing the second amendment, nothing will. At this point, all we can hope to do is make it more difficult.

Then there’s the media. Movies, video games and the news networks.

I’d like to separate the two – movies / video games and the news networks, because in my mind, they do two separate things.

The news networks push fear. Fear sells. Their stories are ALL about “Are you at risk from the new strain of Nuclear HIV? Story at 10!” – if it bleeds, it leads. That’s Selling By Fear. It’s no wonder that so many people want to own guns – the news is ALL ABOUT the latest shooting and all the drama and blood they can drag up.

Movies / Video games don’t make you afraid. They glorify and desensitize. When you have syndicated Television shows (Syndication means you sell your shows as a large block of 100 episodes or more – and it helps if each episode is individual, because more than likely the shows will be shown out of order) where one week, a character gets a gun shot wound and is close to death, but next week, he’s fine with no ill effects, it’s no surprise than an entire generation grows up thinking that a gun shot wound won’t stop you – you just have to grimace and carry on. None of these tv shows / movies show the reality of being in a fire fight – of shock and pain and on going medical problems due to a gun shot wound.

Video game are the same – there’s a argument that movies / TV and Video game are ‘just holding up a mirror to society’ – that they depict is just whats going on in society anyway. However, there’s a lack of understanding that while that might be true, it also perpetuates those attitudes. That gunning down masses of people is all Just Good Clean Fun.

Now to be sure – I’m part of that industry and I don’t want to believe that we are responsible for training mass murderers – and to be clear, I don’t believe that. The fact is that if you are disturbed enough to go on a shooting spree, then playing a few games won’t significantly make that worse. People were blowing stuff up and on killing rampages way before Call of Duty was on the shelves.

But, having said that, we _do_ owe a duty to be more responsible with what we make and show. We aren’t the problem, but we do contribute. And contrary to what the NRA would like everyone to believe, Gun Control would go a hell of a lot further to stopping nutters going berserk with an M-15 than stopping Medal Of Honor being made.

So there you have it. The three prongs of what needs to change to reduce (and note I said reduce – you will NEVER entirely stop events like this from happening – that genie is not going back in the bottle) massacres of this nature.

Tell Biden I sent you.

Posted in Game Development, Life Musings | 1 Comment

Animation Systems 101

So, here’s a post I made on a forum – this is a basic primer on animation systems for video game development. The idea is that it should explain how they are made, what components are part of it

Animation Systems – a Primer

Bone based animations – at it’s root, an animation is basically just a bunch of hierarchical bone rotations (and possibly positions, if you want the capability to scale bones).

An animation file tends to contain several things -
The skeleton default pose – this is a list of bones, in tree order, with positions, when each bone is considered to have “no rotation” – the angles for each bone are 0,0,0 or the matrix is at identity. Usually this pose looks like the T Pose for humanoid characters, but really it’s whatever your animation guys think it should be. The only caveat is that all animation values you ever get are offsets from this original default position.

Then, for each frame of animation, you have a list of bones (in hirearchial tree order) with rotations (and as said, possibly new positions) of each bone. This can be represented in several ways. Simple Euler angles – although if you do this, you do have to indicate what the order of angle operations is (do you apply Roll first? Or Pitch?) – just like you do when creating a matrix from euler angles (which is the final step of the per-frame skeleton generation). Almost all systems use Quaternions for per-bone rotation because a) they are interpolatable (you can take two rotations, one for frame A and one for frame B and just do a linear interpolation of values between the two) which you cannot do for a pure matrix representation, b) they are smaller in size (3 floats for quat vector and one for rotation as opposed for 9 floats for a pure rotation matrix) and c) you can actually encode scale into the vector (ie it’s not normalized) if you want to lengthen bones.

For each frame you wish to display, it’s a very simple calculation to work out which frame you are inside the animation (well, between – usually you are between two frames – I’ll get to frame frequency in a moment), take the two frames rotation for any given bone and interpolate between it, based on percentage of closeness between Frame A and Frame B (Frame A is on time at 300ms, Frame B is on 400ms, we are at 320ms, so we get 80% of Frame A and 20% of Frame B. Bogus example, but you get the idea).

From this, you now have a quaternion for a specific bone. You just then generate a matrix from this bone, multiply that by the parent bone (remember, you are traversing a tree here, from root to extremity and it’s hierarchical, so you need to multiply by the parent to get an absolute rotation for each bone, rather than a parent relative one), and there’s your skeleton.


To skin, you have a mesh which is mapped to the skeleton default pose (as described above) and for every vertex you have two sets of values (beyond simple XYZ position). A list of bones that affect this vertex and percentages for each bone. Usually there are limits to how many bones can affect a given vertex (4 or 8 because of shader usage, but we’ll get to that later. For now, this is a software skinning system.)

So, for each vertex you need to do a transform. What you need to do is transform each vertex by each bone that it says affects it, and then blend together the resulting position based on percentage of the amount each bone affects it.
So if vertex 38 is affected by bones 3 at 20%, 7 at 15%, 9 at 15% and 10 at 50%, you transform the original vertex position by bones 3, 7, 9 and 10, then blend the result using the percentages given.NOTE – you need to blend the result of each of those transforms, not the actual matrices for each bone. You need to physically transform the vertex for each bone – giving you 4 positions, in this example – and then scale blend between the RESULTS, not the initial matrices.

Also note that if you have a per vertex normal, you need to transform that by each bone and do the same blending you are doing for the vertex position.

There are more elegant ways of doing this and generating one matrix to multiply the vertex by, but in my experience the work required to generate that matrix tends to outweigh just doing it repeatedly (particularly if you are using SLL instructions) and doing it that way. Others may well be cleverer than I am.
And you need to do this for every vertex.

That’ll give you your mesh.

At root, that’s the most basic dynamic meshed animation system.

Now it can start getting very complicated very fast.

Optimizing Animation Storage

Firstly, you can vary your key frame rate inside the animation. A key frame is every frame of animation inside the animation. So if your key frame rate is 30hz, that means that you have an actual frame of animation every other tick. 60hz means a frame per frame.
Most animation systems allow for variable rates – so the animation itself tells you “I’m 30hz” or “I’m 20hz” because some animations need higher frame rates than others – an idle pose can be 15fps no problem, but gun animations in an FPS (because they are up close and always on screen at full size) need to be higher and smoother.
Even though you are interpolating in software, it’s relatively easy to miss an extreme frame if you are at low frame rate (e.g. you miss the frame with the hand outstretched to it’s fullest in a punch because it falls on frame that is missed out because you are at 15hz) so your animation doesn’t look right. It also makes circular motions look wrong because you are interpolating between absolute positions – instead of a nice curve you end up with very angular animations.

So what can you do about this? Well, more advanced animation systems allow you to have variable rate frequency animation within the animation itself. So although the animation is set at 15hz, there are some extra, animator marked, frames within it to ensure that extremes of motion aren’t missed. Each frame has a time stamp on it so you know when the last frame and when the next frame are meant to be individually. This is more work for the animator but ultimately results in smoother and better looking animation.

Then there’s per bone LOD and compression – you might actually want to store the per bone quat as 16bit fixed point instead of 4 floats – halves the size and still gives you pretty good results, although less so for dramatic large motions.

Lots of animation systems also don’t store per bone values for every bone per frame. They may well have systems that, when generating the animation in game ready format, say “Hmm, bone 3 didn’t move more than X degrees between frame 10 and frame 11. I’m just going to ignore that and not write any value out at all”. This approach can dramatically reduce out the per frame data for a given skeleton, depending on what your low threshold is for “Any motion below this value, I don’t care about” (or your Epsilon Delta). However, this also can have the effect of loosing certain very subtle aspects of human motion by effectively smoothing them out. It’s usually what makes mocap look like shit – because someone is smoothing out very subtle motion to make the curves look nicer and make the data smaller.
It also has the effect of making the code for generating skeletons have to look harder to find the last / next Quat per bone for frame to frame interpolation, since you can’t guarantee it actually IS in any given frame – you have to keep looking for it in every frame back from where you are till you actually find one (there are ways around this, with caching of data from frame to frame, but it’s a hack to get around thrashing your CPU cache looking for animation data for a given bone).

Anyway, suffice to say there are lots of ways to be clever about bone storage.


Then there’s blending. This can get really complicated, really fast.
So, you’ve already got per frame interpolation going on, so there’s blending going on already. What about blending two animations together? From a standing idle to walk? Well, it’s not really that hard. You just generate two skeletons (quaternion) for each animation, then blend between those two skeletons (using the exact same code you used for generating the interpolation between frame A to frame B ).

Although that does tend to suppose you are doing two full body blend. What if you want to run two different animations? One upper body, one lower body? Then what?
Well, now your animation system has to cope with the fact that, effectively, you might be running any animation on any bone (this is what Ghoul2 does) at differing frame rates.

You will also, probably, want to allow direct override control at any given bone too – so if you have a camera with a rotation bone for the root of the camera, you can direct it to where YOU want it to look, rather than where the animation does.

Again, not hard, but a fair amount of code to get right, and 85% of the time it will go totally unused.

But all this blending does come at a price, and I don’t mean CPU wise. I mean animation wise. The problem comes that animations are hierarchical, which means if you override one specific bone, it’s still starting it’s animation at whatever point the parent was at. So if you override a wrist animation, it’s occurring on the end of an arm that is controlled by something else. You might want to flip someone off, but the flip animation while running on the hand just fine, is running during an run cycle, so the arm is moving according to the run animation.

It gets worse. That the example of the upper and lower body above is a good case in point. While I want to have someone sitting in a chair, I also want that person to be following someone else around a room, so he’s constantly looking at them. You could override the bone at the bottom of the spine, but that’s not realistic – you just twist people around. What you actually have to do is override several bones in the back and twist them each a bit, so the torso rotates more realistically.

At this point you are now into very primitive IK – which is really outside of the animation system itself. The animation system should provide hooks that an IK system can use (like ragdoll) but shouldn’t be doing that work itself, because that means it has knowledge of the meshes / skeletons it’s rendering, which it shouldn’t.

Anyway, you get the point that it starts getting very blurry.

NOTE – there are some very cool things you can do with blending regardless of knowledge. Imagine having a baseball batter. Imagine you have two swings, one high and one low (at the same frame rate). You can run both animations on the batter with a provided percentage blend and what you get out of it is a swing that is at the exact height you want it, based on your percentage. So if the high swing at at 5 feet and the low swing is at 4 feet, a percentage of 50% will give you a swing at 4.5 feet. That’s actually a really cool thing, if you think about it.

Shader Skinning

So what about hardware skinning? Using a shader to do all that matrix math? That’s what vertex shaders are for, right?
Well, yes. However, there are two downsides to this. The first is that you don’t get to see the resulting mesh inside the CPU. If it’s all occurring inside the vertex shader, you don’t get the resulting mesh, which means you can’t be doing per-triangle collision detection against it (well, you can if you do the work on the CPU as WELL as the GPU and there are some higher performance optimization reasons for doing this too which I won’t go into now) – which mostly doesn’t matter but occasionally does.
The second is that now you have to break up your skin. Usually a mesh tends to be one large mesh, all referencing the same texture page. You might break it into two or three – skin being a different mesh / material grouping from clothing – but unless your game has a large runtime dynamic system which generates NPC’s or player characters from lots of smaller pieces, they tend to be 1-3 meshes.

However because of that, if you have a mesh that has a skeleton of 150 bones in it (which is very possible, particularly with articulated faces) you don’t know per vertex which bones that vertex might need; effectively you have to feed all 150 bones into the vertex shader running on that mesh. Which is too many – vertex shaders can’t take that many (that’s why I said there tended to be practical limits on how many bones a given vertex will reference). If any given vertex can reference any given bone (and all bones WILL get referenced by some part of the mesh vertex array) you MUST feed them all in.

The only way around this is to pre-process the mesh and break it up into smaller groups – where group A is known to reference bones 1-16 and you can feed 16 bones into a vertex shader easily. So for each mesh submission you feed in different groups of bones into the vertex shader (and often you’ll submit the same bone several times for different mesh arrays).

This breaks a render into several render calls, which doesn’t sound like much, but given you’ve effectively got about 2000 render calls per frame at 60fps, can make a difference. If you’ve got 20 people on screen, but now you’ve broken the mesh into 5 pieces you’ve just gone from 20 render calls to 100. That *does* make a difference.

Generally you don’t want more than 10 mesh render calls per mesh – and in my experience, those meshes that are about 2000 – 3000 polys with a skeleton of about 100 odd bones tend to come out around that – 7-10 mesh groups.

Other stuff.

Blending frames – generally, my experience has been to blend from one animation over about 200-250ms.

You can do some clever lod systems for animations based on distance – in Ghoul2, we did a camera to object distance and actually halved the rate at which we animated object based on how far it was. We just kept the skeleton / mesh around from the last frame and just re-used that every other frame. We probably could have gotten away with more if I’d really pushed it.

Other stuff you can do is just not animate all extremities – mark certain bones as “do no animate in low lod situations” – we did experiment with having different skeletons / meshes for low lod situations, and that really does work, however you can get cheap purely skeletal animation returns by simply not animating the fingers of a hand, for example. As long as you keep last frames around (or even just use the default frames bone settings) you are golden. Realistically though, unless you are doing a hell of a lot of blending, this doesn’t buy you that much, because skeleton generation isn’t that expensive – it pales next to vertex transformation for example.

Also, another thing that most animation systems have is a event track. This is a timed set of events that can occur over the length of the animation. So, for example, when a foot fully hits the floor, a “foot fall” event is generated. This could fire off a sound, or make a gun fire, or whatever.

What’s interesting is that the event track is actually processed separately from normal animations, because you may be doing clever shit like LODing the animations so not ever frame gets an update – however your DO need to do the animation track every frame because game events maybe happening off the animation (e.g. a weapon firing, or a punch connecting with an NPC).

Note though, event tracks can get a little tricky to handle. If you end up blending two animations together, but you have an event that’s right at the front of the second animation being blended (like fire weapon), what happens? I mean, you are still blending – you have no idea where the weapon is actually pointed at that point. It can get really tricky to know whether to actually fire an event or not based on blend state.

Similarly, the problem of internal motion within an animation can be handled with a per-frame motion track.

The problem is this. If you have a motion captured walk, the per frame motion in it is variable per frame. So, either you retain the motion in the animation itself (ie each frame moves further and further from the 0,0,0 starting point of the animation), which means you aren’t actively moving the object in the world per frame of animation, just moving them at the end of the animation to the new position in the world because the animation is looping OR you try and massage the animation so the motion forward is constant, so you can be moving your game object forward in the world at a constant and game controlled rate.

This is what most people do and where foot sliding comes from.

There is a better solution. Your pre-processor needs to look at the range of animation motion for the root bone of the object over the life of the animation (e.g. a complete walk cycle. It starts at 0,0,0 and ends at 100,0,0). From that you know the delta of the total animation (in this case, 100 in x).
Divide that by the number of frames (in this case 10), then subtract frame * delta from the position of the root bone on each frame.

This moves the animation roughly over 0,0,0 for every frame, offset by whatever little difference there is frame to frame to keep the ‘real’ motion in it.

Now you know what forward motion the average frame has, which your animation can then inform the game of. The game is moving the object through the world at the speed it was always intended to move at, and there’s no foot sliding. As a side effect, you can now increase or decrease that speed and then simply multiply the animation playback speed by the same value, and viola, you’ve made the animation walk faster but again with no footsliding. It’s really very simple.

It doesn’t work well with jumps, or anything that moves a large amount in the animation, but returns to the starting position at the end of the animation, but the way you sort that out is by breaking the animation in two, at the point of most extreme.

There’s a LOAD more about animation systems you can do, but that’s enough for now I think.

Posted in Animation, Game Development | Leave a comment

Dalek Iphone / MP3 Tones

I created these a while ago – they are Dalek Tones for your Iphone / Android phone. For free. Yes indeedy, FREE!

Basically, they are replacements for the standard Incoming Text / Facebook / Twitter / Email / Ring tones on your Iphone or Android phone – a bit of silliness I put together one evening when I was bored and waiting for builds to complete.

Originally they were posted on Kasterborous – one of the premier Doctor Who fan websites on the web (and one that I *cough* write articles for, as well).

Here’s the post that was put up there…

So the other night, while waiting for a build of the latest game I am working on, I was playing with the iOS BBC app that allows you to record your voice as a Dalek voice (iAmADalekfrom BBC Worldwide in iTunes).

I was playing with recording my voice, as a Dalek, to replace the default tones that my iPhone uses when an event occurs – an email comes in, a tweet comes in, a text arrives and so on.

It took some playing around to be able to import these new recordings – the app doesn’t allow you to set these values by default, so I had to email them to myself from the app, then import them in iTunes, convert them to the Tones format (the .m4r format), then _re-import_ them into iTunes, resynch the phone and then they worked. Apple sure doesn’t make it easy for you, but then they want to sell you tones, rather than making it easy for you to create them yourself.

ANYWAY…. I got 20 or so little speech snippets done, and I thought I’d package them up and see if anyone wants to download them themselves? Installation for iPhones is a very simple process (note, the following instructions are the same for Windows and Mac OSX versions of iTunes).

Download the zip file of iPhone samples, and unzip them once you get them (I’m assuming you’ll know how to do that. Windows should unzip the incoming file itself). This should leave you with a directory of 20 or so files in it). For FREE.

Open up iTunes, and select the Tones item, under Library (see pic). Now, newer versions of iTunes ship with that item not displayed by default (or until you’ve bought a tone). To make it viewable, open preferences and in the General Tab, make sure Tones is selected. Once you do that, Tones shows up in the main window.

Once you’ve selected the Tones item (and got an empty pane on the right hand side), go back to your unzipped directory of samples, select them all, and drag them on top of iTunes, depositing them in the empty Tones pane. This will force iTunes to import all those tone files.

Once it’s finished (and iTunes under windows takes a while; I’ve no idea why. I would LOVE a peek at that source code to understand why it’s so slow!), just synch your phone. The new tone samples will show up under Settings on your device, under the Sounds option.

Simply select one for whatever event sound you want to replace.

There you go. Did I mention they are FREE?

And Android users – never fear, you aren’t left out! I’ve also created a zip file of the same samples, but as MP3 files. I don’t have an Android phone, so can’t walk you through the steps of installing, but I’m sure you can work it out.

Have fun and Happy Phone Daleking!

Click here for the Android Dalek Tone zip file (mp3 files).

Click here for the Iphone Dalek Tone zip file (m4r files).

Posted in Doctor Who | Leave a comment

Risking it all.

Over the years, I’ve gone away from games, come back to games, tried to get away from games, (under my own steam, in terms of doing something where I only have to rely on myself) and come back to games.

Why the wishy washy-ness? Well, the fact is that in the Video Game Industry, like every other industry, we are awash with people who haven’t a clue. Or worse still, think they do have a clue.

A few days ago, for example, I had a client who was extolling the virtues of Video Advertising inside of mobile games. Seriously – he thought it was just GREAT. I kept trying to point out to him that, as a gamer, I seriously hate web pages that push video ads at you – I click away instantly (I can _just about_ deal with the way Hulu does it – those news pages that insist you watch a 30 second ad before they give you whatever content is there? Instant click away.) – can you imagine what I’d do if a GAME did that? INSTANT UN-INSTALL.

I actually did a quick informal poll of friends – both game developer and not – about this and they all said the same thing, which was Absolutely No Way. A game that did that, they would un-install instantly.

But my client wouldn’t have it. He thought they were greatbecause of the rate of return you get on forcing eyeballs onto it. By definition, anything that earns him money is great, and everyone will love it. Quite apart from the fact that video ads EAT bandwidth (which on most mobile plans is limited) and therefore the player is quite literally paying for you to waste his/her time – he just couldn’t grasp that players would hate it. “But it has huge stick and click through” he told me, as though that made it ok.


So yeah, getting back to the point, you can see how the industry is full of those people who don’t have a clue. I don’t think anyone can blame me for wanting to go do something where I have a bit more control.

But here’s the problem. I’m 44. I have a mortgage, I have kids, christ alone knows how I am going to pay for college for the kids. My savings got eaten by paying the mortgage on a house in CA that took months to sell… it’s not a great situation.

But still – I’ve written my own game, I’ve written books – I should be able to move into a new area and make a living, right?

Yeah, unfortunately, it’s not as simple as that. Give my age, I have a certain level of incoming requirement to meet the commitments I already have. It’s not super huge, but it’s not entry level either.

And therein lies the rub. In order to really make it in a new industry, you have to essentially start at the bottom. And, at my age, who can afford that?

A good friend of mine who lives in LA was talking about getting a gig in TV / The Movies. His point was that in order to get to the point where you are able to work for a living, you have to act as though you already are, and are in complete control over your time. You have to be available to take an audition, do a lunch meeting, meet someone for afternoon drinks or whatever at the drop of a hat  – because if you don’t, someone else will. Effectively you have to have the resources available to you to actually already live that life in order to get a foot in to start living that life. It’s a catch 22 situation.

The real point about the above story though is that if you intend to switch careers and go do something else, you have to do it 100% – be 100% committed to that new career and totally immerse yourself in it. You cannot do it piecemeal from an existing career. You might write a book in the evenings, but unless you are prepared to do book signings, go places and do readings and spend One Hell Of A Lot Of Time on self promotion, your books is going to sink, more likely than not. That’s just the reality.

There are hundreds of people trying to break into that market who _don’t_ have the overheads you do – college grads, stay at home moms, etc. They ARE on this 100% – why would publishers take a risk on someone who is evidently not 100% committed to this industry?

Then there’s the side aspects of a new career. Writing a book is a great example. Most people think that you write the book, and then you put it on Kindle, and you are done, right? Nope – before you even put it on the kindle, there’s a TON of stuff to work out first.  You need to get your book proof read (and good editors cost $2-3k a WEEK. Your friends might say they’ll do it, but they won’t. Proof reading a whole book is a HUGE job. Friends want to help, but they just won’t once they realise the amount of work involved.), then you need to format it for Kindle (it’s not like there is an Export To Kindle button in Word), then a cover needs to be made – and that’s MASSIVE in terms of getting attention. Then, once you are out,  there’s promotion, there’s watching what is happening in sales, changing the prices, putting the book on Nook and iBooks, keeping up with lots of websites about books and writing – you NEED to be public and available on those websites, etc etc.

Literally, writing the damn thing is only HALF the job – the fun bit. The rest comes as a shock to the system when you’ve just spent over a year doing the hard work of writing – now you aren’t done? You can’t just sit back and start thinking about the next book?

The fact is that every hobby is just that – a hobby. If it were your full time job, you’d realise how much work it is – and most of us have No Clue about how much extraneous work there is to making our hobby a full time job – much less be good enough at it to earn enough to earn a living. Unless you have a celebrity friend who’ll advertise you on twitter, or know someone in a position of authority who is willing to take the risk on you, good luck with it.

And it sucks. Because I desperately want to be in a situation where I only have to depend on myself rather than anyone else – but at this stage, I just don’t see how I can do it, not without risking my children’s future.

And yet – with all that, I’m not going to give up. I am still going to write books in my spare time and release them on Kindle. I’m never going to stop chasing the dream of being truly independent – because if you do, what else is there? Acceptance of a substandard now just makes for a crappy future. And I’m not up for that.

Posted in Game Development, Life Musings | Leave a comment

Yearly roundup of favorite developers.

I saw that my good friends Harvey Smith (https://twitter.com/Harvey1966 on twitter) and his partner Raphael Colantonio (https://twitter.com/rafcolantonio) just won the Best Adventure category at the VGA awards last night, for Dishonored.

Now I couldn’t be more made up for them (full disclosure – I consider Harvey a friend and as such you might want to take the following gushing in that light) – Dishonored is a truly awesome game and frankly, it’s great to see someone who has been absorbed by a publisher produce a AAA game of this caliber. If anyone was going to do it, it was these two, since both are established talents in the game development world.

It’s great to see Dishonored take it’s place along side Assassins Creed and Bioshock as a new IP that is top tier gaming.

But it got me thinking about developers that I really admire, and for what it’s worth, I thought I’d list some.

Bungie. Well, they are Bungie, aren’t they? They created Halo, have been around forever, have an incredible history of being bought by Microsoft, then buying themselves back – and now Destiny appears to be on the horizon and from the concept art, it looks AWESOME. There’s serious pressure on them to reproduce the Halo Magic, but I’m confident that they can do it.

Epic. While I have some reservations about how Epic runs itself as a company, and some of the decisions that have gone into Unreal as an engine, there is just no questioning that these guys know how to make great games. Unreal, Unreal Tourament, Gears of War – these guys are top notch at making the most of their engine and producing engaging content.
It’s also worth noting how they started up a new studio in Boston to help out when 38 Studios went bang, which was very inspiring.

4 Door Lemon. Who? They are a small mobile group in Bradford, in the north of the UK. And they produce some pretty damn awesome Vita games. They are One To Watch for the future. Learn more about them here – http://www.fourdoorlemon.com/

The Pickford Brothers – creators of Magnetic Billards on the Ipad (http://www.zee-3.com/magneticbilliards/) who are frankly pioneers and capable of great things (Look out for Naked War, coming to an Ipad near you soon!).

Human Head up in Madison Wisc. They’d have a crappy year, what with all the shenanigans over Prey II – however they are hiring again, which means they have a gig. Fingers crossed for them!

Gearbox. Gotta admit, I’m a fanboi. They get the job done and they have a sense of humor when doing it. Borderlands is a thing of beauty, no question.

So there you go. Personal favorites when it comes to developers. YMMV.

Posted in Game Development | Leave a comment

Every Rule is Contextual.

I got into an argument yesterday – someone made a generic rule and immediately the pendant in me reared it’s head to contest it.

However, on reflection, I think that my initial thinking “Christ, I need to stop being so pedantic” wasn’t right. The fact is No Rule set that can be expressed in a tweet is complete or generic enough to cover everything.

Now, before we go any further, it’s probably a good thing to understand the difference between an observation and a rule. An observation is when you look at something and describe it’s state. It can sound like a rule, but isn’t one.

For example, “I love you” isn’t a rule – it’s an observation. It’s describing how I am now, and not describing how I will be in the future. “I will always love you” is both a rule AND an observation. As an observation, it’s fine. As a rule, it’s not. There will be circumstances that except this rule – you die, you cheat on me, I cheat on you, I get tired of your shit, etc etc etc. The actual rule being expressed here is actually “I will always love you, assuming that everything stays the same as it is now.”

A pure rule is more like a statement of intent and result. “If you do this, that will result”. However, these kinds of statements are ALWAYS contextual.

Bruce Lee was fond of saying that if you face a man with a sword against a man without a sword, when both have the same ability, who will win? Well, obviously the man with the sword will. Bruce listens to you answer, then nods and says “What if I put that man in a phone box and gave him a gun?”.

The whole point is that the question is so contextual as to be absurd. And rules of the one line variety generally are too. There’s ALWAYS an edge case (and usually a fair number of them) that invalidates the rule or adds an extra dimension to it.

It even goes down to things like the 10 commandments, which are generally thought as basic as you can get  – “Thou shalt not kill” – doesn’t get any more basic than that. But if you had the chance to take out Timothy McVeigh before he dropped off the truck off in Oklahoma, and the only way to stop him was to kill him, hell yes you’d be justified in taking that life. The same with Osama Bin Laden, before he approved the 9/11 hijackings. Those would be completely justifiable homicides (well, they would and they wouldn’t. They would knowing what we know now, but if you just upped and killed these people, even if you knew what the results were, no one else would, and you’d be viewed as a murderer in their eyes).

So ok, now we have an exception that that hard and fast rule. So it’s not all encompassing, after all. You get where I am going here.

People who talk in absolutes when it comes to rules scare me – they’ve convinced themselves that they satisfy the root tenets of the rule, ergo it _should_ apply to everyone in the same application regardless.

Every rule is contextual – remember that the next time you blanket apply one to someone elses situation.

NOTE – that doesn’t mean that lots of rules don’t cover most of the situations they are meant to though. Just because a rule usually requires context doesn’t mean it isn’t right. Worth bearing that mind too.

Posted in Game Development, Life Musings | 1 Comment

Working for Linden Lab on Second Life – Part 2

The reality.

So yesterdays post was about what I had experienced, 3 months into the job. This one is about my experiences post that – what the real issues were, at least from my perspective.

I worked for Linden Lab for roughly 2 years. I worked on building a new international billing system, for a guy who ran their Linden Dollar Exchange (who left immediately after this was done, and it just got abandoned, since no one really picked up where he left off – I was quite annoyed about that.), I worked in the maintenance department for way longer than I had planned (I just enjoyed it and I got on really well with the guy who ran it, Don) and then I got picked up by the Biz Dev team who needed an out facing developer for external entities to talk to. Linden Lab was running big on hype and everyone wanted to work with them, but none of their API’s were really designed to be used by external companies – there was no security on them, so DOS attacks were possible and most of the time they were lacking functionality that external companies would need but that wasn’t required in house. Part of my job was to deal with external entities that wanted to extend and use Second Life functionality – look at what they wanted to do, advise them on how best to do it and design modifications to our own API’s if they needed extending (and try and persuade Linden Lab to actually do it – but more on that in a bit).

Life was good. I was working externally, at home, with family, but getting out to SF periodically.

However, it wasn’t all good. There were several downsides to life at Linden Lab that were starting to become apparent.

1) The remote thing.

It can work and often does, however it also has the effect of marginalizing those who worked on their own. Given that 60% of Linden Lab _was_ located in one place and were working there day in and out – those who were remote did tend to get put to one side. There were discussions about you and your proposals that occurred at head office that you weren’t a part of – or even aware of a lot of the time – that color perceptions of you, simply because you weren’t there every day to combat them. It only took one large personality at head office to dislike a proposal from a remote worker and effectively you were shut down, because they’d have lunches with people, present their point of view and you’d not get the chance.  That happened to me more than once and it was extremely frustrating.

Further to that point, a lot at Linden Lab came from a Linuxy background and used Linux handles instead of real names. That’s great and cute and allows for personalization, but it makes it almost impossible to know who people were when you did arrive on site – since in person they’d use their real names. Linden Lab did have a wiki where people were supposed to put their mug shots against their handles and real names, but not a lot of people actually did it – there was no enforcing of that usage and often it made you feel very stupid when you had to ask everyone who someone was before a meeting with them.

The fact is that there was a degree of remote Out Of Sight – and that’s inevitable when you have some people in an office and some that don’t. I learned a bunch of techniques to ensure that I was still present in people’s minds – being part of mailing lists, a blog, always being on IRC and Skype, sending round weekly “State of the Project” emails and so on, but still, when you miss out on hall way conversations, going out to lunch, going for drinks after work and all the other million and one small social conventions, it’s hard to perceive yourself as part of the team as those who are present on site.

2) It wasn’t the wild west any more.

When I started at Linden Lab, it was about 100 people. They’d just grown from about 40-50 people and doubled in size. Now they’d maintained the same attitudes and development methodology, but it wasn’t scaling. Up-til that point everyone had pretty much just done whatever they’d wanted, and yelled at other people in the studio to tell them what they’d done. There were enough discrete systems that you could afford to just do whatever you wanted in the implementation of any given system, since everyone was busy and only you’d be looking at it. Now it was different – every system was now in place and the success of Second Life was it’s own problem since none of these systems scaled. Now the race was on to build scalable systems, and integrate them properly, only Linden Lab wasn’t used to working in a more disciplined way.

What actually ended up happening was the inner cabal of people-who’d-been-there-forever carried on working in the way they’d always been accustomed to, whereas new arrivals were held to a much higher standard and critiqued. New arrivals were forced into justifications of whatever they wanted to do; those who’d been there a while never were and reacted quite badly when challenged on it by new arrivals.

There were several projects that were done as skunkworks projects, and when revealed when they went live, the residents of Second Life LOVED them, so Linden Lab, as a company, had no choice but to love them too – morphable primitives using a color map was one such project. When that was released in the client there were a LOT of questions asked about why no one knew this was being built, however it was such a resident hit, there was no choice but to move forward with it (and for the record, it was an awesome idea and implementation – the results seen in world, once the residents got to grips with how it worked, were stunning.).

However, events like this were reflected on internal engineers who _did_ attempt to do things the right way, by asking permission, since the programmer elite were smarting from not being asked about the skunkworks projects, and reacted quite badly to new projects being proposed legitimately. There were _always_ some edge case where the proposal wouldn’t hold up to some engineers satisfaction.

3) Colorful Personalities.

Linden Lab hired quite a few colorful personalities – they liked to hire based on results, but after a while and the scale they were getting to, the ability to work with others was becoming more and more important. There were a number of people at Linden Lab who were very competent but also hell on earth to get along with (I was probably one of them, to be fair). Working remotely and interfacing with these people was.. difficult, to say the least. I had one online meeting I held to talk about a proposal I was making and I was literally shouted down by a team leader who felt the proposal should have been under his wing (and he was right; it should, however he’d done nothing with it in a year, and I was getting external pressure from users for this feature set) and I literally couldn’t speak without being shouted down.

Another time I remember, early on,  having a code review from someone I’d never met, who held no authority over me at all, who read me the riot act over skype, chastised me for not following coding standards that didn’t exist and was generally aggressive and rude. I had no idea who this person was, why he was looking over my code, what his positions was or anything, he just showed up on Skype and started laying into me. And I let him because at the time I didn’t really understand that this really was a programmer pissing contest and he was staking his claim.

This kind of thing happened a LOT, however Linden Lab was never going to do anything about these colorful personalities – none of them were disciplined, pointed at better ways to behave or, at a last resort, let go. There was, really, no place to even complain to, since the company was so flat.

Most remote workers I talked to acknowledged that problem people were in Linden Lab, but they each shared with me techniques they had employed to avoid working with these people. However, how effective can you be when you are constantly having to do this? You just learned who you didn’t get on with, and arranged things such that you didn’t encounter them.

4) The data scaling problem.

All of Second Life’s persistent data was sitting on a single MySQL DB, that did NOT scale. Linden Lab was faced with the problem of what to do – they’d tried clustering MySQL boxes but a) they kept falling over and b) you still face the One Apache Server bottle neck problem of all requests going through it.

In the end they decided to split the main DB into multiple parts and have a complex system of administration that each system asked yet another system for info as to where it should be sending it’s queries and communications. It was complex, over engineered and badly communicated to the engineering staff, none of whom knew what they were supposed to be doing or how to use it.

It also revealed the issues with how Second Life had been built to-date – since each system was built by one individual with almost no oversight, each system was built using a language and 3rd party libs that only the original developer could love – it ended up being a system cobbled together using multiple versions of PHP, python, C++, Perl, Apache servers, strains of MySQL and even some C#. Watching a message go from client to the end server and back again often required 8 or more different windows open and a networking degree from MIT.

Another aspect of the problem was that Second Life was LIVE. All the data was flowing in real time. If you brought a replacement system online, it HAD to be at least as complete in functionality as the one it replaced, because there were undoubtedly systems down the line that relied on all functionality it offered. You had to 100% understand what a system did, why, what the history of it was, why there was one edge case that it handled differently for one specific situation (and there were a LOT of these), and then you had to build something better, faster and more scalable. Then bring it online at once, and hope to god it worked. Looking back, I’m actually surprised that the grid didn’t go down far more than it did.

It was complex, hard to understand and I think telling that when one of the most accomplished and experienced engineers in the company – who’d been there from day #1 – announced that he could no longer hold all of the code base – with all it’s spurs and different language requirements – in his head. It was just too complex and crufty at this point.

5) Scaling for Enterprise / Stock Offering / wherever it would end up.

In terms of a comprehensive strategy of where Second Life was going – where they wanted to be 10 or even 5 years from then, there wasn’t one. There were a bunch of projects that were ongoing that promised big things, but none of them were the company focus. Several of these projects were very promising – Grid in a Box was one of them.

Grid in a box was a project where the entire Second Life support system was packaged up so it could be deployed on one box or a combination of boxes. This would enable an enterprise system to be sold, so your company could buy it’s own grid, maintain it, and hold it privately, yet _still_ connect with the main grid. So you could bring in content made in the main grid, yet you owned the machines you were running, could control the content and who could see it and who couldn’t. To be clear, Second Life already had these controls, but the servers were still located where Linden Lab said they would be.

Several companies had expressed interest in being able to run their own grid, but in order to make this a reality – where we could release some of the tools (which were hard coded to work only with Linden Labs grid and some of which were, in the kindest possible way, hacks), and have integration with the main grid, there was a LOT of infrastructure work to do. At the same time, there was a plan to open up the integration API’s, so Second Life could talk to other Virtual Worlds of the time. So the work became double – the integration API’s had to be generic and they had to exist at all. A technical plan was devised that was effectively so high level and abstract – in order to fulfill all the competing requirements of the different systems – as to be incomprehensible in terms of actually building anything.

Part of the point was to get around the new gambling laws. If servers were located in the UK, there was nothing the US could do about shutting down gambling in Second Life – and by providing Grid in a Box, it wasn’t as though Linden Lab was running those servers itself either.

It was a huge undertaking and one that should have consumed and been the focus of the entire company – but it wasn’t. There was another large project to push tools into the web, so you could create and modify an avatar via the web, and yet another one to build the infrastructure so residents could build web stores for their wares – you wouldn’t need to go into the Second Life client at all to buy stuff (in the end, Linden Lab ended up buying a 3rd party company who were already offering this facility).

Linden Lab was also on an acquisition binge – buying and trying to integrate Windward Mark Interactive, and create a Boston Studio at the same time.

There were other projects, all competing for time and resources and money. Effectively the company didn’t know what it wanted to be, where it was to spend it’s time and effort and be cohesive.

It brought on board a new CFO to try and get it’s finances into order, and who did a great job, although he despaired of trying to get any projections together because of the “as and when” attitude Linden Lab had to spending money. The “no travel budget” aspect went away because with no travel budgets, it was impossible to plan financially. It was understood internally that if a Stock Option offering was ever going to happen (and that was always the plan, internally. We were all watching other companies go public and looking at each other, giving each other tight smiles when colleagues across town suddenly became millionaires over night.), Linden Lab had to work as though it was already a public company – but on hiring a ‘real’ CFO with that kind of experience to start preparations, it became obvious very quickly that Linden Lab couldn’t carry on with either it’s transparency or it’s book keeping or the way it handled money, because it was inherently unpredictable, and Wall Street would never have stood for that. Certain aspects just had to change and there were internal battles between the finance group and the rest of the company who were quite happy with the way things had worked to-date, thank you very much.

The CTO was, at this point, effectively missing in action. We’d all get emails from wherever he happened to be in the world at that moment, and what Important Person he was speaking to that day. It was great to see us represented at White House level meetings, but in terms of the CTO actually guiding the technical direction of what Linden Lab Engineering was doing, not so much. He was a terrific ambassador, and was also mostly single handedly responsible for the code base that Second Life sprang from, and also just a nice guy, but being brutally honest there was very little CTO work going on at this point, and lots of travel and meeting people.

The fact is that Linden lab honestly didn’t know what to do with it’s success and had no concerted direction to move in, which really was consistent (if you think about it) with it’s Every Man is an Island of Development culture. It was more concerned – at that time – with how to scale and find new ways to work doing what it was already doing rather than looking to the future about what it wanted to do and be.

6) The pain of Starting new projects or No, We aren’t going to work with external companies.

As I mentioned earlier, I was part of the small 5 man biz dev team at Linden Lab. Basically, at it’s heyday, Linden Lab was being contacted by everyone and his dog eager to get on the band wagon. At the time, Linden was growing, painfully, and was more inward looking than outward looking. However, the biz dev group was run by a powerful and strong personality and he saw all these opportunities and wanted to take advantage of these events. He pursued these enquirers vigorously, and encouraged us to do the same.

However, quite a lot of these projects would have involved quite a lot of modification to internal API’s (or at the least, security strengthening them) – they would have had to be adopted as new projects internally and been something that Linden Lab would throw it’s weight behind as a company. As it happened, Linden Lab’s top level had noticed that the traditional “Ask forgiveness not permission” way of starting new internal projects wasn’t scaling – you tend to get 300 people all frantically rowing in their own direction and getting in each others way doing that, rather than all rowing in the same direction. So, the whole “Individually directed work” process was being retired somewhat, and individual studios given direction on what projects they should work on. You’d still get your choice of what studio to work in, but once there, your work load would be dictated by what the studio was working on.

However the process of proposing a new internal project was… flawed, to say the least. In attempt to be democratic, lots of people were consulted, you had to make a presentation, justify why this project was something that Linden Lab was something it should be working on. However, the process lurched from saying “Yes” to everything presented, to then course correcting and saying “No” to everything while the revolving group of judges got to grips with what the job entailed. There was also a distinct lack of Big Picture thinking – each department was only considering impact to them – operations and maintenance didn’t want to support anything that would add to their burden of support, even if it was good that the company as a whole, for example.

After three such presentations that were no brainers in terms of supporting external companies wanting to do business with Linden Lab were turned down, it had become fairly obvious that while we in the biz dev group were frantically drumming up new business with external companies, Linden Lab as a cohesive group was just not that interested in following up on these opportunities. Now, at the time, I would have considered that a failure of imagination, but hindsight and time has enabled me to view these decisions in a different light.

At the time Linden Lab was trying to grow, scale and find new ways of working. They’d successfully built what they had using non standard culture and they were attempting to do the same at a larger scale, trying this and that method and discarding or modifying as they went along. This is great and exciting, but it’s also painful. Introducing projects to support external entities when you have enough issues internally to keep you grinding for the next 12-18 months isn’t a smart move.

Added to that, most of the projects the biz dev group where bringing to the table were dubious in terms of actual financial return; they were certainly ‘cool’ projects, and would have brand extended Second Life – and looking at the IP now, it would have definitely benefited from it had they gone ahead – but at the time, if it wasn’t going to immediately make Linden Lab money, it wasn’t that important. They didn’t need the brand extension at the time, since the Second Life brand was so big and hyped anyway.

The projects simply weren’t worth the effort that Linden Lab would have put into them and looking back, rightly so, although my personal feeling that some of the No decisions came about due to lack of clarity about the company mission and the New Project Approval process, rather than a comprehensive understanding of what Linden Lab was and should have been working on at the time.

At the end of this, the internal exec group just decided the Biz Dev Group wasn’t that useful and several of us were let go or reassigned – the only thing they carried forward was the development of the Enterprise version of Second Life in conjunction with IBM.

At the time I wasn’t that surprised – it had become obvious that we weren’t operating within the remit of what the Exec Group / Board of Linden Lab felt was important and as such we were an un-necessary luxury. We _should_ have gone when we did – I certainly hold no hard feelings over that personally.


Now I stress that all this was in 2007/2008. This blog is a look back, not any kind of expose of what Linden Lab is now. I’m sure that it’s stabilized and has good processes in place now – I can’t comment as to that myself though.

And yet, for all of the above, I am glad and grateful I got to work there. I got exposed to new ways of doing things, and exposed to a CEO who was as out of the box as he was smart. I learned that you could be successful at the top and NOT be an asshole – thank you Philip, for educating me in that – and I learned that Linden Lab had some truly awesome people in it – people I still like to call friends to this day (Jim Purbrick, thank you for all the conversations. You helped immeasurably).

There were some lessons I learned along the way that were stunning in their simplicity – the grid management system Linden Lab has is amazing – it’s almost 100% automated – one person can effectively manage thousands upon thousands of CPU’s, and the tools provided (in real time) I’ve never seen anywhere else.

The trust they placed in their people was amazing and I’ve never seen anywhere as open and transparent as Linden Lab.

It was an exciting time, literally making the road up as you went along. It’s a shame that it seems to have lost its way and the stock offering never happened, but it’s still The Best virtual World out there and almost certainly one hell of a place to work.

Posted in Linden Lab, Second lIfe | 3 Comments

Working for Linden Lab on Second Life – Part 1

Inspired by this truly wonderful piece of personal history written by Dominic Connor over at The Register about his time working on OS2, (http://www.theregister.co.uk/2012/11/23/why_os2_failed_part_one/ and http://www.theregister.co.uk/2012/11/26/os2_final_fail/ – seriously, if you’ve got 20 minutes and were around then or are curious about the Windows Competitor Made By Microsoft, OS2, this is a must read. Well written and full of information), I thought I’d add a two part post of my own.

This is about my time working at Linden Lab during it’s boom time, when it was at it’s most hyped. I spent 2 years working for Linden Lab on Second Life – working from Arizona and hitting San Francisco every 6 weeks or so.

The first post is basically a post I wrote for a forum I frequent of game developers, where I was trying to drum up people to hire. Linden Lab was exciting, interesting, something completely different and I wanted people to know about this. Hopefully we’ll have some Linden Lab Alumni chiming in.

Anyway, the original post was called “Linden Lab – Company 2.0″ – please remember this was written at the time when I was deeply drinking the koolaid:)

Also Note – this was how Linden Lab was then – in the years of 2007-2009. Linden Lab’s corporate culture changed fairly radically when Philip Rosedale, the then CEO and founder stepped down and Mark Kingdom was hired to replace him. One of the reasons Mark was hired was to prepare and set up Linden Lab for a stock offering (there were always hints and rumors that this was going to happen – indeed all the workers at Linden Lab were gifted stock in preparation for this event) – he was more conservative and wall street would have accepted him in a way it would never have accepted Philip Rosedale. However, we’ll talk about that more in part 2.


The Linden Lab culture.
the pros

Wow, is it ever different. It’s so far from your traditional development model that you can barely see that in the rear view mirror. For a start it’s almost completely flat. I’ve been here 3 months and I’ve still no idea who I report to. There are directors who own ‘studios’ – but that’s less about people as it is about functionality. There’s a studio that owns grid maintenance and upgrades, another that owns bug fixing and so on. You can work within a studio, or not, your choice.

In fact pretty much everything here is your choice. There’s no “here is your task list” kind of approach, there’s just this whacking great Jira task list that people enter bugs, or new feature requests into, which gets prioritized, and you just go in and look for something to do. You get to choose where you spend your time and what you work on – if you have a pet project then you stuff it into Jira and go work on it. There’s an understanding that obviously you try and work on stuff that’s a high priority first, but the understanding that you might want to spend some time experimenting with a new lighting method is ok too.

As a developer it’s cool because you aren’t streamed into one particular area and expected to specialize in just that. I can choose to work on web stuff, grid / simulator stuff, CSR support tools, client code, the whole gamut is open to me. Its up to me where I go, so my experience base suddenly becomes very broad – right now, for example, I’m writing a billing system in PHP, next I’ll be doing maintenance on our grid and then looking at client viewer optimizations. And this is encouraged, which is even cooler.

New developers spend some time in maintenance in order to get familiarity with the codebase and how things work here, which makes sense; it gives you the tools to be able to be productive across the board, plus you get experience in almost all aspects of the Linden Lab business.

Since it’s entirely self driven, obviously the emphasis is on self driven performance. Everyone is expected to contribute and there are people watching to ensure you are, as well as people constantly looking for ways to make the employees more successful – there’s been a particular emphasis lately on the new employee experience since they are expanding a fair bit.

There’s a love machine. Yes, a love machine. What does it do? Well, it’s a way to promote positive feedback and feeling among the people that work at Linden Lab. Basically it’s a web site where you can go and send love to a specific person for a specific thing. “Jakes Sends Love to Billc for helping out with the logging in the new billing system” for example. The system then sends an email to that person saying he’s been sent love, and records it in a big old list that anyone can peruse. Since you have to justify why love was sent you can’t just game the system, nor can you send love to yourself. Every quarter the love is actually tallied up and a dollar value assigned to each love entry (usually 1 or 2 dollars per entry) and the balance is added to your paycheck, and the system reset.
I’ve never actually seen anything like it. It’s quite unique.

The bonus scheme is very forward thinking too. For example, every quarter you are given 2000 points of bonus. 1000 points are just given to you, flat out. Thanks for working here. The other 1000 points you are given, you have to distribute among your co workers, in any way you see fit. You can give them all to one person if you like, or you can hand them out per task (I want to give X bonus points to everyone who worked on adding Voice to Second Life for example), or per group (I want to give X points to everyone who works on the web team) or any combination thereof. Obviously you can’t just work out a scheme where you give it all to your buddy and he you – that’s looked for – but otherwise it’s up to you. The bonus points them selves are valued by making the first grant of 1000 points as a percentage of your salary and the second as a percentage of the amount of money in the bonus pool – usually made up of a percentage of the profit from the last quarter.
The interesting point about this is that while each individual usually knows of about 10% of the company what they really did and where he/she thinks the money should go, everyones 10% covers a different part so it tends to balance out.

They encourage working out of house and working at remote studios. They are actively promoting building smaller satellite studios of smaller teams where you can work if it’s more local to you. For instance they already have offices in Mountain View (south of SF), two in SF, a Seattle studio, one in Boston and they are in the process of building one in Brighton in the UK. Remote working is not the problem it is for most companies because of the way the company is set up in the first place, ie everyone being their own island of work.

There’s no crunch. None. How? Because they roll out new simulator code / client code every couple / three weeks rather than doing the burst based release of a game traditional game development does. Its more like an MMO maintenance team than traditional game development. Don’t get your stuff done for the next roll out? Don’t worry about it – there’s another one coming in a couple of weeks. Wait until then.
Thats not to say that on the roll out of new systems there may not be some late nights of preparation, but nothing like death marches of weeks of crunch. It’s literally a few late nights and you are done and onto the next thing.

There’s no travel budget. You want to go to GDC/Some open worlds expo/whatever? Buy yourself a ticket, go, write up a report, report what it cost and judge whether it was worth it or not, then expense it. You are considered to be your own judge of how money is spent in that way. If you need hardware or software or whatever, then just go get it (or ask ops to get it for you since they may get a better deal) and just do it. One of the surprising things is when you are actually responsible personally for spending money on trips and so on, you don’t tend to do it unless you can really justify it. They actually spend less on travel than most companies do because they don’t follow the traditional travel budget approach.

Everything is completely 100% transparent. Every month everyone is sent a copy of the company financial statement. This covers everything from how much was spent on salaries (obviously not down to the specific person) to how much was spent on rent. Everyone is able to ask questions and an open meeting is held to do so every month.

There is a weekly ‘brown bag” lunch held at the main offices (and sometimes within Second Life itself – I’ll explain more about that later) where the CEO basically poses and question via email before hand, then everyone who wants to sits down at lunch time with a company supplied lunch and discusses that thought / question / whatever that the CEO posed. And boy, some of those questions are just stuff you’d never hear anywhere else. Here’s some examples “Satellite offices – way of the future, or are we just kidding ourselves?” – “The bonus scheme – does it really work? What do you think?” – “Should I still be CEO or should be find a ‘real’ CEO to run LL” – yeah, I doubt *your* head of studio is going to ask you that.

One of the nicest things about this company is the recognition that it’s doing new stuff and working on new paradigms of ways of work. They recognize that what they do doesn’t always scale, or work out great. Every thing is pretty much considered to be a work in progress. Which means when something isn’t working, or scaling, or there’s a better way of doing something they want to hear about it. I just pointed out some issues with new people working remotely and everyone who saw my email on the subject considered it a call to arms to fix this stuff. I’ve never been *anywhere* where that was thought. It was quite quite refreshing.
They even went so far as to say “ok, lets bring you on site again to try integrating you again into the maintenance group and that should clear up some of these issues”. And now, in the middle of it, it was the right thing to do. Now I’m being way more productive.

The cons
Because of the One Man is an Island approach, knowledge sharing tends to be harder to manage. There’s an enormous wiki full off every part of Linden Lab culture and details, but now it’s so large that navigating it and finding the bit of information you want is somewhat hard. Google search is really helping there, but it still needs to be managed full time by a technical writer (which LL has recognized and is hiring someone to do just that).

Communication is often an issue. With everyone spread everywhere (even across two campuses in SF) sometimes getting hold of someone who holds the info you need can be a challenge (and in some cases, working out who has that nugget also can be an issue). Everyone is encouraged to be on an in house IRC channel, and available on Skype, but even then sometimes it can be hard to get what you need in a timely fashion. More often than not information is given when asked for, but not volunteered, which is a problem – but that has to do with busy people more than anything.

Quite often there is some nugget of information about how something is done, or a process, or something of that ilk that is available on the wiki, but you don’t even know to go and look there because the process or tool or whatever is just so esotric that you had no idea it existed to go and ask about it, until you’ve done something that broke it and everyone starts glaring at you (metaphorically speaking). For example I just broke the open source build because I created a project that was more than one directory deep, yet had no idea we even *had* a tool that created an open source version of the code base that only parsed one directory deep. Once I was told I looked in the wiki, and yes, buried in there, there was a an entry on it. However I had no idea to even look for it in the first place.

Further to that, there isn’t as disciplined a technical design approach as you might imagine and sometimes getting something new started can be a bit like running in molasses – because of the flatness and everyman is an island kind of thing, everyone can have an opinion (and usually will) and everyone is entitled to chime in, and that’s encouraged. But when everyone can comment it doesn’t take long before conflicting idea’s come to the fore and things start to get bogged down. It’s harder to just pick a direction and run that way when you’ve got everyone pointing out forks that could be (and should be in their opinion) taken. While it’s YOUR decision as the implementor, it can be hard to justify why you feel the way you do, particularly when there’s two equal methods of doing stuff. It takes a strong personality to just plow ahead anyway, since there’s no way to please everyone and no organized structure to really just hand the word down on high on what is going to be done and how. It’s a great approach, but it does require some bullishness to get moving on although it should also be pointed out that when the right people are together, this freeing up of approach means you can get stuff implemented very quickly and be very nimble too.

This is often one of the hardest things to deal with when you come in from a traditional hierarchical structure.

There is no IT. Since LL is considered to be a technical company, it therefore follows that everyone should be technical and therefore responsible for their own equipment and what goes on it. You are 100% responsible for your dev environment (and you can use whatever you want), but it’s up to you to support yourself and fix any problems that arise. Literally, when you arrive at LL you are given three boxes (containing a PC, a monitor and a Macbook Pro laptop) and a piece of paper with your password on it, and a link to the wiki and it’s all up to you from there on in. It’s scary but exhilarating at the same time. You are actually trusted from moment one to use your time wisely.

Lots of time can be wasted trying to get Parallels to install and work, or navigate your way around an unfamiliar GUI or whatever. When something goes wrong, someone may help you if they have experience, but more likely it is up to you to diagnose the problem and fix it.
I think this is going to get addressed sometime soon – the problem is how do you do IT for a company that prides itself on allowing individuals to create their own dev environment? It’s a hard problem to chew.

The whole cross platform thing can be daunting. The grid runs on linux, but there’s a windows version of it to run locally. The client is both Linux, Mac and PC and all changes have to be compiled against all platforms, so you’ve better be familiar with building in all 3. Because of the cross platform-ness there’s a fairly draconian set of coding standards because things like headers and so on need to be forward slashed and not back slashed and so on. Things that a PC developer would never care about become a priority because of Linux and the Mac. It’s not a bad thing per-se, it’s just a shock to the system because development tends to slow a fair bit because of all this constant need to ensure cross platforming works. When it comes to things like memory allocators and so on, then doing it for 3 different platforms tends to be an inertia factor.

And added to that, the whole Open Source thing. This is a double edged sword. Sure, you get fixes to your client code from the outside world, which is nice, but someone has to vet those fixes and as with most things that lots of people use, any change will be something someone somewhere depends on and you WILL get to hear about it. Plus, it makes including SDK’s and other stuff a royal pain in the ass, since you know need a release agreement with whomever you get the code from so you can release the code, or the lib, or whatever. It does stop you using quite a lot of 3rd party stuff – I wonder how we’d try and use something like Granny in this case.

The codebase itself has plenty of places where people can shine. There is *always* more work on expanding features and fixing stuff that’s incomplete or buggy than there is people, which is why LL is expanding so much. The number of residents that LL is supporting is staggering (Wow numbers) and this means trying to scale systems that were built with one tenth of the people we have now online in mind. Thats no reflection on those people who wrote the original systems (Cory in fact was responsible for many of them), and is in fact a testament to the fact that they are still working now, albeit a bit slowly. But some of the code is a bit crufty, and sometimes it’s bit hard to understand why a particular approach was taken, and chances are there’s no documentation explaining it. Some of the code base is just a thing of beauty. And some is not.

And lastly, there is *always* a vocal minority that use the client that will complain about anything. If you don’t fix teleporting, but add voice feature, someone will complain about the fact that you should have fixed teleporting first. If you fix teleporting but don’t add voice, then someone will complain about that. There’s no way to please everyone and believe me, everyone has an opinion they are all too happy to express to you, particularly when you are in world as we Linden employees increasingly are these days. It can be demoralizing when all you want to do is make things better, but the residents for whom you are doing it just don’t understand what / why you are doing what you are. I think anyone who works on a released MMO can empathize here.

Part 2. What can Second Life do for you?

Second Life isn’t a game. It’s a small nation and rapidly expanding. We now have to tax people for in world transactions and we are doing effectively what a small nation does. Linden Lab took some unusual steps when starting up Second Life in so far as they grant ownership rights to anyone who creates anything. Since creation of objects / scripts is all in world / in client, anyone can build anything they want (and of course they do). But they *retain* the rights to ownership, as well as the responsibility for ownership. If you create stuff that is unacceptable (child porn stuff, gambling etc) and we find it (and we will) you are banned and/or the local authorities alerted. This gets a bit difficult because Second Life is so multi-national – who do we actually alert when we come across stuff? But we are doing everything we can to stop stuff like this, without becoming a censorship driven medium. The whole idea here is to foster individual and social moral responsibility rather than act as an overlord and dictate terms.

There are tools so you can sell stuff and not have people forward that item on and so on to promote the concept of ownership which tends to work pretty well. This is the real secret to Second Life, if there is one. The freedom to build what you want,own it and sell it on, is where the real popularity comes from. Linden Lab actually have no content creators at all (well, one or two I think) – literally everything you see in Second Life comes from the residents building stuff in world. It’s hard to believe when you go in there and wander around, but it’s true. This is why something like Sony@Home is never going to be a threat, because they don’t subscribe to this philosophy.

What does this mean? Well, since you buy in world currency on our Lindex Website (think a commodities website. You want to buy Linden $? Well, that guy over there is selling some at price X – you can just buy them from him at X USD. He gets the USD and you get the Linden$ – this does mean the price for Linden$ fluctuates based on demand – like a real commodity – but we govern that by selling and buying Linden$ ourselves) your linden$ balance is actually worth real money. You can actually cash it out into real money too, only small bits at a time (think ATM withdrawal rules) to avoid people using this as a money laundering operation. We do actually see some currency speculators within Second Life, but although it’s possible it’s not really practical to run a business on that approach.

But the practical upshot of which is that it is possible to actually run a business within Second Life. And many do. There are now at last count over 200 ‘real world’ businesses creating content for others in Second Life, plus thousands of individuals making and selling clothes and so on as one man operations. What does these 200 businesses actually create? Well, everything and anything. You can lease an island in Second Life but that just gets you land. What do you do with that land? You could learn all the building tools and start making buildings yourself, or alternatively you could just commission one of these businesses to make it for you. The fact is that like video game development, how you build your assets will affect the on screen speed that your ‘level’ (or island) runs at. Better building approaches use less primitives and therefore run faster for physics collisions, rendering etc. These content creation companies know how to do this properly. You pay them either in Linden$ or real world currency outside of SL and they’ll come in, design you a building and so on.

You might want small objects you can hand out, or perhaps a game on your island – whatever. These companies exist to build this stuff for you.

Now as an individual that’s probably overkill, but for large companies (like IBM, or the Swiss Consulate) that want a Second Life presence, this kind of help is essential. You might think this is a bit of a one hit wonder kind of business to run, yet every one of these companies is desperate for content creators to help build stuff, since they are over whelmed by work. In fact it’s likely that Millions Of Us (the largest and best known of these content creation companies) may well eclipse Linden Labs profits in the next couple of years or so.

But why would someone like IBM want a Second Life presence in the first place? Well many companies now use Second Life both for advertising – Chrysler sells cars through Second Life. Scary, but true – and also as a place for virtual conferences. It’s far cheaper to use Second Life as a virtual conference tool than it is to buy expensive video equipment. Recently we’ve added voice communication (via Vioxx servers) which is in the process of being rolled out. Internally we at Linden Lab are starting to use this more and more as a conferencing tool, which is particularly effective given the spread out nature of how Linden Labs operates.

I know it sounds a bit, well, weird, but does actually work. You can even reference web pages / streamed video via primitives you can build really easily to demonstrate stuff. We are working on expanding out the capabilities of this kind of approach to make Second Life even more friendly for this kind of thing (being able to draw locally like on a grease board and see it instantly in world, and so on). There are obviously tools to make conferences private and so on.

Right now people are barely scratching the surface of what is possible in Second Life. It’s definitely a work in progress, and always will be, and Linden Lab is always looking for new ways to both shore up whats there and also expand out with new features. Sometimes Linden Lab drives that new feature, sometimes it’s people in world who start doing something in a primitive fashion and LL looks to give them better tools. Either way, Second Life isn’t going away and isn’t sadville (alright, there are parts of it that are), as The Register would like everyone to believe.

I just thought you guys might be interested, since to me this is SUCH a different environment to be in, and it’s just….well, damn interesting.


So there you go. Originally written back in 2007 – a glimpse into what working for Linden Lab was like, in rosey terms. The next post will deal with what my experiences were 2 years later – how things had changed and why I got laid off.

Posted in Linden Lab, Second lIfe | Tagged , , , , | 2 Comments