Inspired by this truly wonderful piece of personal history written by Dominic Connor over at The Register about his time working on OS2, (http://www.theregister.co.uk/2012/11/23/why_os2_failed_part_one/ and http://www.theregister.co.uk/2012/11/26/os2_final_fail/ – seriously, if you’ve got 20 minutes and were around then or are curious about the Windows Competitor Made By Microsoft, OS2, this is a must read. Well written and full of information), I thought I’d add a two part post of my own.
This is about my time working at Linden Lab during it’s boom time, when it was at it’s most hyped. I spent 2 years working for Linden Lab on Second Life – working from Arizona and hitting San Francisco every 6 weeks or so.
The first post is basically a post I wrote for a forum I frequent of game developers, where I was trying to drum up people to hire. Linden Lab was exciting, interesting, something completely different and I wanted people to know about this. Hopefully we’ll have some Linden Lab Alumni chiming in.
Anyway, the original post was called “Linden Lab – Company 2.0″ – please remember this was written at the time when I was deeply drinking the koolaid:)
Also Note – this was how Linden Lab was then – in the years of 2007-2009. Linden Lab’s corporate culture changed fairly radically when Philip Rosedale, the then CEO and founder stepped down and Mark Kingdom was hired to replace him. One of the reasons Mark was hired was to prepare and set up Linden Lab for a stock offering (there were always hints and rumors that this was going to happen – indeed all the workers at Linden Lab were gifted stock in preparation for this event) – he was more conservative and wall street would have accepted him in a way it would never have accepted Philip Rosedale. However, we’ll talk about that more in part 2.
The Linden Lab culture.
Wow, is it ever different. It’s so far from your traditional development model that you can barely see that in the rear view mirror. For a start it’s almost completely flat. I’ve been here 3 months and I’ve still no idea who I report to. There are directors who own ‘studios’ – but that’s less about people as it is about functionality. There’s a studio that owns grid maintenance and upgrades, another that owns bug fixing and so on. You can work within a studio, or not, your choice.
In fact pretty much everything here is your choice. There’s no “here is your task list” kind of approach, there’s just this whacking great Jira task list that people enter bugs, or new feature requests into, which gets prioritized, and you just go in and look for something to do. You get to choose where you spend your time and what you work on – if you have a pet project then you stuff it into Jira and go work on it. There’s an understanding that obviously you try and work on stuff that’s a high priority first, but the understanding that you might want to spend some time experimenting with a new lighting method is ok too.
As a developer it’s cool because you aren’t streamed into one particular area and expected to specialize in just that. I can choose to work on web stuff, grid / simulator stuff, CSR support tools, client code, the whole gamut is open to me. Its up to me where I go, so my experience base suddenly becomes very broad – right now, for example, I’m writing a billing system in PHP, next I’ll be doing maintenance on our grid and then looking at client viewer optimizations. And this is encouraged, which is even cooler.
New developers spend some time in maintenance in order to get familiarity with the codebase and how things work here, which makes sense; it gives you the tools to be able to be productive across the board, plus you get experience in almost all aspects of the Linden Lab business.
Since it’s entirely self driven, obviously the emphasis is on self driven performance. Everyone is expected to contribute and there are people watching to ensure you are, as well as people constantly looking for ways to make the employees more successful – there’s been a particular emphasis lately on the new employee experience since they are expanding a fair bit.
There’s a love machine. Yes, a love machine. What does it do? Well, it’s a way to promote positive feedback and feeling among the people that work at Linden Lab. Basically it’s a web site where you can go and send love to a specific person for a specific thing. “Jakes Sends Love to Billc for helping out with the logging in the new billing system” for example. The system then sends an email to that person saying he’s been sent love, and records it in a big old list that anyone can peruse. Since you have to justify why love was sent you can’t just game the system, nor can you send love to yourself. Every quarter the love is actually tallied up and a dollar value assigned to each love entry (usually 1 or 2 dollars per entry) and the balance is added to your paycheck, and the system reset.
I’ve never actually seen anything like it. It’s quite unique.
The bonus scheme is very forward thinking too. For example, every quarter you are given 2000 points of bonus. 1000 points are just given to you, flat out. Thanks for working here. The other 1000 points you are given, you have to distribute among your co workers, in any way you see fit. You can give them all to one person if you like, or you can hand them out per task (I want to give X bonus points to everyone who worked on adding Voice to Second Life for example), or per group (I want to give X points to everyone who works on the web team) or any combination thereof. Obviously you can’t just work out a scheme where you give it all to your buddy and he you – that’s looked for – but otherwise it’s up to you. The bonus points them selves are valued by making the first grant of 1000 points as a percentage of your salary and the second as a percentage of the amount of money in the bonus pool – usually made up of a percentage of the profit from the last quarter.
The interesting point about this is that while each individual usually knows of about 10% of the company what they really did and where he/she thinks the money should go, everyones 10% covers a different part so it tends to balance out.
They encourage working out of house and working at remote studios. They are actively promoting building smaller satellite studios of smaller teams where you can work if it’s more local to you. For instance they already have offices in Mountain View (south of SF), two in SF, a Seattle studio, one in Boston and they are in the process of building one in Brighton in the UK. Remote working is not the problem it is for most companies because of the way the company is set up in the first place, ie everyone being their own island of work.
There’s no crunch. None. How? Because they roll out new simulator code / client code every couple / three weeks rather than doing the burst based release of a game traditional game development does. Its more like an MMO maintenance team than traditional game development. Don’t get your stuff done for the next roll out? Don’t worry about it – there’s another one coming in a couple of weeks. Wait until then.
Thats not to say that on the roll out of new systems there may not be some late nights of preparation, but nothing like death marches of weeks of crunch. It’s literally a few late nights and you are done and onto the next thing.
There’s no travel budget. You want to go to GDC/Some open worlds expo/whatever? Buy yourself a ticket, go, write up a report, report what it cost and judge whether it was worth it or not, then expense it. You are considered to be your own judge of how money is spent in that way. If you need hardware or software or whatever, then just go get it (or ask ops to get it for you since they may get a better deal) and just do it. One of the surprising things is when you are actually responsible personally for spending money on trips and so on, you don’t tend to do it unless you can really justify it. They actually spend less on travel than most companies do because they don’t follow the traditional travel budget approach.
Everything is completely 100% transparent. Every month everyone is sent a copy of the company financial statement. This covers everything from how much was spent on salaries (obviously not down to the specific person) to how much was spent on rent. Everyone is able to ask questions and an open meeting is held to do so every month.
There is a weekly ‘brown bag” lunch held at the main offices (and sometimes within Second Life itself – I’ll explain more about that later) where the CEO basically poses and question via email before hand, then everyone who wants to sits down at lunch time with a company supplied lunch and discusses that thought / question / whatever that the CEO posed. And boy, some of those questions are just stuff you’d never hear anywhere else. Here’s some examples “Satellite offices – way of the future, or are we just kidding ourselves?” – “The bonus scheme – does it really work? What do you think?” – “Should I still be CEO or should be find a ‘real’ CEO to run LL” – yeah, I doubt *your* head of studio is going to ask you that.
One of the nicest things about this company is the recognition that it’s doing new stuff and working on new paradigms of ways of work. They recognize that what they do doesn’t always scale, or work out great. Every thing is pretty much considered to be a work in progress. Which means when something isn’t working, or scaling, or there’s a better way of doing something they want to hear about it. I just pointed out some issues with new people working remotely and everyone who saw my email on the subject considered it a call to arms to fix this stuff. I’ve never been *anywhere* where that was thought. It was quite quite refreshing.
They even went so far as to say “ok, lets bring you on site again to try integrating you again into the maintenance group and that should clear up some of these issues”. And now, in the middle of it, it was the right thing to do. Now I’m being way more productive.
Because of the One Man is an Island approach, knowledge sharing tends to be harder to manage. There’s an enormous wiki full off every part of Linden Lab culture and details, but now it’s so large that navigating it and finding the bit of information you want is somewhat hard. Google search is really helping there, but it still needs to be managed full time by a technical writer (which LL has recognized and is hiring someone to do just that).
Communication is often an issue. With everyone spread everywhere (even across two campuses in SF) sometimes getting hold of someone who holds the info you need can be a challenge (and in some cases, working out who has that nugget also can be an issue). Everyone is encouraged to be on an in house IRC channel, and available on Skype, but even then sometimes it can be hard to get what you need in a timely fashion. More often than not information is given when asked for, but not volunteered, which is a problem – but that has to do with busy people more than anything.
Quite often there is some nugget of information about how something is done, or a process, or something of that ilk that is available on the wiki, but you don’t even know to go and look there because the process or tool or whatever is just so esotric that you had no idea it existed to go and ask about it, until you’ve done something that broke it and everyone starts glaring at you (metaphorically speaking). For example I just broke the open source build because I created a project that was more than one directory deep, yet had no idea we even *had* a tool that created an open source version of the code base that only parsed one directory deep. Once I was told I looked in the wiki, and yes, buried in there, there was a an entry on it. However I had no idea to even look for it in the first place.
Further to that, there isn’t as disciplined a technical design approach as you might imagine and sometimes getting something new started can be a bit like running in molasses – because of the flatness and everyman is an island kind of thing, everyone can have an opinion (and usually will) and everyone is entitled to chime in, and that’s encouraged. But when everyone can comment it doesn’t take long before conflicting idea’s come to the fore and things start to get bogged down. It’s harder to just pick a direction and run that way when you’ve got everyone pointing out forks that could be (and should be in their opinion) taken. While it’s YOUR decision as the implementor, it can be hard to justify why you feel the way you do, particularly when there’s two equal methods of doing stuff. It takes a strong personality to just plow ahead anyway, since there’s no way to please everyone and no organized structure to really just hand the word down on high on what is going to be done and how. It’s a great approach, but it does require some bullishness to get moving on although it should also be pointed out that when the right people are together, this freeing up of approach means you can get stuff implemented very quickly and be very nimble too.
This is often one of the hardest things to deal with when you come in from a traditional hierarchical structure.
There is no IT. Since LL is considered to be a technical company, it therefore follows that everyone should be technical and therefore responsible for their own equipment and what goes on it. You are 100% responsible for your dev environment (and you can use whatever you want), but it’s up to you to support yourself and fix any problems that arise. Literally, when you arrive at LL you are given three boxes (containing a PC, a monitor and a Macbook Pro laptop) and a piece of paper with your password on it, and a link to the wiki and it’s all up to you from there on in. It’s scary but exhilarating at the same time. You are actually trusted from moment one to use your time wisely.
Lots of time can be wasted trying to get Parallels to install and work, or navigate your way around an unfamiliar GUI or whatever. When something goes wrong, someone may help you if they have experience, but more likely it is up to you to diagnose the problem and fix it.
I think this is going to get addressed sometime soon – the problem is how do you do IT for a company that prides itself on allowing individuals to create their own dev environment? It’s a hard problem to chew.
The whole cross platform thing can be daunting. The grid runs on linux, but there’s a windows version of it to run locally. The client is both Linux, Mac and PC and all changes have to be compiled against all platforms, so you’ve better be familiar with building in all 3. Because of the cross platform-ness there’s a fairly draconian set of coding standards because things like headers and so on need to be forward slashed and not back slashed and so on. Things that a PC developer would never care about become a priority because of Linux and the Mac. It’s not a bad thing per-se, it’s just a shock to the system because development tends to slow a fair bit because of all this constant need to ensure cross platforming works. When it comes to things like memory allocators and so on, then doing it for 3 different platforms tends to be an inertia factor.
And added to that, the whole Open Source thing. This is a double edged sword. Sure, you get fixes to your client code from the outside world, which is nice, but someone has to vet those fixes and as with most things that lots of people use, any change will be something someone somewhere depends on and you WILL get to hear about it. Plus, it makes including SDK’s and other stuff a royal pain in the ass, since you know need a release agreement with whomever you get the code from so you can release the code, or the lib, or whatever. It does stop you using quite a lot of 3rd party stuff – I wonder how we’d try and use something like Granny in this case.
The codebase itself has plenty of places where people can shine. There is *always* more work on expanding features and fixing stuff that’s incomplete or buggy than there is people, which is why LL is expanding so much. The number of residents that LL is supporting is staggering (Wow numbers) and this means trying to scale systems that were built with one tenth of the people we have now online in mind. Thats no reflection on those people who wrote the original systems (Cory in fact was responsible for many of them), and is in fact a testament to the fact that they are still working now, albeit a bit slowly. But some of the code is a bit crufty, and sometimes it’s bit hard to understand why a particular approach was taken, and chances are there’s no documentation explaining it. Some of the code base is just a thing of beauty. And some is not.
And lastly, there is *always* a vocal minority that use the client that will complain about anything. If you don’t fix teleporting, but add voice feature, someone will complain about the fact that you should have fixed teleporting first. If you fix teleporting but don’t add voice, then someone will complain about that. There’s no way to please everyone and believe me, everyone has an opinion they are all too happy to express to you, particularly when you are in world as we Linden employees increasingly are these days. It can be demoralizing when all you want to do is make things better, but the residents for whom you are doing it just don’t understand what / why you are doing what you are. I think anyone who works on a released MMO can empathize here.
Part 2. What can Second Life do for you?
Second Life isn’t a game. It’s a small nation and rapidly expanding. We now have to tax people for in world transactions and we are doing effectively what a small nation does. Linden Lab took some unusual steps when starting up Second Life in so far as they grant ownership rights to anyone who creates anything. Since creation of objects / scripts is all in world / in client, anyone can build anything they want (and of course they do). But they *retain* the rights to ownership, as well as the responsibility for ownership. If you create stuff that is unacceptable (child porn stuff, gambling etc) and we find it (and we will) you are banned and/or the local authorities alerted. This gets a bit difficult because Second Life is so multi-national – who do we actually alert when we come across stuff? But we are doing everything we can to stop stuff like this, without becoming a censorship driven medium. The whole idea here is to foster individual and social moral responsibility rather than act as an overlord and dictate terms.
There are tools so you can sell stuff and not have people forward that item on and so on to promote the concept of ownership which tends to work pretty well. This is the real secret to Second Life, if there is one. The freedom to build what you want,own it and sell it on, is where the real popularity comes from. Linden Lab actually have no content creators at all (well, one or two I think) – literally everything you see in Second Life comes from the residents building stuff in world. It’s hard to believe when you go in there and wander around, but it’s true. This is why something like Sony@Home is never going to be a threat, because they don’t subscribe to this philosophy.
What does this mean? Well, since you buy in world currency on our Lindex Website (think a commodities website. You want to buy Linden $? Well, that guy over there is selling some at price X – you can just buy them from him at X USD. He gets the USD and you get the Linden$ – this does mean the price for Linden$ fluctuates based on demand – like a real commodity – but we govern that by selling and buying Linden$ ourselves) your linden$ balance is actually worth real money. You can actually cash it out into real money too, only small bits at a time (think ATM withdrawal rules) to avoid people using this as a money laundering operation. We do actually see some currency speculators within Second Life, but although it’s possible it’s not really practical to run a business on that approach.
But the practical upshot of which is that it is possible to actually run a business within Second Life. And many do. There are now at last count over 200 ‘real world’ businesses creating content for others in Second Life, plus thousands of individuals making and selling clothes and so on as one man operations. What does these 200 businesses actually create? Well, everything and anything. You can lease an island in Second Life but that just gets you land. What do you do with that land? You could learn all the building tools and start making buildings yourself, or alternatively you could just commission one of these businesses to make it for you. The fact is that like video game development, how you build your assets will affect the on screen speed that your ‘level’ (or island) runs at. Better building approaches use less primitives and therefore run faster for physics collisions, rendering etc. These content creation companies know how to do this properly. You pay them either in Linden$ or real world currency outside of SL and they’ll come in, design you a building and so on.
You might want small objects you can hand out, or perhaps a game on your island – whatever. These companies exist to build this stuff for you.
Now as an individual that’s probably overkill, but for large companies (like IBM, or the Swiss Consulate) that want a Second Life presence, this kind of help is essential. You might think this is a bit of a one hit wonder kind of business to run, yet every one of these companies is desperate for content creators to help build stuff, since they are over whelmed by work. In fact it’s likely that Millions Of Us (the largest and best known of these content creation companies) may well eclipse Linden Labs profits in the next couple of years or so.
But why would someone like IBM want a Second Life presence in the first place? Well many companies now use Second Life both for advertising – Chrysler sells cars through Second Life. Scary, but true – and also as a place for virtual conferences. It’s far cheaper to use Second Life as a virtual conference tool than it is to buy expensive video equipment. Recently we’ve added voice communication (via Vioxx servers) which is in the process of being rolled out. Internally we at Linden Lab are starting to use this more and more as a conferencing tool, which is particularly effective given the spread out nature of how Linden Labs operates.
I know it sounds a bit, well, weird, but does actually work. You can even reference web pages / streamed video via primitives you can build really easily to demonstrate stuff. We are working on expanding out the capabilities of this kind of approach to make Second Life even more friendly for this kind of thing (being able to draw locally like on a grease board and see it instantly in world, and so on). There are obviously tools to make conferences private and so on.
Right now people are barely scratching the surface of what is possible in Second Life. It’s definitely a work in progress, and always will be, and Linden Lab is always looking for new ways to both shore up whats there and also expand out with new features. Sometimes Linden Lab drives that new feature, sometimes it’s people in world who start doing something in a primitive fashion and LL looks to give them better tools. Either way, Second Life isn’t going away and isn’t sadville (alright, there are parts of it that are), as The Register would like everyone to believe.
I just thought you guys might be interested, since to me this is SUCH a different environment to be in, and it’s just….well, damn interesting.
So there you go. Originally written back in 2007 – a glimpse into what working for Linden Lab was like, in rosey terms. The next post will deal with what my experiences were 2 years later – how things had changed and why I got laid off.