Working for Linden Lab on Second Life – Part 2

The reality.

So yesterdays post was about what I had experienced, 3 months into the job. This one is about my experiences post that – what the real issues were, at least from my perspective.

I worked for Linden Lab for roughly 2 years. I worked on building a new international billing system, for a guy who ran their Linden Dollar Exchange (who left immediately after this was done, and it just got abandoned, since no one really picked up where he left off – I was quite annoyed about that.), I worked in the maintenance department for way longer than I had planned (I just enjoyed it and I got on really well with the guy who ran it, Don) and then I got picked up by the Biz Dev team who needed an out facing developer for external entities to talk to. Linden Lab was running big on hype and everyone wanted to work with them, but none of their API’s were really designed to be used by external companies – there was no security on them, so DOS attacks were possible and most of the time they were lacking functionality that external companies would need but that wasn’t required in house. Part of my job was to deal with external entities that wanted to extend and use Second Life functionality – look at what they wanted to do, advise them on how best to do it and design modifications to our own API’s if they needed extending (and try and persuade Linden Lab to actually do it – but more on that in a bit).

Life was good. I was working externally, at home, with family, but getting out to SF periodically.

However, it wasn’t all good. There were several downsides to life at Linden Lab that were starting to become apparent.

1) The remote thing.

It can work and often does, however it also has the effect of marginalizing those who worked on their own. Given that 60% of Linden Lab _was_ located in one place and were working there day in and out – those who were remote did tend to get put to one side. There were discussions about you and your proposals that occurred at head office that you weren’t a part of – or even aware of a lot of the time – that color perceptions of you, simply because you weren’t there every day to combat them. It only took one large personality at head office to dislike a proposal from a remote worker and effectively you were shut down, because they’d have lunches with people, present their point of view and you’d not get the chance.  That happened to me more than once and it was extremely frustrating.

Further to that point, a lot at Linden Lab came from a Linuxy background and used Linux handles instead of real names. That’s great and cute and allows for personalization, but it makes it almost impossible to know who people were when you did arrive on site – since in person they’d use their real names. Linden Lab did have a wiki where people were supposed to put their mug shots against their handles and real names, but not a lot of people actually did it – there was no enforcing of that usage and often it made you feel very stupid when you had to ask everyone who someone was before a meeting with them.

The fact is that there was a degree of remote Out Of Sight – and that’s inevitable when you have some people in an office and some that don’t. I learned a bunch of techniques to ensure that I was still present in people’s minds – being part of mailing lists, a blog, always being on IRC and Skype, sending round weekly “State of the Project” emails and so on, but still, when you miss out on hall way conversations, going out to lunch, going for drinks after work and all the other million and one small social conventions, it’s hard to perceive yourself as part of the team as those who are present on site.

2) It wasn’t the wild west any more.

When I started at Linden Lab, it was about 100 people. They’d just grown from about 40-50 people and doubled in size. Now they’d maintained the same attitudes and development methodology, but it wasn’t scaling. Up-til that point everyone had pretty much just done whatever they’d wanted, and yelled at other people in the studio to tell them what they’d done. There were enough discrete systems that you could afford to just do whatever you wanted in the implementation of any given system, since everyone was busy and only you’d be looking at it. Now it was different – every system was now in place and the success of Second Life was it’s own problem since none of these systems scaled. Now the race was on to build scalable systems, and integrate them properly, only Linden Lab wasn’t used to working in a more disciplined way.

What actually ended up happening was the inner cabal of people-who’d-been-there-forever carried on working in the way they’d always been accustomed to, whereas new arrivals were held to a much higher standard and critiqued. New arrivals were forced into justifications of whatever they wanted to do; those who’d been there a while never were and reacted quite badly when challenged on it by new arrivals.

There were several projects that were done as skunkworks projects, and when revealed when they went live, the residents of Second Life LOVED them, so Linden Lab, as a company, had no choice but to love them too – morphable primitives using a color map was one such project. When that was released in the client there were a LOT of questions asked about why no one knew this was being built, however it was such a resident hit, there was no choice but to move forward with it (and for the record, it was an awesome idea and implementation – the results seen in world, once the residents got to grips with how it worked, were stunning.).

However, events like this were reflected on internal engineers who _did_ attempt to do things the right way, by asking permission, since the programmer elite were smarting from not being asked about the skunkworks projects, and reacted quite badly to new projects being proposed legitimately. There were _always_ some edge case where the proposal wouldn’t hold up to some engineers satisfaction.

3) Colorful Personalities.

Linden Lab hired quite a few colorful personalities – they liked to hire based on results, but after a while and the scale they were getting to, the ability to work with others was becoming more and more important. There were a number of people at Linden Lab who were very competent but also hell on earth to get along with (I was probably one of them, to be fair). Working remotely and interfacing with these people was.. difficult, to say the least. I had one online meeting I held to talk about a proposal I was making and I was literally shouted down by a team leader who felt the proposal should have been under his wing (and he was right; it should, however he’d done nothing with it in a year, and I was getting external pressure from users for this feature set) and I literally couldn’t speak without being shouted down.

Another time I remember, early on,  having a code review from someone I’d never met, who held no authority over me at all, who read me the riot act over skype, chastised me for not following coding standards that didn’t exist and was generally aggressive and rude. I had no idea who this person was, why he was looking over my code, what his positions was or anything, he just showed up on Skype and started laying into me. And I let him because at the time I didn’t really understand that this really was a programmer pissing contest and he was staking his claim.

This kind of thing happened a LOT, however Linden Lab was never going to do anything about these colorful personalities – none of them were disciplined, pointed at better ways to behave or, at a last resort, let go. There was, really, no place to even complain to, since the company was so flat.

Most remote workers I talked to acknowledged that problem people were in Linden Lab, but they each shared with me techniques they had employed to avoid working with these people. However, how effective can you be when you are constantly having to do this? You just learned who you didn’t get on with, and arranged things such that you didn’t encounter them.

4) The data scaling problem.

All of Second Life’s persistent data was sitting on a single MySQL DB, that did NOT scale. Linden Lab was faced with the problem of what to do – they’d tried clustering MySQL boxes but a) they kept falling over and b) you still face the One Apache Server bottle neck problem of all requests going through it.

In the end they decided to split the main DB into multiple parts and have a complex system of administration that each system asked yet another system for info as to where it should be sending it’s queries and communications. It was complex, over engineered and badly communicated to the engineering staff, none of whom knew what they were supposed to be doing or how to use it.

It also revealed the issues with how Second Life had been built to-date – since each system was built by one individual with almost no oversight, each system was built using a language and 3rd party libs that only the original developer could love – it ended up being a system cobbled together using multiple versions of PHP, python, C++, Perl, Apache servers, strains of MySQL and even some C#. Watching a message go from client to the end server and back again often required 8 or more different windows open and a networking degree from MIT.

Another aspect of the problem was that Second Life was LIVE. All the data was flowing in real time. If you brought a replacement system online, it HAD to be at least as complete in functionality as the one it replaced, because there were undoubtedly systems down the line that relied on all functionality it offered. You had to 100% understand what a system did, why, what the history of it was, why there was one edge case that it handled differently for one specific situation (and there were a LOT of these), and then you had to build something better, faster and more scalable. Then bring it online at once, and hope to god it worked. Looking back, I’m actually surprised that the grid didn’t go down far more than it did.

It was complex, hard to understand and I think telling that when one of the most accomplished and experienced engineers in the company – who’d been there from day #1 – announced that he could no longer hold all of the code base – with all it’s spurs and different language requirements – in his head. It was just too complex and crufty at this point.

5) Scaling for Enterprise / Stock Offering / wherever it would end up.

In terms of a comprehensive strategy of where Second Life was going – where they wanted to be 10 or even 5 years from then, there wasn’t one. There were a bunch of projects that were ongoing that promised big things, but none of them were the company focus. Several of these projects were very promising – Grid in a Box was one of them.

Grid in a box was a project where the entire Second Life support system was packaged up so it could be deployed on one box or a combination of boxes. This would enable an enterprise system to be sold, so your company could buy it’s own grid, maintain it, and hold it privately, yet _still_ connect with the main grid. So you could bring in content made in the main grid, yet you owned the machines you were running, could control the content and who could see it and who couldn’t. To be clear, Second Life already had these controls, but the servers were still located where Linden Lab said they would be.

Several companies had expressed interest in being able to run their own grid, but in order to make this a reality – where we could release some of the tools (which were hard coded to work only with Linden Labs grid and some of which were, in the kindest possible way, hacks), and have integration with the main grid, there was a LOT of infrastructure work to do. At the same time, there was a plan to open up the integration API’s, so Second Life could talk to other Virtual Worlds of the time. So the work became double – the integration API’s had to be generic and they had to exist at all. A technical plan was devised that was effectively so high level and abstract – in order to fulfill all the competing requirements of the different systems – as to be incomprehensible in terms of actually building anything.

Part of the point was to get around the new gambling laws. If servers were located in the UK, there was nothing the US could do about shutting down gambling in Second Life – and by providing Grid in a Box, it wasn’t as though Linden Lab was running those servers itself either.

It was a huge undertaking and one that should have consumed and been the focus of the entire company – but it wasn’t. There was another large project to push tools into the web, so you could create and modify an avatar via the web, and yet another one to build the infrastructure so residents could build web stores for their wares – you wouldn’t need to go into the Second Life client at all to buy stuff (in the end, Linden Lab ended up buying a 3rd party company who were already offering this facility).

Linden Lab was also on an acquisition binge – buying and trying to integrate Windward Mark Interactive, and create a Boston Studio at the same time.

There were other projects, all competing for time and resources and money. Effectively the company didn’t know what it wanted to be, where it was to spend it’s time and effort and be cohesive.

It brought on board a new CFO to try and get it’s finances into order, and who did a great job, although he despaired of trying to get any projections together because of the “as and when” attitude Linden Lab had to spending money. The “no travel budget” aspect went away because with no travel budgets, it was impossible to plan financially. It was understood internally that if a Stock Option offering was ever going to happen (and that was always the plan, internally. We were all watching other companies go public and looking at each other, giving each other tight smiles when colleagues across town suddenly became millionaires over night.), Linden Lab had to work as though it was already a public company – but on hiring a ‘real’ CFO with that kind of experience to start preparations, it became obvious very quickly that Linden Lab couldn’t carry on with either it’s transparency or it’s book keeping or the way it handled money, because it was inherently unpredictable, and Wall Street would never have stood for that. Certain aspects just had to change and there were internal battles between the finance group and the rest of the company who were quite happy with the way things had worked to-date, thank you very much.

The CTO was, at this point, effectively missing in action. We’d all get emails from wherever he happened to be in the world at that moment, and what Important Person he was speaking to that day. It was great to see us represented at White House level meetings, but in terms of the CTO actually guiding the technical direction of what Linden Lab Engineering was doing, not so much. He was a terrific ambassador, and was also mostly single handedly responsible for the code base that Second Life sprang from, and also just a nice guy, but being brutally honest there was very little CTO work going on at this point, and lots of travel and meeting people.

The fact is that Linden lab honestly didn’t know what to do with it’s success and had no concerted direction to move in, which really was consistent (if you think about it) with it’s Every Man is an Island of Development culture. It was more concerned – at that time – with how to scale and find new ways to work doing what it was already doing rather than looking to the future about what it wanted to do and be.

6) The pain of Starting new projects or No, We aren’t going to work with external companies.

As I mentioned earlier, I was part of the small 5 man biz dev team at Linden Lab. Basically, at it’s heyday, Linden Lab was being contacted by everyone and his dog eager to get on the band wagon. At the time, Linden was growing, painfully, and was more inward looking than outward looking. However, the biz dev group was run by a powerful and strong personality and he saw all these opportunities and wanted to take advantage of these events. He pursued these enquirers vigorously, and encouraged us to do the same.

However, quite a lot of these projects would have involved quite a lot of modification to internal API’s (or at the least, security strengthening them) – they would have had to be adopted as new projects internally and been something that Linden Lab would throw it’s weight behind as a company. As it happened, Linden Lab’s top level had noticed that the traditional “Ask forgiveness not permission” way of starting new internal projects wasn’t scaling – you tend to get 300 people all frantically rowing in their own direction and getting in each others way doing that, rather than all rowing in the same direction. So, the whole “Individually directed work” process was being retired somewhat, and individual studios given direction on what projects they should work on. You’d still get your choice of what studio to work in, but once there, your work load would be dictated by what the studio was working on.

However the process of proposing a new internal project was… flawed, to say the least. In attempt to be democratic, lots of people were consulted, you had to make a presentation, justify why this project was something that Linden Lab was something it should be working on. However, the process lurched from saying “Yes” to everything presented, to then course correcting and saying “No” to everything while the revolving group of judges got to grips with what the job entailed. There was also a distinct lack of Big Picture thinking – each department was only considering impact to them – operations and maintenance didn’t want to support anything that would add to their burden of support, even if it was good that the company as a whole, for example.

After three such presentations that were no brainers in terms of supporting external companies wanting to do business with Linden Lab were turned down, it had become fairly obvious that while we in the biz dev group were frantically drumming up new business with external companies, Linden Lab as a cohesive group was just not that interested in following up on these opportunities. Now, at the time, I would have considered that a failure of imagination, but hindsight and time has enabled me to view these decisions in a different light.

At the time Linden Lab was trying to grow, scale and find new ways of working. They’d successfully built what they had using non standard culture and they were attempting to do the same at a larger scale, trying this and that method and discarding or modifying as they went along. This is great and exciting, but it’s also painful. Introducing projects to support external entities when you have enough issues internally to keep you grinding for the next 12-18 months isn’t a smart move.

Added to that, most of the projects the biz dev group where bringing to the table were dubious in terms of actual financial return; they were certainly ‘cool’ projects, and would have brand extended Second Life – and looking at the IP now, it would have definitely benefited from it had they gone ahead – but at the time, if it wasn’t going to immediately make Linden Lab money, it wasn’t that important. They didn’t need the brand extension at the time, since the Second Life brand was so big and hyped anyway.

The projects simply weren’t worth the effort that Linden Lab would have put into them and looking back, rightly so, although my personal feeling that some of the No decisions came about due to lack of clarity about the company mission and the New Project Approval process, rather than a comprehensive understanding of what Linden Lab was and should have been working on at the time.

At the end of this, the internal exec group just decided the Biz Dev Group wasn’t that useful and several of us were let go or reassigned – the only thing they carried forward was the development of the Enterprise version of Second Life in conjunction with IBM.

At the time I wasn’t that surprised – it had become obvious that we weren’t operating within the remit of what the Exec Group / Board of Linden Lab felt was important and as such we were an un-necessary luxury. We _should_ have gone when we did – I certainly hold no hard feelings over that personally.


Now I stress that all this was in 2007/2008. This blog is a look back, not any kind of expose of what Linden Lab is now. I’m sure that it’s stabilized and has good processes in place now – I can’t comment as to that myself though.

And yet, for all of the above, I am glad and grateful I got to work there. I got exposed to new ways of doing things, and exposed to a CEO who was as out of the box as he was smart. I learned that you could be successful at the top and NOT be an asshole – thank you Philip, for educating me in that – and I learned that Linden Lab had some truly awesome people in it – people I still like to call friends to this day (Jim Purbrick, thank you for all the conversations. You helped immeasurably).

There were some lessons I learned along the way that were stunning in their simplicity – the grid management system Linden Lab has is amazing – it’s almost 100% automated – one person can effectively manage thousands upon thousands of CPU’s, and the tools provided (in real time) I’ve never seen anywhere else.

The trust they placed in their people was amazing and I’ve never seen anywhere as open and transparent as Linden Lab.

It was an exciting time, literally making the road up as you went along. It’s a shame that it seems to have lost its way and the stock offering never happened, but it’s still The Best virtual World out there and almost certainly one hell of a place to work.

This entry was posted in Linden Lab, Second lIfe. Bookmark the permalink.

3 Responses to Working for Linden Lab on Second Life – Part 2

  1. I came to SL in 2007 (and am still fairly active there) during the “hype” and have experienced it from a resident’s point of view. I enjoyed reading your fascinating “insider’s” account of your time working for Linden Lab.

    Found this blog post via @draxtor on Twitter, btw.

  2. Great post, Jake. As one of those “remote” workers myself, I think you summed up the challenges nicely. While at Linden Lab, I felt it was tragically ironic that a company striving to build the Metaverse would have so many issues with online employees in a globally distributed workplace.

    Now I work for a company that is 100% distributed (ReactionGrid), and we put a lot of effort into knowledge management and online collaboration. As a result, I think we’re a much more effective team than most companies where everyone is working in the same physical room.

  3. Sherlyn says:

    Have you ever thought about including a little bit more than just your articles?
    I mean, what you say is important and all. Nevertheless just imagine if you added some great graphics or videos to give your posts more,
    “pop”! Your content is excellent but with images and video clips, this site could definitely be one of the best
    in its field. Wonderful blog!

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>