Open Positions at Netflix

I have an immediate need for a software engineer in test for the Customer Account group on the Web QA team at Netflix.

I have an upcoming need for a software engineer in test for the Software Infrastructure group on the Web QA team at Netflix.

I also have an upcoming need for a build and release engineer.

Any help from anyone that happens to read this message in a bottle would be most welcome.

For all of them, I need someone who has a proven ability to:
• get the right things done when there's always too much to do
• solve problems in a proactive and elegant way
• communicate clearly and succinctly

First, a description of what I'm looking for in the Software Engineer in Test -- Customer Account

THE ROLE:

Netflix has a goal of 20 million members by 2012. Part of keeping those members happy is helping them manage their own data and efficiently sharing that data with other parts of Netflix. Your team provides services to members, internal Netflix teams and external partners. These services span authentication, account information management, and authoritative answers to questions about member data.

We are in the process of consolidating our business logic into services and SDKs from where it lives today: the presentation tier and stored procedures in the database. As we untangle and streamline our code, we need a safety net of automated tests. The tools and automated tests that you build will help insure that planned changes don't cause unintended consequences. What you build will help prevent defects by finding them in the test environment and detect defects that make it out to production.

At Netflix, you will work with a team of intelligent and capable software engineers who are committed to delivering high quality software through test driven development, test automation and a deep partnership with quality assurance.

REQUIREMENTS:

• Expert in assessing software quality in an ecommerce environment using:
- Test driven development
- Java, Tomcat, JDBC and Apache
- jUnit test automation (or equivalent)
- Web services
• Tests as part of cross-functional teams and supports back-end application interfaces
• Understands relational databases and SQL deeply
• Develops and maintains production system and data monitoring tools
• Desires to work in a fast paced, evolving, growing, dynamic environment

Now, here's the description for Software Engineer in Test -- Software Infrastructure

THE ROLE:

The number of people getting their movie entertainment through the Internet is continuing to grow at a rapid pace. Between DVDs by mail and Instant Watching, Netflix wants most of those subscribers to be Netflix members. Part of keeping those current and future members happy is making sure our systems are built from components and services that will scale to handle the volume.

The Software Infrastructure team builds components that include:
• management of Queue and rental history data
• a Netflix server reference implementation
• security APIs and services
• session management APIs and services
• message bus services for logging and data transfer
• explorations into The Cloud for all of the above

You will be a key part of the core team that ensures our broader development teams are using an intuitive and robust set of services to build the website and operations applications. You will ensure that we capture and accurately deliver data, and that we do it in a timely fashion. You will ensure that our internal services meet the needs of the developers consuming them. At Netflix, you will work with a team of intelligent and capable software engineers who are committed to test driven development, test automation and a deep partnership with quality assurance.

REQUIREMENTS:

Expert in understanding, assessing and providing feedback on the quality of high traffic, n-tier, web based environments including:
• Test driven development using Java, Tomcat, JDBC/Oracle, service buses and Apache with jUnit or TestNG test automation (or equivalent)
• Cross-functional testing of back-end system interfaces

Experienced with testing:
• Traditional and non-traditional e-commerce applications
• High traffic/high availability components
• Service oriented architectures

Deep understanding of:
• relational databases and SQL
• Linux or some other unix flavor
• a scripting language like Perl or Python

Familiarity with:
• testing under open source frameworks, such as Struts, Hibernate, Spring, etc.

Desire to work in a fast paced, evolving, growing, dynamic environment

Finally, I bring you a description for what I'd like to see in a Configuration and Release Engineer:

THE ROLE:

Web Engineering at Netflix provides our public face to our customers, so it's very important that we deliver an easy to use and engaging web experience. As part of the Web QA team, you'll be responsible for the team's test servers and their environment. Other duties include herding the changes to the code base from development, through testing and out to production. You will coordinate the change process across all of the teams involved, so you need not only technical chops, but also excellent communication skills. This position is part Linux admin, part software engineer and part project manager. The ideal candidate should be equally comfortable owning bash scripts, java code and ant configuration files, and will have an insatiable craving for making the deployment process more and more push-button and more and more secure.

REQUIREMENTS:
• Excellent communication and inter-personal skills
• Working experience in web based environments utilizing:
- Java, Tomcat, Apache and other web applications
- Perforce and other source control applications
- Ant and the 'x'Unit family of test automation applications
• Skilled in Unix shell and SQL
• Desire to work in a fast paced, evolving, growing, dynamic environment

DESIRED SKILLS:
• Perl hacking
• Bugzilla maintenance and modification
• Configure and use Kintana (Mercury IT Governance Center) deployment application
• Build and manage automated build/test/deployment environments

What is Applesauce the Gateway To?

In the last few weekends of September, I was making applesauce from the Fuji and Granny Smith apple trees we planted a couple of years ago. I had just picked the last apples, except for one lonely Fuji. At the time, I wasn't sure if it would ever ripen to the point where it would come off easily when one of us happens to be there to test it. It might have just dropped off in the middle of the night, and wind up in the ant's larder instead of ours. As it was, it did come off just before Thanksgiving.

Back in September, I had just finished putting a batch of cooked applesauce into a couple of 2 quart containers and left them on the counter-top, loosely covered. I wanted them to cool a bit before sticking them in the refrigerator. It was very deeply satisfying to pick and wash and peel and core and dice and simmer. Sampling the apples at different stages along the way lit up paths in my brain leading to very comfortable and comforting places. The snap and explosion of biting into a freshly picked and washed apple is like summer fireworks. Peeling the apples, I felt the juice slip over my fingers, making the peeler and the apple slick. I wondered just how much of the juice went down the drain instead of into the sauce. I've asked Jen to put a peeler on my Christmas list and I wonder if it will make next year's applesauce any different.

Lucy had introduced me to chewing some of the peel just after it drops into the sink. You get the tartness of the skin without the moderating sweetness of much juice or flesh. It's work, chewing on the peel, but it recalls other pleasures that are satisfying because they are work and not pure pleasure. Running is like that for me as well: it's hard work, but I feel better afterward.

Coring the apples is always an adventure. Attempting to pop out the heart of the apple with the seeds while minimizing the degree of flesh that goes with them. I ponder whether an apple corer will have a better or worse core/flesh ratio, since it's an indiscriminate cylinder versus the custom "cone plus wedges" that I'm cutting out.

If you compare the dollar cost of a quart of organic applesauce at Trader Joe's to the time and effort of growing, waiting for and harvesting the apples, then processing them all the way into sauce, it seems penny wise and pound foolish to spend the time. However, like many other things in life, the main value comes down to the doing, and not the monetary value of the physical product.

Software As A Manufacturing Activity

Another book that I read recently was Goldratt's The Goal. It's gotten me thinking about software development within the context of manufacturing. If a software development cycle is constrained by one of the intermediate steps, it either will starve the downstream processes or let software of inferior quality out in front of the paying customers. Neither is a desirable outcome.

So, you have two problems from the world of moving around atoms that may be applicable to the world of moving around bits. First, the problem of coordination and scheduling that supply chain management gurus have thought about. Second, the problem of throughput and productivity that constraint based manufacturing theory has gone a long way towards optimizing.

How much can be applied directly, and how much just doesn't apply because turning out software doesn't suffer from the same scarcities as turning out physical products?

I guess that's why I'm trying to think and write about it.

Experience Chain Management

You heard about it here first, folks. I've just created a new discipline: Experience Chain Management.

It was fueled mostly by some reading I did of Lou Carbone's writing, in his book Clued In: How to Keep Customers Coming Back Again and Again.

The core of it is that every business is trying to create a valuable customer experience to keep the money coming in the door. However, nobody thinks about the internal consumers of the business processes that deliver those experiences to the paying customers. Someone upstream can sabotage the effectiveness of the downstream team members and, without even realizing it, impair the customer experience.

Simple example that I'm periodically wrestling with:

• Developer publishes a code change past the start of the testing cycle
• The QA engineer doesn't regress that part of the code because they don't know about the change
• A bad bug slips through to production
• A customer can't get what they want done, and gets angry instead

I'm thinking that there are probably some lessons to be learned from all of the work done on supply chain management. Even though you're shuffling bits instead of atoms, when you have a team turning out software that drives a customer experience, it's still an awful lot like manufacturing. You have:

• raw materials (product ideas)
• manufacturing (coding)
• assembly & testing (build, configure, QA)
• delivery (deployment & operations)

Of course, the thing I need to think about more is how downstream components and processes can block or impair the productivity of upstream partners. For that, you need to look at a different discipline (see next post).

The Morning Report

The dorkiness continues. Caught myself on film with Isaac this morning and was playing around with annotations.

Definite mst3k potential there.

I've Joined Generation Stupid

As is evidenced by the following:

And of course, being stupid, I recorded the following before realizing that the one you see above wasn't broken after all:



And then, of course, the quick obligatory shout-out to a relative.

I'm astounded

I received a multi-forwarded email from a relative older than me, and the content actually interested me! I don't know if this has ever happened to you. It's really eye opening that it's even possible that the content from this person was something worth more than a quick scan and deletion, because the probability of it being something you've seen before approaches 100%.

Regardless, here's the text:


Dear Friends and Family,

I was one of the ones who wrote my friend, Anne (as mentioned below in her letter) as soon as Sarah Palin was chosen to run for Republican V.P. I received her really informative reply plus another informative letter from a friend of hers. I think it's really good info...and not all of it is negative...a lot is strictly informational. Sorry I didn't forward it right away: since then, Sarah Palin emails are flying around me like bats in a belfry!

I have put info about Anne herself at the very bottom of this email, in case you would like to know.

Hope this is useful to you...in whatever way.

xxoome

------------------

Date: Tue, 2 Sep 2008 15:46:40 -0700

From: anne......

Subject: Sarah Palin

Big Alaskan-sized hello to one and all,

Over the holiday weekend I received several emails from friends and family asking about our relatively unknown governor who has suddenly become a name you all recognize. As I started to write an email response, I received a copy of an email sent by a woman I used to know very well. She has a way with words that I think you'll enjoy. Additionally, I will mention:

I like her as governor, but won't be voting for her in November. As another dear friend of mine (who also lives and works in Alaska - and may have even more access to the Palin administration than I do) said, "she's not the one you want just one fluffernutter away from the presidency."

The ethics reform bill she signed recently was actually the work of Alaskan Democrats who pushed it through.

While she turned down our infamous bridge-to-nowhere, she accepted the federal funds to build the road to the bridge, and built the road, which goes to exactly nowhere.

She pronounces nuclear the same incorrect way the our current president does.

Yep, she wears her hair like that all the time. Who doesn't love a sexy librarian?!? There's a reason we have bumper stickers that say, "Alaska... coldest state but the hottest governor."

Trooper-gate... if it hasn't hit the news yet where you are... it will.

Although she didn't fish this year, she and her husband have fished in Bristol Bay for years. Even though her family has been deeply involved in the fishery, she announced a week before the election last week that she would not be supporting the ballot initiative that would have safguarded the water quality of the salmon natal streams in Alaska. A no vote on the initiative made it easier for large-scale metallurgic mines (like the controversial and poorly planned Pebble mine at the headwaters of Bristol Bay) to pollute, redirect and otherwise destroy the water that supports people and salmon resources. Her vote swayed thousands of undecided voters and ultimately defeated the initiative. Our salmon aren't in trouble yet, but yours??

I could go on and on, but... Anyway, below is the email referenced above. Enjoy!

Anne

She is very scary on a number of levels....one big one that Nancy identified is that she'll be underestimated.

D

-----Forwarded Message-----

>Subject: Sarah palin
>
>Hi Everybody,
>
>Five of you have written and asked about our Governor Palin as VP, so
>here's what I know from Halibut Cove. First off ... every single person in
>Alaska is just as surprised and amazed as everyone else about this. I
>was in Anchorage on Friday listening to NPR and when I first heard the news - I
>thought it was a joke or just hypothetical. Then the phone started
>ringing... As many of you know, we Alaskans talk about our politics
>constantly and this year with an indicted senior Senator, our sole
>Congressman under investigation, etc etc etc it's juicier than ever, but the
>Palin announcement takes the cake. Imagine what it would be like if any of
>you was asked to be the Vice President. That is the level of experience
>that Palin has. She's not kidding when she describes herself as a hockey
>mom - she really is. She really did grow up in a small town in Alaska.
>She really was the mayor of Wasilla (population 9,000). I would
>imagine most of you have been in Washington DC more than she has.
>
>On the one hand Palin's nomination is preposterous, but at the same time she
>will be a formidable candidate. We live in a celebrity culture which
>thrives on the personal story, and Palin's story with all the fabulous
>Alaskan scenes as backdrop will be hard to top. I think of poor John
>Kerry trying to carry a gun around in 2004 or Michael Dukakis in the
>tank, and then think of Palin's parents who were out on a gold panning
>expedition when the news broke.... She is the real thing - small town
>girl, father a teacher, grew up in Alaska, goes hunting and fishing,
>likes snow machining because "it pounds the snot out of you", husband
>is the Alaska version of a NASCAR dad, he's handsome and known as First
>Dude. On top of all that Palin is both beautiful and HOT. Look at the
>pictures of her versus Cindy McCain and you'll get the idea.
>
>I did not vote for Sarah Palin and have always wondered if her blowing
>the whistle on the head of the Republican party was pure calculated ambition.
>On the down side she not only opposes abortion (think of the Supreme
>Court
>appointments) but is a creationist. Despite her father being a science
>teacher, she seems to have little regard for the subject and recently
>was accused (like Bush) of overwriting her own staff scientist's
>recommendations on the polar bear endangered listing. On the other
>hand, she picked a very competent unbiased scientist (a friend of mine)
>to be Commissioner of Fish and Game. I asked my friend, the
>Commissioner, about what she's like in Cabinet meetings and he said
>"she's just insecure enough that she actually listens". She is a
>lightweight but a quick study, and a brilliant natural politician.
>
>Which is why I, and 80% of all Alaskans approve of her as Governor.
>(That's down from 87% five months ago.) She has made excellent and
>independent appointments to her Cabinet. She has vetoed our legislature's bloated
>budget items. She has pushed through major changes in the state's
>relationships with the oil companies. She has cajones, no doubt about
>that. In addition, she is what she is, a wonderfully attractive down
>to earth person--Kathryn even saw her shopping with her husband and new
>baby at Costco!
>
>And that will be the killer in this campaign. She may not know
>anything, but I believe more than anyone else (including Barack) she
>will attract people from a wide, wide swath of the population. No one
>can accuse her of being elitist. I looked at that picture of her
>signing photographs after the announcement and I was reminded of Mia Hamm and the
>women's soccer team.
>That's the kind of appeal Sarah Palin will bring to the Republican
>ticket, and it'll be tough to resist. Sarah Palin is the type of
>person that people tend to underestimate, which is partly why she was
>able to come out of nowhere and knock off a 20 year Senator and
>incumbent Republican Governor AND a past two term Democratic Governor.
>
>Wish us luck. Best to all of you, N

Automated Testing and the Mayan Apocalypse

Odd idea that popped out of a conversation yesterday. What if the trend towards test driven development and behavior driven development is a harbinger of the apocalypse? Automated testing could be a first step toward machine self-awareness if it were married with self-modifying code. Genetic algoritms could optimize themselves towards the specified tests and behavior, but what happens when the self-modifying code creeps into the tests (which are, after all, just more code)?

I think you get an increasing rate of accelerated change leading to Skynet launching the nukes in 2012, just in time for the Mayan calendar to be right about the end of the world.

Dr. Horrible is Fantastic

Thank goodness that Saturday on the east coast starts around 9pm on the west coast. Didn't have to wait until tomorrow to see Act III of Dr. Horrible. Recommended is not strong enough of a word. Go watch one, two and three RIGHT NOW BEFORE THEY DISAPPEAR ON SUNDAY AND YOU MIGHT HAVE TO PAY FOR IT!!! (even though paying for it would probably be a good thing because it would encourage more things like it through the action of the free market blah blah blah)

All My Hate Is Washed Away

I was kind of miffed at Yahoo last time I wrote here. Then I found this:



I laughed uproariously. Can't wait until tomorrow. Must remember to watch part III tomorrow!

Many many thanks to the folks at io9 for passing along this gem.

Yahoo has made me angry

Well, maybe not out-and-out angry, but seriously peeved. They've been periodically threatening me with switching my MyYahoo account to some newfangled ajaxy home page. My experience with it so far is that it's slower and shinier at the expense of just putting the text I want on the page in front of me.

The latest assault of trying to force me to use it happened this morning. I tried to go to the /switch URL that has worked in the past, and that failed. I did go to the 'switch back?' page, but when I clicked on the button to switch back, I still had the new UI. I noticed a link at the bottom of the new home page for switching back, and even when I clicked on that, it still failed and left me with the new home page.

Very Annoying(tm).

It's getting to the point where I'm about to switch everything over to gmail and igoogle. Google's creeping me out a little, lately, though. It's all free to use, but maybe the price is too high <backgroundmusic type='ominous'/>.

New Widget and A Shout Out

The new widget on the left is a thingy that shows where people come from when they come in a look at my ramblings. I found it on Joel Mier's blog, Think For A Moment.

He'd posted a commentary about Netflix's recent reversal on the 'kill profiles' issue. As always, well reasoned and articulately stated commentary. Miss ya, Joel. Hope all's well.

Puppet Sex

I was just watching "Team America: World Police" on the Roku box. Didn't get all the way through it, because the kids came home and that movie is FILTHY! Can't say that I wasn't chuckling along like some kind of adolescent. However, there's a reasonably explict sex scene that had me rolling. You haven't seen anything until you've seen marionettes... er... 'engaged'.

Otherwise, the movie is reasonably entertaining. Not a classic, by any means, but the irreverence, inappropriateness and general bad-boy-ness you'd expect from Parker & Stone is all there in spades.

Ongoing Changes

Minor milestones this morning.

1. Eloise rode without a carseat (she's not > 60 pounds, but she is six, and the shoulder strap doesn't cut into her neck)
2. Talked with the swim class people about how old Isaac needs to be for baby-and-me classes (3 months, get on the waitlist one month ahead)
3. ummm. Had a third one but forgot it (I blame sleep dep)

Anyway, feeling the flow of the currents of life today. Seems like the pace has picked up even as the length of days seems longer (for this I also assign some blame to sleep dep...)

exploring wirelessness

So as you can see on the left, I've been dumping occasional tidbits onto Twitter via my blackberry I have from work.

I'm also writing this post from said blackberry. I guess in addition to a new baby around the house, I'm getting in touch with my inner blogger.

That said, I don't generally see the attraction of leaving behind a constant stream of sound bites. It's been a cool novelty, but I don't think I have a strong enough itch for self-expression to need to scratch it that way.

Twittering a birth

I didn't think I was being particularly clever by jotting notes down about Isaac's birth via twitter. I'd just joined and was messing around.

Now I know I wasn't even being that original. Googling 'twitter pitocin' led me to finding this oldest account of twitlivery of a baby on July 15, 2007. I'm a year late to the party, it seems. Also, she's a bit hard-core. Of the other dozen odd accounts, we were all husbands, and Carmen's account was the only account from the driver's seat, so to speak.

Isaac Lorenzo Gilbert-Fagen

We settled on a middle name: Lorenzo

For those of you not following along at home, that would make the following updated thusly....
Tale of the tape:

• Name: Isaac Lorenzo Gilbert-Fagen


• Born: June 2, 2008 18:37 PDT


• Weight: 7lb 8oz


• Height: 20 1/4"


• Mom & baby: happy & healthy


• Dad: ditto

Netflix Player by Roku

I was really thrilled to hear today that people buying Roku's new set-top box for the Netflix Watch Instantly service is backordered by 10 days. Well, not so thrilled since I just ordered mine, but happy that the device is so popular.

Bacterially Generated Buckytubes

I'm a fan of the concept of a space elevator. I'd love to be able to see this technology envisioned by Arthur Clarke in the 70's become reality some time before I die. One of the biggest obstacles to the creation of the a space elevator is a breakthrough in the material science necessary to build a tether.

There are some promising bits of work being done on carbon fiber and buckytubes, but the last I heard, there were scaling problems. Sure, we can generate a fiber a few millimeters long that's strong enough, but getting from there to 100,000km is still a really big leap.

What if someone genetically engineered a bacteria that had some chlorophyll in it and had a predilection for growing a buckytube tail? Now you stick a vat in the sunlight and start pulling the thread... Voila! space elevator tether!

Did I mention that I was hiring?

I'm currently looking for three different people to fill three different jobs here at Netflix?. The first is a QA engineer for our Instant Viewing on the PC client. The second is a QA Engineer for the Netflix API team. The third is a QA Engineer for the Customer Account team.

While each of those job descriptions has a list of technical requirements, I need someone (three someones) who can understand the why of testing just as much as they understand the how of testing. A particular tool in your toolbox isn't as important as knowing how to select the right tool for the job (or even learn a new one). Likewise, having demonstrated prior experience in knowing where to apply your tools for the greatest effect is more important than any particular bit of prior experience with a particular technology.

I've been toying with a mission statement for my group:

Economically review changes in order to prevent, detect and mitigate impaired customer experiences.

If this somehow speaks to you, I'd love to hear from you what you think it means. I might want you to apply yourself to that mission.

Software Plasticity and Risk: The Search

After writing yesterday's post, I decided that it might be a good idea to take a look and see if anyone else was using plasticity in the sense that I was trying to use it, or if there was an already established notion of plasticity that I'd be swimming upstream against.

Again, my notion of plasticity is: how changeable is the system in question balanced against the cumulative risk to the business of the change itself and the risk to the business of actually making the changes.

I found a couple links via Google. The first was from a paper submitted to a 2004 conference in Hamburg, Germany. It discussed the concept of a new widget type, "the comet", that would adapt its function based on the execution context. Not exactly what I was after. The second was a link to the same paper from a different source. Then, surprisingly, the third entry was my blog post from yesterday.

In the fourth position, was a blog page that was just a click through to the real article which was actually discussing the use of aspect oriented programming (AOP) and the concept of software plasticity. In reviewing it, this appeared to be more of a discussion of how to apply AOP as a debugging tool, rather than how to build and maintain a system in the context of keeping a business running and satisfying customers.

After that it seemed to get a little random, except for a link to a PDF on UI plasticity. The article referenced by the top link continued to show up an awful lot from a bunch of different sources as well. I took a look at Yahoo search and Dogpile. Came up with the same set of stuff.

From this survey, it seems like there hasn't really been a lot of thinking published on the web about the development effects of the plastic nature of software, and especially when evaluating plasticity within the context of business risk.

I guess I'll keep flogging it and see if anything useful comes of it.

Software Plasticity and Risk

I've been toying with the concept of plasticity with respect to software. I've also been thinking about the software development process and how plasticity relates to the risks introduced by making changes to software.

Think about using a scale from 1 to 10 to describe how plastic a software product is. A one (the least plastic software product) would be a consumer electronics device where the software is embedded and it's impossible to update the software. Examples would include a portable DVD player or a digital clock radio. A ten would be something like a website (perhaps like the Netflix website or Instant Viewing PC client -- and I happen to be hiring one or two QA Engineers), where if there are defects discovered, they can be fixed quickly by pushing new code to the servers, and instantly, all users get the new code.

Then, think about the risks you run by making changes. My canonical examples to contrast the risk of changes usually go something like this:

1. We're updating the encryption key that we use to encrypt the credit cards for our members. We test this very thoroughly. The risks to the business are very large if there's a screwup and the benefits of avoiding screwups are equally large. Therefore, it makes a lot of sense to invest a lot in testing.

2. We're modifying how we format the pages for different genres of movies so that boxshots are right aligned instead of left aligned. Not a lot of risk for damaging the business there. We may not even do any formal QA and leave it up to the UI engineers to make sure they get that one right.

I haven't worked out the details yet for measuring risk vs. plasticity, but I imagine some kind of matrix with plasticity across the top and different software components down the side. The intersection of each row and column would have some metric that boils down the cost to fix, the cost to deploy, the risk of change and the risk to the business. Then, during the course of normal development, or in the case of an emergency, evaluate the proposed change on those four dimensions to come up the metric for that change and compare to the matrix for where the change lies. If there's a net positive risk/reward, do the change immediately in the case of a fix to production. Likewise, if it's part of the normal development cycle, then if there's a positive risk/reward, allocate the QA resources to fully test. If there's not adequate reward for testing, or the risk to the business is low enough, dedicate the limited QA resources elsewhere in that push cycle.

Comments?