How Two Startups Use Games to Beat the Developer Crunch

Published on ReadWriteWeb | shared via feedly mobile

"You can't judge if someone is one of the best programmers in the country in 1 minute, but it turns out you can in 5 minutes."
Good software developers are hard to find. Startups are all about finding creative solutions to common problems - so why not this one too?
Two startups that have found creative and interesting ways to solve their developer shortage problems are travel photo network Jetpac and app search startup Quixey. Both used contests and games to overcome their challenges and get access to the high-level coding talent they needed. Their efforts may illustrate a part of what people call the gamification of work that's expected to be a big part of the future.

How Jetpac Built a Photo Quality Algorithm for $5k in 3 Weeks

Jetpac is a young San Francisco startup that asks you to log in with your Facebook account, then it searches through all the photos your friends have uploaded. It looks for photos with the names of places in their captions, then builds a personalized travel photo magazine out of your friends' pictures.
One member of the founding team is leading data hacker Pete Warden. (Disclosure, Warden told me this story while I was staying at his house on a trip to SF, but it's such a cool story I've been telling it ever since - and it works well with the Quixey story too.)
Warden says that when the team was first showing off its service in demos, far too many of the photos that came up were terrible. They were blurry, boring, bad photos. It was easy for a human being to look at these photos and know they should be excluded from the collections displayed.
Could a machine be taught to look at new photos and determine whether they were high or low quality? Warden suspected that it was possible, but recognized the limitations of his own knowledge. He didn't have the machine learning skills to build something himself, much less at the pace the company needed a solution.

Here's what they did: They looked at 30,000 photos with their human brains and quickly judged whether each was a good or bad photo for a travel magazine experience.
Then they visited the website Kaggle, where data science challenges gets turned into contests with prizes that anyone in the world can win. The Jetpac team took all the metadata they had about these 30,000 photos, including the dimensions, and they substituted standardized numbers for words that appeared more than once. They uploaded all that data onto Kaggle but they only included the corresponding human judgement of whether a photo was good or bad for 10,000 of the photos.
The challenge they set up was this: could Kaggle participants write code that could analyze the patterns of metadata effectively enough based on the 10k photos they were told the human judgements about well enough to accurately guess whether humans would call the other 20k photos good or bad just based on the other metadata available about them?
The startup put up a tiny $5k bounty, one of the smallest Kaggle had ever hosted, and applied a deadline in 3 weeks.
People loved it. All kinds of computer scientists moonlighting as Kaggle competitors jumped into the fray and wrote algorithms they thought could predict photo quality. They drafted something up, uploaded their "guesses" for the other 20k photos to Kaggle's server, then were told what percentage they got right - how often they accurately predicted a person would deem a photo good. Then they changed their code and tried to improve their results.
212 teams, consisting of 418 people, competed for 3 weeks. The contest leaderboard showed the top ten teams all had more than 85% accuracy.
All the algorithms found that there were some words in photo captions that make them far more likely to be connected to a good travel photo than a bad one. Among the best words: Peru, Cambodia, Michigan, tombs, trails and boats. What photo captions are the most likely to signify a bad photo for a travel magazine? San Jose, mommy, graduation and CEO, Warden says.

All the algorithms found that there were some words in photo captions that make them far more likely to be connected to a good travel photo than a bad one.
Among the best words: Peru, Cambodia, Michigan, tombs, trails and boats. What photo captions are the most likely to signify a bad photo for a travel magazine? San Jose, mommy, graduation and CEO, Warden says.

Bo Yang, a USC PhD whose team had just narrowly lost out on winning the Netflix prize, squeaked out a small improvement in his photo quality algorithm to take the top prize in the very last day. Yang was interviewed by the Kaggle team here.
Part of the Kaggle terms of service are that contest sponsors must have non-exclusive IP rights to the work, so the Jetpac team was able to put code from the contestants directly into their app.
Jetpac's Warden says of the experience as a startup,

"The two biggest enemies of a startup are lack of money and lack of time. Packaging the data didn't take as long as we thought and after we uploaded it to the site, all of the details of dealing with the contestants were automated. So it saved us a massive amount of time compared to finding, hiring and explaining our problem to an outside contractor.

"And we would never have gotten anywhere near the quality from the circle of people we know. The short term nature of the project wouldn't have made it attractive as a project for most - just the overhead of setting up a contract and that sort of stuff. The caliber of people participating in these contests is amazing. They aren't starving college students, many are highly skilled professionals who make a lot more money than I do, in their day jobs. They do this for fun."
Jetpac had to think through how to set up the contest, but the Kaggle team helped them, too. It's hard to imagine a way that such a complex problem could get so much brain power thrown at it so fast and so inexpensively. Warden says the end results have been great.

How Quixey Finds Great Developers with $100, 60 Second Challenges

Quixey, a Silicon Valley app search engine (it's cool, try it - I found this on it), faces the same struggle to find developers that so many startups do. They have high-profile VC backing (Eric Schmidt of Google, among others) and had been paying $20k per developer hire to traditional recruiters.
Liron Shapira, co-founder and CTO of Quixey, says the company came up with a very elegant solution. Called the Quixey Challenge, it's a simple contest. If you can find and fix a bug in the code for an algorithm you're given, in under 60 seconds, the company PayPals you $100.
In order to qualify for the monthly contest, you've got to succeed at least 3 times in challenge rounds over the weeks prior to the big event. If you qualify, then the company calls you on Skype and administers the challenge face-to-face. It only lasts 60 seconds. If, in preparation, you succeed 5 times - then the system automatically contacts you to see if you might be interested in working for Quixey.
Shapira says that 38 prizes were awarded in the December challenge, and it resulted in 3 full time hires and 2 intern hires. Winners also receive Quixey Challenge hoodies, which Shapira says can be seen floating around the elite student body of Carnegie Mellon University.
"We've had about 5k users sign up and practice and we've reached out to 500 or something," Shapira told me. "Those are incredibly valuable leads to have."

"We just hired a guy named Marshall who doesn't have a college degree and lives in Grand Rapids Michigan. He wouldn't come in from a Silicon Valley recruiter, but he reads Hacker News and he nailed the interview.

"You can't judge if someone is one of the best programmers in the country in 1 minute, but it turns out you can in 5 minutes. You only need 3 practices to qualify for the challenge but people take 10. A low percentage like 1 in 15 or 20 users will be good enough to get contacted, so we are able to filter people out with high accuracy.
"We wasted so much time figuring out peoples' skills before. Many times we'll do the challenge or interviews and it will take 15 minutes. The fact that some people can do it under 1 minute and others, also working in Silicon Valley, take 15 minutes, is evidence of the 10x engineer idea. Debugging is something you do every day at work, if you can get more than half of the bug fixes that we put in front of you, fast, then you are probably very good and we want to talk you."
Quixey says it is looking to outsource its process to other startups sometime in the future.

See also: The End of the Résumé: Oracle's Big Plans for Taleo

It's not just these two companies that are using contests and games to get software development done. The US Government has Challenge.gov and SeatGeek has gamified not just their developer hiring but also their communications hires. Then there's the gamification of everyday employee management.
Examples are just beginning to emerge, but they do seem to point towards some relief in the face of a very difficult talent shortage challenge. "We think we're still on the leading edge of this trend and it's going to get bigger," says Quixey's Liron Shapira.
"We didn't know if people would be able to produce a good result out of this," says Jetpac's Warden about his startup's experiment with gamification of development, "but we were amazed by how effective the solutions they come with were."
"The most important choice you make as a data scientist is deciding what problems you're not going to solve," Warden says. That equation changes when you've got access to compelling ways to use other peoples' skills to solve those problems.
Basketball hoop photo from MinimalistPhotography101.com Discuss

feedly. feed your mind. http://www.feedly.com

Sent from my iPhone