The algorithm that judges your fashion sense will soon be an app

Lila MacLellan
Daily Brew
Robotics researcher Edgar Simo-Serra before and after working on the algorithm
Robotics researcher Edgar Simo-Serra before and after working on the algorithm

Two computer science professors at University of Toronto may have caused mild panic among stylists last month when they announced the results of their latest research: an algorithm that analyses fashion choices in a photograph and makes suggestions to up a person’s chic factor.

Sanja Fidler and Raquel Urtasun created the system using two main technologies, graphical modelling and something called deep learning, a subcategory of machine learning, or artificial intelligence, pioneered by University of Toronto professor Geoffrey Hinton. Deep learning allows a computer to “think” for itself using complicated artificial neural networks, or “neural nets”, that function in a similar way to the human brain’s neural connections.

To school itself in the ways of sharp style, the professors’ prototype algorithm—a kind of “thinking” robot—collected data from more than 100,000 photos posted to the website Chictopia, an Instagram-like digital billboard where users around the world upload images of their own sartorial creations and “vote” for ( “like” in Facebook terms) other people’s style. Essentially, Fidler and Urtasun’s robot absorbed this information and used it to figure out what the going trends were for different age groups and regions. The idea is that the robot would continue to learn what’s hot or not as fresh photos are added to the website.

If someone then needed style advice from the algorithm, she would feed a photo of the outfit she’s considering to the fashion-centric system. (For now, the model caters to mainly to women.) It would not only evaluate the apparel it sees, including tops, bottoms, shoes, handbags, scarves and hats, but also “read” the image to gather information about a person’s facial features (thus determining age and ethnicity), as well as one’s facial expression, body type, background imagery and even the angle from which the photo was taken. It next uses all of this data to determine whether someone has made the best decisions she could in front of the mirror that morning, or if a different shade, or an A-line skirt instead of pleated culottes, or a even a new photographic background, would optimize the photo’s attractiveness.

Until now, the project has been an intellectual exercise, a way of demonstrating the way deep learning can be useful in one’s daily life, a personal assistant, say the professors, who are both in their thirties. Soon, though, it will be an app ready for the public to play with. A student team is currently fine-tuning the necessary calculations and expect to have a polished product ready within a few months.

“We realized we didn’t know how to dress for certain occasions,” says Fidler. “Partly, this is what inspired the work. We’re also computer scientists so there’s already a stigma that we don’t dress very well,” she adds.

Fidler moved to Toronto a year and a half ago to take up her current teaching position and is originally from Slovenia. Urtasun grew up in Spain and arrived in Toronto to join the “Canadian Mafia” of deep learning, as tech publication Re/code recently labelled their larger group of academics, at the same time as Fidler. They both say the app was not a response to the styles they saw on Canadian streets, though. In fact, building a new lab has occupied so much of their time that they haven’t had a chance to form an opinion of the going fashions in their new home—or so they claim.

Although their model likely did not improve their own style choices, as far as they’ve noticed, they did see a profound change in one of their students visiting Toronto from Spain, Edgar Simo-Serra. A year ago, Simo-Serra moved to Toronto looking like a “typical general computer scientist—not well dressed,” explains Fidler. “But just by working on this algorithm and being exposed to all these pretty people who are dressed well, he became very fashionable. He’s like a top male model now.” The scientists like to joke that he is their success story. (See before and after photos, below.)

To date, however, neither the Donna Karans and Domenico Dolces, nor the H&Ms and Inditex Groups of the world have contacted Fidler and Urtasun to suggest collaborating, something the professors would like to do. They imagine companies using their system to predict trends and how they will evolve over time among different demographics and in various cities. Or they’d love to create some kind of computer-fashion branding project.

Instead they’ve found designers reluctant to believe that an algorithm can do their job. “

“It’s the same for any other profession, like doctors and automatic systems that can detect cancer,” says Urtasun. Their response is understandable, she adds, but it’s also missing the point. This algorithm is designed to learnwhat people are already wearing, and to give feedback about what has caught on and what didn’t. It’s not going to compete with designers or invent new styles and silhouettes.

(In that case, they may have better luck selling the idea to economists, who have claimed to find market indicators in everything from hemlines and hairstyles, to men’s underwear sales, lipstick, and color choices.)

Since unveiling this research, the scientists have moved on from clothes to cars. They are working on a formula for an autonomous driving system using the same key technologies that went into their robotic fashionista. Sure, these seem like different functions, but the core mathematical application is similar -- allowing a car’s brain to parse video and understand obstacles in context.

AI is ready to help us live better in many domains, the professors say, and scientists are only starting to envision what can be done with it. Inviting a thinking robot to rifle through our closets might be the softest possible introduction to a whole new world.