City model created from images alone
ETH scientists have combined millions of images and videos into a three-dimensional, living model of the city of Zurich. The new technology has many possible applications – for example, it can analyse where and when pedestrians are on the move and parking spaces become free.
Researchers at ETH Zurich have developed a technology platform with which they can create 3D city models using image data alone. The platform, known as "VarCity", is so versatile that it can use, evaluate, and automatically combine all possible kinds of image sources: aerial photographs, 360-degree panoramic images taken with special vehicles, and even standard photos such as those published by tourists on social networks and internet platforms. The technology also uses video material from sources like YouTube and public webcams.
Thanks to machine learning algorithms, the technology can recognise image content such as buildings, windows and doors, streets, bodies of water, people, and cars. Without human assistance, the 3D model "knows", for example, what pavements are and – by evaluating webcam data – which streets are one-way only.
"Bringing city models to life"
"The whole thing goes so far that we can perform a highly detailed analysis of traffic flows on the street and also measure pedestrian traffic, even in real time depending on the application," explains Hayko Riemenschneider, a postdoc and VarCity project manager in the group led by Luc Van Gool, a Professor at ETH Zurich's Computer Vision Lab. This makes counting traffic extremely easy: it can be done exclusively with cameras and no longer requires sensors to be installed on the road.
"Our technology is dynamic. It brings life, people and vehicles to 3D city models, which is one of its major advantages over well-known applications such as Google Street View," says Riemenschneider. The technology was also devised in line with the principle of "privacy by design", with the algorithms generating anonymised information based on image data. Pedestrian flows and vehicle traffic can be shown by avatars, making VarCity compatible with data privacy.
Large volumes of data evaluated
The researchers developed the computer algorithms for the model over the past five years as part of a European Research Council (ERC) project (an ERC Advanced Grant received by ETH Professor Van Gool). To showcase their work, they have created a 3D model of the city of Zurich, which they are now presenting in a video as they wrap up the project on schedule.
The VarCity technology is based on the evaluation of huge volumes of data. "The more images and videos the platform can evaluate, the more precise the model becomes," says Kenneth Vanhoey, a postdoc in Van Gool's group. "The aim of our project was to develop the algorithms for such 3D city models, assuming that the volume of available images and videos will also increase dramatically in the years ahead."
The scientists use triangulation to create a three-dimensional model from image data. This requires several pictures of an object from various angles; an analysis of the distances and angles in the various images then allows the corner points of the object to be precisely located in the relevant space.
Simple parking management technology
The researchers are not offering the models themselves as an application, but the technology behind them has many possible applications, including in urban and transport planning. The project participants have founded several startups to market these applications: Spectando offers virtual building inspections for the real estate market, while Casalva performs virtual damage analyses on buildings for insurance companies, for example after extreme weather events such as hailstorms. No special devices are required for these applications – standard smartphone photos are sufficient.
ETH spin-off Parquery, that has emerged from the VarCity project, also uses this technology. The company developed a parking management system that works exclusively with cameras installed near parking spaces and does not require parking space sensors. The computer evaluates the video data in real-time and recognises cars, so it knows which parking spaces are free and also whether a car has been mistakenly parked across two spaces. This system was first used in the Swiss city of Locarno, with further projects in over 30 cities in Switzerland and abroad.
The ETH spin-off UniqFEED uses the methods to identify perimeter advertising and individual players during television broadcasts of sports events, allowing it to calculate how long they are visible during the broadcast and to personalise the perimeter ads for various user groups.
