Vladlen Koltun

Vladlen Koltun was born on November 21, 1980 in Kiev, Ukraine[15][16] and grew up in Israel, where he completed his B.S. degree in computer science magna cum laude from Tel Aviv University in 2000. Continuing his studies at Tel Aviv University, he finished his Ph.D. with distinction in computer science in 2002 at the age of 21[16] with a thesis, "Arrangements in four dimensions and related structures"; his doctoral adviser was Micha Sharir.[17] He then completed his postdoctoral fellowship under the supervision of Christos Papadimitriou at the University of California, Berkeley, where he conducted research in theoretical computer science in 2002-2005.[13]

Koltun served as an assistant professor at Stanford University from 2005 - 2013, where he lectured in the areas of computer science, computer graphics and geometric algorithms.[7][18] His scientific research interests at Stanford included computer vision, neural networks, data-driven 3D modeling and machine learning (dense random fields), among others.[1] During his tenure at Stanford, he also supervised PhD students and postdoctoral researchers;[17] two of his students now serve as professors at UC Berkeley (Sergey Levine)[19] ​and UT Austin (Philipp Krähenbühl).[20] While at Stanford, Koltun was a recipient of the Sloan Research Fellowship[3] and the National Science Foundation's Career Award.[4]

Koltun's research at Stanford[7] contributed to the development of data-driven 3D modeling technology[21] in close collaboration with Siddhartha Chaudhuri.[22] Chaudhuri's work along with Vladlen Koltun, Evangelos Kalogerakis and Leonidas Guibas resulted in a SIGGRAPH publication in 2011.[23] As a result, Mixamo licensed the technology from Stanford and later Adobe Inc. acquired Mixamo and further developed Adobe Fuse CC, a 3D computer graphics software that enabled users to create 3D characters.[23][24][21][22] In 2014, Koltun was hired by Adobe to conduct research in visual computing with the primary focus on three-dimensional reconstruction.

Koltun left Adobe to join Intel, where he served in various positions until 2021 for the company's R&D projects, including Principal Researcher,[25] Senior Principal Researcher and Chief Scientist for Intelligent Systems.[10][6] According to his profile on the Institute of Electrical and Electronics Engineers' website, "Koltun's lab conducted high-impact basic research on intelligent systems, with emphasis on computer vision, robotics, and machine learning."[10] Since August 2021, Koltun has been serving as distinguished scientist at Apple Inc.[1]

While at Intel, Koltun was instrumental in developing virtual reality simulators for urban autonomous driving, robots and drones. In particular, he directed research on deep reinforcement learning techniques by applying neural networks in virtual simulators. In the research, the neural networks first learnt to make decisions by trial and error while exploring physics in virtual reality. Once the intelligent systems had been trained, they were transferred to robots or drones to be used in natural environments. For instance, the method was applied to the ANYmal robot, a sophisticated quadrupedal machine with proprioceptive feedback in locomotion control.[12][26]

Another example was development of intelligent systems for drones flying in dynamic environments based on the research papers co-published with University of Zurich robotics professor Davide Scaramuzza: Deep Drone Racing: Learning Agile Flight in Dynamic Environments (2018),[27] Deep Drone Racing: From Simulation to Reality With Domain Randomization (2019)[28] and Deep Drone Acrobatics (2020).[29] In this collaborative project of the Intel Labs with the University of Zurich, neural networks were trained in simulation and then deployed on physical quadrotors, in some cases showing maneuverability and speed similar to those controlled manually by human pilots.[28][30][31]

The studies in the domain of urban autonomous driving led Koltun's group[32] to the development of the CARLA project (short for Car Learning to Act), an open-source simulator,[33] powered by Unreal Engine,[34] that can be used to test self-driving technologies in realistic driving environments with random dangerous situations occurring on the roads.[35][36][37] The project has been funded by the Intel Labs and Toyota Research Institute, among others.[38][36] During his activity in Intel Labs, Koltun also contributed to further development of 3D computer graphics in the fields of 3D photorealistic view synthesis and photorealistic rendering.[39][9][40] In their work, Enhancing Photorealism Enhancement,[41][42][43] Vladlen Koltun, Stephan R. Richter and Hassan Abu AlHaija used a view synthesis technique by sourcing information from the Cityscapes Dataset (See Section 1.3 - Objects detection and recognition in List of datasets for machine-learning research),[39][44] ​a collection of stereo video sequences captured by a car’s built-in camera for machine-learning algorithms. As VentureBeat explains: "The full system, displayed below, is composed of several interconnected neural networks. The G-buffer encoder transforms different render maps (G-buffers) into a set of numerical features. G-buffers are maps for surface normal information, depth, albedo, glossiness, atmosphere, and object segmentation. The neural network uses convolution layers to process this information and output a vector of 128 features that improve the performance of the image enhancement network and avoid artifacts that other similar techniques produce. The G-buffers are obtained directly from the game engine. The image enhancement network takes as input the game’s rendered frame and the features from the G-buffer encoder and generates the photorealistic version of the image. The remaining components, the discriminator and the LPIPS loss function, are used during training. They grade the output of the enhancement network by evaluating its consistency with the original game-rendered frame and by comparing its photorealistic quality with real images."[45] As a result, the researchers achieved substantial gains in stability and realism in comparison to alternative image-to-image translation methods and a variety of other baselines.[46] Andrew Liszewski from Gizmodo notes: "Based on dataset, the neural network also uses other rendered data the game’s engine has access to, like the depth of objects in a scene, and information about how the lighting is being processed and rendered."[47] The photorealism enhancement system was tested in the Grand Theft Auto V game and gained the attention of technology experts.[48][49][50][39]

Inspired by Google Cardboard,[2] Vladlen Koltun and Matthias Müller, a German scientist specializing in aerial tracking and sim-to-real transfer for autonomous navigation, developed OpenBot, a software stack for turning Android smart phones into autonomous four-wheeled robots capable of navigating in space and following objects of interest, also avoiding various obstacles. The robot’s parts include a 3D-printable chassis with notches for a controller, microcontroller, LEDs, a smartphone mount, and a USB cable.[51] The software consists of the Arduino Nano board, which bridges the smartphone with the motor actuation tasks and batteries, and an Android app, which is responsible for integration of data.[2][52] The project was released as an open-source concept[53][54] for various robotics-related applications with the software development kit available on GitHub.[55][56]

Koltun has been critical of the reliability of the h-index, drawing attention to the inflation of h-index values since multiple co-authorships became popular in scientific communities.[57] In the collaborative work with David Hafner, The h-index is no longer an effective correlate of scientific reputation, Koltun and Hafner state that " hyper-authorship – a growing phenomenon where global research consortia produce papers with thousands of co-authors – enables people to rack up enormous h-indices very quickly."[8][58][59]

• SR Richter, HA AlHaija, V Koltun; Enhancing Photorealism Enhancement, arXiv:2105.04619 (2021)
• Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, Marco Hutter; Learning Quadrupedal Locomotion over Challenging Terrain, Science Robotics (2020)[12]
• Elia Kaufmann, Antonio Loquercio, René Ranftl, Matthias Müller, Vladlen Koltun, Davide Scaramuzza; Deep Drone Acrobatics, Robotics: Science and Systems (2020)[60]
• Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, Dhruv Batra; Habitat: A Platform for Embodied AI Research, International Conference on Computer Vision (2019)[61]
• Chen Chen, Qifeng Chen, Jia Xu, Vladlen Koltun, Learning to See in the Dark, Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, June 2018
• Shaojie Bai, J Zico Kolter, Vladlen Koltun; An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, arXiv preprint arXiv:1803.01271 (2018)
• Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun, Open3D: A Modern Library for 3D Data Processing, Technical Report, arXiv:1801.09847, 2018.
• Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López, Vladlen Koltun; CARLA: An Open Urban Driving Simulator, Conference on Robot Learning (CoRL) 2017
• F Yu, V Koltun; Multi-Scale Context Aggregation by Dilated Convolutions, International Conference on Learning Representations (ICLR) 2016
• Stephan R Richter, Vibhav Vineet, Stefan Roth, Vladlen Koltun; Playing for Data: Ground Truth from Computer Games, European Conference on Computer Vision (ECCV) 2016
• Sergey Levine, Vladlen Koltun; Guided Policy Search, International Conference on Machine Learning (ICML) 2013
• Philipp Krähenbühl, Vladlen Koltun; Efficient inference in fully connected CRFs with Gaussian edge potentials, Advances in Neural Information Processing Systems (NIPS) 2011

• Patents by Inventor Vladlen Koltun
• Google Scholar Profile
• Semantic Scholar Profile