Profile image
Hsin-Ying Lee
Senior Research Scientist

About Me


I am a senior research scientist in the Creative Vision team at Snap Research. I received my PhD degree in EECS at UC Merced under the advisement of Prof. Ming-Hsuan Yang in 2020. I got my M.S. degree in the Ming Hsieh Department of Electrical Engineering from Univeristy of Southern California in 2016, and my B.S. degree in the Department of Electrical Engineering from National Taiwan University.

My research interests include Computer Vision and Machine Learning, particularly in 2D/3D/4D generation and manipulation. Here is my CV.

News


  • library_books 09/2024: One papers is accepted to NeurIPS 2024
  • library_books 05/2024: One paper is accepted to ECCV 2024
  • library_books 02/2024: Four papers are accepted to CVPR 2024 (1 Spotlight)
  • library_books 09/2023: One paper is accepted to SIGGRAPH Asia 2023
  • library_books 07/2023: Two papers are accepted to ICCV 2023
  • library_books 04/2023: One paper is accepted to ICML 2023
  • library_books 02/2023: Five papers are accepted to CVPR 2023 (1 highlight)
  • library_books 01/2023: One paper is accepted to ICLR 2023 (oral)
  • library_books 07/2022: Three papers are accepted to ECCV 2022
  • library_books 12/2021: Two papers are accepted to CVPR 2022
  • library_books 12/2021: One paper is accepted to ICLR 2022
  • library_books 07/2020: Joining Snap Research!
  • library_books 07/2020: Four papers are accepted to ECCV 2020 (1 spotlight)
  • library_books 12/2019: One paper is accepted to ICLR 2020 (spotlight)
  • library_books 12/2019: One paper is accepted to IJCV 2019
  • library_books 09/2019: One paper is accepted to NeurIPS 2019
  • library_books 05/2019: Google Internship, MTV
  • library_books 04/2019: One paper is accepted to ICIP 2019
  • library_books 02/2019: One paper is accepted to CVPR 2019
  • library_books 12/2018: Google Internship, SNV
  • library_books 07/2018: Two papers are accepted to ECCV 2018 (1 oral)
  • library_books 05/2018: Nvidia Internship, Santa Clara Audio-Visual Modeling
  • library_books 07/2017: One paper is accepted to ICCV 2017
  • library_books 05/2017: Google Internship, MTV Video Highlights / Video Summarization
  • library_books 08/2016: Start my PhD at UC Merced

Publications


4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models
Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Laszlo A Jeni, Sergey Tulyakov, Hsin-Ying Lee
NeurIPS 2024
UpFusion: Novel View Diffusion from Unposed Sparse View Observations
Bharath Raj Nagoor Kani, Hsin-Ying Lee, Sergey Tulyakov, Shubham Tulsiani
ECCV 2024
Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors
Dave Zhenyu Chen, Haoxuan Li, Hsin-Ying Lee, Sergey Tulyakov, Matthias Nießner
CVPR 2024
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, Sergey Tulyakov
CVPR 2024
Scenewiz3d: Towards text-guided 3d scene composition
Qihang Zhang, Chaoyang Wang, Aliaksandr Siarohin, Peiye Zhuang, Yinghao Xu, Ceyuan Yang, Dahua Lin, Bolei Zhou, Sergey Tulyakov, Hsin-Ying Lee
CVPR 2024
Exploiting Diffusion Prior for Generalizable Pixel-Level Semantic Prediction
Hsin-Ying Lee, Hsin-Ying Lee, Hung-Yu Tseng, Ming-Hsuan Yang
CVPR 2024
Text-Guided Synthesis of Eulerian Cinemagraphs
Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, Sergey Tulyakov, Jun-Yan Zhu
SIGGRAPH Asia 2023
Text2tex: Text-driven texture synthesis via diffusion models
Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, Matthias Nießner
ICCV 2023
InfiniCity: Infinite-Scale City Synthesis
Chieh Hubert Lin, Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Yang, Sergey Tulyakov
ICCV 2023
Unveiling The Mask of Position-Information Pattern Through the Mist of Image Features
Chieh Hubert Lin, Hung-Yu Tseng, Hsin-Ying Lee, Maneesh Kumar Singh, Ming-Hsuan Yang
ICML 2023
3DAvatarGAN: Bridging Domains for Personalized Editable Avatars
Rameen Abdal, Hsin-Ying Lee, Peihao Zhu, Menglei Chai, Aliaksandr Siarohin, Peter Wonka, Sergey Tulyakov
CVPR 2023
SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation
Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander Schwing, Liangyan Gui
CVPR 2023
Make-A-Story: Visual Memory Conditioned Consistent Story Generation
Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, Leonid Sigal
CVPR 2023
DiscoScene: Spatially Disentangled Generative Radiance Field for Controllable 3D-aware Scene Synthesis
Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang, Yujun Shen, Hsin-Ying Lee, Bolei Zhou, Sergey Tulyakov
CVPR 2023 (Highlight)
Unsupervised Volumetric Animation
Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Kyle Olszewski, Jian Ren, Hsin-Ying Lee, Menglei Chai, Sergey Tulyakov
CVPR 2023
3D Generation on ImageNet
Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, Sergey Tulyakov
ICLR 2023 (oral)
Cross-Modal 3D Shape Generation and Manipulation
Zezhou Cheng, Menglei Chai, Jian Ren, Hsin-Ying Lee, Kyle Olszewski, Zeng Huang, Subhransu Maji, Sergey Tulyakov
ECCV 2022
Vector Quantized Image-to-Image Translation
Yu-Jie Chen, Shin-I Cheng, Wei-Chen Chiu, Hung-Yu Tseng, Hsin-Ying Lee
ECCV 2022
Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris Metaxas, Sergey Tulyakov
ECCV 2022
In&Out: Diverse Image Outpainting via GAN Inversion
Yen-Chi Cheng, Chieh Hubert Lin, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Ming-Hsuan Yang
CVPR 2022
InfinityGAN: Towards Infinite-Resolution Image Synthesis
Chieh Hubert Lin, Hsin-Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, Ming-Hsuan Yang
ICLR 2022
Continuous and Diverse Image-to-Image Translation via Signed Attribute Vectors
Qi Mao, Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, Siwei Ma, Ming-Hsuan Yang
IJCV 2020
RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval
Hung-Yu Tseng, Hsin-Ying Lee, Lu Jiang, Ming-Hsuan Yang, Weilong Yang
ECCV 2020
Controllable Image Synthesis via SegVAE
Yen-Chi Cheng, Hsin-Ying Lee, Min Sun, Ming-Hsuan Yang
ECCV 2020
Semantic View Synthesis
Hsin-Ping Huang, Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang
ECCV 2020
Neural Design Network: Graphic Layout Generation with Constraints
Hsin-Ying Lee, Weilong Yang, Lu Jiang, Madison Le, Irfan Essa, Haifeng Gong, Ming-Hsuan Yang
ECCV 2020 (spotlight)
Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation
Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, Ming-Hsuan Yang
ICLR 2020
Dancing to Music
Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, Jan Kautz
NeurIPS 2019
Self-supervised Audio Spatialization with Correspondence Classifier
Yu-Ding Lu, Hsin-Ying Lee, Hung-Yu Tseng, Ming-Hsuan Yang
ICIP 2019
Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis
Qi Mao*, Hsin-Ying Lee*, Hung-Yu Tseng*, Siwei Ma, and Ming-Hsuan Yang
CVPR 2019
Diverse Image-to-Image Translation via Disentangled Representations
Hsin-Ying Lee*, Hung-Yu Tseng*, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang
ECCV 2018 (oral)
IJCV 2019
Sub-GAN: An Unsupervised Generative Model via Subspaces.
Jie Liang, Jufeng Yang, Hsin-Ying Lee, Kai Wang, and Ming-Hsuan Yang
ECCV 2018
Video Highlights Using Retention Stats
This project is done during the intership at Google. We explore a Youtube statistics - Retention Stats as an auxiliary training signal for video highlights extraction. Retention stats is a Youtube metric that takes multiple behaviors into consideration including when do audiences start, when do they quit, which parts are viewed multiple times and which parts are skipped frequently. We focus on sports highlights since they often share more consistent consensus among audience than other categories.
Hsin-Ying Lee, Min-Hsuan Tsai, Zheng Sun, and Weilong Yang
Unsupervised Representation Learning by Sorting Sequences
This work proposes a self-supervised sequence sorting task to learn image representation using unlabeled videos.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsuan Yang
ICCV 2017
Graph-Based Logic Bit Slicing for Datapath-Aware Placement
This work proposes a balanced edge-cover algorithm to completely slice any datapath, and an SA-based refinement process to exploit the bit-slice paths as an alignment guidance to placers.
Chau-Chin Huang, Bo-Qiao Lin, Hsin-Ying Lee, Yao-Wen Chang, Kuo-Sheng Wu, and Jun-Zhi Yang
DAC 2017
Soft-Segmentation Guided Object Motion Deblurring
This work aims to use a maximum a posterior formulation in which soft-segmentation is incorporated for object layer estimation to jointly estimate object segmentation and camera motion.
Jinshan Pan, Zhe Hu, Hsin-Ying Lee, and Ming-Hsuan Yang
CVPR 2016
Detailed-routability-driven analytical placement for mixed-size designs with technology and region constraints
This work aims to participate in the 2015 ISPD Blockage-Aware Detailed Routing-Driven Placement Contest, which targeted at detailed-routability-driven analytical placement for mixed-size design.
Chau-Chin Huang, Hsin-Ying Lee, Bo-Qiao Lin, Sheng-Wei Yang, Chin-Hao Chang, Szu-To Chen, Yao-Wen Chang
ICCAD 2015
Bio-inspired proximity discovery and synchronization for D2D communications
This work aims to propose a distributed mechanism for Device-to-Device (D2D) communication, which achieves proximity discovery and synchronization simultaneously
Shih-Lung Chao, Hsin-Ying Lee, Ching-Chun Chou, Hung-Yu Wei
IEEE communications letters, 2013

Education


Ph.D. in EECS, University of California, Merced, California, USA
Sep. 2016 - May. 2020
M.S. in Electrical Engineering, University of Southern California, Los Angeles, California, USA
Sep. 2015 - Aug. 2016
B.S. in Electrical Engineering, National Taiwan University, Taipei, Taiwan
Sep. 2010 - Jun. 2014