Ling Luo

An art lover and a computer science enthusiast, both contributing to my passion about using computational intelligence to fuel art creation. This passion further motivates me to investigate how humans perceive the three-dimensional world through artistic abstraction.

I’ve been exploring the possibilities of 3D sketch-assisted content creation using VR technology and machine learning since joining the SketchX lab in 2019. This brand-new research direction is supervised by Prof. Yi-Zhe Song (Director, SketchX Lab, CVSSP, University of Surrey); co-supervised by Dr. Yulia Gryaditskaya, Prof. Tao Xiang and Dr. Yongxin Yang.

I also enjoy sharing my knowledge and creating videos.

News

Jan 8, 2024	Passed the viva defense!
Mar 22, 2023	Sketching into the Metaverse: The first fine-grained VR sketch based shape retrieval demo is in AIUK 2023!
Mar 1, 2022	The first fine-grained VR Sketch dataset is available now!
Oct 2, 2019	Joined SketchX of Surrey!

Selected Publications

Bibtex

3D shape modeling is labor-intensive and time-consuming and requires years of expertise. Recently, 2D sketches and text inputs were considered as conditional modalities to 3D shape generation networks to facilitate 3D shape modeling. However, text does not contain enough fine-grained information and is more suitable to describe a category or appearance rather than geometry, while 2D sketches are ambiguous, and depicting complex 3D shapes in 2D again requires extensive practice. Instead, we explore virtual reality sketches that are drawn directly in 3D. We assume that the sketches are created by novices, without any art training, and aim to reconstruct physically-plausible 3D shapes. Since such sketches are potentially ambiguous, we tackle the problem of the generation of multiple 3D shapes that follow the input sketch structure. Limited in the size of the training data, we carefully design our method, training the model step-by-step and leveraging multi-modal 3D shape representation. To guarantee the plausibility of generated 3D shapes we leverage the normalizing flow that models the distribution of the latent space of 3D shapes. To encourage the fidelity of the generated 3D models to an input sketch, we propose a dedicated loss that we deploy at different stages of the training process.
```
@inproceedings{luo20233d,
  abbr = {ICCV},
  title = {3D VR Sketch Guided 3D Shape Prototyping and Exploration},
  author = {Luo, Ling and Chowdhury, Pinaki Nath and Xiang, Tao and Song, Yi-Zhe and Gryaditskaya, Yulia},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages = {9267--9276},
  year = {2023},
  pdf = {https://openaccess.thecvf.com/content/ICCV2023/papers/Luo_3D_VR_Sketch_Guided_3D_Shape_Prototyping_and_Exploration_ICCV_2023_paper.pdf},
  code = {https://github.com/Rowl1ng/3Dsketch2shape},
  video = {https://www.youtube.com/watch?v=PCig106t7aM&ab_channel=LingLuo},
  slides = {https://drive.google.com/file/d/1HewX7cbp8YTOA0G7b_OsOTolpwIRnS_d/view?usp=sharing},
  poster = {https://drive.google.com/file/d/1xTWyCWziSBq7fwTV5EiCird3_DSmu3xs/view?usp=sharing},
  img = {../assets/img/publication_preview/interp_98.gif},
  arxiv = {2306.10830},
  selected = {true}
}
```
ICCV

3D VR Sketch Guided 3D Shape Prototyping and Exploration

Luo Ling, Chowdhury Pinaki Nath, Xiang Tao, Song Yi-Zhe, and Gryaditskaya Yulia

In Proceedings of the IEEE/CVF International Conference on Computer Vision 2023

Abs arXiv PDF Video Code Poster Slides

3D shape modeling is labor-intensive and time-consuming and requires years of expertise. Recently, 2D sketches and text inputs were considered as conditional modalities to 3D shape generation networks to facilitate 3D shape modeling. However, text does not contain enough fine-grained information and is more suitable to describe a category or appearance rather than geometry, while 2D sketches are ambiguous, and depicting complex 3D shapes in 2D again requires extensive practice. Instead, we explore virtual reality sketches that are drawn directly in 3D. We assume that the sketches are created by novices, without any art training, and aim to reconstruct physically-plausible 3D shapes. Since such sketches are potentially ambiguous, we tackle the problem of the generation of multiple 3D shapes that follow the input sketch structure. Limited in the size of the training data, we carefully design our method, training the model step-by-step and leveraging multi-modal 3D shape representation. To guarantee the plausibility of generated 3D shapes we leverage the normalizing flow that models the distribution of the latent space of 3D shapes. To encourage the fidelity of the generated 3D models to an input sketch, we propose a dedicated loss that we deploy at different stages of the training process.
Bibtex

We study the practical task of fine-grained 3D-VR-sketch-based 3D shape retrieval. This task is of particular interest as 2D sketches were shown to be effective queries for 2D images. However, due to the domain gap, it remains hard to achieve strong performance in 3D shape retrieval from 2D sketches. Recent work demonstrated the advantage of 3D VR sketching on this task. In our work, we focus on the challenge caused by inherent inaccuracies in 3D VR sketches. We observe that retrieval results obtained with a triplet loss with a fixed margin value, commonly used for retrieval tasks, contain many irrelevant shapes and often just one or few with a similar structure to the query. To mitigate this problem, we for the first time draw a connection between adaptive margin values and shape similarities. In particular, we propose to use a triplet loss with an adaptive margin value driven by a "fitting gap", which is the similarity of two shapes under structure-preserving deformations. We also conduct a user study which confirms that this fitting gap is indeed a suitable criterion to evaluate the structural similarity of shapes. Furthermore, we introduce a dataset of 202 VR sketches for 202 3D shapes drawn from memory rather than from observation.
```
@inproceedings{luo2022,
  title = {Structure-Aware 3D VR Sketch to 3D Shape Retrieval},
  author = {Luo, Ling and Gryaditskaya, Yulia and Xiang, Tao and Song, Yi-Zhe},
  booktitle = {2021 International Conference on 3D Vision (3DV)},
  year = {2022},
  arxiv = {2209.09043},
  supp = {https://drive.google.com/file/d/11rt_fVuqumWUy_jVMAis4di4KW0bRHJr/view?usp=sharing},
  slides = {https://drive.google.com/file/d/1fkKf1N8SoceD_cHdZS0eZutpostHLwyr/view?usp=sharing},
  code = {https://github.com/Rowl1ng/Structure-Aware-VR-Sketch-Shape-Retrieval},
  poster = {https://drive.google.com/file/d/1MdNsnNXScARlap4UdkW8_Nj2tltzVFxH/view?usp=sharing},
  video = {https://youtu.be/hpM9WNLQmjM},
  abbr = {3DV},
  selected = {true}
}
```
3DV

Structure-Aware 3D VR Sketch to 3D Shape Retrieval

Luo Ling, Gryaditskaya Yulia, Xiang Tao, and Song Yi-Zhe

In 2021 International Conference on 3D Vision (3DV) 2022

Abs arXiv Supp Video Code Poster Slides

We study the practical task of fine-grained 3D-VR-sketch-based 3D shape retrieval. This task is of particular interest as 2D sketches were shown to be effective queries for 2D images. However, due to the domain gap, it remains hard to achieve strong performance in 3D shape retrieval from 2D sketches. Recent work demonstrated the advantage of 3D VR sketching on this task. In our work, we focus on the challenge caused by inherent inaccuracies in 3D VR sketches. We observe that retrieval results obtained with a triplet loss with a fixed margin value, commonly used for retrieval tasks, contain many irrelevant shapes and often just one or few with a similar structure to the query. To mitigate this problem, we for the first time draw a connection between adaptive margin values and shape similarities. In particular, we propose to use a triplet loss with an adaptive margin value driven by a "fitting gap", which is the similarity of two shapes under structure-preserving deformations. We also conduct a user study which confirms that this fitting gap is indeed a suitable criterion to evaluate the structural similarity of shapes. Furthermore, we introduce a dataset of 202 VR sketches for 202 3D shapes drawn from memory rather than from observation.
Bibtex

We present the first fine-grained dataset of 1,497 3D VR sketch and 3D shape pairs of a chair category with large shapes diversity. Our dataset supports the recent trend in the sketch community on fine-grained data analysis, and extends it to an actively developing 3D domain. We argue for the most convenient sketching scenario where the sketch consists of sparse lines and does not require any sketching skills, prior training or time-consuming accurate drawing. We then, for the first time, study the scenario of fine-grained 3D VR sketch to 3D shape retrieval, as a novel VR sketching application and a proving ground to drive out generic insights to inform future research. By experimenting with carefully selected combinations of design factors on this new problem, we draw important conclusions to help follow-on work. We hope our dataset will enable other novel applications, especially those that require a fine-grained angle such as fine-grained 3D shape reconstruction.
```
@inproceedings{luo2021fine,
  title = {Fine-Grained VR Sketching: Dataset and Insights.},
  author = {Luo, Ling and Gryaditskaya, Yulia and Yang, Yongxin and Xiang, Tao and Song, Yi-Zhe},
  booktitle = {2021 International Conference on 3D Vision (3DV)},
  pages = {1003--1013},
  year = {2021},
  organization = {IEEE},
  arxiv = {2209.10008},
  supp = {https://drive.google.com/file/d/1JXGO1s8pyT7YR26zruDevJwO2XJ4MAE1/view?usp=sharing},
  slides = {https://drive.google.com/file/d/1ENUhdXvaFRC-YdFI9ZMzNZJHLTMEePqM/view?usp=sharing},
  poster = {https://drive.google.com/file/d/1uMKKy-5-uf1p9kY-WM74mORR0fV-ZCah/view?usp=sharing},
  code = {https://github.com/Rowl1ng/Fine-Grained_VR_Sketching},
  website = {https://cvssp.org/data/VRChairSketch},
  img = {../assets/img/publication_preview/3D_sketch.gif},
  abbr = {3DV},
  selected = {true}
}
```
3DV

Fine-Grained VR Sketching: Dataset and Insights.

Luo Ling, Gryaditskaya Yulia, Yang Yongxin, Xiang Tao, and Song Yi-Zhe

In 2021 International Conference on 3D Vision (3DV) 2021

Abs arXiv Supp Code Poster Slides Website

We present the first fine-grained dataset of 1,497 3D VR sketch and 3D shape pairs of a chair category with large shapes diversity. Our dataset supports the recent trend in the sketch community on fine-grained data analysis, and extends it to an actively developing 3D domain. We argue for the most convenient sketching scenario where the sketch consists of sparse lines and does not require any sketching skills, prior training or time-consuming accurate drawing. We then, for the first time, study the scenario of fine-grained 3D VR sketch to 3D shape retrieval, as a novel VR sketching application and a proving ground to drive out generic insights to inform future research. By experimenting with carefully selected combinations of design factors on this new problem, we draw important conclusions to help follow-on work. We hope our dataset will enable other novel applications, especially those that require a fine-grained angle such as fine-grained 3D shape reconstruction.
Bibtex

Growing free online 3D shapes collections dictated research on 3D retrieval. Active debate has however been had on (i) what the best input modality is to trigger retrieval, and (ii) the ultimate usage scenario for such retrieval. In this paper, we offer a different perspective towards answering these questions – we study the use of 3D sketches as an input modality and advocate a VR-scenario where retrieval is conducted. Thus, the ultimate vision is that users can freely retrieve a 3D model by air-doodling in a VR environment. As a first stab at this new 3D VR-sketch to 3D shape retrieval problem, we make four contributions. First, we code a VR utility to collect 3D VR-sketches and conduct retrieval. Second, we collect the first set of 167 3D VR-sketches on two shape categories from ModelNet. Third, we propose a novel approach to generate a synthetic dataset of human-like 3D sketches of different abstract levels to train deep networks. At last, we compare the common multi-view and volumetric approaches: We show that, in contrast to 3D shape to 3D shape retrieval, volumetric point-based approaches exhibit superior performance on 3D sketch to 3D shape retrieval due to the sparse and abstract nature of 3D VR-sketches. We believe these contributions will collectively serve as enablers for future attempts at this problem.
```
@inproceedings{luo2020towards,
  abbr = {3DV},
  title = {Towards 3D VR-sketch to 3D shape retrieval},
  author = {Luo, Ling and Gryaditskaya, Yulia and Yang, Yongxin and Xiang, Tao and Song, Yi-Zhe},
  booktitle = {2020 International Conference on 3D Vision (3DV)},
  pages = {81--90},
  year = {2020},
  organization = {IEEE},
  arxiv = {2209.10020},
  supp = {https://drive.google.com/file/d/1nm8Taxc4Ji8iHdcjljygEh3v2yIdmbsO/view?usp=sharing},
  slides = {https://drive.google.com/file/d/1rhba2VxabDFz93II0Xo6e88i7S4d-puN/view?usp=sharing},
  code = {https://github.com/ygryadit/Towards3DVRSketch},
  video = {https://youtu.be/7cdij0rSOaE},
  poster = {https://drive.google.com/file/d/1ZUzV2QIHcsePXaJbzKb-KdNzzGpAWTR6/view?usp=sharing},
  img = {../assets/img/publication_preview/3DV20.gif},
  selected = {true}
}
```
3DV

Towards 3D VR-sketch to 3D shape retrieval

Luo Ling, Gryaditskaya Yulia, Yang Yongxin, Xiang Tao, and Song Yi-Zhe

In 2020 International Conference on 3D Vision (3DV) 2020

Abs arXiv Supp Video Code Poster Slides

Growing free online 3D shapes collections dictated research on 3D retrieval. Active debate has however been had on (i) what the best input modality is to trigger retrieval, and (ii) the ultimate usage scenario for such retrieval. In this paper, we offer a different perspective towards answering these questions – we study the use of 3D sketches as an input modality and advocate a VR-scenario where retrieval is conducted. Thus, the ultimate vision is that users can freely retrieve a 3D model by air-doodling in a VR environment. As a first stab at this new 3D VR-sketch to 3D shape retrieval problem, we make four contributions. First, we code a VR utility to collect 3D VR-sketches and conduct retrieval. Second, we collect the first set of 167 3D VR-sketches on two shape categories from ModelNet. Third, we propose a novel approach to generate a synthetic dataset of human-like 3D sketches of different abstract levels to train deep networks. At last, we compare the common multi-view and volumetric approaches: We show that, in contrast to 3D shape to 3D shape retrieval, volumetric point-based approaches exhibit superior performance on 3D sketch to 3D shape retrieval due to the sparse and abstract nature of 3D VR-sketches. We believe these contributions will collectively serve as enablers for future attempts at this problem.
Bibtex
```
@inproceedings{zhang2019cascaded,
  title = {Cascaded Generative and Discriminative Learning for Microcalcification Detection in Breast Mammograms},
  author = {Zhang, Fandong and Luo, Ling and Sun, Xinwei and Zhou, Zhen and Li, Xiuli and Yu, Yizhou and Wang, Yizhou},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages = {12578--12586},
  year = {2019},
  pdf = {http://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_Cascaded_Generative_and_Discriminative_Learning_for_Microcalcification_Detection_in_Breast_CVPR_2019_paper.pdf},
  abbr = {CVPR},
  selected = {true}
}
```
CVPR

Cascaded Generative and Discriminative Learning for Microcalcification Detection in Breast Mammograms

Zhang Fandong, Luo Ling, Sun Xinwei, Zhou Zhen, Li Xiuli, Yu Yizhou, and Wang Yizhou

In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2019

PDF