DISSERTATION DEFENSE
 

Author :  Ziyu Zhao
Advisors: Dr. Song Wang
Date: Aug 17, 2026
Time: 01:00 pm
Location:  2277 (M. Bert Storey Engineering and Innovation Center)
Link:  https://teams.microsoft.com/meet/240422703068254?p=EsBfT7Q4WmKzfnWa8K

Abstract
While the accuracy of semantic segmentation has improved rapidly, extending a trained model to new categories remains expensive: each addition still requires collecting
dense annotations and training again, which limits the use of segmentation where the categories of interest are not known in advance. This dissertation studies how external prior information can assist semantic segmentation in open-vocabulary and few-shot settings, and refers to this as Prior-Conditioned Dense \mbox{Prediction}. In these settings the target category is specified by an input, consumed the same way in training and at test time, rather than learned into the model's parameters, so new categories require no retraining. This is where assistance is needed: the inputs themselves are difficult to use directly, for two different reasons. A category name identifies what to segment but provides no appearance evidence with which to localize the category in a particular image. A labeled example identifies what to segment by showing it, but in 3D segmentation such examples conventionally take the form of point clouds in which the annotation is given point by point, and producing even a few of them is laborious. The dissertation therefore constructs auxiliary prior evidence, generated from the names, transferred from a cheaper modality, or synthesized from what is given, and converts it into a form directly comparable with the scene. Dense matching between the constructed evidence and the scene then yields the labels at every pixel or point. Where the constructed evidence is unreliable, because it crosses modalities or contains generated content, the model additionally adapts it to the scene or weights it by estimated quality.

Three studies instantiate this view, ordered by how much of the evidence must be constructed. In the first study, for open-vocabulary 2D segmentation, category names are paired with diffusion-generated exemplars that supply the appearance the names lack, combined into a dual-prompt cost volume refined at inference. In the second study, for label-efficient 3D segmentation, the point-wise labeled 3D examples are replaced by labeled 2D images, lifted into pseudo RGB-D point clouds by monocular depth estimation and matched to the scene through prototypes in a shared embedding space. In the third study, where a single labeled view leaves the geometry incomplete, further views are synthesized by RGB-D inpainting; since the newly revealed regions are generated rather than observed, learned view and point weights discount them when prototypes are formed. On standard 2D and 3D benchmarks, each study improves over methods that consume the supplied inputs in their original form.

Together, the studies show that auxiliary prior evidence can strengthen dense prediction well beyond what the task's own inputs support, once it is constructed into a comparable
form and consumed through localized matching. They further indicate that adaptation and weighted aggregation are best introduced according to the semantic, cross-modal, or visibility mismatch present in each setting, rather than adopted uniformly