Multi-scale Deep Representation Learning in Synthetic Biology

Tuesday, May 7, 2024 - 09:00 am


Author : Xiaoyi Liu

Advisor : Dr. Jijun Tang and Dr. Yan Tong

Date : May 07, 2024

Time:  9 am – 11: 00 pm

Place : Teams


Meeting ID: 236 573 306 493

Passcode: UTL2Gs



Synthetic biology advances and combines the expertise of engineers and biologists, bridging the gap between engineering and natural life. Synthetic biology has been generally categorized into two broad branches by developing new biological components, networks, and systems to reprogram organisms. The first branch involves using synthetic molecules to mimic natural biological functions. The second branch focuses on assembling natural biological components in novel ways, aiming to produce systems with unique, practical functions. Thus, the de novo engineering of biological modules and synthetic pathways is used in related practical bioengineering applications, such as drug-targeting strategies and microbial product manufacturing. Therefore, synthetic biology represents a new paradigm in scientific exploration and innovation, with widely used implications for our understanding and optimization of biological systems.

Over the past decades, there has been a significant increase in the amount of available whole-genome sequencing data and experimental data due to the emergence of new automation technologies, such as high-content imaging, high-throughput screening, and sequencing. Given the growth of these data sets, researchers are unable to summarize these data simply from experience and memory. Thus, stable and efficient computational methods are required to integrate them to predict or reveal new phenomena or insights that have never been discovered. However, incomplete knowledge of metabolic processes impairs the accuracy of biological systems, hindering advancements in systems biology and metabolic engineering. Additionally, some fundamental challenges still remain. Firstly, problems in systems biology are often cross-scale and multi-modal, yet existing computational methods for problem definition and model design are often single-scale and single-modal. Secondly, biological systems are multi-scale, unbalanced, and noisy, making structuring and benchmarking this complicated data very difficult. Thirdly, most natural or valuable products' complete biosynthetic pathways are unknown. Thus, computer-aided biosynthesis planning holds significant value.

To address the above challenges, we introduce multi-scale deep learning-based representation learning methodologies to understand and optimize the downstream tasks in systems biology, such as metabolic pathway inference, missing reaction prediction in GEMs, and retrosynthesis prediction. Specifically, our first study introduces a novel Multi-View Multi-Label learning framework for Metabolic Pathway Inference (MVML-MPI), which outperforms State-Of-The-Art (SOTA) methods by accurately representing the complex relationships between compounds and pathways. In the second study, to address the limitation of incomplete metabolic knowledge in GEMs, we proposed a novel framework named hypergraph Convolution network and attention mechanism integrated Explorer for GAPS prediction of metabolism termed CLOSEgaps. It is a comprehensive deep learning-driven tool that represents the hyper-topological information of GEMs and effectively fills gaps through hyperlink prediction, thereby enhancing the accuracy of phenotypic predictions. In the third study, we proposed a novel end-to-end framework for one-step retrosynthesis that combines the power of a graph encoder, which integrates learnable structural information, with the capability to sequentially translate drugs, thereby efficiently capturing chemically plausible information (RetroCaptioner). This research presents an advancement in systems biology by introducing a suite of multi-scale deep learning methodologies. These methodologies tackle key challenges such as MVML-MPI enhancing our understanding of complex metabolic pathways, CLOSEgaps innovatively filling gaps in metabolic models, and RetroCaptioner facilitating the process of retrosynthesis. Taken together, they form a comprehensive and integrated approach, and our proposed methods significantly advance the capabilities of synthetic biology.

Looking at continual learning through a dynamical system point of view

Friday, April 19, 2024 - 02:15 pm

Krishnan Raghavan

One of the critical features of an intelligent system is to continually execute tasks in a real-world environment. As a new task is revealed, we seek to efficiently adapt to a new task (improve generalization) and, in the process of generalization, we seek to remember the previous tasks (minimize catastrophic forgetting). Consequentially, there are two key challenges that must be modeled: catastrophic forgetting and generalization. Despite promising methodological advancements, there is a lack of a theoretical approach that enable analysis of these challenges.

In this talk, we discuss modelling and analysis of continual learning using tools from differential equation theory. We discuss the broad applicability of our approach and demonstrate the many applications where such an approach is required. We will derive methods in some of these applications using this point of view and show the effectiveness of such approaches in modelling these applications.

I am an assistant computational mathematician with the mathematics and computer science division at Argonne national laboratory. I received my Ph.D. in computer engineering from missouri university of science and technology in 2019 and have been at Argonne since then. My primary research agenda is to develop a mathematical characterization of machine learning (ML) models, their learning/training behavior and the associated precision achieved by them. Towards this end, I study the two broad facets of ML: theory; through the eyes of tools from systems theory, statistics and optimization; and applied; by building AI/ML models to solve key problems in nuclear physics, material science, HPC and more recently climate. I enjoy rock climbing, outdoors, cycling, love ramen and many other nerdy things including but not limited to fantasy fiction novels -- go Malazan.

Details here.

Casual analysis & decision intelligence for manufacturing at Bosch

Friday, April 12, 2024 - 02:15 pm

Bosch is a multinational engineering and technology company that develops products in various business sectors, including mobility, industrial technology, energy and building technology, and consumer goods. Currently, Bosch employs over 427K workers and generates ~100B/yr in sales revenue. At the Bosch Center for Artificial Intelligence, in Pittsburgh, we focus on research in the area of neuro-symbolic AI, combining machine learning with knowledge engineering technologies. In this talk, we will illustrate recent efforts in the areas of causal analysis and decision intelligence to improve industrial manufacturing processes. More specifically, we discuss the application of neuro-symbolic methods for (1) root-cause analysis and (2) cognitive architectures for decision making.

About the authors
Alessandro Oltramari is president of the Carnegie Bosch Institute and a senior research scientist at Bosch Center for Artificial Intelligence in Pittsburgh, USA. Oltramari joined Bosch Research in 2016, after working as a research associate at Carnegie Mellon University, funded by public agencies like DARPA, NSF, ARL. At Bosch Research, he focuses on neuro-symbolic AI. His primary interest is to investigate how knowledge-based methods and systems can be integrated with learning algorithms, and help humans and machines make sense of the physical and digital worlds. Contact him at

Cory Henson is a lead research scientist at the Bosch Center for Artificial Intelligence in Pittsburgh, USA. His research focuses on knowledge representation and neuro-symbolic AI methods, integrating machine learning with prior domain knowledge. He has led projects to develop and apply this technology for improving autonomous systems, ranging from automated driving to smart manufacturing. More recently, he has become interested in the use of neuro-symbolic methods for representing, learning, and reasoning with causal knowledge. Contact him at

Details at:

The LLM Journey

Friday, March 29, 2024 - 02:15 pm
Zoom or in person at SWGN 2A27.

Large Language Models (LLMs) have dramatically transformed the landscape of Generative AI, making profound impacts across a broad spectrum of domains. From enhancing Recommender Systems to advancing the frontiers of Natural Language Processing (NLP), LLMs have become indispensable. Their versatility extends into specialized sectors, such as finance with the development of BloombergGPT, and healthcare through MedLlama, showcasing their adaptability and potential for industry-specific innovations.

In this presentation, we will embark on a comprehensive exploration of the evolution of Large Language Models. Our journey will trace the origins of LLMs, highlighting key milestones and breakthroughs, and proceed to examine the latest advancements and research directions in the field. To mirror the structured and layered nature of LLMs themselves, our discussion will be organized into distinct sections. We'll begin with the foundational aspect of prompting, delve into the intricacies of their architecture, and discuss pivotal strategies such as Pretraining, Fine-tuning, and Parameter Efficient Fine-Tuning (PEFT). Furthermore, we'll address the challenges and solutions related to the mitigation of hallucination, a critical aspect of ensuring the reliability and accuracy of LLM-generated content.

Speaker Bio:

Vinija Jain brings to the table an extensive background in machine learning, with significant expertise in developing recommender systems at Amazon and spearheading NLP initiatives at Oracle. Her passion for artificial intelligence was ignited during her time in the Stanford AI program, which served as a catalyst for her deep dive into the field. Currently, Vinija is actively engaged in fundamental research and collaborates with the Artificial Intelligence Institute of South Carolina (AIISC) at the University of South Carolina. Her latest work with AIISC on AI-Generated Text Detection has been recognized with an outstanding paper award at EMNLP '23, underscoring her contributions to advancing AI research and application

Realtime Machine Learning on Edge AI Accelerators

Friday, March 22, 2024 - 02:15 pm

Abstract: Several real-world applications of machine learning (ML) systems such as robotics, autonomous cars, assistive technologies, smart manufacturing, and many other Internet-of-Things (IoT) applications require real-time inference with low energy consumption. The surge in demand for specialized hardware for AI applications has resulted in a rapidly expanding industry for edge AI accelerators. Anticipating this trend, several companies have developed their own specialized accelerators such as the NVIDIA Jetson Nano, Intel NCS2, and Google TPU. While many conventional neural networks can be readily deployed on many of these platforms, the support for deploying more advanced and larger models such as transformers on them has yet to be researched and developed. In this talk, we discuss two of our recent projects in which we utilize optimization mechanism such neural architecture search (NAS) and system-level innovations such as modifying the computational graphs, partitioning, and refactoring the unsupported operations to efficiently deploy ML models on edge accelerators for computer vision and natural language processing tasks.

Bio: Dr. Ramtin Zand is an assistant professor of the Computer Science and Engineering and the principal investigator of the Intelligent Circuits, Architectures, and Systems (iCAS) Lab at the University of South Carolina. The iCAS lab has close collaborations with and is supported by several multinational companies including Intel, AMD, and Juniper Networks, as well as federal agencies such as National Science Foundation (NSF). Dr. Zand has authored more than 50 journal and conference articles and two book chapters and received recognition from ACM/IEEE including the best paper runner-up of ACM GLSVLSI’18, the best poster of ACM GLSVLSI’19, and best paper of IEEE ISVLSI’21, as well as featured paper in IEEE Transactions on Emerging Topics in Computing. He has received the NSF CAREER award in 2024. His research focus is on neuromorphic computing, edge computing, processing-in-memory, and AI/ML hardware acceleration.


Online at:

Automated Data-flow optimization for Digital Signal Processors

Monday, March 18, 2024 - 10:00 am


Department of Computer Science and Engineering

University of South Carolina


Author : Madushan Abeysinghe
Advisor : Dr. Jason Bakos
Date : March 18, 2024
Time:  10 am – 11: 00 pm

Place : Teams

Meeting ID: 234 681 711 471

Passcode: wiatfJ




Digital signal processors (DSP), which are characterized by statically-scheduled Very Long Instruction Word architectures and software-defined scratchpad memory, are currently the go-to processor type for low-power embedded vision systems, as exemplified by the DSP processors integrated into systems-on-chips from NVIDIA, Samsung, Qualcomm, Apple, and Texas Instruments. DSPs achieve performance by statically scheduling workloads, both in terms of data movement and instructions. We developed a method for scheduling buffer transactions across a data flow graph using data-driven performance models, yielding a 25% average reduction in execution time and a reduction of up to 85% DRAM utilization for randomly-generated data flow graphs. We also developed a heuristic instruction scheduler that serves as a performance model to guide the selection of loops from a target data flow graph to be fused. By strategically selecting loops to fuse, performance gains can be achieved by eliminating unnecessary transactions with memory and increasing functional unit utilization. This approach has helped us achieve up to 1.9x speedup on average for sufficiently large data flow graphs used in image processing.

CSE Research Symposium

Friday, March 15, 2024 - 10:30 am
Story Innovation Center

CSE Graduate Student research poster presentations. 

Below is a tentative agenda for the day. If you plant to attend the luncheon, please RSVP by at this link  by Monday, March 11.

Location: 550 Assembly St, Room 2277, Columbia, SC 29201

10:30 am - 12:00 pm: Morning Poster session, 2nd floor hallways

1.Information Competition Simulator: A High-Performance Approach to Modeling Opinion Dynamics in Large PopulationsErik Connerty
3.Orthogonal Dictionary Guided Shape Completion Network for Point CloudPingping Cai
5.Expressive and Flexible Simulation of Information Spread Strategies in Social Networks Using PlanningBharath Muppasani
7.Can I eat this food or not? Explainable food recommendation through multi-contextual GroundingRevathy Venkataramanan
9.Automating the Analysis of Rodent Ultrasonic VocalizationsSabah S. Anis
11.Understanding Information Spread in Dynamic Networks: A Graph Neural Network and Reinforcement Learning Based ApproachProtik Nag
13.Promoting Research Collaboration with Open Data Driven Team Recommendation in Response to Call for ProposalsSiva Likitha Valluru
15.Systematic comparison of EEG brain connectivity metricsDiksha Srishyla
17.Smartwatch-Based Smoking Detection Using Accelerometer Data and Masked AutoEncoder based Neural NetworksMusa Azeem
19.Predicting Language Outcomes from MRI Post-Stroke: A Machine Learning ApproachDeepa Tilwani
21.Deep Learning For Human Vascular AnalysisAli Firooz
23.Adaptive Channel Switching for Connected Vehicles under Extreme Weather Conditions: A Reinforcement Learning Based ApproachJian Liu
25.RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly PipelinesChathurangi Shyalika; Renjith Prasad Kaippilly Mana
27.Crystal structure prediction using neural network potential and age-fitness Pareto genetic algorithmSadman Sadeed Omee
29.An Integrated Transformer-Based Framework for Enhanced Longitudinal Prediction of Low BirthweightYang Ren
31.mLIFE Student Preliminary ResultsLexington Whalen
33.SafeChat: A Framework to Build Safe and Trustworthy ChatbotsKausik Lakkaraju
35.Enhancing Injection Molding with Industry 4.0: Towards Smart Manufacturing and Real-Time Quality PredictionXiaoyi Liu
37.Crystal structure prediction using neural network potential and age-fitness Pareto genetic algorithmSadman Sadeed Omee; Lai Wei; Ming Hu; Jianjun Hu
39.Knowledge Graphs Construction & Alignment At ScaleHong Yung Yip
41.Foundation Model for Pathfinding ProblemsVedant Khandelwal
43.PixelPrune: Sparse Object Detection for AIoT Systems via In-Sensor Segmentation and Adaptive Data TransferMohammadreza Mohammadi
45.Computing Heuristics by RelaxationRoss Foultz; Marco Valtorta


12:00 pm - 1:30 pm: Lunch in room 2277


1:30 pm - 3:00 pm: Afternoon poster session, 2nd floor hallway

2.Rethinking Robust Contrastive Learning from the Adversarial PerspectiveFatemeh Ghofrani
4.MilliCar: Accurate 3D Bounding Box Prediction of Vehicles and Pedestrians in All Weather ConditionsReza Tavasoli; Hem Regmi
6.Analysis of cancer patients molecular and clinical dataAli Firooz
8.Cross modal few-shot point cloud semantic segmentationziyu zhao
10.MilliCar: Accurate 3D Bounding Box Prediction of Moving Vehicles with Millimeter-Wave Radar in All Weather ConditionsReza Tavasoli; Hem Regmi; Joseph Telaak
12.Learning Discrete World Models for PlanningMisagh Soltani
14.Realtime Facial Expression Recognition: Neuromorphic Hardware vs. Edge AI AcceleratorsJames Seekings
16.Assessing Cognitive Measures in Aging through Discourse TasksYong Yang
18.Web Application for Searching and Displaying Cancer Patient DataSavannah Noblitt
20.Physics Guided Dual Self-supervised Learning for Structure-based Material Property PredictionNihang Fu
22.Flex-TPU: A Flexible TPU Architecture with Runtime Reconfigurable DataflowPeyton Chandarana
24.Explainable Chemical Reaction Predictions using Deep Approximate Value IterationChristian Geils
26.Flex-TPU: A Flexible TPU Architecture with Runtime Reconfigurable DataflowPeyton Chandarana
28.RDR: the Recap, Deliberate, and Respond Method for Enhanced Language UnderstandingYuxin Zi
30.Efficient Deployment of Transformer Models on Edge TPU Accelerators: A Real System Evaluation​Mohammadreza Mohammadi
32.Deep Learning-Based Classification of Gamma Photon Interaction in Room-Temperature Semiconductor Radiation DetectorsQinyang Li
34.Auto-req: Automatic detection of pre-requisite dependencies between academic videosRushil Thareja
38.CURE: Simulation-Augmented Auto-Tuning in RoboticsMd Abir Hossen
40.Bidirectional Autoregessive Diffusion Model for Dance GenerationCanyu Zhang
42.SMARPchain: A Smart Marker Based Reputational Probabilistic Blockchain for Multi-Agent SystemsMatthew Sharp


3:00 pm - 4:00 pm: Presentations and poster session winners, room 1400 (1st floor)

03:00 PMAbir HossenJamshidi Lab
03:05 PMSiva Likitha ValluruAI4Society / Srivastava Lab
03:10 PMFawad KirmaniRose Lab
03:15 PMXeerak MuhammadTong Lab
03:20 PMMisagh SoltaniAgostinelli Lab
03:25 PMIoannis RekleitisRekleitis Lab
03:30 PMPeng FuFu Lab
03:35 PMDipannoy GuptaZhang Lab
03:40 PMChristian O'ReillyO'Reilly Lab
03:45 PMJian LiuHuang Lab
03:50 PMRongzhi DongHu Lab
03:55 PMRamtin ZandZand Lab
04:00 PMPingping CaiWang Lab

Scene Text Detection and Recognition via Discriminative Representation

Monday, March 11, 2024 - 08:30 am


Author : Liang Zhao
Advisor : Dr. Song Wang
Date : March 11, 2024
Time:  8:30am-10:00am
Place : Teams Link


Scene texts refer to arbitrary texts presented in an image captured by a camera in the real world. The tasks of scene text detection and recognition from complex images play a crucial role in computer vision, with potential applications in scene understanding, information retrieval, robotics, autonomous driving, etc. Despite the notable progress made by existing deep-learning methods, achieving accurate text detection and recognition remains challenging for robust real-world applications. The challenges in scene text detection and recognition stem from: 1) diverse text shapes, fonts, colors, styles, layouts, etc.; 2) countless combinations of characters with unstable attributes for complete detection, coupled with background interruptions that obscure character strokes and shapes in text recognition; and 3) the need for effective coordination of multiple sub-tasks in end-to-end learning. The fundamental issue lies in the absence of a particularly discriminative representation for the detection task, which involves locating exact complete words with unfixed attributes, and for the recognition task, which entails differentiating similar characters within words. Our research aims to address these challenges and enhance scene text detection and recognition by improving text discriminative representation. In this study, we focus on two interconnected problems: 1) Scene Text Recognition (STR), which involves recognizing text from scene images, and 2) Scene Text Spotting (STS), which entails simultaneously detecting and recognizing multiple texts in scene images.

Addressing the challenges of Scene Text Recognition (STR), the presence of text variations and complex backgrounds remain significant hurdles due to their impact on text feature representation. Numerous existing methods attempt to mitigate these issues by employing attentional regions, bounding boxes, or polygons. Despite these efforts, the text regions identified by such methods often retain undesirable background interference. In response, we propose a Background-Insensitive Network (BINet) that explicitly incorporates text Semantic Segmentation (SSN) to enhance the text representation and reduce the background interruptions. This approach eliminates the need for extensive pixel-level annotations in the STR training data. To maximize the benefits of semantic cues, we introduce novel segmentation refinement and embed?ding modules that refine text masks and strengthen visual features. Experimental results demonstrate that our proposed method significantly improves text recognition in the presence of complex backgrounds, achieving state-of-the-art performance across multiple public datasets.

In tackling the problem of Scene Text Spotting (STS), we introduce two novel developments. Given that the task involves a multi-task model dedicated to locating and recognizing texts in scenes, the coordination of multiple sub-tasks can exert a significant impact on each other and, subsequently, on the overall performance. Current end-to-end text spotters commonly incorporate independent sequential pipelines to conduct different multi-tasks. However, this unidirectional pipeline leads to information loss and error propagation among sub-tasks. In light of these observations, we present CommuSpotter, designed to enhance multi-task communication by explicitly and concurrently exchanging compatible information throughout the scene text spotting process. Experimental results demonstrate that our improved text representation for both sub-tasks enhances performance across public datasets.

Another prominent limitation in multi-tasks coordination in Scene Text Spotting (STS) lies in the capability of extracting and refining text representation of instances for multiple sub-tasks. Existing methods often utilize features from Convolutional Neural Networks (CNNs) and shrink the text regions in representation to perform sequential tasks. Nevertheless, the effectiveness of these methods is primarily constrained by the contextual biases inherent in the representation of CNN backbones. These biases are challenging to filter out, complicating the identification of randomly appearing texts and introducing confusion in discerning the similar characters within text instances. In response to these challenges, we propose a novel approach named Assembling Text Spotter (ATS) to mitigate the problem. ATS initially decouples image contextual information from text structure information through the separation of dual backbones. The disentanglement of image and text information eliminates the need for filtering out one from the other. Subsequently, they are dynamically and purposely aligned to generate discriminative representations for different sub-tasks. Extensive experiments conducted on existing scene text datasets demonstrate competitive performance results across multiple benchmarks for scene text spotting.