A curated starting point for recent work in distributed systems, cloud computing, databases, storage systems, and artificial intelligence—with a dedicated emphasis on research from industry labs. Every entry links to its source abstract and paper.
155 papers
Consensus, replication, decentralization, fault tolerance, and distributed coordination.
Introduces replica assignment that protects high-priority sessions while allowing capacity borrowing in constrained replicated database clusters.
Models quorum-formation latency and multimodal block-time distributions to diagnose heterogeneous network behavior in HotStuff-based BFT systems.
Uses decentralized monitoring and declarative actions to add self-healing, self-protection, and self-optimization to microservices.
Combines a token ring with cache-local rendezvous candidates to improve balance and lookup throughput while preserving minimal churn.
The scaling of Large-scale Language Models (LLMs) demands unprecedented computational power and bandwidth. We present UB-Mesh, an innovative AI datacenter network architecture that enhances scalability, performance, and cost-efficiency through a hierarchical nD-FullMesh topology.
Memory disaggregation has become a major trend in cloud-native databases. However, most existing memory disaggregation solutions suffer from read/write amplification, limited bandwidth, inefficient recovery, and challenges in data sharing.
Due to characteristics such as the openness and exposure of inter-satellite links, Low-Earth Orbit (LEO) satellite networks are subject to heightened vulnerability to malicious attacks compared to ground-based networks. Given various security risks, implementing trust management in LEO satellite networks becomes imperative.
Remote data structures built with one-sided Remote Direct Memory Access (RDMA) are at the heart of many disaggregated database management systems today. Concurrent access to these data structures by thousands of remote workers necessitates a highly efficient synchronization scheme.
Presents error-free, information-theoretically secure asynchronous multi-valued Byzantine agreement with near-optimal communication.
Combines reputation, outlier detection, and blockchain-backed identity to defend cross-device federated learning against poisoned updates.
As an important component of the space-air-ground integrated network, aerial base station (AeBS) systems have gained significant attention for their flexibility in mobility and cost-effective construction. Nevertheless, the scarce spectrum resources and difficulty in accessing global information bring necessity and challenges to the deployment and…
Serverless computing is renowned for its computation elasticity, yet its full potential is often constrained by the requirement for functions to operate within local and dedicated background environments, resulting in limited memory elasticity. To address this limitation, this paper introduces TrEnv, a co-designed integration of the serverless platform…
RDMA-based in-memory storage systems offer high performance but are restricted by the capacity of physical memory. In this article, we propose TeRM to extend RDMA-attached memory with SSD.
With the increasing demand for blockchain technology in various industry sectors, there has been a growing interest in the Byzantine Fault Tolerance (BFT) consensus that is the backbone of most of these blockchains. However, many state-of-the-art algorithms that require reliable connections can only offer limited throughput in wide-area networks (WANs),…
Long-running containerized workloads (e. g.
Distributed databases have become indispensable in contemporary computing and data processing, owing to their pivotal role in ensuring high availability and scalability. They effectively cater to the requirements of data management and high-concurrency access.
Relational network verification is a new approach for validating network changes. In contrast to traditional network verification, which analyzes specifications for a single network snapshot, it analyzes specifications that capture similarities and differences between two network snapshots (e.
Distributed tracing has been widely adopted in many microservice systems and plays an important role in monitoring and analyzing the system. However, trace data often come in large volumes, incurring substantial computational and storage costs.
In a geo-distributed database, data shards and their respective replicas are deployed in distinct datacenters across multiple regions, enabling regional-level disaster recovery and the ability to serve global users locally. However, transaction processing in geo-distributed databases requires multiple cross-region communications, especially during the…
Disaggregated memory separates compute and memory resources into independent pools connected by RDMA (Remote Direct Memory Access) networks, which can improve memory utilization, reduce cost, and enable elastic scaling of compute and memory resources. However, existing RDMA-based distributed transactions on disaggregated memory suffer from severe…
Decentralized Storage Network (DSN) is an emerging technology that challenges traditional cloud-based storage systems by consolidating storage capacities from independent providers and coordinating to provide decentralized storage and retrieval services. However, current DSNs face several challenges associated with data privacy and efficiency of the…
While modern internet services, such as chatbots, search engines, and online advertising, demand the use of large-scale deep neural networks (DNNs), distributed training and inference over heterogeneous computing systems are desired to facilitate these DNN models. Mixture-of-Experts (MoE) is one the most common strategies to lower the cost of training…
Deep Learning training jobs process large amounts of training data using many GPU devices, often running for weeks or months. When hardware or software failures happen, these jobs need to restart, losing the memory state for the Deep Neural Network (DNN) model trained so far, unless checkpointing mechanisms are used to save training state periodically.
Large language models (LLMs) have triggered tremendous success to empower our daily life by generative information. The personalization of LLMs could further contribute to their applications due to better alignment with human intents.
Byzantine fault-tolerant (BFT) state machine replication (SMR) is adopted to support blockchain consensus by tolerating arbitrarily faulty behaviours. However, the inherent complexity of BFT protocols makes existing BFT protocols hard to adapt to large-scale applications that require high scalability and performance.
Distributed protocols such as 2PC and Paxos lie at the core of many systems in the cloud, but standard implementations do not scale. New scalable distributed protocols are developed through careful analysis and rewrites, but this process is ad hoc and error-prone.
With the rapid development of blockchain and its applications, the amount of data stored on decentralized storage networks (DSNs) has grown exponentially. DSNs bring together affordable storage resources from around the world to provide robust, decentralized storage services for tens of thousands of decentralized applications (dApps).
Function-as-a-Service (FaaS) offers a fine-grained resource provision model, enabling developers to build highly elastic cloud applications. User requests are handled by a series of serverless functions step by step, which forms a multi-step workflow.
As humanity’s reach in space exploration extends beyond Earth’s orbit, the demand for robust and efficient communication systems intensifies. The traditional paradigm of isolated missions has given way to a complex network of diverse spacecraft engaged in terrestrial research, lunar exploration, and the provision of essential satellite services.
Memory disaggregation is a promising architecture for modern datacenters that separates compute and memory resources into independent pools connected by ultra-fast networks, which can improve memory utilization, reduce cost, and enable elastic scaling of compute and memory resources. However, existing memory disaggregation solutions based on remote…
In the modelling of distributed systems, most Model of Computations (MoCs) rely on blocking communication to preserve determinism. A prominent example is Kahn Process Networks (KPNs), which supports non-blocking writes and blocking reads, and its implementable variant Finite FIFO Platforms (FFPs) which enforces boundedness using blocking writes.
Cloud-native platforms, serverless systems, scheduling, resource management, and edge computing.
Lets a serverless control plane influence Linux CPU scheduling through SchedExt domains, reducing loaded-request latency and energy use.
Designs a hierarchical discovery and negotiation architecture for dispatching agentic scientific workloads across cloud, edge, and HPC resources.
Coordinates predictive, policy-aware resource decisions across clusters to improve utilization, adaptation speed, and workload stability.
Provides event-driven abstractions for decentralized runtime adaptation without imposing a centralized cloud controller.
The scaling of Large-scale Language Models (LLMs) demands unprecedented computational power and bandwidth. We present UB-Mesh, an innovative AI datacenter network architecture that enhances scalability, performance, and cost-efficiency through a hierarchical nD-FullMesh topology.
Despite efforts to improve the quality of service (QoS) and throughput in Large Language Model (LLM) serving systems, progress is often limited by the lack of publicly available real-world workloads. Consequently, evaluations usually depend on synthetic or oversimplified load patterns, and systems that appear promising in testing frequently underperform…
The evolution of cloud computing has given rise to Cloud Native Applications (CNAs), presenting new challenges in governance, particularly when faced with strict compliance requirements. This work explores the unique characteristics of CNAs and their impact on governance.
The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in cloud datacenters. Traditional techniques are often inadequate for LLM inference due to the fine-grained, millisecond-scale execution phases, each with distinct performance, thermal, and power profiles.
Blockchain offers distinct advantages in terms of data credibility and provenance certification, and its fusion with Internet of Things (IoT) technology holds great promise. Nevertheless, IoT environments are marked by extensive node networks and intricate communication patterns, especially the sensing environment.
Conventional cloud services and infrastructures are mainly designed to maximize utilization of resources and provide best-effort Quality-of-Service levels. However, many emerging use cases in both public and private cloud computing scenarios are time-critical in nature.
This paper considers the carbon-aware data center (DC) capacity provisioning problem under uncertain green energy availability and computing demand. To address it, accurate carbon emissions estimation and robust capacity provisioning are necessary.
Schedules deadline-sensitive inference across drones, edge devices, and serverless cloud functions using dropping, stealing, migration, and adaptation.
Adapts server-side rendering quality to network compression so constrained edge servers spend compute where users can perceive it.
An effective auto-scaling framework is essential for microservices to ensure performance stability and resource efficiency under dynamic workloads. As revealed by many prior studies, the key to efficient auto-scaling lies in accurately learning performance patterns, i.
Since the advent of cloud computing, Elastic Computing (EC) has become the standard architecture for resource allocation and scheduling. EC typically allocates computing resources based on predefined specifications, such as virtual machine or container flavors.
Microservices improve the scalability and flexibility of monolithic architectures to accommodate the evolution of software systems, but the complexity and dynamics of microservices challenge system reliability. Ensuring microservice quality requires efficient failure diagnosis, including detection and triage.
Nowadays, there exists a lot of cross-region data transmission demand on the cloud. It is promising to use serverless computing for data compressing to save the total data size.
Serverless computing provides developers with a maintenance-free approach to resource usage, but it also transfers resource management responsibility to the cloud platform. However, the fine granularity of serverless function resources can lead to performance bottlenecks and resource fragmentation on nodes when creating many function containers.
Long-running containerized workloads (e. g.
This paper presents HPN, Alibaba Cloud's data center network for large language model (LLM) training. Due to the differences between LLMs and general cloud computing (e.
Deep learning training (DLT), e. g.
We design, implement, and evaluate YuanRong, the first production general-purpose serverless platform with a unified programming interface, multi-language runtime, and a distributed computing kernel for cloud-based applications. YuanRong addresses many limitations of existing Function-as-a-Service (FaaS) systems, particularly in performance and lack of…
This article overviews some of the key past developments in cloud data center power and energy management, where we are today, and what the future could be. This topic is gaining enormous renewed interest in the context of the conflicting needs of the AI revolution and the climate crisis.
To mitigate climate change, we must reduce carbon emissions from hyperscale cloud computing. We find that cloud compute servers cause the majority of emissions in a general-purpose cloud.
The power consumption flexibility provided by the energy-intensive Internet data centers (IDCs) has been extensively studied as a potential solution for enhancing the flexibility of power systems. In IDCs, computational workloads are further divided into potentially interdependent tasks.
Recent innovation in large language models (LLMs), and their myriad use cases have rapidly driven up the compute demand for datacenter GPUs. Several cloud providers and other enterprises plan to substantially grow their datacenter capacity to support these new workloads.
Due to the fast developments of 5G and IoT technologies, Inter-Datacenter (Inter-DC) networks are facing unprecedented pressure to duplicate large volumes of geographically distributed user data in a real-time manner. Meanwhile, with the expansion of Inter-DC networks scale, link/node failures also become increasingly frequent, negatively affecting the…
Resource management challenges frequently manifest in systems and networks as tough online decision tasks, for which the proper solution is dependent on an understanding of the workload and environment and facilitates smooth use of mobile edge and cloud resources. Due to the geographical dispersion of resources, constrained resource capacity,…
Function-as-a-Service (FaaS) offers a fine-grained resource provision model, enabling developers to build highly elastic cloud applications. User requests are handled by a series of serverless functions step by step, which forms a multi-step workflow.
The wide adoption of Docker containers for supporting agile and elastic enterprise applications has led to a broad proliferation of container images. The associated storage performance and capacity requirements place a high pressure on the infrastructure of container registries that store and distribute images and container storage systems on the Docker…
This study examines Oracle Insurance Policy Administration (OIPA) Coud Migration projects, analyzing 30 implementations that migrated from SQL Server to Oracle Cloud Infrastructure (OCI) environments. The research focuses on Universal Life Insurance systems migrating from AWS-hosted environments to Oracle’s cloud platform, including site upgrades from…
Query processing, indexing, transactions, data quality, and modern database architectures.
Enforces priority differentiation through replica assignment while retaining high utilization through controlled capacity borrowing.
Combines symbolic database analysis, language-model inference, and targeted user questions to construct usable analytic semantic schemas.
Modern database management systems (DBMSs) expose hundreds of configuration knobs that critically influence performance. Existing automated tuning methods either adopt a data-driven paradigm, which incurs substantial overhead, or rely on manual-driven heuristics extracted from database documentation, which are often limited and overly generic.
Reviews recent work that makes the optimal algorithm for acyclic joins practical in modern query engines.
Balances lookup, range-query, update, stability, and space objectives in one learned-index framework.
With the proliferation of Large Language Models (LLMs) in Business Intelligence (BI), existing solutions face critical challenges in industrial deployments: functionality deficiencies from legacy systems failing to meet evolving LLM-era user demands, interaction limitations from single-round SQL generation paradigms inadequate for multi-round…
There has been considerable interest in leveraging GPUs' computational power and high memory bandwidth for analytical database workloads. However, their limited memory capacity remains a fundamental limitation for databases whose sizes far exceed the GPU memory size.
Acyclic conjunctive queries form the backbone of most analytical workloads, and have been extensively studied in the literature from both theoretical and practical angles. However, there is still a large divide between theory and practice.
Graphs are ubiquitous data structures used in a large spectrum of applications, spanning from transportation networks, financial networks, social networks, product-order transactions and biomedical applications [33]. A recent survey on the usage of graph applications from real users has highlighted the fact that analytics is the most time-consuming task…
The recent ISO SQL:2023 standard adopts SQL/PGQ (Property Graph Queries), facilitating graph-like querying within relational databases. This advancement, however, underscores a significant gap in how to effectively optimize SQL/PGQ queries within relational database systems.
Join evaluation is one of the most fundamental operations performed by database systems and arguably the most well-studied problem in the Database community. A staggering number of join algorithms have been developed, and commercial database engines use finely tuned join heuristics that take into account many factors including the selectivity of…
Cardinality estimation (CardEst) is essential for optimizing query execution plans. Recent ML-based CardEst methods achieve high accuracy but face deployment challenges due to high preparation costs and lack of transferability across databases.
Many organizations have embraced the "Lakehouse" data management paradigm, which involves constructing structured data warehouses on top of open, unstructured data lakes. This approach stands in stark contrast to traditional, closed, relational databases and introduces challenges for performance and stability of distributed query processors.
CXL-based disaggregated memory systems offer options to expand the memory beyond the limits of a single server via cache-coherent memory expansion cards or memory pools. Especially, In-Memory Database Management Systems (IMDBMSs) can benefit from alleviating two critical constraints: (1) limited memory capacity in a server and (2) long restart time…
SQL has been extremely successful as the de facto standard language for working with data. Virtually all mainstream database-like systems use SQL as their primary query language.
Distributed databases have become indispensable in contemporary computing and data processing, owing to their pivotal role in ensuring high availability and scalability. They effectively cater to the requirements of data management and high-concurrency access.
In recent years at Tencent, we have observed that the use of multiple NoSQL databases for storing business data with diverse models has led to increased programming and deployment costs, as well as inefficient maintenance and underutilized resources. In this paper, we report X-Stor, a cloud-native NoSQL database system that supports multiple data models…
Maximizing transaction throughput is key to high-performance database systems, which focus on minimizing data access conflicts to improve performance. However, finding efficient schedules that reduce conflicts remains an open problem.
Database research and development is heavily influenced by benchmarks, such as the industry-standard TPC-H and TPC-DS for analytical systems. However, these twenty-year-old benchmarks neither capture how databases are deployed nor what workloads modern cloud data warehouse systems face these days.
Abstract To provide applications with the ability to analyze fresh data and eliminate the time-consuming ETL workflow, hybrid transactional and analytical (HTAP) systems have been developed to serve online transaction processing and online analytical processing workloads in a single system. In recent years, HTAP systems have attracted considerable…
Memory disaggregation can potentially allow memory-optimized range indexes such as B+-trees to scale beyond one machine while attaining high hardware utilization and low cost. Designing scalable indexes on disaggregated memory, however, is challenging due to rudimentary caching, unprincipled offloading and excessive inconsistency among servers.
The evaluation of top-k conjunctive queries, a staple in business analysis, often requires evaluating the conjunctive query prior to filtering the top-k results, leading to a significant computational overhead within Database Management Systems (DBMSs). While efficient algorithms have been proposed, their integration into DBMSs remains arduous.
Index tuning aims to find the optimal index configuration for an input workload. It is often a time-consuming and resource-intensive process, largely attributed to the huge amount of "what-if" calls made to the query optimizer during configuration enumeration.
Query performance (e. g.
Relational tables, where each row corresponds to an entity and each column corresponds to an attribute, have been the standard for tables in relational databases. However, such a standard cannot be taken for granted when dealing with tables "in the wild".
Classifies hybrid transactional/analytical systems by storage architecture and reviews synchronization, optimization, and scheduling techniques.
Uses cardinality-estimation consistency rules to find confirmed performance bugs in widely used database systems.
Distributed protocols such as 2PC and Paxos lie at the core of many systems in the cloud, but standard implementations do not scale. New scalable distributed protocols are developed through careful analysis and rewrites, but this process is ad hoc and error-prone.
Database systems often rely on historical query traces to perform workload-based performance tuning. However, real production workloads are time-evolving, making historical queries ineffective for optimizing future workloads.
Join ordering and query optimization are crucial for query performance but remain challenging due to unknown or changing characteristics of query intermediates, especially for complex queries with many joins. Over the past two decades, a spectrum of techniques for adaptive query processing (AQP)---including inter-/intra-operator adaptivity and tuple…
Efficient query optimization is crucial for database management systems. Recently, machine learning models have been applied in query optimizers to generate better plans, but the unpredictable performance regressions prevent them from being truly applicable.
Caching, checkpointing, tiered memory, I/O, and distributed data management.
Reports checkpoint, NFS, page-cache, telemetry, and deadlock lessons from full fine-tuning a 32.76B model across two B300 nodes.
Introduces Tensor-as-a-Service and a programmable distributed layer for weights, KV caches, checkpoints, and tensor movement.
Synthesizes safe, deployment-specific heuristics for cache eviction, tiered memory, and spot-VM scheduling using restricted generated code.
Measures aggregation, alignment, coalescing, buffered I/O, direct I/O, and io_uring for large-model checkpoint/restore workloads.
Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase.
In this article, we investigated the effects of cell variations, specifically the variations in gate length (${L}_{\text {g}}$), spacer length (${L}_{\text {s}}$), filler oxide thickness (${T}_{\text {f}}$), channel thickness (${T}_{\text {ch}}$), tunneling oxide thickness (${T}_{\text {tox}}$), charge trap nitride thickness (${T}_{\text {ctn}}$), and…
Fully Homomorphic Encryption (FHE) is a technique that allows arbitrary computations to be performed on encrypted data without the need for decryption, making it ideal for secure computation outsourcing. However, computation on FHE-encrypted data is significantly slower than that on plain data, primarily due to the explosive increases in data size and…
Sustainable and cost-effective long-term storage remains an unsolved problem. The most widely used storage technologies today are magnetic (hard disk drives and tape).
This article traces the evolution of SSD (solid-state drive) interfaces, examining the transition from the block storage paradigm inherited from hard disk drives to SSD-specific standards customized to flash memory. Early SSDs conformed to the block abstraction for compatibility with the existing software storage stack, but studies and deployments show…
Applies reinforcement learning to adapt cache size, queue depth, and readahead settings to changing I/O workloads.
Predicts write-only data so cloud block-storage caches can avoid low-value SSD writes and improve utilization.
Holographic Storage is an old idea that has always promised high density and fast random access, but has never been commercially competitive with Hard Disk Drives (HDDs) and Solid State Devices (SSDs). In Project HSD at Microsoft Research we asked the question: “Does holographic storage finally make sense for cloud storage?
The growing volume of graph data may exhaust the main memory. It is crucial to design a disk-based graph storage system to ingest updates and analyze graphs efficiently.
Major cloud providers have committed to lowering carbon emissions by 2030 across their datacenters, and research has contributed many ideas on how this may be achieved. However, a major contributor to datacenter emissions has not received enough attention: storage.
We propose an electron back-tunneling (EBT) method to enhance the retention characteristics of vertical NAND (V-NAND) flash memory. The storage of back-tunneled electrons in the spacer region between adjacent cells is facilitated by the synergistic effect of the fringing electric field between adjacent word-lines.
In this work, we have developed a large memory window (MW) ferroelectric field effect transistor (FeFET) memory for vertical NAND storage. We demonstrate that: 1) by inserting a top functional layer above the ferroelectric, gate side injection pumped by ferroelectric switching event can be enhanced, thus increasing the MW; 2) inspired by the charge trap…
We study the disturb characteristics of ferroelectric field-effect transistors (FEFETs) with band-engineered gate stacks. We demonstrate that integrating a dielectric Al2O3 layer within the ferroelectric (FE) Hf$_{{0}.
We optimize the shape of the pulse to maximally increase threshold voltage (${V}_{\text {th}}\text {)}$during the incremental step pulse programming (ISPP) of vertical NAND (V-NAND) flash memory using neural networks (NNs). NN is trained using data on the increase in${V}_{\text {th}}$of commercial V-NAND flash memory in response to randomly shaped…
CXL-based disaggregated memory systems offer options to expand the memory beyond the limits of a single server via cache-coherent memory expansion cards or memory pools. Especially, In-Memory Database Management Systems (IMDBMSs) can benefit from alleviating two critical constraints: (1) limited memory capacity in a server and (2) long restart time…
This article overviews some of the key past developments in cloud data center power and energy management, where we are today, and what the future could be. This topic is gaining enormous renewed interest in the context of the conflicting needs of the AI revolution and the climate crisis.
A B-Tree is the most widely used range index for larger-than-memory data systems. It organizes data in pages (usually 4 KB) that efficiently align with disk IO operations, fully utilizing each IO operation to narrow down the search space.
This paper presents DDS, a novel disaggregated storage architecture enabled by emerging networking hardware, namely DPUs (Data Processing Units). DPUs can optimize the latency and CPU consumption of disaggregated storage servers.
The Compute Express Link (CXL) is an open industry-standard interconnect between processors and devices such as accelerators, memory buffers, smart network interfaces, persistent memory, and solid-state drives. CXL offers coherency and memory semantics with bandwidth that scales with PCIe bandwidth while achieving significantly lower latency than PCIe.
Learned indexes use machine learning techniques to improve index construction. However, they often face a fundamental trade-off between performance and memory consumption, especially in dynamic environments with frequent insert and delete operations.
The evolving data generation landscape requires faster and more efficient microprocessors, prompting innovative manufacturing methods for smaller and faster transistors. Transistor congestion and rising demand for parallel processing are pushing the thermal design power of microprocessors well beyond 280 W, a limit for air cooling, and are expected to…
Stream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.
With the rapid development of blockchain and its applications, the amount of data stored on decentralized storage networks (DSNs) has grown exponentially. DSNs bring together affordable storage resources from around the world to provide robust, decentralized storage services for tens of thousands of decentralized applications (dApps).
CXL memory brings extra bandwidth and capacity via PCIe-based memory expansion beyond DDR-based DRAM. This paper introduces the CXL 2.
In this article, the endurance characteristic of 3-D NAND Flash memory is investigated. The bitline (BL) current and the word-line (WL) current are measured for erase/write (EW) cycling up to 50k.
Memory disaggregation is a promising architecture for modern datacenters that separates compute and memory resources into independent pools connected by ultra-fast networks, which can improve memory utilization, reduce cost, and enable elastic scaling of compute and memory resources. However, existing memory disaggregation solutions based on remote…
In order to improve the reliability of vertical NAND (V-NAND) flash memory cells, a scheme using adaptive incremental step pulse programming (A-ISPP) and incremental step pulse erasing (ISPE) is proposed. Incremental step pulse programming (ISPP) with adaptive step voltage is used to precisely adjust${V}_{\text {th}}$to a low target value while rapidly…
Foundation models, reasoning, agents, evaluation, reliability, and model architectures.
Frames robustness as selective trust and trains language models to resist misleading signals without ignoring useful context.
Proposes a unified taxonomy for agent perception, planning, action, tool use, collaboration, environments, and evaluation.
Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.
Large language models (LLMs) have significantly advanced the field of natural language processing (NLP), providing a highly useful, task-agnostic foundation for a wide range of applications. However, directly applying LLMs to solve sophisticated problems in specific domains meets many hurdles, caused by the heterogeneity of domain data, the…
Large language model (LLM)-based agents have recently attracted much attention from the research and industry communities. Compared with original LLMs, LLM-based agents are featured in their self-evolving capability, which is the basis for solving real-world problems that need long-term and complex agent-environment interactions.
This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital.
In recent years, applying multi-modal large language models (MLLMs) in various fields has achieved remarkable success. However, as the foundation model for many downstream tasks, MLLMs comprise the well-known Transformer network, which has a less efficient quadratic computation complexity.
Organizes LLM-agent architecture, collaboration, evolution, evaluation, applications, and open research challenges.
Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions. Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.
Code generation aims to automatically generate code snippets of specific programming language according to natural language descriptions. The continuous advancements in deep learning, particularly pre-trained models, have empowered the code generation task to achieve remarkable performance.
Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to scale large language or visual-language models efficiently, these efforts typically…
Studies large-scale reinforcement learning for emergent reasoning and presents a multi-stage path to capable, readable reasoning models.
Heterogeneous multi-agent systems (HMAS) comprise various intelligent agents with specialized functions, such as drones, ground robots, and automated devices, working in coordinated settings. This paper presents AutoHMA-LLM, a novel framework that combines Large Language Models (LLMs) with classical control algorithms to address the challenges of task…
We introduce a language modeling approach for text to speech synthesis (TTS). Specifically, we train aneural codec language model(calledVALL-E) using discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language modeling task rather than continuous signal regression as in previous work.
Adds a neural long-term memory module that learns at inference time and complements attention's short-term context.
Describes a 671B-parameter mixture-of-experts model using latent attention, auxiliary-loss-free load balancing, and multi-token prediction.
Recently, the multimodal large language model (MLLM) represented by GPT-4V has been a new rising research hotspot, which uses powerful large language models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of the MLLM, such as writing stories based on images and optical character recognition-free math reasoning, are…
Abstract Information Extraction (IE) aims to extract structural knowledge from plain natural language texts. Recently, generative Large Language Models (LLMs) have demonstrated remarkable capabilities in text understanding and generation.
Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite significant achievements, they rely on enormous training data to acquire a broad spectrum of open-domain knowledge.
As one of the most advanced techniques in AI, Retrieval-Augmented Generation (RAG) can offer reliable and up-to-date external knowledge, providing huge convenience for numerous tasks. Particularly in the era of AI-Generated Content (AIGC), the powerful capacity of retrieval in providing additional knowledge enables RAG to assist existing generative AI…
Generative large language model (LLM) applications are growing rapidly, leading to large-scale deployments of expensive and power-hungry GPUs. Our characterization of LLM inference shows that each inference request undergoes two phases: a compute-intensive prompt computation phase and a memory intensive token generation phase, each with distinct…
Learning on Graphs has attracted immense attention due to its wide real-world applications. The most popular pipeline for learning on graphs with textual node attributes primarily relies on Graph Neural Networks (GNNs), and utilizes shallow text embedding as initial node representations, which has limitations in general knowledge and profound semantic…
In the era of modern healthcare, swiftly generating medical question summaries is crucial for informed and timely patient care. Despite the increasing complexity and volume of medical data, existing studies have focused solely on text-based summarization, neglecting the integration of visual information.
Text-guided diffusion models such as DALLE-2, Imagen, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality.
Automatic font generation is an imitation task, which aims to create a font library that mimics the style of reference images while preserving the content from source images. Although existing font generation methods have achieved satisfactory performance, they still struggle with complex characters and large style variations.
The incredible generative ability of large-scale text-to-image (T2I) models has demonstrated strong power of learning complex structures and meaningful semantics. However, relying solely on text prompts cannot fully take advantage of the knowledge learned by the model, especially when flexible and accurate controlling (e.
Current generative knowledge graph construction approaches usually fail to capture structural knowledge by simply flattening natural language into serialized texts or a specification language. However, large generative language model trained on structured data such as code has demonstrated impressive capability in understanding natural language for…
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better…
With the emergence of Large Language Models (LLMs) and Vision Foundation Models (VFMs), multimodal AI systems benefiting from large models have the potential to equally perceive the real world, make decisions, and control tools as humans. In recent months, LLMs have shown widespread attention in autonomous driving and map systems.
Abstract Rapid advancements of large language models (LLMs) have enabled the processing, understanding, and generation of human-like text, with increasing integration into systems that touch our social sphere. Despite this success, these models can learn, perpetuate, and amplify harmful social biases.
Large language models (LLMs) have emerged as a new paradigm for Text-to-SQL task. However, the absence of a systematical benchmark inhibits the development of designing effective, efficient and economic LLM-based Text-to-SQL solutions.