Skip to content
Home

Current Research · Self-Improving Data Systems

Self-Improving Data Systems

AI, database learning, and workload feedback for improving queries and data pipelines while preserving semantics and operational constraints.

GenRewrite

Uses large language models and counterexample-guided validation to rewrite SQL queries while preserving semantics.

DAGSmith

Uses pipeline dependencies as optimization signals and separates rewrite generation from equivalence checking.

Research question

How can AI improve queries and data infrastructure while preserving correctness and operational safeguards?

Approach

GenRewrite uses large language models and natural-language rewrite rules to generate efficient, semantically equivalent SQL rewrites. DAGSmith uses pipeline dependencies as optimization signals and separates rewrite generation from equivalence checking.

Key results

  • On the open-source Tuva dbt project, DAGSmith reduced elapsed time by 42.6% and warehouse compute cost by 67.7%.

Publications

Paper ↗Project ↗
Cite
@article{def837ec-1591-44a4-a36d-929442c789f3,
  title = {SlabCity: Whole-Query Optimization using Program Synthesis},
  author = {Rui Dong and Jie Liu and Yuxuan Zhu and Cong Yan and Barzan Mozafari and Xinyu Wang},
  journal = {Proceedings of the VLDB Endowment},
  year = {2023},
  doi = {10.14778/3611479.3611515}
}