Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices

Runhua Zhang; Hongxu Jiang; Fangzheng Tian; Jinkun Geng; Xiaobin Li; Yuhang Ma; Chenhui Zhu; Dong Dong; Xin Li; Haojie Wang

doi:10.1007/978-3-031-30637-2_35

Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices

Runhua Zhang
, Hongxu Jiang^*
, Fangzheng Tian
, Jinkun Geng
, Xiaobin Li
, Yuhang Ma
, Chenhui Zhu
, Dong Dong
, Xin Li
, Haojie Wang

^*Corresponding author for this work

School of Computer Science and Engineering

Beihang University
Stanford University
Tsinghua University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

In this paper, we propose Xenos, a high-performance edge platform for model inference. Unlike the prior works which mainly focus one operator-centric optimization, Xenos can automatically conduct dataflow-centric optimization on the computation graph and accelerate inference in two dimensions. Vertically, Xenos develops operator linking technique to improve data locality by restructuring the inter-operator dataflow. Horizontally, Xenos develops DSP-aware operator split technique to enable higher parallelism across multiple DSP units. Our evaluation proves the effectiveness of Xenos ’ vertical and horizontal dataflow optimization, which reduce the inference time by 15.0%–84.9% and 17.9%–89.9%, respectively.

Original language	English
Title of host publication	Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings
Editors	Xin Wang, Maria Luisa Sapino, Wook-Shin Han, Amr El Abbadi, Gill Dobbie, Zhiyong Feng, Yingxiao Shao, Hongzhi Yin
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	535-545
Number of pages	11
ISBN (Print)	9783031306365
DOIs	https://doi.org/10.1007/978-3-031-30637-2_35
State	Published - 2023
Event	28th International Conference on Database Systems for Advanced Applications, DASFAA 2023 - Tianjin, China Duration: 17 Apr 2023 → 20 Apr 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13943 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	28th International Conference on Database Systems for Advanced Applications, DASFAA 2023
Country/Territory	China
City	Tianjin
Period	17/04/23 → 20/04/23

Keywords

DSP-aware
data locality
dataflow-centric
edge-based inference

Access to Document

10.1007/978-3-031-30637-2_35

Cite this

Zhang, R., Jiang, H., Tian, F., Geng, J., Li, X., Ma, Y., Zhu, C., Dong, D., Li, X., & Wang, H. (2023). Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices. In X. Wang, M. L. Sapino, W.-S. Han, A. El Abbadi, G. Dobbie, Z. Feng, Y. Shao, & H. Yin (Eds.), Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings (pp. 535-545). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13943 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-30637-2_35

Zhang, Runhua ; Jiang, Hongxu ; Tian, Fangzheng et al. / Xenos : Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices. Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. editor / Xin Wang ; Maria Luisa Sapino ; Wook-Shin Han ; Amr El Abbadi ; Gill Dobbie ; Zhiyong Feng ; Yingxiao Shao ; Hongzhi Yin. Springer Science and Business Media Deutschland GmbH, 2023. pp. 535-545 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c979043e7c404aca85eae9f72f72ab92,

title = "Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices",

abstract = "In this paper, we propose Xenos, a high-performance edge platform for model inference. Unlike the prior works which mainly focus one operator-centric optimization, Xenos can automatically conduct dataflow-centric optimization on the computation graph and accelerate inference in two dimensions. Vertically, Xenos develops operator linking technique to improve data locality by restructuring the inter-operator dataflow. Horizontally, Xenos develops DSP-aware operator split technique to enable higher parallelism across multiple DSP units. Our evaluation proves the effectiveness of Xenos {\textquoteright} vertical and horizontal dataflow optimization, which reduce the inference time by 15.0\%{\textendash}84.9\% and 17.9\%{\textendash}89.9\%, respectively.",

keywords = "DSP-aware, data locality, dataflow-centric, edge-based inference",

author = "Runhua Zhang and Hongxu Jiang and Fangzheng Tian and Jinkun Geng and Xiaobin Li and Yuhang Ma and Chenhui Zhu and Dong Dong and Xin Li and Haojie Wang",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 28th International Conference on Database Systems for Advanced Applications, DASFAA 2023 ; Conference date: 17-04-2023 Through 20-04-2023",

year = "2023",

doi = "10.1007/978-3-031-30637-2\_35",

language = "英语",

isbn = "9783031306365",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "535--545",

editor = "Xin Wang and Sapino, \{Maria Luisa\} and Wook-Shin Han and \{El Abbadi\}, Amr and Gill Dobbie and Zhiyong Feng and Yingxiao Shao and Hongzhi Yin",

booktitle = "Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings",

address = "德国",

}

Zhang, R, Jiang, H, Tian, F, Geng, J, Li, X, Ma, Y, Zhu, C, Dong, D, Li, X & Wang, H 2023, Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices. in X Wang, ML Sapino, W-S Han, A El Abbadi, G Dobbie, Z Feng, Y Shao & H Yin (eds), Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13943 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 535-545, 28th International Conference on Database Systems for Advanced Applications, DASFAA 2023, Tianjin, China, 17/04/23. https://doi.org/10.1007/978-3-031-30637-2_35

Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices. / Zhang, Runhua; Jiang, Hongxu; Tian, Fangzheng et al.
Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. ed. / Xin Wang; Maria Luisa Sapino; Wook-Shin Han; Amr El Abbadi; Gill Dobbie; Zhiyong Feng; Yingxiao Shao; Hongzhi Yin. Springer Science and Business Media Deutschland GmbH, 2023. p. 535-545 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13943 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Xenos

T2 - 28th International Conference on Database Systems for Advanced Applications, DASFAA 2023

AU - Zhang, Runhua

AU - Jiang, Hongxu

AU - Tian, Fangzheng

AU - Geng, Jinkun

AU - Li, Xiaobin

AU - Ma, Yuhang

AU - Zhu, Chenhui

AU - Dong, Dong

AU - Li, Xin

AU - Wang, Haojie

PY - 2023

Y1 - 2023

N2 - In this paper, we propose Xenos, a high-performance edge platform for model inference. Unlike the prior works which mainly focus one operator-centric optimization, Xenos can automatically conduct dataflow-centric optimization on the computation graph and accelerate inference in two dimensions. Vertically, Xenos develops operator linking technique to improve data locality by restructuring the inter-operator dataflow. Horizontally, Xenos develops DSP-aware operator split technique to enable higher parallelism across multiple DSP units. Our evaluation proves the effectiveness of Xenos ’ vertical and horizontal dataflow optimization, which reduce the inference time by 15.0%–84.9% and 17.9%–89.9%, respectively.

AB - In this paper, we propose Xenos, a high-performance edge platform for model inference. Unlike the prior works which mainly focus one operator-centric optimization, Xenos can automatically conduct dataflow-centric optimization on the computation graph and accelerate inference in two dimensions. Vertically, Xenos develops operator linking technique to improve data locality by restructuring the inter-operator dataflow. Horizontally, Xenos develops DSP-aware operator split technique to enable higher parallelism across multiple DSP units. Our evaluation proves the effectiveness of Xenos ’ vertical and horizontal dataflow optimization, which reduce the inference time by 15.0%–84.9% and 17.9%–89.9%, respectively.

KW - DSP-aware

KW - data locality

KW - dataflow-centric

KW - edge-based inference

UR - https://www.scopus.com/pages/publications/85161673611

U2 - 10.1007/978-3-031-30637-2_35

DO - 10.1007/978-3-031-30637-2_35

M3 - 会议稿件

AN - SCOPUS:85161673611

SN - 9783031306365

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 535

EP - 545

BT - Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings

A2 - Wang, Xin

A2 - Sapino, Maria Luisa

A2 - Han, Wook-Shin

A2 - El Abbadi, Amr

A2 - Dobbie, Gill

A2 - Feng, Zhiyong

A2 - Shao, Yingxiao

A2 - Yin, Hongzhi

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 17 April 2023 through 20 April 2023

ER -

Zhang R, Jiang H, Tian F, Geng J, Li X, Ma Y et al. Xenos: Dataflow-Centric Optimization to Accelerate Model Inference on Edge Devices. In Wang X, Sapino ML, Han WS, El Abbadi A, Dobbie G, Feng Z, Shao Y, Yin H, editors, Database Systems for Advanced Applications - 28th International Conference, DASFAA 2023, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 535-545. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-30637-2_35