Publications Neural Network Control
Unified Local-Cloud Decision-Making via Reinforcement Learning
18th European Conference on Computer Vision (ECCV 2024), 2024, Milan, Italy
Abstract
Embodied vision-based real-world systems, such as mobile robots, require a careful balance between energy consumption, compute latency, and safety constraints to optimize operation across dynamic tasks and contexts. As local computation tends to be restricted, offloading the computation, i.e., to a remote server, can save local resources while providing access to high-quality predictions from powerful and large models. However, the resulting communication and latency overhead has led to limited usability of cloud models in dynamic, safety-critical, real-time settings. To effectively address this trade-off, we introduce UniLCD, a novel hybrid inference framework for enabling flexible local-cloud collaboration. By efficiently optimizing a flexible routing module via reinforcement learning and a suitable multi-task objective, UniLCD is specifically designed to support the multiple constraints of safety-critical end-to-end mobile systems. We validate the proposed approach using a challenging, crowded navigation task requiring frequent and timely switching between local and cloud operations. UniLCD demonstrates improved overall performance and efficiency, by over 23% compared to state-of-the-art baselines based on various split computing and early exit strategies. Our code is available at https://unilcd.github.io/.
Code and hardware artifacts
Cite this paper
@inproceedings{UniCL_ECCV24,
title = {{Unified Local-Cloud Decision-Making via Reinforcement Learning}},
author = {Sengupta, Kathakoli and Shangguan, Zhongkai and Bharadwaj, Sandesh and Arora, Sanjay and Ohn-Bar, Eshed and Mancuso, Renato},
booktitle = {18th European Conference on Computer Vision (ECCV 2024)},
year = 2024,
pages = {185--203},
publisher = {Springer Nature Switzerland},
address = {Milan, Italy},
url = {https://arxiv.org/pdf/2409.11403}
}
TY - CONF AU - Sengupta, Kathakoli AU - Shangguan, Zhongkai AU - Bharadwaj, Sandesh AU - Arora, Sanjay AU - Ohn-Bar, Eshed AU - Mancuso, Renato TI - Unified Local-Cloud Decision-Making via Reinforcement Learning T2 - 18th European Conference on Computer Vision (ECCV 2024) PY - 2024 PB - Springer Nature Switzerland CY - Milan, Italy SP - 185 EP - 203 AB - Embodied vision-based real-world systems, such as mobile robots, require a careful balance between energy consumption, compute latency, and safety constraints to optimize operation across dynamic tasks and contexts. As local computation tends to be restricted, offloading the computation, i.e., to a remote server, can save local resources while providing access to high-quality predictions from powerful and large models. However, the resulting communication and latency overhead has led to limited usability of cloud models in dynamic, safety-critical, real-time settings. To effectively address this trade-off, we introduce UniLCD, a novel hybrid inference framework for enabling flexible local-cloud collaboration. By efficiently optimizing a flexible routing module via reinforcement learning and a suitable multi-task objective, UniLCD is specifically designed to support the multiple constraints of safety-critical end-to-end mobile systems. We validate the proposed approach using a challenging, crowded navigation task requiring frequent and timely switching between local and cloud operations. UniLCD demonstrates improved overall performance and efficiency, by over 23% compared to state-of-the-art baselines based on various split computing and early exit strategies. Our code is available at https://unilcd.github.io/. ER -
%0 Conference Paper %A Sengupta, Kathakoli %A Shangguan, Zhongkai %A Bharadwaj, Sandesh %A Arora, Sanjay %A Ohn-Bar, Eshed %A Mancuso, Renato %T Unified Local-Cloud Decision-Making via Reinforcement Learning %B 18th European Conference on Computer Vision (ECCV 2024) %D 2024 %P 185-203 %I Springer Nature Switzerland %C Milan, Italy %X Embodied vision-based real-world systems, such as mobile robots, require a careful balance between energy consumption, compute latency, and safety constraints to optimize operation across dynamic tasks and contexts. As local computation tends to be restricted, offloading the computation, i.e., to a remote server, can save local resources while providing access to high-quality predictions from powerful and large models. However, the resulting communication and latency overhead has led to limited usability of cloud models in dynamic, safety-critical, real-time settings. To effectively address this trade-off, we introduce UniLCD, a novel hybrid inference framework for enabling flexible local-cloud collaboration. By efficiently optimizing a flexible routing module via reinforcement learning and a suitable multi-task objective, UniLCD is specifically designed to support the multiple constraints of safety-critical end-to-end mobile systems. We validate the proposed approach using a challenging, crowded navigation task requiring frequent and timely switching between local and cloud operations. UniLCD demonstrates improved overall performance and efficiency, by over 23% compared to state-of-the-art baselines based on various split computing and early exit strategies. Our code is available at https://unilcd.github.io/.
[
{
"id": "UniCL_ECCV24",
"type": "paper-conference",
"title": "Unified Local-Cloud Decision-Making via Reinforcement Learning",
"author": [
{
"family": "Sengupta",
"given": "Kathakoli"
},
{
"family": "Shangguan",
"given": "Zhongkai"
},
{
"family": "Bharadwaj",
"given": "Sandesh"
},
{
"family": "Arora",
"given": "Sanjay"
},
{
"family": "Ohn-Bar",
"given": "Eshed"
},
{
"family": "Mancuso",
"given": "Renato"
}
],
"container-title": "18th European Conference on Computer Vision (ECCV 2024)",
"issued": {
"date-parts": [
[
2024
]
]
},
"page": "185-203",
"publisher": "Springer Nature Switzerland",
"publisher-place": "Milan, Italy",
"abstract": "Embodied vision-based real-world systems, such as mobile robots, require a careful balance between energy consumption, compute latency, and safety constraints to optimize operation across dynamic tasks and contexts. As local computation tends to be restricted, offloading the computation, i.e., to a remote server, can save local resources while providing access to high-quality predictions from powerful and large models. However, the resulting communication and latency overhead has led to limited usability of cloud models in dynamic, safety-critical, real-time settings. To effectively address this trade-off, we introduce UniLCD, a novel hybrid inference framework for enabling flexible local-cloud collaboration. By efficiently optimizing a flexible routing module via reinforcement learning and a suitable multi-task objective, UniLCD is specifically designed to support the multiple constraints of safety-critical end-to-end mobile systems. We validate the proposed approach using a challenging, crowded navigation task requiring frequent and timely switching between local and cloud operations. UniLCD demonstrates improved overall performance and efficiency, by over 23% compared to state-of-the-art baselines based on various split computing and early exit strategies. Our code is available at https://unilcd.github.io/."
}
]
K. Sengupta, Z. Shangguan, S. Bharadwaj, S. Arora, E. Ohn-Bar, and R. Mancuso, “Unified Local-Cloud Decision-Making via Reinforcement Learning,” in 18th European Conference on Computer Vision (ECCV 2024), pp. 185–203, 2024.
Sengupta, K., Shangguan, Z., Bharadwaj, S., Arora, S., Ohn-Bar, E., & Mancuso, R. (2024). Unified Local-Cloud Decision-Making via Reinforcement Learning. In 18th European Conference on Computer Vision (ECCV 2024) (pp. 185–203). Springer Nature Switzerland.