工業物聯網（Industrial Internet of Things, IIoT）是將物聯網（Internet of Things, IoT）技術運用於工業環境，並透過網路串聯各項設備，實現即時資料的收集、分析與交換，以此有效提升工廠自動化生產與營運流程的效率。IIoT 中的應用相對多元，服務請求亦不盡相同，如何設計並優化資源分配的策略尤其重要，若未妥善處理，可能導致資源使用效率不彰而影響系統整體的效能。為了解決上述問題，本研究導入服務功能鏈（Service Function Chain, SFC），使資料流依序流經一系列預先定義執行順序的虛擬網路功能（Virtual Network Functions, VNFs），提供具彈性的部署方法，支援不同服務的資源需求。我們提出了一種基於深度強化學習之動態服務功能鏈部署的資源分配架構（DRL-based Resource Allocation for Dynamic SFC Embedding, DRL-RADSE），並整合部署及遷移兩種不同的決策模型，以降低系統在處理服務請求過程中所產生的部署延遲。經模擬結果顯示，本文所提出的方法能有效處理不同長度的服務請求，部署及遷移延遲均能收斂，且收斂穩定後的延遲表現優於既有文獻。

The Industrial Internet of Things (IIoT) combines Internet of Things (IoT) technologies into industrial environments. By interconnecting devices through networks, IIoT facilitates real-time data collection, analysis, and exchange, thereby significantly enhancing the efficiency of automated production and operational workflows. The diversity of IIoT applications and the differences of service requests make the design and optimization of resource allocation strategies particularly critical. The unsuitability of allocation strategies may lead to reduced resource utilization and degraded system performance. This thesis introduces Service Function Chaining (SFC), in which data flow sequentially traverses a series of Virtual Network Functions (VNFs) defined in a predetermined execution order. This approach improves the flexibility of function deployment and supports resource requirements across various service types. We propose a DRL-based Resource Allocation framework for Dynamic SFC Embedding, referred to as DRL-RADSE, which integrates two distinct decision models for placement and migration, with the objective of minimizing the average embedding time of total service requests. Simulation results show that the proposed method can effectively handle service requests of varying lengths, achieve convergence in placement and migration embedding time, and outperform existing work with improved time performance.

[1] J. Halpern and C. Pignataro, “Service function chaining (sfc) architecture,” Internet Engineering Task Force (IETF), Tech. Rep., 2015.
[2] H. Hantouti, N. Benamar, and T. Taleb, “Service function chaining in 5g & beyond networks: Challenges and open research issues,” IEEE Network, vol. 34, no. 4, pp. 320–327, 2020.
[3] H. Hantouti, N. Benamar, T. Taleb, and A. Laghrissi, “Traffic steering for service function chaining,” IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 487–507, 2019.
[4] S. Sezer, S. Scott-Hayward, P. K. Chouhan, B. Fraser, D. Lake, J. Finnegan, N. Viljoen, M. Miller, and N. Rao, “Are we ready for sdn? implementation challenges for software-defined networks,” IEEE Communications Magazine, vol. 51, no. 7, pp. 36–43, 2013.
[5] B. A. A. Nunes, M. Mendonca, X.-N. Nguyen, K. Obraczka, and T. Turletti, “A survey of software-defined networking: Past, present, and future of programmable networks,” IEEE Communications Surveys & Tutorials, vol. 16, no. 3, pp. 1617–1634, 2014.
[6] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. The MIT Press, 2015.
[7] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, 2015.
[8] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” 2019. [Online]. Available: https://arxiv.org/abs/1509.02971
[9] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller., “Deterministic policy gradient algorithms,” International Conference on Machine Learning, vol. 9, 2014.
[10] T.-W. Kuo, B.-H. Liou, K. C.-J. Lin, and M.-J. Tsai, “Deploying chains of virtual network functions: On the relation between link and server usage,” IEEE/ACM Transactions on Networking, vol. 26, no. 4, pp. 1562–1576, 2018.
[11] Y. Wu, Z. Jia, Q. Wu, and Z. Lu, “Adaptive qoe-aware sfc orchestration in uav networks: A deep reinforcement learning approach,” IEEE Transactions on Network Science and Engineering, vol. 11, no. 6, pp. 6052–6065, 2024.
[12] Y.-H. Hsu, T.-R. Tsai, T.-C. Yeh, and Y.-L. Wang, “Deep reinforcement learning based mobility-aware sfc embedding for mec in 5g and beyond,” in 2023 IEEE Wireless Communications and Networking Conference (WCNC), 2023, pp. 1–6.
[13] X. Yu, R. Wang, J. Hao, Q. Wu, C. Yi, P. Wang, and D. Niyato, “Priority-aware deployment of autoscaling service function chains based on deep reinforcement learning,” IEEE Transactions on Cognitive Communications and Networking, vol. 10, no. 3, pp. 1050–1062, 2024.
[14] J. Li, R. Wang, and K. Wang, “Service function chaining in industrial internet of things with edge intelligence: A natural actor-critic approach,” IEEE Transactions on Industrial Informatics, vol. 19, no. 1, pp. 491–502, 2023.
[15] R. Behravesh, D. Harutyunyan, E. Coronado, and R. Riggio, “Time-sensitive mobile user association and sfc placement in mec-enabled 5g networks,” IEEE Transactions on Network and Service Management, vol. 18, no. 3, pp. 3006–3020, 2021.
[16] T. Jing, Z. Liu, M. Zhu, X. Li, B. Gao, Q. Gao, and Y. Huo, “P-drr: Ppo-based efficient dynamic resource reallocation scheme in industrial internet of things,” in 2023 IEEE 98th Vehicular Technology Conference (VTC2023-Fall), 2023, pp. 1–5.
[17] L. Stahlbock, J. Weber, and F. Köster, “An optimization approach of container startup times for time-sensitive embedded systems,” in 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), 2022, pp. 2019–2026.
[18] S. He, X. Lyu, W. Ni, H. Tian, R. P. Liu, and E. Hossain, “Virtual service placement for edge computing under finite memory and bandwidth,” IEEE Transactions on Communications, vol. 68, no. 12, pp. 7702–7718, 2020.
[19] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” CoRR, 09 2015.
[20] L. Zhang, S. Zhuge, Y. Wang, H. Xu, and E. Sun, “Energy-delay tradeoff for virtual machine placement in virtualized multi-access edge computing: A two-sided matching approach,” TechRxiv, 12 2020.