0 Introduction
Considering the integration of a high proportion of renewable energy resources, power distribution systems require precise energy scheduling to ensure safe and stable operation [1].Accurate time synchronization is essential for precise energy scheduling.Beidou 3 (BD3), with its highstability hydrogen atomic clock and new communication link, can provide accurate and reliable positioning and timing.Comparison between Global Position System (GPS)and a BD3 system, based on independent research, proves that the time service of a BD3 system ensures the security of distribution network operations and achieves a 10 ns time synchronization for energy scheduling [2-4].However,many widely deployed devices have not been equipped with BD3 receiving modules due to the high operation and maintenance costs and complex electromagnetic environments [5-6].To achieve time synchronization of devices without BD3 receiving modules on the 220/380 V low-voltage user side, a ground power reference station first receives synchronization signals from BD3, and then sends synchronization data packets to devices via a wireless network in a multi-hop routing fashion [7].Therefore,studying how to use the BD3-based power reference station to achieve high-precision time synchronization for large numbers of various devices is urgently required.
Compared with equipping all devices with BD3 receivers, a BD3 system-based power reference station has several advantages.First, not installing BD3 receivers in a device significantly reduces its cost, physical size,and power consumption [8].Second, the synchronization packet is delivered from the power reference station to the destination device through multi-hop routing.Synchronization reliability is guaranteed if the single route does not function properly.However, despite the advantages, research on route selection optimization for high-precision time synchronization of power reference stations based on BD3 still faces several challenges.
First, time synchronization error and time synchronization delay are coupled with each other [9-11].Selecting a route with a small time synchronization error may lead to a large time synchronization delay due to poor channel conditions between the power reference station and destination device, and vice versa [12].Therefore,the minimization of time synchronization error and delay should be jointly considered.Second, some researchers have attempted to optimize route selection with incomplete information based on reinforcement learning [13].However,traditional reinforcement learning algorithms which use fixed learning parameters suffer from slow convergence and poor route selection optimization performance due to the mismatch between learning parameter adjustment and dynamic environment changes.
Several studies on time synchronization for power reference stations have been carried out.Pei et al.[14]proposed a common-view time transfer method based on satellites, where receivers at the power reference station receive the time signals and the offsets are determined with high precision, but the multi-hop time synchronization packet transmission between the power reference station and large-scale deployed devices is ignored.Li et al.[15]proposed a pair-wise broadcast time synchronization algorithm and a multi-hop route protocol to reduce synchronization energy costs and improve synchronization precision.However, it relies on perfect knowledge of complete information to derive the optimal route selection decision and cannot be extended to the multi-hop time synchronization scenario with incomplete information.
The upper confidence bound (UCB) algorithm in reinforcement learning provides a feasible solution for route selection optimization without complete information[16].Zhou et al.[17] proposed a UCB-based jammingaware route-selection mechanism to solve the utility optimal scheduling problem for multi-hop wireless networks.Zhang et al. [18] used a mobile sink to solve the problem of energy efficiency based on UCB.However, these studies focus only on the optimization of routing delay and energy efficiency while ignoring the time synchronization error.As a result,the unilateral optimization of synchronization delay leads to the selection of a route link with poor synchronization precision and a large cumulative synchronization delay.Furthermore, these studies apply classic learning algorithms with fixed exploration factors that are not dynamically adjusted in accordance with environmental or topological changes.They suffer from slow convergence and poor learning accuracy.
To address these challenges, we propose a back propagation (BP) neural network-improved intelligent time synchronization route selection algorithm named BPRS.First, we establish a software-defined network (SDN)-enabled power reference station time synchronization framework based on BD3 and construct models for synchronization delay and error.SDN can distinguish data forwarding from control logic to reduce communication and computing costs at power reference stations.Therefore,this study uses a centralized programmable SDN controller to collect topology as well as other network information,optimize routing decisions, and issue policies to the power reference station.Second, we formulate the joint problem to minimize cumulative synchronization error and delay through multi-hop route selection optimization.The proposed BP-RS algorithm enables the SDN controller to learn the optimal synchronization route based on historical information and continuously update the exploration factor to improve learning performance based on a BP neural network.The contributions of this paper are summarized as follows:
Intelligent multi-hop synchronization route selection:BP-RS intelligently learns the multi-hop synchronization route selection by continuously interacting with the environment and utilizing the BP neural network to dynamically adjust the exploration factor to achieve a better balance between exploration and exploitation.
Minimization of time synchronization error and time synchronization delay: The optimization objective is constructed as the weighted sum of the cumulative time synchronization delay and error.We comprehensively consider the influence of synchronization delay and error to achieve high-precision and low-latency synchronization for large-scale deployed devices.
The rest of the paper is organized as follows: Section II introduces the system model and problem formulation.Section III introduces the proposed BP-RS algorithm in detail.Section IV provides simulation results.Section V concludes this paper.
1 System model and problem formulation
In this section, the system model of SDN-based time synchronization for power reference stations and the joint optimization problem formulation are introduced.
1.1 System model
The considered scenario of SDN-enabled time synchronization for a power reference station based on a BD3 system, shown in Fig.1, consists of an SDN controller,a power reference station, several relay devices, and a destination device.The power reference station receives the time synchronization signal from the BD3 satellite and transmits the time synchronization packet to the destination device through multi-hop relay devices to provide synchronization [19].The SDN controller collects network topology information and makes the route selection decision to minimize cumulative synchronization delay and error.
Fig.1 Scenario of SDN-enabled time synchronization for power reference station based on BD3 system
1.2 Synchronization delay model
where ξ ∈(0,2] is the characteristic factor that represents the tail thickness of the distribution function.λ ∈[- 1,1] is the skew parameter, σ ＞0 is the scale parameter, and μ ∈R represents the median of the distribution function when ξ ∈(0,1] and the mean value when ξ ∈（1,2][21].
The total transmission delay of the synchronization packet from the power reference station to the destination device on link fm is
The total synchronization delay from the power reference station to the destination device in slot t is
1.3 Synchronization error model
1.4 Problem formulation
This paper aims to solve the accurate time synchronization problem of power reference stations based on the BD3 system.The optimization goal is to minimize the weighted sum of cumulative synchronization delay and error by optimizing route selection decisions.The optimization problem is formulated as
where, V is the weight of the synchronization error.C1 and C2 indicate that only one route link can be selected in each slot.
2 BP neural network-improved intelligent time synchronization route selection algorithm
In this section, first, the optimization problem is modeled as a multi-armed bandit (MAB) problem.Then,the proposed BP-RS algorithm is introduced.
2.1 MAB problem modeling
The optimization problem is modeled as an MAB problem [23, 24], as shown in Fig.2.
Decision maker: The SDN controller is defined as the decision maker.
Arms: The set of route links of the time synchronization packet are defined as the set of arms, which is expressed as F={ f1 ,…, fM }.
Reward: The reward θm( t) of selecting the route link fm is defined as the reciprocal of the weighted sum of synchronization delay and error, which is given by
Fig.2 Schematic diagram of the proposed BP-RS
2.2 Intelligent time synchronization route selection
The BP-RS enables the SDN controller to intelligently make route selection decisions based on historical empirical information [25].The flowchart, shown in Fig.3, consists of the following three steps:
whereθm( t- 1) represents the average reward of fm up to slot t-1.The second term allows the SDN controller to explore other routes, where β(t) is the exploration factor, whose initial value β(0) is set to 1.Finally, the SDN controller selects the route link with the maximum preference value for synchronization packet transmission in slot t [26].
Improved learning: At the end of each slot, the SDN controller calculates the reward based on the observed synchronization delay and error.Then, the average reward and the number of route selection times are updated as
β( t) is dynamically adjusted based on the BP neural network to improve learning performance.The details are introduced in the next subsection.
2.3 BP neural network-improved learning
The exploration factor of traditional UCB is fixed,resulting in poor learning performance and slow convergence [27, 28].The proposed BP-RS algorithm utilizes a neural network to dynamically adjust β(t ) to obtain a better exploration and exploitation balance and improve learning performance.
Fig.3 Flowchart of BP-RS
As shown in Fig.4, the BP neural network for improved learning includes an input layer [29], a hidden layer, and an output layer, which are introduced as follows:
Fig.4 Schematic of BP neural network-improved learning
The output layer outputs the exploration factor of slot t+1.The number of neurons in the output layer is P=1,and the neuron vector is Z =(β(t+ 1) )T.
Based on the numbers of neurons in the input and output layers, the number of neurons in the hidden layer is
where κ is the activation function parameter.Based on (17),the output of the q-th neuron in the hidden layer is
Then, based on the activation function of the hidden layer, the output of the output layer, i.e., the exploration factor of slot t+1 is
where wqout is the weight between the hidden layer neuron Nq and the output layer neuron.
3 Simulation results
In this section, we validate the performance of the proposed algorithm through simulations.We considered a power reference station with one SDN controller,10 power base stations, and a large amount of power terminal equipment.The total time synchronization process was divided into 500 slots.In accordance with the actual environment of the power reference station time synchronization network, the channel gain was set as g i ,j ( t )=127 +30log( ri ,j ), where ri ,j (in km) is the distance between vi and vj.The remaining simulation parameters are shown in Table 1 [30-32].In this paper, two comparison algorithms were considered: the traditional UCB based route selection algorithm, which has a fixed exploration coefficient [33], and the shortest route selection algorithm,named SRS, which selects the route with the shortest transmission distance [34].
Table 1 Simulation Parameters
The performance of the proposed algorithm was verified through a scenario with a fixed routing topology.Then, we consider a scenario where the routing topology changes in the 300th slot.The scenario was used to verify the effectiveness and robustness of the proposed algorithm by dynamically changing the routing topology.
Fig.5 shows the weighted sum of synchronization delay and error under a fixed routing topology.Compared with UCB and SRS, the weighted sum calculated using the proposed algorithm was reduced by 14.22% and 43.88%,respectively; synchronization delay was reduced by 13.47%and 48.26%, respectively, and synchronization error was reduced by 14.72% and 40.50%, respectively.The proposed algorithm can dynamically adjust the exploration factor by using a neural network, which can better realize the dynamic balance between exploration and exploitation and quickly find the optimal link selection strategy.SRS selects the route only according to the transmission distance and does not consider the dynamic change in synchronization error and channel state.UCB considers channel state and synchronization error comprehensively, showing better optimization performance than SRS.However, UCB cannot realize the dynamic adjustment of the exploration factor,which results in worse optimization performance than the proposed algorithm.
Fig.6 shows the weighted sum of synchronization delay and error in a time-varying routing topology where a node fails in the 300th slot.Compared with UCB and SRS,using the proposed algorithm reduced the weighted sum by 21.88% and 44.67%, respectively; synchronization delay by 25.72% and 49.26%, respectively, and synchronization error by 19.10% and 41.13%, respectively.When a node fails and the network topology changes, the proposed algorithm can increase the exploration coefficient in time, re-explore the optimal link selection strategy, quickly find the optimal link, and use the optimal link to improve synchronization performance.UCB cannot respond in time to a link failure and find the optimal link after the topology changes.SRS continues using the previous route, which is no longer optional.Its synchronization delay and error performances are the worst.
Fig.5 Performance comparison with fixed routing topology
Fig.6 Performance comparison in a time-varying routing topology
Fig.7 Weighted sum of synchronization delay and error comparison with N=20 and M=8
Fig.7 shows the weighted sum of synchronization delay and error comparison for N=20 and M=8.Simulation results show that after changing the number of network nodes and links, the proposed algorithm still performs optimally.With fixed routing topologies, the proposed algorithm’s performance was 16.45% better than UCB and 46.25% better than SRS.In the case of time-varying routing topologies, the proposed algorithm’s performance was 22.38% better than UCB and 44.71% better than SRS.Simulation results show that the proposed algorithm can adapt to different network environments and is robust.
4 Conclusions
This paper addresses the accurate time synchronization problem of power reference stations based on the BD3 system.A BP-RS algorithm was proposed to minimize the weighted sum of cumulative synchronization delay and error by optimizing route selection decisions between the power reference station and destination devices.Compared with UCB and SRS, the synchronization delay calculated using the BP-RS algorithm was reduced by 13.47% and 48.26%,respectively, and the synchronization error was reduced by 14.72% and 40.50%, respectively.Moreover, simulation results verified adaptability with routing topology changes.In the future, we will consider power control to further reduce the impact of electromagnetic interference and improve synchronization delay performance.
Acknowledgements
This work was supported by the Science and Technology Project of the China Southern Power Grid Company Limited under grant number GDKJXM20202032.
Declaration of Competing Interest
We declare that we have no conflicts of interest.
References
[1] Huo Z, Zhang Z (2019) Scheduling and control co-design for networked wind energy conversion systems.Global Energy Interconnection, 2(4): 328-335
[2] Qiu W, Zeng Q, Gao C, et al.(2020) Fine doppler shift acquisition algorithm for BeiDou software receiver by a lookup table.Journal of Systems Engineering and Electronics, 31(3):612-625
[3] Li S, Li J, Gu X, et al.(2015) Reconfigurable all-band RF CMOS transceiver for GPS/GLONASS/Galileo/Beidou with digitally assisted calibration.IEEE Transactions on Very Large Scale Integration Systems, 23(9): 1814-1827
[4] Wei H, Pascale D (2016) BeiDou time transfer with the standard CGGTTS.IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 63(7): 1005-1012
[5] Zhang D, Cheng E, Wan H, et al.(2019) Prediction of electromagnetic compatibility for dynamic datalink of UAV.IEEE Transactions on Electromagnetic Compatibility, 61(5):1474-1482
[6] Hu C, Tian Y, Yang X, et al.(2016) Background ionosphere effects on geosynchronous SAR focusing: theoretical analysis and verification based on the BeiDou navigation satellite system(BDS).IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 9(3): 1143-1162
[7] Ramezan G, Leung C, Wang Z.(2018) A survey of secure routing protocols in multi-hop cellular networks.IEEE Communications Surveys & Tutorials, 20(4): 3510-3541
[8] Wei D, Li X, Shen G, et al.(2021) A high-altitude platform airground wireless communication system based on Beidou.Journal of Communications and Information Networks, 6(3): 312-320
[9] Han L and Hua N (2013) A distributed time synchronization solution without satellite time reference for mobile communication.IEEE Communications Letters, 17(7): 1447-1450
[10] Luo R, Hua N, Zheng X, et al.(2019) High-reliability subnanosecond network time synchronization method enabled by double-frequency distributed time synchronization.Journal of Optical Communications and Networking, 11(1): A40-A51
[11] Wang Y, Huang L, Cao Y, et al.(2018) The equivalent model of controller in synchronous frame to stationary frame.Global Energy Interconnection, 1(2): 122-129
[12] Zachariah D, Dwivedi S, Händel P, et al.(2017) Scalable and passive wireless network clock synchronization in LoS environments.IEEE Transactions on Wireless Communications,16(6): 3536-3546
[13] Li F, Song X, Chen H, et al.(2019) Hierarchical routing for vehicular ad hoc networks via reinforcement learning.IEEE Transactions on Vehicular Technology, 68(2): 1852-1865
[14] Wei P, Yang C, Yang X, et al.(2020) Common-view time transfer using geostationary satellite.IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 67(9): 1938-1945
[15] Li N, Yang L (2010) A research on synchronization algorithm and highly reliable routine protocol of industrial wireless sensor network, In: Proceeding of the 2nd International Conference on Information Science and Engineering, Hangzhou, 3-5, Dec 2010,5p
[16] Liao H, Mu Y, Zhou Z, et al.(2021) Blockchain and learningbased secure and intelligent task offloading for vehicular fog computing.IEEE Transactions on Intelligent Transportation Systems, 22(7): 4051-4063
[17] Zhou P, Yan Q, Wang K, et al.(2020) Jamsa: A utility optimal contextual online learning framework for anti-jamming wireless scheduling under reactive jamming attack.IEEE Transactions on Network Science and Engineering, 7(3): 1862-1878
[18] Zhang J, Tang J, Wang F (2020) Cooperative relay selection for load balancing with mobility in hierarchical WSNs: A multiarmed bandit approach.IEEE Access, 8: 18110-18122
[19] Yun L, Yao W, Li H, et al.(2019) Design and verification of a satellite-terrestrial integrated IP network model for the Global Energy Interconnection.Global Energy Interconnection, 2(6):497-504
[20] Liao H, Zhou Z, Kong W, et al.(2021) Learning-based intentaware task offloading for air-ground integrated vehicular edge computing.IEEE Transactions on Intelligent Transportation Systems, 22(8): 5127-5139
[21] Zhou Z, Guo Y, He Y, et al.(2019) Access Control and Resource Allocation for M2M Communications in Industrial Automation.in IEEE Transactions on Industrial Informatics, 15(5): 3093-3103
[22] Zhou Z, Chen Y, Pan C, et al.(2020) High reliability and low delay mobile edge computing technology for intelligent power inspection.High Voltage Technology, 46 (06): 1895-1902
[23] Chen S, Wang X, Zhou P, et al.(2022) Heterogeneous Semi-Asynchronous Federated Learning in Internet of Things: A Multi-Armed Bandit Approach.IEEE Transactions on Emerging Topics in Computational Intelligence, 6(5): 1113-1124
[24] Mohamed E, Hashima S, Hatano K, et al.(2021) Sleeping contextual/non-contextual Thompson sampling MAB for mmWave D2D two-hop relay probing.IEEE Transactions on Vehicular Technology, 70(11): 12101-12112
[25] Xu C, Zhang T, Kuang X, et al.(2021) Context-Aware Adaptive Route Mutation Scheme: A Reinforcement Learning Approach.IEEE Internet of Things Journal, 8(17): 13528-13541
[26] Zhou Z, Chen X, Liao H, et al.(2020) Context-Aware Learning-Based Resource Allocation for Ubiquitous Power IoT.IEEE Internet of Things Magazine, 3(4): 46-52
[27] Wu B, Chen T, Ni W, et al.(2021) Multi-agent multi-armed bandit learning for online management of edge-assisted computing.IEEE Transactions on communications, 69(12):8188-8199
[28] Zhou Z, Chen X, Liao H, et al.(2020) Context-aware learningbased resource allocation for ubiquitous power IoT.IEEE Internet of Things Magazine, 3(4): 46-52
[29] Zhang Y, Wang X, Li J, et al.(2020) Small-current grounding fault location method based on transient main resonance frequency analysis.Global Energy Interconnection, 3(4): 324-334
[30] Ji P, Jia J, Chen J (2019) Joint optimization on both routing and resource allocation for millimeter wave cellular networks.IEEE Access, 7: 93631-93642
[31] Ding Z, Shen L, Chen H, et al.(2020) Energy-efficient relayselection-based dynamic routing algorithm for IoT-oriented software-defined WSNs.IEEE Internet of Things Journal, 7(9):9050-9065
[32] Zhang S, Cui G, Long Y, et al.(2021) Joint computing and communication resource allocation for satellite communication networks with edge computing, 18(7): 236-252
[33] Zhao T, Wang L, Chin K-W, et al.(2021) Routing in energy harvesting wireless sensor networks with dual alternative batteries.IEEE Systems Journal, 15(3): 3970-3979
[34] Huang G, Lu W, Xie J, et al.(2019) Improved route selection strategy based on K shortest path.In: Proceeding of 2019 International Symposium on Networks, Computers and Communications (ISNCC), Istanbul, Turkey, 18-20 Jun 2019
Received: 17 August 2022/ Accepted: 24 November 2022/ Published: 25 June 2023
Yuchen Huang
mudi87@163.com
Ting Zou
79673798@qq.com
Zhanqiang Cheng
286010917@qq.com
Jinshen Liu
zhengwenjie@gddky.csg.cn
Hongwei Guo
hongwei_g@outlook.com
2096-5117/© 2023 Global Energy Interconnection Development and Cooperation Organization.Production and hosting by Elsevier B.V.on behalf of KeAi Communications Co., Ltd.This is an open access article under the CC BY-NC-ND license (http: //creativecommons.org/licenses/by-nc-nd/4.0/ ).
Biographies
Ting Zou received his M.S.degree in Power Systems and Automation from the South China University of Technology, Guangdong, China.He is currently the General Manager of the System Operation Department of Guangdong Power Grid Co., Ltd.Shaoguan Power Supply Bureau, Shaoguan, China.His research interests include power communication, power grid operation, and power regulation.
Yuchen Huang received his M.S.degree in Electrical Engineering and Automation from the Northern University for Nationalities,Ningxia, China.He is currently the Dispatching Autonomous Specialist of the System Operation Department of Guangdong Power Grid Co., Ltd.Shaoguan Power Supply Bureau, Shaoguan, China.His research interests include power communication and power dispatch automation control.
Zhanqiang Cheng received his M.S.degree in Electrical Engineering and Automation from North China Electric Power University,Beijing, China.He is currently the Main Grid Relay Protection Specialist of the System Operation Department of Guangdong Power Grid Co., Ltd.Shaoguan Power Supply Bureau, Shaoguan, China.His research interests include power communication and relay protection.
Jinshen Liu received his M.S.degree in Communication from Xiamen University,Xiamen, China.He is currently the Communication Technology Specialist of the System Operation Department of Guangdong Power Grid Co., Ltd.Shaoguan Power Supply Bureau, Shaoguan, China.His research interests include the power communication network technology.
Hongwei Guo received his M.S.degree in Electrical Engineering and Automation from Nanchang Institute of Technology, Nanchang,China.He is currently the Deputy Manager of Shaoguan Nanxiong Power Supply Bureau,Guangdong Power Grid Co., Ltd., Nanxiong,China.His research interest includes distribution network automation.
(Editor Yanbo Wang)