基于DDPG算法的发电企业报价策略研究

doi:10.11985/2023.02.019

摘要/Abstract

摘要：

随着智能代理算法在解决发电企业代理报价策略问题中的优势不断凸显，国内外相关研究层出不穷。由于我国电力市场发展成熟度不够高，目前多数研究采用的是国外电力市场的交易模式，这不符合我国电力市场交易的实际情况，因此提出一种针对国内电力市场中长期集中竞价交易的报价模型。该报价模型建立在深度确定性策略梯度算法(Deep deterministic policy gradient，DDPG)的基础上，提出兼顾社会总效用最大化和发电企业自身收益的报价策略，建立了以市场环境和发电企业自身情况为参考的状态空间，同时还建立了按照统一边际价格出清的市场出清模型。通过仿真算例验证了该模型的可行性，并与Q-Learning算法的结果进行对比，同时也展现了发电企业自身情况对报价模型的市场出清结果和企业收益的影响。

关键词: 电力市场, 报价策略, 强化学习, DDPG算法

Abstract:

With the advantages of intelligent agent algorithm in solving the problem of agent quotation strategy in power generation enterprises, there are many relevant researches at domestic and abroad. Due to the immaturity of China’s power market, most of the researches are based on the foreign power market transaction mode, which does not accord with the actual situation of China’s power market transaction, so a medium-long term centralized bidding quotation model is put forward for domestic power market. This quotation model is based on deep deterministic policy gradient(DDPG) algorithm, a quotation strategy is proposed considering the maximization of total social utility and the income of power generation enterprises. The state space is established with the market environment and the situation of the power generation enterprise as the reference, and the market clearing model is established according to the unified marginal price. The feasibility of the model is verified by simulation examples, and the results are compared with those of Q-Learning algorithm. At the same time, the influence of the power generation enterprise’s own situation on the market clearing results of the quotation model and the enterprise income is also shown.

Key words: Electricity market, quotation strategy, reinforcement learning, DDPG algorithm

中图分类号:

TP391

马丽莹, 魏云冰. 基于DDPG算法的发电企业报价策略研究[J]. 电气工程学报, 2023, 18(2): 192-200.

MA Liying, WEI Yunbing. Research on Bidding Strategy of Power Generation Enterprise Based on DDPG Algorithm[J]. Journal of Electrical Engineering, 2023, 18(2): 192-200.

图/表 13

图1

图2

图3

图4

表1

发电企业技术参数表"

发电企业序号	额定容量 $P E /MW$	$a / (× 10 − 6)$	$b / (× 10 − 3)$	$c$
G1	300	0.088	305	1 150
G2	500	0.057	285	1 280
G3	550	0.054	281	1 340
G4	550	0.058	279	1 300
G5	600	0.052	274	1 420
G6	600	0.048	272	1 400
G7	650	0.050	267	1 480
G8	700	0.046	264	1 550
G9	850	0.045	260	1 650
G10	1 000	0.036	252	1 800

表1

表2

1—12月月度市场电量需求表"

月份	1月	2月	3月	4月	5月	6月
电量需求/ (万 $kW ⋅ h$ )	70 794	65 068	85 469	78 059	72 764	85 502
月份	7月	8月	9月	10月	11月	12月
电量需求/ (万 $kW ⋅ h$ )	56 407	64 982	71 045	79 004	82 443	76 760

表2

表3

表4

表5

图5

图6

图7

图8

参考文献 23

[1]	张紫鹤, 卢继平. 国外电力市场建设对我国新时期电力市场化改革的启示与建议[J]. 甘肃科技, 2021, 37(6):8-15.
	ZHANG Zihe, LU Jiping. Power market construction in foreign countries:Inspiration and suggestions for China’s power market reform in the new era[J]. Gansu Science and Technology, 2021, 37(6):8-15.
[2]	曾嘉志, 赵雄飞, 李静, 等. 用电侧市场放开下的电力市场多主体博弈[J]. 电力系统自动化, 2017, 41(24):129-136.
	ZENG Jiazhi, ZHAO Xiongfei, LI Jing, et al. Game among multiple entities in electricity market with liberalization of power demand side market[J]. Automation of Electric Power Systems, 2017, 41(24):129-136.
[3]	陈慧, 杨勇, 张颖. 基于agent的电力市场仿真中决策模块的实现[J]. 自动化学报, 2008, 32(2):22-26.
	CHEN Hui, YANG Yong, ZHANG Ying. Realization of decision-making module in agent-based simulation of power markets[J]. Acta Automatica Sinica, 2008, 32(2):22-26.
[4]	刘梅招, 杨莉, 甘德强. 基于agent的电力市场仿真研究综述[J]. 电网技术, 2005, 9(4):76-80.
	LIU Meizhao, YANG Li, GAN Deqiang. A review of power market simulation based on agent[J]. Power System Technology, 2005, 9(4):76-80.
[5]	DAVID A K. Competitive bidding in electricity supply[J]. IEEE Proceeding GT & D, 1993, 140(5):421-426.
[6]	马天男, 杜英, 苟全峰, 等. 基于Berge-NS均衡的电力市场多主体非合作博弈竞争模型[J]. 电力自动化设备, 2019, 39(6):192-204.
	MA Tiannan, DU Ying, GOU Quanfeng, et al. Multi agent non-cooperative game competition model of power market based on Berge-NS equilibrium[J]. Electric Power Automation Equipment, 2019, 39(6):192-204.
[7]	张森林, 卢智. 关于全国统一电力市场框架体系的思考[J]. 中国电力企业管理, 2021, 622(1):49-52.
	ZHANG Senlin, LU Zhi. Thinking on the framework system of national unified electricity market[J]. China Electric Power Enterprise Management, 2021, 622(1):49-52.
[8]	董莎. 基于智能代理算法的发电商报价策略研究[A]. 江西省电机工程学会, 2018.
	DONG Sha. Research on pricing strategy of power supplier based on intelligent agent algorithm[A]. Jiangxi Institute of Electrical Engineering, 2018.
[9]	SUN Junjie, TESFATSION L. Dynamic testing of wholesale power market designs:An open-source agent-based framework[J]. Computational Economics, 2007, 30(3):291-327. doi: 10.1007/s10614-007-9095-1
[10]	唐成鹏, 张粒子, 刘方, 等. 基于多智能体强化学习的电力现货市场定价机制研究(一):不同定价机制下发电商报价双层优化模型[J]. 中国电机工程学报, 2021, 41(2):536-553.
	TANG Chengpeng, ZHANG Lizi, LIU Fang, et al. Research on power spot market pricing mechanism based on multi-agent reinforcement learning(I):Two-tier optimization model of power supplier quotation under different pricing mechanisms[J]. Proceedings of the CSEE, 2021, 41(2):536-553.
[11]	江婕, 荆朝霞, 朱继松. 基于多代理仿真的日前市场发电侧报价方式研究[A]. 中国电机工程学会电力市场专业委员会, 2019:7.
	JIANG Jie, JING Zhaoxia, ZHU Jisong. Research on generation side quotation method of day-ahead market based on multi-agent simulation[A]. Electric Power Market Committee of Chinese Society of Electrical Engineering, 2019:7.
[12]	刘妍, 谭建成. 南方区域大用户参与电力市场交易的现状及展望[J]. 南方电网技术, 2017, 11(11):68-74.
	LIU Yan, TAN Jiancheng. Current situation and prospect of large users’ participation in power market transaction in southern region[J]. China Southern Power Grid Technology, 2017, 11(11):68-74.
[13]	张军. 基于复杂自适应系统架构的发电商竞价仿真研究[D]. 北京: 华北电力大学, 2020.
	ZHANG Jun. Research on bidding simulation of generator based on complex adaptive system architecture[D]. Beijing: North China Electric Power University, 2020.
[14]	罗歆尧. 消费者在日常活动中的效用最大化[J]. 财讯, 2017, 19:155.
	LUO Xinyao. Consumer utility maximization in daily activities[J]. Financial News, 2017, 19:155.
[15]	LILLICRAP T P, HUNT J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]// The International Conference on Learning Representations,San Juan,Puerto Rico, 2016.
[16]	HOU Yuenan, LIU Lifeng, WEI Qing, et al. A novel DDPG method with prioritized experience replay[C]// IEEE International Conference on Systems IEEE, 2017.
[17]	JIN Liqiang, TIAN Duanyang, ZHANG Qixiang, et al. Optimal torque distribution control of multi-axle electric vehicles with in-wheel motors based on DDPG algorithm[J]. Energies, 2020, 13(6):1-19. doi: 10.3390/en13010001
[18]	王成亮, 李澄, 葛永高. 基于多智能体的配电台区智能决策系统研究[J]. 信息技术, 2020, 44(10):49-55.
	WANG Chengliang, LI Cheng, GE Yonggao. Research on intelligent decision system of distribution platform area based on multi-agent[J]. Information Technology, 2020, 44(10):49-55.
[19]	SILVER D, LEVER G, HEESS N, et al. Deterministic policy gradient algorithms[J]. Journal of Machine Learning, 2014, 31:387-395.
[20]	荆朝霞, 朱继松. 月度电量集中竞价市场规则的仿真实验分析[J]. 电力系统自动化, 2017, 41(24):42-48.
	JING Zhaoxia, ZHU Jisong. Simulation experiment analysis of the rules of monthly electricity concentration bidding market[J]. Automation of Electric Power Systems, 2017, 41(24):42-48.
[21]	何建华, 何祖威, 胡效雷. 基于成本分析的火电厂日前市场竞价上网报价[J]. 重庆大学学报, 2008, 31(3):276-279.
	HE Jianhua, HE Zuwei, HU Xiaolei. Pricing of power plants based on cost analysis[J]. Journal of Chongqing University, 2008, 31(3):276-279.
[22]	丁亮. 竞争性电力市场的发电商报价策略研究[D]. 杭州: 杭州电子科技大学, 2019.
	DING Liang. Research on bidding strategy of generator in competitive power market[D]. Hangzhou: Hangzhou Dianzi University, 2019.
[23]	范晨凯. 南方区域跨区跨省中长期电力市场开发策略[D]. 北京: 华北电力大学, 2020.
	FAN Chenkai. Development strategy of medium and long-term power market in southern region[D]. Beijing: North China Electric Power University, 2020.

发电企业	1月	2月	3月	4月	5月	6月	7月	8月	9月	10月	11月	12月
1	5 170	4 438	5 170	4 920	5 170	4 920	5 170	2 006	4 920	5 170	4 920	5 170
2	8 858	1 882	1 385	8 507	8 858	8 507	8 858	8 858	8 507	8 858	8 507	8 858
3	9 963	8 620	2 543	9 506	9 963	9 506	9 963	9 963	9 506	9 963	9 506	9 963
4	8 091	6 883	8 091	1 979	8 091	7 678	8 091	8 091	7 678	8 091	7 678	8 091
5	6 622	5 549	6 622	6 254	6 622	6 254	6 622	6 622	6 254	6 622	399	6 622
6	8 894	7 577	8 894	8 444	8 894	8 444	4 885	8 894	4 544	8 894	8 444	8 894
7	10 081	8 583	10 081	9 569	10 081	9 569	10 081	10 081	9 569	10 081	9 569	3 711
8	11 255	9 792	11 255	10 758	3 016	10 758	11 255	4 742	10 758	11 255	10 758	11 255
9	15 740	13 672	15 740	15 038	15 740	15 038	15 740	15 740	15 038	2 868	15 038	15 740
10	6 308	10 285	12 276	11 593	12 276	5 816	12 276	12 276	11 593	12 276	11 593	12 276

发电企业	1	2	3	4	5
边际成本	0.347 2	0.330 6	0.328 5	0.33	0.323 9
发电企业	6	7	8	9	10
边际成本	0.318 1	0.319	0.315 5	0.321 2	0.309 6

发电企业	1	2	3	4	5
年度总收益	0	21.78	613.44	200.03	615.19
发电企业	6	7	8	9	10
年度总收益	1 542.82	1 772.81	2 298.16	2 908.35	3 386.92