Assume you already have historical stock data for selected stocks, you may want to append new daily stock data to historical data. Again, we will use Yahoo finance as an example. Please note that Yahoo does not like this kind of transaction, thus, it just sometimes terminate the transaction. In this case, just wait for 10-20 seconds and try again.
YouTube demo video here:
import time
from numpy import random
# import pandas_datareader as dr
# Note: This is version 2 of the source code. Use yfinance module instead of pandas_datareader. This is different than the YouTube
import yfinance as yf
start_date = '2022-07-29'
end_date = '2022-07-29'
to_write_csv = True
empty_list = []
stock_list1 = ['0027.HK', '0038.HK', '0116.HK', '0123.HK', '0135.HK', '0148.HK', '0168.HK', '0175.HK',
'0187.HK', '0200.HK', '0257.HK', '0268.HK', '0285.HK', '0303.HK', '0316.HK', '0322.HK',
'0323.HK', '0347.HK', '0358.HK', '0384.HK', '0388.HK', '0392.HK', '0398.HK', '0451.HK']
stock_list2 = ['0489.HK', '0512.HK', '0590.HK', '0639.HK', '0669.HK', '0670.HK', '0688.HK', '0696.HK',
'0698.HK', '0700.HK', '0716.HK', '0732.HK', '0746.HK', '0751.HK', '0754.HK', '0763.HK',
'0823.HK', '0826.HK', '0836.HK', '0838.HK', '0855.HK', '0867.HK', '0868.HK', '0880.HK']
stock_list3 = ['0883.HK', '0914.HK', '0921.HK', '0939.HK', '0960.HK', '0966.HK', '1044.HK', '1053.HK',
'1057.HK', '1066.HK', '1072.HK', '1083.HK', '1093.HK', '1109.HK', '1133.HK', '1171.HK',
'1177.HK', '1193.HK', '1208.HK', '1211.HK', '1308.HK', '1378.HK', '1478.HK', '1600.HK']
stock_list4 = ['1812.HK', '1818.HK', '1882.HK', '1919.HK', '1999.HK', '2002.HK', '2018.HK', '2020.HK',
'2128.HK', '2208.HK', '2238.HK', '2313.HK', '2314.HK', '2318.HK', '2319.HK', '2331.HK',
'2333.HK', '2338.HK', '2343.HK', '2357.HK', '2380.HK', '2382.HK', '2388.HK', '2600.HK']
stock_list5 = ['2678.HK', '2688.HK', '2689.HK', '2883.HK', '2899.HK', '3311.HK', '3323.HK', '3339.HK',
'3800.HK', '3808.HK', '3888.HK', '3899.HK', '3900.HK', '8083.HK']
stock_listhsi = ['^HSI']
stock_listx = []
for stock_num in stock_list1:
stockname = 'HK' + stock_num[0:4]
# stockname = 'HKHSI'
print('--> ' + stockname + ': <--')
# df = dr.data.get_data_yahoo(stock_num, start_date, end_date).reset_index()
df = yf.download(stock_num, start_date, end_date).reset_index()
df2 = df.dropna(axis=0, how="any", inplace=False)
print(df2)
df3 = df2[['Date', 'Open', 'High', 'Low', 'Close', 'Adj Close', 'Volume']]
df3 = df3.round(decimals=4)
print(df3)
# print('HK' + stock_num2[0:4] + '.csv')
if to_write_csv:
writefilepath = 'C:/pythonprojects/stocks/' + stockname + '.csv'
df3.to_csv(writefilepath, mode='a', header=False, index=False)
r1 = random.uniform(0.49, 1.59)
time.sleep(r1)
print('')
print('')
'''
# to remove last duplicate row:
for stock_num in stock_list1:
stockname = 'HK' + stock_num[0:4]
print('--> ' + stockname + ': <--')
writefilepath = 'C:/pythonprojects/stocks/' + stockname + '.csv'
df = pd.read_csv(writefilepath, index_col=0, parse_dates=True)
df = df[:-1]
print(df.tail(2))
df.to_csv(writefilepath)
'''
Line 6-7, change the date of getting daily stock data. Yahoo finance uses date format yyyy-mm-dd
Line 13-30, due to Yahoo sometimes kill the transaction, I put the stock codes into 5 smaller groups. If Yahoo terminates the transaction in the middle, I will put the rest of the codes into line 34 stock_listx[] and re-run the program. Please watch the video above.
Line 13-30, these are Hong Kong stock code format. User should make change for themselves such as ‘GOOGL’, ‘AMD’, etc.
Line 37-56, for python new user, remember to put 4 space with same intent after for loop. Otherwise, the codes will exit the loop.
Line 41, this is the actual code that get data from yahoo finance server. It will get a row of data (day of 2022-07-29) into variable df.
Line 44, change the column order. Daily data column is different than the data when manually download from yahoo finance. Change column order to match the columns same as last demo before append.
Line 45, round any column with too many decimal points data.
Line 50, user needs to change your path name and file name here.
Line 51, write the daily data, append to the historical data and save it in the csv file.
Line 53-54, just to make a random sleep time (temporarily stop execution) for the program, so that yahoo finance server will less likely to kill the transaction.
Line 58-70, in case duplicate download daily data, uncomment this part to remove the last row of data in csv file.
*** Same as previous post, Yahoo finance server does not response correctly with pandas_datareader module, and no data can be downloaded. To fix this problem, we will use another module “yfinance” and make slightly changes on the source code. Please refer to the previous post source code that added at the bottom.