%{
Script for Bringing yahoo SP500 (date=01/07/2007) stocks Data into Matlab Workspace

This is a simple algorithm that downloads all most updated data from the
stocks in the SP500 index, for as many day you want and for different frequencies (dailly,
weeklly and monthly). 

The main advantage of this function is that it controls for bad data or for
the cases where yahoo server doesn't have the data you want. You can also set a
critera for bad data (eg. missing 5% of valid prices, when comparing to a full date ticker).
When bad data is found, the function skips that particular asset.

It is basicly a large scale application of sqq.m which is originally submited by
Michael Boldin, link:

http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=4069&objectType=file

The sqq.m works beautifully. Thanks Michael.

The steps are:

1) Get SP500 tickers from tickers.txt (you can change this file for you own purpose)
1) Download data using sqq() (controlling for bad data and missing prices)
2) Substitute values of 0 (price=0) for the most closest previous price
(this way you set return on that date = 0. This substitution shouldn't happen
much. You can control the filter for this event by using a reasonable value for variable Critera)

Please notes that SP500 changes composition over time. The composition 
for the function is from 01-july-07. I'm trying to set a solution for this
(any suggestions are welcome..)

INPUT:

    n_days_back - Integer. Number of days to go back (including weekends and holidays)

    n_stocks    - Number of stocks which to get data from sp500 (in the same order as ticker.txt)

    Freq        - (optional,default='d'). Frequency of data. Options: 'd' dailly, 'w' weekly, 'm', monthlly

    Criteria    - (optional, default=.05). The percentage of a full valid price vector (from a valid ticker eg. IBM)
                  that will set how many missing prices is enough to set bad data event (and cut such stock
                  from the downloaded database.

OUTPUT:

    SPData - A structure with the folowning fields:

    Date
    Close
    Open
    High
    Low
    Volume
    Closeadj
    Valid_Tickers - The downloaded valid tickers (those with enough valid prices and no yahoo problems)

    In such matrices, each collum represents each asset

Author: Marcelo Scherer Perlin
Email:  marceloperlin@gmail.com
Phd Student in finance ICMA/UK (Starting october 2007)
Created: July/2007

Fell free to use it and/or modify it for your own interest.

Any comments are welcome, if you have a suggestion that will significantly
improve the code, please write it and send it to me. If the changes are interesting,
I'll repost the file with full aknowledgements.

%}

function [SPData]=Get_Tickers_Data(n_days_back,n_stocks,Freq,P_Criteria)
    
    addpath([pwd,'\sqq_msp']);    
    
    if nargin()<2
        error('The function needs at least 2 inputs');
    end
    
    if nargin()==2
        Freq='d';
        P_Criteria=.05;
    end
    
    if nargin()==3
        P_Criteria=.05;
    end
        
    if (exist('tickers.txt','file'))==0
        error('The required tickers.txt file was not found at working directory. Please check it.');
    end
    
    if n_days_back<0||n_stocks<0||P_Criteria<0
        error('The inputs n_days, n_stocks and Criteria should be all positive');
    end
    
    if P_Criteria>1
        error('The input Criteria is a percentage and therefore lower than 1');
    end
    
    % Getting ticker from ticker.txt
    
    fid = fopen('tickers.txt', 'r');
    a=textscan(fid, '%s','delimiter',' ');
    fclose(fid);

    tickers=a{1,1};

    fprintf(1,'\nDownloading IBM stock for Date comparisons <-');
    
    % Downloading IBM for date comparinson (from my tests IBM is a reliable
    % stock regarding dates)
    
    try
        first_out=sqq_msp('ibm',today-n_days_back,today,Freq);
    catch
        error('Problem with downloading the IBM stock for dates comparison. Please check internet conection and try again..');
    end
    
    % creting Criteria for bad data. When number of missing prices is higher than Criteria, 
    % then the stock is removed from database
    
    Criteria=floor(P_Criteria*length(first_out));
    
    fprintf(1,['Download complete:\n First Date in data -> ',datestr(first_out(1,1)),'\n Last  Date in data -> ',datestr(first_out(end,1))])
    fprintf(1,['\nCriteria for bad data is: ',num2str(Criteria),' missing prices.\n']);

    Date(:,1)=first_out(:,1);

    fprintf(1,'\n-> Starting Yahoo Downloads <-');

    idx=0;
    for i=1:n_stocks
        try
            fprintf(1,['\nAsset #',num2str(i),' - Ticker: ',tickers{i},' --> ']);
            
            out=sqq_msp(tickers{i},today-n_days_back,today,Freq);
            
            if size(out,1)==size(first_out(:,1),1) % cases where the dates match

                idx=idx+1;

                Date(:,idx)=out(:,1);
                Close(:,idx)=out(:,2);
                Open(:,idx)=out(:,3);
                Low(:,idx)=out(:,4);
                High(:,idx)=out(:,5);
                Volume(:,idx)=out(:,6);
                Closeadj(:,idx)=out(:,7);

                fprintf(1,'DL OK. Dates fully matching.');
                Valid_Tickers{1,idx}=tickers{i};
                
            elseif (length(first_out(:,1))-length(out(:,7)))<Criteria % cases where the dates DONT match

                idx=idx+1;

                for j=1:size(out,1) % this loop will set all prices according to matching dates of valid ticker (in this case IBM)

                    [a]=find(out(j,1)==Date(:,1));

                    Date(a,idx)=out(j,1);
                    Close(a,idx)=out(j,2);
                    Open(a,idx)=out(j,3);
                    Low(a,idx)=out(j,4);
                    High(a,idx)=out(j,5);
                    Volume(a,idx)=out(j,6);
                    Closeadj(a,idx)=out(j,7);

                end

                fprintf(1,['DL OK, but is missing ',num2str(length(first_out(:,1))-length(out(:,7))),' price(s)']);
                Valid_Tickers{1,idx}=tickers{i};

            else
                fprintf(1,'DL OK, but number of missing prices is higher than criteria. Skipping this one.');
            end

        catch
            fprintf(1,'Problem with yahoo Data, skipping this one');
            continue
        end
    end

    % fixing cases where the first price = 0 (this would be a problem in the
    % substituion part
    
    [a]=find(Close(1,:)==0);

    for j=1:size(a,2)
        for i=1:size(Close,1)
            if Close(i,j)~=0

                Close(1,a(j))=Close(i,j);
                Closeadj(1,a(j))=Closeadj(i,j);
                High(1,a(j))=High(i,j);
                Low(1,a(j))=Low(i,j);
                Open(i,a(j))=Open(i,j);

                break;
            end
        end

    end
    
    % Replacing prices = 0 with the oldest availabe price
    
    for i=2:size(Close,1)
        for j=1:size(Close,2)
            if Close(i,j)==0

                Close(i,j)=Close(i-1,j);
                Closeadj(i,j)=Closeadj(i-1,j);
                High(i,j)=High(i-1,j);
                Low(i,j)=Low(i-1,j);
                Open(i,j)=Open(i-1,j);

            end
        end
    end
    
    % Saving it all in the structure

    SPData.Close=Close;
    SPData.Closeadj=Closeadj;
    SPData.Date=Date;
    SPData.Open=Open;
    SPData.Volume=Volume;
    SPData.Low=Low;
    SPData.High=High;
    SPData.Valid_Tickers=Valid_Tickers;
    
    fprintf(1,'\n');