Author : Sanjoga Sahu. Date of Modification : 5th, Aug, 2014. (v1.0.0)

Operating System(s) : Linux

Description : Checks the Market Data feeds for the correctness of data.

Types of Checks Perfomed ========================

1. Feed Stale Check 2. Feed Time Stamp Date Check 3. Feed Arrival Time Stamp Check 4. Feed Size Check 5. Feed Number of Records Check 6. Header Line Check 7. Header Line Date Check 8. Number of Columns Check 9. All Records Check 10. Duplicate Record Check 11. Trailer Line Check 12. Trailer Line Date Check 13. Feed Variance Check

Execution Method :

Note: The Config File Location in Prod : /apps/config//chk_feeds/chk_feeds_config.xml (Default) The Script Location in prod : /apps//batch/bin/chk_feeds.pl The Wrapper Script Location in prod : /apps//batch/bin/_run_chk_feeds.pl The Wrapper Config File Location in Prod : /apps/config//chk_feeds/_run_chk_feeds.cfg (Default)

1. chk_feeds.pl -f /ppb/data/LATEST/Prices/sobratePrices.dat

i. No need to Provide run date if the execution date is same as business date.
ii. The feed name provided should be the absolute.
iii. NOTE : <-f> Option is Mandatory

2. chk_feeds.pl -f /ppb/data/LATEST/Prices/sobratePrices.dat -d 20091119

				i. Provide run date if the execution date Not same as business date.
 

3. chk_feeds.pl -f /ppb/data/LATEST/yield_history/EMEA/MCBenchmarks.dat -d 20091119 -p DEM -e /apps/rvg/lib/holidays/DEM.HOL

i. Provide the prefix of the feed if the config file is same for multiple currency (case Sensitive)
ii. -e escape the holidays for the currency

4. chk_feeds.pl -f /ppb/data/LATEST/Prices/sobratePrices.dat -d 20091119 -c cfg/chk_feeds_config_sobrate_govtPrices.xml

				i. specify the config file if one want to use a particular config file of interest.
                ii. NOTE : default config file is "chk_feeds_config.xml"

---------------------------------- Execution from own directory: Ex:

$SCRIPTSROOT/market_feeds/_run_chk_feeds.pl -rGLOBAL -yGENERAL -d 20110207 -x /home/n209263/dev/chkfeeds/ -c /home/n209263/dev/chkfeeds/run_chk_feeds.cfg -s /home/n209263/dev/chkfeeds/chk_feeds.pl -k /home/n209263/dev/chkfeeds/log/ $SCRIPTSROOT/market_feeds/_run_chk_feeds.pl -rASIA -yGENERAL -x /home/n209263/dev/chkfeeds/ -c /home/n209263/dev/chkfeeds/run_chk_feeds.cfg -s /home/n209263/dev/chkfeeds/chk_feeds.pl -k /home/n209263/dev/chkfeeds/log/ -d 201103221

/apps/rvg/bin/market_feeds/_run_chk_feeds.pl -rASIA -yTHB -x /home/n209263/dev/chkfeeds/ -c /home/n209263/dev/chkfeeds/run_chk_feeds.cfg -s /home/n209263/dev/chkfeeds/chk_feeds.pl -k /home/n209263/dev/chkfeeds/log/ -d 20110503

----------------------------------

Sending Check Report E-mail : /apps/config//chk_feeds/chk_feeds_config.xml (Default) <feed_check_email check="y"> <send_email_always>/send_email_always ----> Always sends Check Report E-mail. ("N" for Not always) <send_email_when_fail>/send_email_when_fail ----> Sends Check Report if Any Check Fails. ("N" NO E-mail if fail) <feed_check_email_to>abc@xyz.com</feed_check_email_to> <feed_check_email_cc>sanjogsahu@xyz.com</feed_check_email_cc> ---> update "NA" if no cc required </feed_check_email>

This subroutine basically checks the main (parent) Check tag and redirects the check to "check_action" subroutine for further processing if the check is "y"

This subroutine basically checks the main (parent) Check tag and redirects the check to "check_action" subroutine for further processing if the check is "y"

This Subroutine routes the correct subrotine to a particular feed check task

This Subroutine compare between two feeds of similar nature

Arguments: 1. $feed_name_actual (Feed name with prefix and postfix) 2. $comp_feed (feed name without prefix and postfix) 3. $check (feed_variance_check)

This Subroutine sends e-mail

Arguments:

1. $feed_name_actual (Actual feed name with pre-post fix) 2. $comp_feed (feed name without pre-post fix) 3. $tag (Checker tag)

This Subroutine adds prefix to the feed

Arguments: 1. $comp_feed (Feed Name) 2. $check (Cheker tag) 3. $type (this_day | prev_day)

This Subroutine adds post fix date to the feed

Arguments: 1. $comp_feed (Feed Name) 2. $checker_tag (Cheker tag) 3. $type (this_day | prev_day)

This Subroutine adds prefix to the feed

Arguments: 1. $comp_feed (Feed Name) 2. $check (Cheker tag) 3. $type (this_day | prev_day)

This Subroutine adds postfix to the feed

Arguments: 1. $comp_feed (Feed Name) 2. $check (Cheker tag) 3. $type (this_day | prev_day)

This Subroutine validates whether the prefix provided in command line <-p> is a valid one or not

if the prefix sring in config is like Example: <prev_prefix_str>DEM|GBP|EUR|CHF|AUD|JPY|CAD|SEK|HKD|NOK|NZD|DK/prev_prefix_str then have to provide the -p option in commad line the propper prifix (case sensitive)

Arguments: 1. $comp_feed (Feed Name) 2. $check (Cheker tag) 3. $info (information string to print)

This Subroutine adds post fix date to the feed

Arguments: 1. $comp_feed (Feed Name) 2. $checker_tag (Cheker tag) 3. $type (this_day | prev_day)

This Subroutine checks whether the feed is stale or not ( current feed vs Previous Days Feed) if there is not difference between current and provious feed, then the check fails

Arguments: 1. $feed_today (Current Feed) 2. $feed_prev (Previous Days' Feed [bussiness Date])

This Subroutine checks all feed attribute like "date", "size", "time" of feed on the disk

Arguments: 1. $feed_name_actual (Feed name with prefix and postfix) 2. $comp_feed (feed name without prefix and postfix) 3. $check (date | size | time)

This Subroutine checks Number of Lines in the feed

Arguments: 1. $feed_name_actual (Feed name with prefix and postfix) 2. $comp_feed (feed name without prefix and postfix)

This Subroutine checks and evalute all records, basically (header line, all line of feed, trailer line etc.)

Arguments: 1. $feed_name_actual (Feed name with prefix and postfix) 2. $comp_feed (feed name without prefix and postfix) 3. $check (feed_header_line | feed_header_line_date | feed_num_column | feed_all_line_check | feed_duplicate_line | feed_tailer_line | feed_tailer_line_date)

This Subroutine checks the date in the feed line given

Arguments: 1. $date_type_tag (date type tag defeind in config.) 2. $date_format_tag (Date format tag defined in the config) 3. $date_regex_tag (regular Expression tag defined in config.) 4. $date_line_num_tag (line number to check tag defined in config.) 5. $all_lines (reference of @all_lines) 6. $info (information to print)

This Subroutine checks all the line of feed with escape line and matching string given

Arguments:

1. $all_lines (reference of @all_lines) 2. $index_hash (Hash Reference which contains all escape line index number) 3. $chk_line_regex (Regular Expression to match lines) 4. $case (Case Sensitivity while matching lines) 5. $type (escape)

This Subroutine gets the Value of the cheker tag provided

Arguments:

1. $comp_feed (feed name without prefix and postfix) 2. $chk_tag (tag name) 3. $info (information to print) 4. $type (conf | any String) to print. 5. $type_print (date_type_print) to print curent date or bussiness date (optional)

This Subroutine checks whether the file exists on the disck or not

Arguments:

1. $feed_name (Actual feed name with pre-post fix) 2. $type (this_day | prev_day)

This Subroutine trims pre-post white Spaces/new line from the string

Arguments:

1. $var (input String)

This Subroutine parses the xml config file (start tag)

Arguments:

1. $parseinst (class reference) 2. $element (xml tag of interest) 3. %attrs (all xml tags from config and it's values)

This Subroutine parses the xml config file (values of tag)

Arguments:

1. $parseinst (class reference) 2. $data (tag of interest)

This Subroutine parses the xml config file (end tag)

Arguments:

1. $parseinst (class reference) 2. $element (tag of interest)

This Subroutine parses the xml config file escape lines (Comment lines new lines)

Arguments: 1. $parseinst (class reference) 2. $element (tag of interest)

This Subroutine returns the directory name and the file name of given file name having absolute path

Arguments: 1. $feed (feed name with absolute path)

This Subroutine returns feed Properties like below: Ex: Feed Properties : -rw-rw-r-- 1 pvmuser primus 74450 2009-11-17 20:13:55 /ppb/data/LATEST/Prices/sobratePrices.dat.20091117

Arguments:

1. $feed_name (Actual feed name with pre-postfix)

This Subroutine returns "file size in bytes", "date" and "time stamp" of the given input like below: -rw-rw-r-- 1 pvmuser primus 74450 2009-11-17 20:13:55 /ppb/data/LATEST/Prices/sobratePrices.dat.20091117

Arguments:

1. $file (Actual feed name with pre-post fix)

This Subroutine whether date type provided is Bussness date or Current date

Arguments:

1. $str (cur_date... | bus_date...)

input string : key1 : Start date => value1 : YYYY-MM-DD hh:mm:ss key2 : End date => value2 : YYYY-MM-DD hh:mm:ss

returns : days, hours, min, and sec.

This Subroutine displayes the Usage information about the script

This Subroutine write master log file

Arguments:

1. $log_type [prefix the messasge type](info | dum |err | conf) [ dum-> prefixes nothing) 2. $message (printing messaxge)

This Subroutine retuns the hash having index number of the array to be used for checking records

Arguments:

1. $array (reference of the array) 2. $chk_line_num (1,2... | L , L-1...)

This Subroutine sends e-mail

Arguments:

1. $feed_name_actual (Actual feed name with pre-post fix) 2. $comp_feed (feed name without pre-post fix) 3. $tag (Checker tag)

This Subroutine creates the e-mail Body (appending mode)

Arguments:

1. $msg (message to append in the body) 2. $status (OK | Failed) 3. $info (information regarding feed)

This Subroutine get all valid prefixes provided in the config file in a hash)

Arguments:

1. $str (feed prefix string) 2. $delim (delimitter used to separate the pre fixes of feed in the config file)

This Subroutine generates unique string using the absolute path of the feed

Arguments:

1. $feed_dir (actual feed name with pre-postfix)

This Subroutine generates unique feed log name

Arguments:

1. $feed_name (actual feed name with pre-postfix)

This Subroutine generates unique feed log file (contains only current feed check information)

Arguments:

1. $feed_log (log for individual feed)

This Subroutine shows the summary of all checks

This Subroutine exits the programe by displaying the proper reason of exit) Also handles the loging , email and summary of check

Arguments: $exit_code (input exit code)

This Subroutine Alerts by sending e-mail and displaying the proper reason of failure) Also handles the loging , email and summary of check

Arguments: $exit_code (input exit code)

This Subroutine checks xml tags exists or not

Arguments:

1. $feed (feed name) 2. $chk_tag (tag name) &check_xml_tag_exist($hash,$feed,'feed_column_name_line_num')

cpanm BusDayUtil

perl -MCPAN -e shell
install BusDayUtil