X-Git-Url: http://gitweb.michael.orlitzky.com/?a=blobdiff_plain;f=doc%2Fman1%2Fhtsn-import.1;h=693cf8f98976e533f7f0d12c3555b997f07fb8ba;hb=4a394d2733e0b8bb92d49200858e02bd95827e61;hp=f1edf446f5b20b5a68dddf9722aec2006869c1dd;hpb=a599e73b762cc14239c2dc22be9bec7c1df90548;p=dead%2Fhtsn-import.git diff --git a/doc/man1/htsn-import.1 b/doc/man1/htsn-import.1 index f1edf44..693cf8f 100644 --- a/doc/man1/htsn-import.1 +++ b/doc/man1/htsn-import.1 @@ -48,29 +48,233 @@ pickle/unpickle everything already, this should be impossible. The XML document types obtained from the feed are uniquely identified by their DTDs. We currently support documents with the following DTDs: .IP \[bu] 2 -Heartbeat.dtd +AutoRacingResultsXML.dtd .IP \[bu] -newsxml.dtd +Auto_Racing_Schedule_XML.dtd +.IP \[bu] +Heartbeat.dtd .IP \[bu] Injuries_Detail_XML.dtd .IP \[bu] injuriesxml.dtd .IP \[bu] +newsxml.dtd +.IP \[bu] Odds_XML.dtd .IP \[bu] +scoresxml.dtd +.IP \[bu] weatherxml.dtd +.IP \[bu] +GameInfo +.RS +.IP \[bu] +CBASK_Lineup_XML.dtd +.IP \[bu] +cbaskpreviewxml.dtd +.IP \[bu] +cflpreviewxml.dtd +.IP \[bu] +Matchup_NBA_NHL_XML.dtd +.IP \[bu] +MLB_Gaming_Matchup_XML.dtd +.IP \[bu] +MLB_Lineup_XML.dtd +.IP \[bu] +MLB_Matchup_XML.dtd +.IP \[bu] +MLS_Preview_XML.dtd +.IP \[bu] +mlbpreviewxml.dtd +.IP \[bu] +NBA_Gaming_Matchup_XML.dtd +.IP \[bu] +NBA_Playoff_Matchup_XML.dtd +.IP \[bu] +NBALineupXML.dtd +.IP \[bu] +nbapreviewxml.dtd +.IP \[bu] +NCAA_FB_Preview_XML.dtd +.IP \[bu] +NFL_NCAA_FB_Matchup_XML.dtd +.IP \[bu] +nflpreviewxml.dtd +.IP \[bu] +nhlpreviewxml.dtd +.IP \[bu] +recapxml.dtd +.IP \[bu] +WorldBaseballPreviewXML.dtd +.RE +.IP \[bu] +SportInfo +.RS +.IP \[bu] +CBASK_3PPctXML.dtd +.IP \[bu] +Cbask_All_Tourn_Teams_XML.dtd +.IP \[bu] +CBASK_AssistsXML.dtd +.IP \[bu] +Cbask_Awards_XML.dtd +.IP \[bu] +CBASK_BlocksXML.dtd +.IP \[bu] +Cbask_Conf_Standings_XML.dtd +.IP \[bu] +Cbask_DivII_III_Indv_Stats_XML.dtd +.IP \[bu] +Cbask_DivII_Team_Stats_XML.dtd +.IP \[bu] +Cbask_DivIII_Team_Stats_XML.dtd +.IP \[bu] +CBASK_FGPctXML.dtd +.IP \[bu] +CBASK_FoulsXML.dtd +.IP \[bu] +CBASK_FTPctXML.dtd +.IP \[bu] +Cbask_Indv_Scoring_XML.dtd +.IP \[bu] +CBASK_MinutesXML.dtd +.IP \[bu] +Cbask_Polls_XML.dtd +.IP \[bu] +CBASK_ReboundsXML.dtd +.IP \[bu] +CBASK_ScoringLeadersXML.dtd +.IP \[bu] +Cbask_Team_ThreePT_Made_XML.dtd +.IP \[bu] +Cbask_Team_ThreePT_PCT_XML.dtd +.IP \[bu] +Cbask_Team_Win_Pct_XML.dtd +.IP \[bu] +Cbask_Top_Twenty_Five_XML.dtd +.IP \[bu] +CBASK_TopTwentyFiveResult_XML.dtd +.IP \[bu] +Cbask_Tourn_Awards_XML.dtd +.IP \[bu] +Cbask_Tourn_Champs_XML.dtd +.IP \[bu] +Cbask_Tourn_Indiv_XML.dtd +.IP \[bu] +Cbask_Tourn_Leaders_XML.dtd +.IP \[bu] +Cbask_Tourn_MVP_XML.dtd +.IP \[bu] +Cbask_Tourn_Records_XML.dtd +.IP \[bu] +LeagueScheduleXML.dtd +.IP \[bu] +minorscoresxml.dtd +.IP \[bu] +Minor_Baseball_League_Leaders_XML.dtd +.IP \[bu] +Minor_Baseball_Standings_XML.dtd +.IP \[bu] +Minor_Baseball_Transactions_XML.dtd +.IP \[bu] +mlbbattingavgxml.dtd +.IP \[bu] +mlbdoublesleadersxml.dtd +.IP \[bu] +MLBGamesPlayedXML.dtd +.IP \[bu] +MLBGIDPXML.dtd +.IP \[bu] +MLBHitByPitchXML.dtd +.IP \[bu] +mlbhitsleadersxml.dtd +.IP \[bu] +mlbhomerunsxml.dtd +.IP \[bu] +MLBHRFreqXML.dtd +.IP \[bu] +MLBIntWalksXML.dtd +.IP \[bu] +MLBKORateXML.dtd +.IP \[bu] +mlbonbasepctxml.dtd +.IP \[bu] +MLBOPSXML.dtd +.IP \[bu] +MLBPlateAppsXML.dtd +.IP \[bu] +mlbrbisxml.dtd +.IP \[bu] +mlbrunsleadersxml.dtd +.IP \[bu] +MLBSacFliesXML.dtd +.IP \[bu] +MLBSacrificesXML.dtd +.IP \[bu] +MLBSBSuccessXML.dtd +.IP \[bu] +mlbsluggingpctxml.dtd +.IP \[bu] +mlbstandxml.dtd +.IP \[bu] +mlbstandxml_preseason.dtd +.IP \[bu] +mlbstolenbasexml.dtd +.IP \[bu] +mlbtotalbasesleadersxml.dtd +.IP \[bu] +mlbtriplesleadersxml.dtd +.IP \[bu] +MLBWalkRateXML.dtd +.IP \[bu] +mlbwalksleadersxml.dtd +.IP \[bu] +MLBXtraBaseHitsXML.dtd +.IP \[bu] +MLB_Pitching_Appearances_Leaders.dtd +.IP \[bu] +MLB_ERA_Leaders.dtd +.IP \[bu] +MLB_Pitching_Balks_Leaders.dtd +.IP \[bu] +MLB_Pitching_CG_Leaders.dtd +.IP \[bu] +MLB_Pitching_ER_Allowed_Leaders.dtd +.IP \[bu] +MLB_Pitching_Hits_Allowed_Leaders.dtd +.IP \[bu] +MLB_Pitching_Hit_Batters_Leaders.dtd +.IP \[bu] +MLB_Pitching_HR_Allowed_Leaders.dtd +.IP \[bu] +MLB_Pitching_IP_Leaders.dtd +.IP \[bu] +MLB_Pitching_Runs_Allowed_Leaders.dtd +.IP \[bu] +MLB_Pitching_Saves_Leaders.dtd +.IP \[bu] +MLB_Pitching_Shut_Outs_Leaders.dtd +.IP \[bu] +MLB_Pitching_Starts_Leaders.dtd +.IP \[bu] +MLB_Pitching_Strike_Outs_Leaders.dtd +.RE +.P +The GameInfo and SportInfo types do not have their own top-level +tables in the database. Instead, their raw XML is stored in either the +\(dqgame_info\(dq or \(dqsport_info\(dq table respectively. .SH DATABASE SCHEMA .P -At the top level, we have one table for each of the XML document types -that we import. For example, the documents corresponding to -\fInewsxml.dtd\fR will have a table called \(dqnews\(dq. All top-level -tables contain two important fields, \(dqxml_file_id\(dq and -\(dqtime_stamp\(dq. The former is unique and prevents us from -inserting the same data twice. The time stamp on the other hand lets -us know when the data is old and can be removed. The database schema -make it possible to delete only the outdated top-level records; all -transient children should be removed by triggers. +At the top level (with two notable exceptions), we have one table for +each of the XML document types that we import. For example, the +documents corresponding to \fInewsxml.dtd\fR will have a table called +\(dqnews\(dq. All top-level tables contain two important fields, +\(dqxml_file_id\(dq and \(dqtime_stamp\(dq. The former is unique and +prevents us from inserting the same data twice. The time stamp on the +other hand lets us know when the data is old and can be removed. The +database schema make it possible to delete only the outdated top-level +records; all transient children should be removed by triggers. .P These top-level tables will often have children. For example, each news item has zero or more locations associated with it. The child @@ -102,9 +306,41 @@ to delete the old games (through an ON DELETE CASCADE, tied to unique constraint in the top-level table's \(dqxml_file_id\(dq will prevent duplication in this case anyway. .P +The aforementioned exceptions are the \(dqgame_info\(dq and +\(dqsport_info\(dq tables. These tables contain the raw XML for a +number of DTDs that are not handled individually. This is partially +for backwards-compatibility with a legacy implementation, but is +mostly a stopgap due to a lack of resources at the moment. These two +tables (game_info and sport_info) still possess timestamps that allow +us to prune old data. +.P UML diagrams of the resulting database schema for each XML document type are provided with the \fBhtsn-import\fR documentation. +.SH XML Schema Oddities +.P +There are a number of problems with the XML on the wire. Even if we +construct the DTDs ourselves, the results are sometimes +inconsistent. Here we document a few of them. + +.IP \[bu] 2 +Odds_XML.dtd + +The elements here are supposed to be associated with a set of + elements, but since the pair +(......) can appear zero or more times, +this leads to ambiguity in parsing. We therefore ignore the notes +entirely (although a hack is employed to facilitate parsing). + +.IP \[bu] +weatherxml.dtd + +There appear to be two types of weather documents; the first has + contained within and the second has +contained within . While it would be possible to parse both, +it would greatly complicate things. The first form is more common, so +that's all we support for now. + .SH OPTIONS .IP \fB\-\-backend\fR,\ \fB\-b\fR @@ -128,10 +364,11 @@ will not be auto-rotated; use something like logrotate for that. Default: none .IP \fB\-\-log-level\fR -How verbose should the logs be? We log notifications at three levels: -INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of +How verbose should the logs be? We log notifications at four levels: +DEBUG, INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of notifications you would like to receive (in all-caps); more -interesting notifications will be logged as well. +interesting notifications will be logged as well. The debug output is +extremely verbose and will not be written to syslog even if you try. Default: INFO