X-Git-Url: http://gitweb.michael.orlitzky.com/?a=blobdiff_plain;f=doc%2Fman1%2Fhtsn-import.1;h=1bf75ff1ab8aa44bb03bb5a00bf7aa5b0bbf880a;hb=f2822a93d9f78a2f4c16b272f905b3c36bb4870d;hp=f1edf446f5b20b5a68dddf9722aec2006869c1dd;hpb=a599e73b762cc14239c2dc22be9bec7c1df90548;p=dead%2Fhtsn-import.git diff --git a/doc/man1/htsn-import.1 b/doc/man1/htsn-import.1 index f1edf44..1bf75ff 100644 --- a/doc/man1/htsn-import.1 +++ b/doc/man1/htsn-import.1 @@ -48,29 +48,67 @@ pickle/unpickle everything already, this should be impossible. The XML document types obtained from the feed are uniquely identified by their DTDs. We currently support documents with the following DTDs: .IP \[bu] 2 -Heartbeat.dtd +Auto_Racing_Schedule_XML.dtd +.IP \[bu] 2 +CBASK_Lineup_XML.dtd (GameInfo) +.IP \[bu] 2 +cbaskpreviewxml.dtd (GameInfo) +.IP \[bu] 2 +cflpreviewxml.dtd (GameInfo) .IP \[bu] -newsxml.dtd +Heartbeat.dtd .IP \[bu] Injuries_Detail_XML.dtd .IP \[bu] injuriesxml.dtd +.IP \[bu] 2 +Matchup_NBA_NHL_XML.dtd (GameInfo) +.IP \[bu] +MLB_Gaming_Matchup_XML.dtd (GameInfo) +.IP \[bu] +MLB_Lineup_XML.dtd (GameInfo) +.IP \[bu] +MLB_Matchup_XML.dtd (GameInfo) +.IP \[bu] +MLS_Preview_XML.dtd (GameInfo) +.IP \[bu] +mlbpreviewxml.dtd (GameInfo) +.IP \[bu] +NBA_Gaming_Matchup_XML.dtd (GameInfo) +.IP \[bu] +NBA_Playoff_Matchup_XML.dtd (GameInfo) +.IP \[bu] +NBALineupXML.dtd (GameInfo) +.IP \[bu] +nbapreviewxml.dtd (GameInfo) +.IP \[bu] +newsxml.dtd +.IP \[bu] +nhlpreviewxml.dtd (GameInfo) .IP \[bu] Odds_XML.dtd .IP \[bu] +recapxml.dtd (GameInfo) +.IP \[bu] +scoresxml.dtd +.IP \[bu] weatherxml.dtd +.P +The GameInfo and SportsInfo types do not have their own top-level +tables in the database. Instead, their raw XML is stored in either the +\(dqgame_info\(dq or \(dqsports_info\(dq table respectively. .SH DATABASE SCHEMA .P -At the top level, we have one table for each of the XML document types -that we import. For example, the documents corresponding to -\fInewsxml.dtd\fR will have a table called \(dqnews\(dq. All top-level -tables contain two important fields, \(dqxml_file_id\(dq and -\(dqtime_stamp\(dq. The former is unique and prevents us from -inserting the same data twice. The time stamp on the other hand lets -us know when the data is old and can be removed. The database schema -make it possible to delete only the outdated top-level records; all -transient children should be removed by triggers. +At the top level (with two notable exceptions), we have one table for +each of the XML document types that we import. For example, the +documents corresponding to \fInewsxml.dtd\fR will have a table called +\(dqnews\(dq. All top-level tables contain two important fields, +\(dqxml_file_id\(dq and \(dqtime_stamp\(dq. The former is unique and +prevents us from inserting the same data twice. The time stamp on the +other hand lets us know when the data is old and can be removed. The +database schema make it possible to delete only the outdated top-level +records; all transient children should be removed by triggers. .P These top-level tables will often have children. For example, each news item has zero or more locations associated with it. The child @@ -102,9 +140,41 @@ to delete the old games (through an ON DELETE CASCADE, tied to unique constraint in the top-level table's \(dqxml_file_id\(dq will prevent duplication in this case anyway. .P +The aforementioned exceptions are the \(dqgame_info\(dq and +\(dqsports_info\(dq tables. These tables contain the raw XML for a +number of DTDs that are not handled individually. This is partially +for backwards-compatibility with a legacy implementation, but is +mostly a stopgap due to a lack of resources at the moment. These two +tables (game_info and sports_info) still possess timestamps that allow +us to prune old data. +.P UML diagrams of the resulting database schema for each XML document type are provided with the \fBhtsn-import\fR documentation. +.SH XML Schema Oddities +.P +There are a number of problems with the XML on the wire. Even if we +construct the DTDs ourselves, the results are sometimes +inconsistent. Here we document a few of them. + +.IP \[bu] 2 +Odds_XML.dtd + +The elements here are supposed to be associated with a set of + elements, but since the pair +(......) can appear zero or more times, +this leads to ambiguity in parsing. We therefore ignore the notes +entirely (although a hack is employed to facilitate parsing). + +.IP \[bu] +weatherxml.dtd + +There appear to be two types of weather documents; the first has + contained within and the second has +contained within . While it would be possible to parse both, +it would greatly complicate things. The first form is more common, so +that's all we support for now. + .SH OPTIONS .IP \fB\-\-backend\fR,\ \fB\-b\fR @@ -128,10 +198,11 @@ will not be auto-rotated; use something like logrotate for that. Default: none .IP \fB\-\-log-level\fR -How verbose should the logs be? We log notifications at three levels: -INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of +How verbose should the logs be? We log notifications at four levels: +DEBUG, INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of notifications you would like to receive (in all-caps); more -interesting notifications will be logged as well. +interesting notifications will be logged as well. The debug output is +extremely verbose and will not be written to syslog even if you try. Default: INFO