X-Git-Url: http://gitweb.michael.orlitzky.com/?a=blobdiff_plain;f=doc%2Fman1%2Fhtsn-import.1;h=66f7ae5d1e769b23357f9fcdea9aaf1beb69cc83;hb=16d86e7a3c1eda08b91752f92510a1de0b952a17;hp=3dcf2bec9315cc44d692a48d5802812af610cda0;hpb=181833623b7924580a98089d12d36abd7330e175;p=dead%2Fhtsn-import.git diff --git a/doc/man1/htsn-import.1 b/doc/man1/htsn-import.1 index 3dcf2be..66f7ae5 100644 --- a/doc/man1/htsn-import.1 +++ b/doc/man1/htsn-import.1 @@ -106,6 +106,29 @@ type are provided with the \fBhtsn-import\fR documentation, in the should be considered a bug if they are incorrect. The diagrams are created using the pgModeler tool. +.SH NULL POLICY +.P +Normally in a database one makes a distinction between fields that +simply don't exist, and those fields that are +\(dqempty\(dq. Translating from XML, there is a natural way to +determine which one should be used: if an element is present in the +XML document but its contents are empty, then an empty string should +be inserted into the corresponding field. If on the other hand the +element is missing entirely, the corresponding database entry should +be NULL to indicate that fact. +.P +This sounds well and good, but the XML must be consistent for the +database consumer to make any sense of what he sees. The feed XML uses +optional and blank elements interchangeably, and without any +discernable pattern. To propagate this pattern into the database would +only cause confusion. +.P +As a result, a policy was adopted: both optional elements and elements +whose contents can be empty will be considered nullable in the +database. If the element is missing, the corresponding field is +NULL. Likewise if the content is simply missing. That means there +should never be a (completely) empty string in a database column. + .SH XML SCHEMA GENERATION .P In order to parse XML, you need to know the structure of your @@ -245,6 +268,23 @@ construct the DTDs ourselves, the results are sometimes inconsistent. Here we document a few of them. .IP \[bu] 2 +\fInewsxml.dtd\fR + +The TSN DTD for news (and almost all XML on the wire) suggests that +there is a exactly one (possibly-empty) element present in each +message. However, we have seen an example (XML_File_ID 21232353) where +an empty followed a non-empty one: + +.fi +Odd Man Rush: Snow under pressure to improve Isles quickly + +.nf + +We don't parse this case at the moment, but we do recognize it and report +it as unsupported so that offending documents can be removed. An example +is provided as test/xml/newsxml-multiple-sms.xml. + +.IP \[bu] \fIOdds_XML.dtd\fR The elements here are supposed to be associated with a set of @@ -262,7 +302,172 @@ There appear to be two types of weather documents; the first has contained within . While it would be possible to parse both, it would greatly complicate things. The first form is more common, so that's all we support for now. An example is provided as -schemagen/weatherxml/20143655.xml. +test/xml/weatherxml-type2.xml. + +We are however able to identify the second type. When one is +encountered, an informational message (that it is unsupported) will be +printed. If the \fI\-\-remove\fR flag is used, the file will be +deleted. This prevents documents that we know we can't import from +building up. + +Another problem that comes up occasionally is that the home and away +team elements appear in the reverse order. As in the other case, we +report these as unsupported and then \(dqsucceed\(dq so that the +offending document can be removed if desired. An example is provided +as test/xml/weatherxml-backwards-teams.xml. + +.SH DATE/TIME ISSUES + +Dates and times appear in a number of places on the feed. The date +portions are usually, fine, but the times often lack important +information such as the time zone, or whether \(dq8 o'clock\(dq means +a.m. or p.m. + +The most pervasive issue occurs with the timestamps that are included +in every message. A typical timestamp looks like, + +.nf + May 24, 2014, at 04:18 PM ET +.fi + +The \(dqtime zone\(dq is given as \(dqET\(dq, but unfortunately +\(dqET\(dq is not a valid time zone. It stands for \(dqEastern +Time\(dq, which can belong to either of two time zones, EST or EDT, +based on the time of the year (i.e. whether or not daylight savings +time is in effect). Since we can't tell from the timestamp, we always +parse these as EST which is UTC-5. When daylight savings is in effect, +they will be off by an hour. + +Here's a list of the ones that may cause surprises: + +.IP \[bu] 2 +\fIAutoRacingResultsXML.dtd\fR + +The elements contain a full date and time, but no time zone +information: + +.nf +5/24/2014 2:45:00 PM +.fi + +We parse them as UTC, which will be wrong when stored, +but \(dqcorrect\(dq if the new UTC time zone is ignored. + +.IP \[bu] +\fIAuto_Racing_Schedule_XML.dtd\fR + +The and elements are combined into on field in +the database, but no time zone information is given. For example, + +.nf +02/16/2013 +08:10 PM +.fi + +As a result, we parse and store the times as UTC. The race times are +not always present in the database, but when they are missing, they +are presented as \(dqTBA\(dq (to be announced): + +.nf +TBA +.fi + +Since the dates do not appear to be optional, we store only the race +date in that case. + +.IP \[bu] +\fIearlylineXML.dtd\fR + +The