]> gitweb.michael.orlitzky.com - dead/htsn-import.git/blob - doc/man1/htsn-import.1
Use UTC instead of EST to store the timestamps; they're both wrong anyway.
[dead/htsn-import.git] / doc / man1 / htsn-import.1
1 .TH htsn-import 1
2
3 .SH NAME
4 htsn-import \- Import XML files from The Sports Network into an RDBMS.
5
6 .SH SYNOPSIS
7
8 \fBhtsn-import\fR [OPTIONS] [FILES]
9
10 .SH DESCRIPTION
11 .P
12 The Sports Network <http://www.sportsnetwork.com/> offers an XML feed
13 containing various sports news and statistics. Our sister program
14 \fBhtsn\fR is capable of retrieving the feed and saving the individual
15 XML documents contained therein. But what to do with them?
16 .P
17 The purpose of \fBhtsn-import\fR is to take these XML documents and
18 get them into something we can use, a relational database management
19 system (RDBMS), otherwise known as a SQL database. The structure of
20 relational database, is, well, relational, and the feed XML is not. So
21 there is some work to do before the data can be imported into the
22 database.
23 .P
24 First, we must parse the XML. Each supported document type (see below)
25 has a full pickle/unpickle implementation (\(dqpickle\(dq is simply a
26 synonym for \(dqserialize\(dq here). That means that we parse the
27 entire document into a data structure, and if we pickle (serialize)
28 that data structure, we get the exact same XML document tha we started
29 with.
30 .P
31 This is important for two reasons. First, it serves as a second level
32 of validation. The first validation is performed by the XML parser,
33 but if that succeeds and unpicking fails, we know that something is
34 fishy. Second, we don't ever want to be surprised by some new element
35 or attribute showing up in the XML. The fact that we can unpickle the
36 whole thing now means that we won't be surprised in the future.
37 .P
38 The aforementioned feature is especially important because we
39 automatically migrate the database schema every time we import a
40 document. If you attempt to import a \(dqnewsxml.dtd\(dq document, all
41 database objects relating to the news will be created if they do not
42 exist. We don't want the schema to change out from under us without
43 warning, so it's important that no XML be parsed that would result in
44 a different schema than we had previously. Since we can
45 pickle/unpickle everything already, this should be impossible.
46 .P
47 A list of supported document types is given in the appendix.
48 .P
49 The GameInfo and SportInfo types do not have their own top-level
50 tables in the database. Instead, their raw XML is stored in either the
51 \(dqgame_info\(dq or \(dqsport_info\(dq table respectively.
52
53 .SH DATABASE SCHEMA
54 .P
55 At the top level (with two notable exceptions), we have one table for
56 each of the XML document types that we import. For example, the
57 documents corresponding to \fInewsxml.dtd\fR will have a table called
58 \(dqnews\(dq. All top-level tables contain two important fields,
59 \(dqxml_file_id\(dq and \(dqtime_stamp\(dq. The former is unique and
60 prevents us from inserting the same data twice. The time stamp on the
61 other hand lets us know when the data is old and can be removed. The
62 database schema make it possible to delete only the outdated top-level
63 records; all transient children should be removed by triggers.
64 .P
65 These top-level tables will often have children. For example, each
66 news item has zero or more locations associated with it. The child
67 table will be named <parent>_<children>, which in this case
68 corresponds to \(dqnews_locations\(dq.
69 .P
70 To relate the two, a third table may exist with name
71 <parent>__<child>. Note the two underscores. This prevents ambiguity
72 when the child table itself contains underscores. The table joining
73 \(dqnews\(dq with \(dqnews_locations\(dq is thus called
74 \(dqnews__news_locations\(dq. This is necessary when the child table
75 has a unique constraint; we don't want to blindly insert duplicate
76 records keyed to the parent. Instead we'd like to use the third table
77 to map an existing child to the new parent.
78 .P
79 Where it makes sense, children are kept unique to prevent pointless
80 duplication. This slows down inserts, and speeds up reads (which are
81 much more frequent). There is a tradeoff to be made, however. For a
82 table with a small, fixed upper bound on the number of rows (like
83 \(dqodds_casinos\(dq), there is great benefit to de-duplication. The
84 total number of rows stays small, so inserts are still quick, and many
85 duplicate rows are eliminated.
86 .P
87 But, with a table like \(dqodds_games\(dq, the number of games grows
88 quickly and without bound. It is therefore more beneficial to be able
89 to delete the old games (through an ON DELETE CASCADE, tied to
90 \(dqodds\(dq) than it is to eliminate duplication. A table like
91 \(dqnews_locations\(dq is somewhere in-between. It is hoped that the
92 unique constraint in the top-level table's \(dqxml_file_id\(dq will
93 prevent duplication in this case anyway.
94 .P
95 The aforementioned exceptions are the \(dqgame_info\(dq and
96 \(dqsport_info\(dq tables. These tables contain the raw XML for a
97 number of DTDs that are not handled individually. This is partially
98 for backwards-compatibility with a legacy implementation, but is
99 mostly a stopgap due to a lack of resources at the moment. These two
100 tables (game_info and sport_info) still possess timestamps that allow
101 us to prune old data.
102 .P
103 UML diagrams of the resulting database schema for each XML document
104 type are provided with the \fBhtsn-import\fR documentation, in the
105 \fIdoc/dbschema\fR directory. These are not authoritative, but it
106 should be considered a bug if they are incorrect. The diagrams are
107 created using the pgModeler <http://www.pgmodeler.com.br/> tool.
108
109 .SH DATABASE SCHEMA COMPROMISES
110
111 There are a few places that the database schema isn't exactly how we'd
112 like it to be:
113
114 .IP \[bu] 2
115 \fIearlylineXML.dtd\fR
116
117 The database representations for earlylineXML.dtd and
118 MLB_earlylineXML.dtd are the same; that is, they share the same
119 tables. The two document types represent team names in different
120 ways. In order to accomodate both types with one parser, we had to
121 make both ways optional, and then merge the two together before
122 converting to the database representation.
123
124 Unfortunately, when we merge two optional things together, we get
125 another optional thing back. There's no way to say that \(dqat least
126 one is not optional.\(dq So the team names in the database schema are
127 optional as well, even though they should always be present.
128
129 .SH NULL POLICY
130 .P
131 Normally in a database one makes a distinction between fields that
132 simply don't exist, and those fields that are
133 \(dqempty\(dq. Translating from XML, there is a natural way to
134 determine which one should be used: if an element is present in the
135 XML document but its contents are empty, then an empty string should
136 be inserted into the corresponding field. If on the other hand the
137 element is missing entirely, the corresponding database entry should
138 be NULL to indicate that fact.
139 .P
140 This sounds well and good, but the XML must be consistent for the
141 database consumer to make any sense of what he sees. The feed XML uses
142 optional and blank elements interchangeably, and without any
143 discernable pattern. To propagate this pattern into the database would
144 only cause confusion.
145 .P
146 As a result, a policy was adopted: both optional elements and elements
147 whose contents can be empty will be considered nullable in the
148 database. If the element is missing, the corresponding field is
149 NULL. Likewise if the content is simply missing. That means there
150 should never be a (completely) empty string in a database column.
151
152 .SH XML SCHEMA GENERATION
153 .P
154 In order to parse XML, you need to know the structure of your
155 documents. Usually this is given in the form of a DTD or schema. The
156 Sports Network does provide DTDs for their XML, but unfortunately many
157 of them do not match the XML found on the feed.
158 .P
159 We need to construct a database into which to insert the XML. How do
160 we know if <game> should be a column, or if it should have its own
161 table? We need to know how many times it can appear in the
162 document. So we need some form of specification. Since the supplied
163 DTDs are incorrect, we would like to generate them automatically.
164 .P
165 The process should go something like,
166 .IP 1.
167 Generate a DTD from the first foo.xml file we see. Call it foo.dtd.
168 .IP 2.
169 Validate future foo documents against foo.dtd. If they all validate,
170 great. If one fails, add it to the corpus and update foo.dtd so
171 that both the original and the new foo.xml validate.
172 .IP 3.
173 Repeat until no more failures occur. This can never be perfect:
174 tomorrow we could get a foo.xml that's wildly different from what
175 we've seen in the past. But it's the best we can hope for under
176 the circumstances.
177 .P
178 Enter XML-Schema-learner
179 <https://github.com/kore/XML-Schema-learner>. This tool can infer a
180 DTD from a set of sample XML files. The top-level \(dqschemagen\(dq
181 folder (in this project) contains a number of subfolders\(emone for
182 each type of document that we want to parse. Contained therein are XML
183 samples for that particular document type. These were hand-picked one
184 at a time according to the procedure above, and the complete set of
185 XML is what we use to generate the DTDs used by htsn-import.
186 .P
187 To generate them, run `make schema` at the project
188 root. XML-Schema-learner will be invoked on each subfolder of
189 \(dqschemagen\(dq and will output the corresponding DTDs to the
190 \(dqschemagen\(dq folder.
191 .P
192 Most of the production schemas are generated this way; however, a few
193 needed manual tweaking. The final, believed-to-be-correct schemas for
194 all supported document types can be found in the \(dqschema\(dq folder in
195 the project root. Having the correct DTDs available means you
196 don't need XML-Schema-learner available to install \fBhtsn-import\fR.
197
198 .SH XML SCHEMA UPDATES
199 .P
200 If a new tag is added to an XML document type, \fBhtsn-import\fR will
201 most likely refuse to parse it, since the new documents no longer
202 match the existing DTD.
203 .P
204 The first thing to do in that case is add the unparseable document to
205 the \(dqschemagen\(dq directory, and generate a new DTD that matches
206 both the old and new samples. Once a new, correct DTD has been
207 generated, it should be added to the \(dqschema\(dq directory. Then,
208 the parser can be updated and \fBhtsn-import\fR rebuilt.
209 .P
210 At this point, \fBhtsn-import\fR should be capable of importing the
211 new document. But the addition of the new tag will most require new
212 fields in the database. Fortunately, easy migrations like this are
213 handled automatically. As an example, at one point, \fIOdds_XML.dtd\fR
214 did not contain the \(dqHStarter\(dq and \(dqAStarter\(dq elements
215 associated with its games. Suppose we parse one of the old documents
216 (without \(dqHStarter\(dq and \(dqAStarter\(dq) using an old version
217 of \fBhtsn-import\fR:
218 .P
219 .nf
220 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
221 .I " schemagen/Odds_XML/19996433.xml"
222 Migration: CREATE TABLE \(dqodds\(dq ...
223 Successfully imported schemagen/Odds_XML/19996433.xml.
224 Processed 1 document(s) total.
225 .fi
226 .P
227 At this point, the database schema matches the old documents, that is,
228 the ones without \fIAStarter\fR and \fIHStarter\fR. If we use a new
229 version of \fBhtsn-import\fR, supporting the new fields, the migration
230 is handled gracefully:
231 .P
232 .nf
233 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
234 .I " schemagen/Odds_XML/21315768.xml"
235 Migration: ALTER TABLE \(dqodds_games\(dq
236 ADD COLUMN \(dqaway_team_starter_id\(dq INTEGER;
237 Migration: ALTER TABLE \(dqodds_games\(dq
238 ADD COLUMN \(dqaway_team_starter_name\(dq VARCHAR;
239 Migration: ALTER TABLE \(dqodds_games\(dq
240 ADD COLUMN \(dqhome_team_starter_id\(dq INTEGER;
241 Migration: ALTER TABLE \(dqodds_games\(dq
242 ADD COLUMN \(dqhome_team_starter_name\(dq VARCHAR;
243 Successfully imported schemagen/Odds_XML/21315768.xml.
244 Processed 1 document(s) total.
245 .fi
246 .P
247 If fields are removed from the schema, then manual intervention may be
248 necessary:
249 .P
250 .nf
251 .I $ htsn-import -b Postgres -c 'dbname=htsn user=postgres' \\\\
252 .I " schemagen/Odds_XML/19996433.xml"
253 ERROR: Database migration: manual intervention required.
254 The following actions are considered unsafe:
255 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqaway_team_starter_id\(dq
256 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqaway_team_starter_name\(dq
257 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqhome_team_starter_id\(dq
258 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqhome_team_starter_name\(dq
259
260 ERROR: Failed to import file schemagen/Odds_XML/19996433.xml.
261 Processed 0 document(s) total.
262 .fi
263 .P
264 To fix these errors, manually invoke the SQL commands that were
265 considered unsafe:
266 .P
267 .nf
268 .I $ psql -U postgres -d htsn \\\\
269 .I " -c 'ALTER TABLE odds_games DROP COLUMN away_team_starter_id;'"
270 ALTER TABLE
271 .I $ psql -U postgres -d htsn \\\\
272 .I " -c 'ALTER TABLE odds_games DROP COLUMN away_team_starter_name;'"
273 ALTER TABLE
274 .I $ psql -U postgres -d htsn \\\\
275 .I " -c 'ALTER TABLE odds_games DROP COLUMN home_team_starter_id;'"
276 ALTER TABLE
277 .I $ psql -U postgres -d htsn \\\\
278 .I " -c 'ALTER TABLE odds_games DROP COLUMN home_team_starter_name;'"
279 ALTER TABLE
280 .fi
281 .P
282 After manually adjusting the schema, the import should succeed.
283
284 .SH XML SCHEMA ODDITIES
285 .P
286 There are a number of problems with the XML on the wire. Even if we
287 construct the DTDs ourselves, the results are sometimes
288 inconsistent. Here we document a few of them.
289
290 .IP \[bu] 2
291 \fInewsxml.dtd\fR
292
293 The TSN DTD for news (and almost all XML on the wire) suggests that
294 there is a exactly one (possibly-empty) <SMS> element present in each
295 message. However, we have seen an example (XML_File_ID 21232353) where
296 an empty <SMS> followed a non-empty one:
297
298 .fi
299 <SMS>Odd Man Rush: Snow under pressure to improve Isles quickly</SMS>
300 <SMS></SMS>
301 .nf
302
303 We don't parse this case at the moment, but we do recognize it and report
304 it as unsupported so that offending documents can be removed. An example
305 is provided as test/xml/newsxml-multiple-sms.xml.
306
307 .IP \[bu]
308 \fIMLB_earlylineXML.dtd\fR
309
310 Unlike earlylineXML.dtd, this document type has more than one <game>
311 associated with each <date>. Moreover, each <date> has a bunch of
312 <note> children that are supposed to be associated with the <game>s,
313 but the document structure indicates no explicit relationship. For
314 example,
315
316 .nf
317 <date>
318 <note>...</note>
319 <game>...</game>
320 <game>...</game>
321 <note>...</note>
322 <game>...</game>
323 </date>
324 .fi
325
326 Here the first <note> is inferred to apply to the two <game>s that
327 follow it, and the second <note> applies to the single <game> that
328 follows it. But this is very fragile to parse. Instead, we use a hack
329 to facilitate (un)pickling, and then drop the notes entirely during
330 the database conversion.
331
332 A similar workaround is implemented for Odds_XML.dtd.
333
334 .IP \[bu]
335 \fIOdds_XML.dtd\fR
336
337 The <Notes> elements here are supposed to be associated with a set of
338 <Game> elements, but since the pair
339 (<Notes>...</Notes><Game>...</Game>) can appear zero or more times,
340 this leads to ambiguity in parsing. We therefore ignore the notes
341 entirely (although a hack is employed to facilitate parsing). The same
342 thing goes for the newer <League_Name> element.
343
344 .IP \[bu]
345 \fIweatherxml.dtd\fR
346
347 There appear to be two types of weather documents; the first has
348 <listing> contained within <forecast> and the second has <forecast>
349 contained within <listing>. While it would be possible to parse both,
350 it would greatly complicate things. The first form is more common, so
351 that's all we support for now. An example is provided as
352 test/xml/weatherxml-type2.xml.
353
354 We are however able to identify the second type. When one is
355 encountered, an informational message (that it is unsupported) will be
356 printed. If the \fI\-\-remove\fR flag is used, the file will be
357 deleted. This prevents documents that we know we can't import from
358 building up.
359
360 Another problem that comes up occasionally is that the home and away
361 team elements appear in the reverse order. As in the other case, we
362 report these as unsupported and then \(dqsucceed\(dq so that the
363 offending document can be removed if desired. An example is provided
364 as test/xml/weatherxml-backwards-teams.xml.
365
366 .SH DATE/TIME ISSUES
367
368 Dates and times appear in a number of places on the feed. The date
369 portions are usually, fine, but the times often lack important
370 information such as the time zone, or whether \(dq8 o'clock\(dq means
371 a.m. or p.m.
372
373 The most pervasive issue occurs with the timestamps that are included
374 in every message. A typical timestamp looks like,
375
376 .nf
377 <time_stamp> May 24, 2014, at 04:18 PM ET </time_stamp>
378 .fi
379
380 The \(dqtime zone\(dq is given as \(dqET\(dq, but unfortunately
381 \(dqET\(dq is not a valid time zone. It stands for \(dqEastern
382 Time\(dq, which can belong to either of two time zones, EST or EDT,
383 based on the time of the year (that is, whether or not daylight
384 savings time is in effect) and one's location (for example, Arizona
385 doesn't observe daylight savings time). It's not much more useful to
386 be off by one hour than it is to be off by five hours, and since we
387 can't determine the true offset from the timestamp, we always parse
388 and store these as UTC.
389
390 Here's a list of the ones that may cause surprises:
391
392 .IP \[bu] 2
393 \fIAutoRacingResultsXML.dtd\fR
394
395 The <RaceDate> elements contain a full date and time, but no time zone
396 information:
397
398 .nf
399 <RaceDate>5/24/2014 2:45:00 PM</RaceDate>
400 .fi
401
402 We parse them as UTC, which will be wrong when stored,
403 but \(dqcorrect\(dq if the new UTC time zone is ignored.
404
405 .IP \[bu]
406 \fIAuto_Racing_Schedule_XML.dtd\fR
407
408 The <Race_Date> and <Race_Time> elements are combined into on field in
409 the database, but no time zone information is given. For example,
410
411 .nf
412 <Race_Date>02/16/2013</Race_Date>
413 <Race_Time>08:10 PM</Race_Time>
414 .fi
415
416 As a result, we parse and store the times as UTC. The race times are
417 not always present in the database, but when they are missing, they
418 are presented as \(dqTBA\(dq (to be announced):
419
420 .nf
421 <Race_Time>TBA</Race_Time>
422 .fi
423
424 Since the dates do not appear to be optional, we store only the race
425 date in that case.
426
427 .IP \[bu]
428 \fIearlylineXML.dtd\fR
429
430 The <time> elements in the early lines contain neither a time zone nor
431 an am/pm identifier:
432
433 .nf
434 <time>8:30</time>
435 .fi
436
437 The times are parsed and stored as UTC, since we
438 don't have any other information upon which to base a guess. Even if
439 one ignores the UTC time zone, the time can possibly be off by 12
440 hours (due to the a.m./p.m. issue).
441
442 .IP \[bu]
443 \fIjfilexml.dtd\fR
444
445 The <Game_Date> and <Game_Time> elements are combined into on field in
446 the database, but no time zone information is given. For example,
447
448 .nf
449 <Game_Date>06/15/2014</Game_Date>
450 <Game_Time>08:00 PM</Game_Time>
451 .fi
452
453 As a result, we parse and store the times as UTC.
454
455 The <CurrentTimestamp> elements suffer a similar problem, sans the
456 date:
457
458 .nf
459 <CurrentTimeStamp>11:30 A.M.</CurrentTimeStamp>
460 .fi
461
462 They are also stored as UTC.
463
464 .IP \[bu]
465 \fIMLB_earlylineXML.dtd\fR
466
467 See earlylineXML.dtd.
468
469 .IP \[bu]
470 \fIOdds_XML.dtd\fR
471
472 The <Game_Date> and <Game_Time> elements are combined into on field in
473 the database, but no time zone information is given. For example,
474
475 .nf
476 <Game_Date>01/04/2014</Game_Date>
477 <Game_Time>04:35 PM</Game_Time>
478 .fi
479
480 As a result, we parse and store the times as UTC.
481
482 .IP \[bu]
483 \fISchedule_Changes_XML.dtd\fR
484
485 The <Game_Date> and <Game_Time> elements are combined into on field in
486 the database, but no time zone information is given. For example,
487
488 .nf
489 <Game_Date>06/06/2014</Game_Date>
490 <Game_Time>04:00 PM</Game_Time>
491 .fi
492
493 As a result, we parse and store the times as UTC. The game times are
494 not always present in the database, but when they are missing, they
495 are presented as \(dqTBA\(dq (to be announced):
496
497 .nf
498 <Game_Time>TBA</Game_Time>
499 .fi
500
501 Since the dates do not appear to be optional, we store only the game
502 date in that case.
503
504 .SH DEPLOYMENT
505 .P
506 When deploying for the first time, the target database will most
507 likely be empty. The schema will be migrated when a new document type
508 is seen, but this has a downside: it can be months before every
509 supported document type has been seen once. This can make it difficult
510 to test the database permissions.
511 .P
512 Since all of the test XML documents have old timestamps, one easy
513 workaround is the following: simply import all of the test XML
514 documents, and then delete them using whatever script is used to prune
515 old entries. This will force the migration of the schema, after which
516 you can set and test the database permissions.
517 .P
518 Something as simple as,
519 .P
520 .nf
521 .I $ find ./test/xml -iname '*.xml' | xargs htsn-import -c foo.sqlite
522 .fi
523 .P
524 should do it.
525
526 .SH OPTIONS
527
528 .IP \fB\-\-backend\fR,\ \fB\-b\fR
529 The RDBMS backend to use. Valid choices are \fISqlite\fR and
530 \fIPostgres\fR. Capitalization is important, sorry.
531
532 Default: Sqlite
533
534 .IP \fB\-\-connection-string\fR,\ \fB\-c\fR
535 The connection string used for connecting to the database backend
536 given by the \fB\-\-backend\fR option. The default is appropriate for
537 the \fISqlite\fR backend.
538
539 Default: \(dq:memory:\(dq
540
541 .IP \fB\-\-log-file\fR
542 If you specify a file here, logs will be written to it (possibly in
543 addition to syslog). Can be either a relative or absolute path. It
544 will not be auto-rotated; use something like logrotate for that.
545
546 Default: none
547
548 .IP \fB\-\-log-level\fR
549 How verbose should the logs be? We log notifications at four levels:
550 DEBUG, INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of
551 notifications you would like to receive (in all-caps); more
552 interesting notifications will be logged as well. The debug output is
553 extremely verbose and will not be written to syslog even if you try.
554
555 Default: INFO
556
557 .IP \fB\-\-remove\fR,\ \fB\-r\fR
558 Remove successfully processed files. If you enable this, you can see
559 at a glance which XML files are not being processed, because they're
560 all that should be left.
561
562 Default: disabled
563
564 .IP \fB\-\-syslog\fR,\ \fB\-s\fR
565 Enable logging to syslog. On Windows this will attempt to communicate
566 (over UDP) with a syslog daemon on localhost, which will most likely
567 not work.
568
569 Default: disabled
570
571 .SH CONFIGURATION FILE
572 .P
573 Any of the command-line options mentioned above can be specified in a
574 configuration file instead. We first look for \(dqhtsn-importrc\(dq in
575 the system configuration directory. We then look for a file named
576 \(dq.htsn-importrc\(dq in the user's home directory. The latter will
577 override the former.
578 .P
579 The user's home directory is simply $HOME on Unix; on Windows it's
580 wherever %APPDATA% points. The system configuration directory is
581 determined by Cabal; the \(dqsysconfdir\(dq parameter during the
582 \(dqconfigure\(dq step is used.
583 .P
584 The file's syntax is given by examples in the htsn-importrc.example file
585 (included with \fBhtsn-import\fR).
586 .P
587 Options specified on the command-line override those in either
588 configuration file.
589
590 .SH EXAMPLES
591 .IP \[bu] 2
592 Import newsxml.xml into a preexisting sqlite database named \(dqfoo.sqlite3\(dq:
593
594 .nf
595 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
596 .I " test/xml/newsxml.xml"
597 Successfully imported test/xml/newsxml.xml.
598 Imported 1 document(s) total.
599 .fi
600 .IP \[bu]
601 Repeat the previous example, but delete newsxml.xml afterwards:
602
603 .nf
604 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
605 .I " --remove test/xml/newsxml.xml"
606 Successfully imported test/xml/newsxml.xml.
607 Imported 1 document(s) total.
608 Removed processed file test/xml/newsxml.xml.
609 .fi
610 .IP \[bu]
611 Use a Postgres database instead of the default Sqlite. This assumes
612 that you have a database named \(dqhtsn\(dq accessible to user
613 \(dqpostgres\(dq locally:
614
615 .nf
616 .I $ htsn-import --connection-string='dbname=htsn user=postgres' \\\\
617 .I " --backend=Postgres test/xml/newsxml.xml"
618 Successfully imported test/xml/newsxml.xml.
619 Imported 1 document(s) total.
620 .fi
621
622 .SH BUGS
623
624 .P
625 Send bugs to michael@orlitzky.com.
626
627 .SH APPENDIX: SUPPORTED DOCUMENT TYPES
628 .P
629 The XML document types obtained from the feed are uniquely identified
630 by their DTDs. We currently support documents with the following DTDs:
631 .IP \[bu] 2
632 AutoRacingResultsXML.dtd
633 .IP \[bu]
634 Auto_Racing_Schedule_XML.dtd
635 .IP \[bu]
636 earlylineXML.dtd
637 .IP \[bu]
638 Heartbeat.dtd
639 .IP \[bu]
640 Injuries_Detail_XML.dtd
641 .IP \[bu]
642 injuriesxml.dtd
643 .IP \[bu]
644 jfilexml.dtd
645 .IP \[bu]
646 MLB_earlylineXML.dtd
647 .IP \[bu]
648 newsxml.dtd
649 .IP \[bu]
650 Odds_XML.dtd
651 .IP \[bu]
652 Schedule_Changes_XML.dtd
653 .IP \[bu]
654 scoresxml.dtd
655 .IP \[bu]
656 weatherxml.dtd
657 .IP \[bu]
658 GameInfo
659 .RS
660 .IP \[bu] 2
661 CBASK_Lineup_XML.dtd
662 .IP \[bu]
663 cbaskpreviewxml.dtd
664 .IP \[bu]
665 cflpreviewxml.dtd
666 .IP \[bu]
667 Matchup_NBA_NHL_XML.dtd
668 .IP \[bu]
669 MLB_Fielding_XML.dtd
670 .IP \[bu]
671 MLB_Gaming_Matchup_XML.dtd
672 .IP \[bu]
673 MLB_Lineup_XML.dtd
674 .IP \[bu]
675 MLB_Matchup_XML.dtd
676 .IP \[bu]
677 MLS_Preview_XML.dtd
678 .IP \[bu]
679 mlbpreviewxml.dtd
680 .IP \[bu]
681 NBA_Gaming_Matchup_XML.dtd
682 .IP \[bu]
683 NBA_Playoff_Matchup_XML.dtd
684 .IP \[bu]
685 NBALineupXML.dtd
686 .IP \[bu]
687 nbapreviewxml.dtd
688 .IP \[bu]
689 NCAA_FB_Preview_XML.dtd
690 .IP \[bu]
691 NFL_NCAA_FB_Matchup_XML.dtd
692 .IP \[bu]
693 nflpreviewxml.dtd
694 .IP \[bu]
695 nhlpreviewxml.dtd
696 .IP \[bu]
697 recapxml.dtd
698 .IP \[bu]
699 WorldBaseballPreviewXML.dtd
700 .RE
701 .IP \[bu]
702 SportInfo
703 .RS
704 .IP \[bu] 2
705 CBASK_3PPctXML.dtd
706 .IP \[bu]
707 Cbask_All_Tourn_Teams_XML.dtd
708 .IP \[bu]
709 CBASK_AssistsXML.dtd
710 .IP \[bu]
711 Cbask_Awards_XML.dtd
712 .IP \[bu]
713 CBASK_BlocksXML.dtd
714 .IP \[bu]
715 Cbask_Conf_Standings_XML.dtd
716 .IP \[bu]
717 Cbask_DivII_III_Indv_Stats_XML.dtd
718 .IP \[bu]
719 Cbask_DivII_Team_Stats_XML.dtd
720 .IP \[bu]
721 Cbask_DivIII_Team_Stats_XML.dtd
722 .IP \[bu]
723 CBASK_FGPctXML.dtd
724 .IP \[bu]
725 CBASK_FoulsXML.dtd
726 .IP \[bu]
727 CBASK_FTPctXML.dtd
728 .IP \[bu]
729 Cbask_Indv_Scoring_XML.dtd
730 .IP \[bu]
731 CBASK_MinutesXML.dtd
732 .IP \[bu]
733 Cbask_Polls_XML.dtd
734 .IP \[bu]
735 CBASK_ReboundsXML.dtd
736 .IP \[bu]
737 CBASK_ScoringLeadersXML.dtd
738 .IP \[bu]
739 Cbask_Team_ThreePT_Made_XML.dtd
740 .IP \[bu]
741 Cbask_Team_ThreePT_PCT_XML.dtd
742 .IP \[bu]
743 Cbask_Team_Win_Pct_XML.dtd
744 .IP \[bu]
745 Cbask_Top_Twenty_Five_XML.dtd
746 .IP \[bu]
747 CBASK_TopTwentyFiveResult_XML.dtd
748 .IP \[bu]
749 Cbask_Tourn_Awards_XML.dtd
750 .IP \[bu]
751 Cbask_Tourn_Champs_XML.dtd
752 .IP \[bu]
753 Cbask_Tourn_Indiv_XML.dtd
754 .IP \[bu]
755 Cbask_Tourn_Leaders_XML.dtd
756 .IP \[bu]
757 Cbask_Tourn_MVP_XML.dtd
758 .IP \[bu]
759 Cbask_Tourn_Records_XML.dtd
760 .IP \[bu]
761 LeagueScheduleXML.dtd
762 .IP \[bu]
763 minorscoresxml.dtd
764 .IP \[bu]
765 Minor_Baseball_League_Leaders_XML.dtd
766 .IP \[bu]
767 Minor_Baseball_Standings_XML.dtd
768 .IP \[bu]
769 Minor_Baseball_Transactions_XML.dtd
770 .IP \[bu]
771 mlbbattingavgxml.dtd
772 .IP \[bu]
773 mlbdoublesleadersxml.dtd
774 .IP \[bu]
775 MLBGamesPlayedXML.dtd
776 .IP \[bu]
777 MLBGIDPXML.dtd
778 .IP \[bu]
779 MLBHitByPitchXML.dtd
780 .IP \[bu]
781 mlbhitsleadersxml.dtd
782 .IP \[bu]
783 mlbhomerunsxml.dtd
784 .IP \[bu]
785 MLBHRFreqXML.dtd
786 .IP \[bu]
787 MLBIntWalksXML.dtd
788 .IP \[bu]
789 MLBKORateXML.dtd
790 .IP \[bu]
791 mlbonbasepctxml.dtd
792 .IP \[bu]
793 MLBOPSXML.dtd
794 .IP \[bu]
795 MLBPlateAppsXML.dtd
796 .IP \[bu]
797 mlbrbisxml.dtd
798 .IP \[bu]
799 mlbrunsleadersxml.dtd
800 .IP \[bu]
801 MLBSacFliesXML.dtd
802 .IP \[bu]
803 MLBSacrificesXML.dtd
804 .IP \[bu]
805 MLBSBSuccessXML.dtd
806 .IP \[bu]
807 mlbsluggingpctxml.dtd
808 .IP \[bu]
809 mlbstandxml.dtd
810 .IP \[bu]
811 mlbstandxml_preseason.dtd
812 .IP \[bu]
813 mlbstolenbasexml.dtd
814 .IP \[bu]
815 mlbtotalbasesleadersxml.dtd
816 .IP \[bu]
817 mlbtriplesleadersxml.dtd
818 .IP \[bu]
819 MLBWalkRateXML.dtd
820 .IP \[bu]
821 mlbwalksleadersxml.dtd
822 .IP \[bu]
823 MLBXtraBaseHitsXML.dtd
824 .IP \[bu]
825 MLB_Pitching_Appearances_Leaders.dtd
826 .IP \[bu]
827 MLB_ERA_Leaders.dtd
828 .IP \[bu]
829 MLB_Pitching_Balks_Leaders.dtd
830 .IP \[bu]
831 MLB_Pitching_CG_Leaders.dtd
832 .IP \[bu]
833 MLB_Pitching_ER_Allowed_Leaders.dtd
834 .IP \[bu]
835 MLB_Pitching_Hits_Allowed_Leaders.dtd
836 .IP \[bu]
837 MLB_Pitching_Hit_Batters_Leaders.dtd
838 .IP \[bu]
839 MLB_Pitching_HR_Allowed_Leaders.dtd
840 .IP \[bu]
841 MLB_Pitching_IP_Leaders.dtd
842 .IP \[bu]
843 MLB_Pitching_Runs_Allowed_Leaders.dtd
844 .IP \[bu]
845 MLB_Pitching_Saves_Leaders.dtd
846 .IP \[bu]
847 MLB_Pitching_Shut_Outs_Leaders.dtd
848 .IP \[bu]
849 MLB_Pitching_Starts_Leaders.dtd
850 .IP \[bu]
851 MLB_Pitching_Strike_Outs_Leaders.dtd
852 .IP \[bu]
853 MLB_Pitching_Walks_Leaders.dtd
854 .IP \[bu]
855 MLB_Pitching_WHIP_Leaders.dtd
856 .IP \[bu]
857 MLB_Pitching_Wild_Pitches_Leaders.dtd
858 .IP \[bu]
859 MLB_Pitching_Win_Percentage_Leaders.dtd
860 .IP \[bu]
861 MLB_Pitching_WL_Leaders.dtd
862 .IP \[bu]
863 NBA_Team_Stats_XML.dtd
864 .IP \[bu]
865 NBA3PPctXML.dtd
866 .IP \[bu]
867 NBAAssistsXML.dtd
868 .IP \[bu]
869 NBABlocksXML.dtd
870 .IP \[bu]
871 nbaconfrecxml.dtd
872 .IP \[bu]
873 nbadaysxml.dtd
874 .IP \[bu]
875 nbadivisionsxml.dtd
876 .IP \[bu]
877 NBAFGPctXML.dtd
878 .IP \[bu]
879 NBAFoulsXML.dtd
880 .IP \[bu]
881 NBAFTPctXML.dtd
882 .IP \[bu]
883 NBAMinutesXML.dtd
884 .IP \[bu]
885 NBAReboundsXML.dtd
886 .IP \[bu]
887 NBAScorersXML.dtd
888 .IP \[bu]
889 nbastandxml.dtd
890 .IP \[bu]
891 NBAStealsXML.dtd
892 .IP \[bu]
893 nbateamleadersxml.dtd
894 .IP \[bu]
895 nbatripledoublexml.dtd
896 .IP \[bu]
897 NBATurnoversXML.dtd
898 .IP \[bu]
899 NCAA_Conference_Schedule_XML.dtd
900 .IP \[bu]
901 nflfirstdownxml.dtd
902 .IP \[bu]
903 NFLFumbleLeaderXML.dtd
904 .IP \[bu]
905 NFLGiveTakeXML.dtd
906 .IP \[bu]
907 NFLInside20XML.dtd
908 .IP \[bu]
909 NFLKickoffsXML.dtd
910 .IP \[bu]
911 NFLMondayNightXML.dtd
912 .IP \[bu]
913 NFLPassLeadXML.dtd
914 .IP \[bu]
915 NFLQBStartsXML.dtd
916 .IP \[bu]
917 NFLSackLeadersXML.dtd
918 .IP \[bu]
919 nflstandxml.dtd
920 .IP \[bu]
921 NFLTeamRankingsXML.dtd
922 .IP \[bu]
923 NFLTopPerformanceXML.dtd
924 .IP \[bu]
925 NFLTotalYardageXML.dtd
926 .IP \[bu]
927 NFL_KickingLeaders_XML.dtd
928 .IP \[bu]
929 NFL_NBA_Draft_XML.dtd
930 .IP \[bu]
931 NFL_Roster_XML.dtd
932 .IP \[bu]
933 NFL_Team_Stats_XML.dtd
934 .IP \[bu]
935 Transactions_XML.dtd
936 .IP \[bu]
937 Weekly_Sched_XML.dtd
938 .IP \[bu]
939 WNBA_Team_Leaders_XML.dtd
940 .IP \[bu]
941 WNBA3PPctXML.dtd
942 .IP \[bu]
943 WNBAAssistsXML.dtd
944 .IP \[bu]
945 WNBABlocksXML.dtd
946 .IP \[bu]
947 WNBAFGPctXML.dtd
948 .IP \[bu]
949 WNBAFoulsXML.dtd
950 .IP \[bu]
951 WNBAFTPctXML.dtd
952 .IP \[bu]
953 WNBAMinutesXML.dtd
954 .IP \[bu]
955 WNBAReboundsXML.dtd
956 .IP \[bu]
957 WNBAScorersXML.dtd
958 .IP \[bu]
959 wnbastandxml.dtd
960 .IP \[bu]
961 WNBAStealsXML.dtd
962 .IP \[bu]
963 WNBATurnoversXML.dtd
964 .RE