]> gitweb.michael.orlitzky.com - dead/htsn-import.git/blob - doc/man1/htsn-import.1
f0ce47c668a91950113d6696b5912560fc36dd97
[dead/htsn-import.git] / doc / man1 / htsn-import.1
1 .TH htsn-import 1
2
3 .SH NAME
4 htsn-import \- Import XML files from The Sports Network into an RDBMS.
5
6 .SH SYNOPSIS
7
8 \fBhtsn-import\fR [OPTIONS] [FILES]
9
10 .SH DESCRIPTION
11 .P
12 The Sports Network <http://www.sportsnetwork.com/> offers an XML feed
13 containing various sports news and statistics. Our sister program
14 \fBhtsn\fR is capable of retrieving the feed and saving the individual
15 XML documents contained therein. But what to do with them?
16 .P
17 The purpose of \fBhtsn-import\fR is to take these XML documents and
18 get them into something we can use, a relational database management
19 system (RDBMS), otherwise known as a SQL database. The structure of
20 relational database, is, well, relational, and the feed XML is not. So
21 there is some work to do before the data can be imported into the
22 database.
23 .P
24 First, we must parse the XML. Each supported document type (see below)
25 has a full pickle/unpickle implementation (\(dqpickle\(dq is simply a
26 synonym for \(dqserialize\(dq here). That means that we parse the
27 entire document into a data structure, and if we pickle (serialize)
28 that data structure, we get the exact same XML document tha we started
29 with.
30 .P
31 This is important for two reasons. First, it serves as a second level
32 of validation. The first validation is performed by the XML parser,
33 but if that succeeds and unpicking fails, we know that something is
34 fishy. Second, we don't ever want to be surprised by some new element
35 or attribute showing up in the XML. The fact that we can unpickle the
36 whole thing now means that we won't be surprised in the future.
37 .P
38 The aforementioned feature is especially important because we
39 automatically migrate the database schema every time we import a
40 document. If you attempt to import a \(dqnewsxml.dtd\(dq document, all
41 database objects relating to the news will be created if they do not
42 exist. We don't want the schema to change out from under us without
43 warning, so it's important that no XML be parsed that would result in
44 a different schema than we had previously. Since we can
45 pickle/unpickle everything already, this should be impossible.
46 .P
47 A list of supported document types is given in the appendix.
48 .P
49 The GameInfo and SportInfo types do not have their own top-level
50 tables in the database. Instead, their raw XML is stored in either the
51 \(dqgame_info\(dq or \(dqsport_info\(dq table respectively.
52
53 .SH DATABASE SCHEMA
54 .P
55 At the top level (with two notable exceptions), we have one table for
56 each of the XML document types that we import. For example, the
57 documents corresponding to \fInewsxml.dtd\fR will have a table called
58 \(dqnews\(dq. All top-level tables contain two important fields,
59 \(dqxml_file_id\(dq and \(dqtime_stamp\(dq. The former is unique and
60 prevents us from inserting the same data twice. The time stamp on the
61 other hand lets us know when the data is old and can be removed. The
62 database schema make it possible to delete only the outdated top-level
63 records; all transient children should be removed by triggers.
64 .P
65 These top-level tables will often have children. For example, each
66 news item has zero or more locations associated with it. The child
67 table will be named <parent>_<children>, which in this case
68 corresponds to \(dqnews_locations\(dq.
69 .P
70 To relate the two, a third table may exist with name
71 <parent>__<child>. Note the two underscores. This prevents ambiguity
72 when the child table itself contains underscores. The table joining
73 \(dqnews\(dq with \(dqnews_locations\(dq is thus called
74 \(dqnews__news_locations\(dq. This is necessary when the child table
75 has a unique constraint; we don't want to blindly insert duplicate
76 records keyed to the parent. Instead we'd like to use the third table
77 to map an existing child to the new parent.
78 .P
79 Where it makes sense, children are kept unique to prevent pointless
80 duplication. This slows down inserts, and speeds up reads (which are
81 much more frequent). There is a tradeoff to be made, however. For a
82 table with a small, fixed upper bound on the number of rows (like
83 \(dqodds_casinos\(dq), there is great benefit to de-duplication. The
84 total number of rows stays small, so inserts are still quick, and many
85 duplicate rows are eliminated.
86 .P
87 But, with a table like \(dqodds_games\(dq, the number of games grows
88 quickly and without bound. It is therefore more beneficial to be able
89 to delete the old games (through an ON DELETE CASCADE, tied to
90 \(dqodds\(dq) than it is to eliminate duplication. A table like
91 \(dqnews_locations\(dq is somewhere in-between. It is hoped that the
92 unique constraint in the top-level table's \(dqxml_file_id\(dq will
93 prevent duplication in this case anyway.
94 .P
95 The aforementioned exceptions are the \(dqgame_info\(dq and
96 \(dqsport_info\(dq tables. These tables contain the raw XML for a
97 number of DTDs that are not handled individually. This is partially
98 for backwards-compatibility with a legacy implementation, but is
99 mostly a stopgap due to a lack of resources at the moment. These two
100 tables (game_info and sport_info) still possess timestamps that allow
101 us to prune old data.
102 .P
103 UML diagrams of the resulting database schema for each XML document
104 type are provided with the \fBhtsn-import\fR documentation, in the
105 \fIdoc/dbschema\fR directory. These are not authoritative, but it
106 should be considered a bug if they are incorrect. The diagrams are
107 created using the pgModeler <http://www.pgmodeler.com.br/> tool.
108
109 .SH DATABASE SCHEMA COMPROMISES
110
111 There are a few places that the database schema isn't exactly how we'd
112 like it to be:
113
114 .IP \[bu] 2
115 \fIearlylineXML.dtd\fR
116
117 The database representations for earlylineXML.dtd and
118 MLB_earlylineXML.dtd are the same; that is, they share the same
119 tables. The two document types represent team names in different
120 ways. In order to accomodate both types with one parser, we had to
121 make both ways optional, and then merge the two together before
122 converting to the database representation.
123
124 Unfortunately, when we merge two optional things together, we get
125 another optional thing back. There's no way to say that \(dqat least
126 one is not optional.\(dq So the team names in the database schema are
127 optional as well, even though they should always be present.
128
129 .SH NULL POLICY
130 .P
131 Normally in a database one makes a distinction between fields that
132 simply don't exist, and those fields that are
133 \(dqempty\(dq. Translating from XML, there is a natural way to
134 determine which one should be used: if an element is present in the
135 XML document but its contents are empty, then an empty string should
136 be inserted into the corresponding field. If on the other hand the
137 element is missing entirely, the corresponding database entry should
138 be NULL to indicate that fact.
139 .P
140 This sounds well and good, but the XML must be consistent for the
141 database consumer to make any sense of what he sees. The feed XML uses
142 optional and blank elements interchangeably, and without any
143 discernable pattern. To propagate this pattern into the database would
144 only cause confusion.
145 .P
146 As a result, a policy was adopted: both optional elements and elements
147 whose contents can be empty will be considered nullable in the
148 database. If the element is missing, the corresponding field is
149 NULL. Likewise if the content is simply missing. That means there
150 should never be a (completely) empty string in a database column.
151
152 .SH XML SCHEMA GENERATION
153 .P
154 In order to parse XML, you need to know the structure of your
155 documents. Usually this is given in the form of a DTD or schema. The
156 Sports Network does provide DTDs for their XML, but unfortunately many
157 of them do not match the XML found on the feed.
158 .P
159 We need to construct a database into which to insert the XML. How do
160 we know if <game> should be a column, or if it should have its own
161 table? We need to know how many times it can appear in the
162 document. So we need some form of specification. Since the supplied
163 DTDs are incorrect, we would like to generate them automatically.
164 .P
165 The process should go something like,
166 .IP 1.
167 Generate a DTD from the first foo.xml file we see. Call it foo.dtd.
168 .IP 2.
169 Validate future foo documents against foo.dtd. If they all validate,
170 great. If one fails, add it to the corpus and update foo.dtd so
171 that both the original and the new foo.xml validate.
172 .IP 3.
173 Repeat until no more failures occur. This can never be perfect:
174 tomorrow we could get a foo.xml that's wildly different from what
175 we've seen in the past. But it's the best we can hope for under
176 the circumstances.
177 .P
178 Enter XML-Schema-learner
179 <https://github.com/kore/XML-Schema-learner>. This tool can infer a
180 DTD from a set of sample XML files. The top-level \(dqschemagen\(dq
181 folder (in this project) contains a number of subfolders\(emone for
182 each type of document that we want to parse. Contained therein are XML
183 samples for that particular document type. These were hand-picked one
184 at a time according to the procedure above, and the complete set of
185 XML is what we use to generate the DTDs used by htsn-import.
186 .P
187 To generate them, run `make schema` at the project
188 root. XML-Schema-learner will be invoked on each subfolder of
189 \(dqschemagen\(dq and will output the corresponding DTDs to the
190 \(dqschemagen\(dq folder.
191 .P
192 Most of the production schemas are generated this way; however, a few
193 needed manual tweaking. The final, believed-to-be-correct schemas for
194 all supported document types can be found in the \(dqschema\(dq folder in
195 the project root. Having the correct DTDs available means you
196 don't need XML-Schema-learner available to install \fBhtsn-import\fR.
197
198 .SH XML SCHEMA UPDATES
199 .P
200 If a new tag is added to an XML document type, \fBhtsn-import\fR will
201 most likely refuse to parse it, since the new documents no longer
202 match the existing DTD.
203 .P
204 The first thing to do in that case is add the unparseable document to
205 the \(dqschemagen\(dq directory, and generate a new DTD that matches
206 both the old and new samples. Once a new, correct DTD has been
207 generated, it should be added to the \(dqschema\(dq directory. Then,
208 the parser can be updated and \fBhtsn-import\fR rebuilt.
209 .P
210 At this point, \fBhtsn-import\fR should be capable of importing the
211 new document. But the addition of the new tag will most require new
212 fields in the database. Fortunately, easy migrations like this are
213 handled automatically. As an example, at one point, \fIOdds_XML.dtd\fR
214 did not contain the \(dqHStarter\(dq and \(dqAStarter\(dq elements
215 associated with its games. Suppose we parse one of the old documents
216 (without \(dqHStarter\(dq and \(dqAStarter\(dq) using an old version
217 of \fBhtsn-import\fR:
218 .P
219 .nf
220 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
221 .I " schemagen/Odds_XML/19996433.xml"
222 Migration: CREATE TABLE \(dqodds\(dq ...
223 Successfully imported schemagen/Odds_XML/19996433.xml.
224 Processed 1 document(s) total.
225 .fi
226 .P
227 At this point, the database schema matches the old documents, that is,
228 the ones without \fIAStarter\fR and \fIHStarter\fR. If we use a new
229 version of \fBhtsn-import\fR, supporting the new fields, the migration
230 is handled gracefully:
231 .P
232 .nf
233 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
234 .I " schemagen/Odds_XML/21315768.xml"
235 Migration: ALTER TABLE \(dqodds_games\(dq
236 ADD COLUMN \(dqaway_team_starter_id\(dq INTEGER;
237 Migration: ALTER TABLE \(dqodds_games\(dq
238 ADD COLUMN \(dqaway_team_starter_name\(dq VARCHAR;
239 Migration: ALTER TABLE \(dqodds_games\(dq
240 ADD COLUMN \(dqhome_team_starter_id\(dq INTEGER;
241 Migration: ALTER TABLE \(dqodds_games\(dq
242 ADD COLUMN \(dqhome_team_starter_name\(dq VARCHAR;
243 Successfully imported schemagen/Odds_XML/21315768.xml.
244 Processed 1 document(s) total.
245 .fi
246 .P
247 If fields are removed from the schema, then manual intervention may be
248 necessary:
249 .P
250 .nf
251 .I $ htsn-import -b Postgres -c 'dbname=htsn user=postgres' \\\\
252 .I " schemagen/Odds_XML/19996433.xml"
253 ERROR: Database migration: manual intervention required.
254 The following actions are considered unsafe:
255 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqaway_team_starter_id\(dq
256 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqaway_team_starter_name\(dq
257 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqhome_team_starter_id\(dq
258 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqhome_team_starter_name\(dq
259
260 ERROR: Failed to import file schemagen/Odds_XML/19996433.xml.
261 Processed 0 document(s) total.
262 .fi
263 .P
264 To fix these errors, manually invoke the SQL commands that were
265 considered unsafe:
266 .P
267 .nf
268 .I $ psql -U postgres -d htsn \\\\
269 .I " -c 'ALTER TABLE odds_games DROP COLUMN away_team_starter_id;'"
270 ALTER TABLE
271 .I $ psql -U postgres -d htsn \\\\
272 .I " -c 'ALTER TABLE odds_games DROP COLUMN away_team_starter_name;'"
273 ALTER TABLE
274 .I $ psql -U postgres -d htsn \\\\
275 .I " -c 'ALTER TABLE odds_games DROP COLUMN home_team_starter_id;'"
276 ALTER TABLE
277 .I $ psql -U postgres -d htsn \\\\
278 .I " -c 'ALTER TABLE odds_games DROP COLUMN home_team_starter_name;'"
279 ALTER TABLE
280 .fi
281 .P
282 After manually adjusting the schema, the import should succeed.
283
284 .SH XML SCHEMA ODDITIES
285 .P
286 There are a number of problems with the XML on the wire. Even if we
287 construct the DTDs ourselves, the results are sometimes
288 inconsistent. Here we document a few of them.
289
290 .IP \[bu] 2
291 \fInewsxml.dtd\fR
292
293 The TSN DTD for news (and almost all XML on the wire) suggests that
294 there is a exactly one (possibly-empty) <SMS> element present in each
295 message. However, we have seen an example (XML_File_ID 21232353) where
296 an empty <SMS> followed a non-empty one:
297
298 .fi
299 <SMS>Odd Man Rush: Snow under pressure to improve Isles quickly</SMS>
300 <SMS></SMS>
301 .nf
302
303 We don't parse this case at the moment, but we do recognize it and report
304 it as unsupported so that offending documents can be removed. An example
305 is provided as test/xml/newsxml-multiple-sms.xml.
306
307 .IP \[bu]
308 \fIMLB_earlylineXML.dtd\fR
309
310 Unlike earlylineXML.dtd, this document type has more than one <game>
311 associated with each <date>. Moreover, each <date> has a bunch of
312 <note> children that are supposed to be associated with the <game>s,
313 but the document structure indicates no explicit relationship. For
314 example,
315
316 .nf
317 <date>
318 <note>...</note>
319 <game>...</game>
320 <game>...</game>
321 <note>...</note>
322 <game>...</game>
323 </date>
324 .fi
325
326 Here the first <note> is inferred to apply to the two <game>s that
327 follow it, and the second <note> applies to the single <game> that
328 follows it. But this is very fragile to parse. Instead, we use a hack
329 to facilitate (un)pickling, and then drop the notes entirely during
330 the database conversion.
331
332 A similar workaround is implemented for Odds_XML.dtd.
333
334 .IP \[bu]
335 \fIOdds_XML.dtd\fR
336
337 The <Notes> elements here are supposed to be associated with a set of
338 <Game> elements, but since the pair
339 (<Notes>...</Notes><Game>...</Game>) can appear zero or more times,
340 this leads to ambiguity in parsing. We therefore ignore the notes
341 entirely (although a hack is employed to facilitate parsing). The same
342 thing goes for the newer <League_Name> element.
343
344 .IP \[bu]
345 \fIweatherxml.dtd\fR
346
347 There appear to be two types of weather documents; the first has
348 <listing> contained within <forecast> and the second has <forecast>
349 contained within <listing>. While it would be possible to parse both,
350 it would greatly complicate things. The first form is more common, so
351 that's all we support for now. An example is provided as
352 test/xml/weatherxml-type2.xml.
353
354 We are however able to identify the second type. When one is
355 encountered, an informational message (that it is unsupported) will be
356 printed. If the \fI\-\-remove\fR flag is used, the file will be
357 deleted. This prevents documents that we know we can't import from
358 building up.
359
360 Another problem that comes up occasionally is that the home and away
361 team elements appear in the reverse order. As in the other case, we
362 report these as unsupported and then \(dqsucceed\(dq so that the
363 offending document can be removed if desired. An example is provided
364 as test/xml/weatherxml-backwards-teams.xml.
365
366 .SH DATE/TIME ISSUES
367
368 Dates and times appear in a number of places on the feed. The date
369 portions are usually, fine, but the times often lack important
370 information such as the time zone, or whether \(dq8 o'clock\(dq means
371 a.m. or p.m.
372
373 The most pervasive issue occurs with the timestamps that are included
374 in every message. A typical timestamp looks like,
375
376 .nf
377 <time_stamp> May 24, 2014, at 04:18 PM ET </time_stamp>
378 .fi
379
380 The \(dqtime zone\(dq is given as \(dqET\(dq, but unfortunately
381 \(dqET\(dq is not a valid time zone. It stands for \(dqEastern
382 Time\(dq, which can belong to either of two time zones, EST or EDT,
383 based on the time of the year (that is, whether or not daylight
384 savings time is in effect) and one's location (for example, Arizona
385 doesn't observe daylight savings time). It's not much more useful to
386 be off by one hour than it is to be off by five hours, and since we
387 can't determine the true offset from the timestamp, we always parse
388 and store these as UTC.
389
390 Here's a list of the ones that may cause surprises:
391
392 .IP \[bu] 2
393 \fIAutoRacingResultsXML.dtd\fR
394
395 The <RaceDate> elements contain a full date and time, but no time zone
396 information:
397
398 .nf
399 <RaceDate>5/24/2014 2:45:00 PM</RaceDate>
400 .fi
401
402 We parse them as UTC, which will be wrong when stored,
403 but \(dqcorrect\(dq if the new UTC time zone is ignored.
404
405 .IP \[bu]
406 \fIAuto_Racing_Schedule_XML.dtd\fR
407
408 The <Race_Date> and <Race_Time> elements are combined into on field in
409 the database, but no time zone information is given. For example,
410
411 .nf
412 <Race_Date>02/16/2013</Race_Date>
413 <Race_Time>08:10 PM</Race_Time>
414 .fi
415
416 As a result, we parse and store the times as UTC. The race times are
417 not always present in the database, but when they are missing, they
418 are presented as \(dqTBA\(dq (to be announced):
419
420 .nf
421 <Race_Time>TBA</Race_Time>
422 .fi
423
424 Since the dates do not appear to be optional, we store only the race
425 date in that case.
426
427 .IP \[bu]
428 \fIearlylineXML.dtd\fR
429
430 The <time> elements in the early lines contain neither a time zone nor
431 an am/pm identifier:
432
433 .nf
434 <time>8:30</time>
435 .fi
436
437 The times are parsed and stored as UTC, since we
438 don't have any other information upon which to base a guess. Even if
439 one ignores the UTC time zone, the time can possibly be off by 12
440 hours (due to the a.m./p.m. issue).
441
442 The game <time> elements can also be empty. Since we store the
443 combined game date/time in one field, these games will appear to begin
444 at midnight on the day they occur.
445
446 .IP \[bu]
447 \fIjfilexml.dtd\fR
448
449 The <Game_Date> and <Game_Time> elements are combined into on field in
450 the database, but no time zone information is given. For example,
451
452 .nf
453 <Game_Date>06/15/2014</Game_Date>
454 <Game_Time>08:00 PM</Game_Time>
455 .fi
456
457 As a result, we parse and store the times as UTC.
458
459 The <CurrentTimestamp> elements suffer a similar problem, sans the
460 date:
461
462 .nf
463 <CurrentTimeStamp>11:30 A.M.</CurrentTimeStamp>
464 .fi
465
466 They are also stored as UTC.
467
468 .IP \[bu]
469 \fIMLB_earlylineXML.dtd\fR
470
471 See earlylineXML.dtd.
472
473 .IP \[bu]
474 \fIOdds_XML.dtd\fR
475
476 The <Game_Date> and <Game_Time> elements are combined into on field in
477 the database, but no time zone information is given. For example,
478
479 .nf
480 <Game_Date>01/04/2014</Game_Date>
481 <Game_Time>04:35 PM</Game_Time>
482 .fi
483
484 As a result, we parse and store the times as UTC.
485
486 .IP \[bu]
487 \fISchedule_Changes_XML.dtd\fR
488
489 The <Game_Date> and <Game_Time> elements are combined into on field in
490 the database, but no time zone information is given. For example,
491
492 .nf
493 <Game_Date>06/06/2014</Game_Date>
494 <Game_Time>04:00 PM</Game_Time>
495 .fi
496
497 As a result, we parse and store the times as UTC. The game times are
498 not always present in the database, but when they are missing, they
499 are presented as \(dqTBA\(dq (to be announced):
500
501 .nf
502 <Game_Time>TBA</Game_Time>
503 .fi
504
505 Since the dates do not appear to be optional, we store only the game
506 date in that case.
507
508 .SH DEPLOYMENT
509 .P
510 When deploying for the first time, the target database will most
511 likely be empty. The schema will be migrated when a new document type
512 is seen, but this has a downside: it can be months before every
513 supported document type has been seen once. This can make it difficult
514 to test the database permissions.
515 .P
516 Since all of the test XML documents have old timestamps, one easy
517 workaround is the following: simply import all of the test XML
518 documents, and then delete them using whatever script is used to prune
519 old entries. This will force the migration of the schema, after which
520 you can set and test the database permissions.
521 .P
522 Something as simple as,
523 .P
524 .nf
525 .I $ find ./test/xml -iname '*.xml' | xargs htsn-import -c foo.sqlite
526 .fi
527 .P
528 should do it.
529
530 .SH OPTIONS
531
532 .IP \fB\-\-backend\fR,\ \fB\-b\fR
533 The RDBMS backend to use. Valid choices are \fISqlite\fR and
534 \fIPostgres\fR. Capitalization is important, sorry.
535
536 Default: Sqlite
537
538 .IP \fB\-\-connection-string\fR,\ \fB\-c\fR
539 The connection string used for connecting to the database backend
540 given by the \fB\-\-backend\fR option. The default is appropriate for
541 the \fISqlite\fR backend.
542
543 Default: \(dq:memory:\(dq
544
545 .IP \fB\-\-log-file\fR
546 If you specify a file here, logs will be written to it (possibly in
547 addition to syslog). Can be either a relative or absolute path. It
548 will not be auto-rotated; use something like logrotate for that.
549
550 Default: none
551
552 .IP \fB\-\-log-level\fR
553 How verbose should the logs be? We log notifications at four levels:
554 DEBUG, INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of
555 notifications you would like to receive (in all-caps); more
556 interesting notifications will be logged as well. The debug output is
557 extremely verbose and will not be written to syslog even if you try.
558
559 Default: INFO
560
561 .IP \fB\-\-remove\fR,\ \fB\-r\fR
562 Remove successfully processed files. If you enable this, you can see
563 at a glance which XML files are not being processed, because they're
564 all that should be left.
565
566 Default: disabled
567
568 .IP \fB\-\-syslog\fR,\ \fB\-s\fR
569 Enable logging to syslog. On Windows this will attempt to communicate
570 (over UDP) with a syslog daemon on localhost, which will most likely
571 not work.
572
573 Default: disabled
574
575 .SH CONFIGURATION FILE
576 .P
577 Any of the command-line options mentioned above can be specified in a
578 configuration file instead. We first look for \(dqhtsn-importrc\(dq in
579 the system configuration directory. We then look for a file named
580 \(dq.htsn-importrc\(dq in the user's home directory. The latter will
581 override the former.
582 .P
583 The user's home directory is simply $HOME on Unix; on Windows it's
584 wherever %APPDATA% points. The system configuration directory is
585 determined by Cabal; the \(dqsysconfdir\(dq parameter during the
586 \(dqconfigure\(dq step is used.
587 .P
588 The file's syntax is given by examples in the htsn-importrc.example file
589 (included with \fBhtsn-import\fR).
590 .P
591 Options specified on the command-line override those in either
592 configuration file.
593
594 .SH EXAMPLES
595 .IP \[bu] 2
596 Import newsxml.xml into a preexisting sqlite database named \(dqfoo.sqlite3\(dq:
597
598 .nf
599 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
600 .I " test/xml/newsxml.xml"
601 Successfully imported test/xml/newsxml.xml.
602 Imported 1 document(s) total.
603 .fi
604 .IP \[bu]
605 Repeat the previous example, but delete newsxml.xml afterwards:
606
607 .nf
608 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
609 .I " --remove test/xml/newsxml.xml"
610 Successfully imported test/xml/newsxml.xml.
611 Imported 1 document(s) total.
612 Removed processed file test/xml/newsxml.xml.
613 .fi
614 .IP \[bu]
615 Use a Postgres database instead of the default Sqlite. This assumes
616 that you have a database named \(dqhtsn\(dq accessible to user
617 \(dqpostgres\(dq locally:
618
619 .nf
620 .I $ htsn-import --connection-string='dbname=htsn user=postgres' \\\\
621 .I " --backend=Postgres test/xml/newsxml.xml"
622 Successfully imported test/xml/newsxml.xml.
623 Imported 1 document(s) total.
624 .fi
625
626 .SH BUGS
627
628 .P
629 Send bugs to michael@orlitzky.com.
630
631 .SH APPENDIX: SUPPORTED DOCUMENT TYPES
632 .P
633 The XML document types obtained from the feed are uniquely identified
634 by their DTDs. We currently support documents with the following DTDs:
635 .IP \[bu] 2
636 AutoRacingResultsXML.dtd
637 .IP \[bu]
638 Auto_Racing_Schedule_XML.dtd
639 .IP \[bu]
640 earlylineXML.dtd
641 .IP \[bu]
642 Heartbeat.dtd
643 .IP \[bu]
644 Injuries_Detail_XML.dtd
645 .IP \[bu]
646 injuriesxml.dtd
647 .IP \[bu]
648 jfilexml.dtd
649 .IP \[bu]
650 MLB_earlylineXML.dtd
651 .IP \[bu]
652 newsxml.dtd
653 .IP \[bu]
654 Odds_XML.dtd
655 .IP \[bu]
656 Schedule_Changes_XML.dtd
657 .IP \[bu]
658 scoresxml.dtd
659 .IP \[bu]
660 weatherxml.dtd
661 .IP \[bu]
662 GameInfo
663 .RS
664 .IP \[bu] 2
665 CBASK_Lineup_XML.dtd
666 .IP \[bu]
667 cbaskpreviewxml.dtd
668 .IP \[bu]
669 cflpreviewxml.dtd
670 .IP \[bu]
671 Matchup_NBA_NHL_XML.dtd
672 .IP \[bu]
673 MLB_Fielding_XML.dtd
674 .IP \[bu]
675 MLB_Gaming_Matchup_XML.dtd
676 .IP \[bu]
677 MLB_Lineup_XML.dtd
678 .IP \[bu]
679 MLB_Matchup_XML.dtd
680 .IP \[bu]
681 MLS_Preview_XML.dtd
682 .IP \[bu]
683 mlbpreviewxml.dtd
684 .IP \[bu]
685 NBA_Gaming_Matchup_XML.dtd
686 .IP \[bu]
687 NBA_Playoff_Matchup_XML.dtd
688 .IP \[bu]
689 NBALineupXML.dtd
690 .IP \[bu]
691 nbapreviewxml.dtd
692 .IP \[bu]
693 NCAA_FB_Preview_XML.dtd
694 .IP \[bu]
695 NFL_NCAA_FB_Matchup_XML.dtd
696 .IP \[bu]
697 nflpreviewxml.dtd
698 .IP \[bu]
699 nhlpreviewxml.dtd
700 .IP \[bu]
701 recapxml.dtd
702 .IP \[bu]
703 WorldBaseballPreviewXML.dtd
704 .RE
705 .IP \[bu]
706 SportInfo
707 .RS
708 .IP \[bu] 2
709 CBASK_3PPctXML.dtd
710 .IP \[bu]
711 Cbask_All_Tourn_Teams_XML.dtd
712 .IP \[bu]
713 CBASK_AssistsXML.dtd
714 .IP \[bu]
715 Cbask_Awards_XML.dtd
716 .IP \[bu]
717 CBASK_BlocksXML.dtd
718 .IP \[bu]
719 Cbask_Conf_Standings_XML.dtd
720 .IP \[bu]
721 Cbask_DivII_III_Indv_Stats_XML.dtd
722 .IP \[bu]
723 Cbask_DivII_Team_Stats_XML.dtd
724 .IP \[bu]
725 Cbask_DivIII_Team_Stats_XML.dtd
726 .IP \[bu]
727 CBASK_FGPctXML.dtd
728 .IP \[bu]
729 CBASK_FoulsXML.dtd
730 .IP \[bu]
731 CBASK_FTPctXML.dtd
732 .IP \[bu]
733 Cbask_Indv_No_Avg_XML
734 .IP \[bu]
735 Cbask_Indv_Scoring_XML.dtd
736 .IP \[bu]
737 CBASK_MinutesXML.dtd
738 .IP \[bu]
739 Cbask_Polls_XML.dtd
740 .IP \[bu]
741 CBASK_ReboundsXML.dtd
742 .IP \[bu]
743 CBASK_ScoringLeadersXML.dtd
744 .IP \[bu]
745 Cbask_Team_ThreePT_Made_XML.dtd
746 .IP \[bu]
747 Cbask_Team_ThreePT_PCT_XML.dtd
748 .IP \[bu]
749 Cbask_Team_Win_Pct_XML.dtd
750 .IP \[bu]
751 Cbask_Top_Twenty_Five_XML.dtd
752 .IP \[bu]
753 CBASK_TopTwentyFiveResult_XML.dtd
754 .IP \[bu]
755 Cbask_Tourn_Awards_XML.dtd
756 .IP \[bu]
757 Cbask_Tourn_Champs_XML.dtd
758 .IP \[bu]
759 Cbask_Tourn_Indiv_XML.dtd
760 .IP \[bu]
761 Cbask_Tourn_Leaders_XML.dtd
762 .IP \[bu]
763 Cbask_Tourn_MVP_XML.dtd
764 .IP \[bu]
765 Cbask_Tourn_Records_XML.dtd
766 .IP \[bu]
767 LeagueScheduleXML.dtd
768 .IP \[bu]
769 minorscoresxml.dtd
770 .IP \[bu]
771 Minor_Baseball_League_Leaders_XML.dtd
772 .IP \[bu]
773 Minor_Baseball_Standings_XML.dtd
774 .IP \[bu]
775 Minor_Baseball_Transactions_XML.dtd
776 .IP \[bu]
777 mlbbattingavgxml.dtd
778 .IP \[bu]
779 mlbdoublesleadersxml.dtd
780 .IP \[bu]
781 MLBGamesPlayedXML.dtd
782 .IP \[bu]
783 MLBGIDPXML.dtd
784 .IP \[bu]
785 MLBHitByPitchXML.dtd
786 .IP \[bu]
787 mlbhitsleadersxml.dtd
788 .IP \[bu]
789 mlbhomerunsxml.dtd
790 .IP \[bu]
791 MLBHRFreqXML.dtd
792 .IP \[bu]
793 MLBIntWalksXML.dtd
794 .IP \[bu]
795 MLBKORateXML.dtd
796 .IP \[bu]
797 mlbonbasepctxml.dtd
798 .IP \[bu]
799 MLBOPSXML.dtd
800 .IP \[bu]
801 MLBPlateAppsXML.dtd
802 .IP \[bu]
803 mlbrbisxml.dtd
804 .IP \[bu]
805 mlbrunsleadersxml.dtd
806 .IP \[bu]
807 MLBSacFliesXML.dtd
808 .IP \[bu]
809 MLBSacrificesXML.dtd
810 .IP \[bu]
811 MLBSBSuccessXML.dtd
812 .IP \[bu]
813 mlbsluggingpctxml.dtd
814 .IP \[bu]
815 mlbstandxml.dtd
816 .IP \[bu]
817 mlbstandxml_preseason.dtd
818 .IP \[bu]
819 mlbstolenbasexml.dtd
820 .IP \[bu]
821 mlbtotalbasesleadersxml.dtd
822 .IP \[bu]
823 mlbtriplesleadersxml.dtd
824 .IP \[bu]
825 MLBWalkRateXML.dtd
826 .IP \[bu]
827 mlbwalksleadersxml.dtd
828 .IP \[bu]
829 MLBXtraBaseHitsXML.dtd
830 .IP \[bu]
831 MLB_Pitching_Appearances_Leaders.dtd
832 .IP \[bu]
833 MLB_ERA_Leaders.dtd
834 .IP \[bu]
835 MLB_Pitching_Balks_Leaders.dtd
836 .IP \[bu]
837 MLB_Pitching_CG_Leaders.dtd
838 .IP \[bu]
839 MLB_Pitching_ER_Allowed_Leaders.dtd
840 .IP \[bu]
841 MLB_Pitching_Hits_Allowed_Leaders.dtd
842 .IP \[bu]
843 MLB_Pitching_Hit_Batters_Leaders.dtd
844 .IP \[bu]
845 MLB_Pitching_HR_Allowed_Leaders.dtd
846 .IP \[bu]
847 MLB_Pitching_IP_Leaders.dtd
848 .IP \[bu]
849 MLB_Pitching_Runs_Allowed_Leaders.dtd
850 .IP \[bu]
851 MLB_Pitching_Saves_Leaders.dtd
852 .IP \[bu]
853 MLB_Pitching_Shut_Outs_Leaders.dtd
854 .IP \[bu]
855 MLB_Pitching_Starts_Leaders.dtd
856 .IP \[bu]
857 MLB_Pitching_Strike_Outs_Leaders.dtd
858 .IP \[bu]
859 MLB_Pitching_Walks_Leaders.dtd
860 .IP \[bu]
861 MLB_Pitching_WHIP_Leaders.dtd
862 .IP \[bu]
863 MLB_Pitching_Wild_Pitches_Leaders.dtd
864 .IP \[bu]
865 MLB_Pitching_Win_Percentage_Leaders.dtd
866 .IP \[bu]
867 MLB_Pitching_WL_Leaders.dtd
868 .IP \[bu]
869 NBA_Team_Stats_XML.dtd
870 .IP \[bu]
871 NBA3PPctXML.dtd
872 .IP \[bu]
873 NBAAssistsXML.dtd
874 .IP \[bu]
875 NBABlocksXML.dtd
876 .IP \[bu]
877 nbaconfrecxml.dtd
878 .IP \[bu]
879 nbadaysxml.dtd
880 .IP \[bu]
881 nbadivisionsxml.dtd
882 .IP \[bu]
883 NBAFGPctXML.dtd
884 .IP \[bu]
885 NBAFoulsXML.dtd
886 .IP \[bu]
887 NBAFTPctXML.dtd
888 .IP \[bu]
889 NBAMinutesXML.dtd
890 .IP \[bu]
891 NBAReboundsXML.dtd
892 .IP \[bu]
893 NBAScorersXML.dtd
894 .IP \[bu]
895 nbastandxml.dtd
896 .IP \[bu]
897 NBAStealsXML.dtd
898 .IP \[bu]
899 nbateamleadersxml.dtd
900 .IP \[bu]
901 nbatripledoublexml.dtd
902 .IP \[bu]
903 NBATurnoversXML.dtd
904 .IP \[bu]
905 NCAA_Conference_Schedule_XML.dtd
906 .IP \[bu]
907 nflfirstdownxml.dtd
908 .IP \[bu]
909 NFLFumbleLeaderXML.dtd
910 .IP \[bu]
911 NFLGiveTakeXML.dtd
912 .IP \[bu]
913 NFLGrassTurfDomeOutsideXML.dtd
914 .IP \[bu]
915 NFLInside20XML.dtd
916 .IP \[bu]
917 NFLInterceptionLeadersXML.dtd
918 .IP \[bu]
919 NFLKickoffsXML.dtd
920 .IP \[bu]
921 NFLMondayNightXML.dtd
922 .IP \[bu]
923 NFLPassingLeadersXML.dtd
924 .IP \[bu]
925 NFLPassLeadXML.dtd
926 .IP \[bu]
927 NFLQBStartsXML.dtd
928 .IP \[bu]
929 NFLReceivingLeadersXML.dtd
930 .IP \[bu]
931 NFLRushingLeadersXML.dtd
932 .IP \[bu]
933 NFLSackLeadersXML.dtd
934 .IP \[bu]
935 nflstandxml.dtd
936 .IP \[bu]
937 NFLTackleFFLeadersXML.dtd
938 .IP \[bu]
939 NFLTeamRankingsXML.dtd
940 .IP \[bu]
941 NFLTopKickoffReturnXML.dtd
942 .IP \[bu]
943 NFLTopPerformanceXML.dtd
944 .IP \[bu]
945 NFLTopPuntReturnXML.dtd
946 .IP \[bu]
947 NFLTotalYardageXML.dtd
948 .IP \[bu]
949 NFLYardsXML.dtd
950 .IP \[bu]
951 NFL_KickingLeaders_XML.dtd
952 .IP \[bu]
953 NFL_NBA_Draft_XML.dtd
954 .IP \[bu]
955 NFL_PuntingLeaders_XML.dtd
956 .IP \[bu]
957 NFL_Roster_XML.dtd
958 .IP \[bu]
959 NFL_Team_Stats_XML.dtd
960 .IP \[bu]
961 Transactions_XML.dtd
962 .IP \[bu]
963 Weekly_Sched_XML.dtd
964 .IP \[bu]
965 WNBA_Team_Leaders_XML.dtd
966 .IP \[bu]
967 WNBA3PPctXML.dtd
968 .IP \[bu]
969 WNBAAssistsXML.dtd
970 .IP \[bu]
971 WNBABlocksXML.dtd
972 .IP \[bu]
973 WNBAFGPctXML.dtd
974 .IP \[bu]
975 WNBAFoulsXML.dtd
976 .IP \[bu]
977 WNBAFTPctXML.dtd
978 .IP \[bu]
979 WNBAMinutesXML.dtd
980 .IP \[bu]
981 WNBAReboundsXML.dtd
982 .IP \[bu]
983 WNBAScorersXML.dtd
984 .IP \[bu]
985 wnbastandxml.dtd
986 .IP \[bu]
987 WNBAStealsXML.dtd
988 .IP \[bu]
989 WNBATurnoversXML.dtd
990 .RE