]> gitweb.michael.orlitzky.com - dead/htsn-import.git/blob - doc/man1/htsn-import.1
Add a new module, TSN.XML.MLBEarlyLines supporting MLB_earlylinesXML.dtd.
[dead/htsn-import.git] / doc / man1 / htsn-import.1
1 .TH htsn-import 1
2
3 .SH NAME
4 htsn-import \- Import XML files from The Sports Network into an RDBMS.
5
6 .SH SYNOPSIS
7
8 \fBhtsn-import\fR [OPTIONS] [FILES]
9
10 .SH DESCRIPTION
11 .P
12 The Sports Network <http://www.sportsnetwork.com/> offers an XML feed
13 containing various sports news and statistics. Our sister program
14 \fBhtsn\fR is capable of retrieving the feed and saving the individual
15 XML documents contained therein. But what to do with them?
16 .P
17 The purpose of \fBhtsn-import\fR is to take these XML documents and
18 get them into something we can use, a relational database management
19 system (RDBMS), otherwise known as a SQL database. The structure of
20 relational database, is, well, relational, and the feed XML is not. So
21 there is some work to do before the data can be imported into the
22 database.
23 .P
24 First, we must parse the XML. Each supported document type (see below)
25 has a full pickle/unpickle implementation (\(dqpickle\(dq is simply a
26 synonym for \(dqserialize\(dq here). That means that we parse the
27 entire document into a data structure, and if we pickle (serialize)
28 that data structure, we get the exact same XML document tha we started
29 with.
30 .P
31 This is important for two reasons. First, it serves as a second level
32 of validation. The first validation is performed by the XML parser,
33 but if that succeeds and unpicking fails, we know that something is
34 fishy. Second, we don't ever want to be surprised by some new element
35 or attribute showing up in the XML. The fact that we can unpickle the
36 whole thing now means that we won't be surprised in the future.
37 .P
38 The aforementioned feature is especially important because we
39 automatically migrate the database schema every time we import a
40 document. If you attempt to import a \(dqnewsxml.dtd\(dq document, all
41 database objects relating to the news will be created if they do not
42 exist. We don't want the schema to change out from under us without
43 warning, so it's important that no XML be parsed that would result in
44 a different schema than we had previously. Since we can
45 pickle/unpickle everything already, this should be impossible.
46 .P
47 A list of supported document types is given in the appendix.
48 .P
49 The GameInfo and SportInfo types do not have their own top-level
50 tables in the database. Instead, their raw XML is stored in either the
51 \(dqgame_info\(dq or \(dqsport_info\(dq table respectively.
52
53 .SH DATABASE SCHEMA
54 .P
55 At the top level (with two notable exceptions), we have one table for
56 each of the XML document types that we import. For example, the
57 documents corresponding to \fInewsxml.dtd\fR will have a table called
58 \(dqnews\(dq. All top-level tables contain two important fields,
59 \(dqxml_file_id\(dq and \(dqtime_stamp\(dq. The former is unique and
60 prevents us from inserting the same data twice. The time stamp on the
61 other hand lets us know when the data is old and can be removed. The
62 database schema make it possible to delete only the outdated top-level
63 records; all transient children should be removed by triggers.
64 .P
65 These top-level tables will often have children. For example, each
66 news item has zero or more locations associated with it. The child
67 table will be named <parent>_<children>, which in this case
68 corresponds to \(dqnews_locations\(dq.
69 .P
70 To relate the two, a third table may exist with name
71 <parent>__<child>. Note the two underscores. This prevents ambiguity
72 when the child table itself contains underscores. The table joining
73 \(dqnews\(dq with \(dqnews_locations\(dq is thus called
74 \(dqnews__news_locations\(dq. This is necessary when the child table
75 has a unique constraint; we don't want to blindly insert duplicate
76 records keyed to the parent. Instead we'd like to use the third table
77 to map an existing child to the new parent.
78 .P
79 Where it makes sense, children are kept unique to prevent pointless
80 duplication. This slows down inserts, and speeds up reads (which are
81 much more frequent). There is a tradeoff to be made, however. For a
82 table with a small, fixed upper bound on the number of rows (like
83 \(dqodds_casinos\(dq), there is great benefit to de-duplication. The
84 total number of rows stays small, so inserts are still quick, and many
85 duplicate rows are eliminated.
86 .P
87 But, with a table like \(dqodds_games\(dq, the number of games grows
88 quickly and without bound. It is therefore more beneficial to be able
89 to delete the old games (through an ON DELETE CASCADE, tied to
90 \(dqodds\(dq) than it is to eliminate duplication. A table like
91 \(dqnews_locations\(dq is somewhere in-between. It is hoped that the
92 unique constraint in the top-level table's \(dqxml_file_id\(dq will
93 prevent duplication in this case anyway.
94 .P
95 The aforementioned exceptions are the \(dqgame_info\(dq and
96 \(dqsport_info\(dq tables. These tables contain the raw XML for a
97 number of DTDs that are not handled individually. This is partially
98 for backwards-compatibility with a legacy implementation, but is
99 mostly a stopgap due to a lack of resources at the moment. These two
100 tables (game_info and sport_info) still possess timestamps that allow
101 us to prune old data.
102 .P
103 UML diagrams of the resulting database schema for each XML document
104 type are provided with the \fBhtsn-import\fR documentation, in the
105 \fIdoc/dbschema\fR directory. These are not authoritative, but it
106 should be considered a bug if they are incorrect. The diagrams are
107 created using the pgModeler <http://www.pgmodeler.com.br/> tool.
108
109 .SH DATABASE SCHEMA COMPROMISES
110
111 There are a few places that the database schema isn't exactly how we'd
112 like it to be:
113
114 .IP \[bu] 2
115 \fIearlylineXML.dtd\fR
116
117 The database representations for earlylineXML.dtd and
118 MLB_earlylineXML.dtd are the same; that is, they share the same
119 tables. The two document types represent team names in different
120 ways. In order to accomodate both types with one parser, we had to
121 make both ways optional, and then merge the two together before
122 converting to the database representation.
123
124 Unfortunately, when we merge two optional things together, we get
125 another optional thing back. There's no way to say that \(dqat least
126 one is not optional.\(dq So the team names in the database schema are
127 optional as well, even though they should always be present.
128
129 .SH NULL POLICY
130 .P
131 Normally in a database one makes a distinction between fields that
132 simply don't exist, and those fields that are
133 \(dqempty\(dq. Translating from XML, there is a natural way to
134 determine which one should be used: if an element is present in the
135 XML document but its contents are empty, then an empty string should
136 be inserted into the corresponding field. If on the other hand the
137 element is missing entirely, the corresponding database entry should
138 be NULL to indicate that fact.
139 .P
140 This sounds well and good, but the XML must be consistent for the
141 database consumer to make any sense of what he sees. The feed XML uses
142 optional and blank elements interchangeably, and without any
143 discernable pattern. To propagate this pattern into the database would
144 only cause confusion.
145 .P
146 As a result, a policy was adopted: both optional elements and elements
147 whose contents can be empty will be considered nullable in the
148 database. If the element is missing, the corresponding field is
149 NULL. Likewise if the content is simply missing. That means there
150 should never be a (completely) empty string in a database column.
151
152 .SH XML SCHEMA GENERATION
153 .P
154 In order to parse XML, you need to know the structure of your
155 documents. Usually this is given in the form of a DTD or schema. The
156 Sports Network does provide DTDs for their XML, but unfortunately many
157 of them do not match the XML found on the feed.
158 .P
159 We need to construct a database into which to insert the XML. How do
160 we know if <game> should be a column, or if it should have its own
161 table? We need to know how many times it can appear in the
162 document. So we need some form of specification. Since the supplied
163 DTDs are incorrect, we would like to generate them automatically.
164 .P
165 The process should go something like,
166 .IP 1.
167 Generate a DTD from the first foo.xml file we see. Call it foo.dtd.
168 .IP 2.
169 Validate future foo documents against foo.dtd. If they all validate,
170 great. If one fails, add it to the corpus and update foo.dtd so
171 that both the original and the new foo.xml validate.
172 .IP 3.
173 Repeat until no more failures occur. This can never be perfect:
174 tomorrow we could get a foo.xml that's wildly different from what
175 we've seen in the past. But it's the best we can hope for under
176 the circumstances.
177 .P
178 Enter XML-Schema-learner
179 <https://github.com/kore/XML-Schema-learner>. This tool can infer a
180 DTD from a set of sample XML files. The top-level \(dqschemagen\(dq
181 folder (in this project) contains a number of subfolders\(emone for
182 each type of document that we want to parse. Contained therein are XML
183 samples for that particular document type. These were hand-picked one
184 at a time according to the procedure above, and the complete set of
185 XML is what we use to generate the DTDs used by htsn-import.
186 .P
187 To generate them, run `make schema` at the project
188 root. XML-Schema-learner will be invoked on each subfolder of
189 \(dqschemagen\(dq and will output the corresponding DTDs to the
190 \(dqschemagen\(dq folder.
191 .P
192 Most of the production schemas are generated this way; however, a few
193 needed manual tweaking. The final, believed-to-be-correct schemas for
194 all supported document types can be found in the \(dqschema\(dq folder in
195 the project root. Having the correct DTDs available means you
196 don't need XML-Schema-learner available to install \fBhtsn-import\fR.
197
198 .SH XML SCHEMA UPDATES
199 .P
200 If a new tag is added to an XML document type, \fBhtsn-import\fR will
201 most likely refuse to parse it, since the new documents no longer
202 match the existing DTD.
203 .P
204 The first thing to do in that case is add the unparseable document to
205 the \(dqschemagen\(dq directory, and generate a new DTD that matches
206 both the old and new samples. Once a new, correct DTD has been
207 generated, it should be added to the \(dqschema\(dq directory. Then,
208 the parser can be updated and \fBhtsn-import\fR rebuilt.
209 .P
210 At this point, \fBhtsn-import\fR should be capable of importing the
211 new document. But the addition of the new tag will most require new
212 fields in the database. Fortunately, easy migrations like this are
213 handled automatically. As an example, at one point, \fIOdds_XML.dtd\fR
214 did not contain the \(dqHStarter\(dq and \(dqAStarter\(dq elements
215 associated with its games. Suppose we parse one of the old documents
216 (without \(dqHStarter\(dq and \(dqAStarter\(dq) using an old version
217 of \fBhtsn-import\fR:
218 .P
219 .nf
220 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
221 .I " schemagen/Odds_XML/19996433.xml"
222 Migration: CREATE TABLE \(dqodds\(dq ...
223 Successfully imported schemagen/Odds_XML/19996433.xml.
224 Processed 1 document(s) total.
225 .fi
226 .P
227 At this point, the database schema matches the old documents, that is,
228 the ones without \fIAStarter\fR and \fIHStarter\fR. If we use a new
229 version of \fBhtsn-import\fR, supporting the new fields, the migration
230 is handled gracefully:
231 .P
232 .nf
233 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
234 .I " schemagen/Odds_XML/21315768.xml"
235 Migration: ALTER TABLE \(dqodds_games\(dq
236 ADD COLUMN \(dqaway_team_starter_id\(dq INTEGER;
237 Migration: ALTER TABLE \(dqodds_games\(dq
238 ADD COLUMN \(dqaway_team_starter_name\(dq VARCHAR;
239 Migration: ALTER TABLE \(dqodds_games\(dq
240 ADD COLUMN \(dqhome_team_starter_id\(dq INTEGER;
241 Migration: ALTER TABLE \(dqodds_games\(dq
242 ADD COLUMN \(dqhome_team_starter_name\(dq VARCHAR;
243 Successfully imported schemagen/Odds_XML/21315768.xml.
244 Processed 1 document(s) total.
245 .fi
246 .P
247 If fields are removed from the schema, then manual intervention may be
248 necessary:
249 .P
250 .nf
251 .I $ htsn-import -b Postgres -c 'dbname=htsn user=postgres' \\\\
252 .I " schemagen/Odds_XML/19996433.xml"
253 ERROR: Database migration: manual intervention required.
254 The following actions are considered unsafe:
255 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqaway_team_starter_id\(dq
256 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqaway_team_starter_name\(dq
257 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqhome_team_starter_id\(dq
258 ALTER TABLE \(dqodds_games\(dq DROP COLUMN \(dqhome_team_starter_name\(dq
259
260 ERROR: Failed to import file schemagen/Odds_XML/19996433.xml.
261 Processed 0 document(s) total.
262 .fi
263 .P
264 To fix these errors, manually invoke the SQL commands that were
265 considered unsafe:
266 .P
267 .nf
268 .I $ psql -U postgres -d htsn \\\\
269 .I " -c 'ALTER TABLE odds_games DROP COLUMN away_team_starter_id;'"
270 ALTER TABLE
271 .I $ psql -U postgres -d htsn \\\\
272 .I " -c 'ALTER TABLE odds_games DROP COLUMN away_team_starter_name;'"
273 ALTER TABLE
274 .I $ psql -U postgres -d htsn \\\\
275 .I " -c 'ALTER TABLE odds_games DROP COLUMN home_team_starter_id;'"
276 ALTER TABLE
277 .I $ psql -U postgres -d htsn \\\\
278 .I " -c 'ALTER TABLE odds_games DROP COLUMN home_team_starter_name;'"
279 ALTER TABLE
280 .fi
281 .P
282 After manually adjusting the schema, the import should succeed.
283
284 .SH XML SCHEMA ODDITIES
285 .P
286 There are a number of problems with the XML on the wire. Even if we
287 construct the DTDs ourselves, the results are sometimes
288 inconsistent. Here we document a few of them.
289
290 .IP \[bu] 2
291 \fInewsxml.dtd\fR
292
293 The TSN DTD for news (and almost all XML on the wire) suggests that
294 there is a exactly one (possibly-empty) <SMS> element present in each
295 message. However, we have seen an example (XML_File_ID 21232353) where
296 an empty <SMS> followed a non-empty one:
297
298 .fi
299 <SMS>Odd Man Rush: Snow under pressure to improve Isles quickly</SMS>
300 <SMS></SMS>
301 .nf
302
303 We don't parse this case at the moment, but we do recognize it and report
304 it as unsupported so that offending documents can be removed. An example
305 is provided as test/xml/newsxml-multiple-sms.xml.
306
307 .IP \[bu]
308 \fIMLB_earlylineXML.dtd\fR
309
310 Unlike earlylineXML.dtd, this document type has more than one <game>
311 associated with each <date>. Moreover, each <date> has a bunch of
312 <note> children that are supposed to be associated with the <game>s,
313 but the document structure indicates no explicit relationship. For
314 example,
315
316 .nf
317 <date>
318 <note>...</note>
319 <game>...</game>
320 <game>...</game>
321 <note>...</note>
322 <game>...</game>
323 </date>
324 .fi
325
326 Here the first <note> is inferred to apply to the two <game>s that
327 follow it, and the second <note> applies to the single <game> that
328 follows it. But this is very fragile to parse. Instead, we use a hack
329 to facilitate (un)pickling, and then drop the notes entirely during
330 the database conversion.
331
332 A similar workaround is implemented for Odds_XML.dtd.
333
334 .IP \[bu]
335 \fIOdds_XML.dtd\fR
336
337 The <Notes> elements here are supposed to be associated with a set of
338 <Game> elements, but since the pair
339 (<Notes>...</Notes><Game>...</Game>) can appear zero or more times,
340 this leads to ambiguity in parsing. We therefore ignore the notes
341 entirely (although a hack is employed to facilitate parsing). The same
342 thing goes for the newer <League_Name> element.
343
344 .IP \[bu]
345 \fIweatherxml.dtd\fR
346
347 There appear to be two types of weather documents; the first has
348 <listing> contained within <forecast> and the second has <forecast>
349 contained within <listing>. While it would be possible to parse both,
350 it would greatly complicate things. The first form is more common, so
351 that's all we support for now. An example is provided as
352 test/xml/weatherxml-type2.xml.
353
354 We are however able to identify the second type. When one is
355 encountered, an informational message (that it is unsupported) will be
356 printed. If the \fI\-\-remove\fR flag is used, the file will be
357 deleted. This prevents documents that we know we can't import from
358 building up.
359
360 Another problem that comes up occasionally is that the home and away
361 team elements appear in the reverse order. As in the other case, we
362 report these as unsupported and then \(dqsucceed\(dq so that the
363 offending document can be removed if desired. An example is provided
364 as test/xml/weatherxml-backwards-teams.xml.
365
366 .SH DATE/TIME ISSUES
367
368 Dates and times appear in a number of places on the feed. The date
369 portions are usually, fine, but the times often lack important
370 information such as the time zone, or whether \(dq8 o'clock\(dq means
371 a.m. or p.m.
372
373 The most pervasive issue occurs with the timestamps that are included
374 in every message. A typical timestamp looks like,
375
376 .nf
377 <time_stamp> May 24, 2014, at 04:18 PM ET </time_stamp>
378 .fi
379
380 The \(dqtime zone\(dq is given as \(dqET\(dq, but unfortunately
381 \(dqET\(dq is not a valid time zone. It stands for \(dqEastern
382 Time\(dq, which can belong to either of two time zones, EST or EDT,
383 based on the time of the year (that is, whether or not daylight
384 savings time is in effect). Since we can't tell from the timestamp, we
385 always parse these as EST which is UTC-5. When daylight savings is in
386 effect, they will be off by an hour.
387
388 Here's a list of the ones that may cause surprises:
389
390 .IP \[bu] 2
391 \fIAutoRacingResultsXML.dtd\fR
392
393 The <RaceDate> elements contain a full date and time, but no time zone
394 information:
395
396 .nf
397 <RaceDate>5/24/2014 2:45:00 PM</RaceDate>
398 .fi
399
400 We parse them as UTC, which will be wrong when stored,
401 but \(dqcorrect\(dq if the new UTC time zone is ignored.
402
403 .IP \[bu]
404 \fIAuto_Racing_Schedule_XML.dtd\fR
405
406 The <Race_Date> and <Race_Time> elements are combined into on field in
407 the database, but no time zone information is given. For example,
408
409 .nf
410 <Race_Date>02/16/2013</Race_Date>
411 <Race_Time>08:10 PM</Race_Time>
412 .fi
413
414 As a result, we parse and store the times as UTC. The race times are
415 not always present in the database, but when they are missing, they
416 are presented as \(dqTBA\(dq (to be announced):
417
418 .nf
419 <Race_Time>TBA</Race_Time>
420 .fi
421
422 Since the dates do not appear to be optional, we store only the race
423 date in that case.
424
425 .IP \[bu]
426 \fIearlylineXML.dtd\fR
427
428 The <time> elements in the early lines contain neither a time zone nor
429 an am/pm identifier:
430
431 .nf
432 <time>8:30</time>
433 .fi
434
435 The times are parsed and stored as UTC, since we
436 don't have any other information upon which to base a guess. Even if
437 one ignores the UTC time zone, the time can possibly be off by 12
438 hours (due to the a.m./p.m. issue).
439
440 .IP \[bu]
441 \fIjfilexml.dtd\fR
442
443 The <Game_Date> and <Game_Time> elements are combined into on field in
444 the database, but no time zone information is given. For example,
445
446 .nf
447 <Game_Date>06/15/2014</Game_Date>
448 <Game_Time>08:00 PM</Game_Time>
449 .fi
450
451 As a result, we parse and store the times as UTC.
452
453 The <CurrentTimestamp> elements suffer a similar problem, sans the
454 date:
455
456 .nf
457 <CurrentTimeStamp>11:30 A.M.</CurrentTimeStamp>
458 .fi
459
460 They are also stored as UTC.
461
462 .IP \[bu]
463 \fIMLB_earlylineXML.dtd\fR
464
465 See earlylineXML.dtd.
466
467 .IP \[bu]
468 \fIOdds_XML.dtd\fR
469
470 The <Game_Date> and <Game_Time> elements are combined into on field in
471 the database, but no time zone information is given. For example,
472
473 .nf
474 <Game_Date>01/04/2014</Game_Date>
475 <Game_Time>04:35 PM</Game_Time>
476 .fi
477
478 As a result, we parse and store the times as UTC.
479
480 .IP \[bu]
481 \fISchedule_Changes_XML.dtd\fR
482
483 The <Game_Date> and <Game_Time> elements are combined into on field in
484 the database, but no time zone information is given. For example,
485
486 .nf
487 <Game_Date>06/06/2014</Game_Date>
488 <Game_Time>04:00 PM</Game_Time>
489 .fi
490
491 As a result, we parse and store the times as UTC. The game times are
492 not always present in the database, but when they are missing, they
493 are presented as \(dqTBA\(dq (to be announced):
494
495 .nf
496 <Game_Time>TBA</Game_Time>
497 .fi
498
499 Since the dates do not appear to be optional, we store only the game
500 date in that case.
501
502 .SH DEPLOYMENT
503 .P
504 When deploying for the first time, the target database will most
505 likely be empty. The schema will be migrated when a new document type
506 is seen, but this has a downside: it can be months before every
507 supported document type has been seen once. This can make it difficult
508 to test the database permissions.
509 .P
510 Since all of the test XML documents have old timestamps, one easy
511 workaround is the following: simply import all of the test XML
512 documents, and then delete them using whatever script is used to prune
513 old entries. This will force the migration of the schema, after which
514 you can set and test the database permissions.
515 .P
516 Something as simple as,
517 .P
518 .nf
519 .I $ find ./test/xml -iname '*.xml' | xargs htsn-import -c foo.sqlite
520 .fi
521 .P
522 should do it.
523
524 .SH OPTIONS
525
526 .IP \fB\-\-backend\fR,\ \fB\-b\fR
527 The RDBMS backend to use. Valid choices are \fISqlite\fR and
528 \fIPostgres\fR. Capitalization is important, sorry.
529
530 Default: Sqlite
531
532 .IP \fB\-\-connection-string\fR,\ \fB\-c\fR
533 The connection string used for connecting to the database backend
534 given by the \fB\-\-backend\fR option. The default is appropriate for
535 the \fISqlite\fR backend.
536
537 Default: \(dq:memory:\(dq
538
539 .IP \fB\-\-log-file\fR
540 If you specify a file here, logs will be written to it (possibly in
541 addition to syslog). Can be either a relative or absolute path. It
542 will not be auto-rotated; use something like logrotate for that.
543
544 Default: none
545
546 .IP \fB\-\-log-level\fR
547 How verbose should the logs be? We log notifications at four levels:
548 DEBUG, INFO, WARN, and ERROR. Specify the \(dqmost boring\(dq level of
549 notifications you would like to receive (in all-caps); more
550 interesting notifications will be logged as well. The debug output is
551 extremely verbose and will not be written to syslog even if you try.
552
553 Default: INFO
554
555 .IP \fB\-\-remove\fR,\ \fB\-r\fR
556 Remove successfully processed files. If you enable this, you can see
557 at a glance which XML files are not being processed, because they're
558 all that should be left.
559
560 Default: disabled
561
562 .IP \fB\-\-syslog\fR,\ \fB\-s\fR
563 Enable logging to syslog. On Windows this will attempt to communicate
564 (over UDP) with a syslog daemon on localhost, which will most likely
565 not work.
566
567 Default: disabled
568
569 .SH CONFIGURATION FILE
570 .P
571 Any of the command-line options mentioned above can be specified in a
572 configuration file instead. We first look for \(dqhtsn-importrc\(dq in
573 the system configuration directory. We then look for a file named
574 \(dq.htsn-importrc\(dq in the user's home directory. The latter will
575 override the former.
576 .P
577 The user's home directory is simply $HOME on Unix; on Windows it's
578 wherever %APPDATA% points. The system configuration directory is
579 determined by Cabal; the \(dqsysconfdir\(dq parameter during the
580 \(dqconfigure\(dq step is used.
581 .P
582 The file's syntax is given by examples in the htsn-importrc.example file
583 (included with \fBhtsn-import\fR).
584 .P
585 Options specified on the command-line override those in either
586 configuration file.
587
588 .SH EXAMPLES
589 .IP \[bu] 2
590 Import newsxml.xml into a preexisting sqlite database named \(dqfoo.sqlite3\(dq:
591
592 .nf
593 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
594 .I " test/xml/newsxml.xml"
595 Successfully imported test/xml/newsxml.xml.
596 Imported 1 document(s) total.
597 .fi
598 .IP \[bu]
599 Repeat the previous example, but delete newsxml.xml afterwards:
600
601 .nf
602 .I $ htsn-import --connection-string='foo.sqlite3' \\\\
603 .I " --remove test/xml/newsxml.xml"
604 Successfully imported test/xml/newsxml.xml.
605 Imported 1 document(s) total.
606 Removed processed file test/xml/newsxml.xml.
607 .fi
608 .IP \[bu]
609 Use a Postgres database instead of the default Sqlite. This assumes
610 that you have a database named \(dqhtsn\(dq accessible to user
611 \(dqpostgres\(dq locally:
612
613 .nf
614 .I $ htsn-import --connection-string='dbname=htsn user=postgres' \\\\
615 .I " --backend=Postgres test/xml/newsxml.xml"
616 Successfully imported test/xml/newsxml.xml.
617 Imported 1 document(s) total.
618 .fi
619
620 .SH BUGS
621
622 .P
623 Send bugs to michael@orlitzky.com.
624
625 .SH APPENDIX: SUPPORTED DOCUMENT TYPES
626 .P
627 The XML document types obtained from the feed are uniquely identified
628 by their DTDs. We currently support documents with the following DTDs:
629 .IP \[bu] 2
630 AutoRacingResultsXML.dtd
631 .IP \[bu]
632 Auto_Racing_Schedule_XML.dtd
633 .IP \[bu]
634 earlylineXML.dtd
635 .IP \[bu]
636 Heartbeat.dtd
637 .IP \[bu]
638 Injuries_Detail_XML.dtd
639 .IP \[bu]
640 injuriesxml.dtd
641 .IP \[bu]
642 jfilexml.dtd
643 .IP \[bu]
644 MLB_earlylineXML.dtd
645 .IP \[bu]
646 newsxml.dtd
647 .IP \[bu]
648 Odds_XML.dtd
649 .IP \[bu]
650 Schedule_Changes_XML.dtd
651 .IP \[bu]
652 scoresxml.dtd
653 .IP \[bu]
654 weatherxml.dtd
655 .IP \[bu]
656 GameInfo
657 .RS
658 .IP \[bu] 2
659 CBASK_Lineup_XML.dtd
660 .IP \[bu]
661 cbaskpreviewxml.dtd
662 .IP \[bu]
663 cflpreviewxml.dtd
664 .IP \[bu]
665 Matchup_NBA_NHL_XML.dtd
666 .IP \[bu]
667 MLB_Fielding_XML.dtd
668 .IP \[bu]
669 MLB_Gaming_Matchup_XML.dtd
670 .IP \[bu]
671 MLB_Lineup_XML.dtd
672 .IP \[bu]
673 MLB_Matchup_XML.dtd
674 .IP \[bu]
675 MLS_Preview_XML.dtd
676 .IP \[bu]
677 mlbpreviewxml.dtd
678 .IP \[bu]
679 NBA_Gaming_Matchup_XML.dtd
680 .IP \[bu]
681 NBA_Playoff_Matchup_XML.dtd
682 .IP \[bu]
683 NBALineupXML.dtd
684 .IP \[bu]
685 nbapreviewxml.dtd
686 .IP \[bu]
687 NCAA_FB_Preview_XML.dtd
688 .IP \[bu]
689 NFL_NCAA_FB_Matchup_XML.dtd
690 .IP \[bu]
691 nflpreviewxml.dtd
692 .IP \[bu]
693 nhlpreviewxml.dtd
694 .IP \[bu]
695 recapxml.dtd
696 .IP \[bu]
697 WorldBaseballPreviewXML.dtd
698 .RE
699 .IP \[bu]
700 SportInfo
701 .RS
702 .IP \[bu] 2
703 CBASK_3PPctXML.dtd
704 .IP \[bu]
705 Cbask_All_Tourn_Teams_XML.dtd
706 .IP \[bu]
707 CBASK_AssistsXML.dtd
708 .IP \[bu]
709 Cbask_Awards_XML.dtd
710 .IP \[bu]
711 CBASK_BlocksXML.dtd
712 .IP \[bu]
713 Cbask_Conf_Standings_XML.dtd
714 .IP \[bu]
715 Cbask_DivII_III_Indv_Stats_XML.dtd
716 .IP \[bu]
717 Cbask_DivII_Team_Stats_XML.dtd
718 .IP \[bu]
719 Cbask_DivIII_Team_Stats_XML.dtd
720 .IP \[bu]
721 CBASK_FGPctXML.dtd
722 .IP \[bu]
723 CBASK_FoulsXML.dtd
724 .IP \[bu]
725 CBASK_FTPctXML.dtd
726 .IP \[bu]
727 Cbask_Indv_Scoring_XML.dtd
728 .IP \[bu]
729 CBASK_MinutesXML.dtd
730 .IP \[bu]
731 Cbask_Polls_XML.dtd
732 .IP \[bu]
733 CBASK_ReboundsXML.dtd
734 .IP \[bu]
735 CBASK_ScoringLeadersXML.dtd
736 .IP \[bu]
737 Cbask_Team_ThreePT_Made_XML.dtd
738 .IP \[bu]
739 Cbask_Team_ThreePT_PCT_XML.dtd
740 .IP \[bu]
741 Cbask_Team_Win_Pct_XML.dtd
742 .IP \[bu]
743 Cbask_Top_Twenty_Five_XML.dtd
744 .IP \[bu]
745 CBASK_TopTwentyFiveResult_XML.dtd
746 .IP \[bu]
747 Cbask_Tourn_Awards_XML.dtd
748 .IP \[bu]
749 Cbask_Tourn_Champs_XML.dtd
750 .IP \[bu]
751 Cbask_Tourn_Indiv_XML.dtd
752 .IP \[bu]
753 Cbask_Tourn_Leaders_XML.dtd
754 .IP \[bu]
755 Cbask_Tourn_MVP_XML.dtd
756 .IP \[bu]
757 Cbask_Tourn_Records_XML.dtd
758 .IP \[bu]
759 LeagueScheduleXML.dtd
760 .IP \[bu]
761 minorscoresxml.dtd
762 .IP \[bu]
763 Minor_Baseball_League_Leaders_XML.dtd
764 .IP \[bu]
765 Minor_Baseball_Standings_XML.dtd
766 .IP \[bu]
767 Minor_Baseball_Transactions_XML.dtd
768 .IP \[bu]
769 mlbbattingavgxml.dtd
770 .IP \[bu]
771 mlbdoublesleadersxml.dtd
772 .IP \[bu]
773 MLBGamesPlayedXML.dtd
774 .IP \[bu]
775 MLBGIDPXML.dtd
776 .IP \[bu]
777 MLBHitByPitchXML.dtd
778 .IP \[bu]
779 mlbhitsleadersxml.dtd
780 .IP \[bu]
781 mlbhomerunsxml.dtd
782 .IP \[bu]
783 MLBHRFreqXML.dtd
784 .IP \[bu]
785 MLBIntWalksXML.dtd
786 .IP \[bu]
787 MLBKORateXML.dtd
788 .IP \[bu]
789 mlbonbasepctxml.dtd
790 .IP \[bu]
791 MLBOPSXML.dtd
792 .IP \[bu]
793 MLBPlateAppsXML.dtd
794 .IP \[bu]
795 mlbrbisxml.dtd
796 .IP \[bu]
797 mlbrunsleadersxml.dtd
798 .IP \[bu]
799 MLBSacFliesXML.dtd
800 .IP \[bu]
801 MLBSacrificesXML.dtd
802 .IP \[bu]
803 MLBSBSuccessXML.dtd
804 .IP \[bu]
805 mlbsluggingpctxml.dtd
806 .IP \[bu]
807 mlbstandxml.dtd
808 .IP \[bu]
809 mlbstandxml_preseason.dtd
810 .IP \[bu]
811 mlbstolenbasexml.dtd
812 .IP \[bu]
813 mlbtotalbasesleadersxml.dtd
814 .IP \[bu]
815 mlbtriplesleadersxml.dtd
816 .IP \[bu]
817 MLBWalkRateXML.dtd
818 .IP \[bu]
819 mlbwalksleadersxml.dtd
820 .IP \[bu]
821 MLBXtraBaseHitsXML.dtd
822 .IP \[bu]
823 MLB_Pitching_Appearances_Leaders.dtd
824 .IP \[bu]
825 MLB_ERA_Leaders.dtd
826 .IP \[bu]
827 MLB_Pitching_Balks_Leaders.dtd
828 .IP \[bu]
829 MLB_Pitching_CG_Leaders.dtd
830 .IP \[bu]
831 MLB_Pitching_ER_Allowed_Leaders.dtd
832 .IP \[bu]
833 MLB_Pitching_Hits_Allowed_Leaders.dtd
834 .IP \[bu]
835 MLB_Pitching_Hit_Batters_Leaders.dtd
836 .IP \[bu]
837 MLB_Pitching_HR_Allowed_Leaders.dtd
838 .IP \[bu]
839 MLB_Pitching_IP_Leaders.dtd
840 .IP \[bu]
841 MLB_Pitching_Runs_Allowed_Leaders.dtd
842 .IP \[bu]
843 MLB_Pitching_Saves_Leaders.dtd
844 .IP \[bu]
845 MLB_Pitching_Shut_Outs_Leaders.dtd
846 .IP \[bu]
847 MLB_Pitching_Starts_Leaders.dtd
848 .IP \[bu]
849 MLB_Pitching_Strike_Outs_Leaders.dtd
850 .IP \[bu]
851 MLB_Pitching_Walks_Leaders.dtd
852 .IP \[bu]
853 MLB_Pitching_WHIP_Leaders.dtd
854 .IP \[bu]
855 MLB_Pitching_Wild_Pitches_Leaders.dtd
856 .IP \[bu]
857 MLB_Pitching_Win_Percentage_Leaders.dtd
858 .IP \[bu]
859 MLB_Pitching_WL_Leaders.dtd
860 .IP \[bu]
861 NBA_Team_Stats_XML.dtd
862 .IP \[bu]
863 NBA3PPctXML.dtd
864 .IP \[bu]
865 NBAAssistsXML.dtd
866 .IP \[bu]
867 NBABlocksXML.dtd
868 .IP \[bu]
869 nbaconfrecxml.dtd
870 .IP \[bu]
871 nbadaysxml.dtd
872 .IP \[bu]
873 nbadivisionsxml.dtd
874 .IP \[bu]
875 NBAFGPctXML.dtd
876 .IP \[bu]
877 NBAFoulsXML.dtd
878 .IP \[bu]
879 NBAFTPctXML.dtd
880 .IP \[bu]
881 NBAMinutesXML.dtd
882 .IP \[bu]
883 NBAReboundsXML.dtd
884 .IP \[bu]
885 NBAScorersXML.dtd
886 .IP \[bu]
887 nbastandxml.dtd
888 .IP \[bu]
889 NBAStealsXML.dtd
890 .IP \[bu]
891 nbateamleadersxml.dtd
892 .IP \[bu]
893 nbatripledoublexml.dtd
894 .IP \[bu]
895 NBATurnoversXML.dtd
896 .IP \[bu]
897 NCAA_Conference_Schedule_XML.dtd
898 .IP \[bu]
899 nflfirstdownxml.dtd
900 .IP \[bu]
901 NFLFumbleLeaderXML.dtd
902 .IP \[bu]
903 NFLGiveTakeXML.dtd
904 .IP \[bu]
905 NFLInside20XML.dtd
906 .IP \[bu]
907 NFLKickoffsXML.dtd
908 .IP \[bu]
909 NFLMondayNightXML.dtd
910 .IP \[bu]
911 NFLPassLeadXML.dtd
912 .IP \[bu]
913 NFLQBStartsXML.dtd
914 .IP \[bu]
915 NFLSackLeadersXML.dtd
916 .IP \[bu]
917 nflstandxml.dtd
918 .IP \[bu]
919 NFLTeamRankingsXML.dtd
920 .IP \[bu]
921 NFLTopPerformanceXML.dtd
922 .IP \[bu]
923 NFLTotalYardageXML.dtd
924 .IP \[bu]
925 NFL_KickingLeaders_XML.dtd
926 .IP \[bu]
927 NFL_NBA_Draft_XML.dtd
928 .IP \[bu]
929 NFL_Roster_XML.dtd
930 .IP \[bu]
931 NFL_Team_Stats_XML.dtd
932 .IP \[bu]
933 Transactions_XML.dtd
934 .IP \[bu]
935 Weekly_Sched_XML.dtd
936 .IP \[bu]
937 WNBA_Team_Leaders_XML.dtd
938 .IP \[bu]
939 WNBA3PPctXML.dtd
940 .IP \[bu]
941 WNBAAssistsXML.dtd
942 .IP \[bu]
943 WNBABlocksXML.dtd
944 .IP \[bu]
945 WNBAFGPctXML.dtd
946 .IP \[bu]
947 WNBAFoulsXML.dtd
948 .IP \[bu]
949 WNBAFTPctXML.dtd
950 .IP \[bu]
951 WNBAMinutesXML.dtd
952 .IP \[bu]
953 WNBAReboundsXML.dtd
954 .IP \[bu]
955 WNBAScorersXML.dtd
956 .IP \[bu]
957 wnbastandxml.dtd
958 .IP \[bu]
959 WNBAStealsXML.dtd
960 .IP \[bu]
961 WNBATurnoversXML.dtd
962 .RE