4 author: Michael Orlitzky
5 maintainer: Michael Orlitzky <michael@orlitzky.com>
8 license-file: doc/LICENSE
12 doc/htsn-importrc.example
13 doc/man1/htsn-import.1
19 schemagen/Auto_Racing_Schedule_XML/*.xml
20 schemagen/Heartbeat/*.xml
21 schemagen/injuriesxml/*.xml
22 schemagen/Injuries_Detail_XML/*.xml
23 schemagen/newsxml/*.xml
24 schemagen/Odds_XML/*.xml
25 schemagen/scoresxml/*.xml
26 schemagen/weatherxml/*.xml
31 Import XML files from The Sports Network into an RDBMS.
36 htsn-import [OPTIONS] [FILES]
39 The Sports Network <http://www.sportsnetwork.com/> offers an XML feed
40 containing various sports news and statistics. Our sister program
41 /htsn/ is capable of retrieving the feed and saving the individual
42 XML documents contained therein. But what to do with them?
44 The purpose of /htsn-import/ is to take these XML documents and
45 get them into something we can use, a relational database management
46 system (RDBMS), loosely known as a SQL database. The structure of
47 relational database, is, well, relational, and the feed XML is not. So
48 there is some work to do before the data can be inserted.
50 First, we must parse the XML. Each supported document type (see below)
51 has a full pickle/unpickle implementation (\"pickle\" is simply a
52 synonym for serialize here). That means that we parse the entire
53 document into a data structure, and if we pickle (serialize) that data
54 structure, we get the exact same XML document tha we started with.
56 This is important for two reasons. First, it serves as a second level
57 of validation. The first validation is performed by the XML parser,
58 but if that succeeds and unpicking fails, we know that something is
59 fishy. Second, we don't ever want to be surprised by some new element
60 or attribute showing up in the XML. The fact that we can unpickle the
61 whole thing now means that we won't be surprised in the future.
63 The aforementioned feature is especially important because we
64 automatically migrate the database schema every time we import a
65 document. If you attempt to import a \"newsxml.dtd\" document, all
66 database objects relating to the news will be created if they do not
67 exist. We don't want the schema to change out from under us without
68 warning, so it's important that no XML be parsed that would result in
69 a different schema than we had previously. Since we can
70 pickle/unpickle everything already, this should be impossible.
72 Examples and usage documentation are available in the man page.
74 executable htsn-import
78 configurator == 0.2.*,
85 groundhog-postgresql == 0.4.*,
86 groundhog-sqlite == 0.4.*,
87 groundhog-th == 0.4.*,
93 transformers == 0.3.*,
108 OptionalConfiguration
114 TSN.XML.AutoRacingSchedule
117 TSN.XML.InjuriesDetail
127 -fwarn-missing-signatures
128 -fwarn-name-shadowing
132 -fwarn-incomplete-record-updates
133 -fwarn-monomorphism-restriction
134 -fwarn-unused-do-bind
141 -- The following unbreak profiling with template haskell. We have
142 -- to build the program twice; once without profile and again with
149 type: exitcode-stdio-1.0
150 hs-source-dirs: src test
151 main-is: TestSuite.hs
155 configurator == 0.2.*,
159 htsn-common == 0.0.1,
162 groundhog-postgresql == 0.4.*,
163 groundhog-sqlite == 0.4.*,
164 groundhog-th == 0.4.*,
168 tasty-hunit == 0.4.*,
170 transformers == 0.3.*,
173 -- It's not entirely clear to me why I have to reproduce all of this.
177 -fwarn-missing-signatures
178 -fwarn-name-shadowing
182 -fwarn-incomplete-record-updates
183 -fwarn-monomorphism-restriction
184 -fwarn-unused-do-bind
189 type: exitcode-stdio-1.0
194 -- Additional test dependencies.
197 -- It's not entirely clear to me why I have to reproduce all of this.
201 -fwarn-missing-signatures
202 -fwarn-name-shadowing
206 -fwarn-incomplete-record-updates
207 -fwarn-monomorphism-restriction
208 -fwarn-unused-do-bind
216 -- These won't work without shelltestrunner installed in your
217 -- $PATH. Maybe there is some way to tell Cabal that.
218 test-suite shelltests
219 type: exitcode-stdio-1.0
221 main-is: ShellTests.hs
226 configurator == 0.2.*,
230 htsn-common == 0.0.1,
233 groundhog-postgresql == 0.4.*,
234 groundhog-sqlite == 0.4.*,
235 groundhog-th == 0.4.*,
240 tasty-hunit == 0.4.*,
242 transformers == 0.3.*,
247 source-repository head
249 location: http://michael.orlitzky.com/git/htsn-import.git