4 author: Michael Orlitzky
5 maintainer: Michael Orlitzky <michael@orlitzky.com>
8 license-file: doc/LICENSE
12 doc/htsn-importrc.example
13 doc/man1/htsn-import.1
19 schemagen/Auto_Racing_Schedule_XML/*.xml
20 schemagen/Heartbeat/*.xml
21 schemagen/injuriesxml/*.xml
22 schemagen/Injuries_Detail_XML/*.xml
23 schemagen/newsxml/*.xml
24 schemagen/Odds_XML/*.xml
25 schemagen/weatherxml/*.xml
30 Import XML files from The Sports Network into an RDBMS.
35 htsn-import [OPTIONS] [FILES]
38 The Sports Network <http://www.sportsnetwork.com/> offers an XML feed
39 containing various sports news and statistics. Our sister program
40 /htsn/ is capable of retrieving the feed and saving the individual
41 XML documents contained therein. But what to do with them?
43 The purpose of /htsn-import/ is to take these XML documents and
44 get them into something we can use, a relational database management
45 system (RDBMS), loosely known as a SQL database. The structure of
46 relational database, is, well, relational, and the feed XML is not. So
47 there is some work to do before the data can be inserted.
49 First, we must parse the XML. Each supported document type (see below)
50 has a full pickle/unpickle implementation (\"pickle\" is simply a
51 synonym for serialize here). That means that we parse the entire
52 document into a data structure, and if we pickle (serialize) that data
53 structure, we get the exact same XML document tha we started with.
55 This is important for two reasons. First, it serves as a second level
56 of validation. The first validation is performed by the XML parser,
57 but if that succeeds and unpicking fails, we know that something is
58 fishy. Second, we don't ever want to be surprised by some new element
59 or attribute showing up in the XML. The fact that we can unpickle the
60 whole thing now means that we won't be surprised in the future.
62 The aforementioned feature is especially important because we
63 automatically migrate the database schema every time we import a
64 document. If you attempt to import a \"newsxml.dtd\" document, all
65 database objects relating to the news will be created if they do not
66 exist. We don't want the schema to change out from under us without
67 warning, so it's important that no XML be parsed that would result in
68 a different schema than we had previously. Since we can
69 pickle/unpickle everything already, this should be impossible.
71 Examples and usage documentation are available in the man page.
73 executable htsn-import
77 configurator == 0.2.*,
84 groundhog-postgresql == 0.4.*,
85 groundhog-sqlite == 0.4.*,
86 groundhog-th == 0.4.*,
92 transformers == 0.3.*,
107 OptionalConfiguration
113 TSN.XML.AutoRacingSchedule
116 TSN.XML.InjuriesDetail
125 -fwarn-missing-signatures
126 -fwarn-name-shadowing
130 -fwarn-incomplete-record-updates
131 -fwarn-monomorphism-restriction
132 -fwarn-unused-do-bind
139 -- The following unbreak profiling with template haskell. We have
140 -- to build the program twice; once without profile and again with
147 type: exitcode-stdio-1.0
148 hs-source-dirs: src test
149 main-is: TestSuite.hs
153 configurator == 0.2.*,
157 htsn-common == 0.0.1,
160 groundhog-postgresql == 0.4.*,
161 groundhog-sqlite == 0.4.*,
162 groundhog-th == 0.4.*,
166 tasty-hunit == 0.4.*,
168 transformers == 0.3.*,
171 -- It's not entirely clear to me why I have to reproduce all of this.
175 -fwarn-missing-signatures
176 -fwarn-name-shadowing
180 -fwarn-incomplete-record-updates
181 -fwarn-monomorphism-restriction
182 -fwarn-unused-do-bind
187 type: exitcode-stdio-1.0
192 -- Additional test dependencies.
195 -- It's not entirely clear to me why I have to reproduce all of this.
199 -fwarn-missing-signatures
200 -fwarn-name-shadowing
204 -fwarn-incomplete-record-updates
205 -fwarn-monomorphism-restriction
206 -fwarn-unused-do-bind
214 -- These won't work without shelltestrunner installed in your
215 -- $PATH. Maybe there is some way to tell Cabal that.
216 test-suite shelltests
217 type: exitcode-stdio-1.0
219 main-is: ShellTests.hs
224 configurator == 0.2.*,
228 htsn-common == 0.0.1,
231 groundhog-postgresql == 0.4.*,
232 groundhog-sqlite == 0.4.*,
233 groundhog-th == 0.4.*,
238 tasty-hunit == 0.4.*,
240 transformers == 0.3.*,
245 source-repository head
247 location: http://michael.orlitzky.com/git/htsn-import.git