4 author: Michael Orlitzky
5 maintainer: Michael Orlitzky <michael@orlitzky.com>
8 license-file: doc/LICENSE
12 doc/htsn-importrc.example
13 doc/man1/htsn-import.1
19 schemagen/Heartbeat/*.xml
20 schemagen/injuriesxml/*.xml
21 schemagen/Injuries_Detail_XML/*.xml
22 schemagen/newsxml/*.xml
23 schemagen/Odds_XML/*.xml
24 schemagen/weatherxml/*.xml
29 Import XML files from The Sports Network into an RDBMS.
34 htsn-import [OPTIONS] [FILES]
37 The Sports Network <http://www.sportsnetwork.com/> offers an XML feed
38 containing various sports news and statistics. Our sister program
39 /htsn/ is capable of retrieving the feed and saving the individual
40 XML documents contained therein. But what to do with them?
42 The purpose of /htsn-import/ is to take these XML documents and
43 get them into something we can use, a relational database management
44 system (RDBMS), loosely known as a SQL database. The structure of
45 relational database, is, well, relational, and the feed XML is not. So
46 there is some work to do before the data can be inserted.
48 First, we must parse the XML. Each supported document type (see below)
49 has a full pickle/unpickle implementation (\"pickle\" is simply a
50 synonym for serialize here). That means that we parse the entire
51 document into a data structure, and if we pickle (serialize) that data
52 structure, we get the exact same XML document tha we started with.
54 This is important for two reasons. First, it serves as a second level
55 of validation. The first validation is performed by the XML parser,
56 but if that succeeds and unpicking fails, we know that something is
57 fishy. Second, we don't ever want to be surprised by some new element
58 or attribute showing up in the XML. The fact that we can unpickle the
59 whole thing now means that we won't be surprised in the future.
61 The aforementioned feature is especially important because we
62 automatically migrate the database schema every time we import a
63 document. If you attempt to import a \"newsxml.dtd\" document, all
64 database objects relating to the news will be created if they do not
65 exist. We don't want the schema to change out from under us without
66 warning, so it's important that no XML be parsed that would result in
67 a different schema than we had previously. Since we can
68 pickle/unpickle everything already, this should be impossible.
70 Examples and usage documentation are available in the man page.
72 executable htsn-import
76 configurator == 0.2.*,
83 groundhog-postgresql == 0.4.*,
84 groundhog-sqlite == 0.4.*,
85 groundhog-th == 0.4.*,
91 transformers == 0.3.*,
106 OptionalConfiguration
114 TSN.XML.InjuriesDetail
123 -fwarn-missing-signatures
124 -fwarn-name-shadowing
128 -fwarn-incomplete-record-updates
129 -fwarn-monomorphism-restriction
130 -fwarn-unused-do-bind
137 -- The following unbreak profiling with template haskell. We have
138 -- to build the program twice; once without profile and again with
145 type: exitcode-stdio-1.0
146 hs-source-dirs: src test
147 main-is: TestSuite.hs
151 configurator == 0.2.*,
155 htsn-common == 0.0.1,
158 groundhog-postgresql == 0.4.*,
159 groundhog-sqlite == 0.4.*,
160 groundhog-th == 0.4.*,
164 tasty-hunit == 0.4.*,
166 transformers == 0.3.*,
169 -- It's not entirely clear to me why I have to reproduce all of this.
173 -fwarn-missing-signatures
174 -fwarn-name-shadowing
178 -fwarn-incomplete-record-updates
179 -fwarn-monomorphism-restriction
180 -fwarn-unused-do-bind
185 type: exitcode-stdio-1.0
190 -- Additional test dependencies.
193 -- It's not entirely clear to me why I have to reproduce all of this.
197 -fwarn-missing-signatures
198 -fwarn-name-shadowing
202 -fwarn-incomplete-record-updates
203 -fwarn-monomorphism-restriction
204 -fwarn-unused-do-bind
212 -- These won't work without shelltestrunner installed in your
213 -- $PATH. Maybe there is some way to tell Cabal that.
214 test-suite shelltests
215 type: exitcode-stdio-1.0
217 main-is: ShellTests.hs
222 configurator == 0.2.*,
226 htsn-common == 0.0.1,
229 groundhog-postgresql == 0.4.*,
230 groundhog-sqlite == 0.4.*,
231 groundhog-th == 0.4.*,
236 tasty-hunit == 0.4.*,
238 transformers == 0.3.*,
243 source-repository head
245 location: http://michael.orlitzky.com/git/htsn-import.git