One of the foremost goals that must be achieved is to model the
average population density throughout the United States. Using
this data, we would like to be able to calculate the risk
- associated with an <em>event</em> taking place somewhere in the
+ associated with an <q>event</q> taking place somewhere in the
United States. This will, in general, be an accident or other
unexpected event that causes some damage to the surrounding
population and environment.
</p>
<ol>
+ <li>
+ Most of the application code is written in <a
+ href="http://www.python.org/">Python</a>, and so the Python
+ runtime is required to run it.
+
+ <ol>
+ <li>
+ We utilize a third-party library called <a
+ href="http://pypi.python.org/pypi/Shapely">Shapely</a> for
+ Python/GEOS integration. GEOS is required by PostGIS (see
+ below), so it is not listed as a separate requirement, even
+ though Shapely does depend on it.
+ </li>
+ </ol>
+ </li>
+
<li>
The build system utilizes <a
href="http://www.gnu.org/software/make/">GNU Make</a>. The
</li>
<li>
- A redundant field, called <em>blkidfp00</em>, which contains the
+ A redundant field, called <q>blkidfp00</q>, which contains the
concatenation of block/state/county/tract. This is our unique
identifier.
</li>
We need to correlate the TIGER/Line geometric information with the
demographic information contained in the Summary File 1 geographic
header records. To do this, we need to rely on the unique
- <em>blkidfp00</em> identifier.
+ <q>blkidfp00</q> identifier.
</p>
<p>
<p>
<em>
Note: the <a href="../../makefile">makefile</a> provides a task
- for creation/import of the databases, but its use is strictly
- required.
+ for creation/import of the databases, but its use is not
+ strictly required.
</em>
</p>
<p>
A Postgres/PostGIS database is required to store our Census
- data. The database name is unimportant (default: <em>census</em>),
+ data. The database name is unimportant (default: <q>census</q>),
but several of the scripts refer to the table names. For
- simplicity, we will call the database <em>census</em> from now on.
+ simplicity, we will call the database <q>census</q> from now on.
</p>
<p>
Once the database has been created, we need to import two PostGIS
tables so that we can support the GIS functionality. These two
- files are <em>lwpostgis.sql</em> and
- <
em>spatial_ref_sys.sql</em>. See the <a
+ files are <q>lwpostgis.sql</q> and
+ <
q>spatial_ref_sys.sql</q>. See the <a
href="../../makefile">makefile</a> for an example of their import.
</p>
<p>
Since the shapefiles are in a standard format, we can use
pre-existing tools to import the data in to our SQL
- database. PostGIS provides a binary, <em>shp2pgsql</em>, that will
+ database. PostGIS provides a binary, <q>shp2pgsql</q>, that will
parse and convert the shapefiles to SQL.
</p>
<p>
- There is one caveat here: the <em>shp2pgsql</em> program requires
+ There is one caveat here: the <q>shp2pgsql</q> program requires
an SRID as an argument; this SRID is assigned to each record it
imports. We have designated an SRID of 4269, which denotes
<q>NAD83</q>, or the North American Datum (1983). There may be
States.
</p>
+ <h2>Possible Optimizations</h2>
+ <p>
+ There are a number of possible optimizations that can be made
+ should performance ever become prohibitive. To date, these have
+ been eschewed for lack of flexibility and/or development time.
+ </p>
+
+ <h3>De-normalization of TIGER/SF1 Block Data</h3>
+ <p>
+ Currently, the TIGER/Line block data is stored in a separate table
+ from the Summary File 1 block data. The two are combined at query
+ time via <a href="http://en.wikipedia.org/wiki/Join_(SQL)">SQL
+ JOIN</a>s. Since we import the TIGER data first, and use a custom
+ import script for SF1, we could <a
+ href="http://en.wikipedia.org/wiki/Denormalization">de-normalize</a>
+ this design to increase query speed.
+ </p>
+
+ <p>
+ This would slow down the SF1 import, of course; but the import
+ only needs to be performed once. The procedure would look like the
+ following:
+ </p>
+
+ <ol>
+ <li>
+ Add the SF1 columns to the TIGER table, allowing them to be
+ nullable initially (since they will all be NULL at first).
+ </li>
+
+ <li>
+ Within the SF1 import, we would,
+ <ol>
+ <li>Parse a block</li>
+ <li>
+ Use that block's blkidfp00 to find the corresponding row in
+ the TIGER table.
+ </li>
+ <li>
+ Update the TIGER row with the values from SF1.
+ </li>
+ </ol>
+ </li>
+
+ <li>
+ Optionally set the SF1 columns to NOT NULL. This may have
+ <em>some</em> performance benefit, but I wouldn't count on it.
+ </li>
+
+ <li>
+ Fix all the SQL queries to use the schema.
+ </li>
+ </ol>
+
+
+ <h3>Switch from GiST to GIN Indexes</h3>
+ <p>
+ When the TIGER data is imported via <q>shp2pgsql</q>, a <a
+ href="http://www.postgresql.org/docs/8.4/static/textsearch-indexes.html">GiST</a>
+ index is added to the geometry column by means of the
+ <strong>-I</strong> flag. This improves the performance of the
+ population calculations by a (wildly-estimates) order of
+ magnitude.
+ </p>
+
+ <p>
+ Postgres, however, offers another type of similar index —
+ the <a
+ href="http://www.postgresql.org/docs/8.4/static/textsearch-indexes.html">GIN
+ Index</a>. If performance degrades beyond what is acceptable, it
+ may be worth evaluating the benefit of a GIN index versus the GiST
+ one.
+ </p>
+
</body>
</html>
projects / dead / census-tools.git / blobdiff