Selecting the georeferenced image to add to ArcMap 10

Add a georeferenced map or image to ArcMap. For this exercise add Haida.jpg. Download the georeferenced map in a .zip file from http://spatial.scholarslab.org/wp-content/uploads/2012/08/Haida.zip (15MB) and save to your computer.

The zip file format reduces file size and preserves folder structure but the contents must be extracted before use.

Extract the zip file contents to your computer using Windows Explorer, WinZip, or another program.

In ArcMap use the “Add Data” button to browse to the folder where you extracted the zip contents and highlight the file by single clicking Haida.jpg and then click Add.

Map of Haida Villages

Catalog Window

Open the Catalog window.

New Shape File

Browse to your desired folder location and create a new point data set by right-clicking on that folder and selecting New > Shapefile. Give it a name and Edit the coordinate system. The easiest option is to simply import the coordinate system from the georeferenced map. Click Add and then OK twice.

Create New Shape File Dialog

Spatial Reference Properties

Browse for Dataset

The new point shapefile will now appear in the Table of Contents, but nothing will appear on the map because the new data set has not yet had information added into it.

Updated Table of Contents

Table of Contents Open Attribute Table

Open the attribute table for the newly created point shapefile and add a new Text field called Name. As you can see the table is empty. Leave the table open.

Table Add Field

Table Add Field

Editor Toolbar

If not already present within your ArcMap window, turn on the Editor toolbar and then select Start Editing from the toolbar.

Start Editing

Create Features

We can now start the process of adding the features that we want to digitize. Zoom into the first feature that you want to add to your new layer. In the Create Features box that has opened, select your empty shapefile and then at the bottom of the section underneath that, select Point.

Construction Tools

With the arrow tool that appears you can now create your point features and add a name to the new feature in the open attribute table.

Create Feature Tool

Type Feature Name

Haida Villages

After adding all the features, stop editing and save your edits. You have now finished creating a point feature shapefile. This new layer can be added to any kind of GIS map (basemap, thematic map, digital elevation model, imagery) and the points can be labeled. All sorts of spatial queries and analyses can now be performed using these point features: buffers, distance measurements, shortest paths, etc.

Finished Product: Vectorized points on a topographic map

a. ArcGIS.com/home

a. Sign up for a free ESRI personal account at www.arcgis.com/home. Click the ‘Sign In’ link and create an account if you do not already have one.
Click the ‘Map’ tab. Once the webmap has loaded, follow the brief instructions in the sidebar: Zoom to your area of interest, select your basemap, and for the purposes of this exercise, save your map. Give it a title, and add tags to allow others to find it. Click the ‘My Content’ tab.
 
 

b. My Content

b. In ‘My Content’, select the checkbox to the left of your new map. The ‘Share’ tab will become active.

c. Sharing

c. Click ‘Share’ and in the dialog that appears, check the box next to ‘Everyone’. The map will become public once it has been shared. Alternatively, you could create an ArcGIS.com ‘Group’ so that only group members have access. All members must have ESRI accounts.

d. Now, click on the title of your map listed in the ‘My Content’ data grid to return to your webmap.

a. Shapefile

a. A shapefile is a multipart data storage format for geospatial data. The easiest way to understand the various files which comprise a shapefile is to navigate to the location of a shapefile on your hard drive using Windows Explorer. Here is an example of a shapefile which represents roads in the City of Charlottesville (TIGER Shapefile, US Census Bureau). As you can see, a shapefile consists of several file types (at least .shp, .shx, and .dbf) which store information related to feature geometry, attributes, and map projection, or coordinate system.

b. 7 Zip

b. In order to add a shapefile to your ArcGIS.com webmap use a program such as 7-Zip to zip your shapefile. Ensure that all of the files which constitute your shapefile are placed on the root of the zipped folder. If you do not have a shapefile, try downloading a TIGER shapefile from the US Census Bureau: http://www.census.gov/geo/www/tiger/tgrshp2010/tgrshp2010.html or a local government website. You may have to clip it first if the shapefile contains over 1000 features (search the ArcGIS Resource Center to learn more about geoprocessing tools such as ‘Clip’).
Alternatively, you can drag and drop .csv, .txt, or .gpx files directly into your ArcGIS.com webmap from your computer. Your .csv, or .txt file must include either latitude and longitude fields, or address fields such as Address, City, State, Zipcode for ArcGIS.com to geocode your data and display it on the map. A .csv file may contain the addresses of local businesses, fire stations, or any other point of interest. A .gpx file (GPS Exchange Format) allows you to display data captured using a GPS device.

c. Add Data

c. Now you’re ready to add data. Click the ‘Add’ tab located in the upper left corner of your map, and select ‘Add Layer from File’. Navigate to the location of your zipped shapefile and select it. You should see your shapefile displayed in your webmap.
 

d. Map with counties

d. Now, remember to save your map- click on the ‘Save’ link located directly above your new webmap. At the right you will see my county boundaries shapefile in my webmap.

Explore ArcGIS.com to discover additional functionality you might like, such as adding points and features.

a. Share webmap

a. You may want to share your webmap with others. In this case, ArcGIS.com makes it easy to embed your webmap in your blog, or personal website. In ArcGIS.com, open your newly created webmap. Click the ‘Share’ tab.
 

b. Embed in Website

b. In the dialog that appears, click ‘Embed in Website’.
 

c. Copy HTML

c. Copy the HTML and navigate to your website or blog. If you use a popular blog service like Blogger, compose a post and select ‘HTML’. Paste the code into your post where you’d like your map to display.

 
 
d. View your blog with your newly embedded ArcGIS.com webmap.

a. Find Maps

a. This example uses the iPhone but the process is similar for other smartphones. If you have an iPhone, go to the App Store and enter ‘ESRI’ in the search box. Download the free ESRI ArcGIS app. Open the app, and select the map icon. Select ‘Gallery’ and then click the search icon at the top of the Gallery page.

b. Webmap

b. Enter the name of your webmap and scroll until you find it. Select and open your webmap. Explore the application for additional functionality you might like, such as measuring distances.

Trolley Map

Download, Extract and Open

a.  Download the ArcMap project in a zip file named creatingGISdata.zip from http://spatial.scholarslab.org/wp-content/uploads/2012/02/cvilleTrolley1.zip (32MB) and save to your local drive.  A zip file is a compression format that reduces file size and preserves directory structure.  More information on Zip files here.

b.  Extract the zip file to a local drive using WinZip, Windows Explorer or another program.

Trolley Map in ArcGIS

c.  You should now find a directory called cvilleTrolley on your local drive in the location you designated in the last step.

NOTE:  If you do not extract the zip file to your local drive, the ArcMap project will not work properly because the directory structure required for geodatabases and ArcMap is not available.

d.  Browse into the cvilleTrolley directory and double-click cvilleTrolleyMap.mxd.  This will launch ArcMap with cvilleTrolleyMap open.  Alternatively, you can start ArcMap using the Start menu and then browse to cvilleTrolleyMap.mxd by clicking File > Open….  You should see something similar to the image shown here.

Create File Geodatabase

Create a Geodatabase and Spatial Layers

A geodatabase is just what it sounds like, a place to store spatial data.  There are several types of geodatabase; we will be using a file geodatabase.  More on geodatabases here.

a.  Launch ArcCatalog either by clicking the Catalog tab on the right side of your ArcMap window or clicking the Catalog Window button near the top of the ArcMap window in the Standard toolbar.

b.  Browse to the directory where you extracted your ArcMap project.

NOTE: ArcCatalog emphasizes the use of links to working directories.  The use of the list of “shortcuts” is why you may not see your directory in the tree in ArcCatalog.   Once set up, ArcCatalog’s links can be very helpful but are often frustrating for novice users in the beginning.  To create a direct link to your working directory, click the Connect to Folder button and browse to your directory through the Windows directory tree.  You will only have to do this once; ArcGIS remembers the links and adds it to the list for next time.

c.  In the Catalog pane, right-click on the cvilleTrolley directory > New > File Geodatabase.

d.  Give your new file geodatabase a meaningful name like cvilleTrolley.gdb.

e.  Right-click on your new database > New > Feature Dataset….

NOTE:  A Feature Dataset is actually a collection of spatial layers (feature classes) grouped for a purpose.  Maybe they are all the same theme like the roads from four adjacent counties or different themes of the same area.  Feature Datasets have projections.  Therefore, all the feature classes within must be in the same projection.  When you add a Feature Dataset using the Add Data function, ArcGIS adds all of the feature classes within it to the Table of Contents (TOC).   More on feature datasets and projections here.

f.  Name it cvilleTrolleyDatasets and click Next.

g.  We want to use the same projection as our georeferenced map.  Click Import….

h.  Select 1920mapWGS.tif, click Add.

i.  Click Next.  Click Next again to skip Vertical Coordinate Systems.  Click Finish to accept defaults on other settings.

Adding the Field Name attribute to Routes (step 3.h)

Add the feature classes.  There will be three:

CityLimits – a polygon layer
Routes – a line layer
SystemFeatures – a point layer

a.  Right-click on  cvilleTrolleyDatasets > New > Feature Class….

b.  In the New Feature Class dialog, add CityLimits for Name.  Accept default of Polygon Features for Type.  Click Next.

c.  Click Next to accept Default for Configuration Keyword.

Project Directory (step 3.k)

d.  Click Finish.

e.  Repeat for Routes.  Right-click on cvilleTrolleyDatasets > New > Feature Class….

f.  In the New Feature Class dialog, add Routes for Name.  Select Line Features for Type. Click Next.

g.  Click Next to accept Default for Configuration Keyword.

We want to store the name of the line in the Routes layer using attributes.  We will add the field by entering a Field Name and selecting the Data Type.

Table of Contents (step 3.l)

h.  Type Name in next available row under Field Name and select Text as Data Type.

i.  Hit Finish.

j.  Now, go back and make another feature class named SystemFeatures of Type Point Features with two new fields Name (Text) and Type (Text).

k. The tree for your project directory should now look like the the image shown here.

l. NOTE: ArcMap may have also loaded the new layers into your TOC.  If not, you can drag and drop the whole dataset in the TOC.

Adding Symbology Categories (step 4g)

Creating Features

To add features to the feature classes we need to use ArcEditor.  To begin using the editor, we must begin an edit session.  An edit session is a safe place to work because you have to commit any changes before the feature class is changed.  This is good because if you make a bad mistake, you can always back out of a session with no harm done.  The flip side is that if you don’t commit your changes, you have lost all your work.  More on editing here.

We are also going to use a template to help us fill in some of the fields in a feature attribute table.  Templates define where new features are stored, default attribute values, and default tools for creating features.  When we start an edit session, ArcMap creates default templates for each layer if none already exist.  We can also modify existing templates and create new templates.  First, we want to categorize the types of features in the SystemFeatures layer and have ArcMap fill in the feature type in the attribute table.

a.  Right-click on SystemFeatures in the Table Of Contents (TOC) > Properties > Symbology.

b.  Click Categories.

c.  Select Type from Value Field.

e.  Click the Add Values… button.

f.  In the New Value area of the Add Values dialog box, enter Turn-around and hit Add to List.

g.  Do the same for Pass-by, Car Barn, Union Station, and C&O Station. Make sure all the values in the list are selected (highlighted), then click OK.  Feel free to change the symbols by adjusting the color ramp and/or right-clicking on individual values and selecting Properties for Selected Symbols.

Digitize City Limits Layer (step 4l)

h.  Hit OK to dismiss Layer Properties.

Notice your TOC has now changed to reflect your newly-created symbology.

i.  Load the Editor toolbar by clicking Customize > Toolbars > Editor.

j.  Start an edit session by clicking on the Editor toolbar Editor > Start Editing.  This launches the Create Features pane showing the default templates for each layer.

k. We will start with CityLimits which is a polygon layer.  Click on the layer in the Create Features pane to highlight it.  Notice your cursor switches to a crosshair symbol.

Zoom to Main St. Route (step 4q)

l. Start tracing the perimeter of the polygon making sure to double-click on the last point to end the drawing of the feature.  Since our historic map does not show the city limit boundary in the area of the map legend we must make a decision on how to proceed.  With additional research, period maps showing this part of the boundary could be found and brought into our project to trace a more accurate city limit.   Since the trolley route is our primary focus for this project we will draw a straight line boundary in this area but should note this cartographic choice in any description of this work.

m. You can either toggle off the CityLimits layer by unchecking the box next to the name in the TOC or you can change the symbology to something hollow to enable you to see through to the map.

n. We are going to digitize three features for the Routes layer: Main St., Jefferson Park Ave (JPA), and University St. west of the Main St. and JPA intersection.

o. Open the Attribute Table for Routes by right-clicking on the name in the TOC and selecting Open Attribute Table.

Digitize Main St. Route (step 4s)

p. Highlight Routes in Create Features.

q. Zoom in as necessary to include the Main St. line from the fork in the west to the turn-around in the east where the Main St. line ends.

r. Start digitizing Main St. by starting at the fork on the west end and working east.  Click to set vertices. The more points you put in a curve, the smoother it will look.  Double-click to end.

You’ll notice the historic map shows double lines in some sections of the route where the tracks split to allow trolley cars to pass in both directions or “Pass-by”. To simplify, we will ignore these double lines and digitize only a single line. Later we’ll add point features to mark the Pass_by locations.  Digitizing multiple lines for the Pass-by features could make sense for a more detailed analysis.

Digitize Routes (step 4w)

s.  Notice the Attribute Table now shows a record for the feature you just created.  Type “Main St.” (no quotes) into the Name attribute, then from the Edit toolbar choose Edit / Save Edits to save your edits.

t. Position the map to include the whole JPA leg (southern fork).  Start at the turn-around at Fry’s Spring and work back to the Main St. line.

NOTE: Notice that your cursor snaps to the end of the Main St. line and the symbol changes when you get close enough.  This is good and ensures there are no gaps or dangles in our data.

u. Enter in JPA for Name in Attribute Table.

v. Repeat steps for University line.

Final Map (step 4z)

w. You can categories the lines by the Name attribute to get different symbology for each line.

x.  SystemFeatures are next.  Highlight Turn-around under SystemFeatures in the Create Features and click on each turn-around.  There are three.  Notice the Attribute Table for this layer.  Type is filled in automatically.  You can add a name if you wish.

y. Repeat the steps for the pass-by, car barn, and train station features.

z. Save your edits and close the editor.  You should know have something like the Final Map at right.

Import your .csv file to Google Fusion Tables

Import the Data into Google Fusion Tables

First, you’ll need to have a Google account to complete this tutorial. Google accounts are free.  Sign in or sign up.

If you didn’t create it yourself in Part 1, the dataset for this tutorial (~2 MB) can be downloaded here: http://spatial.scholarslab.org/wp-content/uploads/2011/08/AfghanWarDiary_EnemiesDetained.csv. 

Go ahead and download this .csv file now (note: a .csv file can be accessed using Excel, but we should have no need of that program).

After downloading the dataset, go to http://www.google.com/fusiontables.  Sign in if you haven’t already, then select the “Import Table” option in the “New Table” drop down menu at the top left of the page.  From there, browse to find the .csv file you just downloaded.  Make sure “Comma” is selected as the “Separator character” and the “Character Encoding” is “UTF-8.”  Click “next”.

Importing may take a minute or so, depending on the speed of your connection.  After the information is uploaded, an import dialog will start and automatically select the first row as the row for determining column names.  Just click “next” here (the .csv file is set up so that the columns’ titles are listed in the first row), and then click “finish” on the last page, unless you want to change the name of the document or provide more documentation.

Click “Save”. Now that you’ve Geocoded the data and modified your columns, your Fusion Table is ready to visualize!

Format the data in Google Fusion Tables

Once your information is imported, you’ll need to format it so it will be ready for mapping. 

In the File menu, select the “Geocode” option.  This will automatically select the “Latitude” column, so just click the Geocode button.  Once the bar hits 100%, you can close this dialog. 

You’ll also want to modify the columns so that Google is recognizing each column for what it is.  You can do this by selecting the “Modify columns” option in the Edit menu. Google gives you four options for each column: text, number, location, and date/time.  Make sure the columns in your able are correctly labeled (you might need to change the Type of the “tracking number” to “text). The latitude column should be labeled location and the box next to “Two column location:” should be checked.  This should happen automatically, but if you want to check, your dialog should look like the image you see to the right. 

Click “Save”.  Now that you’ve Geocoded the data and modified your columns, your Fusion Table is ready to visualize!

First view of our map with default markers. Next we’ll customize!

Map the data

Select the “Map” option from the Visualize menu at the top of your spreadsheet.
 
This should automatically drop you into a Google map centered on your data points (in our case, Afghanistan).  The markers may not immediately appear when you first view your map in Google Fusion Tables.  If you don’t see the map markers, zoom in or out and they will appear.

Here's an example of my (simple) html layout.

Visualize the data – Customize the Pop-Up Windows

Our dataset provides much more than a list of coordinates, so we’ll want to figure out how to display additional information via the pop-up windows and markers.  In this step we will customize the pop-up windows.

Google will automatically select certain fields to appear in the windows that pop up if you click on a marker.  To configure this yourself, go to the “Configure info window” link at the top.   Here, you can click on the columns you’d like displayed and delete those you don’t want displayed– for our purposes we definitely want to add the “Enemies Detained” column to the list of columns to be included. 

After choosing the columns you want displayed, use the custom tab to order and format the columns any way you like.  If you are unfamiliar with html, I would suggest simply arranging the rows in the order you’d like them to be listed (see my simple example).  If you do know html, this is a really powerful tool and can be used to add links, images, etc.

Here’s an example of how my customizations look (note the first two buckets are not shown). I use smaller divisions towards the low end of the amount of detainees and larger ones towards the top of the range.

Visualize the data – Customize the Markers

Although your map will now mark the relevant locations and display the information you want to display in the pop-up windows, all the markers will look the same and give no hint as to which markers mark locations where greater numbers of detainees were taken into custody. 

To highlight this information, you’ll need to select the “Configure styles” option above the map.  Here you can customize your marker icons by how many detainees were taken.

Since we are only dealing with points on the map, you’ll want to adjust the settings for the “Marker icon” (the default selection).  Once you’re on that option, select the “Buckets” tab to the right, then divide the markers into 8 buckets and change the final number to 150. 

This will divide the markers equally, and assign smaller colored markers to the lower numbers and larger markers to the higher numbers.  You can customize the way these numbers are divided as well (see my example below), but you’ll see changes even if you don’t customize them. 

 Once you’ve determined your buckets, click save.  The markers will now not only show where detainees were picked up, but also indicate how many detainees were picked up in those locations.

If you’d like to embed a larger or smaller version of the map, simply change the “width” and “height” attributes in the iframe embeddable code to the size you desire.

Share/Embed the Map:

After you’re finished, you’ll want to share your map.  You can do so by clicking the “Share” button at the top right, which will let you send an email to friends or colleagues. 

You can also click the “Get link” button to obtain a link to send or publish – if you do send the link, you’ll want to change the settings for the map from “Private” to either “Public” or “Unlisted” via the same “Share” button mentioned above, so that other users are allowed to view your map.  

Here’s a link to the map I created following this process: http://www.google.com/fusiontables/DataSource?snapid=S197759ENg-&pli=1

If you’d like to embed the map onto your own website, you can select the “Get embeddable link” option just above the map.  This will provide you with code that you can embed within a website, blog, or other online resource.  If you’d like the map to appear bigger, you just need to change the width and height options at the beginning of the code. 

Finally, if you’d like to view your map in Google Earth or in a GIS program you can “Export to KML” or “Get EML network link.”  The KML file can be viewed by opening it with Google Earth.

Choose the download option "All entries, CSV format".

Download the Data

Given how often the main Wikileaks site (and how often most of the many mirrored versions of the Wikileaks site are also down), getting the data is sometimes the most difficult step.  The Afghan War Diary data, however, is usually accessible via the mirror.wikileaks.info page, which currently serves as the wiki of Wikileaks.

The main page for the Afghan War Diary in the wiki is here: http://mirror.wikileaks.info/wiki/Afghan_War_Diary,_2004-2010/

Go to the page, and then download the data in .csv (comma-separated values) format.  This should be the second option listed in the list at the bottom of the page.

Note:  If for some reason these sites are down and you’re having problems accessing the dataset, try using this public Google Fusion Table, “Wikileaks Afghan War Diary, 2004-2010.”  First click on the link and then click on File / Export via the menu at the top left.  This should lead to your downloading the entire dataset in a .csv format.  If you take this step, you may skip Step 2.

The 7-Zip File Manager

Decompress the Data

But wait, the file you just downloaded isn’t a .csv file, it’s a 7z file.

What is a .7z file?

It is a compressed file format much like a .zip file.  You will need to download special software to extract data from this .7z file.

Windows users can use the 7-zip site to download their own file archiver.

Mac users will have to use a different program, as the 7-zip file archiver is only built for windows.  Ez7z seems to be the preferred archiver for the Mac user.  Mac users may need to ensure that the compressed filename ends in .7z.

So go ahead and download an archiver, and then use it to extract the .csv file.

Use the list to the left and the “Text to Columns” option in Excel to create your column headings.

Understand the Data

Like most data, the Wikileaks .csv file needs some clean up.  First of all, the columns have no titles, and this lack of documentation/metadata makes some of the entries unintelligible on first glance.  Luckily, this dataset has been uploaded and published in a variety of different places, so simply copying a string from one of the more arcane cells, such as the first cell (“D92871CA-D217-4124-B8FB-89B9A2CFFCB4“), and then Googling that string will often lead you to a site that documents the war logs.

One site I’ve found particularly helpful is here: http://afghanwarleak.org/.  From this site we can look at individual events, such as the event connected to the string above, to get the categories of information:  http://afghanwarleak.org/war_events/D92871CA-D217-4124-B8FB-89B9A2CFFCB4

Using a site like this, we can deduce that the columns correspond to the following fields (in .csv format):

Report Key,Date,Type,Category,Tracking Number,Title,Summary,Region,Attack On,Complex Attack,Reporting Unit,Unit Name,Type of Unit,Friendly WIA,Friendly KIA,Host nation WIA,Host nation KIA,Civilian WIA,Civilian KIA,Enemy WIA,Enemy KIA,Enemy Detained,MGRS,Latitude,Longitude,Originator Group,Updated by Group,Ccir,Sigact,Affiliation,D Color,Classification

(If you’d like to import these into your excel sheet, insert a blank row, then paste this list into the first cell.  After pasting, select the “Text toColumns” option in the “Data” menu and separate the text via each “comma” into separate columns.  Here’s a tutorial on that process.)

Now that we know what information is listed in the columns, we will also benefit from understanding the acronyms used within the columns. The Guardian has published a handy glossary of these terms here: http://www.guardian.co.uk/world/datablog/2010/jul/25/wikileaks-afghanistan-war-logs-glossary.

That’s a big Excel sheet! Let’s make it more manageable.

Parse the Data

Now that we understand the data fields and have a reference sheet to decipher what certain acronyms mean, we can go ahead and determine what aspect of the spreadsheet we’d like to visualize.

While visualizing the 76000+ events currently listed in the .csv file might give one a good sense of the overall picture of events in Afghanistan, limiting the visualization to one type of event might help you and your audience to see something more specific and more understandable.  So let’s limit our dataset to only those events where enemies were detained.

First, sort your excel sheet by the “Enemies Detained” field, and then copy and paste all rows listing events where at least 1 enemy was detained (i.e. all cells that are greater or equal to 1) into a new Excel sheet.  This should leave you with about 2373 rows of information, or 2372 events and one title row.

Save this file as a .csv spreadsheet (choose save as in excel, then scroll down in the “Save as Type” dropdown menu until you see the .csv option).  We’re saving it as a .csv file rather than an Excel file because the Google Fusion tables work better with this simpler type of file.