ArcPy Is Great, And...¶
Problem one: row[0]¶
def _update_year_built(layer, year_fields):
with arcpy.da.UpdateCursor(layer, year_fields) as cursor:
for row in cursor:
if row[0] is None or row[0] < 1 or row[0] == '':
row[0] = f'{row[1]}_{row[2]}'
cursor.updateRow(row)
Problem 2: Transfering data between feature classes and tables¶
with arcpy.da.SearchCursor(new_data_fc, fields) as new_data_cursor, \
arcpy.da.InsertCursor(current_data_fc, fields) as current_data_cursor:
for row in new_data_cursor:
current_data_cursor.insertRow(row)
copied_records += 1
Problem 3: Renaming/reordering fields¶
fieldmappings = arcpy.FieldMappings()
fieldmappings.addTable(energov_parcels_fc)
fieldmappings.addTable(tville_parcels_fc)
fields_list = [
('PIN', 'parcel_id'),
('own_cityst', 'own_citystate'),
('own_zip_fo', 'own_zip_four'),
('prop_locat', 'prop_location'),
('property_t', 'property_type'),
('neighborho', 'neighborhood_code'),
('adjusted_p', 'adjusted_prcl_total'),
#: ...
]
for field_map in fields_list:
field_to_map_index = fieldmappings.findFieldMapIndex(field_map[0])
field_to_map = fieldmappings.getFieldMap(field_to_map_index)
field_to_map.addInputField(tville_parcels_fc, field_map[1])
fieldmappings.replaceFieldMap(field_to_map_index, field_to_map)
Problem 4: Intermediate feature classes¶
ssa_summarized_roads = fr'{output_gdb}\ssa_bike_lanes_roads'
ssa_summarized_paths = fr'{output_gdb}\ssa_bike_lanes_paths'
ssa_summarized_lengths = fr'{output_gdb}b\SmallStatisticalAreas_2018_bike_lane_lengths'
tract_summarized_roads = fr'{output_gdb}\tract_bike_lanes_roads'
tract_summarized_paths = fr'{output_gdb}\tract_bike_lanes_paths'
tract_summarized_lengths = fr'{output_gdb}\census_tracts_2020_bike_lane_lengths'
buffered_tracts = fr'{output_gdb}\census_tracts_2020_buffered_30ft'
buffered_areas = fr'{output_gdb}\small_areas_buffered_200ft'
bike_lanes = fr'{output_gdb}\bike_lanes_20220111'
major_paths = fr'{output_gdb}\major_paths'
Enter the Pandas!¶
pandas gives you the tools to work with tables of data defined by rows and columns, called a DataFrame
In [1]:
import pandas as pd
medians_df = pd.read_csv('assets/median_age.csv')
medians_df.head()
Out[1]:
We can access individual rows and columns using .loc (with index labels) or .iloc (with indices)
medians_df.loc[row labels, column labels]
medians_df.iloc[row indices, column indices]
In [2]:
medians_df.loc[[0, 1, 2, 5], 'County']
Out[2]:
In [3]:
medians_df.iloc[10:15, :4]
Out[3]:
We can also get just a few columns from all rows
In [4]:
medians_df[['Median_age', 'Avg_MonthlyIncome']].head()
Out[4]:
Extending pandas Spatially¶
The ArcGIS API for Python provides Spatially Enabled DataFrames, which include geometry information.
In [5]:
from arcgis.features import GeoAccessor, GeoSeriesAccessor
counties_fc_path = r'C:\Users\jdadams\AppData\Roaming\Esri\ArcGISPro\Favorites\opensgid.agrc.utah.gov.sde\opensgid.boundaries.county_boundaries'
counties_df = pd.DataFrame.spatial.from_featureclass(counties_fc_path)
counties_df.head()
Out[5]:
pandas lets you work on rows that meet a certain condition
In [6]:
counties_df.loc[counties_df['stateplane'] == 'Central', ['name', 'stateplane', 'fips_str']]
Out[6]:
You can easily add new columns
In [7]:
counties_df['emperor'] = 'Jake'
counties_df.head()
Out[7]:
pandas provides powerful built in grouping and aggregation tools, along with Spatially Enabled DataFrames' geometry operations
In [8]:
counties_df.groupby('stateplane').count()
Out[8]:
In [9]:
counties_df['acres'] = counties_df['SHAPE'].apply(lambda shape: shape.area / 4046.8564)
counties_df.groupby('stateplane')['acres'].sum()
Out[9]:
pandas Solutions to our Arcpy Problems¶
row[0] Solution: Field Names¶
Let's make Erik the emperor of the small counties that use State Plane North
In [10]:
counties_df.loc[(counties_df['pop_lastcensus'] < 100000) & (counties_df['stateplane'] == 'North'), 'emperor'] = 'Erik'
counties_df[['name', 'pop_lastcensus', 'stateplane', 'emperor']].sort_values('name').head()
Out[10]:
Joined Tables Solution: Merged DataFrames¶
Let's add census data to our counties
In [11]:
census_fc_path = r'C:\Users\jdadams\AppData\Roaming\Esri\ArcGISPro\Favorites\opensgid.agrc.utah.gov.sde\opensgid.demographic.census_counties_2020'
census_df = pd.DataFrame.spatial.from_featureclass(census_fc_path)
counties_with_census_df = counties_df.merge(census_df[['geoid20', 'aland20']], left_on='fips_str', right_on='geoid20')
counties_with_census_df.head()
Out[11]:
Renaming/Reordering Fields Solution: df.rename() and df.reindex()¶
"Emperor" is too bold; let's use "Benevolent Dictator for Life" instead.
In [12]:
renames = {
'name': 'County Name',
'pop_lastcensus': 'Last Census Population',
'emperor': 'Benevolent Dictator for Life',
'acres': 'Acres',
'aland20': 'Land Area',
}
counties_with_census_df.rename(columns=renames, inplace=True)
counties_with_census_df.head()
Out[12]:
Now that we've got it all looking good, let's reorder the fields and get rid of the ones we don't want
In [13]:
field_order = [
'County Name',
'Benevolent Dictator for Life',
'Acres',
'Land Area',
'Last Census Population',
'SHAPE'
]
final_counties_df = counties_with_census_df.reindex(columns=field_order)
final_counties_df.head()
Out[13]:
Intermediate Feature Classes: New DataFrame Variables¶
With everything we've done, we've not written a single feature class to either disk or in_memory
counties_df
counties_with_census_df
final_counties_df
Finally, Write It All To Disk¶
In [14]:
final_counties_df.spatial.to_featureclass(r'C:\gis\Projects\HousingInventory\HousingInventory.gdb\counties_ugic')
Out[14]:
Spatial Joins¶
centroids_df = pd.DataFrame.spatial.from_featureclass(centroids_fc)
walksheds_df = pd.DataFrame.spatial.from_featureclass(walksheds_fc)
walk_centroids_df = centroids_df.spatial.join(walksheds_df, 'left', 'within')
Update an AGOL Hosted Feature Layer¶
feature_layer_item = gis.content.get(feature_layer_itemid)
feature_layer = arcgis.features.FeatureLayer.fromitem(feature_layer_item)
live_dataframe = pd.DataFrame.spatial.from_layer(feature_layer)
#: maniuplate/transform the existing data
cleaned_dataframe = do_stuff(live_dataframe)
feature_layer.edit_features(updates=cleaned_dataframe.spatial.to_featureset())
Resources?¶
Official Docs¶
- Pandas docs: https://pandas.pydata.org/docs/user_guide/index.html
- ArcGIS API for Python Reference: https://developers.arcgis.com/python/api-reference/arcgis.features.toc.html#geoaccessor
Example Code¶
- Erik's 2020 UGIC Intro to Pandas presentation: https://agrc.github.io/Presentations/UGIC/2020/pandas.pdf
- Updating AGOL with dataframe: https://github.com/agrc/palletjack/blob/main/src/palletjack/updaters.py#L175
- So much dataframe craziness: https://github.com/agrc/housing-unit-inventory/tree/first-dev/src/housing_unit_inventory
