Finding The Key For The Keyhole: (or how to make KML files retain attributes)

It has happened to everyone who uses GIS. You're minding your own business, enjoying a strong cup of coffee and happily opening your shapefiles and saving them into proprietary formats so that only those who are 'in the club' can open... oh wait, I mean saving your files into a flexible and friendly

PostGIS

database when a co-worker or client sends you a KML file... and you have the same well-practiced response:

Because you know more often than not, it won't work and you'll spend the rest of your day futzing around with it, only to have moderate success with elaborate joins or editing in a text-editor. Or you could just hit delete and pretend you never received it...

In the past I've had limited success with

ogr2ogr

converting from KML to SHP after the 1.8 GDAL update BUT I never understood the underlying issue (or why it - seemingly - worked randomly). I recently received a KML file that threw this error when opening in QGIS: 

So I tried GlobalMapper, which opened the file but didn't preserve any attributes just the geometry and then ArcMap (because what better way to store your data than a layer file....) and that too only offered a "

Background Server Threw An Exception

" - ArcMap's cousin to the lovely "Segmentation Violation" - so informative. So I turned to the University of Google and discovered

this

post by Dane Springmeyer (who I find frequently ferrets out the weirdest little issues). 

He describes an issue with the encoding of KML files as they get exported where the KML claims it is encoded in UTF-8 but is actually encoded in iso-8859-1 (or Latin-1). If you have files with unknown - or inaccurate - encoding, he also shares a tool to identify the likely encoding called

Chardet

. Once you know the encoding you can re-encode the KML with

one line of code

and it will open properly with all the attributes properly preserved. Here is the re-encoding applied to my KML file: 

Dane's simple explanation and the easy to use

iconv

 library might just save you hours of tinkering and frustration. That way you can get back to work fighting the good fight against data silos...

GIS Educators Day 2014 - A Clarification

I really like GIS. AND I really like teaching GIS; you know, the theory and concepts and ideas that are transferable to a variety of desktop/web-based tools. So why wouldn't I be excited for NEARC's annual GIS Educators Day? Well... I am in spirit but not in practice and this post is meant to serve as a warning to folks who can't read between the lines in the proposal criteria (I certainly couldn't last year).

My peeve is with this language (unchanged from last year):

"GIS Educators Day, sponsored by NEARC and Esri, is a full-day conference devoted exclusively to topics and issues pertaining to the use of GIS and other geospatial technologies in K-12, college, and informal education."

I understand the acronym stands for Northeast Arc User. I understand that it is sponsored by ESRI. I don't understand why the language wasn't changed this year to reflect criteria and better guide proposal submissions that aren't listed online and only discovered via email:

Response To Proposal

My Suggestion

Apparently allowing students to explore a variety of web-based mapping tools (yes, including ArcGIS Online) and asking them to compare and contrast their functionality doesn't count as 'education' and all you folks teaching non-ESRI products aren't really GIS Educators. As I've mentioned in previous posts, I use ArcMap on a daily basis - I have no issue with ESRI's products per se, I do have an issue with hiding criteria that require presenters to explicitly use or integrate a specific software package under the umbrella of GIS Education. I'll refer you to this sarcastic rant for my reasoning. From my perspective, this is less about exploring education and instead co-opting the language of GIS Education (making that acronym synonymous with one vendor) to ensure everyone is drinking the same Kool-Aid. So to all you wanna-be presenters - make sure you're teaching GIS correctly before you go submitting any proposals..

Creating Geologic Maps in QGIS: Strike and Dip Symbols

Last week I had a brief Twitter conversation with @polarwander about the feasibility of using QGIS for geologic mapping.  If you've read any of my posts (rants?) before, it will come as no surprise that my usual response to any question that starts out "Can I use QGIS for..." is yes. There are limitations and things that are easier in other software packages but the integration of GRASS, Saga, TauDEM, etc., within the Processing Toolbox makes it a viable option for most situations. However I couldn't find a quick link to share that explained how to use a basic strike and dip symbol in QGIS - so this is my attempt at providing a little guidance.

By default QGIS doesn't install with a library of geologic symbols. I started with SVG symbols from the QGIS repository here. The symbols worked fine when I added them to the view but when I tried rotating them I ended up with weird red lines next to each symbol (Figure 1).

Figure 1: Odd red stroke adjacent to each strike and dip symbol in QGIS.

I opened each SVG with Inkscape but didn't find anything except the original symbols, but when I opened them in Illustrator I was able to both see and delete the red lines (Figure 2). I can't for the life of me figure out why they didn't show up in Inkscape - if you've seen this before or have thoughts, please post a comment to help clarify.

Figure 2: Stroke visible in Illustrator. I un-grouped the elements and deleted the offending red line.

The updated SVG library can be downloaded here: https://github.com/vanhoesenj/FGDC-Joints-SVG. To be clear these are the same files available on the QGIS repository, all I did was open each file, ungroup the elements, delete the red line and re-save. You can download only the SVG files you need, the zip file or the entire repository if you're comfortable using a GitHub client.

Once you download the files of interest you need to copy them into the folder where you QGIS SVG libraries reside. I flip-flop between a PC and Mac these days so if you're using a PC you'll usually find these folders in one of three places depending on how you installed QGIS:

1. If you used the OSGEO installer on a 64-bit machine:  C:\OSGeo4W64\apps\qgis-dev\svg
2. If you used the OSGEO installer on a 64-bit machine:  C:\OSGeo4W\apps\qgis-dev\svg
3. If you used the standalone installer: C:\Program Files (x86)\Quantum GIS Valmiera\apps\qgis\svg

Note: Your version of QGIS may be different (e.g. - Lisboa instead of Valmiera, etc).

If you're using a Mac, you need to open Finder and click on Applications, scroll down to QGIS, right-click and 'Show Package Contents." You will want to copy the new SVG library into: /Applications/QGIS.app/Contents/Resources/SVG (Figure 3).

Figure 3: Location where you need to copy new SVG libraries to use in QGIS (running OSX).

To add strike and dip symbols to your map, you need to open up the Properties - Style dialog box of your point layer, select SVG Markers and select the appropriate symbol in the Joints folder (Figure 4).

 Figure 4: Style tab under Layer Properties where you can select SVG Marker as the layer type and then the appropriate joint symbol.

This will add the symbol to your map but it won't properly orient your strike and dip symbol so you need to make sure your attribute table has a rotation column (e.g. - strike, azimuth, etc) and go back to the marker dialog, select Advanced - Rotation Field (Figure 5). This is where you need to pay attention to how your data is formatted. All the joint symbols are initially oriented North and follow the right-hand-rule, so make sure you know how the data was collected and ensure your symbols are properly oriented.

Figure 5: Marker dialog where you can set the rotation field (the advanced button is next to the symbol button).

All that's left is resizing the symbols so they look appropriate with the scale of your map. Finally, you will likely want to add dip values to the map. You can use the label tab to set the label field to whatever the name of the field is containing your dip measurements. Another option is to add the Easy Custom Labeling plugin to create a new layer containing dip labels. The advantage of using this approach is you can start editing this layer and manually move each dip label so that it is properly situated next to every dip symbol (Figure 6).

 Figure 6: Simple bedrock geologic map with rotated and labeled strike and dip symbols.

Hopefully this is helpful for folks contemplating migrating at least some of their GIS workflow over to QGIS. There are a lot other ways to streamline labeling and fill patterns using PostGIS too. Feel free to send me 'how to' questions @taconic_musings and when time permits I'll create an equivalent tutorial answering your questions (if I know the answer). If you are looking for additional geology symbols, Ryan Clark (@worbly) started a GitHub repository that contains a variety of FGDC compliant symbols - you can explore and contribute to if you want!
 

GIS Was Never Simple...

I know the boat sailed in early October but still felt compelled to respond given I have an interest in both disciplines. I saved Thierry Gregorius' blog post to Pocket and finally read it last week. In general I get the sentiment and frustration but found myself disagreeing with a number of points, which is good because I wouldn't want him retiring from blogging either! So this is less of a critique and meant more to initiate (instigate?) more dialogue - Thierry, you did ask for more comments.

First I disagree that the field of GIS itself is less simple now - it was never simple. I'm referring to the underlying science related to map projections, re-projections, neighborhood statistics, autocorrelation - pick your topic. Tobler's 1970 seminal paper on autocorrelation is instructive - it was never simple. If you consider the GIS software itself, the GUIs have certainly become more cluttered but in every package I have used it was possible to turn off toolbars and extensions/plugins. If this is really a concern, vilifying software and vendors should cut across all sectors; why is no one clamoring for Cricket Graph, a return to Fortran or Internet Explorer? Ok, that last one wasn't nice. I completely agree with the final conclusion that 'spatial is not special' but then again, neither are geoscientists (I just implied I wasn't special too, don't get feisty).

Geoscientists are not the only folks who consume data that has been 'digitized from analog sources, scraped from literature or intelligence reports, imported from spreadsheets, dug up from archives, copied from network drives, extracted from databases, downloaded from web feeds" or have to develop a story using a variety of data formats and datasets (Gregorius, 2013). The same holds true for town planners (not necessarily GIS professionals), historians, biologists, transportation specialists, and many (many) other disciplines. 'Our' issue is not ours alone to claim. 

The data format issue is real and it is challenging but certainly not insurmountable. I find it unrealistic to assume that if geoscientists "don't consider themselves to be GIS professionals" but that "no GIS professional today can survive without FME or GDAL" that somehow the vendors have ignored the needs of our profession. Is this issue related to vendors producing software capable of consuming the data or that many data sources are stored in proprietary data formats with complex schemas that require translation? FME and GDAL are completely different in my mind: FME offers translation capabilities in addition to conversion while GDAL supports raster conversion and manipulation (OGR is the vector equivalent). GDAL and OGR are a means to address the issue of data formats not complicate it - both serve us well in the sea of data formats. So, vendors nor software 'require' us to create silos, we (yes, geoscientists too) choose that route (link, link). Criticizing open standards or open-source GIS packages is akin to couples bemoaning how challenging the adoption process is... there are plenty of children in foster care, they just aren't adopted.

The GeoWeb can be abused and neglected by any profession - geologic clearinghouses  commit some of the more egregious sins, which result in slow and unfriendly user interfaces (I won't point to at any particular state survey here, but do a random sampling on your own). Eric Edmonds and Brian Timony have offered thoughts on why these portals are so cumbersome but that isn't because of the GIS industry or GIS professionals per se. Similarly, I'm not sure how developers creating mobile applications - who aren't 'GIS people' or geologists - would know what geoscientists need so they didn't feel like they fell through the cracks. The implication here is that we need apps that can perform 'geological data interpretation' in the field on a smartphone or tablet versus having the capability to collect points, lines and polygons and easily enter data using something like a data dictionary (those apps exist).

Open-source GIS does provide a viable alternative to a proprietary vendor marriage (this isn't 'new' - GRASS has offered this for 30-years). The various FOSS-GEO options are not the reason we struggle with proprietary formats, again it is the people who choose store them that way. And Google Earth certainly offers an easier interface to work in but I don't think Brian claimed that 'most enterprise GIS requirements would be met' but instead referred to 'data viz' needs. It's a semantic issue but in my mind Google Earth is not a GIS. You can create, store, and visualize spatial data but is missing the most fundamental (in my mind) tools: the ability to query and analyze spatial relationships. So if you want to create a visual depiction of your data, store it in network KMLs and distribute, Timoney's right (again!). But if you want to buffer private water wells, use zonal statistics to characterize the dominant bedrock within a watershed, interpolate an isopach map, etc... you can't do that using Google Earth. If you're 'really' nostalgic for ArcView 3.x then take a look at gvSIG, it has a very similar interface and functionality.

I'm a little unclear who the GIS behind the curtain actually IS but here are some thoughts on Thierry's questions:

1. Where are the innovative solutions for dealing with the variety of geoscience data?
   
   Timoney referred to this in one of his Tweets (circa 2007): Store your data in a PostGIS enabled PostgresSQL database and become a spatial data hipster.
   
   
2. Where are the productivity tools to simply assemble digital scrapbooks of georeferenced information?
   
   As far as I can tell, QGIS isn't planning a coup of ESRI (nor does their mission allude to that), the interface is entirely customizable, it consumes data stored in PostGIS AND it has the ability to create map books using the Atlas function.
   
   
3. Where are the flexible data models that enable thematic harvesting and analysis irrespective of data type?
   
   PostGIS via command line, the SPIT and DB Manager plugins for QQIS or ST-Links Spatial Links for ESRI.
   
   
4. Where are the analytical tools that can handle dirty and incomplete data without hours of pre-processing?
   
   If it is dirty, it will probably take some tinkering... I guess I'd have to see an example but my default would be Open Refine. Or just import it into a PostGIS database...
   
5. Where are the predictive user interfaces that only show relevant options?
   
   Personally I don't want a software package deciding what is relevant to me, I'd much rather selectively turn off things I don't need and set a default workspace. I'm more frustrated by this in Pages or Excel than any GIS. By the way, check out PostGIS.

So I don't think the GIS community has ignored or let geoscientists fall through the cracks and I certainly don't think geologic data is special or any messier than other disciplines. If you're looking for a geology-flavored GIS, check out BeeGIS it's built on JGrass so you get the spatial tools of GRASS in a simple interface that has tools specific to hydrology and geomorphology. And that's the ultimate rub for me in this discussion - if the existing GIS packages aren't sufficient, perhaps the geoscience community should follow Matt Hall's example and build their own. Waiting for a vendor or non-geoscientist to create a custom tailored, predictive GIS will likely happen sometime in 20... never. BUT, I encourage you to explore PostGIS and let Pele change your world - she's cute, she's talented and she'll eat up all your spatial data.

http://j-vh.me/1fkUa3n  

 

Exploring the Vast Sea of Open Source GIS: There Be Dragons No More!

This is a repost of an article I wrote for GeoFocus, CT's Quarterly Geospatial Newsletter - the original can be found here.

This year the Geographic Resources Analysis Support System - a.k.a GRASS - celebrated its 30^th anniversary, establishing itself as the Bob Barker of free and open-source geospatial tools. It is this enduring alternative to a proprietary GIS that often leaves me bewildered by the surprise at discovering there are other options beyond the typical mainstream GIS suites. Surprise is often followed by questions about reliability, maturity and support – after all, “there is no such thing as a free lunch” right?

I think some of the hesitancy to embrace Free and Open Source Software for Geographical Information Systems (FOSSGIS) is summarized well by Gregy Mankiw’s statement that “to get one thing that we like, we usually have to give up another thing that we like. Making decisions requires trading off one goal against another." So if something is ‘free’ and I have access to the underlying code, there must be a catch. The issue with that assumption lies in the literal interpretation of the word free; it is best to consider FOSSGIS options through Richard Stallman’s mantra of "free speech, not free beer." It is also likely that earlier forays into open source options resulted in frustration with a graphical user interface that responded differently than more familiar Windows-based GUIs and less mature (sometimes non-existent) documentation or man pages (for those folks who explored GRASS on a UNIX/Linux OS). So there are indeed tradeoffs but depending on your needs and the structure of your organization, the opportunities for harnessing the power of both free alternatives are only limited by the null loop that you define (meaning you can utilize one small tool or migrate completely). Perhaps more importantly, through supportive user forums, YouTube, blog tutorials, and social media – we are experiencing what Dan Sui has described as “the wikification of GIS.” Open access to this ever growing free and archived ‘support staff’ has fostered the growth of a neogeography, where journalists, statisticians, graphic designers, web developers, etc. are creating and consuming spatial data to produce stunning visuals that even the most staunch cartographer would (and do) praise. If the so-called ‘non-experts’ can utilize these options with success, imagine how the geospatial community might fair if we used them more often!

My intention isn’t to encourage anyone to drop their current software merely because an alternative exists or to convince you that FOSSGIS is inherently better, but rather assuage some of the common concerns I hear regarding the alternatives. Most of the questions are usually process specific, like: “but can I do [insert common GIS task here] using that software?” And without hesitation, my response is almost always a resounding YES. In fact, depending on the license you have with many proprietary suites, you often have access to more tools and analyses. For example, GRASS offers access to powerful watershed analyses tools and SAGA offers advanced geostatistical analyses that aren’t provided by default in many proprietary suites. But in response to the overarching question of “what can I do” - basically any of the most common geoprocessing tasks (clip, merge, union, etc.), create and edit shapefiles/attribute tables, georeferencing, geocoding, spatial joins, topological editing, and - depending on the FOSSGIS tool - raster analyses and map algebra functions. Deciding on the tool often leaves people overwhelmed given the extensive list maintained by the Open Source Geospatial Foundation (OSGEO): and I think the effort to discern which to choose is often a barrier to exploring new tools.

The second most common question I hear is “so what is the best option?” As you might imagine, this decision depends on the goal. If you would like to streamline your workflow using batch scripting of common translation or projection tasks, you might just be interested in using the standalone ogr2ogr and GDAL libraries. These libraries can be accessed via command line, ogr2gui or from within a number of desktop packages. Translating between formats is relatively easy with these libraries and is further facilitated by migrating proprietary formats into a PostGIS database running on PostgreSQL. If you are wondering about an alternative to proprietary geodatabase solutions, PostgreSQL/PostGIS is a powerful object-relational database system that can support geospatial queries and analyses; I highly recommend “PostGIS in Action” as a great primer. 

Three desktop packages rise to my list of alternatives - again depending on your needs and application. For a gentle introduction to open-source alternatives I recommend MapWindow GIS, it has fewer tools installed by default but through numerous plugins you gain access to more advanced tools. Saga GIS offers access to advanced geostatistical and terrain analysis functions and specific algorithms for erosion modeling, fire risk analysis and hydrologic modeling and others. However the package I recommend most often is Quantum GIS for a variety of reasons: (1) GRASS tools are seamlessly integrated and accessible via the GRASS plugin or using the Sextante plugin (which also provides access to many SAGA functions), (2) user-created plugins are frequently added to the repositories, (3) I find the interface the most user friendly, (3) it is available for Windows, Apple and Linux operating systems, and (4) it includes cartographic functionality. Many (not all) open source alternatives don’t include tools for making cartographically pleasing maps, but QGIS does and version 2 offers even more functionality. If you’re not happy with the final layout you can export to Adobe Illustrator or a nice alternative called Inkscape or you could use QGIS to manipulate your data and then use TileMill to produce the final map, which is also cross platform.

There are but a few of the many options and purely based on my opinion and experience. If you are curious about a specific software package or how it might increase your ROI or align with your current workflow, feel free to shoot me an email but your best option would be to explore the user forums or post your questions there: the community is incredibly supportive and helpful. So I encourage you to slay those dragons of uncertainty and jump into what is no longer uncharged waters!