Showing posts with label Tools. Show all posts
Showing posts with label Tools. Show all posts

Saturday, September 24, 2022

Genealogical DNA Analysis Tool (GDAT) - Part 1: Getting started

 

When I first started undertaking DNA analysis back in 2011, I would often duplicate my effort by revisiting key matches, running queries and would sometimes spend many hours before I realised I had gone down the same rabbit hole, again.  I started many 'master' spreadsheets, often deciding to start again as I kept wanting to view the data in different ways.   Jim Bartlett blogged about his spreadsheet method in 2014 and I once again adapted my process to try and get some order into my data management, but I still had so much other information, email folders, pedigrees and notes in different places.  I was in a mess.  The spreadsheet method suits many but the volume of data can make it unmanageable particularly if you are not used to manipulating large amounts of data.  

I had heard about a database called Genome Mate Pro (GMP) but hadn't been able to use it as it only ran on Windows.  It was developed by Becky Walker and was free!  In 2015 a version that worked on a Mac was released and I started using the product, I've not looked back since.  I was delighted as it gave me order and structure, plus I immediately saw results.  Being specifically built to support DNA analysis it had all the views I wanted.  I was able to capture text and pedigree information and it gave me a 'one stop shop' for storing all my information with the ability to also record my progress with notes and status settings.

Fast forward to today, we now have the new and improved Genealogical DNA Analysis Tool (GDAT)  with separate Windows and Mac versions.  Back in 2011, I was only dealing with data from FamilyTreeDNA and GEDmatch.  In 2022 we have many more testing companies and as a result many more thousands of DNA matches.  I really don't know where I would be without GDAT!  I use it to manage other kits as well as my own and you can see from my latest stats that I now have nearly half a million DNA relatives in my database.  How I could have ever done that in a spreadsheet is beyond me!  Thank you Becky Walker for all you do!!  


Pre requisites

To be able to use the GDAT database you first need to understand the DNA Analysis Framework and the underlying concepts of chromosome analysis.  You must also be computer literate and adept with handling spreadsheets and large amounts of data.

You can find more information about the DNA Framework and chromosome analysis on this page.  If you are unfamiliar with these concepts I suggest you revisit the posts included in the 'DNA Research Framework Modules - Chromosome Analysis' section before attempting to use the GDAT product.


SAG GDAT User Group

This new series of GDAT posts have been developed to support the new user group we are establishing at the Society of Australian Genealogists (SAG) for members using the GDAT product.   It does not replace the extensive user guide and help video tutorials already developed by the GDAT team.  Members of the User Group should always consult those resources first.


To get started:

* Watch the video and undertake all the tasks in the Quick Start Guide (Setting up);

* Read how the developer (Becky) approaches her DNA analysis, recognising the tool allows for multiple variations.


To get help:

* Make sure you first consult the extensive GDAT User Guide;

* Join the Facebook Group for GDAT and search the site to see if the question has been asked before.  If not, ask the question there;

* SAG members can join the 'private' SAG GDAT User Group on Facebook for general collaboration between members.


Remember:

* GDAT is a database

* You need to constantly feed it information

* Without loading the relevant report - it can’t tell you anything

* You need to understand chromosome analysis to be able to use it

* Avoid re-work: before you start, decide how you are going to mark groups, use status settings and standardise notes. 


Site overview

The GDAT site is divided into workspaces, the key areas highlighted in the following chart.  Points 1-6 were the focus of our first meeting as these are needed to get started.  Follow the instructions in the User Guide to set up, including the video tutorials.  





The following are some key things to remember as you work with the software.
  1. File - open, close, back-up, restore.  Back up regularly!
  2. Help - lots of resources, use them;
  3. Preferences - check this area if things aren't behaving as you would expect, you probably need to adjust some of the defaults;
  4. Edit - this is where you set up profiles and activate import templates.  Profiles are people whose DNA results you are tracking.  It is important to ensure you have set up 'match keys' for each DNA source you want to import.  Before you can get any data into the database, you must also tell GDAT the type of imports you will do and activate the relevant templates;
  5. Utilities - alternative way to import gedcom, plus ability to set standard surname and variations;
  6. Import Data - this is where you load DNA data.  Remember to set up your import templates before attempting to import your data files (see 4).
When you first start using GDAT it is best to trial it with one profile and one DNA data source.  When you are comfortable with how the software is working, branch out with more profiles or new data sources.



Before you start analysing and marking segments

Once you have set up your profiles and imported data you have the information you need to start your analysis.  However, before you start reviewing your relative list or chromosome browser views (F8 - total cMs; or F7 - cMs by chromosome), make sure you have also loaded a pedigree for each profile.  

Once the pedigree is loaded, it is important to review your group names and make amendments to the ancestral couple names if needed.  As you mark segments to ancestral groups GDAT has a number of tools that use these MRCA group names, including database interrogation tools, chromosome maps, etc so it's important they are consistent.  These tools will be discussed in more detail in future meetings.

As indicated earlier, the more data you feed into GDAT the more useful it will be to you.  You should focus on importing as many pedigrees for your matches as possible if you want to use the GDAT software to work to its fullest extent.   The following chart may assist in deciding how to import pedigree information for your matches.  DNAGedcom can be extremely slow to generate reports, but the advantage is you can import trees in bulk, whereas most other methods are one at a time.  Be aware that these third party tools can be subject to change.





Not all methods of extracting DNA segment data from the source company are the same.  Some are more time consuming than others.  It may depend on the platform you are using.  Many GDAT users use different approaches, you need to decide what works best for you.  Remember whilst GDAT has a triangulation feature, unless you have loaded the relevant ICW and triangulation data it cannot provide accurate reports. Be conscious of when your GDAT system can/cannot provide this, it may not be possible for all DNA sources at all times.  Again, be aware that the third party tools for extracting this information are constantly subject to change.




The last three workspaces will be the focus of future meetings when the genetic and genealogical tools of the GDAT software will be discussed in more detail.

  1. Relative - views relevant to your DNA matches;
  2. Profile - views relevant to people whose DNA you are monitoring;
  3. MRCA - tools to assist your research.

GDAT is a complicated product, but we are dealing with a complicated subject matter.  It has many benefits in increasing your productively when reviewing your DNA results.  It is of particular benefit when managing multiple kits.  If used regularly it provides a 'one stop shop' and makes it much easier to quickly pick up where you left off.  Don't keep re inventing the wheel, it might be time consuming to start but bear with it.  If you are serious about your analysis and plan to be in it for the long haul it will definitely be worth it!


Veronica Williams
GDAT Group Convenor
23rd September 2022



Other GMP references:

There are some other documents that may be useful when learning about GDAT that were developed for the old GMP system.  Whilst not exactly applicable, many of the concepts are the same.  Most I know about have been collated onto the DNA Sydney blogsite, you can access them via this page.


Sunday, October 24, 2021

Tools: Clustering for Chromosome Analysis

Clustering tools are a great way to quickly identify groups of matches with a relationship to each other.  

As we know shared matches are 'clues', whilst shared segments are 'evidence' of a shared ancestor.  All cluster tools identify matches that are likely to be clues of common relationships.   The best types of clusters in my opinion, are shared segment clusters, however every cluster analysis you do is likely to give you new ideas about how your matches might relate to you.

This post is aimed at identifying where clustering can be undertaken and whether they provide shared match or shared segment clusters.  Click on the heading hyperlinks for more information, plus please refer to the blog posts at the end of this page for more information about using each of the different tools.

  

DNAGedcom Client 

Using the cluster tool at DNAGedcom requires a subscription which can be taken out for a minimum of one month.  The DNAGedcom client provides clusters based on the Collins Leeds Method for AncestryDNA and FamilyTreeDNA.  

As AncestryDNA does not provide segment data, the clusters generated are 'shared match' clusters.  Generating clusters with matches >30cMs will generally given an indication of shared ancestry, however clusters generated with less than that amount may give false leads with matches potentially sharing different more distant ancestors.   


DNAGedcom has an another advantage when working with AncestryDNA data.  It allows you to identify other shared matches that are below the 20cMs threshold, provided they share >20cMs with the match you have in common.  This can be confusing, hopefully this cheat sheet helps. 

The FamilyTreeDNA clusters generated by DNAGedcom are based on shared segments and most matches in the cluster are usually triangulated groups.  Due to this, smaller cMs defaults can be used and each cluster will provide clues of a shared common ancestor.  Additionally, shared matches are also shown in grey.


  

GEDmatch 

There are now 2 cluster tools at GEDmatch, they require a Tier 1 subscription which can be taken out for a minimum of one month. 

The first tool is the AutoTree Clustering tool which is based on shared segments and each cluster of the same colour is usually a triangulated group.  Similar to the DNAGedcom tool for FTDNA, smaller cMs defaults can be used and each cluster will provide matches of interest sharing common ancestors.



In Oct 2021 a second tool was released called AutoSegment.  It provides triangulated data.  Refer to the Roberta Estes blogpost for more information on this great new tool.





The free tool at My Heritage is an example of a 'shared match' cluster, so whilst the matches in each cluster have something in common, they may not all descend from the same common ancestor.  The advantage of this cluster tool is that it also provides segment detail.  In the example below interrogation of Cluster 3 (yellow) revealed four distinct triangulated groups, each providing separate clues regarding the identity of the common ancestor.  The main limitation with this tool is that you are unable to change or refine the search parameters. 

Genetic Affairs

Genetic Affairs is a third-party tool created by Evert-Jan Blom that provides great tools for use with 23andMe, FTDNA, GEDmatch and My Heritage DNA data.  Evert also developed the free My Heritage cluster tool and the new GEDmatch AutoSegment tool.  

The Genetic Affairs site is another subscription site that provides a range of tools to assist with your analysis.  Running the tools at Genetic Affairs provides more flexibility and in particular for the My Heritage cluster also shows interrelationships, something not provided by the free tool.

Genetic Affairs provides many other useful tools on its site, including AutoKinship, AutoTree and AutoPedigree.


Shelley Crawford Clusters 

Connected DNA used to provide fabulous network maps of shared match clusters for AncestryDNA, FamilyTreeDNA, and 23andMe.  However, as at October 2021 Shelley is taking a break and no orders are being taken at the moment.  We hope she will be back soon.


DNA Painter Cluster Auto Painter 

You can upload clusters created from DNAGedcom (FTDNA only), My Heritage, GEDmatch and Genetic Affairs to the DNA Painter Cluster Auto Painter to visualise the segments and make notes about the results of your analysis.  This allows for a more visual approach to analysing your segment data.  The image below has been generated from the output of the AutoSegment at GEDmatch.



Shared Clustering Tool

The Shared Clustering tool by Jonathan Brecher works from AncestryDNA shared match lists. Rather than providing a single list of matches ordered only by the strength of the match, Shared Clustering divides that list into smaller clusters of matches that are likely related to each other.  Downloading directly from AncestryDNA has been disabled, but files extracted using DNAGedcom can be uploaded.  It only works using Windows.



RootsFinder 

RootsFinder is a family tree building and DNA analysis website. The premium level has DNA features for a subscription fee.  The triangulation (cluster) view allows you to view your matches in clusters – otherwise known as a network graph.



Ideas for exploring your cluster

* Explore the matches in the cluster and check if the shared matches are also sharing the same segments;

* Are there any triangulated groups within the cluster?  Explore these matches first.  Remember you can expand each triangulated group by checking your segment data for others not appearing in the cluster report, who may not have met the cluster criteria.

* Are there any 'bridge' matches in the cluster?  Use these to help you to find others who match in the same segment area at other sites.

* If the cluster includes matches not triangulating with the core group, explore those segment areas as these may provide additional clues to the possible relationship of those in the cluster. 

* Do the genealogy!  Build research trees for your matches and revisit your own pedigree to search for the what is in common between the cluster group - surnames, locations, ethnicities?

* Once you have identified a 'side' or MRCA make notes on your master list and at the DNA site for all the matches in the triangulated group. 

* You may also wish to allocate a reference number to your Cluster for future reference.  Remember all the tools have their own numbering systems which constantly change with each report.

* Make sure you are systematic with your notes so that next time you generate a cluster report you can easily see matches that have been worked on before.

* Don't forget to consider the other side of the chromosome - use what you now know to mark the segments on the opposing side.   Explore the other side of the 'specific segment area' for more triangulated groups.  By looking at both sides concurrently you can then mark others not triangulating as potentially false positive matches, saving analysis time in the future.

My group numbering system has changed many times over the years, it has been adapted from the Jim Bartlett method.  It doesn't need to be as complex as this, but it needs to be meaningful for you.



More information:

Check out these links for useful blogs that may help you interpret how the different cluster sites work:

* The Leeds Method, Dana Leeds, 2018 

* What to do after Clustering (You Tube), Dana Leeds, 2024

* DNA Sleuth, 2019

* Walking the Clusters Back, Jim Bartlett 2019

* Cluster Auto Painter, Jonny Perl 2019

* Shared Clustering - A great tool! Jim Bartlett, 2019

* 10 ways to group your genetic clusters, Family Locket, 2019

* Connecting the Dots, My Heritage 2020

* Fast Ways to Cluster your DNA Matches, Family Locket 2020

* Genetic Affairs Tools and how to use them, Roberta Estes 2020

* Walking Back the Clusters, Veronica Williams 2020.  Demonstration of applying Jim's method.

* Using DNA tools to solve a family mystery, Vicki Hails 2020

* Annotating a Cluster Auto Painter Map, Jonny Pearl 2020

* Auto Segment Triangulation Tool at GEDmatch, Roberta Estes 18 Oct 2021

* RootsFinder Network Graphs, Family Locket Oct 2021

* RootsFinder - Making Triangulated Network Graphs, Family Locket Dec 2021


AncestryDNA

Whilst AncestryDNA does not provide chromosome information it is always best to start your analysis there, due to the large numbers in the database and its many pedigrees.  AncestryDNA Clustering can be done using the DNAGedcom CLM tool, Shared Clustering Tool and the DNA2 Tree app.  You will often find Ancestry testers at other sites, particularly GEDmatch and My Heritage (a growing database with lots of family trees).  Always look for 'bridge matches' between the sites as they can help to expand your research pool and help you tie your triangulated groups together with broader shared match clusters.


Veronica Williams

First published: 24 Oct 2021

Last updated: January 2025


2024 NOTE:  Since September 2023 many downloads have been unavailable due to restrictions at the DNA companies, this includes the very handy DNA2Tree Tool.  GEDmatch has remained available throughout this time and finally FTDNA returned their reports in late August 2024.  We hope things will change at My Heritage and 23andMe soon.  

2025 NOTE: The release in late 2024 of AncestryDNA's new Pro Tools feature, Enhanced Shared Matching (ESM) is proving to be a 'game changer' for identifying family groups and clusters within your shared match lists.  Using the data in ESM you can often work out relationships and family groups for matches that don't have trees!  As always, interrogate AncestryDNA clusters to find people who have their data on chromosome sites.  The reverse is also recommended, if you have triangulated groups (TG) for a segment you know belongs to a 'brick wall' ancestor who you can't identify, look for people in the TG's who might also be on AncestryDNA and try to drill into their shared matches to broaden the clues for your search.