<< Click to Display Table of Contents >> Navigation:  More Frequently Asked Questions > FAQs Common to Multiple Assessor Series Applications > Assessor Series FAQ #31 Frequently Asked Questions

QUESTION: How are data from multiple salary surveys combined in the Assessor Series?

 

Combining Survey Data

When ERI combines the data from multiple surveys into a single result, many steps are taken to ensure that the result is accurate.  To accomplish this, a type of advanced data analysis method called Meta-Analysis is conducted to accurately combine information from multiple salary surveys.  This respected and commonly used technique is a standard tool used by researchers in areas of investigation such as cancer research.  Below, Meta Analysis is discussed, but it should be noted that the particular type of analysis technique used is one small part of an overall research process.  As with any form of research, the final result is far more dependent on the quality of the data going into the analysis than the analysis itself.  The process of collecting, culling, and matching data is the most time intensive part of compensation analysis and will be discussed following a description of Meta-Analysis.

 

Meta-Analysis

 

Meta-Analysis is a well established statistical procedure used to analyze the results from different studies.  This technique can be used to combine multiple studies in a manner which yields results that are more accurate than any one of the component studies.  

 

In this methodology, studies which ask the same research questions are matched, and common data elements from each study are collected for analysis.  Elements such as weighted means, sample sizes, and measures of variance are cataloged and used in the analysis process to ensure that appropriate weight is given to each data point.  The dependent variable for the analysis is salary.  The independent variable is occupation.  Additional variables which change the relationship between the occupation and salary, called moderator variables, are also used in the analysis.  These variables include years of experience, revenue, occupation level, geography, and industry.  The inclusion of these variables allows us to show results by each of these specifications.  The specific algorithms used for the Meta- Analysis process are proprietary, though they do follow standard statistical practices.  

 

Data Matching

 

One of the most critical components of Meta-Analysis is ensuring that surveys are combined across similar data.  Essentially, we are making sure that we are comparing apples to apples.  The data elements which ERI matches between surveys are occupation, years of experience or company revenue, geography, and industry.  When care is taken to match these elements, the Meta-Analysis process can yield accurate results.

 

Job Matching

 

Proper job matching is central to the success of a Meta-Analysis.  This is the process of matching a specific job to one of the jobs listed in a survey.  In the context of ERI’s analyses, it is the process of matching ERI’s internal occupational definitions to those of the incoming survey.  Because this process requires human judgment, it is necessary to make this process as systematic as possible.

 

To do this, multiple independent raters go through the job descriptions in the surveys and match the jobs in the surveys to ERI jobs we have developed.  These initial matches are conducted based on job descriptions.  After the initial match, the raters sit down as a group, discuss the independent ratings, and come to a consensus as to whether a particular survey job actually matches the internal job.  An internal job matching manual is used in this process to assist the raters in making objective determinations.  Factors such as level, education, SVP, industry, and 97 additional hard metrics are also considered for each job.  These factors are provided by a job analysis firm which is owned by ERI. 

 

Selection of Surveys

 

One advantage of Meta-Analysis is that it can consider variability between surveys.  This does assume that the variability is not due to poor survey techniques.  The use of surveys which do not follow well accepted survey procedures may be expected to contribute noise in the final analysis.  Because of this, great care is taken in the selection of surveys to ensure an appropriate level of methodological rigor.  The specific methods of each survey are examined to determine whether the researchers used sound methodological principles to conduct the survey.