Behavioral interventions for treating symptoms of psychiatric and developmental disorders have been helped enormously by the increased expectation of scientific rigor in design and analysis of these kinds of studies over the past decade. The standards for testing of efficacy and effectiveness of behavioral treatments now mirror the rigor of standards for testing of efficacy and effectiveness of pharmacological treatments and include randomized controlled trials, power analyses, preregistration, independent data and safety monitoring boards, independent replication, intent-to-treat designs, and data sharing. Positive results from meta-analyses are fast becoming the new standard of efficacy. The results of this increase in rigor combined with the ethical standards to use evidence-based interventions seen in many health-related professions, institutions and agencies serves the public by promoting and funding treatment approaches that have high quality data behind them, and by helping patients to seek out and prioritize those interventions with the strongest evidence.

This commentary addresses a recently published replication of a comprehensive intervention designed for very young children with ASD to be implemented at the point of first diagnosis and designed to address all areas of need [1]. The commentary will address three key issues that have been raised by the study, points that will likely be applicable to clinical trials of behavioral treatments and their replications to many different childhood conditions. The issues are: rationale for choosing targets of the interventions and measures of effects; expected effects of replication methods and procedures on outcome data; and the nature of comparison groups and conditions involved in clinical treatment trials. The commentary will begin with an introduction to the treatment and its development, and then proceed to address these three issues.

The treatment examined in the paper is one of several evidence-based interventions for very young children with ASD that uses a naturalistic interaction approach to deliver a treatment plan for which both the content and the treatment procedures are derived from an integration of developmental and behavioral science principles -an NDBI approach (naturalistic developmental/behavioral intervention; [2]. The Early Start Denver Model’s (ESDM) roots originated in the 1980s, a time when expectations for autism early intervention were strongly affected by the reports of Lovaas’s 1987 intervention findings of large changes in IQ, language ability, and adaptive behavior, a degree of improvement that was unheard of at the time. Early studies of the Denver Model began with descriptive papers, pre-post studies, and replications using non-rigorous designs considered acceptable designs for studying intervention outcomes several decades ago. The intervention was revised and improved in collaboration with Geraldine Dawson during 2004-2007, increasing its applicability to toddlers as young as 12 months, adding considerable measurement and procedural rigor, and integrating positive behavioral approaches from applied behavior analysis. The resulting model, renamed the Early Start Denver Model, was tested in a rigorous randomized controlled trial (RCT; [3,4]). Compared to a control group receiving community intervention, children in the ESDM demonstrated growth in a two-year period on standardized measures in: cognition, receptive and expressive language, and adaptive behavior, gains that compared favorably to the results published by Lovaas [5].

An independent multisite replication study resulted in the 2019 JAACAP paper that is the focus of this commentary. Study findings supported the prespecified primary hypothesis of the paper, that the group of children receiving ESDM would demonstrate greater language improvement than the community treatment comparison group. Findings did not support secondary hypotheses concerning greater cognitive or adaptive behavior gains in the ESDM groups; both groups of children made significant gains in these areas. We neither expected to find, or found, significant group differences in autism severity scores, though these also improved in both groups.

Some methodological concerns are simply not addressable within the requirements of a study. For example, parents in this study were not naive to the intervention their child is receiving, and they are also providing some of the data being used to measure outcomes. One methodological question that has been raised and can be explained is why the primary hypotheses in both the 2010 and the 2019 papers focused on developmental changes rather than specifying changes in ASD symptoms. In both studies, language and developmental gains were chose as primary hypotheses because of (1) findings from previous studies of both this and other ASD early interventions that demonstrated gains in these areas; (2) findings from past studies demonstrating the long-term power of these two variables for predicting adult outcomes in ASD (e.g. Magiati et al, [6]), and (3) findings from early childhood studies that demonstrate strong relationships between preschool IQ and language scores and later school success. To our knowledge, there are no long-term longitudinal studies that demonstrate that an autism severity measure is a better predictor of adult outcomes than IQ and language ability, which are strongly correlated in studies of ASD. At a deeper level, we understand language as the human behavior that most successfully allows for people to communicate with and learn from other people - areas of difficulty that lie at the heart of symptoms of ASD in young children. This was clearly articulated by the parents in this study, most of whose children were not speaking at the start of the study. The question “will he learn to talk?” is one of the most pressing concerns that parents voice. It is a question that we hear far more often than questions about specific ASD symptoms like eye contact. They are well aware that many children with ASD do not acquire functional speech, and they invariably ask about this risk. Thus, our focus on language as the primary outcome also has social validity. Thus, we understand that the primary effects associated with ASD early in life are seen in everyday functioning in play, language, social communication, and overall developmental rates, operating through mechanisms that impair social learning in the earliest years of life. For children, developing communication and social skills, problem solving skills, literacy skills, and adaptive behavior skills are the platforms upon which development is built. ESDM focuses on improved development and function in all these areas in everyday life, not symptom reduction per se.

In an important editorial in 2011, Sally Ozonoff [7] discussed a pattern that she suggested typifies replication results: reduced evidence of efficacy compared to the original finding. She offers several reasons for this: replications typically use more rigorous designs than the original paper - larger groups, independent authors, more rigorous approaches to measurement, independence in the analytic team, and intent-to treat analyses, among others. The 2019 ESDM replication study was carried out in three sites, spread out across the country and directed and staffed by independent teams, with initial varying levels of expertise in delivering the intervention, including two sites that were completely naive to delivering the intervention. A multisite design allows for much more site-based variability than a single site study, both by the research team and also in terms of the population from which the sample is being drawn. The 2019 replication used an independent statistical group, an independent data safety and monitoring group, an intent-to-treat design, and respecified hypotheses and procedures including prespecified statistical analyses and measures, a level or rigor not required in first trials of a new treatment. The intent-to-treat design involves including all participants originally enrolled in the study at all measurement points, regardless of whether or how much treatment they received. Thus, the children who dropped out of the study early on are included in all measurement points, and their data fully included in the analyses of their group, which can be expected to reduce the effects of the experimental intervention on the group overall. It is for these reasons that the results of replication studies often show smaller group differences than the original studies.

Like drug studies, a new intervention should be compared to the current available treatments in order to assess its possible benefits and costs. In the current study, available community treatment was the comparison condition, However, “available treatment” is certainly not synonymous with “no treatment”, and public community treatments are strongly influenced by the results of treatment studies. High quality ASD early intervention trials have been going on for the past two decades, and their results are steadily improving the state of community practice across the country. Over time, we can expect that most community interventions will be influenced is some ways by research findings and dissemination. Additionally, public information campaigns inform families about best practices in early ASD intervention, about the range of outcomes possible, and about methods for locating and advocating for best practices within community treatment. Public laws have been passed requiring insurance companies to cover expenses of intensive behavioral intervention for young children with ASD, with resulting increased services in many towns and cities across the country. Families have won many lawsuits requiring public schools to implement evidence-based services to children with ASD. Families sign up for research trials with the hope, of gaining access to the best new treatments, and being assigned to a community treatment group rather than the experimental group can be devastating to them. Families may respond to this disappointment by seeking out the very best intervention that they can find (and purchase) in their community. This reaction of comparison families thus changes the nature of the “community treatment” that their child receives from “typical community services” to “enhanced community services”.

Findings from the current study shows these comparison group effects by: (1) the number of families assigned to the control condition who dropped out of the study; (2) the higher-than-expected number of treatment hours that the community treatment group averaged in the study; and (3) a bias in the comparison group families who did not return for assessments towards those with greater severity. We see these effects in the 2019 study when we compare the longitudinal data from this study with the 2010 original study. The 2019 ESDM group made similar developmental progress as was made in the 2010 study, while the comparison group makes much more progress in the 2019 study than was made in the 2010 study. This reduction in the gap between the groups is reflected in the efficacy data.

Taking all this into account, the fact that the 2019 study replicated the significant language advantages associated with ESDM treatment is an important finding, even though there were fewer significant comparisons and site differences existed. An additional meta-analytic paper published recently also supports the efficacy of the ESDM intervention in supporting language and cognitive gains [8].

The methods used in this study were chosen to support the quality of this lab-based clinical trial. The effectiveness of this intervention approach delivered in real-world settings cannot be determined from the data provided in the published study. Community implementation studies use specific research designs and methods, including measures, to examine the effects of an intervention delivered by community agents and within the funding capacity of community agencies to a much wider range of participants than was used in the replication study. Some of the differences In community implementation studies compared to clinical efficacy trials involve the level of staff training and supervision, greater variability among the participant groups, more variability in fidelity of implementation, less rigorous adherence to the intervention protocol, less control over administration of outcome and progress measures - in short - greater degrees of freedom at every level of the study [9]. We cannot assume the generalization of positive effects found in clinical trials to outcomes seen in community usage of an intervention. Yet a large-scale replication clinical trial such as this one provides the evidence needed to justify the need for a community implementation study.

In comparison to past studies documenting minimal changes in functioning of young children with ASD over the first decade or more of life [10,11] rigorous studies of intensive early intervention like this one has documented considerable plasticity in developmental trajectories of language and cognitive development, with children on average gaining 15-20 DQ points in a two-year period [1,3,12,13]. It is quite possible that many young children with ASD and no other significant biological conditions can be ready to attend public school at age 5 with adequate language and cognitive skills to access the curriculum. Early developmental delays do not moderate the effect of high-quality interventions on young autistic children’s language and cognitive progress within high quality interventions [13,14]. It is important that clinicians understand the multiple effects of early autism on early social and play skills, language learning, and cognitive development, and appreciate the plasticity that many young children show in their development in response to high quality learning opportunities via early intervention and parent efforts.

The questions that parents pose when they first receive a diagnosis of ASD for their young child, and the degree of fear and worry that parents express in their questions, often stress clinicians who understand well the range of outcomes that occur in ASD and the multitude of interventions (or lack of them) in communities. One clear recommendation from the findings of this and other high-quality studies of intensive early intervention is to be very cautious in predicting a child’s outcomes and level of impairment over time from first, pre-treatment contact. Children who resemble those in study samples may change substantially over time, in language, social, and intellectual abilities. Child progress over two years of high-quality intervention will provide both clinician and parents with a much better sense of the child’s learning and adaptive abilities than will the initial diagnostic data.

A second recommendation has to do with the nature of intervention being recommended. The strongest data come from intervention studies that apply principles from applied behavior analysis and developmental science. Contrary to many clinical opinions, findings from this study and other recent studies [14] document that children with more severe delays and autism symptoms respond well to a naturalistic, play and routines-based developmental behavioral intervention (NDBI, [2]. Good practice guides families to look for interventions that incorporate behavioral and developmental principles and are delivered by highly trained people, based on high quality evidence, adhere to fidelity of implementation standards, include parent training or coaching as part of the intervention, and focus on embedding delivery across multiple natural environments. It may well be the quality of the interventions other than the specific type (brand-name) of intervention that is most important for child outcomes.

In summary, a large replication study used a rigorous multi-site design to test efficacy of intensive early intervention for toddlers just diagnosed with ASD. The study partially replicated the amount of change from age 2 to age 4 in language development, one of the best predictors of long-term outcomes, in young children with ASD who received a high quality, intensive, evidence-based intervention approach, the Early Start Denver Model, during their toddler and preschool years. The commentary describes several methodological aspects of the study that increase the rigor of the study and also influence the outcomes in terms of site differences, choice of measures, and effects on the treatment-as-usual condition in the midst of a surge of interest and progress in providing and improving interventions for a specific disorder. These three methodological aspects illustrate some of the many challenges that arise in designing and carrying out a large scale, multisite, replication study.