- Research article
- Open Access
- Open Peer Review
Logistics of community smallpox control through contact tracing and ring vaccination: a stochastic network model
© Porco et al; licensee BioMed Central Ltd. 2004
- Received: 03 January 2004
- Accepted: 06 August 2004
- Published: 06 August 2004
Previous smallpox ring vaccination models based on contact tracing over a network suggest that ring vaccination would be effective, but have not explicitly included response logistics and limited numbers of vaccinators.
We developed a continuous-time stochastic simulation of smallpox transmission, including network structure, post-exposure vaccination, vaccination of contacts of contacts, limited response capacity, heterogeneity in symptoms and infectiousness, vaccination prior to the discontinuation of routine vaccination, more rapid diagnosis due to public awareness, surveillance of asymptomatic contacts, and isolation of cases.
We found that even in cases of very rapidly spreading smallpox, ring vaccination (when coupled with surveillance) is sufficient in most cases to eliminate smallpox quickly, assuming that 95% of household contacts are traced, 80% of workplace or social contacts are traced, and no casual contacts are traced, and that in most cases the ability to trace 1–5 individuals per day per index case is sufficient. If smallpox is assumed to be transmitted very quickly to contacts, it may at times escape containment by ring vaccination, but could be controlled in these circumstances by mass vaccination.
Small introductions of smallpox are likely to be easily contained by ring vaccination, provided contact tracing is feasible. Uncertainties in the nature of bioterrorist smallpox (infectiousness, vaccine efficacy) support continued planning for ring vaccination as well as mass vaccination. If initiated, ring vaccination should be conducted without delays in vaccination, should include contacts of contacts (whenever there is sufficient capacity) and should be accompanied by increased public awareness and surveillance.
- Latin Hypercube Sample
- Household Contact
- Secondary Case
- Mass Vaccination
Concerns about intentional releases of smallpox have prompted extensive preparations to improve our ability to detect and respond to an outbreak of smallpox [1, 3, 4, 2]. Many factors contribute to the public health challenge of understanding and preparing for smallpox, including the age and quality of epidemiological data on native smallpox and the smallpox vaccine, the difficulty of extrapolating that data to our current populations, the possible terrorist use of altered smallpox, our ignorance of terrorist methods of release, and the relatively high risk of adverse events caused by the smallpox vaccine.
The Centers for Disease Control and Prevention (CDC) established ring vaccination (selective epidemiological control ), a strategy in which contacts of cases are identified and vaccinated, as the preferred control measure in the event of a smallpox outbreak (interim plan). The successful use of ring vaccination during the smallpox eradication campaign and its logical emphasis of case-contacts for immediate vaccination support its use (though the attribution of the success of the eradication program to ring vaccination has been challenged ). Health Officers should initiate ring vaccination upon identification of the first cases of smallpox. However, there are legitimate concerns regarding the ability of public health practitioners to mount a quick, comprehensive and successful ring vaccination program, particularly in the face of a moderate-sized or large smallpox outbreak. To guide preparation efforts and inform incident decision-making, we attempt to identify outbreak characteristics and response capacities that significantly impact the ability of ring vaccination to control a smallpox outbreak and to determine whether ring vaccination is useful in the presence of a mass vaccination campaign. Our analysis uses a newly developed mathematical model: a continuous-time, event-driven network simulation model of smallpox ring vaccination.
Mathematical models can advance our understanding of how a smallpox outbreak might progress. Several mathematical and computer models address the question of smallpox transmission [7–13]. The first model to appear  concluded that ring vaccination would be effective, but did not treat response logistics in detail; the model was linear and did not treat the depletion of susceptibles as the epidemic progressed (appropriate, however, for assessing control early in an epidemic, when the number infected is small compared to the number of susceptibles, e.g. ). The innovative model by Kaplan et al.  emphasized the importance of resource limitation and the logistics of smallpox response, but assumed that full infectiousness began before the onset of symptoms (and the subsequent identification and removal), and did not separately monitor close epidemiological contacts of patients (which are at greatest risk, but also easiest to find and vaccinate); the conclusions were highly critical of ring vaccination. The model by Halloran et al. , a stochastic, discrete-time network model omitted the explicit inclusion of response logistics while otherwise used parameter values similar to those in Kaplan et al. ; the inclusion of residual immunity from individuals vaccinated prior to the discontinuation of routine vaccination, however, led to a more favorable view of ring vaccination. The model by Bozzette et al.  assumed that ring vaccination would reduce the number of transmissions and focused on health care workers (but did not explicitly include the network structure of the population nor the response logistics of ring vaccination). The model by Eichner  did not explicitly include the network structure of the population nor the logistics of ring vaccination, but did use parameters based on data from an outbreak in Nigeria, and did distinguish close and casual contacts, case isolation, and surveillance of contacts; it concluded that case isolation and contact tracing could prevent the spread of smallpox. Finally, the individual-based model by Epstein et al.  presented scenarios illustrating certain alternatives to pure mass vaccination and ring vaccination of contacts of cases in preventing smallpox transmission in small populations of 800 individuals; this model includes no homogeneity assumptions, but did not analyze tracing of contacts of contacts.
Because none of the available models includes both network structure (with explicit contact tracing) and response logistics limited by the number of available disease control investigators , we included these features in a continuous-time event-driven network simulation model of smallpox ring vaccination. Specifically, the model we developed includes the following features:
Smallpox was primarily a disease of close contact, especially household contacts . Such contacts are both the most important epidemiologically, and also the easiest to identify.
Ring vaccination may involve not only vaccinating contacts of cases, but also contacts of contacts of cases [21, 22] – potentially allowing the public health authorities to "outrun" the chain of transmission.
Limited case-finding and vaccination capabilities lead to the possibility that it may be impossible to find newly exposed individuals and vaccinate them in time, resulting in a "race to trace" .
Heterogeneity in natural history
Mild, ambulatory cases of smallpox may spread disease because such cases may be harder to recognize.
Vaccination of individuals prior to the discontinuation of routine vaccination may provide some, possibly considerable, protection against infection [11, 23, 24], although it may also result in more mild cases which may be harder to detect.
Public awareness may lead to more rapid detection of cases.
We use this model to determine what factors promote or hinder the success of ring vaccination during a smallpox outbreak, and whether ring vaccination is useful in the presence of a mass vaccination campaign. In particular, the goal of this paper is to examine the control of smallpox by contact tracing and ring vaccination using a network model which includes response logistics .
Natural history of smallpox
We briefly review relevant features of the natural history and epidemiology of smallpox [17, 25–27, 8, 28]. Following infection by the variola virus, individuals exhibited an incubation period of approximately 7–19 days with 10–14 being most typical. Sudden onset of fever and malaise, often with accompanying headache and backache, began the initial (or pre-eruptive) phase of smallpox. After 2–3 (or perhaps 4) days, individuals with the most common form, ordinary type smallpox, developed the characteristic focal rash, preceded in many cases by oropharyngeal lesions. In fatal cases of ordinary smallpox, death often occurred between the tenth and sixteenth day of symptoms; among survivors, most scabs had separated by day 22–27 of illness .
The course of smallpox varied widely between individuals, and several different clinical classifications were developed [29–31, 17, 26]. Consideration of the clinical features and severity of smallpox is important from the standpoint of mathematical transmission modeling because (1) the clinical features affect the ease of diagnosis (and thus of case identification), (2) more severe forms of smallpox may result in more transmission, (3) vaccinated individuals may develop less severe disease. We utilize a modified or simplified version of the classification system developed by Rao [32, 31, 26]; for the mathematical model, we will classify smallpox into five categories: early hemorrhagic, flat and late hemorrhagic, ordinary, modified, and mild. However, the clinical features and severity of smallpox in different populations may have been affected by underlying host factors, differences in viral strains, or differences in the infectious dose owing to different prevailing modes of transmission, and thus robust and precise quantitative estimates of the effects of (pre- or post-exposure) vaccination on the resulting smallpox severity, or of the infectivity differences between individuals exhibiting different forms of smallpox, are not available. The significance of such differences will be revealed through sensitivity analysis. Further details are given in Appendix 1 [see Additional file: 1].
Vaccination with vaccinia virus provided substantial protection against infection. Dixon assessed the risk of infection for an individual successfully vaccinated 3 years prior to exposure to be 0.1% the infection risk of an unvaccinated individual . However, smallpox vaccination did not always take when applied, and moreover, in many instances, individuals who experienced a repeated vaccination failure developed severe smallpox upon exposure. The probability of a successful take depended on the vaccination method used; we assume that the take rate is between 95% and 100% [22, 28]. In addition to protection against infection, vaccination could in many cases modify the course of infection and reduce the severity. Vaccine protection waned over time, but individuals vaccinated 20 years prior to exposure were believed to still have half the infection probability that an unvaccinated person had , and to have some protection against the most severe manifestations of smallpox. Dixon  believed that vaccine protection had at least three components, which decayed at different rates; for the purpose of this paper, we will assume that the severity of smallpox in previously any (recently or otherwise) vaccinated individuals follows the same distribution as for the vaccinated subjects seen in the case series observed by Rao in Madras , except that anywhere from 0 to 5% of vaccinated subjects develop smallpox too mild to diagnose without special surveillance or awareness. Observe that the vaccinated cases studied by Rao were vaccinated (at some point in their lives) before exposure, rather than after exposure to smallpox.
Smallpox was largely a disease of close contacts [17, 26, 33], spread primarily through face to face contact with an infected person (or occasionally through contaminated clothing). Individuals in the incubation period of smallpox were not infectious, and long term carriers did not exist. Patients were believed to be infectious following the development of oropharyngeal lesions, which could precede the rash by 24 hours . However, patients were believed to be most infectious during the first week of the rash ; Dixon (1962) believed that patients could be infectious from the onset of acute viremia, but most evidence suggested that little transmission occurred prior to the development of the rash [26, 33]. The more severe the case, the more infectious they appeared to be ; mild cases were believed to have very little infectiousness. While scabs contained infectious material and patients were considered to be infectious until the last scab fell off, in practice patients were not highly infectious during the scabbing phase. Importantly, patients who had been vaccinated were found to cause fewer secondary cases . Very severe cases, such as hemorrhagic or flat smallpox, occasionally resulted in considerable transmission, owing to diagnostic difficulties; mild cases, in which the patient remained ambulant during the course of the disease, could cause considerable spread as well [35, 36]. Within a household or family dwelling, the secondary attack rate of unvaccinated susceptibles depended on the time and place, occasionally below 50% , but often approaching 100% . Drier conditions were often believed to favor transmission [17, 27], so that lower rates of transmission derived from tropical regions may not be applicable to the temperate zone . The number of secondary cases resulting from a given importation into Europe varied widely , with most importations yielding few cases, but with the occasional large outbreak being seen.
We simulate the transmission of smallpox on a "small-worlds" network (highly clustered, but with short characteristic path lengths) . Specifically, we assume that each person is located in a single household, and that the transmission rates were greatest in the household. We also assume that a fraction of the population are grouped into workplace or social groups, in which transmission may also occur, but with a lower rate per unit time than for household contacts. Finally, we assume that with a still smaller probability, any individual may transmit infection to any other individual in the population (casual contacts).
Medical and public health intervention
We assume that even in the absence of specific case investigations, the presence of smallpox symptoms will prompt patients to be diagnosed; we assume, however, a higher diagnosis rate for all forms of ordinary smallpox than for the severe flat and hemorrhagic forms, or for the mildest form. We assume that once an individual is diagnosed, their household and workplace contacts are investigated and detected with some probability; we assume that a high fraction (such as 95%) of household contacts are assumed to be traceable (see below). We assume that the fraction of workplace/social contacts that are traceable is less than the fraction of household contacts that are traceable; we assume that no casual contacts are traceable.
High contact-finding rates may be plausible; we examined San Francisco Department of Public Health records of contact investigations for meningococcal disease (like smallpox, a potentially fatal disease for which rapid intervention may prevent mortality and morbidity). Records were available from December 2001 to April 2002; 13 such investigations during this period resulted in identification of 62 household contacts, all of which were contacted; out of 38 workplace/social contacts identified, 32 were contacted (84%).
In our model, we assume that identified asymptomatic contacts are vaccinated, quarantined, and monitored for symptom development, while symptomatic patients are isolated and treated as necessary ; thus, the modeled interventions include more than ring vaccination alone. Finally, we include the possibility that all contacts (of both symptomatic and asymptomatic) traced and the same procedure applied, so that all contacts of contacts would be investigated. We assume that uninfected or asymptomatic individuals who are visited or traced individuals will be diagnosed more rapidly than if they had not been traced; in fact, such individuals would be isolated and would not be able to continue a chain of transmission. We follow previous models  in assuming a limited vaccination capability of K r per day for ring vaccination. We assumed one of two strategies for contact tracing: (1) tracing only of direct contacts of diagnosed cases, and (2) tracing of contacts of contacts of diagnosed cases as well as direct contacts.
We analyzed the model in three ways. First, we selected a Latin Hypercube sample [44–46] of parameters chosen uniformly from the parameter ranges given in Appendix 2 [see additional file 2], and simulated the transmission and control of smallpox to determine which parameters were most important for contact tracing and ring vaccination to be effective. Second, we used the same Latin Hypercube Sample of input parameters, but assumed that all disease control efforts were inactive. We used these parameters to simulate smallpox transmission, but then iteratively selected transmission parameters so that (1) between 1% and 10% of new infections resulted from casual (random) transmission, and (2) each index case resulted in between two and five secondary cases (thought to be plausible for historic smallpox;  suggest three secondary cases). For each of the resulting smallpox parameter sets using 100 stochastic simulations per set, we determined the daily ring vaccination/case tracking capacity needed to contain all simulated smallpox epidemics (i.e., keep the total number of cases below 500 within 250 days). Third and finally, we chose parameter values to yield an moderately large smallpox epidemic (with each index case causing approximately six secondary cases), and present illustrative scenarios for ring vaccination. These scenarios are intended to complement the simulations which were calibrated to historic smallpox, since the characteristics of smallpox that may be used in a deliberate release are not known. It is important to realize that in our model, the case finding time determines the fraction of contacts that will become infected, and that our model parameters have been chosen to yield quite rapid transmission to close contacts; in reality, much transmission of natural smallpox occurred through "sickbed" routes which would not occur in a modern setting , so that in this sense our model errs considerably on the side of caution and pessimism.
Most important parameters (sensitivity analysis)
Most important parameters. PRCC: partial rank correlation coefficient (see Appendix 2 [see additional file 2] for definition and references).
Mean Household Size
Transmission Rate from Close Contacts
Infectivity prior to rash
Ring Vaccination Capacity
Casual Transmission Probability
Pre-eruptive infectious period (lower bound)
Number of Casual Contacts per Day
Relative Infectiousness of Social/Workplace Contacts
Fraction of Individuals in Social/Workplace Groups
Faster Diagnosis due to Awareness of Outbreak
Household Tracing Delay
Pre-eruptive Diagnosis Probability
Diagnosis Probability after Rash
To explore factors which contribute to the success of ring vaccination, we chose smallpox scenarios which resulted in severe and fast-moving epidemics in the absence of disease control; these simulated epidemics are considerably more severe than is believed likely under present circumstances.
Effect of contact tracing and ring vaccination
Selected parameter values for Figure 3A and other illustrative scenarios. The notations "Other" or "Other factors" in the column "See also" refers to the text section "Other factors". The symbols are defined in Appendix 2 [see additional file 2] and are included for reference.
Number of index cases
Mean household size
Workplace/social group size
Ring vaccinations per day
Fig. 3A, 3B, Other
Monitored diagnosis rate
Figure 5A, 5B
Prob. of finding household contact
Prob. of finding workplace/social contact
Fig. 3B, 4AB; Tb. 4
Delay, tracing household contacts
Delay, tracing workplace/social contacts
Relative diagnosis rate after 1st diagnosed case
Infectivity, stage 4 relative to stage 5
Infectivity, stage 5 relative to stage 6
Infection hazard for close contacts
Relative hazard for workplace/social contacts
Casual transmission rate
Prior vaccination fraction
Fraction of mild cases
Vaccine success rate (for very recent vaccination)
Vaccine success rate (vaccination prior to discontinuation of routine vaccination)
Vaccine success rate full protection
Because we assumed nonzero diagnosis probabilities during the prodromal period for all individuals in Figure 3A, we repeated the simulation assuming no diagnosis in the prodromal period unless individuals were under specific surveillance. The results were nearly identical: assuming 30 contact tracings (ring vaccinations) per day, we found 26% of the scenarios in Figure 3A exhibited decontainment, and 28% assuming no diagnosis during the prodromal period; assuming 40 contact tracings per day, we found 1 out of 100 scenarios showed loss of containment in Figure 3A and when we repeated the scenario of Figure 3A assuming no diagnosis during the prodromal period.
In Figure 3B, we illustrate control of an epidemic for which all parameters are identical to Figure 3A, except that the workplace/social group size is 12 (instead of 8, as in Figure 3A), and the probability of finding workplace/social contacts is 60% (instead of 80%, as in Figure 3A). In this case, the larger size of the workplace/social groups and the lower contact finding probability makes it necessary to have a higher ring vaccination capacity to attain a high probability of containing the epidemic, and on average it takes longer for eradication to finally occur.
Finally, in Figure 3C, we show control of an epidemic in a population of 100,000, beginning with 1000 initial infectives, keeping all other parameters the same as in Figure 3A. Each curve corresponds to the indicated number of possible ring vaccinations per day. This figure shows that assuming sufficient capacity, ring vaccination is in principle capable of containing even epidemics beginning with very many infected individuals. However, mass vaccination in such cases is justified because of the far larger number of individuals at risk and the inability to perform such extensive contact tracing.
In Figure 3D, we compare the effect of tracing contacts of contacts (as described in Appendix 2 [see additional file 2]) at different levels of ring vaccination capacity. Thin lines in red correspond to the presence of tracing contacts of contacts; thick lines in black correspond to tracing direct contacts of cases only. Each simulation was performed 100 times, with 10 initial infectives, and for 20 and 40 ring vaccinations possible per day (as indicated). The average number infected on each day is plotted in the Figure. The figure illustrates that when ring vaccination capacity is low, tracing contacts of contacts (as modeled) yields a more severe average epidemic; when ring vaccination capacity is large, tracing contacts of contacts results in a less severe average epidemic; if the contact tracing/ring vaccination capacity is too low to cover adequately the contacts of contacts in addition to the contacts of cases, extension of tracing to the contacts of contacts (the second ring) is harmful; however, if there is sufficient capacity to cover the contacts of contacts, then the tracing of contacts of contacts is beneficial.
Estimated decontainment probability for different levels of ring vaccination capacity (K r ) and relative hazard for infection due to workplace/social contacts (h 2), for different levels of the baseline hazard for infection from household contacts λ (based on replications of 100 simulations for each level). For each scenario, 10 index cases were introduced into a population of size 10000. All other parameters were the same as for Figure 3A. As before, we define decontainment to mean that the total number of cases from 10 index cases eventually exceeded 500 by day 250.
Relative hazard for workplace
or social contacts
Ring vaccinations per day
Because of considerable uncertainty in the model parameters, we chose a collection of parameter values, and for each, estimated the containment probability (operationally defined as fewer than 500 total cases as a result of 10 index cases, within 250 days). We estimated this containment probability by simulating the smallpox epidemic 100 times for the same parameter values, and computing the frequency out of these 100 realizations for which fewer than 500 index cases resulted within 250 days. (Using a 1000 day window produces slightly smaller containment estimates; for 3 out of 1000 parameter set choices, this difference was greater than 0.06; the maximum difference seen was 0.23; the mean absolute difference was 0.0029; in only one case out of 1000 did we see containment in all 100 cases for the 250-day window, but not in all 100 cases for the 1000-day window).
One thousand scenarios chosen from a Latin Hypercube sample were analyzed, and as indicated before, we chose the hazard for close contact transmission and the hazard for random transmission to guarantee that between 2 and 5 secondary cases per case occur, and that no more than 5% of cases are attributable to random transmission (we refer to this set as the "calibrated" scenarios further in this text). Having chosen this collection of 1000 parameter sets, we considered two levels of two different control parameters which were applied to each (so that each of the 1000 parameter sets were simulated under four different control conditions). The first of the two control parameters was the probability of workplace/social group contact finding; we chose values of 0.8 and 0.9 for this parameter (the household contact finding probability was 0.95 in all cases). The second of the control parameters was the rate of diagnosis (and effective removal) from the community of cases developing among previously identified and traced contacts who were initially asymptomatic (we refer to this as the monitored diagnosis rate); we assumed first a low level corresponding to a mean diagnosis time of one day from the onset of symptoms, and a high level corresponding to a mean time of 3 hours from the onset of symptoms (high levels of the monitored diagnosis rate correspond effectively to isolation of contacts). Finally, we assumed a doubling of the diagnosis rate after the beginning of widespread community awareness of smallpox. We then computed the containment fraction at different levels of ring vaccination capacity (contact tracing capacity per day). Thus, for each of 1000 scenarios (parameter set choices), we assigned the workplace/social group contact tracing success probability (υ2), the monitored diagnosis rate φ (Appendix 2 [see additional file 2]), and the contact tracing/ring vaccination capacity per day (K r ). We then performed 100 realizations beginning with 10 index cases, and computed the containment fraction (fraction showing fewer than 500 cases in 250 days, beginning with 10 index cases). Thus, for each of the two choices each of υ2 and φ, and for each value of K r we examined, we obtained 1000 values of the containment fraction. We use the resulting distributions in Figure 5A (averaging over these 1000 containment fractions), and Figure 5B (displaying the minimum value of the 1000 containment fractions).
In Figure 5A, we plot the mean containment fraction (averaging the containment fraction over all 1000 scenarios), as ring vaccination capacity varies, for the two levels of workplace/social group contact finding probabilities (0.8 and 0.9), and for the two levels of monitored diagnosis rate among initially asymptomatic contacts (1 day-1 and 8 day-1). For low levels of ring vaccination (traceable contacts per day), the epidemic is almost never contained, but for ring vaccination levels near 50–60 per day (5–6 per index case per day), the average containment fraction became close to 1. However, this average conceals the fact that for some scenarios (parameter sets chosen from the calibrated uncertainty analysis), control remains difficult or impossible even at high levels of ring vaccination. Therefore, in Figure 5B, we plotted the single lowest containment fraction seen out of the 1000 computed; focusing on the single worst scenarios reveals a different picture, and shows that isolation of asymptomatic contacts and very high probabilities of finding workplace or social contacts would be needed to control smallpox under these most pessimistic parameter choices.
Effect of contact tracing speed
While Figure 6 illustrates the possibility that rapid contact tracing may be of decisive importance in some scenarios (parameter set choices), this is not always the case. For some parameter sets, the probability of tracing contacts (household or workplace/social) may be too low, or the transmission rate too high, for more rapid contact tracing to make any difference. Conversely, for other parameter sets, the smallpox transmission rate may be so low that smallpox is easily contained even with slow contact tracing. While rapid contact tracing is never harmful, overall, how typical are the results of Figure 6 (in which rapid contact tracing was important in ensuring the efficacy of ring vaccination)? To address this question, we simulated the growth of smallpox for the 1000 "calibrated" scenarios we used in Figure 5A and 5B. As before, we assumed ten initial cases, and (as in Figure 6) that 30 ring vaccinations were possible per day; then we simulated 100 epidemics assuming one day to find a household contact (and 2 days to find a workplace/social contact). We then simulated 100 epidemics assuming that it takes five days to find a household contact and 10 days to find a workplace/social contact (as in Figure 6). For each of these 1000 scenarios, we calculated the fraction of simulations for which the total number of cases is 500 or less within 250 days, i.e. the containment fraction. For nearly all scenarios (parameter set choices), the containment fraction was smaller (sometimes much smaller) when the contact finding time is faster (since faster contact finding, all else being equal, improves smallpox control, as illustrated in Figure 6). However, for 64.5% of the scenarios (parameter set choices) examined, the difference was less than 2.5% in absolute terms (smallpox was either contained or not contained depending on other factors, and rapid contact tracing did not make the difference). On the other hand, for 18.7% of the scenarios examined, the absolute difference in the containment probability was 20% or more; thus, a substantial difference in containment probability is occasionally attributable to the difference between fast and slow contact tracing.
Effect of more rapid diagnosis
In many cases, however, more rapid diagnosis was not required for ring vaccination to be effective. As before, we simulated smallpox epidemics for each of 1000 calibrated scenarios, performing 100 realizations each beginning with 10 index cases, and computed the fraction of scenarios for which the epidemic was always contained (as defined earlier), assuming no change in diagnosis rates. We assumed 80 ring vaccinators per day, contact finding probabilities of 0.95 for households and 0.8 for workplace/social contacts (as in Figure 3A). Under these assumptions, for 83.4% of the scenarios, the epidemic was contained within 500 total cases in each of the 100 realizations, even with no change in diagnosis rates. Uncertainty analysis (using the 1000 calibrated scenarios, and based on the fraction of 100 replications showing decontainment) revealed the most important parameters which predict the failure of ring vaccination without more rapid diagnosis were the same as we found in the earlier uncertainty analysis; a higher fraction vaccinated before the epidemic, smaller households or workplace/social groups, less transmissibility, lowered infectivity prior to the rash, more rapid diagnosis, and a higher rate of diagnosis for alerted individuals all contribute to a greater containment probability even without an overall increase in the diagnosis rate.
Effect of continued surveillance of contacts
We have been assuming that whenever an individual is contacted during an investigation, the individual will be diagnosed more quickly should they subsequently develop symptoms. When transmission is assumed to be very rapid (smallpox is assumed to be highly contagious), most individuals may already be infected when identified through contact tracing from an infective. Using the scenario we examined in Figure 3A, we see that continued surveillance of contacts is an essential component of effective ring vaccination designed to control rapidly spreading smallpox: if smallpox in a contact is not diagnosed any more quickly than for a non-contact, containment by ring vaccination requires over 98% contact finding probabilities for both household and workplace/social contacts – even if unlimited numbers of ring vaccinators are available; containment cannot be guaranteed by adding additional ring vaccination capacity if the contact finding rates are too low and/or the follow-up for contacts is insufficient. Smallpox which is transmitted less rapidly to contacts would, however, be containable with a lower contact finding probability (results not shown).
Containment of severe smallpox at different levels of contact finding. The first three columns are assumed levels for the probability of finding a household contact, the probability of finding a workplace/social (W/S) contact, and for the number of contact tracings/ring vaccinations possible per day; the last two columns express (as percentages) the resulting probability of containment given the assumed contact finding probabilities and contact tracing capacities; two containment probabilities are given: the containment probability when only contacts of cases are traced (first column, "Contacts"), and the containment probability when contacts of contacts of cases are traced in addition to the contacts of cases (second column, "Contacts of Contacts"). All other parameters are the same as in Figure 3A.
Probability of finding
Number of Ring vacc. per day
Probability when Tracing Contacts of contacts
Transmission prior to rash
Scenario b was motivated by the possibility that individuals with severe forms of smallpox may be more difficult to diagnose, and thus remain infectious in the community longer (despite the much greater degree of illness of such patients), or that such patients may be more infectious. In this particular case, quadrupling the mean diagnosis time led to one additional replication out of 100 in which containment was not achieved (2/100, compared to the baseline of 1/100). However, we assumed that community awareness of smallpox leads to the same relative rate of increased diagnosis among severe cases as for ordinary cases, and that the most severe forms are relatively rare. In addition to the scenario shown in the figure, we also replicated the same 1000 "calibrated" simulations, assuming that in each case 40 contact tracings per day are possible and that the diagnosis time for severe cases was four times that of ordinary cases. Finally, we repeated each "calibrated" scenario 100 times assuming long diagnosis times for severe cases, and not making this assumption, and found that the difference in the decontainment fraction was not large (results not shown).
Scenario c illustrates that vaccination prior to the discontinuation of routine vaccination does play a role in smallpox control by ring vaccination; there were more decontainment scenarios (5/100) when no prior protection exists in the population. The results suggest that prior vaccination aids in the control of smallpox, but that it is not strictly necessary for control (in this scenario, 95% of the replications exhibited containment). In Figure 3A, we assumed 25% of individuals had protection due to vaccination prior to the discontinuation of routine vaccination; in scenario c of Figure 9, we assumed this fraction was zero.
Scenario d demonstrates that if 10% more smallpox infections (in absolute terms, i.e. 13% compared to 3% in Figure 3A) lead to mild cases among individuals with no prior protection, the epidemic is more difficult to contain (13/100 replications showed loss of containment).
Finally, scenario e demonstrates that containment is still possible even when the vaccine is completely ineffective in everyone – because of case isolation and isolation of contacts (and of contacts of contacts). Here, with 40 contact tracings possible per day, 55% of the replications nevertheless exhibited containment even with a vaccine which offered no protection whatever. With 90 contact tracings possible per day, all replications exhibited containment even assuming no vaccine protection.
Effect of mass vaccination
We constructed a simple network model of smallpox transmission, and addressed the question of what circumstances contribute to the success of a ring vaccination campaign designed to control smallpox. Our analysis focused on the use of contact tracing/ring vaccination to prevent a widespread epidemic following a deliberate release.
We conducted a sensitivity analysis based on particular, but reasonable, ranges for the unknown parameters. Our results are consistent with prior vaccination models in identifying prior vaccination and ring vaccination capacity as significant factors in determining the spread of smallpox. Unsurprisingly, we also find that household size and ring vaccination speed are particularly important parameters; these results are intuitively plausible. The contact finding probability did not appear important in this analysis only because a narrow range of values was chosen.
We illustrated smallpox control by presenting scenarios based on control of moderately severe smallpox epidemics. We find that swift, aggressive contact tracing and ring vaccination is is usually sufficient to bring the infection under control. Provided that there is sufficient capacity, vaccination of contacts of contacts is beneficial, and results in fewer infected individuals and more rapid elimination of infection; investigating contacts of contacts allows the chain of transmission to be outrun to some extent. When ring vaccination capacity is small, diversion of crucial resources away from contacts is harmful; contacts of contacts should only be traced and vaccinated provided that no resources are diverted away from contacts of cases. The increased surveillance (or isolation) of contacts, together with improved rates of diagnosis due to community awareness, play important roles in smallpox control; we note that in some cases, lowered diagnosis rates among severe cases contributed to a small extent to loss of epidemic control, and suggest that any public awareness campaign include information to help the public be more aware of the full spectrum of the clinical features of smallpox.
One limitation of our analysis is that we chose not to explicitly incorporate the specific epidemiology of health care workers (or mortuary workers), who are likely to be exposed to infected individuals during any smallpox epidemic (e.g. [17, 22]), and who may then infect further members of the community  (as was also seen in the recent outbreak of SARS, e.g. ). Transmission to health care workers may be considered to amplify the initial attack or to be simply accounted among the exposures we considered (and thus be approximated by the behavior of our model), since health care workers and their household contacts are in all likelihood traceable contacts, and ring vaccination/contact tracing would identify and halt these chains of transmission as in our model. The disruption of smallpox control and patient care that may occur is not accounted for in our analysis, however, causing our model in this sense to err on the side of optimism. The appropriateness of pre-event vaccination of health care workers or other first responders has been addressed by other analyses [12, 49], and is beyond the scope of our model.
While we analyzed the effect of contact tracing, case and contact isolation, and ring vaccination (together with mass vaccination), in a real smallpox epidemic, in practice, control efforts are unlikely to be limited strictly to vaccinating contacts (and health care workers, as likely contacts) and isolating cases. Indeed, making vaccine available to individuals who believe they live near cases or to others on a voluntary basis occurred in smallpox control efforts in the past . Vaccination of such individuals can only harm the disease control effort if it hinders or delays the diagnosis of cases or the investigation and vaccination of contacts; our results show that even relatively low levels of vaccination of the general population may have a beneficial effect in preventing the epidemic from escaping control.
More serious is the possibility that individuals who should be vaccinated or isolated would be missed; this could occur either because individuals or institutions did not cooperate with the disease control effort, or because the individuals simply could not be found. Our analysis suggests that ring vaccination need not be perfect to successfully contain the epidemic, and yet, under conditions where there is a high rate of infection among contacts, or a relatively high rate of casual transmission, high rates of contact finding (in excess of 90%), together with increased surveillance and contact isolation, are needed to contain the epidemic.
Finally, the vaccination of individuals at low risk of contracting smallpox will cause harm due to adverse events of the vaccine; in our model, the assumed death rate due to vaccination was small compared to the probability of death from smallpox, and played essentially no role in the analysis. In practice, individuals suspected to be at high-risk for vaccine complications, but at relatively low risk for contracting smallpox, might simply be isolated or closely monitored even during an outbreak; while the presence of individuals in the population at higher risk for vaccine complications would increase the death rate during an outbreak, such individuals are unlikely to impair the containment of the epidemic (the primary focus of this analysis).
Our results support ring vaccination against epidemics of smallpox (even assuming high rates of transmission to close contacts), but do note that stochastically, for severe (rapidly transmissible) smallpox, scenarios of loss of control are seen, with resulting widespread epidemics. In scenarios in which the transmission potential of smallpox is smaller, such loss-of-control scenarios occur less frequently (results not shown). Mass vaccination campaigns, when conducted quickly and with very high coverage, do not result in loss of control in our model. Nevertheless, fewer deaths due to smallpox result when ring vaccination is conducted along with mass vaccination.
Simulated smallpox epidemics with ring vaccination suggest that aggressive, fast ring vaccination can control epidemics of smallpox. To do so, however, smallpox must be identified quickly and contacts vaccinated promptly. We also identify public awareness of smallpox – leading to prompt identification of cases – as a major factor in smallpox control; in some simulations, it may play a role as significant as ring vaccination itself . However, we also found that uncertainty in (1) transmission from mild cases, (2) the household size, and (3) casual transmission contributed to the overall uncertainty in the epidemic size. Other parameters to which the number of infections were highly sensitive were the prior vaccination fraction, parameters related to infectiousness, and parameters related to transmission prior to the rash.
Because our model combines network structure with response logistics, our results support and complement the results of other investigators. Our results support the notion that prior vaccine protection may play an important role in slowing the epidemic , despite the possibility that some vaccinated individuals may develop mild cases which are harder to identify, but which nevertheless transmit disease. Likewise, our results provide support for the view that ring vaccination should play a central part in smallpox control. If initiated, ring vaccination should be conducted without delays in vaccination, should include contacts of contacts (whenever there is sufficient capacity to cover all contacts of cases), and should be accompanied by a vigorous campaign of public awareness which can facilitate more rapid identification and isolation of cases. We assumed that ring vaccination could be fast (little delay between identification of a case and vaccination of the contacts), effective (nearly all household contacts can be found, and most of workplace and social contacts), and available (there is sufficient capacity). To be effective, ring vaccination planning must yield a system capable of meeting these benchmarks; we should not only be able to assess the number of contact vaccinations that will be possible per day, but should have a plan in place to (1) identify contacts by working with individuals, employers, schools, community representatives, and authorities or businesses who may have access to information facilitating contact tracing, (2) rapidly investigate and vaccinate such individuals, perhaps using field teams managed by central dispatch. It is important to realize that for high-risk, transient, or unstably housed populations where reliable contact tracing is impossible, the conclusions of the model we present cannot be applied. It is important to note that while our model suggests that ring vaccination together with contact tracing and isolation is likely to be successful, we found that for some scenarios (where smallpox was more transmissible, or was relatively more transmissible before the rash), epidemic containment required not only ring vaccination, but increased public awareness, the isolation of contacts, and tracing of contacts of contacts. For scenarios in which the smallpox was less transmissible, epidemic containment was possible at lower contact finding probabilities. Thus, while our simulations suggest that contact tracing/ring vaccination need not be perfect to succeed, because of uncertainties in our knowledge of the behavior of bioterrorist smallpox, it is impossible to know in advance how good it will have to be. Thus, that high contact finding rates, mass public awareness leading to early identification of cases, isolation of contacts, and investigation of contacts of contacts should all be conducted with maximum effectiveness to reduce the probability of a widespread epidemic.
While the possibility of smallpox uncontrollable by ring vaccination has made mass vaccination preparations wise, and while mass vaccination may be unavoidable in the event of a deliberate release of smallpox, we believe that ring vaccination is essential in any case. This is not only because individuals recently exposed to smallpox may be protected if they are vaccinated promptly, but because each contact identified potentially lies in the immediate future of the transmission chain. From the standpoint of epidemic control, it is far more valuable to vaccinate individuals next in the transmission chain than to vaccinate other persons. Our results support the idea that ring vaccination/case isolation may in many, if not most cases, eliminate smallpox even without mass vaccination, but also support planning for mass vaccination (so that the vastly more costly and difficult policy of mass vaccination will be available in the event of an explosive epidemic). When faced with the unknown, multiple redundant preparations are appropriate; case investigation/isolation may control smallpox even if the vaccine does not work at all, but mass vaccination is useful in the event of an explosive epidemic for which case tracking becomes impossible.
TCP acknowledges the support of NIDA grant DA 5-R01-13510. The authors gratefully acknowledge the support of the computing facility at the Stanford Linear Accelerator Center, Stanford University, and thank Richard Mount, Randy Melen, and Tor Raubenheimer for their assistance.
- Alibek K: Biohazard. 1999, New York: Random HouseGoogle Scholar
- Koopman J: Controlling smallpox. Science. 2002, 298: 1342-1344. 10.1126/science.1079370.View ArticlePubMedGoogle Scholar
- Henderson DA: The looming threat of bioterrorism. Science. 1999, 283: 1279-1282. 10.1126/science.283.5406.1279.View ArticlePubMedGoogle Scholar
- O'Toole T: Smallpox: an attack scenario. Emerging Infectious Diseases. 1999, 5: 540-546.View ArticlePubMedPubMed CentralGoogle Scholar
- Foege WH, Millar JD, Lane JM: Selective epidemiologic control in smallpox eradication. American Journal of Epidemiology. 1971, 94: 311-315.PubMedGoogle Scholar
- Kaplan EH, Wein LM: Smallpox eradication in West and Central Africa: surveillance-containment or herd immunity?. Epidemiology. 2003, 14: 90-92. 10.1097/00001648-200301000-00021.View ArticlePubMedGoogle Scholar
- Gani R, Leach S: Transmission potential of smallpox in contemporary populations. Nature. 2001, 414: 748-751. 10.1038/414748a.View ArticlePubMedGoogle Scholar
- Meltzer MI, Damon I, LeDuc JW, Millar JD: Modeling potential responses to smallpox as a bioterrorist weapon. Emerging Infectious Diseases. 2001, 7: 959-969.View ArticlePubMedPubMed CentralGoogle Scholar
- Kaplan EH, Craft DL, Wein LM: Emergency response to a smallpox attack: the case for mass vaccination. Proc Nat Acad Sci. 2002, 99: 10935-10940. 10.1073/pnas.162282799.View ArticlePubMedPubMed CentralGoogle Scholar
- Kaplan EH, Craft DL, Wein LM: Analyzing bioterror response logistics: the case of smallpox. Mathematical Biosciences. 2003, 185: 33-72. 10.1016/S0025-5564(03)00090-7.View ArticlePubMedGoogle Scholar
- Halloran ME, Longini IM, Nizam A, Yang Y: Containing bioterrorist smallpox. Science. 2002, 298: 1428-1432. 10.1126/science.1074674.View ArticlePubMedGoogle Scholar
- Bozzette SA, Boer R, Bhatnagar V, Brower JL, Keeler EB, Morton SC, Stoto MA: A model for a smallpox-vaccination policy. New England Journal of Medicine. 2003, 348: 416-425. 10.1056/NEJMsa025075.View ArticlePubMedGoogle Scholar
- Ferguson N, Keeling MJ, Edmunds WJ, Gani R, Grenfell BT, Anderson RM, Leach S: Planning for smallpox outbreaks. Nature. 2003, 425: 681-685. 10.1038/nature02007.View ArticlePubMedGoogle Scholar
- Porco TC, Small PM, Blower SM: Amplification dynamics: predicting the effect of HIV on tuberculosis outbreaks. Journal of Acquired Immune Deficiency Syndromes. 2001, 28: 437-444.View ArticlePubMedGoogle Scholar
- Eichner M: Case isolation and contact tracing can prevent the spread of smallpox. American Journal of Epidemiology. 1971, 158: 118-128. 10.1093/aje/kwg104.View ArticleGoogle Scholar
- Epstein JM, Cummings DAT, Chakravarty S, Singa RM, Burke DS: Toward a containment strategy for smallpox bioterror: an individual-based computational approach. Center on Social and Economic Dynamics Working Paper No. 31. 2002, [http://www.brook.edu/dybdocroot/es/dynamics/papers/bioterrorism.pdf]Google Scholar
- Dixon CW: Smallpox. 1962, London: J & A ChurchillGoogle Scholar
- Massoudi MS, Barker L, Schwartz : Effectiveness of postexposure vaccination for the prevention of smallpox: results of a Delphi analysis. Journal of Infectious Diseases. 2003, 188: 973-976. 10.1086/378357.View ArticlePubMedGoogle Scholar
- Mortimer PP: Can postexposure vaccination against smallpox succeed?. Clinical Infectious Diseases. 2003, 36: 622-629. 10.1086/374054.View ArticlePubMedGoogle Scholar
- Bicknell WJ, James K: Smallpox vaccination after a bioterrorism-based exposure. Clinical Infectious Diseases. 2003, 37: 467-10.1086/376650.View ArticlePubMedGoogle Scholar
- Centers for Disease Control and Prevention: Interim smallpox response plan and guidelines. 2002Google Scholar
- Zetterberg B, Ringertz O, Svedmyr A, Wallmark G, Alin K: Epidemiology of smallpox in Stockholm 1963. Acta Med Scand Suppl. 1966, 464: 9-42.Google Scholar
- Eichner M: Analysis of historical data suggests long-lasting protective effects of smallpox vaccination. American Journal of Epidemiology. 2003, 158: 717-723. 10.1093/aje/kwg225.View ArticlePubMedGoogle Scholar
- Hsieh SM, Pan SC, Chen SY, Huang PF, Chang SC: Age distribution for T-cell reactivity to vaccinia virus in a healthy population. Clinical Infectious Diseases. 2004, 38: 86-89. 10.1086/380460.View ArticlePubMedGoogle Scholar
- Benenson AS: Smallpox. In Communicable and infectious diseases. Edited by: Top FH, Wehrle PF. 1976, St. Louis: CV Mosby, 623-635.Google Scholar
- Fenner F, Henderson DA, Arita I, Ježek Z, Ladnyi LD: Smallpox and its eradication. 1988, Geneva: World Health OrganizationGoogle Scholar
- Henderson DA, Inglesby TV, Bartlett JG, Ascher MS, Eitzen E, Jahrling PB, Hauer J, Layton M, McDade J, Osterholm MT, O'Toole T, Parker G, Perl T, Russell PK, Tonat K: Smallpox as a biological weapon. Medical and public health management. Journal of the American Medical Association. 1999, 281: 2127-2137. 10.1001/jama.281.22.2127.View ArticlePubMedGoogle Scholar
- Breman JG, Henderson DA: Diagnosis and management of smallpox. New England Journal of Medicine. 2002, 346: 1300-1308. 10.1056/NEJMra020025.View ArticlePubMedGoogle Scholar
- Vaccination Commission: Final Report of the Royal Commission appointed to inquire into the subject of Vaccination. 1896, London: Her Majesty's Stationery OfficeGoogle Scholar
- Thomas DB, McCormack WM, Arita I, Khan MM, Islam MS, Mack TM: Endemic smallpox in rural East Pakistan. I. Methodology, clinical and epidemiologic characteristics of cases, and intervillage transmission. American Journal of Epidemiology. 1971, 93: 361-372.PubMedGoogle Scholar
- Rao AR: Smallpox. 1972, Bombay: Kothari Book DepotGoogle Scholar
- Rao AR, Jacob ES, Kamalakshi S, Appaswamy S, Bradbury : Epidemiological studies of smallpox. a study of intrafamilial transmission in a series of 254 infected families. Indian Journal of Medical Research. 1968, 56: 1826-1854.PubMedGoogle Scholar
- Eichner M, Dietz K: Transmission potential of smallpox: estimates based on detailed data from an outbreak. American Journal of Epidemiology. 1971, 158: 110-117. 10.1093/aje/kwg103.View ArticleGoogle Scholar
- Heiner GG, Fatima N, McCrumb FR: A study of intrafamilial transmission of smallpox. American Journal of Epidemiology. 1971, 94: 316-326.PubMedGoogle Scholar
- de Jongh CL: Over zeer lichte gevallen van pokken en hun beteekenis voor de verspreiding dezer ziekte. Nederlandsch Tijdschrift voor Geneeskunde. 1929, 73: 5709-5719.Google Scholar
- Litvinjenko S, Arsić B, Borjanović S: Epidemiologic aspects of smallpox in Yugoslavia in 1972. World Health Organization WHO/SE/73.57. 1973Google Scholar
- Mack TM, Thomas DB, Ali A, Khan MM: Epidemiology of smallpox in West Pakistan. I. Acquired immunity and distribution of the disease. American Journal of Epidemiology. 1972, 95: 157-168.PubMedGoogle Scholar
- Thomas DB, Arita I, McCormack WM, Khan MM, Islam MS, Mack TM: Endemic smallpox in rural East Pakistan. II. Intravillage transmission and infectiousness. American Journal of Epidemiology. 1971, 93: 373-383.PubMedGoogle Scholar
- Mack TM: Smallpox in Europe, 1950–1971. Journal of Infectious Diseases. 1972, 125: 161-169.View ArticlePubMedGoogle Scholar
- Watts D: Small worlds. 1999, Princeton: Princeton University PressGoogle Scholar
- Becker NG, Dietz K: The effect of household distribution on transmission and control of highly infectious diseases. Mathematical Biosciences. 1995, 127: 207-219. 10.1016/0025-5564(94)00055-5.View ArticlePubMedGoogle Scholar
- Ball F, Mollison D, Scalia-Tomba G: Epidemics with two levels of mixing. Annals of Applied Probability. 1997, 7: 46-89. 10.1214/aoap/1034625252.View ArticleGoogle Scholar
- Eisenberg JNS, Lewis BL, Porco TC, Hubbard AH, Colford JM: Bias due to secondary transmission in estimation of attributable risk from intervention trials. Epidemiology. 2003, 14: 442-450. 10.1097/00001648-200307000-00013.PubMedGoogle Scholar
- Iman RL, Helton JC, Campbell JE: An approach to sensitivity analysis of computer models: Part I – Introduction, input variable selection and preliminary variable assessment. Journal of Quality Technology. 1981, 13: 174-183.Google Scholar
- Iman RL, Helton JC, Campbell JE: An approach to sensitivity analysis of computer models: Part II – Ranking of input variables, response surface validation, distribution effect, and technique synopsis variable assessment. Journal of Quality Technology. 1981, 13: 232-240.Google Scholar
- Blower SM, Dowlatabadi H: Sensitivity and uncertainty analysis of complex models of disease transmission: an HIV model, as an example. International Statistical Review. 1994, 62: 229-243.View ArticleGoogle Scholar
- Mack TM, Thomas DB, Khan MM: Epidemiology of smallpox in West Pakistan. II. Determinants of intravillage spread other than acquired immunity. American Journal of Epidemiology. 1972, 95: 169-177.PubMedGoogle Scholar
- Lloyd-Smith JO, Galvani AP, Getz WM: Curtailing transmission of severe acute respiratory syndrome within a community and its hospital. Proc R Soc Lond Biol Sci. 2003, 270: 1979-1989. 10.1098/rspb.2003.2481.View ArticleGoogle Scholar
- Meltzer MI: Risks and benefits of preexposure and postexposure smallpox vaccination. Emerg Infect Dis. 2003, 9: 1673-1670.View ArticleGoogle Scholar
- Dixon CW: Smallpox in Tripolitania, 1946: an epidemiological and clinical study of 500 cases, including trials of penicillin treatment. Journal of Hygiene (Cambridge). 1948, 46: 351-377.View ArticleGoogle Scholar
- Chapin CV: Variation in type of infectious disease as shown by the history of smallpox in the United States, 1895–1912. Journal of Infectious Diseases. 1913, 13: 171-196.View ArticleGoogle Scholar
- Adams F: The epidemic of virulent smallpox in Windsor and the vicinity. Canadian Medical Association Journal. 1924, 14: 692-696.PubMedPubMed CentralGoogle Scholar
- Glasserman P, Yao DD: Monotone structure in discrete-event systems. 1994, New York: J WileyGoogle Scholar
- Henderson RH, Yekpe M: Smallpox transmission in southern Dahomey. American Journal of Epidemiology. 1969, 90: 423-428.PubMedGoogle Scholar
- Aragón TJ, Ulrich S, Fernyak S, Rutherford GW: Risks of serious complications and death from smallpox vaccination: a systematic review of the United States experience, 1963–1968. BMC Public Health. 2003, 3: 26-10.1186/1471-2458-3-26.View ArticlePubMedPubMed CentralGoogle Scholar
- Blower SM, McLean AR: Prophylactic vaccines, risk behavior change, and the probability of eradicating HIV in San Francisco. Science. 1994, 265: 1451-1454.View ArticlePubMedGoogle Scholar
- Mardia KV, Kent JT, Bibby JM: Multivariate analysis. 1979, London: Academic PressGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://0-www.biomedcentral.com.brum.beds.ac.uk/1471-2458/4/34/prepub
This article is published under license to BioMed Central Ltd. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.