Accuracy: A Relative Measure of Translation Quality
Throughout my professional career, I have had numerous discussions about the relationship between an accurate translation and the perceived quality. I have had these discussions as a client working with language service providers (LSPs) and as an LSP working with clients.
Often, the fundamental misconception is that accuracy is an absolute measure of quality. And it certainly applies to many situations where high quality requires a certain level of accuracy. When it comes to translations, there is no guarantee that accuracy automatically constitutes high quality.
Let’s take a brief look at the quality standard SAE J2450. It was developed by the Society of Automotive Engineers (SAE) to measure and quantify translation quality. This standard provides a generic framework for categorizing translation errors and applying weights based on error type and significance. Also, it uses a calculated relative quality score to give you a sense of translation quality. However, it does not consider writing style since it would represent a subjective measure and rely on preferential taste for a translation. This limitation makes the standard not suitable for creative content where style is a key feature of perceived quality.
Overall, SAE J2450 offers an objective framework for quantifying translation quality. The standard’s approach reminds me of methodologies that we typically encounter in manufacturing environments. In such environments, quantity, tolerance, and sampling are common concepts used to define and measure quality. I have seen clients combine Six Sigma and SAE J2450 to measure translation quality and process control. Both frameworks can certainly provide great insights into the robustness of a process and give you a sense of relative quality.
Comprehension Versus Accuracy
Translation accuracy is a partial indicator of translation quality. For instance, a high quality score based on SAE J2450 does not necessarily mean that translated content is easy to understand. It just means that the translation accurately matches the source content. If the goal is to create useful and non-ambiguous global content that is easy to understand, then we should also consider comprehension as a quality indicator.
To illustrate the concept of comprehension and how it differs from accuracy, I will use the context of Linguistic Validation (LV). Testing comprehension is an important activity of Clinical Outcome Assessments (COAs) to confirm drug efficacy during clinical trials. If you are not familiar with COAs, just think of them as a questionnaire-based tool for collecting the health information of patients in response to the drug or treatment that is being studied during a clinical trial.
You can imagine that in order to collect valid and representative health data across all patients and locales, the design of the health questions is crucial. If a patient does not understand a question, she might not provide an answer or simply offer a best-guess response, which of course undermines the validity of the collected data.
Comprehension Starts with the Source Content
Generally, comprehension starts with the quality of the source content (the original content). Good, non-ambiguous source content establishes a much more robust foundation for the translation process. Any comprehension issues in the source content will likely appear in the translation.
To refer back to the LV example, even native speakers of the source language could run into comprehension issues. Comprehension issues can have many causes that are not always rooted in the translation itself. For instance, I recall a questionnaire that was targeting patients in the US, the UK, and Australia. The original questionnaire was prepared for a US audience. One of the health questions was designed to assess the patient’s level of fatigue after walking a “block.” Obviously, the concept of distance as a “block” was familiar to all US patients.
However, Australian patients indicated comprehension issues during the cognitive debriefing (comprehension testing). Their feedback revealed that they were struggling to provide a definitive response to this question. Some Australian patients did not understand the question, while others could not say for sure what distance a “block” represents. In the end, the original question was revised for Australian patients to include an actual distance (e.g., 100 meters). This of course meant that the Australian questionnaire was no longer an accurate rendition of the original questionnaire, but now the patients understood the question.
The presented example demonstrates that an accurate translation does not guarantee comprehension. So when we talk about translation quality, we should be aware that frameworks such as SAE J2450 or our own understanding of “accuracy” might not offer an ideal assessment of translation quality. In fact, we might gain a false sense of quality.
Applying an Adaptive Approach to Quality
My intent is not to question the validity of SAE J2450 or other quantitative frameworks. Such frameworks have their place within the larger question of quality control and assurance. The goal here is to raise awareness about accuracy and comprehension. I also want to point out that these are two different indicators for translation quality.
This also brings up the basic question about the definition of quality, which might be different depending on the type and purpose of the content we are creating. For instance, technical documentation is usually very literal, and the consistent use of terminology is important. In contrast, marketing content can be very creative and requires adaptation to achieve the same response across regional target audiences. One could say that technical documentation aims for accuracy and creative marketing content aims for comprehension.
In an ideal world, all content should be accurate and understandable. Creating content, however, is not always a black and white undertaking. Therefore, we should be aware that accuracy and comprehension can be mutually exclusive and still produce quality content.
Creating and translating content might require us to apply a more flexible approach to quality. If accuracy is the primary goal, we might not be able to achieve full comprehension. And if comprehension is the primary goal, we might not achieve full accuracy. This realization could help clients and service providers develop better quality models for different content types. It would also result in a more effective measurement of quality for original content and translated content.