Understanding Code-Mixing vs. Code-Switching

Before diving into annotation practices, it is important to clarify two commonly confused linguistic phenomena: code-mixing and code-switching. While both involve the blending of multiple languages in speech, they differ in structure and intent.

• Code-Switching refers to the alternation between two or more languages across phrases, sentences, or discourse units. For example, a Spanish-English bilingual might say: “I was walking to the tienda when I saw my friend.” Here, “tienda” is used seamlessly in an otherwise English sentence, but the shift is clear and structured.
• Code-Mixing, on the other hand, happens at the word or morpheme level, often resulting in hybrid forms. For example, South African speakers might combine English with isiZulu in ways that create entirely new expressions or blend grammatical systems. Unlike code-switching, code-mixing does not always respect sentence boundaries and often emerges in casual, spontaneous speech.

For linguists, transcription coordinators, and annotation teams, distinguishing between these two is vital. Code-switching might require marking larger segments with clear boundaries, while code-mixing demands finer-grained tagging down to syllables or morphemes. Confusion between the two can result in inconsistent annotations that reduce corpus quality and compromise ASR training.

Moreover, the prevalence of code-mixing differs by community. In urban multilingual environments—such as Nairobi, Mumbai, or Johannesburg—speakers frequently mix languages at the word level. In contrast, academic interviews or formal political speeches may lean more towards code-switching at sentence boundaries. Recognising these nuances ensures more accurate multilingual audio annotation that truly reflects linguistic reality.

Annotation Frameworks for Mixed Speech

Once the difference between code-mixing and code-switching is clear, the next step is designing annotation frameworks that can capture both phenomena consistently. This involves establishing rules and conventions for tagging languages in audio transcripts.

Core Principles of Mixed-Speech Annotation

• Segment Identification: Define the smallest unit that will be tagged (sentence, phrase, word, or morpheme). Code-mixing often requires annotations at the word or sub-word level.
• Language Labels: Decide on a consistent labelling scheme. For example, [ENG] for English, [SPA] for Spanish, [ZUL] for isiZulu, etc.
• Overlap Handling: In natural speech, speakers may overlap, creating segments where multiple languages are spoken simultaneously. Annotation frameworks should clarify how to tag these overlaps without losing information.
• Hybrid Words: When words themselves contain elements of two languages (such as borrowed roots with local suffixes), annotators should agree on whether to mark them as mixed tokens or split by morpheme.

Example Annotation Schema

A bilingual Hindi-English utterance might look like this:

• [HIN] Mujhe [ENG] appointment [HIN] kal mil sakta hai.
  This indicates that “Mujhe” and “kal mil sakta hai” are Hindi, while “appointment” is tagged as English.

Standardisation and Inter-Annotator Agreement

Consistency is critical. Different annotators might interpret mixed segments differently if guidelines are vague. Developing a detailed annotation manual, combined with training sessions and test cases, helps establish inter-annotator agreement. Quality assurance processes, such as random sampling and review by senior linguists, further improve accuracy.

Ultimately, a robust annotation framework should not only handle straightforward switches but also account for the messiness of real-world multilingual speech—where hesitations, borrowings, and incomplete sentences abound.

Tooling and File Structures

Annotation is not only about linguistic clarity but also about the tools and file structures that support the process. Several widely used tools and formats have become standard for code-mixing tagging and multilingual audio annotation.

Common Tools

• ELAN (EUDICO Linguistic Annotator): One of the most popular tools for linguistic research, ELAN allows multi-tier annotation. Each tier can represent a language, and annotators can tag word-level switches with time-aligned precision.
• TranscriberAG: Useful for transcription tasks requiring segmentation, speaker labelling, and metadata annotation. While less sophisticated than ELAN, it can be adapted for multilingual contexts.
• Praat: Primarily a phonetic analysis tool, Praat can also be used for manual tagging of speech segments, although it is less efficient for large multilingual corpora.

File Structures and Schemas

• XML-based Schemas: XML formats allow detailed metadata storage, making it easy to represent segment-level language information. Many ASR training pipelines rely on XML or JSON schemas to import annotated corpora.
• TEI (Text Encoding Initiative): Widely used in digital humanities, TEI guidelines can be adapted for multilingual transcripts.
• Custom JSON Frameworks: Some teams create custom schemas where each segment is annotated with attributes such as start time, end time, speaker ID, and language label.

Best Practices for Tool Use

• Ensure annotations are time-aligned so that speech recognition models can map acoustic features to the correct language labels.
• Maintain a metadata layer that documents speaker background, proficiency, and context of recording. This contextual information is often invaluable when interpreting mixed-language use.
• Build export compatibility into the workflow. Tools should allow exporting to formats compatible with ASR pipelines and NLP research environments.

Without proper tooling, even the most rigorous annotation frameworks risk being inconsistent, difficult to scale, or unusable in downstream applications.

Training Considerations for ASR and Language Identification

The ultimate goal of tagging multilingual code-mixing is not simply to create neat transcripts. Instead, it is to generate training data that improves ASR systems and multilingual language identification models. The way annotation is done directly impacts the quality of these models.

Why Code-Mixing Matters for Training

• ASR Robustness: Real-world audio rarely comes in neat monolingual packages. ASR systems trained only on single-language data perform poorly in multilingual contexts.
• Language Identification: Labelling switches at precise points helps models learn how to detect and adapt to changes in language, rather than forcing users into rigid monolingual categories.
• Hybrid Word Handling: Code-mixing often introduces borrowed or hybrid words. If these are tagged properly, ASR models can learn to generalise across language boundaries.

Training Pipeline Implications

• Balanced Datasets: Models must be trained on a representative mix of languages and contexts, not just dominant languages.
• Sub-Word Modelling: Annotation of morphemes in code-mixed words helps sub-word tokenisation approaches (e.g., Byte Pair Encoding or SentencePiece).
• Cross-Script Learning: Many languages use different scripts. Annotating script switches explicitly can guide models in learning to disambiguate writing systems alongside phonetic cues.

For researchers and developers, the message is clear: accurate and fine-grained language switch labelling is not just an academic exercise. It is the foundation for building tools that truly reflect how people speak in multilingual societies.

Common Pitfalls and QA Practices

Despite best intentions, multilingual annotation projects often encounter recurring challenges. Identifying these pitfalls in advance can save considerable time and ensure higher data quality.

Frequent Pitfalls

• Inconsistent Tagging: Annotators may differ in how they classify mixed tokens. Without strong guidelines, one might tag “WhatsApping” as English while another treats it as code-mixed.
• Script Confusion: In regions where multiple scripts exist (e.g., Hindi in Devanagari and Romanised script), inconsistencies in transcription can undermine annotations.
• Overlooking Prosody: Tagging may focus solely on words while ignoring prosodic features like intonation that signal language shifts.
• Annotator Bias: Annotators may unconsciously impose their own language preferences, resulting in skewed data.

QA Best Practices

• Double Annotation: Having two independent annotators work on the same file before reconciliation increases accuracy.
• Regular Review Sessions: Weekly or bi-weekly team reviews ensure consistency across annotators.
• Validation Sets: Build small validation datasets reviewed by expert linguists to test inter-annotator agreement.
• Clear Error Protocols: Develop rules for handling uncertainty. For example, if a token cannot be confidently classified, it may be tagged as [MIX] with a note for later review.

Ultimately, quality control must be embedded throughout the project lifecycle. Annotation is not a one-off task but a continuous cycle of tagging, reviewing, and refining.