UltraTrace: A free/open-source cross-platform tool for manual annotation of ultrasound tongue imaging data

Kevin Murphy,¹ Nathaniel Ziv Stern,¹ Daniel Swanson,² Caleb Ho,¹ Jonathan Washington²
  1. Independent Researcher (USA)
  2. Swarthmore College (USA)

Overview

Our talk in a nutshell

We made a new tool
for manually annotating UltraSound data

that works on multiple desktop platforms

and is released
to the public under a free/open-source license.

The next ~18 minutes:

  • Background
  • Main features
    • Freedoms
    • Platform support
    • Format support
    • User features
  • Future work
  • Discussion

Background

  • 2014: very simple UI by Jonathan Washington for dissertation work
  • 2015: redesign to make it easier to work on
  • 2016: added features
  • 2018: redesign by Kevin Murphy
  • 2018-2019: intense development by Ziv Stern
  • 2019-2020: intense development by Daniel Swanson
    & beginning of UI redesign by Caleb Ho and Kevin Murphy

Freedoms

  • UltraTrace is:
    • released as Free/Open Source Software (FOSS)
    • licensed under the General Public License (GPL) v3
    • written entirely in Python, entirely FOSS libraries
UltraTraceGetContoursWASLPalatoglossotronAAA
Freely available✔✔✔✔✘
Source code available✔✔✔✔✘
Free license✔✔✘✔✘
Free dependencies✔✘✘✔—

Why FOSS

  • GPLv3 grants the freedom to …
    • run the software
    • study the software
    • share the software
    • modify the software
    • (even sell the software)
  • … as long as you …
    • name © holders
    • state any changes
    • keep the source open
    • keep same/similar license

This encourages:

  • sharing of data, formats, algorithms, etc.
  • continuity between possibly temporally distant projects
  • fewer conflicting standards (e.g. file formats)
  • collaboration between people doing different things
  • optimal usage of limited resources
Free dependencies also an issue of social justice:
e.g., no access to MATLAB does not inhibit use.

Native Platform support

UltraTraceGetContoursWASLPalatoglossotronAAA
Windows✔✔✔✔✔
macOS✔✔✔✘?✘
GNU/Linux✔✔✔✘?✘

Format support

UltraTraceGetContoursWASLPalatoglossotronAAA
DICOM images✔✔✔✘✘
native Philips✔✘✔✘✘
AAA-exported✔✘✘✘✔

Feature Comparison: User Features

UltraTraceGetContoursWASLPalatoglossotronAAA
Text annotations✔✔✘✘✔
Adjust audio sync.✔✘✔✘✘
Tracing completion status✔✘✘✘✘
Aribtrary zoom and pan✔✘✘✘✘

Project Management

  • Automatic matching of US + audio + TextGrid by filename
  • Organise your files how you like!
    • Participant A1/
      • wordlist1/
        • 2020-10-22_175001.wav
        • 2020-10-22_175001.TextGrid
        • 2020-10-22_175001.dicom
      • wordlist2/
        • 2020-10-22_175235.wav
        • 2020-10-22_175235.TextGrid
        • 2020-10-22_175235.dicom
    • Participant A1/
      • Audio/
        • 2018-09-01_002108.wav
        • 2020-10-22_175235.wav
      • Ultrasound/
        • 2018-09-01_002108.dicom
        • 2020-10-22_175235.dicom
      • TextGrids/
        • 2018-09-01_002108.TextGrid
        • 2020-10-22_175235.TextGrid
  • Or don't! It'll still work ☺
  • Participant A1/
    • 2018-09-01_002108.wav
    • 2018-09-01_002108.TextGrid
    • 2018-09-01_002108.dicom
    • 2020-10-22_175235.wav
    • 2020-10-22_175235.TextGrid
    • 2020-10-22_175235.dicom

Main tracing features

  • Tracing
  • Adjust points
  • Select/drag multiple points
  • Deletion of points
  • Full undo/redo buffer
  • Arbitrary colour of points
  • Arbitrary number of landmarks

Touch-screen support

Copy/pasting of points

Arbitrary Zoom and Pan

Added portability:

  • coordinates stored in percentages, not pixels or physical dimensions
  • Autosaves data
  • Stored in json

Text Annotations: Zooming/panning

Text Annotations: Navigating by interval

Text Annotations: Collapsible Tiers

Interactive Spectrogram

Features:

  • Click for nearest frame
  • Adjust:
    • max frequency
    • window length
    • dynamic range

Synchronised playback of intervals

Future work

  • Direct manipulation of interval annotations
  • Waveform collapsible as spectrogram
  • General UI improvements (perhaps rewrite)
  • Automatic tracing algorithm(s: SLURP? neural approach?)
  • Eventually: 3D/4D support

Discussion

  • Authors of (most) alternatives present—no ill will meant!
  • We acknowledge many advantages offered by alternatives
  • UltraTrace — an additional available tool, maybe filling a niche?
    • limited resources?
    • more comfortable with Python?
    • care more about our specific set of features?
  • Available on GitHub (issues and pull requests wecome)!
    http://github.com/SwatPhonLab/UltraTrace

Thank you!

With generous support from:
  • Joel Dean Summer Research Fellowship
  • Eugene M. Lang Summer Research Fellowship
  • Swarthmore College Faculty Research Support Grants

Also thanks to:

  • reviewer for UltraFest

Appendix slides

Aligning ultrasound frames with audio

  • Frame positions are stored in a TextGrid point tier
  • Generated when ultrasound file first opened
  • AAA exported data:
    • calculate from framerate
  • Reading DICOM:
    • read individual frametimes from native Philips data
    • fallback to calculation based on framerate if not present