@realkimhansen @signlfm If you hadn't been aware, this came out earlier in the week and may be of overlapping interest:
Three recommendations to enable Annotations on the Web | W3C News https://www.w3.org/blog/news/archives/6156
https://www.w3.org/annotation/
Their method of breaking down the problem and their vocabulary may be particularly useful; the overlay onto audio may be another problem altogether...

I also thought this post (with code) by Aaron Parecki, while a relatively simple concept, was exceptionally cool. https://aaronparecki.com/2017/02/19/4/day-61-media-fragments