stash/docs/dev/AUDIO.md

Audio Datatype

The Audio datatype is similar to Scene but stores audio-only media (i.e. Audiobooks, music, ASMR, etc).

Scope

• This ticket adds backend support for Audio Only, future tickets can add the UI elements

  • Database design
  • Graphql Support
  • Scanner Support
    • No transcodes right now, but will keep the infrastructure to more easily support adding transcodes in the future

• Audio metadata:

  • Title
  • Date
  • Studio
  • Performers
  • Tags
  • Details
  • Urls
  • Rating
  • Organized
  • O History
  • Play History
  • Groups
  • Captions

• Audio File metadata:

  • duration
  • audio codec
  • bitrate
  • sample rate

Open Questions

• Should Audio's have cover photo?
• Should Legacy/Deprecate features be copied over?
  • Since Audio's is NEW, it doesn't have to support deprecated features/naming/etc
  • I suggest removing them if easy to do, and for the more complicated ones to defer to a separate ticket
• What should be done for sortByOCounter/sortByPlayCount?
  • These assume SCENES
  • I see 3 options
    • ignore
    • add audios into the calculation
    • split into sortBySceneOCounter and sortByAudioOCounter

Future Tickets

• UI
  • Audio using video.js (ref: https://videojs.org/blog/video-js-4-9-now-can-join-the-party)
  • Audio Waveform (ref: https://github.com/collab-project/videojs-wavesurfer)
  • New AudioPlayer.tsx (copy ui/v2.5/src/components/ScenePlayer/ScenePlayer.tsx)

General TODO

• Setup Database
• Scanner to scan Audio Files and create Audios
  • FFProbe for Audio Files
• Graphql to return Audios (queries)
• Graphql to update Audios (mutations)
• Update test files

Notes

• Phashes cannot be used on audio files; A future ticket might introduce Chromaprint (AcoustID)
• Gallery could be added to Audio, but I am removing to reduce PR complexity
• StashIDs was removed, audio is unlikely to be added immediately to stashbox
• Audio's could have interactive components, but removed to reduce PR complexity

Last Steps

• Delete this file upon completion of the feature

Manual Tests

Setup

1. Copy .mp3 files into .local-data
2. make server-clean
3. make server-start OR run go debugger (VSCode F5)
4. Create new instance with library at ./.local-data/
5. go to http://127.0.0.1:9999/playground
   • Perform manual tests here

Check Query

This is a manual test with all fields. The test ensures that the Querying is setup correctly.

Later you can reuse this to ensure that mutations correctly updated the database.

query {
  findAudios(filter:{sort:"title" direction:DESC}){
    count
    audios {
			id title code details urls date rating100 organized o_counter created_at updated_at last_played_at resume_time play_duration play_count play_history o_history custom_fields

      files{
          id path basename mod_time size format duration audio_codec sample_rate bit_rate created_at updated_at
          parent_folder{id}
          zip_file{id}
          fingerprints{type value}
      }
      captions{language_code caption_type}
      paths{caption stream}
      studio{id}
      groups{group{id} audio_index}
      tags{id}
      performers{id}
      audioStreams{url mime_type label}
    }
  }
  # findScenes(filter:{sort:"title" direction:DESC}){
  #   count
  #   scenes {
  #     id sceneStreams{url mime_type label}
  #     files{id path fingerprints{type value}}
  #   }
  # }
}

Check Mutations

mutation audio_mut {
  audioAddO(id:1){count history}
  audioUpdate(input:{id:1 title:"testing 1"}){id title o_history}
  audiosUpdate(input:[{id:1 details:"details 1"}]){id title details}
}

Check Streams

Currently only direct streams are implemented. Use the following to get the Stream URL.

1. Execute this GraphQL
2. Paste the Direct stream url into the browser, ensure that the audio plays

query {
  findAudios(filter:{sort:"title" direction:DESC}){
    count
    audios {id audioStreams{url mime_type label}
    }
  }
}

HTML Confirmation

<audio controls>
  <source src="http://127.0.0.1:9999/audio/1/stream" type="audio/mp3">
Your browser does not support the audio element.
</audio>

You can also listen to audio using VIDEO tag

<video controls>
  <source src="http://127.0.0.1:9999/audio/1/stream" type="audio/mp3">
Your browser does not support the video element.
</video>