tts models support (#2033)

Co-authored-by: luowei <glpat-EjySCyNjWiLqAED-YmwM> Co-authored-by: crazywoola <427733928@qq.com> Co-authored-by: crazywoola <100913391+crazywoola@users.noreply.github.com> Co-authored-by: Yeuoly <45712896+Yeuoly@users.noreply.github.com>
2024-01-24 01:05:37 +08:00
parent 27828f44b9
commit 6355e61eb8
86 changed files with 1645 additions and 133 deletions
--- a/web/app/components/develop/template/template_chat.en.mdx
+++ b/web/app/components/develop/template/template_chat.en.mdx
@@ -6,7 +6,7 @@ import { Row, Col, Properties, Property, Heading, SubProperty, Paragraph } from
 Chat applications support session persistence, allowing previous chat history to be used as context for responses. This can be applicable for chatbots, customer service AI, etc.

 <div>
-  ### Base URL 
+  ### Base URL
  <CodeGroup title="Code" targetCode={props.appDetail.api_base_url}>
    ```javascript
    ```
@@ -14,10 +14,10 @@ Chat applications support session persistence, allowing previous chat history to

  ### Authentication

-  The Service API uses `API-Key` authentication.  
+  The Service API uses `API-Key` authentication.
  <i>**Strongly recommend storing your API Key on the server-side, not shared or stored on the client-side, to avoid possible API-Key leakage that can lead to serious consequences.**</i>

-  For all API requests, include your API Key in the `Authorization`HTTP Header, as shown below: 
+  For all API requests, include your API Key in the `Authorization`HTTP Header, as shown below:

  <CodeGroup title="Code">
    ```javascript
@@ -46,18 +46,18 @@ Chat applications support session persistence, allowing previous chat history to
        User Input/Question content
      </Property>
      <Property name='inputs' type='object' key='inputs'>
-          Allows the entry of various variable values defined by the App.  
+          Allows the entry of various variable values defined by the App.
          The `inputs` parameter contains multiple key/value pairs, with each key corresponding to a specific variable and each value being the specific value for that variable.
      </Property>
      <Property name='response_mode' type='string' key='response_mode'>
        The mode of response return, supporting:
        - `streaming` Streaming mode (recommended), implements a typewriter-like output through SSE ([Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events)).
-        - `blocking` Blocking mode, returns result after execution is complete. (Requests may be interrupted if the process is long)  
-        Due to Cloudflare restrictions, the request will be interrupted without a return after 100 seconds.  
+        - `blocking` Blocking mode, returns result after execution is complete. (Requests may be interrupted if the process is long)
+        Due to Cloudflare restrictions, the request will be interrupted without a return after 100 seconds.
        <i>Note: blocking mode is not supported in Agent Assistant mode</i>
      </Property>
      <Property name='user' type='string' key='user'>
-          User identifier, used to define the identity of the end-user for retrieval and statistics.  
+          User identifier, used to define the identity of the end-user for retrieval and statistics.
          Should be uniquely defined by the developer within the application.
      </Property>
      <Property name='conversation_id' type='string' key='conversation_id'>
@@ -75,9 +75,9 @@ Chat applications support session persistence, allowing previous chat history to
      Can achieve async title generation by calling the conversation rename API and setting `auto_generate` to true.
      </Property>
    </Properties>
-  
+
    ### Response
-    When response_mode is blocking, return a CompletionResponse object.  
+    When response_mode is blocking, return a CompletionResponse object.
    When response_mode is streaming, return a ChunkCompletionResponse stream.

    ### ChatCompletionResponse
@@ -122,7 +122,7 @@ Chat applications support session persistence, allowing previous chat history to
      - `tool` (string) A list of tools represents which tools are called，split by ;
      - `tool_input` (string) Input of tools in JSON format. Like: `{"dalle3": {"prompt": "a cute cat"}}`.
      - `created_at` (int) Creation timestamp, e.g., 1705395332
-      - `message_files` (array[string])  Refer to message_file event 
+      - `message_files` (array[string])  Refer to message_file event
        - `file_id` (string) File ID
      - `conversation_id` (string) Conversation ID
    - `event: message_file` Message file event, a new file has created by tool
@@ -260,7 +260,7 @@ Chat applications support session persistence, allowing previous chat history to
 <Row>
  <Col>
  Upload a file (currently only images are supported) for use when sending messages, enabling multimodal understanding of images and text.
-  Supports png, jpg, jpeg, webp, gif formats.  
+  Supports png, jpg, jpeg, webp, gif formats.
  Uploaded files are for use by the current end-user only.

  ### Request Body
@@ -269,7 +269,7 @@ Chat applications support session persistence, allowing previous chat history to
    The file to be uploaded.
  - `user` (string) Required
    User identifier, defined by the developer's rules, must be unique within the application.
-  
+
  ### Response
  After a successful upload, the server will return the file's ID and related information.
  - `id` (uuid) ID
@@ -291,7 +291,7 @@ Chat applications support session persistence, allowing previous chat history to
  - 503, `s3_permission_denied`, no permission to upload files to S3
  - 503, `s3_file_too_large`, file exceeds S3 size limit
  - 500, internal server error
-  
+

  </Col>
  <Col sticky>
@@ -311,12 +311,12 @@ Chat applications support session persistence, allowing previous chat history to
  <CodeGroup title="Response">
    ```json {{ title: 'Response' }}
    {
-      "id": "72fa9618-8f89-4a37-9b33-7e1178a24a67",  
+      "id": "72fa9618-8f89-4a37-9b33-7e1178a24a67",
      "name": "example.png",
      "size": 1024,
      "extension": "png",
      "mime_type": "image/png",
-      "created_by": "6ad1ab0a-73ff-4ac1-b9e4-cdb312f71f13",  
+      "created_by": "6ad1ab0a-73ff-4ac1-b9e4-cdb312f71f13",
      "created_at": 1577836800,
    }
  ```
@@ -347,8 +347,8 @@ Chat applications support session persistence, allowing previous chat history to
  <CodeGroup title="Request" tag="POST" label="/chat-messages/:task_id/stop" targetCode={`curl -X POST 'https://cloud.dify.ai/v1/chat-messages/:task_id/stop' \\\n-H 'Authorization: Bearer {api_key}' \\\n-H 'Content-Type: application/json' \\\n--data-raw '{ "user": "abc-123"}`}>
    ```bash {{ title: 'cURL' }}
    curl -X POST 'https://cloud.dify.ai/v1/chat-messages/:task_id/stop' \
-    -H 'Authorization: Bearer {api_key}' \  
-    -H 'Content-Type: application/json' \  
+    -H 'Authorization: Bearer {api_key}' \
+    -H 'Content-Type: application/json' \
    --data-raw '{
        "user": "abc-123"
    }'
@@ -444,7 +444,7 @@ Chat applications support session persistence, allowing previous chat history to
        Conversation ID
      </Property>
      <Property name='user' type='string' key='user'>
-        User identifier, used to define the identity of the end-user for retrieval and statistics.  
+        User identifier, used to define the identity of the end-user for retrieval and statistics.
        Should be uniquely defined by the developer within the application.
      </Property>
      <Property name='first_id' type='string' key='first_id'>
@@ -475,7 +475,7 @@ Chat applications support session persistence, allowing previous chat history to
      - `tool` (string) A list of tools represents which tools are called，split by ;
      - `tool_input` (string) Input of tools in JSON format. Like: `{"dalle3": {"prompt": "a cute cat"}}`.
      - `created_at` (int) Creation timestamp, e.g., 1705395332
-      - `message_files` (array[string])  Refer to message_file event 
+      - `message_files` (array[string])  Refer to message_file event
        - `file_id` (string) File ID
    - `answer` (string) Response message content
    - `created_at` (timestamp) Creation timestamp, e.g., 1705395332
@@ -609,7 +609,7 @@ Chat applications support session persistence, allowing previous chat history to

    <Properties>
      <Property name='user' type='string' key='user'>
-          User identifier, used to define the identity of the end-user for retrieval and statistics.  
+          User identifier, used to define the identity of the end-user for retrieval and statistics.
          Should be uniquely defined by the developer within the application.
      </Property>
      <Property name='last_id' type='string' key='last_id'>
@@ -800,8 +800,8 @@ Chat applications support session persistence, allowing previous chat history to

    <Properties>
      <Property name='file' type='file' key='file'>
-        Audio file.  
-        Supported formats: `['mp3', 'mp4', 'mpeg', 'mpga', 'm4a', 'wav', 'webm']`  
+        Audio file.
+        Supported formats: `['mp3', 'mp4', 'mpeg', 'mpga', 'm4a', 'wav', 'webm']`
        File size limit: 15MB
      </Property>
      <Property name='user' type='string' key='user'>
@@ -837,6 +837,54 @@ Chat applications support session persistence, allowing previous chat history to

 ---

+<Heading
+  url='/text-to-audio'
+  method='POST'
+  title='text to audio'
+  name='#audio'
+/>
+<Row>
+  <Col>
+    Text to speech, only supports openai model.
+
+    ### Request Body
+
+    <Properties>
+      <Property name='text' type='str' key='text'>
+        Speech generated content。
+      </Property>
+      <Property name='user' type='string' key='user'>
+        The user identifier, defined by the developer, must ensure uniqueness within the app.
+      </Property>
+      <Property name='streaming' type='bool' key='streaming'>
+        Whether to enable streaming output, true、false。
+      </Property>
+    </Properties>
+  </Col>
+  <Col sticky>
+
+    <CodeGroup title="Request" tag="POST" label="/text-to-audio" targetCode={`curl --location --request POST '${props.appDetail.api_base_url}/text-to-audio' \\\n--header 'Authorization: Bearer ENTER-YOUR-SECRET-KEY' \\\n--form 'text=Hello Dify;user=abc-123;streaming=false`}>
+
+    ```bash {{ title: 'cURL' }}
+    curl --location --request POST '${props.appDetail.api_base_url}/text-to-audio' \
+    --header 'Authorization: Bearer ENTER-YOUR-SECRET-KEY' \
+    --form 'file=Hello Dify;user=abc-123;streaming=false'
+    ```
+
+    </CodeGroup>
+
+    <CodeGroup title="headers">
+    ```json {{ title: 'headers' }}
+    {
+      "Content-Type": "audio/wav"
+    }
+    ```
+    </CodeGroup>
+  </Col>
+</Row>
+
+---
+
 <Heading
  url='/parameters'
  method='GET'
@@ -969,7 +1017,7 @@ Chat applications support session persistence, allowing previous chat history to
      </Properties>
  ### Response
  - `tool_icons`(object[string]) tool icons
-    - `tool_name` (string) 
+    - `tool_name` (string)
      - `icon` (object|string)
        - (object) icon object
          - `background` (string) background color in hex format