K100_AI两卡全离线部署超长音视频说话人角色确认转录系统（多轮推理）

一、引言之前我发表的一篇文章https://blog.csdn.net/pla88888888/article/details/162817260K100_AI两卡全离线部署音视频说话人角色确认转录系统采用SoulX-Transcriber多模态大模型实现在实测中发现超长音视频文件分析时会出现重复输出。这次我尝试用Speaker-Reasoner-4194h多模态大模型来解决该问题自编程序思路来源于官方代码代码链接为https://github.com/ASLP-lab/Speaker-Reasoner/tree/main如果您觉得该大模型对本工作有用请引用article{lin2026speakerreasoner,title{Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR},author{Zhennan Lin and Shuai Wang and Zhaokai Sun and Pengyuan Xie and Chuan Xie and Jie Liu and Qiang Zhang and Lei Xie},year{2026},eprint{2604.03074},archivePrefix{arXiv},primaryClass{eess.AS},url{https://arxiv.org/abs/2604.03074},}二、方案设计1. 项目背景与目标在会议记录、访谈整理等场景中精确的说话人分离与时间轴对齐是核心需求。本系统旨在利用多模态大模型“Speaker-Reasoner-4194h”的音频理解与推理能力通过多轮对话机制实现端到端的智能转录用户只需上传音频或视频文件即可自动获得带有说话人ID、性别、精确时间戳的完整转写文本。2. 系统架构系统采用B/S架构前端基于HTML模板提供简洁的上传界面后端基于Flask框架负责请求接收、任务调度与结果返回。核心组件包括文件预处理模块使用ffmpeg将任意格式视频/音频统一转码为16kHz单声道WAV保证模型输入一致性。推理引擎通过HTTP接口调用自部署的vLLM服务承载Speaker-Reasoner-4194h模型。多轮对话控制模块实现自适应温度调节、重复输出检测、最大重试与边界校验确保推理鲁棒性。转录解析模块支持秒数、HH:MM:SS、MM:SS.ss等多类时间戳格式并过滤占位符等无效行输出结构化片段。3. 核心工作流程1用户通过Web页面上传文件后端进行安全校验并保存为临时文件。2ffmpeg转换为16kHz单声道WAV格式创建临时工作目录用于存放后续音频片段。3发起首轮对话模型接收完整音频与初始提示被要求概括整体并给出第一个片段的起止时间边界。4进入循环每轮解析模型返回的box标签获取边界用ffmpeg精确切出片段音频构造“当前片段作为O_turn”的新消息驱动模型逐步推导该片段的说话人、性别与文本。5模型在最终回复中输出answer标签包含所有片段的完整转录。若未正常结束则从最后一轮提取后备结果。6解析转录文本转换为带有start、end、speaker、text的标准JSON分段数组连同原始文本一并返回前端展示。7全程自动清理所有临时文件保证服务器空间安全。4. 关键技术细节鲁棒性设计单轮最多重试5次前两次使用极低温度0.01和受限采样以抑制随机性后续逐步放宽引入重复惩罚系数并检测连续重复10次以上的模式一旦发现即触发重试。边界校验严格检查box中结束时间必须大于起始时间否则终止对话防止无效片段。资源管理上传限制500MB所有临时文件使用try-finally确保释放避免残留。多格式兼容解析模块同时支持Speaker [0.0, 3.68]: text和[00:01:23 - 00:01:45] Speaker1: text等并通过统一的秒数→时间码转换函数保证输出一致。5. 部署与展望系统部署于Flask内置服务器监听5000端口可与vLLM服务同机或分布式部署。后续可扩展为异步任务队列支持更大规模并发处理并增加说话人声纹注册与自动命名等功能进一步提升转录体验。下图展示了整体数据流三、实施方法及代码3.1硬件环境本方案的硬件平台为一台H3C服务器配置如下组件规格CPU2×海光74902.7GHz64C内存16×32GDDR5GPU8×海光DCU64GBK100_AI需占用两张K100_AI显卡3.2软件栈本方案的软件栈基于Docker容器化技术构建使用经过海光DCU适配的vLLM推理镜像镜像Speaker-Reasoner-4194hharbor.sourcefind.cn:5443/dcu/admin/base/custom:vllm0.15.1-ubuntu22.04-dtk26.04-0130-py3.10-20260220该镜像基于vLLM0.15.1推理框架、DTK26.04Python3.10环境双卡部署Speaker-Reasoner-4194h多模态大模型。软件项目1、Speaker-Reasoner-4194h环境部署参照https://developer.sourcefind.cn/modelzoo/list/qwen3-omni_vllm/detail?post_id3563caba-356c-11f1-be71-0242ac150003里面操作步骤项目下载链接为https://developer.sourcefind.cn/codes/modelzoo/qwen3-omni_vllm/-/archive/v1.0/qwen3-omni_vllm-v1.0.zip。2、离线音视频说话人角色确认转录系统自编程序代码如下Speaker-Reasoner-web-2.py代码如下import os import re import tempfile import subprocess import requests import logging from flask import Flask, request, jsonify, render_template from werkzeug.utils import secure_filename app Flask(__name__) app.config[MAX_CONTENT_LENGTH] 500 * 1024 * 1024 # 500 MB VLLM_URL http://localhost:8082/v1/chat/completions MODEL_NAME Speaker-Reasoner-4194h # 提示词来自 plugin/prompt_4194h.json INITIAL_USER_TEXT ( 分析这段音频$X$将其划分为合适的片段。对于每个片段 请逐步推导说话人ID、性别、时间戳及转录内容。 现在请先概括音频整体情况并给出第一个片段的边界。 ) TURN_USER_TEXT 当前片段作为O_{turn}。 logging.basicConfig(levellogging.INFO) logger logging.getLogger(__name__) def _seconds_to_hms(seconds_str: str) - str: 将秒数字符串转换为 HH:MM:SS.sss 格式便于前端统一处理 try: total_seconds float(seconds_str) except ValueError: return seconds_str # 转换失败则原样返回 hours int(total_seconds // 3600) minutes int((total_seconds % 3600) // 60) secs total_seconds % 60 return f{hours:02d}:{minutes:02d}:{secs:06.3f} def parse_transcript(text: str): 解析转录文本支持多种时间戳格式过滤无意义的占位行。 增加支持Speaker [秒数, 秒数]: 内容 lines text.strip().split(\n) segments [] # 匹配 --:--:-- -- --:--:-- 占位行 placeholder_pattern re.compile(r^--:--:--\s*(?:--|→)\s*--:--:--$) # 新增匹配 Female 1 [0.0, 3.68]: 王姐我爸妈来看你了。 格式 pattern_sec re.compile( r(.?)\s*\[\s*(\d(?:\.\d)?)\s*,\s*(\d(?:\.\d)?)\s*\]\s*:\s*(.*) ) # 原有格式 pattern_hms re.compile(r\[(\d{2}:\d{2}:\d{2})\s*-\s*(\d{2}:\d{2}:\d{2})\]\s*(Speaker\d):\s*(.*)) pattern_ms re.compile(r\[(\d{2}:\d{2}\.\d)\s*--\s*(\d{2}:\d{2}\.\d)\]\s*(Speaker\d):\s*(.*)) pattern_no_time re.compile(r(Speaker\d):\s*(.*)) for line in lines: line line.strip() if not line: continue if placeholder_pattern.match(line): continue # 1. 秒数格式 match pattern_sec.match(line) if match: speaker, start_sec, end_sec, content match.groups() segments.append({ start: _seconds_to_hms(start_sec), end: _seconds_to_hms(end_sec), speaker: speaker.strip(), text: content.strip() }) continue # 2. HH:MM:SS 格式 match pattern_hms.match(line) if match: start, end, speaker, content match.groups() segments.append({start: start, end: end, speaker: speaker, text: content}) continue # 3. MM:SS.ss 格式 match pattern_ms.match(line) if match: start, end, speaker, content match.groups() segments.append({start: start, end: end, speaker: speaker, text: content}) continue # 4. 仅有 Speaker 无时间 match pattern_no_time.match(line) if match: speaker, content match.groups() segments.append({start: None, end: None, speaker: speaker, text: content}) else: segments.append({start: None, end: None, speaker: None, text: line}) return segments def cut_audio_segment(wav_path: str, start_sec: float, end_sec: float, tmp_dir: str) - str: 使用 ffmpeg 从 wav 文件中切出一段音频 segment_path os.path.join( tmp_dir, f{os.path.splitext(os.path.basename(wav_path))[0]}_{start_sec:.3f}_{end_sec:.3f}.wav ) cmd [ ffmpeg, -i, wav_path, -ss, str(start_sec), -to, str(end_sec), -c:a, pcm_s16le, -y, segment_path ] subprocess.run(cmd, checkTrue, stdoutsubprocess.PIPE, stderrsubprocess.PIPE) return segment_path def call_speaker_reasoner(wav_path: str, tmp_dir: str) - str: 通过 HTTP 调用 vLLM 服务完成多轮对话 返回最终 answer 标签内的文本。 messages [ { role: user, content: [ {type: audio_url, audio_url: {url: ffile://{os.path.abspath(wav_path)}}}, {type: text, text: INITIAL_USER_TEXT} ] } ] max_turns 100 max_retries_per_turn 5 current_turn 0 answer None last_response segment_files [] while current_turn max_turns: turn_success False content # 用于记录本轮响应 for attempt in range(max_retries_per_turn): try: # 动态调整温度 if attempt 2: temperature 0.01 top_p 0.1 top_k 1 else: temperature min(0.1 (attempt - 2) * 0.1, 1.0) top_p 0.9 top_k -1 # 增加 repetition_penalty 以减少重复输出 repetition_penalty 1.1 attempt * 0.05 payload { model: MODEL_NAME, messages: messages, temperature: temperature, top_p: top_p, top_k: top_k, max_tokens: 8192, repetition_penalty: repetition_penalty } resp requests.post(VLLM_URL, jsonpayload, timeout600) resp.raise_for_status() result resp.json() content result[choices][0][message][content] # 简单重复检测 if re.search(r(.?)\1{10,}, content.strip()): raise RuntimeError(Repeated output detected) messages.append({role: assistant, content: content}) # 检查 answer answer_match re.search(ranswer(.*?)/answer, content, re.DOTALL) if answer_match: answer answer_match.group(1).strip() current_turn max_turns # 跳出外层循环 turn_success True break # 检查 box获取下一段边界 box_match re.search(rbox\[\s*(\d(?:\.\d)?)\s*,\s*(\d(?:\.\d)?)\s*\]/box, content) if not box_match: # 尝试宽松匹配任何 [数字, 数字] 形式 fallback re.search(r\[\s*(\d(?:\.\d)?)\s*,\s*(\d(?:\.\d)?)\s*\], content) if fallback: start_time float(fallback.group(1)) end_time float(fallback.group(2)) else: # 彻底找不到边界视为对话结束 logger.info(No box or time boundary found, ending conversation.) turn_success True current_turn max_turns break else: start_time float(box_match.group(1)) end_time float(box_match.group(2)) # 验证边界有效性结束时间必须大于起始时间 if end_time start_time: logger.warning(Invalid time boundary: [%s, %s]. Ending conversation., start_time, end_time) turn_success True current_turn max_turns break # 切出音频片段 segment_path cut_audio_segment(wav_path, start_time, end_time, tmp_dir) segment_files.append(segment_path) turn_text TURN_USER_TEXT.format(turncurrent_turn 1) messages.append({ role: user, content: [ {type: audio_url, audio_url: {url: ffile://{os.path.abspath(segment_path)}}}, {type: text, text: turn_text} ] }) turn_success True current_turn 1 break except Exception as e: logger.warning(Turn %d, attempt %d failed: %s, current_turn, attempt, e) last_response content if content else last_response # 回滚可能已添加的错误 assistant 消息 if messages and messages[-1][role] assistant: messages.pop() # 如果是重复输出错误可以额外记录详细内容以便调试 if Repeated output in str(e): logger.debug(Repeated content sample: %s, content[:200] if content else N/A) if not turn_success: logger.error(Turn %d failed after all retries., current_turn) break # 若未获得 answer尝试从最后一条 assistant 消息提取 if answer is None: if messages and messages[-1][role] assistant: last_content messages[-1][content] else: last_content last_response answer_match re.search(ranswer(.*?)/answer, last_content, re.DOTALL) if answer_match: answer answer_match.group(1).strip() else: # 若根本没有 answer则返回全部最后回复作为后备 answer last_content.strip() if last_content else # 清理临时片段 for f in segment_files: if os.path.exists(f): os.unlink(f) return answer app.route(/) def index(): return render_template(index.html) app.route(/transcribe, methods[POST]) def transcribe(): if video not in request.files: return jsonify({error: No video/audio file provided}), 400 video_file request.files[video] if video_file.filename : return jsonify({error: Empty filename}), 400 filename secure_filename(video_file.filename) input_tmp None wav_path None try: # 1. 保存上传文件 suffix os.path.splitext(filename)[1] or .tmp input_tmp tempfile.NamedTemporaryFile(deleteFalse, suffixsuffix) video_file.save(input_tmp.name) input_tmp.close() # 2. 创建 WAV 临时文件16kHz 单声道 wav_tmp tempfile.NamedTemporaryFile(deleteFalse, suffix.wav) wav_path wav_tmp.name wav_tmp.close() # 3. ffmpeg 转码 cmd [ ffmpeg, -i, input_tmp.name, -vn, -acodec, pcm_s16le, -ar, 16000, -ac, 1, -y, wav_path ] subprocess.run(cmd, checkTrue, stdoutsubprocess.PIPE, stderrsubprocess.PIPE) # 4. 创建临时目录存放可能的音频片段 tmp_dir tempfile.mkdtemp() # 5. 调用 Speaker-Reasoner 多轮推理 try: raw_answer call_speaker_reasoner(wav_path, tmp_dir) finally: # 清理临时目录 if os.path.exists(tmp_dir): for f in os.listdir(tmp_dir): fp os.path.join(tmp_dir, f) try: os.unlink(fp) except Exception: pass os.rmdir(tmp_dir) segments parse_transcript(raw_answer) return jsonify({ success: True, segments: segments, raw_text: raw_answer }) except subprocess.CalledProcessError as e: error_msg e.stderr.decode() if e.stderr else ffmpeg failed return jsonify({error: fAudio conversion failed: {error_msg}}), 500 except Exception as e: logger.exception(Transcription error) return jsonify({error: str(e)}), 500 finally: # 清理临时文件 if input_tmp and os.path.exists(input_tmp.name): os.unlink(input_tmp.name) if wav_path and os.path.exists(wav_path): os.unlink(wav_path) if __name__ __main__: app.run(host0.0.0.0, port5000, debugTrue)templates/index.html代码如下!DOCTYPE html html langzh-CN head meta charsetUTF-8 meta nameviewport contentwidthdevice-width, initial-scale1.0 title超长离线音视频说话人角色确认转录系统强推理版/title style * { box-sizing: border-box; margin: 0; padding: 0; } body { font-family: Segoe UI, Roboto, system-ui, sans-serif; background: #f5f7fa; color: #1e293b; padding: 2rem; display: flex; flex-direction: column; align-items: center; } .container { max-width: 1400px; width: 100%; background: white; border-radius: 24px; box-shadow: 0 20px 40px -12px rgba(0,0,0,0.2); padding: 2rem; } h1 { font-weight: 600; margin-bottom: 1.5rem; display: flex; align-items: center; gap: 0.5rem; } .upload-area { border: 2px dashed #cbd5e1; border-radius: 16px; padding: 2rem; text-align: center; margin-bottom: 2rem; transition: background 0.2s; cursor: pointer; } .upload-area:hover { background: #f8fafc; } .upload-area input[typefile] { display: none; } .upload-label { font-size: 1.1rem; color: #475569; } .upload-label strong { color: #2563eb; } .player-wrapper { display: flex; flex-direction: row; gap: 2rem; margin-top: 1rem; } .video-section { flex: 1.2; min-width: 0; } .video-section video { width: 100%; border-radius: 12px; background: #0f172a; display: block; } .transcript-section { flex: 1.5; max-height: 500px; overflow-y: auto; background: #f1f5f9; border-radius: 12px; padding: 0.5rem 0; scroll-behavior: smooth; } .transcript-line { padding: 0.75rem 1.2rem; border-left: 3px solid transparent; cursor: pointer; transition: background 0.15s, border-color 0.15s; font-size: 0.95rem; line-height: 1.6; display: flex; align-items: baseline; gap: 0.8rem; } .transcript-line:hover { background: #e2e8f0; } .transcript-line.active { background: #dbeafe; border-left-color: #2563eb; } .transcript-line .time { font-family: JetBrains Mono, monospace; font-size: 0.8rem; color: #64748b; white-space: nowrap; min-width: 120px; flex-shrink: 0; margin-right: 0.5rem; } .transcript-line .speaker { font-weight: 600; color: #7c3aed !important; margin-right: 0.25rem; white-space: nowrap; flex-shrink: 0; } .transcript-line .text { word-break: break-word; } .status { margin-top: 1rem; padding: 0.75rem 1rem; background: #f1f5f9; border-radius: 8px; color: #475569; font-size: 0.95rem; } .status .error { color: #dc2626; } .status .success { color: #16a34a; } .loader { display: inline-block; width: 16px; height: 16px; border: 3px solid #e2e8f0; border-top-color: #2563eb; border-radius: 50%; animation: spin 0.6s linear infinite; margin-right: 0.5rem; vertical-align: middle; } keyframes spin { to { transform: rotate(360deg); } } media (max-width: 768px) { .player-wrapper { flex-direction: column; } .transcript-section { max-height: 300px; } } /style /head body div classcontainer h1 超长离线音视频说话人角色确认转录系统强推理版/h1 div classupload-area iduploadArea div classupload-label strong点击选择音视频文件/strong 或拖拽到这里br span stylefont-size:0.9rem; color:#94a3b8;支持WAV / MP3 / MP4 / MOV / AVI 等格式/span /div input typefile idvideoInput acceptaudio/*,video/* /div div idstatus classstatus 请上传视频文件以开始转录/div div classplayer-wrapper idplayerWrapper styledisplay:none; div classvideo-section video idvideoPlayer controls/video /div div classtranscript-section idtranscriptContainer !-- 转录行将动态渲染在这里 -- /div /div /div script (function() { const videoInput document.getElementById(videoInput); const uploadArea document.getElementById(uploadArea); const statusDiv document.getElementById(status); const playerWrapper document.getElementById(playerWrapper); const videoPlayer document.getElementById(videoPlayer); const transcriptContainer document.getElementById(transcriptContainer); let segments []; let timeUpdateHandler null; // 上传区域 uploadArea.addEventListener(click, () videoInput.click()); uploadArea.addEventListener(dragover, (e) { e.preventDefault(); uploadArea.style.borderColor #2563eb; }); uploadArea.addEventListener(dragleave, () { uploadArea.style.borderColor #cbd5e1; }); uploadArea.addEventListener(drop, (e) { e.preventDefault(); uploadArea.style.borderColor #cbd5e1; if (e.dataTransfer.files.length) { videoInput.files e.dataTransfer.files; videoInput.dispatchEvent(new Event(change)); } }); // 文件选择与转录 videoInput.addEventListener(change, async function() { const file this.files[0]; if (!file) return; if (timeUpdateHandler) { videoPlayer.removeEventListener(timeupdate, timeUpdateHandler); timeUpdateHandler null; } const url URL.createObjectURL(file); videoPlayer.src url; videoPlayer.style.display block; playerWrapper.style.display flex; statusDiv.innerHTML ⏳ 正在上传并转录请稍候...; const formData new FormData(); formData.append(video, file); try { const response await fetch(/transcribe, { method: POST, body: formData }); const data await response.json(); if (!response.ok || !data.success) { throw new Error(data.error || 转录失败); } segments data.segments || []; renderTranscript(segments); statusDiv.innerHTML ✅ 转录完成点击句子可跳转播放; timeUpdateHandler syncHighlight; videoPlayer.addEventListener(timeupdate, timeUpdateHandler); } catch (err) { statusDiv.innerHTML ❌ 错误${err.message}; console.error(err); } }); // 事件委托点击行跳转 transcriptContainer.addEventListener(click, (event) { const line event.target.closest(.transcript-line); if (!line) return; const startStr line.dataset.start; console.log(点击行>3.3模型下载与准备Speaker-Reasoner-4194h面向带时间戳说话人归属自动语音识别的交互轮次与推理模式扩展多模态大模型下载链接https://modelscope.cn/models/ASLP-lab/Speaker-Reasoner-4194h/files3.4模型启动参数Speaker-Reasoner-4194h启动参数cat SoulX-Transcriber.sh vllm serve /home/models/Speaker-Reasoner-4194h \ --port 8082 \ --served-model-name Speaker-Reasoner-4194h \ --trust-remote-code \ --tensor-parallel-size 2 \ --dtype bfloat16 \ --max-model-len 65536 \ --gpu-memory-utilization 0.95 \ --allowed-local-media-path / \ --chat-template /home/models/qwen3-omni_vllm-main/Speaker-Reasoner-web/chat_template.jinja cat /home/models/qwen3-omni_vllm-main/Speaker-Reasoner-web/chat_template.jinja {% for message in messages %}{{ |im_start| message[role] \n }}{% if message[content] is string %}{{ message[content] }}{% else %}{% for item in message[content] %}{% if item[type] text %}{{ item[text] }}{% elif item[type] audio %}{{ |audio_start||audio_pad||audio_end| }}{% elif item[type] image %}{{ |vision_start||image_pad||vision_end| }}{% elif item[type] video %}{{ |vision_start||video_pad||vision_end| }}{% endif %}{% endfor %}{% endif %}{{ |im_end|\n }}{% endfor %}{% if add_generation_prompt %}{{ |im_start|assistant\n }}{% endif %}四、运行测试1、启动Speaker-Reasoner-4194hdocker exec -it Speaker-Reasoner-4194h bash cd /home/models/ nohup ./Speaker-Reasoner-4194h.sh 2、启动超长离线音视频说话人角色确认转录系统自编程序python Speaker-Reasoner-web-2.py * Serving Flask app Speaker-Reasoner-web-2 * Debug mode: on INFO:werkzeug:WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:5000 * Running on http://192.168.222.65:5000 INFO:werkzeug:Press CTRLC to quit INFO:werkzeug: * Restarting with stat WARNING:werkzeug: * Debugger is active! INFO:werkzeug: * Debugger PIN: 105-818-0563、访问超长离线音视频说话人角色确认转录系统自编程序用谷歌浏览器访问转录完成后点击右边有时间段的文本记录播放器会自动跳转到指定时间位置进行播放