diff --git a/.github/workflows/build-with-all-capacity.yml b/.github/workflows/build-with-all-capacity.yml
new file mode 100644
index 0000000..7b2ee6a
--- /dev/null
+++ b/.github/workflows/build-with-all-capacity.yml
@@ -0,0 +1,44 @@
+# https://docs.github.com/en/actions/publishing-packages/publishing-docker-images#publishing-images-to-github-packages
+name: build-with-all-capacity
+
+on:
+ push:
+ branches:
+ - 'master'
+
+env:
+ REGISTRY: ghcr.io
+ IMAGE_NAME: ${{ github.repository }}_with_all_capacity
+
+jobs:
+ build-and-push-image:
+ runs-on: ubuntu-latest
+ permissions:
+ contents: read
+ packages: write
+
+ steps:
+ - name: Checkout repository
+ uses: actions/checkout@v3
+
+ - name: Log in to the Container registry
+ uses: docker/login-action@v2
+ with:
+ registry: ${{ env.REGISTRY }}
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+
+ - name: Extract metadata (tags, labels) for Docker
+ id: meta
+ uses: docker/metadata-action@v4
+ with:
+ images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
+
+ - name: Build and push Docker image
+ uses: docker/build-push-action@v4
+ with:
+ context: .
+ push: true
+ file: docs/GithubAction+AllCapacity
+ tags: ${{ steps.meta.outputs.tags }}
+ labels: ${{ steps.meta.outputs.labels }}
diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml
new file mode 100644
index 0000000..717f254
--- /dev/null
+++ b/.github/workflows/stale.yml
@@ -0,0 +1,25 @@
+# This workflow warns and then closes issues and PRs that have had no activity for a specified amount of time.
+#
+# You can adjust the behavior by modifying this file.
+# For more information, see:
+# https://github.com/actions/stale
+
+name: 'Close stale issues and PRs'
+on:
+ schedule:
+ - cron: '*/5 * * * *'
+
+jobs:
+ stale:
+ runs-on: ubuntu-latest
+ permissions:
+ issues: write
+ pull-requests: read
+
+ steps:
+ - uses: actions/stale@v8
+ with:
+ stale-issue-message: 'This issue is stale because it has been open 100 days with no activity. Remove stale label or comment or this will be closed in 1 days.'
+ days-before-stale: 100
+ days-before-close: 1
+ debug-only: true
diff --git a/README.md b/README.md
index 761e303..ef463bd 100644
--- a/README.md
+++ b/README.md
@@ -10,13 +10,13 @@
**如果喜欢这个项目，请给它一个Star；如果您发明了好用的快捷键或函数插件，欢迎发pull requests！**
If you like this project, please give it a Star. If you've come up with more useful academic shortcuts or functional plugins, feel free to open an issue or pull request. We also have a README in [English|](docs/README_EN.md)[日本語|](docs/README_JP.md)[한국어|](https://github.com/mldljyh/ko_gpt_academic)[Русский|](docs/README_RS.md)[Français](docs/README_FR.md) translated by this project itself.
-To translate this project to arbitary language with GPT, read and run [`multi_language.py`](multi_language.py) (experimental).
+To translate this project to arbitrary language with GPT, read and run [`multi_language.py`](multi_language.py) (experimental).
> **Note**
>
-> 1.请注意只有 **高亮(如红色)** 标识的函数插件（按钮）才支持读取文件，部分插件位于插件区的**下拉菜单**中。另外我们以**最高优先级**欢迎和处理任何新插件的PR。
+> 1.请注意只有 **高亮** 标识的函数插件（按钮）才支持读取文件，部分插件位于插件区的**下拉菜单**中。另外我们以**最高优先级**欢迎和处理任何新插件的PR。
>
-> 2.本项目中每个文件的功能都在[自译解报告`self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告)详细说明。随着版本的迭代，您也可以随时自行点击相关函数插件，调用GPT重新生成项目的自我解析报告。常见问题汇总在[`wiki`](https://github.com/binary-husky/gpt_academic/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98)当中。[安装方法](#installation)。
+> 2.本项目中每个文件的功能都在[自译解报告`self_analysis.md`](https://github.com/binary-husky/gpt_academic/wiki/GPT‐Academic项目自译解报告)详细说明。随着版本的迭代，您也可以随时自行点击相关函数插件，调用GPT重新生成项目的自我解析报告。常见问题[`wiki`](https://github.com/binary-husky/gpt_academic/wiki)。[安装方法](#installation) | [配置说明](https://github.com/binary-husky/gpt_academic/wiki/%E9%A1%B9%E7%9B%AE%E9%85%8D%E7%BD%AE%E8%AF%B4%E6%98%8E)。
>
> 3.本项目兼容并鼓励尝试国产大语言模型ChatGLM和Moss等等。支持多个api-key共存，可在配置文件中填写如`API_KEY="openai-key1,openai-key2,azure-key3,api2d-key4"`。需要临时更换`API_KEY`时，在输入区输入临时的`API_KEY`然后回车键提交后即可生效。
@@ -53,7 +53,8 @@ Latex论文一键校对 | [函数插件] 仿Grammarly对Latex文章进行语法
[多LLM模型](https://www.bilibili.com/video/BV1wT411p7yf)支持 | 同时被GPT3.5、GPT4、[清华ChatGLM2](https://github.com/THUDM/ChatGLM2-6B)、[复旦MOSS](https://github.com/OpenLMLab/MOSS)同时伺候的感觉一定会很不错吧？
⭐ChatGLM2微调模型 | 支持加载ChatGLM2微调模型，提供ChatGLM2微调辅助插件
更多LLM模型接入，支持[huggingface部署](https://huggingface.co/spaces/qingxu98/gpt-academic) | 加入Newbing接口(新必应)，引入清华[Jittorllms](https://github.com/Jittor/JittorLLMs)支持[LLaMA](https://github.com/facebookresearch/llama)和[盘古α](https://openi.org.cn/pangu/)
-⭐[虚空终端](https://github.com/binary-husky/void-terminal)pip包 | 脱离GUI，在Python中直接调用本项目的函数插件（开发中）
+⭐[void-terminal](https://github.com/binary-husky/void-terminal) pip包 | 脱离GUI，在Python中直接调用本项目的所有函数插件（开发中）
+⭐虚空终端插件 | [函数插件] 用自然语言，直接调度本项目其他插件
更多新功能展示 (图像生成等) …… | 见本文档结尾处 ……
@@ -148,11 +149,14 @@ python main.py
### 安装方法II：使用Docker
+[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
+
1. 仅ChatGPT（推荐大多数人选择，等价于docker-compose方案1）
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-without-local-llms.yml)
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-latex.yml)
[](https://github.com/binary-husky/gpt_academic/actions/workflows/build-with-audio-assistant.yml)
+
``` sh
git clone --depth=1 https://github.com/binary-husky/gpt_academic.git # 下载项目
cd gpt_academic # 进入路径
@@ -249,10 +253,13 @@ Tip：不指定文件直接点击 `载入对话历史存档` 可以查看历史h
-3. 生成报告。大部分插件都会在执行结束后，生成工作报告
+3. 虚空终端（从自然语言输入中，理解用户意图+自动调用其他插件）
+
+- 步骤一：输入 “ 请调用插件翻译PDF论文，地址为https://storage.googleapis.com/deepmind-media/alphago/AlphaGoNaturePaper.pdf ”
+- 步骤二：点击“虚空终端”
+
4. 模块化功能设计，简单的接口却能支持强大的功能
@@ -299,8 +306,10 @@ Tip：不指定文件直接点击 `载入对话历史存档` 可以查看历史h
+
### II：版本:
-- version 3.5(Todo): 使用自然语言调用本项目的所有函数插件（高优先级）
+- version 3.60（todo）: 优化虚空终端，引入code interpreter和更多插件
+- version 3.50: 使用自然语言调用本项目的所有函数插件（虚空终端），支持插件分类，改进UI，设计新主题
- version 3.49: 支持百度千帆平台和文心一言
- version 3.48: 支持阿里达摩院通义千问，上海AI-Lab书生，讯飞星火
- version 3.46: 支持完全脱手操作的实时语音对话
diff --git a/check_proxy.py b/check_proxy.py
index 474988c..b6fe99f 100644
--- a/check_proxy.py
+++ b/check_proxy.py
@@ -5,7 +5,7 @@ def check_proxy(proxies):
try:
response = requests.get("https://ipapi.co/json/", proxies=proxies, timeout=4)
data = response.json()
- print(f'查询代理的地理位置，返回的结果是{data}')
+ # print(f'查询代理的地理位置，返回的结果是{data}')
if 'country_name' in data:
country = data['country_name']
result = f"代理配置 {proxies_https}, 代理所在地：{country}"
diff --git a/config.py b/config.py
index d2d5db6..1e91e39 100644
--- a/config.py
+++ b/config.py
@@ -43,7 +43,11 @@ API_URL_REDIRECT = {}
DEFAULT_WORKER_NUM = 3
-# 对话窗的高度
+# 色彩主题，可选 ["Default", "Chuanhu-Small-and-Beautiful", "High-Contrast"]
+THEME = "Default"
+
+
+# 对话窗的高度 （仅在LAYOUT="TOP-DOWN"时生效）
CHATBOT_HEIGHT = 1115
@@ -68,6 +72,10 @@ WEB_PORT = -1
MAX_RETRY = 2
+# 插件分类默认选项
+DEFAULT_FN_GROUPS = ['对话', '编程', '学术']
+
+
# 模型选择是 (注意: LLM_MODEL是默认选中的模型, 它*必须*被包含在AVAIL_LLM_MODELS列表中 )
LLM_MODEL = "gpt-3.5-turbo" # 可选 ↓↓↓
AVAIL_LLM_MODELS = ["gpt-3.5-turbo-16k", "gpt-3.5-turbo", "azure-gpt-3.5", "api2d-gpt-3.5-turbo",
@@ -83,7 +91,7 @@ BAIDU_CLOUD_QIANFAN_MODEL = 'ERNIE-Bot' # 可选 "ERNIE-Bot"(文心一言), "
# 如果使用ChatGLM2微调模型，请把 LLM_MODEL="chatglmft"，并在此处指定模型路径
-ChatGLM_PTUNING_CHECKPOINT = "" # 例如"/home/hmp/ChatGLM2-6B/ptuning/output/6b-pt-128-1e-2/checkpoint-100"
+CHATGLM_PTUNING_CHECKPOINT = "" # 例如"/home/hmp/ChatGLM2-6B/ptuning/output/6b-pt-128-1e-2/checkpoint-100"
# 本地LLM模型如ChatGLM的执行方式 CPU/GPU
@@ -99,10 +107,6 @@ CONCURRENT_COUNT = 100
AUTO_CLEAR_TXT = False
-# 色彩主体，可选 ["Default", "Chuanhu-Small-and-Beautiful", "High-Contrast"]
-THEME = "Default"
-
-
# 加一个live2d装饰
ADD_WAIFU = False
@@ -213,6 +217,18 @@ ALLOW_RESET_CONFIG = False
├── NEWBING_STYLE
└── NEWBING_COOKIES
+
+用户图形界面布局依赖关系示意图
+│
+├── CHATBOT_HEIGHT 对话窗的高度
+├── CODE_HIGHLIGHT 代码高亮
+├── LAYOUT 窗口布局
+├── DARK_MODE 暗色模式 / 亮色模式
+├── DEFAULT_FN_GROUPS 插件分类默认选项
+├── THEME 色彩主题
+├── AUTO_CLEAR_TXT 是否在提交时自动清空输入框
+├── ADD_WAIFU 加一个live2d装饰
+├── ALLOW_RESET_CONFIG 是否允许通过自然语言描述修改本页的配置，该功能具有一定的危险性
插件在线服务配置依赖关系示意图
diff --git a/core_functional.py b/core_functional.py
index b04e1e0..c4519ef 100644
--- a/core_functional.py
+++ b/core_functional.py
@@ -63,6 +63,7 @@ def get_core_functions():
"英译中": {
"Prefix": r"翻译成地道的中文：" + "\n\n",
"Suffix": r"",
+ "Visible": False,
},
"找图片": {
"Prefix": r"我需要你找一张网络图片。使用Unsplash API(https://source.unsplash.com/960x640/?<英语关键词>)获取图片URL，" +
@@ -78,6 +79,7 @@ def get_core_functions():
"Prefix": r"Here are some bibliography items, please transform them into bibtex style." +
r"Note that, reference styles maybe more than one kind, you should transform each item correctly." +
r"Items need to be transformed:",
+ "Visible": False,
"Suffix": r"",
}
}
diff --git a/crazy_functional.py b/crazy_functional.py
index fcc4e52..c657855 100644
--- a/crazy_functional.py
+++ b/crazy_functional.py
@@ -34,87 +34,108 @@ def get_crazy_functions():
from crazy_functions.Latex全文翻译 import Latex中译英
from crazy_functions.Latex全文翻译 import Latex英译中
from crazy_functions.批量Markdown翻译 import Markdown中译英
+ from crazy_functions.虚空终端 import 虚空终端
function_plugins = {
+ "虚空终端": {
+ "Group": "对话|编程|学术",
+ "Color": "stop",
+ "AsButton": True,
+ "Function": HotReload(虚空终端)
+ },
"解析整个Python项目": {
+ "Group": "编程",
"Color": "stop",
"AsButton": True,
"Info": "解析一个Python项目的所有源文件(.py) | 输入参数为路径",
"Function": HotReload(解析一个Python项目)
},
"载入对话历史存档（先上传存档或输入路径）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"Info": "载入对话历史存档 | 输入参数为路径",
"Function": HotReload(载入对话历史存档)
},
"删除所有本地对话历史记录（谨慎操作）": {
+ "Group": "对话",
"AsButton": False,
"Info": "删除所有本地对话历史记录，谨慎操作 | 不需要输入参数",
"Function": HotReload(删除所有本地对话历史记录)
},
"清除所有缓存文件（谨慎操作）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "清除所有缓存文件，谨慎操作 | 不需要输入参数",
"Function": HotReload(清除缓存)
},
"批量总结Word文档": {
+ "Group": "学术",
"Color": "stop",
"AsButton": True,
"Info": "批量总结word文档 | 输入参数为路径",
"Function": HotReload(总结word文档)
},
"解析整个C++项目头文件": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个C++项目的所有头文件(.h/.hpp) | 输入参数为路径",
"Function": HotReload(解析一个C项目的头文件)
},
"解析整个C++项目（.cpp/.hpp/.c/.h）": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个C++项目的所有源文件（.cpp/.hpp/.c/.h）| 输入参数为路径",
"Function": HotReload(解析一个C项目)
},
"解析整个Go项目": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个Go项目的所有源文件 | 输入参数为路径",
"Function": HotReload(解析一个Golang项目)
},
"解析整个Rust项目": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个Rust项目的所有源文件 | 输入参数为路径",
"Function": HotReload(解析一个Rust项目)
},
"解析整个Java项目": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个Java项目的所有源文件 | 输入参数为路径",
"Function": HotReload(解析一个Java项目)
},
"解析整个前端项目（js,ts,css等）": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个前端项目的所有源文件（js,ts,css等） | 输入参数为路径",
"Function": HotReload(解析一个前端项目)
},
"解析整个Lua项目": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个Lua项目的所有源文件 | 输入参数为路径",
"Function": HotReload(解析一个Lua项目)
},
"解析整个CSharp项目": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Info": "解析一个CSharp项目的所有源文件 | 输入参数为路径",
"Function": HotReload(解析一个CSharp项目)
},
"解析Jupyter Notebook文件": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False,
"Info": "解析Jupyter Notebook文件 | 输入参数为路径",
@@ -123,92 +144,125 @@ def get_crazy_functions():
"ArgsReminder": "若输入0，则不解析notebook中的Markdown块", # 高级参数输入区的显示提示
},
"读Tex论文写摘要": {
+ "Group": "学术",
"Color": "stop",
- "AsButton": True,
+ "AsButton": False,
+ "Info": "读取Tex论文并写摘要 | 输入参数为路径",
"Function": HotReload(读文章写摘要)
},
- "翻译README或.MD": {
+ "翻译README或MD": {
+ "Group": "编程",
"Color": "stop",
"AsButton": True,
"Info": "将Markdown翻译为中文 | 输入参数为路径或URL",
"Function": HotReload(Markdown英译中)
},
"翻译Markdown或README（支持Github链接）": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False,
+ "Info": "将Markdown或README翻译为中文 | 输入参数为路径或URL",
"Function": HotReload(Markdown英译中)
},
"批量生成函数注释": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "批量生成函数的注释 | 输入参数为路径",
"Function": HotReload(批量生成函数注释)
},
"保存当前的对话": {
+ "Group": "对话",
"AsButton": True,
"Info": "保存当前的对话 | 不需要输入参数",
"Function": HotReload(对话历史存档)
},
- "[多线程Demo] 解析此项目本身（源码自译解）": {
+ "[多线程Demo]解析此项目本身（源码自译解）": {
+ "Group": "对话|编程",
"AsButton": False, # 加入下拉菜单中
+ "Info": "多线程解析并翻译此项目的源码 | 不需要输入参数",
"Function": HotReload(解析项目本身)
},
- "[插件demo] 历史上的今天": {
+ "[插件demo]历史上的今天": {
+ "Group": "对话",
"AsButton": True,
+ "Info": "查看历史上的今天事件 | 不需要输入参数",
"Function": HotReload(高阶功能模板函数)
},
"精准翻译PDF论文": {
+ "Group": "学术",
"Color": "stop",
- "AsButton": True, # 加入下拉菜单中
+ "AsButton": True,
+ "Info": "精准翻译PDF论文为中文 | 输入参数为路径",
"Function": HotReload(批量翻译PDF文档)
},
"询问多个GPT模型": {
+ "Group": "对话",
"Color": "stop",
"AsButton": True,
"Function": HotReload(同时问询)
},
"批量总结PDF文档": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "批量总结PDF文档的内容 | 输入参数为路径",
"Function": HotReload(批量总结PDF文档)
},
"谷歌学术检索助手（输入谷歌学术搜索页url）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "使用谷歌学术检索助手搜索指定URL的结果 | 输入参数为谷歌学术搜索页的URL",
"Function": HotReload(谷歌检索小助手)
},
"理解PDF文档内容 （模仿ChatPDF）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "理解PDF文档的内容并进行回答 | 输入参数为路径",
"Function": HotReload(理解PDF文档内容标准文件输入)
},
"英文Latex项目全文润色（输入路径或上传压缩包）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "对英文Latex项目全文进行润色处理 | 输入参数为路径或上传压缩包",
"Function": HotReload(Latex英文润色)
},
"英文Latex项目全文纠错（输入路径或上传压缩包）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "对英文Latex项目全文进行纠错处理 | 输入参数为路径或上传压缩包",
"Function": HotReload(Latex英文纠错)
},
"中文Latex项目全文润色（输入路径或上传压缩包）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "对中文Latex项目全文进行润色处理 | 输入参数为路径或上传压缩包",
"Function": HotReload(Latex中文润色)
},
"Latex项目全文中译英（输入路径或上传压缩包）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "对Latex项目全文进行中译英处理 | 输入参数为路径或上传压缩包",
"Function": HotReload(Latex中译英)
},
"Latex项目全文英译中（输入路径或上传压缩包）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "对Latex项目全文进行英译中处理 | 输入参数为路径或上传压缩包",
"Function": HotReload(Latex英译中)
},
"批量Markdown中译英（输入路径或上传压缩包）": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "批量将Markdown文件中文翻译为英文 | 输入参数为路径或上传压缩包",
"Function": HotReload(Markdown中译英)
},
}
@@ -218,8 +272,10 @@ def get_crazy_functions():
from crazy_functions.下载arxiv论文翻译摘要 import 下载arxiv论文并翻译摘要
function_plugins.update({
"一键下载arxiv论文并翻译摘要（先在input输入编号，如1812.10695）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ # "Info": "下载arxiv论文并翻译摘要 | 输入参数为arxiv编号如1812.10695",
"Function": HotReload(下载arxiv论文并翻译摘要)
}
})
@@ -230,16 +286,20 @@ def get_crazy_functions():
from crazy_functions.联网的ChatGPT import 连接网络回答问题
function_plugins.update({
"连接网络回答问题（输入问题后点击该插件，需要访问谷歌）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ # "Info": "连接网络回答问题（需要访问谷歌）| 输入参数是一个问题",
"Function": HotReload(连接网络回答问题)
}
})
from crazy_functions.联网的ChatGPT_bing版 import 连接bing搜索回答问题
function_plugins.update({
"连接网络回答问题（中文Bing版，输入问题后点击该插件）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
+ "Info": "连接网络回答问题（需要访问中文Bing）| 输入参数是一个问题",
"Function": HotReload(连接bing搜索回答问题)
}
})
@@ -250,6 +310,7 @@ def get_crazy_functions():
from crazy_functions.解析项目源代码 import 解析任意code项目
function_plugins.update({
"解析项目源代码（手动指定和筛选源代码文件类型）": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True, # 调用时，唤起高级参数输入区（默认False）
@@ -264,6 +325,7 @@ def get_crazy_functions():
from crazy_functions.询问多个大语言模型 import 同时问询_指定模型
function_plugins.update({
"询问多个GPT模型（手动指定询问哪些模型）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True, # 调用时，唤起高级参数输入区（默认False）
@@ -278,6 +340,7 @@ def get_crazy_functions():
from crazy_functions.图片生成 import 图片生成
function_plugins.update({
"图片生成（先切换模型到openai或api2d）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True, # 调用时，唤起高级参数输入区（默认False）
@@ -293,6 +356,7 @@ def get_crazy_functions():
from crazy_functions.总结音视频 import 总结音视频
function_plugins.update({
"批量总结音视频（输入路径或上传压缩包）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -308,8 +372,10 @@ def get_crazy_functions():
from crazy_functions.数学动画生成manim import 动画生成
function_plugins.update({
"数学动画生成（Manim）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
+ "Info": "按照自然语言描述生成一个动画 | 输入参数是一段话",
"Function": HotReload(动画生成)
}
})
@@ -320,6 +386,7 @@ def get_crazy_functions():
from crazy_functions.批量Markdown翻译 import Markdown翻译指定语言
function_plugins.update({
"Markdown翻译（手动指定语言）": {
+ "Group": "编程",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -334,6 +401,7 @@ def get_crazy_functions():
from crazy_functions.Langchain知识库 import 知识库问答
function_plugins.update({
"构建知识库（请先上传文件素材）": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -348,6 +416,7 @@ def get_crazy_functions():
from crazy_functions.Langchain知识库 import 读取知识库作答
function_plugins.update({
"知识库问答": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -362,6 +431,7 @@ def get_crazy_functions():
from crazy_functions.交互功能函数模板 import 交互功能模板函数
function_plugins.update({
"交互功能模板函数": {
+ "Group": "对话",
"Color": "stop",
"AsButton": False,
"Function": HotReload(交互功能模板函数)
@@ -374,6 +444,7 @@ def get_crazy_functions():
from crazy_functions.Latex输出PDF结果 import Latex英文纠错加PDF对比
function_plugins.update({
"Latex英文纠错+高亮修正位置 [需Latex]": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -384,6 +455,7 @@ def get_crazy_functions():
from crazy_functions.Latex输出PDF结果 import Latex翻译中文并重新编译PDF
function_plugins.update({
"Arixv论文精细翻译（输入arxivID）[需Latex]": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -391,11 +463,13 @@ def get_crazy_functions():
"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 " +
"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: " +
'If the term "agent" is used in this section, it should be translated to "智能体". ',
+ "Info": "Arixv论文精细翻译 | 输入参数arxiv论文的ID，比如1812.10695",
"Function": HotReload(Latex翻译中文并重新编译PDF)
}
})
function_plugins.update({
"本地Latex论文精细翻译（上传Latex项目）[需Latex]": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False,
"AdvancedArgs": True,
@@ -403,6 +477,7 @@ def get_crazy_functions():
"如果有必要, 请在此处给出自定义翻译命令, 解决部分词汇翻译不准确的问题。 " +
"例如当单词'agent'翻译不准确时, 请尝试把以下指令复制到高级参数区: " +
'If the term "agent" is used in this section, it should be translated to "智能体". ',
+ "Info": "本地Latex论文精细翻译 | 输入参数是路径",
"Function": HotReload(Latex翻译中文并重新编译PDF)
}
})
@@ -416,8 +491,10 @@ def get_crazy_functions():
from crazy_functions.语音助手 import 语音助手
function_plugins.update({
"实时音频采集": {
+ "Group": "对话",
"Color": "stop",
"AsButton": True,
+ "Info": "开始语言对话 | 没有输入参数",
"Function": HotReload(语音助手)
}
})
@@ -425,17 +502,32 @@ def get_crazy_functions():
print('Load function plugin failed')
try:
- from crazy_functions.虚空终端 import 自动终端
+ from crazy_functions.批量翻译PDF文档_NOUGAT import 批量翻译PDF文档
function_plugins.update({
- "自动终端": {
+ "精准翻译PDF文档（NOUGAT）": {
+ "Group": "学术",
"Color": "stop",
"AsButton": False,
- "Function": HotReload(自动终端)
+ "Function": HotReload(批量翻译PDF文档)
}
})
except:
print('Load function plugin failed')
+
+ # try:
+ # from crazy_functions.CodeInterpreter import 虚空终端CodeInterpreter
+ # function_plugins.update({
+ # "CodeInterpreter（开发中，仅供测试）": {
+ # "Group": "编程|对话",
+ # "Color": "stop",
+ # "AsButton": False,
+ # "Function": HotReload(虚空终端CodeInterpreter)
+ # }
+ # })
+ # except:
+ # print('Load function plugin failed')
+
# try:
# from crazy_functions.chatglm微调工具 import 微调数据集生成
# function_plugins.update({
@@ -449,4 +541,24 @@ def get_crazy_functions():
# })
# except:
# print('Load function plugin failed')
+
+
+
+ """
+ 设置默认值:
+ - 默认 Group = 对话
+ - 默认 AsButton = True
+ - 默认 AdvancedArgs = False
+ - 默认 Color = secondary
+ """
+ for name, function_meta in function_plugins.items():
+ if "Group" not in function_meta:
+ function_plugins[name]["Group"] = '对话'
+ if "AsButton" not in function_meta:
+ function_plugins[name]["AsButton"] = True
+ if "AdvancedArgs" not in function_meta:
+ function_plugins[name]["AdvancedArgs"] = False
+ if "Color" not in function_meta:
+ function_plugins[name]["Color"] = 'secondary'
+
return function_plugins
diff --git a/crazy_functions/CodeInterpreter.py b/crazy_functions/CodeInterpreter.py
new file mode 100644
index 0000000..3c970f3
--- /dev/null
+++ b/crazy_functions/CodeInterpreter.py
@@ -0,0 +1,231 @@
+from collections.abc import Callable, Iterable, Mapping
+from typing import Any
+from toolbox import CatchException, update_ui, gen_time_str, trimmed_format_exc, promote_file_to_downloadzone, clear_file_downloadzone
+from .crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
+from .crazy_utils import input_clipping, try_install_deps
+from multiprocessing import Process, Pipe
+import os
+import time
+
+templete = """
+```python
+import ... # Put dependencies here, e.g. import numpy as np
+
+class TerminalFunction(object): # Do not change the name of the class, The name of the class must be `TerminalFunction`
+
+ def run(self, path): # The name of the function must be `run`, it takes only a positional argument.
+ # rewrite the function you have just written here
+ ...
+ return generated_file_path
+```
+"""
+
+def inspect_dependency(chatbot, history):
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+ return True
+
+def get_code_block(reply):
+ import re
+ pattern = r"```([\s\S]*?)```" # regex pattern to match code blocks
+ matches = re.findall(pattern, reply) # find all code blocks in text
+ if len(matches) == 1:
+ return matches[0].strip('python') # code block
+ for match in matches:
+ if 'class TerminalFunction' in match:
+ return match.strip('python') # code block
+ raise RuntimeError("GPT is not generating proper code.")
+
+def gpt_interact_multi_step(txt, file_type, llm_kwargs, chatbot, history):
+ # 输入
+ prompt_compose = [
+ f'Your job:\n'
+ f'1. write a single Python function, which takes a path of a `{file_type}` file as the only argument and returns a `string` containing the result of analysis or the path of generated files. \n',
+ f"2. You should write this function to perform following task: " + txt + "\n",
+ f"3. Wrap the output python function with markdown codeblock."
+ ]
+ i_say = "".join(prompt_compose)
+ demo = []
+
+ # 第一步
+ gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
+ inputs=i_say, inputs_show_user=i_say,
+ llm_kwargs=llm_kwargs, chatbot=chatbot, history=demo,
+ sys_prompt= r"You are a programmer."
+ )
+ history.extend([i_say, gpt_say])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
+
+ # 第二步
+ prompt_compose = [
+ "If previous stage is successful, rewrite the function you have just written to satisfy following templete: \n",
+ templete
+ ]
+ i_say = "".join(prompt_compose); inputs_show_user = "If previous stage is successful, rewrite the function you have just written to satisfy executable templete. "
+ gpt_say = yield from request_gpt_model_in_new_thread_with_ui_alive(
+ inputs=i_say, inputs_show_user=inputs_show_user,
+ llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
+ sys_prompt= r"You are a programmer."
+ )
+ code_to_return = gpt_say
+ history.extend([i_say, gpt_say])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
+
+ # # 第三步
+ # i_say = "Please list to packages to install to run the code above. Then show me how to use `try_install_deps` function to install them."
+ # i_say += 'For instance. `try_install_deps(["opencv-python", "scipy", "numpy"])`'
+ # installation_advance = yield from request_gpt_model_in_new_thread_with_ui_alive(
+ # inputs=i_say, inputs_show_user=inputs_show_user,
+ # llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
+ # sys_prompt= r"You are a programmer."
+ # )
+ # # # 第三步
+ # i_say = "Show me how to use `pip` to install packages to run the code above. "
+ # i_say += 'For instance. `pip install -r opencv-python scipy numpy`'
+ # installation_advance = yield from request_gpt_model_in_new_thread_with_ui_alive(
+ # inputs=i_say, inputs_show_user=i_say,
+ # llm_kwargs=llm_kwargs, chatbot=chatbot, history=history,
+ # sys_prompt= r"You are a programmer."
+ # )
+ installation_advance = ""
+
+ return code_to_return, installation_advance, txt, file_type, llm_kwargs, chatbot, history
+
+def make_module(code):
+ module_file = 'gpt_fn_' + gen_time_str().replace('-','_')
+ with open(f'gpt_log/{module_file}.py', 'w', encoding='utf8') as f:
+ f.write(code)
+
+ def get_class_name(class_string):
+ import re
+ # Use regex to extract the class name
+ class_name = re.search(r'class (\w+)\(', class_string).group(1)
+ return class_name
+
+ class_name = get_class_name(code)
+ return f"gpt_log.{module_file}->{class_name}"
+
+def init_module_instance(module):
+ import importlib
+ module_, class_ = module.split('->')
+ init_f = getattr(importlib.import_module(module_), class_)
+ return init_f()
+
+def for_immediate_show_off_when_possible(file_type, fp, chatbot):
+ if file_type in ['png', 'jpg']:
+ image_path = os.path.abspath(fp)
+ chatbot.append(['这是一张图片, 展示如下:',
+ f'本地文件地址:
`{image_path}`
'+
+ f'本地文件预览:
'
+ ])
+ return chatbot
+
+def subprocess_worker(instance, file_path, return_dict):
+ return_dict['result'] = instance.run(file_path)
+
+def have_any_recent_upload_files(chatbot):
+ _5min = 5 * 60
+ if not chatbot: return False # chatbot is None
+ most_recent_uploaded = chatbot._cookies.get("most_recent_uploaded", None)
+ if not most_recent_uploaded: return False # most_recent_uploaded is None
+ if time.time() - most_recent_uploaded["time"] < _5min: return True # most_recent_uploaded is new
+ else: return False # most_recent_uploaded is too old
+
+def get_recent_file_prompt_support(chatbot):
+ most_recent_uploaded = chatbot._cookies.get("most_recent_uploaded", None)
+ path = most_recent_uploaded['path']
+ return path
+
+@CatchException
+def 虚空终端CodeInterpreter(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, web_port):
+ """
+ txt 输入栏用户输入的文本，例如需要翻译的一段话，再例如一个包含了待处理文件的路径
+ llm_kwargs gpt模型参数，如温度和top_p等，一般原样传递下去就行
+ plugin_kwargs 插件模型的参数，暂时没有用武之地
+ chatbot 聊天显示框的句柄，用于显示给用户
+ history 聊天历史，前情提要
+ system_prompt 给gpt的静默提醒
+ web_port 当前软件运行的端口号
+ """
+ raise NotImplementedError
+
+ # 清空历史，以免输入溢出
+ history = []; clear_file_downloadzone(chatbot)
+
+ # 基本信息：功能、贡献者
+ chatbot.append([
+ "函数插件功能？",
+ "CodeInterpreter开源版, 此插件处于开发阶段, 建议暂时不要使用, 插件初始化中 ..."
+ ])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+
+ if have_any_recent_upload_files(chatbot):
+ file_path = get_recent_file_prompt_support(chatbot)
+ else:
+ chatbot.append(["文件检索", "没有发现任何近期上传的文件。"])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+
+ # 读取文件
+ if ("recently_uploaded_files" in plugin_kwargs) and (plugin_kwargs["recently_uploaded_files"] == ""): plugin_kwargs.pop("recently_uploaded_files")
+ recently_uploaded_files = plugin_kwargs.get("recently_uploaded_files", None)
+ file_path = recently_uploaded_files[-1]
+ file_type = file_path.split('.')[-1]
+
+ # 粗心检查
+ if 'private_upload' in txt:
+ chatbot.append([
+ "...",
+ f"请在输入框内填写需求，然后再次点击该插件（文件路径 {file_path} 已经被记忆）"
+ ])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+ return
+
+ # 开始干正事
+ for j in range(5): # 最多重试5次
+ try:
+ code, installation_advance, txt, file_type, llm_kwargs, chatbot, history = \
+ yield from gpt_interact_multi_step(txt, file_type, llm_kwargs, chatbot, history)
+ code = get_code_block(code)
+ res = make_module(code)
+ instance = init_module_instance(res)
+ break
+ except Exception as e:
+ chatbot.append([f"第{j}次代码生成尝试，失败了", f"错误追踪\n```\n{trimmed_format_exc()}\n```\n"])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+
+ # 代码生成结束, 开始执行
+ try:
+ import multiprocessing
+ manager = multiprocessing.Manager()
+ return_dict = manager.dict()
+
+ p = multiprocessing.Process(target=subprocess_worker, args=(instance, file_path, return_dict))
+ # only has 10 seconds to run
+ p.start(); p.join(timeout=10)
+ if p.is_alive(): p.terminate(); p.join()
+ p.close()
+ res = return_dict['result']
+ # res = instance.run(file_path)
+ except Exception as e:
+ chatbot.append(["执行失败了", f"错误追踪\n```\n{trimmed_format_exc()}\n```\n"])
+ # chatbot.append(["如果是缺乏依赖，请参考以下建议", installation_advance])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+ return
+
+ # 顺利完成，收尾
+ res = str(res)
+ if os.path.exists(res):
+ chatbot.append(["执行成功了，结果是一个有效文件", "结果：" + res])
+ new_file_path = promote_file_to_downloadzone(res, chatbot=chatbot)
+ chatbot = for_immediate_show_off_when_possible(file_type, new_file_path, chatbot)
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
+ else:
+ chatbot.append(["执行成功了，结果是一个字符串", "结果：" + res])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面 # 界面更新
+
+"""
+测试：
+ 裁剪图像，保留下半部分
+ 交换图像的蓝色通道和红色通道
+ 将图像转为灰度图像
+ 将csv文件转excel表格
+"""
\ No newline at end of file
diff --git a/crazy_functions/Latex输出PDF结果.py b/crazy_functions/Latex输出PDF结果.py
index e79cf82..8686f7e 100644
--- a/crazy_functions/Latex输出PDF结果.py
+++ b/crazy_functions/Latex输出PDF结果.py
@@ -6,7 +6,7 @@ pj = os.path.join
ARXIV_CACHE_DIR = os.path.expanduser(f"~/arxiv_cache/")
# =================================== 工具函数 ===============================================
-专业词汇声明 = 'If the term "agent" is used in this section, it should be translated to "智能体". '
+# 专业词汇声明 = 'If the term "agent" is used in this section, it should be translated to "智能体". '
def switch_prompt(pfg, mode, more_requirement):
"""
Generate prompts and system prompts based on the mode for proofreading or translating.
@@ -109,7 +109,7 @@ def arxiv_download(chatbot, history, txt):
url_ = txt # https://arxiv.org/abs/1707.06690
if not txt.startswith('https://arxiv.org/abs/'):
- msg = f"解析arxiv网址失败, 期望格式例如: https://arxiv.org/abs/1707.06690。实际得到格式: {url_}"
+ msg = f"解析arxiv网址失败, 期望格式例如: https://arxiv.org/abs/1707.06690。实际得到格式: {url_}。"
yield from update_ui_lastest_msg(msg, chatbot=chatbot, history=history) # 刷新界面
return msg, None
# <-------------- set format ------------->
@@ -255,7 +255,7 @@ def Latex翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot,
project_folder = txt
else:
if txt == "": txt = '空空如也的输入栏'
- report_execption(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
+ report_execption(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无法处理: {txt}")
yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
return
@@ -291,7 +291,7 @@ def Latex翻译中文并重新编译PDF(txt, llm_kwargs, plugin_kwargs, chatbot,
yield from update_ui(chatbot=chatbot, history=history); time.sleep(1) # 刷新界面
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
else:
- chatbot.append((f"失败了", '虽然PDF生成失败了, 但请查收结果（压缩包）, 内含已经翻译的Tex文档, 也是可读的, 您可以到Github Issue区, 用该压缩包+对话历史存档进行反馈 ...'))
+ chatbot.append((f"失败了", '虽然PDF生成失败了, 但请查收结果（压缩包）, 内含已经翻译的Tex文档, 您可以到Github Issue区, 用该压缩包进行反馈。如系统是Linux，请检查系统字体（见Github wiki） ...'))
yield from update_ui(chatbot=chatbot, history=history); time.sleep(1) # 刷新界面
promote_file_to_downloadzone(file=zip_res, chatbot=chatbot)
diff --git a/crazy_functions/crazy_utils.py b/crazy_functions/crazy_utils.py
index ffe95e2..5a314b3 100644
--- a/crazy_functions/crazy_utils.py
+++ b/crazy_functions/crazy_utils.py
@@ -591,11 +591,16 @@ def get_files_from_everything(txt, type): # type='.md'
# 网络的远程文件
import requests
from toolbox import get_conf
+ from toolbox import get_log_folder, gen_time_str
proxies, = get_conf('proxies')
- r = requests.get(txt, proxies=proxies)
- with open('./gpt_log/temp'+type, 'wb+') as f: f.write(r.content)
- project_folder = './gpt_log/'
- file_manifest = ['./gpt_log/temp'+type]
+ try:
+ r = requests.get(txt, proxies=proxies)
+ except:
+ raise ConnectionRefusedError(f"无法下载资源{txt}，请检查。")
+ path = os.path.join(get_log_folder(plugin_name='web_download'), gen_time_str()+type)
+ with open(path, 'wb+') as f: f.write(r.content)
+ project_folder = get_log_folder(plugin_name='web_download')
+ file_manifest = [path]
elif txt.endswith(type):
# 直接给定文件
file_manifest = [txt]
diff --git a/crazy_functions/json_fns/pydantic_io.py b/crazy_functions/json_fns/pydantic_io.py
index db92412..4e300d6 100644
--- a/crazy_functions/json_fns/pydantic_io.py
+++ b/crazy_functions/json_fns/pydantic_io.py
@@ -37,10 +37,19 @@ Here is the output schema:
{schema}
```"""
+
+PYDANTIC_FORMAT_INSTRUCTIONS_SIMPLE = """The output should be formatted as a JSON instance that conforms to the JSON schema below.
+```
+{schema}
+```"""
+
+class JsonStringError(Exception): ...
+
class GptJsonIO():
- def __init__(self, schema):
+ def __init__(self, schema, example_instruction=True):
self.pydantic_object = schema
+ self.example_instruction = example_instruction
self.format_instructions = self.generate_format_instructions()
def generate_format_instructions(self):
@@ -53,9 +62,11 @@ class GptJsonIO():
if "type" in reduced_schema:
del reduced_schema["type"]
# Ensure json in context is well-formed with double quotes.
- schema_str = json.dumps(reduced_schema)
-
- return PYDANTIC_FORMAT_INSTRUCTIONS.format(schema=schema_str)
+ if self.example_instruction:
+ schema_str = json.dumps(reduced_schema)
+ return PYDANTIC_FORMAT_INSTRUCTIONS.format(schema=schema_str)
+ else:
+ return PYDANTIC_FORMAT_INSTRUCTIONS_SIMPLE.format(schema=schema_str)
def generate_output(self, text):
# Greedy search for 1st json candidate.
@@ -95,6 +106,6 @@ class GptJsonIO():
except Exception as e:
# 没辙了，放弃治疗
logging.info('Repaire json fail.')
- raise RuntimeError('Cannot repair json.', str(e))
+ raise JsonStringError('Cannot repair json.', str(e))
return result
diff --git a/crazy_functions/live_audio/aliyunASR.py b/crazy_functions/live_audio/aliyunASR.py
index 9641057..ed67fcd 100644
--- a/crazy_functions/live_audio/aliyunASR.py
+++ b/crazy_functions/live_audio/aliyunASR.py
@@ -1,4 +1,4 @@
-import time, threading, json
+import time, logging, json
class AliyunASR():
@@ -12,14 +12,14 @@ class AliyunASR():
message = json.loads(message)
self.parsed_sentence = message['payload']['result']
self.event_on_entence_end.set()
- print(self.parsed_sentence)
+ # print(self.parsed_sentence)
def test_on_start(self, message, *args):
# print("test_on_start:{}".format(message))
pass
def test_on_error(self, message, *args):
- print("on_error args=>{}".format(args))
+ logging.error("on_error args=>{}".format(args))
pass
def test_on_close(self, *args):
@@ -36,7 +36,6 @@ class AliyunASR():
# print("on_completed:args=>{} message=>{}".format(args, message))
pass
-
def audio_convertion_thread(self, uuid):
# 在一个异步线程中采集音频
import nls # pip install git+https://github.com/aliyun/alibabacloud-nls-python-sdk.git
diff --git a/crazy_functions/pdf_fns/parse_pdf.py b/crazy_functions/pdf_fns/parse_pdf.py
index 00016be..8a7117a 100644
--- a/crazy_functions/pdf_fns/parse_pdf.py
+++ b/crazy_functions/pdf_fns/parse_pdf.py
@@ -20,6 +20,11 @@ def get_avail_grobid_url():
def parse_pdf(pdf_path, grobid_url):
import scipdf # pip install scipdf_parser
if grobid_url.endswith('/'): grobid_url = grobid_url.rstrip('/')
- article_dict = scipdf.parse_pdf_to_dict(pdf_path, grobid_url=grobid_url)
+ try:
+ article_dict = scipdf.parse_pdf_to_dict(pdf_path, grobid_url=grobid_url)
+ except GROBID_OFFLINE_EXCEPTION:
+ raise GROBID_OFFLINE_EXCEPTION("GROBID服务不可用，请修改config中的GROBID_URL，可修改成本地GROBID服务。")
+ except:
+ raise RuntimeError("解析PDF失败，请检查PDF是否损坏。")
return article_dict
diff --git a/crazy_functions/vt_fns/vt_call_plugin.py b/crazy_functions/vt_fns/vt_call_plugin.py
index c1c1976..455ac88 100644
--- a/crazy_functions/vt_fns/vt_call_plugin.py
+++ b/crazy_functions/vt_fns/vt_call_plugin.py
@@ -1,9 +1,9 @@
from pydantic import BaseModel, Field
from typing import List
-from toolbox import update_ui_lastest_msg, get_conf
+from toolbox import update_ui_lastest_msg, disable_auto_promotion
from request_llm.bridge_all import predict_no_ui_long_connection
-from crazy_functions.json_fns.pydantic_io import GptJsonIO
-import copy, json, pickle, os, sys
+from crazy_functions.json_fns.pydantic_io import GptJsonIO, JsonStringError
+import copy, json, pickle, os, sys, time
def read_avail_plugin_enum():
@@ -11,37 +11,85 @@ def read_avail_plugin_enum():
plugin_arr = get_crazy_functions()
# remove plugins with out explaination
plugin_arr = {k:v for k, v in plugin_arr.items() if 'Info' in v}
- plugin_arr_info = {"F{:04d}".format(i):v["Info"] for i, v in enumerate(plugin_arr.values(), start=1)}
- plugin_arr_dict = {"F{:04d}".format(i):v for i, v in enumerate(plugin_arr.values(), start=1)}
+ plugin_arr_info = {"F_{:04d}".format(i):v["Info"] for i, v in enumerate(plugin_arr.values(), start=1)}
+ plugin_arr_dict = {"F_{:04d}".format(i):v for i, v in enumerate(plugin_arr.values(), start=1)}
+ plugin_arr_dict_parse = {"F_{:04d}".format(i):v for i, v in enumerate(plugin_arr.values(), start=1)}
+ plugin_arr_dict_parse.update({f"F_{i}":v for i, v in enumerate(plugin_arr.values(), start=1)})
prompt = json.dumps(plugin_arr_info, ensure_ascii=False, indent=2)
prompt = "\n\nThe defination of PluginEnum:\nPluginEnum=" + prompt
- return prompt, plugin_arr_dict
+ return prompt, plugin_arr_dict, plugin_arr_dict_parse
+def wrap_code(txt):
+ txt = txt.replace('```','')
+ return f"\n```\n{txt}\n```\n"
+
+def have_any_recent_upload_files(chatbot):
+ _5min = 5 * 60
+ if not chatbot: return False # chatbot is None
+ most_recent_uploaded = chatbot._cookies.get("most_recent_uploaded", None)
+ if not most_recent_uploaded: return False # most_recent_uploaded is None
+ if time.time() - most_recent_uploaded["time"] < _5min: return True # most_recent_uploaded is new
+ else: return False # most_recent_uploaded is too old
+
+def get_recent_file_prompt_support(chatbot):
+ most_recent_uploaded = chatbot._cookies.get("most_recent_uploaded", None)
+ path = most_recent_uploaded['path']
+ prompt = "\nAdditional Information:\n"
+ prompt = "In case that this plugin requires a path or a file as argument,"
+ prompt += f"it is important for you to know that the user has recently uploaded a file, located at: `{path}`"
+ prompt += f"Only use it when necessary, otherwise, you can ignore this file."
+ return prompt
+
+def get_inputs_show_user(inputs, plugin_arr_enum_prompt):
+ # remove plugin_arr_enum_prompt from inputs string
+ inputs_show_user = inputs.replace(plugin_arr_enum_prompt, "")
+ inputs_show_user += plugin_arr_enum_prompt[:200] + '...'
+ inputs_show_user += '\n...\n'
+ inputs_show_user += '...\n'
+ inputs_show_user += '...}'
+ return inputs_show_user
def execute_plugin(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, user_intention):
- plugin_arr_enum_prompt, plugin_arr_dict = read_avail_plugin_enum()
+ plugin_arr_enum_prompt, plugin_arr_dict, plugin_arr_dict_parse = read_avail_plugin_enum()
class Plugin(BaseModel):
- plugin_selection: str = Field(description="The most related plugin from one of the PluginEnum.", default="F0000000000000")
- plugin_arg: str = Field(description="The argument of the plugin. A path or url or empty.", default="")
-
+ plugin_selection: str = Field(description="The most related plugin from one of the PluginEnum.", default="F_0000")
+ reason_of_selection: str = Field(description="The reason why you should select this plugin.", default="This plugin satisfy user requirement most")
# ⭐ ⭐ ⭐ 选择插件
yield from update_ui_lastest_msg(lastmsg=f"正在执行任务: {txt}\n\n查找可用插件中...", chatbot=chatbot, history=history, delay=0)
gpt_json_io = GptJsonIO(Plugin)
+ gpt_json_io.format_instructions = "The format of your output should be a json that can be parsed by json.loads.\n"
+ gpt_json_io.format_instructions += """Output example: {"plugin_selection":"F_1234", "reason_of_selection":"F_1234 plugin satisfy user requirement most"}\n"""
+ gpt_json_io.format_instructions += "The plugins you are authorized to use are listed below:\n"
gpt_json_io.format_instructions += plugin_arr_enum_prompt
- inputs = "Choose the correct plugin and extract plugin_arg, the user requirement is: \n\n" + \
- ">> " + txt.rstrip('\n').replace('\n','\n>> ') + '\n\n' + \
- gpt_json_io.format_instructions
+ inputs = "Choose the correct plugin according to user requirements, the user requirement is: \n\n" + \
+ ">> " + txt.rstrip('\n').replace('\n','\n>> ') + '\n\n' + gpt_json_io.format_instructions
+
run_gpt_fn = lambda inputs, sys_prompt: predict_no_ui_long_connection(
inputs=inputs, llm_kwargs=llm_kwargs, history=[], sys_prompt=sys_prompt, observe_window=[])
- plugin_sel = gpt_json_io.generate_output_auto_repair(run_gpt_fn(inputs, ""), run_gpt_fn)
-
- if plugin_sel.plugin_selection not in plugin_arr_dict:
- msg = f'找不到合适插件执行该任务'
+ try:
+ gpt_reply = run_gpt_fn(inputs, "")
+ plugin_sel = gpt_json_io.generate_output_auto_repair(gpt_reply, run_gpt_fn)
+ except JsonStringError:
+ msg = f"抱歉, {llm_kwargs['llm_model']}无法理解您的需求。"
+ msg += "请求的Prompt为：\n" + wrap_code(get_inputs_show_user(inputs, plugin_arr_enum_prompt))
+ msg += "语言模型回复为：\n" + wrap_code(gpt_reply)
+ msg += "\n但您可以尝试再试一次\n"
+ yield from update_ui_lastest_msg(lastmsg=msg, chatbot=chatbot, history=history, delay=2)
+ return
+ if plugin_sel.plugin_selection not in plugin_arr_dict_parse:
+ msg = f"抱歉, 找不到合适插件执行该任务, 或者{llm_kwargs['llm_model']}无法理解您的需求。"
+ msg += f"语言模型{llm_kwargs['llm_model']}选择了不存在的插件：\n" + wrap_code(gpt_reply)
+ msg += "\n但您可以尝试再试一次\n"
yield from update_ui_lastest_msg(lastmsg=msg, chatbot=chatbot, history=history, delay=2)
return
# ⭐ ⭐ ⭐ 确认插件参数
- plugin = plugin_arr_dict[plugin_sel.plugin_selection]
+ if not have_any_recent_upload_files(chatbot):
+ appendix_info = ""
+ else:
+ appendix_info = get_recent_file_prompt_support(chatbot)
+
+ plugin = plugin_arr_dict_parse[plugin_sel.plugin_selection]
yield from update_ui_lastest_msg(lastmsg=f"正在执行任务: {txt}\n\n提取插件参数...", chatbot=chatbot, history=history, delay=0)
class PluginExplicit(BaseModel):
plugin_selection: str = plugin_sel.plugin_selection
@@ -50,7 +98,7 @@ def execute_plugin(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prom
gpt_json_io.format_instructions += "The information about this plugin is:" + plugin["Info"]
inputs = f"A plugin named {plugin_sel.plugin_selection} is selected, " + \
"you should extract plugin_arg from the user requirement, the user requirement is: \n\n" + \
- ">> " + txt.rstrip('\n').replace('\n','\n>> ') + '\n\n' + \
+ ">> " + (txt + appendix_info).rstrip('\n').replace('\n','\n>> ') + '\n\n' + \
gpt_json_io.format_instructions
run_gpt_fn = lambda inputs, sys_prompt: predict_no_ui_long_connection(
inputs=inputs, llm_kwargs=llm_kwargs, history=[], sys_prompt=sys_prompt, observe_window=[])
@@ -60,7 +108,7 @@ def execute_plugin(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prom
# ⭐ ⭐ ⭐ 执行插件
fn = plugin['Function']
fn_name = fn.__name__
- msg = f'正在调用插件: {fn_name}\n\n插件说明：{plugin["Info"]}\n\n插件参数：{plugin_sel.plugin_arg}'
+ msg = f'{llm_kwargs["llm_model"]}为您选择了插件: `{fn_name}`\n\n插件说明：{plugin["Info"]}\n\n插件参数：{plugin_sel.plugin_arg}\n\n假如偏离了您的要求，按停止键终止。'
yield from update_ui_lastest_msg(lastmsg=msg, chatbot=chatbot, history=history, delay=2)
yield from fn(plugin_sel.plugin_arg, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, -1)
return
\ No newline at end of file
diff --git a/crazy_functions/vt_fns/vt_state.py b/crazy_functions/vt_fns/vt_state.py
new file mode 100644
index 0000000..1818728
--- /dev/null
+++ b/crazy_functions/vt_fns/vt_state.py
@@ -0,0 +1,28 @@
+import pickle
+
+class VoidTerminalState():
+ def __init__(self):
+ self.reset_state()
+
+ def reset_state(self):
+ self.has_provided_explaination = False
+
+ def lock_plugin(self, chatbot):
+ chatbot._cookies['lock_plugin'] = 'crazy_functions.虚空终端->虚空终端'
+ chatbot._cookies['plugin_state'] = pickle.dumps(self)
+
+ def unlock_plugin(self, chatbot):
+ self.reset_state()
+ chatbot._cookies['lock_plugin'] = None
+ chatbot._cookies['plugin_state'] = pickle.dumps(self)
+
+ def set_state(self, chatbot, key, value):
+ setattr(self, key, value)
+ chatbot._cookies['plugin_state'] = pickle.dumps(self)
+
+ def get_state(chatbot):
+ state = chatbot._cookies.get('plugin_state', None)
+ if state is not None: state = pickle.loads(state)
+ else: state = VoidTerminalState()
+ state.chatbot = chatbot
+ return state
\ No newline at end of file
diff --git a/crazy_functions/批量翻译PDF文档_NOUGAT.py b/crazy_functions/批量翻译PDF文档_NOUGAT.py
new file mode 100644
index 0000000..ed15121
--- /dev/null
+++ b/crazy_functions/批量翻译PDF文档_NOUGAT.py
@@ -0,0 +1,271 @@
+from toolbox import CatchException, report_execption, gen_time_str
+from toolbox import update_ui, promote_file_to_downloadzone, update_ui_lastest_msg, disable_auto_promotion
+from toolbox import write_history_to_file, get_log_folder
+from .crazy_utils import request_gpt_model_in_new_thread_with_ui_alive
+from .crazy_utils import request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency
+from .crazy_utils import read_and_clean_pdf_text
+from .pdf_fns.parse_pdf import parse_pdf, get_avail_grobid_url
+from colorful import *
+import os
+import math
+import logging
+
+def markdown_to_dict(article_content):
+ import markdown
+ from bs4 import BeautifulSoup
+ cur_t = ""
+ cur_c = ""
+ results = {}
+ for line in article_content:
+ if line.startswith('#'):
+ if cur_t!="":
+ if cur_t not in results:
+ results.update({cur_t:cur_c.lstrip('\n')})
+ else:
+ # 处理重名的章节
+ results.update({cur_t + " " + gen_time_str():cur_c.lstrip('\n')})
+ cur_t = line.rstrip('\n')
+ cur_c = ""
+ else:
+ cur_c += line
+ results_final = {}
+ for k in list(results.keys()):
+ if k.startswith('# '):
+ results_final['title'] = k.split('# ')[-1]
+ results_final['authors'] = results.pop(k).lstrip('\n')
+ if k.startswith('###### Abstract'):
+ results_final['abstract'] = results.pop(k).lstrip('\n')
+
+ results_final_sections = []
+ for k,v in results.items():
+ results_final_sections.append({
+ 'heading':k.lstrip("# "),
+ 'text':v if len(v) > 0 else f"The beginning of {k.lstrip('# ')} section."
+ })
+ results_final['sections'] = results_final_sections
+ return results_final
+
+
+@CatchException
+def 批量翻译PDF文档(txt, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt, web_port):
+
+ disable_auto_promotion(chatbot)
+ # 基本信息：功能、贡献者
+ chatbot.append([
+ "函数插件功能？",
+ "批量翻译PDF文档。函数插件贡献者: Binary-Husky"])
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+
+ # 尝试导入依赖，如果缺少依赖，则给出安装建议
+ try:
+ import nougat
+ import tiktoken
+ except:
+ report_execption(chatbot, history,
+ a=f"解析项目: {txt}",
+ b=f"导入软件依赖失败。使用该模块需要额外依赖，安装方法```pip install --upgrade nougat-ocr tiktoken```。")
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+ return
+
+ # 清空历史，以免输入溢出
+ history = []
+
+ from .crazy_utils import get_files_from_everything
+ success, file_manifest, project_folder = get_files_from_everything(txt, type='.pdf')
+ # 检测输入参数，如没有给定输入参数，直接退出
+ if not success:
+ if txt == "": txt = '空空如也的输入栏'
+
+ # 如果没找到任何文件
+ if len(file_manifest) == 0:
+ report_execption(chatbot, history,
+ a=f"解析项目: {txt}", b=f"找不到任何.tex或.pdf文件: {txt}")
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+ return
+
+ # 开始正式执行任务
+ yield from 解析PDF_基于NOUGAT(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt)
+
+
+def nougat_with_timeout(command, cwd, timeout=3600):
+ import subprocess
+ process = subprocess.Popen(command, shell=True, cwd=cwd)
+ try:
+ stdout, stderr = process.communicate(timeout=timeout)
+ except subprocess.TimeoutExpired:
+ process.kill()
+ stdout, stderr = process.communicate()
+ print("Process timed out!")
+ return False
+ return True
+
+
+def NOUGAT_parse_pdf(fp):
+ import glob
+ from toolbox import get_log_folder, gen_time_str
+ dst = os.path.join(get_log_folder(plugin_name='nougat'), gen_time_str())
+ os.makedirs(dst)
+ nougat_with_timeout(f'nougat --out "{os.path.abspath(dst)}" "{os.path.abspath(fp)}"', os.getcwd())
+ res = glob.glob(os.path.join(dst,'*.mmd'))
+ if len(res) == 0:
+ raise RuntimeError("Nougat解析论文失败。")
+ return res[0]
+
+
+def 解析PDF_基于NOUGAT(file_manifest, project_folder, llm_kwargs, plugin_kwargs, chatbot, history, system_prompt):
+ import copy
+ import tiktoken
+ TOKEN_LIMIT_PER_FRAGMENT = 1280
+ generated_conclusion_files = []
+ generated_html_files = []
+ DST_LANG = "中文"
+ for index, fp in enumerate(file_manifest):
+ chatbot.append(["当前进度：", f"正在解析论文，请稍候。（第一次运行时，需要花费较长时间下载NOUGAT参数）"]); yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+ fpp = NOUGAT_parse_pdf(fp)
+
+ with open(fpp, 'r', encoding='utf8') as f:
+ article_content = f.readlines()
+ article_dict = markdown_to_dict(article_content)
+ logging.info(article_dict)
+
+ prompt = "以下是一篇学术论文的基本信息:\n"
+ # title
+ title = article_dict.get('title', '无法获取 title'); prompt += f'title:{title}\n\n'
+ # authors
+ authors = article_dict.get('authors', '无法获取 authors'); prompt += f'authors:{authors}\n\n'
+ # abstract
+ abstract = article_dict.get('abstract', '无法获取 abstract'); prompt += f'abstract:{abstract}\n\n'
+ # command
+ prompt += f"请将题目和摘要翻译为{DST_LANG}。"
+ meta = [f'# Title:\n\n', title, f'# Abstract:\n\n', abstract ]
+
+ # 单线，获取文章meta信息
+ paper_meta_info = yield from request_gpt_model_in_new_thread_with_ui_alive(
+ inputs=prompt,
+ inputs_show_user=prompt,
+ llm_kwargs=llm_kwargs,
+ chatbot=chatbot, history=[],
+ sys_prompt="You are an academic paper reader。",
+ )
+
+ # 多线，翻译
+ inputs_array = []
+ inputs_show_user_array = []
+
+ # get_token_num
+ from request_llm.bridge_all import model_info
+ enc = model_info[llm_kwargs['llm_model']]['tokenizer']
+ def get_token_num(txt): return len(enc.encode(txt, disallowed_special=()))
+ from .crazy_utils import breakdown_txt_to_satisfy_token_limit_for_pdf
+
+ def break_down(txt):
+ raw_token_num = get_token_num(txt)
+ if raw_token_num <= TOKEN_LIMIT_PER_FRAGMENT:
+ return [txt]
+ else:
+ # raw_token_num > TOKEN_LIMIT_PER_FRAGMENT
+ # find a smooth token limit to achieve even seperation
+ count = int(math.ceil(raw_token_num / TOKEN_LIMIT_PER_FRAGMENT))
+ token_limit_smooth = raw_token_num // count + count
+ return breakdown_txt_to_satisfy_token_limit_for_pdf(txt, get_token_fn=get_token_num, limit=token_limit_smooth)
+
+ for section in article_dict.get('sections'):
+ if len(section['text']) == 0: continue
+ section_frags = break_down(section['text'])
+ for i, fragment in enumerate(section_frags):
+ heading = section['heading']
+ if len(section_frags) > 1: heading += f' Part-{i+1}'
+ inputs_array.append(
+ f"你需要翻译{heading}章节，内容如下: \n\n{fragment}"
+ )
+ inputs_show_user_array.append(
+ f"# {heading}\n\n{fragment}"
+ )
+
+ gpt_response_collection = yield from request_gpt_model_multi_threads_with_very_awesome_ui_and_high_efficiency(
+ inputs_array=inputs_array,
+ inputs_show_user_array=inputs_show_user_array,
+ llm_kwargs=llm_kwargs,
+ chatbot=chatbot,
+ history_array=[meta for _ in inputs_array],
+ sys_prompt_array=[
+ "请你作为一个学术翻译，负责把学术论文准确翻译成中文。注意文章中的每一句话都要翻译。" for _ in inputs_array],
+ )
+ res_path = write_history_to_file(meta + ["# Meta Translation" , paper_meta_info] + gpt_response_collection, file_basename=None, file_fullname=None)
+ promote_file_to_downloadzone(res_path, rename_file=os.path.basename(fp)+'.md', chatbot=chatbot)
+ generated_conclusion_files.append(res_path)
+
+ ch = construct_html()
+ orig = ""
+ trans = ""
+ gpt_response_collection_html = copy.deepcopy(gpt_response_collection)
+ for i,k in enumerate(gpt_response_collection_html):
+ if i%2==0:
+ gpt_response_collection_html[i] = inputs_show_user_array[i//2]
+ else:
+ gpt_response_collection_html[i] = gpt_response_collection_html[i]
+
+ final = ["", "", "一、论文概况", "", "Abstract", paper_meta_info, "二、论文翻译", ""]
+ final.extend(gpt_response_collection_html)
+ for i, k in enumerate(final):
+ if i%2==0:
+ orig = k
+ if i%2==1:
+ trans = k
+ ch.add_row(a=orig, b=trans)
+ create_report_file_name = f"{os.path.basename(fp)}.trans.html"
+ html_file = ch.save_file(create_report_file_name)
+ generated_html_files.append(html_file)
+ promote_file_to_downloadzone(html_file, rename_file=os.path.basename(html_file), chatbot=chatbot)
+
+ chatbot.append(("给出输出文件清单", str(generated_conclusion_files + generated_html_files)))
+ yield from update_ui(chatbot=chatbot, history=history) # 刷新界面
+
+
+
+class construct_html():
+ def __init__(self) -> None:
+ self.css = """
+.row {
+ display: flex;
+ flex-wrap: wrap;
+}
+
+.column {
+ flex: 1;
+ padding: 10px;
+}
+
+.table-header {
+ font-weight: bold;
+ border-bottom: 1px solid black;
+}
+
+.table-row {
+ border-bottom: 1px solid lightgray;
+}
+
+.table-cell {
+ padding: 5px;
+}
+ """
+ self.html_string = f'翻译结果'
+
+
+ def add_row(self, a, b):
+ tmp = """
+
+
REPLACE_A
+
REPLACE_B
+
GPT 学术优化 {get_current_version()}
{theme_declaration}"
- description = """代码开源和更新[地址🚀](https://github.com/binary-husky/chatgpt_academic)，感谢热情的[开发者们❤️](https://github.com/binary-husky/chatgpt_academic/graphs/contributors)"""
+ description = "代码开源和更新[地址🚀](https://github.com/binary-husky/gpt_academic)，"
+ description += "感谢热情的[开发者们❤️](https://github.com/binary-husky/gpt_academic/graphs/contributors)"
# 问询记录, python 版本建议3.9+（越新越好）
import logging, uuid
@@ -34,7 +34,10 @@ def main():
# 高级函数插件
from crazy_functional import get_crazy_functions
- crazy_fns = get_crazy_functions()
+ DEFAULT_FN_GROUPS, = get_conf('DEFAULT_FN_GROUPS')
+ plugins = get_crazy_functions()
+ all_plugin_groups = list(set([g for _, plugin in plugins.items() for g in plugin['Group'].split('|')]))
+ match_group = lambda tags, groups: any([g in groups for g in tags.split('|')])
# 处理markdown文本格式的转变
gr.Chatbot.postprocess = format_io
@@ -83,25 +86,33 @@ def main():
if ("Visible" in functional[k]) and (not functional[k]["Visible"]): continue
variant = functional[k]["Color"] if "Color" in functional[k] else "secondary"
functional[k]["Button"] = gr.Button(k, variant=variant)
+ functional[k]["Button"].style(size="sm")
with gr.Accordion("函数插件区", open=True, elem_id="plugin-panel") as area_crazy_fn:
with gr.Row():
gr.Markdown("插件可读取“输入区”文本/路径作为参数（上传文件自动修正路径）")
+ with gr.Row(elem_id="input-plugin-group"):
+ plugin_group_sel = gr.Dropdown(choices=all_plugin_groups, label='', show_label=False, value=DEFAULT_FN_GROUPS,
+ multiselect=True, interactive=True, elem_classes='normal_mut_select').style(container=False)
with gr.Row():
- for k in crazy_fns:
- if not crazy_fns[k].get("AsButton", True): continue
- variant = crazy_fns[k]["Color"] if "Color" in crazy_fns[k] else "secondary"
- crazy_fns[k]["Button"] = gr.Button(k, variant=variant)
- crazy_fns[k]["Button"].style(size="sm")
+ for k, plugin in plugins.items():
+ if not plugin.get("AsButton", True): continue
+ visible = True if match_group(plugin['Group'], DEFAULT_FN_GROUPS) else False
+ variant = plugins[k]["Color"] if "Color" in plugin else "secondary"
+ plugin['Button'] = plugins[k]['Button'] = gr.Button(k, variant=variant, visible=visible).style(size="sm")
with gr.Row():
with gr.Accordion("更多函数插件", open=True):
- dropdown_fn_list = [k for k in crazy_fns.keys() if not crazy_fns[k].get("AsButton", True)]
+ dropdown_fn_list = []
+ for k, plugin in plugins.items():
+ if not match_group(plugin['Group'], DEFAULT_FN_GROUPS): continue
+ if not plugin.get("AsButton", True): dropdown_fn_list.append(k) # 排除已经是按钮的插件
+ elif plugin.get('AdvancedArgs', False): dropdown_fn_list.append(k) # 对于需要高级参数的插件，亦在下拉菜单中显示
with gr.Row():
dropdown = gr.Dropdown(dropdown_fn_list, value=r"打开插件列表", label="", show_label=False).style(container=False)
with gr.Row():
plugin_advanced_arg = gr.Textbox(show_label=True, label="高级参数输入区", visible=False,
placeholder="这里是特殊函数插件的高级参数输入区").style(container=False)
with gr.Row():
- switchy_bt = gr.Button(r"请先从插件列表中选择", variant="secondary")
+ switchy_bt = gr.Button(r"请先从插件列表中选择", variant="secondary").style(size="sm")
with gr.Row():
with gr.Accordion("点击展开“文件上传区”。上传本地文件/压缩包供函数插件调用。", open=False) as area_file_up:
file_upload = gr.Files(label="任何文件, 但推荐上传压缩文件(zip, tar)", file_count="multiple")
@@ -112,7 +123,6 @@ def main():
max_length_sl = gr.Slider(minimum=256, maximum=8192, value=4096, step=1, interactive=True, label="Local LLM MaxLength",)
checkboxes = gr.CheckboxGroup(["基础功能区", "函数插件区", "底部输入区", "输入清除键", "插件参数区"], value=["基础功能区", "函数插件区"], label="显示/隐藏功能区")
md_dropdown = gr.Dropdown(AVAIL_LLM_MODELS, value=LLM_MODEL, label="更换LLM模型/请求源").style(container=False)
-
gr.Markdown(description)
with gr.Accordion("备选输入区", open=True, visible=False, elem_id="input-panel2") as area_input_secondary:
with gr.Row():
@@ -123,6 +133,7 @@ def main():
resetBtn2 = gr.Button("重置", variant="secondary"); resetBtn2.style(size="sm")
stopBtn2 = gr.Button("停止", variant="secondary"); stopBtn2.style(size="sm")
clearBtn2 = gr.Button("清除", variant="secondary", visible=False); clearBtn2.style(size="sm")
+
# 功能区显示开关与功能区的互动
def fn_area_visibility(a):
ret = {}
@@ -160,19 +171,19 @@ def main():
click_handle = functional[k]["Button"].click(fn=ArgsGeneralWrapper(predict), inputs=[*input_combo, gr.State(True), gr.State(k)], outputs=output_combo)
cancel_handles.append(click_handle)
# 文件上传区，接收文件后与chatbot的互动
- file_upload.upload(on_file_uploaded, [file_upload, chatbot, txt, txt2, checkboxes], [chatbot, txt, txt2])
+ file_upload.upload(on_file_uploaded, [file_upload, chatbot, txt, txt2, checkboxes, cookies], [chatbot, txt, txt2, cookies])
# 函数插件-固定按钮区
- for k in crazy_fns:
- if not crazy_fns[k].get("AsButton", True): continue
- click_handle = crazy_fns[k]["Button"].click(ArgsGeneralWrapper(crazy_fns[k]["Function"]), [*input_combo, gr.State(PORT)], output_combo)
+ for k in plugins:
+ if not plugins[k].get("AsButton", True): continue
+ click_handle = plugins[k]["Button"].click(ArgsGeneralWrapper(plugins[k]["Function"]), [*input_combo, gr.State(PORT)], output_combo)
click_handle.then(on_report_generated, [cookies, file_upload, chatbot], [cookies, file_upload, chatbot])
cancel_handles.append(click_handle)
# 函数插件-下拉菜单与随变按钮的互动
def on_dropdown_changed(k):
- variant = crazy_fns[k]["Color"] if "Color" in crazy_fns[k] else "secondary"
+ variant = plugins[k]["Color"] if "Color" in plugins[k] else "secondary"
ret = {switchy_bt: gr.update(value=k, variant=variant)}
- if crazy_fns[k].get("AdvancedArgs", False): # 是否唤起高级插件参数区
- ret.update({plugin_advanced_arg: gr.update(visible=True, label=f"插件[{k}]的高级参数说明：" + crazy_fns[k].get("ArgsReminder", [f"没有提供高级参数功能说明"]))})
+ if plugins[k].get("AdvancedArgs", False): # 是否唤起高级插件参数区
+ ret.update({plugin_advanced_arg: gr.update(visible=True, label=f"插件[{k}]的高级参数说明：" + plugins[k].get("ArgsReminder", [f"没有提供高级参数功能说明"]))})
else:
ret.update({plugin_advanced_arg: gr.update(visible=False, label=f"插件[{k}]不需要高级参数。")})
return ret
@@ -183,13 +194,26 @@ def main():
# 随变按钮的回调函数注册
def route(request: gr.Request, k, *args, **kwargs):
if k in [r"打开插件列表", r"请先从插件列表中选择"]: return
- yield from ArgsGeneralWrapper(crazy_fns[k]["Function"])(request, *args, **kwargs)
+ yield from ArgsGeneralWrapper(plugins[k]["Function"])(request, *args, **kwargs)
click_handle = switchy_bt.click(route,[switchy_bt, *input_combo, gr.State(PORT)], output_combo)
click_handle.then(on_report_generated, [cookies, file_upload, chatbot], [cookies, file_upload, chatbot])
cancel_handles.append(click_handle)
# 终止按钮的回调函数注册
stopBtn.click(fn=None, inputs=None, outputs=None, cancels=cancel_handles)
stopBtn2.click(fn=None, inputs=None, outputs=None, cancels=cancel_handles)
+ plugins_as_btn = {name:plugin for name, plugin in plugins.items() if plugin.get('Button', None)}
+ def on_group_change(group_list):
+ btn_list = []
+ fns_list = []
+ if not group_list: # 处理特殊情况：没有选择任何插件组
+ return [*[plugin['Button'].update(visible=False) for _, plugin in plugins_as_btn.items()], gr.Dropdown.update(choices=[])]
+ for k, plugin in plugins.items():
+ if plugin.get("AsButton", True):
+ btn_list.append(plugin['Button'].update(visible=match_group(plugin['Group'], group_list))) # 刷新按钮
+ if plugin.get('AdvancedArgs', False): dropdown_fn_list.append(k) # 对于需要高级参数的插件，亦在下拉菜单中显示
+ elif match_group(plugin['Group'], group_list): fns_list.append(k) # 刷新下拉列表
+ return [*btn_list, gr.Dropdown.update(choices=fns_list)]
+ plugin_group_sel.select(fn=on_group_change, inputs=[plugin_group_sel], outputs=[*[plugin['Button'] for name, plugin in plugins_as_btn.items()], dropdown])
if ENABLE_AUDIO:
from crazy_functions.live_audio.audio_io import RealtimeAudioDistribution
rad = RealtimeAudioDistribution()
@@ -221,8 +245,10 @@ def main():
auto_opentab_delay()
demo.queue(concurrency_count=CONCURRENT_COUNT).launch(
- server_name="0.0.0.0", server_port=PORT,
- favicon_path="docs/logo.png", auth=AUTHENTICATION,
+ server_name="0.0.0.0",
+ server_port=PORT,
+ favicon_path="docs/logo.png",
+ auth=AUTHENTICATION if len(AUTHENTICATION) != 0 else None,
blocked_paths=["config.py","config_private.py","docker-compose.yml","Dockerfile"])
# 如果需要在二级路径下运行
diff --git a/multi_language.py b/multi_language.py
index 7786b86..c4ed36e 100644
--- a/multi_language.py
+++ b/multi_language.py
@@ -478,6 +478,8 @@ def step_2_core_key_translate():
up = trans_json(need_translate, language=LANG, special=False)
map_to_json(up, language=LANG)
cached_translation = read_map_from_json(language=LANG)
+ LANG_STD = 'std'
+ cached_translation.update(read_map_from_json(language=LANG_STD))
cached_translation = dict(sorted(cached_translation.items(), key=lambda x: -len(x[0])))
# ===============================================
diff --git a/request_llm/bridge_chatglmft.py b/request_llm/bridge_chatglmft.py
index 4e21c98..71af942 100644
--- a/request_llm/bridge_chatglmft.py
+++ b/request_llm/bridge_chatglmft.py
@@ -63,9 +63,9 @@ class GetGLMFTHandle(Process):
# if not os.path.exists(conf): raise RuntimeError('找不到微调模型信息')
# with open(conf, 'r', encoding='utf8') as f:
# model_args = json.loads(f.read())
- ChatGLM_PTUNING_CHECKPOINT, = get_conf('ChatGLM_PTUNING_CHECKPOINT')
- assert os.path.exists(ChatGLM_PTUNING_CHECKPOINT), "找不到微调模型检查点"
- conf = os.path.join(ChatGLM_PTUNING_CHECKPOINT, "config.json")
+ CHATGLM_PTUNING_CHECKPOINT, = get_conf('CHATGLM_PTUNING_CHECKPOINT')
+ assert os.path.exists(CHATGLM_PTUNING_CHECKPOINT), "找不到微调模型检查点"
+ conf = os.path.join(CHATGLM_PTUNING_CHECKPOINT, "config.json")
with open(conf, 'r', encoding='utf8') as f:
model_args = json.loads(f.read())
if 'model_name_or_path' not in model_args:
@@ -78,9 +78,9 @@ class GetGLMFTHandle(Process):
config.pre_seq_len = model_args['pre_seq_len']
config.prefix_projection = model_args['prefix_projection']
- print(f"Loading prefix_encoder weight from {ChatGLM_PTUNING_CHECKPOINT}")
+ print(f"Loading prefix_encoder weight from {CHATGLM_PTUNING_CHECKPOINT}")
model = AutoModel.from_pretrained(model_args['model_name_or_path'], config=config, trust_remote_code=True)
- prefix_state_dict = torch.load(os.path.join(ChatGLM_PTUNING_CHECKPOINT, "pytorch_model.bin"))
+ prefix_state_dict = torch.load(os.path.join(CHATGLM_PTUNING_CHECKPOINT, "pytorch_model.bin"))
new_prefix_state_dict = {}
for k, v in prefix_state_dict.items():
if k.startswith("transformer.prefix_encoder."):
diff --git a/request_llm/bridge_qianfan.py b/request_llm/bridge_qianfan.py
index e2cdb0e..be73976 100644
--- a/request_llm/bridge_qianfan.py
+++ b/request_llm/bridge_qianfan.py
@@ -49,16 +49,17 @@ def get_access_token():
def generate_message_payload(inputs, llm_kwargs, history, system_prompt):
conversation_cnt = len(history) // 2
+ if system_prompt == "": system_prompt = "Hello"
messages = [{"role": "user", "content": system_prompt}]
messages.append({"role": "assistant", "content": 'Certainly!'})
if conversation_cnt:
for index in range(0, 2*conversation_cnt, 2):
what_i_have_asked = {}
what_i_have_asked["role"] = "user"
- what_i_have_asked["content"] = history[index]
+ what_i_have_asked["content"] = history[index] if history[index]!="" else "Hello"
what_gpt_answer = {}
what_gpt_answer["role"] = "assistant"
- what_gpt_answer["content"] = history[index+1]
+ what_gpt_answer["content"] = history[index+1] if history[index]!="" else "Hello"
if what_i_have_asked["content"] != "":
if what_gpt_answer["content"] == "": continue
if what_gpt_answer["content"] == timeout_bot_msg: continue
diff --git a/request_llm/bridge_spark.py b/request_llm/bridge_spark.py
index 1a3d43d..0fe925f 100644
--- a/request_llm/bridge_spark.py
+++ b/request_llm/bridge_spark.py
@@ -2,11 +2,17 @@
import time
import threading
import importlib
-from toolbox import update_ui, get_conf
+from toolbox import update_ui, get_conf, update_ui_lastest_msg
from multiprocessing import Process, Pipe
model_name = '星火认知大模型'
+def validate_key():
+ XFYUN_APPID, = get_conf('XFYUN_APPID', )
+ if XFYUN_APPID == '00000000' or XFYUN_APPID == '':
+ return False
+ return True
+
def predict_no_ui_long_connection(inputs, llm_kwargs, history=[], sys_prompt="", observe_window=[], console_slience=False):
"""
⭐多线程方法
@@ -15,6 +21,9 @@ def predict_no_ui_long_connection(inputs, llm_kwargs, history=[], sys_prompt="",
watch_dog_patience = 5
response = ""
+ if validate_key() is False:
+ raise RuntimeError('请配置讯飞星火大模型的XFYUN_APPID, XFYUN_API_KEY, XFYUN_API_SECRET')
+
from .com_sparkapi import SparkRequestInstance
sri = SparkRequestInstance()
for response in sri.generate(inputs, llm_kwargs, history, sys_prompt):
@@ -32,6 +41,10 @@ def predict(inputs, llm_kwargs, plugin_kwargs, chatbot, history=[], system_promp
chatbot.append((inputs, ""))
yield from update_ui(chatbot=chatbot, history=history)
+ if validate_key() is False:
+ yield from update_ui_lastest_msg(lastmsg="[Local Message]: 请配置讯飞星火大模型的XFYUN_APPID, XFYUN_API_KEY, XFYUN_API_SECRET", chatbot=chatbot, history=history, delay=0)
+ return
+
if additional_fn is not None:
from core_functional import handle_core_functionality
inputs, history = handle_core_functionality(additional_fn, inputs, history, chatbot)
diff --git a/request_llm/com_sparkapi.py b/request_llm/com_sparkapi.py
index 308aa64..0b8d655 100644
--- a/request_llm/com_sparkapi.py
+++ b/request_llm/com_sparkapi.py
@@ -58,7 +58,7 @@ class Ws_Param(object):
class SparkRequestInstance():
def __init__(self):
XFYUN_APPID, XFYUN_API_SECRET, XFYUN_API_KEY = get_conf('XFYUN_APPID', 'XFYUN_API_SECRET', 'XFYUN_API_KEY')
-
+ if XFYUN_APPID == '00000000' or XFYUN_APPID == '': raise RuntimeError('请配置讯飞星火大模型的XFYUN_APPID, XFYUN_API_KEY, XFYUN_API_SECRET')
self.appid = XFYUN_APPID
self.api_secret = XFYUN_API_SECRET
self.api_key = XFYUN_API_KEY
diff --git a/requirements.txt b/requirements.txt
index e6d27d2..5ff40cc 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -20,4 +20,4 @@ arxiv
rich
pypdf2==2.12.1
websocket-client
-scipdf_parser==0.3
+scipdf_parser>=0.3
diff --git a/tests/test_plugins.py b/tests/test_plugins.py
index 2207f71..ec28af1 100644
--- a/tests/test_plugins.py
+++ b/tests/test_plugins.py
@@ -9,9 +9,10 @@ validate_path() # 返回项目根路径
from tests.test_utils import plugin_test
if __name__ == "__main__":
- # plugin_test(plugin='crazy_functions.虚空终端->自动终端', main_input='修改api-key为sk-jhoejriotherjep')
+ # plugin_test(plugin='crazy_functions.虚空终端->虚空终端', main_input='修改api-key为sk-jhoejriotherjep')
+ plugin_test(plugin='crazy_functions.批量翻译PDF文档_NOUGAT->批量翻译PDF文档', main_input='crazy_functions/test_project/pdf_and_word/aaai.pdf')
- plugin_test(plugin='crazy_functions.虚空终端->自动终端', main_input='调用插件，对C:/Users/fuqingxu/Desktop/旧文件/gpt/chatgpt_academic/crazy_functions/latex_fns中的python文件进行解析')
+ # plugin_test(plugin='crazy_functions.虚空终端->虚空终端', main_input='调用插件，对C:/Users/fuqingxu/Desktop/旧文件/gpt/chatgpt_academic/crazy_functions/latex_fns中的python文件进行解析')
# plugin_test(plugin='crazy_functions.命令行助手->命令行助手', main_input='查看当前的docker容器列表')
diff --git a/themes/common.css b/themes/common.css
index fdb8d71..5880d00 100644
--- a/themes/common.css
+++ b/themes/common.css
@@ -1,15 +1,21 @@
-/* hide remove all button */
-.remove-all.svelte-aqlk7e.svelte-aqlk7e.svelte-aqlk7e {
- visibility: hidden;
-}
-
-/* hide selector border */
-#input-plugin-group .wrap.svelte-aqlk7e.svelte-aqlk7e.svelte-aqlk7e {
- border: 0px;
- box-shadow: none;
-}
-
-/* hide selector label */
-#input-plugin-group .svelte-1gfkn6j {
- visibility: hidden;
-}
+/* hide remove all button */
+.remove-all.svelte-aqlk7e.svelte-aqlk7e.svelte-aqlk7e {
+ visibility: hidden;
+}
+
+/* hide selector border */
+#input-plugin-group .wrap.svelte-aqlk7e.svelte-aqlk7e.svelte-aqlk7e {
+ border: 0px;
+ box-shadow: none;
+}
+
+/* hide selector label */
+#input-plugin-group .svelte-1gfkn6j {
+ visibility: hidden;
+}
+
+
+/* height of the upload box */
+.wrap.svelte-xwlu1w {
+ min-height: var(--size-32);
+}
diff --git a/toolbox.py b/toolbox.py
index 73e3e8d..ab8d046 100644
--- a/toolbox.py
+++ b/toolbox.py
@@ -24,6 +24,19 @@ pj = os.path.join
class ChatBotWithCookies(list):
def __init__(self, cookie):
+ """
+ cookies = {
+ 'top_p': top_p,
+ 'temperature': temperature,
+ 'lock_plugin': bool,
+ "files_to_promote": ["file1", "file2"],
+ "most_recent_uploaded": {
+ "path": "uploaded_path",
+ "time": time.time(),
+ "time_str": "timestr",
+ }
+ }
+ """
self._cookies = cookie
def write_list(self, list):
@@ -47,6 +60,8 @@ def ArgsGeneralWrapper(f):
# 引入一个有cookie的chatbot
cookies.update({
'top_p':top_p,
+ 'api_key': cookies['api_key'],
+ 'llm_model': llm_model,
'temperature':temperature,
})
llm_kwargs = {
@@ -69,7 +84,7 @@ def ArgsGeneralWrapper(f):
# 处理个别特殊插件的锁定状态
module, fn_name = cookies['lock_plugin'].split('->')
f_hot_reload = getattr(importlib.import_module(module, fn_name), fn_name)
- yield from f_hot_reload(txt_passon, llm_kwargs, plugin_kwargs, chatbot_with_cookie, history, system_prompt, *args)
+ yield from f_hot_reload(txt_passon, llm_kwargs, plugin_kwargs, chatbot_with_cookie, history, system_prompt, request)
return decorated
@@ -266,8 +281,7 @@ def report_execption(chatbot, history, a, b):
向chatbot中添加错误信息
"""
chatbot.append((a, b))
- history.append(a)
- history.append(b)
+ history.extend([a, b])
def text_divide_paragraph(text):
@@ -290,6 +304,7 @@ def text_divide_paragraph(text):
text = "".join(lines)
return pre + text + suf
+
@lru_cache(maxsize=128) # 使用 lru缓存 加快转换速度
def markdown_convertion(txt):
"""
@@ -344,19 +359,41 @@ def markdown_convertion(txt):
content = content.replace('\n', '')
return content
- def no_code(txt):
- if '```' not in txt:
- return True
- else:
- if '```reference' in txt: return True # newbing
- else: return False
+ def is_equation(txt):
+ """
+ 判定是否为公式 | 测试1 写出洛伦兹定律，使用tex格式公式 测试2 给出柯西不等式，使用latex格式 测试3 写出麦克斯韦方程组
+ """
+ if '```' in txt and '```reference' not in txt: return False
+ if '$' not in txt and '\\[' not in txt: return False
+ mathpatterns = {
+ r'(? 接入百度千帆平台和文心一言 <-> 接入阿里通义千问、讯飞星火、上海AI-Lab书生 <-> 优化一键升级 <-> 提高arxiv翻译速度和成功率 <-> 支持自定义APIKEY格式 <-> 临时修复theme的文件丢失问题 <-> 新增实时语音对话插件（自动断句，脱手对话） <-> 支持加载自定义的ChatGLM2微调模型 <-> 动态ChatBot窗口高度 <-> 修复Azure接口的BUG <-> 完善多语言模块"
+ "new_feature": "支持插件分类！ <-> 支持用户使用自然语言调度各个插件(虚空终端) ！ <-> 改进UI，设计新主题 <-> 支持借助GROBID实现PDF高精度翻译 <-> 接入百度千帆平台和文心一言 <-> 接入阿里通义千问、讯飞星火、上海AI-Lab书生 <-> 优化一键升级 <-> 提高arxiv翻译速度和成功率"
}
-
+