161 Commits
Author SHA1 Message Date
binary-husky a528c35d1f Revert "feat: 使用CSS完善表格、列表、代码块、对话气泡显示样式" 2023-04-01 19:16:42 +08:00
binary-husky 623b0c83a1 Merge pull request #236 from Keldos-Li/CSS
feat: 使用CSS完善表格、列表、代码块、对话气泡显示样式
2023-04-01 19:10:55 +08:00
Your Name c64dbb03fd fix bug 2023-04-01 19:07:58 +08:00
Your Name 8575c82ed7 README 2023-04-01 18:07:26 +08:00
Your Name bca61754e3 Typo in Prompt 2023-04-01 17:29:30 +08:00
Your Name f61ea1559c python3.7 compat 2023-04-01 17:11:59 +08:00
Keldos 12c36a68ce feat: 调整表格样式 2023-04-01 16:58:51 +08:00
Keldos 998e127b2f feat: 使用CSS完善表格、列表、代码块、对话气泡显示样式
移植了 川虎ChatGPT 的CSS——但是川虎ChatGPT的CSS也是我写的~
2023-04-01 16:51:38 +08:00
Your Name c9c9449a59 README up 2023-04-01 16:36:57 +08:00
Your Name 722b538261 update README 2023-04-01 16:35:45 +08:00
Your Name 29fb436e76 Merge branch 'dev' 2023-04-01 16:31:57 +08:00
binary-husky 6789eaee45 Update README.md 2023-04-01 04:25:03 +08:00
binary-husky 937823ec64 Update README.md 2023-04-01 04:19:02 +08:00
Your Name 3c95299f48 交互优化 2023-04-01 04:11:31 +08:00
Your Name 639e24fc82 将css样式移动到theme文件,减少main.py的代码行数 2023-04-01 03:39:43 +08:00
binary-husky 364810983a Merge pull request #209 from jr-shen/dev-1
(1)修改语法检查的prompt,确保输出格式统一。

之前使用时经常发现输出没有把修改的部分加粗,或者在表格中把整段文字输出了,影响阅读。因此在之前的prompt基础上增加了一个example,确保输出格式统一。

(2)表格内增加了边框线,使行/列之间的分隔更清楚。

使用时发现没有边框的表格在里面文字较多时难以区分。因此增加表格内边框线。
2023-04-01 03:37:02 +08:00
Your Name cb9404c4de 优化Token溢出时的处理 2023-04-01 03:36:05 +08:00
Your Name 17a18e99fa 隐藏、显示功能区 2023-04-01 00:21:27 +08:00
Your Name 30ea77a496 更清朗些的UI 2023-03-31 23:54:25 +08:00
Your Name 01931b0bd2 更清朗的UI 2023-03-31 23:51:17 +08:00
Junru Shen 6a2c7db7c1 add markdown table border line to make text boundary more clear 2023-03-31 23:40:21 +08:00
Junru Shen 9c15c446a6 make grammar correction prompt more clear 2023-03-31 23:38:49 +08:00
Your Name 44ed8a46ad 对word和pdf进行简易的支持 2023-03-31 23:18:45 +08:00
Your Name 41801c017c Merge branch 'master' into dev 2023-03-31 23:08:30 +08:00
Your Name 3e88422ab6 Merge branch 'master' of github.com:binary-husky/chatgpt_academic 2023-03-31 22:49:45 +08:00
Your Name 7c8b8b95b2 修复bug 2023-03-31 22:49:39 +08:00
Your Name 65b1d78516 优化自译解功能 2023-03-31 22:36:46 +08:00
binary-husky e815afb792 Update README.md 2023-03-31 21:48:45 +08:00
Your Name ac681d3201 移动函数到调用模组 2023-03-31 21:46:47 +08:00
binary-husky ecebdf3ab5 Merge pull request #204 from Eralien/dev-clean_pdf
feat: clean pdf fitz text
2023-03-31 21:42:18 +08:00
binary-husky 8aea6536e0 add contributor 2023-03-31 21:41:17 +08:00
binary-husky 9c90b28bed Merge pull request #147 from JasonGuo1/master
feat(toolbox.py,总结word文档.py): 支持rar格式与7z格式解压;word读取
2023-03-31 21:39:05 +08:00
Your Name 36ef2fa884 JasonGuo1 2023-03-31 21:37:46 +08:00
Your Name 89ca010a11 Merge branch 'master' of https://github.com/JasonGuo1/chatgpt_academic into JasonGuo1-master 2023-03-31 21:31:31 +08:00
Siyuan Feng 60fc2bf4c1 feat: clean pdf fitz text 2023-03-31 21:26:55 +08:00
binary-husky 838c3dc881 Merge pull request #117 from XMB-7/better_prompt
feat: better prompt
2023-03-31 21:19:25 +08:00
Your Name 16c59e1bf6 Merge branch 'better_prompt' of https://github.com/XMB-7/chatgpt_academic into XMB-7-better_prompt 2023-03-31 21:18:28 +08:00
binary-husky fb889cb4ce Merge pull request #174 from Euclid-Jie/Euclid_Test
feature(read pdf paper then write summary)
2023-03-31 21:06:02 +08:00
Your Name c91cfc64d9 整合 2023-03-31 21:05:18 +08:00
Your Name dcab956cff Merge branch 'dev' into Euclid-Jie-Euclid_Test 2023-03-31 21:03:43 +08:00
Your Name da55ae68f6 pdfminer整合到一个文件中 2023-03-31 21:03:12 +08:00
Your Name 201f53c0f0 Merge branch 'Euclid_Test' of https://github.com/Euclid-Jie/chatgpt_academic into Euclid-Jie-Euclid_Test 2023-03-31 20:26:59 +08:00
Your Name 77a98f03d0 修改文本 2023-03-31 20:12:27 +08:00
Your Name 9ee5545ad5 fix import error 2023-03-31 20:05:31 +08:00
Your Name d37b0ce447 Merge branch 'dev' of github.com:binary-husky/chatgpt_academic into dev 2023-03-31 20:04:11 +08:00
Your Name 0abb84ae4b config新增说明 2023-03-31 20:02:12 +08:00
binary-husky e6034b6928 Merge pull request #194 from fulyaec/enhance-chataca
修改AUTHENTICATION的判断,使得AUTHENTICATION为None/[]/""时都可以正确判断
2023-03-31 19:49:40 +08:00
Your Name ac9e72b9f8 revert toolbox 2023-03-31 19:46:01 +08:00
Your Name cb1ac5d13d Merge branch 'enhance-chataca' of https://github.com/fulyaec/chatgpt_academic into fulyaec-enhance-chataca 2023-03-31 19:45:23 +08:00
binary-husky 3497addc7b Merge pull request #198 from oneLuckyman/feature-match-API_KEY
一个小改进:更精准的 API_KEY 确认机制
2023-03-31 19:28:21 +08:00
Your Name f17c580fd9 Merge remote-tracking branch 'origin/hot-reload-test' 2023-03-31 19:21:15 +08:00
Jia Xinglong 808e23c98a 使用 re 模块的 match 函数可以更精准的匹配和确认 API_KEY 是否正确 2023-03-31 17:38:39 +08:00
fulyaec e3e4fa19a2 refactor and enhance 2023-03-31 16:24:40 +08:00
binary-husky 3cc0635628 Update main.py 2023-03-31 13:33:03 +08:00
binary-husky 7149f9fe90 Update README.md 2023-03-31 13:29:37 +08:00
binary-husky d2f009ba8d Update README.md 2023-03-31 13:11:10 +08:00
欧玮杰 8f4f13efd5 fix(the ".PDF" file can not be recognized): 2023-03-31 10:26:40 +08:00
欧玮杰 fefe96144f fix(fix "gbk" encode error in 批量总结PDF文档 line14):
由于不可编码字符,导致报错,添加软解码,处理原始文本。
2023-03-31 10:03:10 +08:00
欧玮杰 5521f0e41c feature(read pdf paper then write summary):
add a func called readPdf in toolbox, which can read pdf paper to str. then use bs4.BeautifulSoup to clean content.
2023-03-31 00:54:01 +08:00
binary-husky 2d5d719696 Merge pull request #171 from RoderickChan/add-deploy-instruction
在README中添加远程部署的指导
2023-03-31 00:00:35 +08:00
binary-husky 2b339464ee Update README.md 2023-03-30 23:59:01 +08:00
binary-husky 43ad798c68 Update README.md 2023-03-30 23:34:17 +08:00
RoderickChan a441c90436 在README中添加远程部署的指导方案 2023-03-30 23:31:44 +08:00
JasonGuo1 5ca623e9c5 feat(总结word文档):增加读取docx、doc格式的功能 2023-03-30 23:23:41 +08:00
binary-husky c74a8b04b5 添加Wiki链接 2023-03-30 23:09:45 +08:00
JasonGuo1 34c693a571 feat(toolbox):调整了空格的问题 2023-03-30 20:28:15 +08:00
binary-husky d4cd1877f4 Merge pull request #151 from SadPencil/patch-1
Fix a typo
2023-03-30 19:14:48 +08:00
qingxu fu 57a668bf30 自译解报告 2023-03-30 18:21:17 +08:00
qingxu fu 700d14be10 Merge branch 'hot-reload-test' of https://github.com/binary-husky/chatgpt_academic into hot-reload-test 2023-03-30 18:05:03 +08:00
qingxu fu 3fe96956ce 新增热更新功能 2023-03-30 18:04:20 +08:00
qingxu fu c358c95630 新增热更新功能 2023-03-30 18:01:06 +08:00
Sad Pencil 70c02a08e9 Fix a typo 2023-03-30 15:55:46 +08:00
JasonGuo1 27f7153f37 feat(toolbox): 支持rar格式与7z格式解压,修改了下注释 2023-03-30 15:48:55 +08:00
JasonGuo1 eb69704c20 feat(toolbox): 支持rar格式与7z格式解压,修改了下注释 2023-03-30 15:48:00 +08:00
JasonGuo1 d616915348 feat(toolbox): 支持rar格式与7z格式解压,修改了下注释 2023-03-30 15:47:18 +08:00
JasonGuo1 5381df8f35 feat(toolbox): 支持rar格式与7z格式解压,修改了下注释 2023-03-30 15:45:58 +08:00
JasonGuo1 a7c857d4d9 feat(支持rar格式与7z格式解压) 2023-03-30 15:24:01 +08:00
binary-husky 1c3ce18eff Update README.md 2023-03-30 14:48:46 +08:00
binary-husky a689960e31 Update README.md 2023-03-30 14:48:20 +08:00
binary-husky 07eca9fe55 Update README.md 2023-03-30 14:47:19 +08:00
binary-husky 00f29f2120 Update README.md 2023-03-30 14:08:24 +08:00
binary-husky 10ea3daaa5 Update README.md 2023-03-30 13:24:30 +08:00
binary-husky eb95eec122 Merge pull request #124 from Freddd13/master
feat: 添加wsl2使用windows proxy的方法
2023-03-30 12:56:13 +08:00
qingxu fu e03634f9e2 查找语法错误之前先清除换行符 2023-03-30 12:52:28 +08:00
qingxu fu 77d4628877 语法错误查找prompt更新 2023-03-30 12:16:18 +08:00
qingxu fu 8c3d37bbce up 2023-03-30 11:51:55 +08:00
qingxu fu 28f6801870 标准化代码格式 2023-03-30 11:50:11 +08:00
qingxu fu 6d7fb20b5a 修改配置的读取方式 2023-03-30 11:05:38 +08:00
Freddd13 9dc0d3273b update: 修改readme 2023-03-30 02:00:15 +08:00
Freddd13 13e976774f Merge remote-tracking branch 'upstream/master' 2023-03-30 01:58:39 +08:00
Freddd13 35d3346a1c feat: 支持wsl2使用windows proxy 2023-03-30 01:47:39 +08:00
Your Name eaec1c3728 Merge branch 'master' of github.com:binary-husky/chatgpt_academic 2023-03-30 00:15:37 +08:00
Your Name 0140b0cda8 change UI layout 2023-03-30 00:15:31 +08:00
binary-husky f98fe5b9ab Update README.md 2023-03-30 00:09:58 +08:00
Xiaoming Bai 932c430d5d better prompt 2023-03-30 00:06:02 +08:00
binary-husky 0063f8dd82 Update README.md 2023-03-29 23:53:33 +08:00
binary-husky 23b8c47c54 Update README.md 2023-03-29 23:50:20 +08:00
binary-husky 5353a32345 Update README.md 2023-03-29 23:48:58 +08:00
binary-husky d569d2f9c1 Update README.md 2023-03-29 23:44:37 +08:00
binary-husky 67e6070d80 Update README.md 2023-03-29 23:44:01 +08:00
binary-husky 01873f7811 Merge pull request #108 from sjiang95/condainstall
readme: update
2023-03-29 23:22:44 +08:00
Your Name c2318bc197 Merge branch 'dev' 2023-03-29 23:15:29 +08:00
binary-husky 86dc4ca3af Merge pull request #102 from ValeriaWong/master
feat(读文章写摘要):支持pdf文件批量阅读及总结 #101
2023-03-29 23:14:12 +08:00
Your Name 4a2f73d007 change UI layout 2023-03-29 23:04:37 +08:00
Your Name 9c643e6d8c change ui layout 2023-03-29 23:00:16 +08:00
Your Name 34323c9d36 Merge https://github.com/ValeriaWong/chatgpt_academic into ValeriaWong-master 2023-03-29 21:49:56 +08:00
Your Name 48cc2349e3 add pip package check 2023-03-29 21:47:56 +08:00
Your Name 7d57967519 Merge branch 'master' of https://github.com/ValeriaWong/chatgpt_academic 2023-03-29 21:44:59 +08:00
Shengjiang Quan 68ffa54c84 readme: update
Re-format a part of the markdown content
and add conda instruction for installation.

Signed-off-by: Shengjiang Quan <[email protected]>
2023-03-29 22:36:15 +09:00
ValeriaWong 02ddcdf731 Merge branch 'master' of https://github.com/ValeriaWong/chatgpt_academic 2023-03-29 21:05:25 +08:00
ValeriaWong 14ea206a5a Merge branch 'binary-husky:master' into master 2023-03-29 20:57:07 +08:00
ValeriaWong 814c93bb75 feat(读文章写摘要):支持pdf文件批量阅读及总结 #101 2023-03-29 20:55:13 +08:00
binary-husky fb953148b9 Update main.py 2023-03-29 20:47:34 +08:00
Your Name 2bcd34360a bug quick fix 2023-03-29 20:41:07 +08:00
binary-husky b3211362f9 Merge pull request #82 from Okabe-Rintarou-0/master
支持暂停按钮 #53
2023-03-29 20:38:06 +08:00
Your Name be1bf6cb77 提交后不清空输入栏,添加停止键 2023-03-29 20:36:58 +08:00
Your Name 5f771d8acf Merge branch 'master' of https://github.com/Okabe-Rintarou-0/chatgpt_academic into Okabe-Rintarou-0-master 2023-03-29 20:26:13 +08:00
Your Name 095579c323 Merge branch 'master' of https://github.com/Okabe-Rintarou-0/chatgpt_academic into Okabe-Rintarou-0-master 2023-03-29 20:07:38 +08:00
Your Name f9308300f3 handle ip location lookup error 2023-03-29 19:37:39 +08:00
binary-husky 6ca28fbff2 Merge pull request #87 from Okabe-Rintarou-0/fix-markdown-display
正确显示多行输入的 markdown #84
2023-03-29 19:32:19 +08:00
binary-husky 11f619f76f Merge pull request #96 from eltociear/patch-1
fix typo in predict.py
2023-03-29 18:47:51 +08:00
Your Name 39d0176673 新增代理配置说明 2023-03-29 18:07:33 +08:00
Ikko Eltociear Ashimine 043bd59ffc fix typo in predict.py
refleshing -> refreshing
2023-03-29 18:57:37 +09:00
ValeriaWong 2ac602e0aa feat(读文章写摘要):支持pdf文件批量阅读及总结 2023-03-29 17:57:17 +08:00
Your Name 50ee37f23c error message change 2023-03-29 16:50:37 +08:00
Your Name f0d9098df5 dev 2023-03-29 16:47:15 +08:00
okabe 2401cf2136 fix: markdown display bug #84 2023-03-29 15:29:40 +08:00
okabe dd3c4f988c feat: support stop generate button (#53) 2023-03-29 14:53:53 +08:00
Your Name 22e7dc617b fix directory return bug 2023-03-29 14:28:57 +08:00
Your Name a3c937c202 update comments 2023-03-29 14:16:59 +08:00
505030475 1b01d0fd8a 修复变量名 2023-03-29 13:58:30 +08:00
505030475 0b018bf871 Merge remote-tracking branch 'origin/test-3-29' 2023-03-29 13:44:57 +08:00
505030475 88d41ab4ca config comments 2023-03-29 13:43:07 +08:00
binary-husky 6ebadeb2a6 Update README.md 2023-03-29 13:38:43 +08:00
binary-husky 515045a8d1 Merge pull request #57 from GaiZhenbiao/master
Adding a bunch of nice-to-have features
2023-03-29 13:37:58 +08:00
Your Name 74d5061969 Merge branch 'test-3-29' of github.com:binary-husky/chatgpt_academic into test-3-29 2023-03-29 12:29:52 +08:00
Your Name 0b7c1c50ca 优化Unsplash API的使用 2023-03-29 12:28:45 +08:00
Your Name f45e0d3486 优化Unsplash API的使用 2023-03-29 12:27:47 +08:00
Your Name 881132557e 历史上的今天,带图片 2023-03-29 12:21:47 +08:00
Your Name 6d852d76b0 更新一个更有意思的模板函数 2023-03-29 11:36:55 +08:00
Your Name b588291cdf [实验] 历史上的今天(高级函数demo) 2023-03-29 11:34:03 +08:00
Your Name b4e0fe39ea bug fix 2023-03-29 01:42:11 +08:00
Your Name 7295978c93 change description 2023-03-29 01:39:15 +08:00
Your Name f2359e0442 更好的多线程交互性 2023-03-29 01:32:28 +08:00
Your Name ced6898daa introduce project self-translation 2023-03-29 01:11:53 +08:00
Tuchuanhuhuhu 39ad492a65 增加“重置”按钮,提交之后自动清空输入框 2023-03-28 23:33:19 +08:00
Tuchuanhuhuhu dd8ec0d511 temprature的取值范围为[0, 2] 2023-03-28 23:20:54 +08:00
Tuchuanhuhuhu 43d59d936f 增加并行处理与权限控制 2023-03-28 23:17:12 +08:00
Your Name 4bca8b4f82 simplify codes 2023-03-28 23:09:25 +08:00
Chuan Hu 46eba1f399 Improve the way to open webbrowser 2023-03-28 22:47:30 +08:00
Your Name 5a6877f9fa 界面色彩自定义 2023-03-28 22:35:55 +08:00
Your Name 2c15b51ea4 explain color and theme 2023-03-28 22:31:43 +08:00
Your Name 9666b9b99b Merge branch 'master' of github.com:binary-husky/chatgpt_academic 2023-03-28 22:25:27 +08:00
Your Name baf61477d2 remove .vscode from git 2023-03-28 22:24:59 +08:00
Your Name 00e411bd68 update todo 2023-03-28 22:10:22 +08:00
Your Name 37bcdf684d fix unicode bug 2023-03-28 20:31:44 +08:00
binary-husky c80808a0c5 Merge pull request #46 from mambaHu/master
Markdown analysis report garbled issue
2023-03-28 20:04:01 +08:00
luca hu b69313884d improving garbled words issue with utf8 2023-03-28 19:34:18 +08:00
binary-husky 3b6675755e Delete jpeg-compressor.tps 2023-03-28 17:21:14 +08:00
binary-husky 3e2e4937db Delete JpegLibrary.tps 2023-03-28 17:21:06 +08:00
binary-husky e6f7e75e19 Delete UElibJPG.Build.cs 2023-03-28 17:20:54 +08:00
25 changed files with 6058 additions and 324 deletions
+3
View File
@@ -131,9 +131,12 @@ dmypy.json
# Pyre type checker
.pyre/
.vscode
history
ssr_conf
config_private.py
gpt_log
private.md
private_upload
other_llms
-16
View File
@@ -1,16 +0,0 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Python: Current File",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": false
}
]
}
+80 -32
View File
@@ -1,20 +1,31 @@
# ChatGPT 学术优化
**如果喜欢这个项目,请给它一个Star;如果你发明了更好用的学术快捷键,欢迎发issue或者pull requests**
**如果喜欢这个项目,请给它一个Star;如果你发明了更好用的学术快捷键,欢迎发issue或者pull requestsdev分支)**
If you like this project, please give it a Star. If you've come up with more useful academic shortcuts, feel free to open an issue or pull request.
If you like this project, please give it a Star. If you've come up with more useful academic shortcuts, feel free to open an issue or pull request to `dev` branch.
```
代码中参考了很多其他优秀项目中的设计,主要包括:
# 借鉴项目1:借鉴了mdtex2html中公式处理的方法
https://github.com/polarwinkel/mdtex2html
# 借鉴项目2:借鉴了ChuanhuChatGPT中读取OpenAI json的方法、记录历史问询记录的方法以及gradio queue的使用技巧
# 借鉴项目1:借鉴了ChuanhuChatGPT中读取OpenAI json的方法、记录历史问询记录的方法以及gradio queue的使用技巧
https://github.com/GaiZhenbiao/ChuanhuChatGPT
# 借鉴项目2:借鉴了mdtex2html中公式处理的方法
https://github.com/polarwinkel/mdtex2html
项目使用OpenAI的gpt-3.5-turbo模型,期待gpt-4早点放宽门槛😂
```
> **Note**
>
> 1.请注意只有“红颜色”标识的函数插件(按钮)才支持读取文件。目前暂不能完善地支持pdf/word格式文献的翻译解读,相关函数函件正在测试中。
>
> 2.本项目中每个文件的功能都在自译解[`project_self_analysis.md`](https://github.com/binary-husky/chatgpt_academic/wiki/chatgpt-academic%E9%A1%B9%E7%9B%AE%E8%87%AA%E8%AF%91%E8%A7%A3%E6%8A%A5%E5%91%8A)详细说明。随着版本的迭代,您也可以随时自行点击相关函数插件,调用GPT重新生成项目的自我解析报告。常见问题汇总在[`wiki`](https://github.com/binary-husky/chatgpt_academic/wiki/%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98)当中。
>
> 3.如果您不太习惯部分中文命名的函数,您可以随时点击相关函数插件,调用GPT一键生成纯英文的项目源代码。
<div align="center">
功能 | 描述
@@ -33,16 +44,17 @@ chat分析报告生成 | [实验性功能] 运行后自动生成总结汇报
公式显示 | 可以同时显示公式的tex形式和渲染形式
图片显示 | 可以在markdown中显示图片
支持GPT输出的markdown表格 | 可以输出支持GPT的markdown表格
…… | ……
</div>
<!-- - 新界面(左:master主分支, 右:dev开发前沿) -->
- 新界面
<div align="center">
<img src="https://user-images.githubusercontent.com/96192199/227851398-fab5a158-aaf6-4151-95ac-a8172ce611c7.png" width="700" >
<img src="https://user-images.githubusercontent.com/96192199/229222589-b30ff298-adb1-4e1e-8352-466085919bfb.png" width="700" >
</div>
- 所有按钮都通过读取functional.py动态生成,可随意加自定义功能,解放粘贴板
<div align="center">
<img src="img/公式.gif" width="700" >
@@ -70,33 +82,57 @@ chat分析报告生成 | [实验性功能] 运行后自动生成总结汇报
<img src="https://user-images.githubusercontent.com/96192199/226935232-6b6a73ce-8900-4aee-93f9-733c7e6fef53.png" width="700" >
</div>
## 直接运行 (Windows or Linux or MacOS)
## 直接运行 (Windows, Linux or MacOS)
### 1. 下载项目
```sh
# 下载项目
git clone https://github.com/binary-husky/chatgpt_academic.git
cd chatgpt_academic
# 在config.py中,配置 海外Proxy 和 OpenAI API KEY
- 1.如果你在国内,需要设置海外代理才能够使用 OpenAI API,你可以通过 config.py 文件来进行设置。
- 2.配置 OpenAI API KEY。你需要在 OpenAI 官网上注册并获取 API KEY。一旦你拿到了 API KEY,在 config.py 文件里配置好即可。
# 安装依赖
python -m pip install -r requirements.txt
# 运行
python main.py
# 测试实验性功能
## 测试C++项目头文件分析
input区域 输入 ./crazy_functions/test_project/cpp/libJPG 然后点击 "[实验] 解析整个C++项目(input输入项目根路径)"
## 测试给Latex项目写摘要
input区域 输入 ./crazy_functions/test_project/latex/attention 然后点击 "[实验] 读tex论文写摘要(input输入项目根路径)"
## 测试Python项目分析
input区域 输入 ./crazy_functions/test_project/python/dqn 然后点击 "[实验] 解析整个py项目(input输入项目根路径)"
## 测试自我代码解读
点击 "[实验] 请解析并解构此项目本身"
## 测试实验功能模板函数(要求gpt回答几个数的平方是什么),您可以根据此函数为模板,实现更复杂的功能
点击 "[实验] 实验功能函数模板"
```
### 2. 配置API_KEY和代理设置
`config.py`中,配置 海外Proxy 和 OpenAI API KEY,说明如下
```
1. 如果你在国内,需要设置海外代理才能够顺利使用 OpenAI API,设置方法请仔细阅读config.py1.修改其中的USE_PROXY为True; 2.按照说明修改其中的proxies)。
2. 配置 OpenAI API KEY。你需要在 OpenAI 官网上注册并获取 API KEY。一旦你拿到了 API KEY,在 config.py 文件里配置好即可。
3. 与代理网络有关的issue(网络超时、代理不起作用)汇总到 https://github.com/binary-husky/chatgpt_academic/issues/1
```
(P.S. 程序运行时会优先检查是否存在名为`config_private.py`的私密配置文件,并用其中的配置覆盖`config.py`的同名配置。因此,如果您能理解我们的配置读取逻辑,我们强烈建议您在`config.py`旁边创建一个名为`config_private.py`的新配置文件,并把`config.py`中的配置转移(复制)到`config_private.py`中。`config_private.py`不受git管控,可以让您的隐私信息更加安全。)
### 3. 安装依赖
```sh
# (选择一)推荐
python -m pip install -r requirements.txt
# (选择二)如果您使用anaconda,步骤也是类似的:
# (选择二.1conda create -n gptac_venv python=3.11
# (选择二.2conda activate gptac_venv
# (选择二.3python -m pip install -r requirements.txt
# 备注:使用官方pip源或者阿里pip源,其他pip源(如清华pip)有可能出问题,临时换源方法:
# python -m pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/
```
### 4. 运行
```sh
python main.py
```
### 5. 测试实验性功能
```
- 测试C++项目头文件分析
input区域 输入 `./crazy_functions/test_project/cpp/libJPG` 然后点击 "[实验] 解析整个C++项目(input输入项目根路径)"
- 测试给Latex项目写摘要
input区域 输入 `./crazy_functions/test_project/latex/attention` 然后点击 "[实验] 读tex论文写摘要(input输入项目根路径)"
- 测试Python项目分析
input区域 输入 `./crazy_functions/test_project/python/dqn` 然后点击 "[实验] 解析整个py项目(input输入项目根路径)"
- 测试自我代码解读
点击 "[实验] 请解析并解构此项目本身"
- 测试实验功能模板函数(要求gpt回答历史上的今天发生了什么),您可以根据此函数为模板,实现更复杂的功能
点击 "[实验] 实验功能函数模板"
```
## 使用docker (Linux)
@@ -105,7 +141,7 @@ input区域 输入 ./crazy_functions/test_project/python/dqn 然后点击 "[
git clone https://github.com/binary-husky/chatgpt_academic.git
cd chatgpt_academic
# 配置 海外Proxy 和 OpenAI API KEY
config.py
用任意文本编辑器编辑 config.py
# 安装
docker build -t gpt-academic .
# 运行
@@ -114,7 +150,7 @@ docker run --rm -it --net=host gpt-academic
# 测试实验性功能
## 测试自我代码解读
点击 "[实验] 请解析并解构此项目本身"
## 测试实验功能模板函数(要求gpt回答几个数的平方是什么),您可以根据此函数为模板,实现更复杂的功能
## 测试实验功能模板函数(要求gpt回答历史上的今天发生了什么),您可以根据此函数为模板,实现更复杂的功能
点击 "[实验] 实验功能函数模板"
##(请注意在docker中运行时,需要额外注意程序的文件访问权限问题)
## 测试C++项目头文件分析
@@ -126,6 +162,13 @@ input区域 输入 ./crazy_functions/test_project/python/dqn 然后点击 "[
```
## 其他部署方式
- 使用WSL2Windows Subsystem for Linux 子系统)
请访问[部署wiki-1](https://github.com/binary-husky/chatgpt_academic/wiki/%E4%BD%BF%E7%94%A8WSL2%EF%BC%88Windows-Subsystem-for-Linux-%E5%AD%90%E7%B3%BB%E7%BB%9F%EF%BC%89%E9%83%A8%E7%BD%B2)
- nginx远程部署
请访问[部署wiki-2](https://github.com/binary-husky/chatgpt_academic/wiki/%E8%BF%9C%E7%A8%8B%E9%83%A8%E7%BD%B2%E7%9A%84%E6%8C%87%E5%AF%BC)
## 自定义新的便捷按钮(学术快捷键自定义)
打开functional.py,添加条目如下,然后重启程序即可。(如果按钮已经添加成功并可见,那么前缀、后缀都支持热修改,无需重启程序即可生效。)
@@ -165,11 +208,12 @@ python check_proxy.py
## 兼容性测试
### 图片显示:
<div align="center">
<img src="https://user-images.githubusercontent.com/96192199/226906087-b5f1c127-2060-4db9-af05-487643b21ed9.png" height="200" >
<img src="https://user-images.githubusercontent.com/96192199/226906703-7226495d-6a1f-4a53-9728-ce6778cbdd19.png" height="200" >
<img src="https://user-images.githubusercontent.com/96192199/228737599-bf0a9d9c-1808-4f43-ae15-dfcc7af0f295.png" width="800" >
</div>
### 如果一个程序能够读懂并剖析自己:
<div align="center">
@@ -207,5 +251,9 @@ python check_proxy.py
<img src="https://user-images.githubusercontent.com/96192199/227504931-19955f78-45cd-4d1c-adac-e71e50957915.png" height="400" >
</div>
## Todo:
- (Top Priority) 调用另一个开源项目text-generation-webui的web接口,使用其他llm模型
- 总结大工程源代码时,文本过长、token溢出的问题(目前的方法是直接二分丢弃处理溢出,过于粗暴,有效信息大量丢失)
- UI不够美观
+7 -2
View File
@@ -6,8 +6,11 @@ def check_proxy(proxies):
response = requests.get("https://ipapi.co/json/", proxies=proxies, timeout=4)
data = response.json()
print(f'查询代理的地理位置,返回的结果是{data}')
if 'country_name' in data:
country = data['country_name']
result = f"代理配置 {proxies_https}, 代理所在地:{country}"
elif 'error' in data:
result = f"代理配置 {proxies_https}, 代理所在地:未知,IP查询频率受限"
print(result)
return result
except:
@@ -17,6 +20,8 @@ def check_proxy(proxies):
if __name__ == '__main__':
try: from config_private import proxies # 放自己的秘密如API和代理网址 os.path.exists('config_private.py')
except: from config import proxies
import os; os.environ['no_proxy'] = '*' # 避免代理网络产生意外污染
from toolbox import get_conf
proxies, = get_conf('proxies')
check_proxy(proxies)
+29 -12
View File
@@ -1,19 +1,31 @@
# API_KEY = "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" 此key无效
API_KEY = "sk-此处填API"
API_URL = "https://api.openai.com/v1/chat/completions"
# [step 1]>> 例如: API_KEY = "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" 此key无效
API_KEY = "sk-此处填API"
# 改为True应用代理
# [step 2]>> 改为True应用代理,如果直接在海外服务器部署,此处不修改
USE_PROXY = False
if USE_PROXY:
# 填写格式是 [协议]:// [地址] :[端口],填写之前不要忘记把USE_PROXY改成True,如果直接在海外服务器部署,此处不修改
# 例如 "socks5h://localhost:11284"
# [协议] 常见协议无非socks5h/http; 例如 v2**y 和 ss* 的默认本地协议是socks5h; 而cl**h 的默认本地协议是http
# [地址] 懂的都懂,不懂就填localhost或者127.0.0.1肯定错不了(localhost意思是代理软件安装在本机上)
# [端口] 在代理软件的设置里找。虽然不同的代理软件界面不一样,但端口号都应该在最显眼的位置上
# 代理网络的地址,打开你的科学上网软件查看代理的协议(socks5/http)、地址(localhost)和端口(11284)
proxies = { "http": "socks5h://localhost:11284", "https": "socks5h://localhost:11284", }
print('网络代理状态:运行。')
proxies = {
# [协议]:// [地址] :[端口]
"http": "socks5h://localhost:11284",
"https": "socks5h://localhost:11284",
}
else:
proxies = None
print('网络代理状态:未配置。无代理状态下很可能无法访问。')
# [step 3]>> 以下配置可以优化体验,但大部分场合下并不需要修改
# 对话窗的高度
CHATBOT_HEIGHT = 1116
# 发送请求到OpenAI后,等待多久判定为超时
TIMEOUT_SECONDS = 20
TIMEOUT_SECONDS = 25
# 网页的端口, -1代表随机端口
WEB_PORT = -1
@@ -21,9 +33,14 @@ WEB_PORT = -1
# 如果OpenAI不响应(网络卡顿、代理失败、KEY失效),重试的次数限制
MAX_RETRY = 2
# 选择的OpenAI模型是(gpt4现在只对申请成功的人开放)
# OpenAI模型选择是(gpt4现在只对申请成功的人开放)
LLM_MODEL = "gpt-3.5-turbo"
# 检查一下是不是忘了改config
if API_KEY == "sk-此处填API秘钥":
assert False, "请在config文件中修改API密钥, 添加海外代理之后再运行"
# OpenAI的API_URL
API_URL = "https://api.openai.com/v1/chat/completions"
# 设置并行使用的线程数
CONCURRENT_COUNT = 100
# 设置用户名和密码
AUTHENTICATION = [] # [("username", "password"), ("username2", "password2"), ...]
@@ -1,15 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<TpsData xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Name>Jpeg Library</Name>
<Location>/Engine/Source/ThirdParty/libJPG/</Location>
<Date>2016-06-10T14:04:17.9005402-04:00</Date>
<Function>We need it because it is a 3rd party lib in GFx</Function>
<Justification />
<Eula> See license in download: http://www.ijg.org/</Eula>
<RedistributeTo>
<EndUserGroup>Licensees</EndUserGroup>
<EndUserGroup>Git</EndUserGroup>
<EndUserGroup>P4</EndUserGroup>
</RedistributeTo>
<LicenseFolder>/Engine/Source/ThirdParty/Licenses/JPEG_License.txt</LicenseFolder>
</TpsData>
@@ -1,17 +0,0 @@
// Copyright Epic Games, Inc. All Rights Reserved.
using UnrealBuildTool;
public class UElibJPG : ModuleRules
{
public UElibJPG(ReadOnlyTargetRules Target) : base(Target)
{
Type = ModuleType.External;
string libJPGPath = Target.UEThirdPartySourceDirectory + "libJPG";
PublicIncludePaths.Add(libJPGPath);
ShadowVariableWarningLevel = WarningLevel.Off;
}
}
@@ -1,15 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<TpsData xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<Name>jpeg-compressor</Name>
<Location>/Engine/Source/ThirdParty/libJPG/</Location>
<Date>2016-06-10T14:07:13.8351319-04:00</Date>
<Function>Allows JPEG compression and decompression.</Function>
<Justification>Compressing video frames at runtime for reduced memory usage. Decompression to access the data afterwards.</Justification>
<Eula>https://code.google.com/archive/p/jpeg-compressor/</Eula>
<RedistributeTo>
<EndUserGroup>Licensees</EndUserGroup>
<EndUserGroup>Git</EndUserGroup>
<EndUserGroup>P4</EndUserGroup>
</RedistributeTo>
<LicenseFolder>None</LicenseFolder>
</TpsData>
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,433 @@
#pragma once
#include <atomic>
#include <utility>
#include <cstring>
#include <type_traits>
#include <cstdint>
#include "libipc/def.h"
#include "libipc/platform/detail.h"
#include "libipc/circ/elem_def.h"
#include "libipc/utility/log.h"
#include "libipc/utility/utility.h"
namespace ipc {
////////////////////////////////////////////////////////////////
/// producer-consumer implementation
////////////////////////////////////////////////////////////////
template <typename Flag>
struct prod_cons_impl;
template <>
struct prod_cons_impl<wr<relat::single, relat::single, trans::unicast>> {
template <std::size_t DataSize, std::size_t AlignSize>
struct elem_t {
std::aligned_storage_t<DataSize, AlignSize> data_ {};
};
alignas(cache_line_size) std::atomic<circ::u2_t> rd_; // read index
alignas(cache_line_size) std::atomic<circ::u2_t> wt_; // write index
constexpr circ::u2_t cursor() const noexcept {
return 0;
}
template <typename W, typename F, typename E>
bool push(W* /*wrapper*/, F&& f, E* elems) {
auto cur_wt = circ::index_of(wt_.load(std::memory_order_relaxed));
if (cur_wt == circ::index_of(rd_.load(std::memory_order_acquire) - 1)) {
return false; // full
}
std::forward<F>(f)(&(elems[cur_wt].data_));
wt_.fetch_add(1, std::memory_order_release);
return true;
}
/**
* In single-single-unicast, 'force_push' means 'no reader' or 'the only one reader is dead'.
* So we could just disconnect all connections of receiver, and return false.
*/
template <typename W, typename F, typename E>
bool force_push(W* wrapper, F&&, E*) {
wrapper->elems()->disconnect_receiver(~static_cast<circ::cc_t>(0u));
return false;
}
template <typename W, typename F, typename R, typename E>
bool pop(W* /*wrapper*/, circ::u2_t& /*cur*/, F&& f, R&& out, E* elems) {
auto cur_rd = circ::index_of(rd_.load(std::memory_order_relaxed));
if (cur_rd == circ::index_of(wt_.load(std::memory_order_acquire))) {
return false; // empty
}
std::forward<F>(f)(&(elems[cur_rd].data_));
std::forward<R>(out)(true);
rd_.fetch_add(1, std::memory_order_release);
return true;
}
};
template <>
struct prod_cons_impl<wr<relat::single, relat::multi , trans::unicast>>
: prod_cons_impl<wr<relat::single, relat::single, trans::unicast>> {
template <typename W, typename F, typename E>
bool force_push(W* wrapper, F&&, E*) {
wrapper->elems()->disconnect_receiver(1);
return false;
}
template <typename W, typename F, typename R,
template <std::size_t, std::size_t> class E, std::size_t DS, std::size_t AS>
bool pop(W* /*wrapper*/, circ::u2_t& /*cur*/, F&& f, R&& out, E<DS, AS>* elems) {
byte_t buff[DS];
for (unsigned k = 0;;) {
auto cur_rd = rd_.load(std::memory_order_relaxed);
if (circ::index_of(cur_rd) ==
circ::index_of(wt_.load(std::memory_order_acquire))) {
return false; // empty
}
std::memcpy(buff, &(elems[circ::index_of(cur_rd)].data_), sizeof(buff));
if (rd_.compare_exchange_weak(cur_rd, cur_rd + 1, std::memory_order_release)) {
std::forward<F>(f)(buff);
std::forward<R>(out)(true);
return true;
}
ipc::yield(k);
}
}
};
template <>
struct prod_cons_impl<wr<relat::multi , relat::multi, trans::unicast>>
: prod_cons_impl<wr<relat::single, relat::multi, trans::unicast>> {
using flag_t = std::uint64_t;
template <std::size_t DataSize, std::size_t AlignSize>
struct elem_t {
std::aligned_storage_t<DataSize, AlignSize> data_ {};
std::atomic<flag_t> f_ct_ { 0 }; // commit flag
};
alignas(cache_line_size) std::atomic<circ::u2_t> ct_; // commit index
template <typename W, typename F, typename E>
bool push(W* /*wrapper*/, F&& f, E* elems) {
circ::u2_t cur_ct, nxt_ct;
for (unsigned k = 0;;) {
cur_ct = ct_.load(std::memory_order_relaxed);
if (circ::index_of(nxt_ct = cur_ct + 1) ==
circ::index_of(rd_.load(std::memory_order_acquire))) {
return false; // full
}
if (ct_.compare_exchange_weak(cur_ct, nxt_ct, std::memory_order_acq_rel)) {
break;
}
ipc::yield(k);
}
auto* el = elems + circ::index_of(cur_ct);
std::forward<F>(f)(&(el->data_));
// set flag & try update wt
el->f_ct_.store(~static_cast<flag_t>(cur_ct), std::memory_order_release);
while (1) {
auto cac_ct = el->f_ct_.load(std::memory_order_acquire);
if (cur_ct != wt_.load(std::memory_order_relaxed)) {
return true;
}
if ((~cac_ct) != cur_ct) {
return true;
}
if (!el->f_ct_.compare_exchange_strong(cac_ct, 0, std::memory_order_relaxed)) {
return true;
}
wt_.store(nxt_ct, std::memory_order_release);
cur_ct = nxt_ct;
nxt_ct = cur_ct + 1;
el = elems + circ::index_of(cur_ct);
}
return true;
}
template <typename W, typename F, typename E>
bool force_push(W* wrapper, F&&, E*) {
wrapper->elems()->disconnect_receiver(1);
return false;
}
template <typename W, typename F, typename R,
template <std::size_t, std::size_t> class E, std::size_t DS, std::size_t AS>
bool pop(W* /*wrapper*/, circ::u2_t& /*cur*/, F&& f, R&& out, E<DS, AS>* elems) {
byte_t buff[DS];
for (unsigned k = 0;;) {
auto cur_rd = rd_.load(std::memory_order_relaxed);
auto cur_wt = wt_.load(std::memory_order_acquire);
auto id_rd = circ::index_of(cur_rd);
auto id_wt = circ::index_of(cur_wt);
if (id_rd == id_wt) {
auto* el = elems + id_wt;
auto cac_ct = el->f_ct_.load(std::memory_order_acquire);
if ((~cac_ct) != cur_wt) {
return false; // empty
}
if (el->f_ct_.compare_exchange_weak(cac_ct, 0, std::memory_order_relaxed)) {
wt_.store(cur_wt + 1, std::memory_order_release);
}
k = 0;
}
else {
std::memcpy(buff, &(elems[circ::index_of(cur_rd)].data_), sizeof(buff));
if (rd_.compare_exchange_weak(cur_rd, cur_rd + 1, std::memory_order_release)) {
std::forward<F>(f)(buff);
std::forward<R>(out)(true);
return true;
}
ipc::yield(k);
}
}
}
};
template <>
struct prod_cons_impl<wr<relat::single, relat::multi, trans::broadcast>> {
using rc_t = std::uint64_t;
enum : rc_t {
ep_mask = 0x00000000ffffffffull,
ep_incr = 0x0000000100000000ull
};
template <std::size_t DataSize, std::size_t AlignSize>
struct elem_t {
std::aligned_storage_t<DataSize, AlignSize> data_ {};
std::atomic<rc_t> rc_ { 0 }; // read-counter
};
alignas(cache_line_size) std::atomic<circ::u2_t> wt_; // write index
alignas(cache_line_size) rc_t epoch_ { 0 }; // only one writer
circ::u2_t cursor() const noexcept {
return wt_.load(std::memory_order_acquire);
}
template <typename W, typename F, typename E>
bool push(W* wrapper, F&& f, E* elems) {
E* el;
for (unsigned k = 0;;) {
circ::cc_t cc = wrapper->elems()->connections(std::memory_order_relaxed);
if (cc == 0) return false; // no reader
el = elems + circ::index_of(wt_.load(std::memory_order_relaxed));
// check all consumers have finished reading this element
auto cur_rc = el->rc_.load(std::memory_order_acquire);
circ::cc_t rem_cc = cur_rc & ep_mask;
if ((cc & rem_cc) && ((cur_rc & ~ep_mask) == epoch_)) {
return false; // has not finished yet
}
// consider rem_cc to be 0 here
if (el->rc_.compare_exchange_weak(
cur_rc, epoch_ | static_cast<rc_t>(cc), std::memory_order_release)) {
break;
}
ipc::yield(k);
}
std::forward<F>(f)(&(el->data_));
wt_.fetch_add(1, std::memory_order_release);
return true;
}
template <typename W, typename F, typename E>
bool force_push(W* wrapper, F&& f, E* elems) {
E* el;
epoch_ += ep_incr;
for (unsigned k = 0;;) {
circ::cc_t cc = wrapper->elems()->connections(std::memory_order_relaxed);
if (cc == 0) return false; // no reader
el = elems + circ::index_of(wt_.load(std::memory_order_relaxed));
// check all consumers have finished reading this element
auto cur_rc = el->rc_.load(std::memory_order_acquire);
circ::cc_t rem_cc = cur_rc & ep_mask;
if (cc & rem_cc) {
ipc::log("force_push: k = %u, cc = %u, rem_cc = %u\n", k, cc, rem_cc);
cc = wrapper->elems()->disconnect_receiver(rem_cc); // disconnect all invalid readers
if (cc == 0) return false; // no reader
}
// just compare & exchange
if (el->rc_.compare_exchange_weak(
cur_rc, epoch_ | static_cast<rc_t>(cc), std::memory_order_release)) {
break;
}
ipc::yield(k);
}
std::forward<F>(f)(&(el->data_));
wt_.fetch_add(1, std::memory_order_release);
return true;
}
template <typename W, typename F, typename R, typename E>
bool pop(W* wrapper, circ::u2_t& cur, F&& f, R&& out, E* elems) {
if (cur == cursor()) return false; // acquire
auto* el = elems + circ::index_of(cur++);
std::forward<F>(f)(&(el->data_));
for (unsigned k = 0;;) {
auto cur_rc = el->rc_.load(std::memory_order_acquire);
if ((cur_rc & ep_mask) == 0) {
std::forward<R>(out)(true);
return true;
}
auto nxt_rc = cur_rc & ~static_cast<rc_t>(wrapper->connected_id());
if (el->rc_.compare_exchange_weak(cur_rc, nxt_rc, std::memory_order_release)) {
std::forward<R>(out)((nxt_rc & ep_mask) == 0);
return true;
}
ipc::yield(k);
}
}
};
template <>
struct prod_cons_impl<wr<relat::multi, relat::multi, trans::broadcast>> {
using rc_t = std::uint64_t;
using flag_t = std::uint64_t;
enum : rc_t {
rc_mask = 0x00000000ffffffffull,
ep_mask = 0x00ffffffffffffffull,
ep_incr = 0x0100000000000000ull,
ic_mask = 0xff000000ffffffffull,
ic_incr = 0x0000000100000000ull
};
template <std::size_t DataSize, std::size_t AlignSize>
struct elem_t {
std::aligned_storage_t<DataSize, AlignSize> data_ {};
std::atomic<rc_t > rc_ { 0 }; // read-counter
std::atomic<flag_t> f_ct_ { 0 }; // commit flag
};
alignas(cache_line_size) std::atomic<circ::u2_t> ct_; // commit index
alignas(cache_line_size) std::atomic<rc_t> epoch_ { 0 };
circ::u2_t cursor() const noexcept {
return ct_.load(std::memory_order_acquire);
}
constexpr static rc_t inc_rc(rc_t rc) noexcept {
return (rc & ic_mask) | ((rc + ic_incr) & ~ic_mask);
}
constexpr static rc_t inc_mask(rc_t rc) noexcept {
return inc_rc(rc) & ~rc_mask;
}
template <typename W, typename F, typename E>
bool push(W* wrapper, F&& f, E* elems) {
E* el;
circ::u2_t cur_ct;
rc_t epoch = epoch_.load(std::memory_order_acquire);
for (unsigned k = 0;;) {
circ::cc_t cc = wrapper->elems()->connections(std::memory_order_relaxed);
if (cc == 0) return false; // no reader
el = elems + circ::index_of(cur_ct = ct_.load(std::memory_order_relaxed));
// check all consumers have finished reading this element
auto cur_rc = el->rc_.load(std::memory_order_relaxed);
circ::cc_t rem_cc = cur_rc & rc_mask;
if ((cc & rem_cc) && ((cur_rc & ~ep_mask) == epoch)) {
return false; // has not finished yet
}
else if (!rem_cc) {
auto cur_fl = el->f_ct_.load(std::memory_order_acquire);
if ((cur_fl != cur_ct) && cur_fl) {
return false; // full
}
}
// consider rem_cc to be 0 here
if (el->rc_.compare_exchange_weak(
cur_rc, inc_mask(epoch | (cur_rc & ep_mask)) | static_cast<rc_t>(cc), std::memory_order_relaxed) &&
epoch_.compare_exchange_weak(epoch, epoch, std::memory_order_acq_rel)) {
break;
}
ipc::yield(k);
}
// only one thread/process would touch here at one time
ct_.store(cur_ct + 1, std::memory_order_release);
std::forward<F>(f)(&(el->data_));
// set flag & try update wt
el->f_ct_.store(~static_cast<flag_t>(cur_ct), std::memory_order_release);
return true;
}
template <typename W, typename F, typename E>
bool force_push(W* wrapper, F&& f, E* elems) {
E* el;
circ::u2_t cur_ct;
rc_t epoch = epoch_.fetch_add(ep_incr, std::memory_order_release) + ep_incr;
for (unsigned k = 0;;) {
circ::cc_t cc = wrapper->elems()->connections(std::memory_order_relaxed);
if (cc == 0) return false; // no reader
el = elems + circ::index_of(cur_ct = ct_.load(std::memory_order_relaxed));
// check all consumers have finished reading this element
auto cur_rc = el->rc_.load(std::memory_order_acquire);
circ::cc_t rem_cc = cur_rc & rc_mask;
if (cc & rem_cc) {
ipc::log("force_push: k = %u, cc = %u, rem_cc = %u\n", k, cc, rem_cc);
cc = wrapper->elems()->disconnect_receiver(rem_cc); // disconnect all invalid readers
if (cc == 0) return false; // no reader
}
// just compare & exchange
if (el->rc_.compare_exchange_weak(
cur_rc, inc_mask(epoch | (cur_rc & ep_mask)) | static_cast<rc_t>(cc), std::memory_order_relaxed)) {
if (epoch == epoch_.load(std::memory_order_acquire)) {
break;
}
else if (push(wrapper, std::forward<F>(f), elems)) {
return true;
}
epoch = epoch_.fetch_add(ep_incr, std::memory_order_release) + ep_incr;
}
ipc::yield(k);
}
// only one thread/process would touch here at one time
ct_.store(cur_ct + 1, std::memory_order_release);
std::forward<F>(f)(&(el->data_));
// set flag & try update wt
el->f_ct_.store(~static_cast<flag_t>(cur_ct), std::memory_order_release);
return true;
}
template <typename W, typename F, typename R, typename E, std::size_t N>
bool pop(W* wrapper, circ::u2_t& cur, F&& f, R&& out, E(& elems)[N]) {
auto* el = elems + circ::index_of(cur);
auto cur_fl = el->f_ct_.load(std::memory_order_acquire);
if (cur_fl != ~static_cast<flag_t>(cur)) {
return false; // empty
}
++cur;
std::forward<F>(f)(&(el->data_));
for (unsigned k = 0;;) {
auto cur_rc = el->rc_.load(std::memory_order_acquire);
if ((cur_rc & rc_mask) == 0) {
std::forward<R>(out)(true);
el->f_ct_.store(cur + N - 1, std::memory_order_release);
return true;
}
auto nxt_rc = inc_rc(cur_rc) & ~static_cast<rc_t>(wrapper->connected_id());
bool last_one = false;
if ((last_one = (nxt_rc & rc_mask) == 0)) {
el->f_ct_.store(cur + N - 1, std::memory_order_release);
}
if (el->rc_.compare_exchange_weak(cur_rc, nxt_rc, std::memory_order_release)) {
std::forward<R>(out)(last_one);
return true;
}
ipc::yield(k);
}
}
};
} // namespace ipc
@@ -0,0 +1,75 @@
import threading
from predict import predict_no_ui_long_connection
from toolbox import CatchException, write_results_to_file
@CatchException
def 全项目切换英文(txt, top_p, temperature, chatbot, history, sys_prompt, WEB_PORT):
history = [] # 清空历史,以免输入溢出
# 集合文件
import time, glob, os
os.makedirs('gpt_log/generated_english_version', exist_ok=True)
os.makedirs('gpt_log/generated_english_version/crazy_functions', exist_ok=True)
file_manifest = [f for f in glob.glob('./*.py') if ('test_project' not in f) and ('gpt_log' not in f)] + \
[f for f in glob.glob('./crazy_functions/*.py') if ('test_project' not in f) and ('gpt_log' not in f)]
i_say_show_user_buffer = []
# 随便显示点什么防止卡顿的感觉
for index, fp in enumerate(file_manifest):
# if 'test_project' in fp: continue
with open(fp, 'r', encoding='utf-8') as f:
file_content = f.read()
i_say_show_user =f'[{index}/{len(file_manifest)}] 接下来请将以下代码中包含的所有中文转化为英文,只输出代码: {os.path.abspath(fp)}'
i_say_show_user_buffer.append(i_say_show_user)
chatbot.append((i_say_show_user, "[Local Message] 等待多线程操作,中间过程不予显示."))
yield chatbot, history, '正常'
# 任务函数
mutable_return = [None for _ in file_manifest]
def thread_worker(fp,index):
with open(fp, 'r', encoding='utf-8') as f:
file_content = f.read()
i_say = f'接下来请将以下代码中包含的所有中文转化为英文,只输出代码,文件名是{fp},文件代码是 ```{file_content}```'
# ** gpt request **
gpt_say = predict_no_ui_long_connection(inputs=i_say, top_p=top_p, temperature=temperature, history=history, sys_prompt=sys_prompt)
mutable_return[index] = gpt_say
# 所有线程同时开始执行任务函数
handles = [threading.Thread(target=thread_worker, args=(fp,index)) for index, fp in enumerate(file_manifest)]
for h in handles:
h.daemon = True
h.start()
chatbot.append(('开始了吗?', f'多线程操作已经开始'))
yield chatbot, history, '正常'
# 循环轮询各个线程是否执行完毕
cnt = 0
while True:
time.sleep(1)
th_alive = [h.is_alive() for h in handles]
if not any(th_alive): break
stat = ['执行中' if alive else '已完成' for alive in th_alive]
stat_str = '|'.join(stat)
cnt += 1
chatbot[-1] = (chatbot[-1][0], f'多线程操作已经开始,完成情况: {stat_str}' + ''.join(['.']*(cnt%4)))
yield chatbot, history, '正常'
# 把结果写入文件
for index, h in enumerate(handles):
h.join() # 这里其实不需要join了,肯定已经都结束了
fp = file_manifest[index]
gpt_say = mutable_return[index]
i_say_show_user = i_say_show_user_buffer[index]
where_to_relocate = f'gpt_log/generated_english_version/{fp}'
with open(where_to_relocate, 'w+', encoding='utf-8') as f: f.write(gpt_say.lstrip('```').rstrip('```'))
chatbot.append((i_say_show_user, f'[Local Message] 已完成{os.path.abspath(fp)}的转化,\n\n存入{os.path.abspath(where_to_relocate)}'))
history.append(i_say_show_user); history.append(gpt_say)
yield chatbot, history, '正常'
time.sleep(1)
# 备份一个文件
res = write_results_to_file(history)
chatbot.append(("生成一份任务执行报告", res))
yield chatbot, history, '正常'
+127
View File
@@ -0,0 +1,127 @@
from predict import predict_no_ui
from toolbox import CatchException, report_execption, write_results_to_file, predict_no_ui_but_counting_down
fast_debug = False
def 解析docx(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt):
import time, os
# pip install python-docx 用于docx格式,跨平台
# pip install pywin32 用于doc格式,仅支持Win平台
print('begin analysis on:', file_manifest)
for index, fp in enumerate(file_manifest):
if fp.split(".")[-1] == "docx":
from docx import Document
doc = Document(fp)
file_content = "\n".join([para.text for para in doc.paragraphs])
else:
import win32com.client
word = win32com.client.Dispatch("Word.Application")
word.visible = False
# 打开文件
print('fp', os.getcwd())
doc = word.Documents.Open(os.getcwd() + '/' + fp)
# file_content = doc.Content.Text
doc = word.ActiveDocument
file_content = doc.Range().Text
doc.Close()
word.Quit()
print(file_content)
prefix = "接下来请你逐文件分析下面的论文文件," if index == 0 else ""
# private_upload里面的文件名在解压zip后容易出现乱码(rar和7z格式正常),故可以只分析文章内容,不输入文件名
i_say = prefix + f'请对下面的文章片段用中英文做概述,文件名是{os.path.relpath(fp, project_folder)},' \
f'文章内容是 ```{file_content}```'
i_say_show_user = prefix + f'[{index+1}/{len(file_manifest)}] 假设你是论文审稿专家,请对下面的文章片段做概述: {os.path.abspath(fp)}'
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
yield chatbot, history, '正常'
if not fast_debug:
msg = '正常'
# ** gpt request **
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature,
history=[]) # 带超时倒计时
chatbot[-1] = (i_say_show_user, gpt_say)
history.append(i_say_show_user);
history.append(gpt_say)
yield chatbot, history, msg
if not fast_debug: time.sleep(2)
"""
# 可按需启用
i_say = f'根据你上述的分析,对全文进行概括,用学术性语言写一段中文摘要,然后再写一篇英文的。'
chatbot.append((i_say, "[Local Message] waiting gpt response."))
yield chatbot, history, '正常'
i_say = f'我想让你做一个论文写作导师。您的任务是使用人工智能工具(例如自然语言处理)提供有关如何改进其上述文章的反馈。' \
f'您还应该利用您在有效写作技巧方面的修辞知识和经验来建议作者可以更好地以书面形式表达他们的想法和想法的方法。' \
f'根据你之前的分析,提出建议'
chatbot.append((i_say, "[Local Message] waiting gpt response."))
yield chatbot, history, '正常'
"""
if not fast_debug:
msg = '正常'
# ** gpt request **
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say, chatbot, top_p, temperature,
history=history) # 带超时倒计时
chatbot[-1] = (i_say, gpt_say)
history.append(i_say)
history.append(gpt_say)
yield chatbot, history, msg
res = write_results_to_file(history)
chatbot.append(("完成了吗?", res))
yield chatbot, history, msg
@CatchException
def 总结word文档(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT):
import glob, os
# 基本信息:功能、贡献者
chatbot.append([
"函数插件功能?",
"批量总结Word文档。函数插件贡献者: JasonGuo1"])
yield chatbot, history, '正常'
# 尝试导入依赖,如果缺少依赖,则给出安装建议
try:
from docx import Document
except:
report_execption(chatbot, history,
a=f"解析项目: {txt}",
b=f"导入软件依赖失败。使用该模块需要额外依赖,安装方法```pip install --upgrade python-docx pywin32```。")
yield chatbot, history, '正常'
return
# 清空历史,以免输入溢出
history = []
# 检测输入参数,如没有给定输入参数,直接退出
if os.path.exists(txt):
project_folder = txt
else:
if txt == "": txt = '空空如也的输入栏'
report_execption(chatbot, history, a=f"解析项目: {txt}", b=f"找不到本地项目或无权访问: {txt}")
yield chatbot, history, '正常'
return
# 搜索需要处理的文件清单
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.docx', recursive=True)] + \
[f for f in glob.glob(f'{project_folder}/**/*.doc', recursive=True)]
# [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)] + \
# [f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)] + \
# [f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
# 如果没找到任何文件
if len(file_manifest) == 0:
report_execption(chatbot, history, a=f"解析项目: {txt}", b=f"找不到任何.docx或doc文件: {txt}")
yield chatbot, history, '正常'
return
# 开始正式执行任务
yield from 解析docx(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt)
+154
View File
@@ -0,0 +1,154 @@
from predict import predict_no_ui
from toolbox import CatchException, report_execption, write_results_to_file, predict_no_ui_but_counting_down
import re
import unicodedata
fast_debug = False
def is_paragraph_break(match):
"""
根据给定的匹配结果来判断换行符是否表示段落分隔
如果换行符前为句子结束标志句号感叹号问号且下一个字符为大写字母则换行符更有可能表示段落分隔
也可以根据之前的内容长度来判断段落是否已经足够长
"""
prev_char, next_char = match.groups()
# 句子结束标志
sentence_endings = ".!?"
# 设定一个最小段落长度阈值
min_paragraph_length = 140
if prev_char in sentence_endings and next_char.isupper() and len(match.string[:match.start(1)]) > min_paragraph_length:
return "\n\n"
else:
return " "
def normalize_text(text):
"""
通过把连字ligatures等文本特殊符号转换为其基本形式来对文本进行归一化处理
例如将连字 "fi" 转换为 "f" "i"
"""
# 对文本进行归一化处理,分解连字
normalized_text = unicodedata.normalize("NFKD", text)
# 替换其他特殊字符
cleaned_text = re.sub(r'[^\x00-\x7F]+', '', normalized_text)
return cleaned_text
def clean_text(raw_text):
"""
对从 PDF 提取出的原始文本进行清洗和格式化处理
1. 对原始文本进行归一化处理
2. 替换跨行的连词例如 Espe-\ncially 转换为 Especially
3. 根据 heuristic 规则判断换行符是否是段落分隔并相应地进行替换
"""
# 对文本进行归一化处理
normalized_text = normalize_text(raw_text)
# 替换跨行的连词
text = re.sub(r'(\w+-\n\w+)', lambda m: m.group(1).replace('-\n', ''), normalized_text)
# 根据前后相邻字符的特点,找到原文本中的换行符
newlines = re.compile(r'(\S)\n(\S)')
# 根据 heuristic 规则,用空格或段落分隔符替换原换行符
final_text = re.sub(newlines, lambda m: m.group(1) + is_paragraph_break(m) + m.group(2), text)
return final_text.strip()
def 解析PDF(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt):
import time, glob, os, fitz
print('begin analysis on:', file_manifest)
for index, fp in enumerate(file_manifest):
with fitz.open(fp) as doc:
file_content = ""
for page in doc:
file_content += page.get_text()
file_content = clean_text(file_content)
print(file_content)
prefix = "接下来请你逐文件分析下面的论文文件,概括其内容" if index==0 else ""
i_say = prefix + f'请对下面的文章片段用中文做一个概述,文件名是{os.path.relpath(fp, project_folder)},文章内容是 ```{file_content}```'
i_say_show_user = prefix + f'[{index}/{len(file_manifest)}] 请对下面的文章片段做一个概述: {os.path.abspath(fp)}'
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
print('[1] yield chatbot, history')
yield chatbot, history, '正常'
if not fast_debug:
msg = '正常'
# ** gpt request **
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature, history=[]) # 带超时倒计时
print('[2] end gpt req')
chatbot[-1] = (i_say_show_user, gpt_say)
history.append(i_say_show_user); history.append(gpt_say)
print('[3] yield chatbot, history')
yield chatbot, history, msg
print('[4] next')
if not fast_debug: time.sleep(2)
all_file = ', '.join([os.path.relpath(fp, project_folder) for index, fp in enumerate(file_manifest)])
i_say = f'根据以上你自己的分析,对全文进行概括,用学术性语言写一段中文摘要,然后再写一段英文摘要(包括{all_file})。'
chatbot.append((i_say, "[Local Message] waiting gpt response."))
yield chatbot, history, '正常'
if not fast_debug:
msg = '正常'
# ** gpt request **
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say, chatbot, top_p, temperature, history=history) # 带超时倒计时
chatbot[-1] = (i_say, gpt_say)
history.append(i_say); history.append(gpt_say)
yield chatbot, history, msg
res = write_results_to_file(history)
chatbot.append(("完成了吗?", res))
yield chatbot, history, msg
@CatchException
def 批量总结PDF文档(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT):
import glob, os
# 基本信息:功能、贡献者
chatbot.append([
"函数插件功能?",
"批量总结PDF文档。函数插件贡献者: ValeriaWongEralien"])
yield chatbot, history, '正常'
# 尝试导入依赖,如果缺少依赖,则给出安装建议
try:
import fitz
except:
report_execption(chatbot, history,
a = f"解析项目: {txt}",
b = f"导入软件依赖失败。使用该模块需要额外依赖,安装方法```pip install --upgrade pymupdf```。")
yield chatbot, history, '正常'
return
# 清空历史,以免输入溢出
history = []
# 检测输入参数,如没有给定输入参数,直接退出
if os.path.exists(txt):
project_folder = txt
else:
if txt == "": txt = '空空如也的输入栏'
report_execption(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
yield chatbot, history, '正常'
return
# 搜索需要处理的文件清单
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.pdf', recursive=True)] # + \
# [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)] + \
# [f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)] + \
# [f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
# 如果没找到任何文件
if len(file_manifest) == 0:
report_execption(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex或.pdf文件: {txt}")
yield chatbot, history, '正常'
return
# 开始正式执行任务
yield from 解析PDF(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt)
@@ -0,0 +1,151 @@
from predict import predict_no_ui
from toolbox import CatchException, report_execption, write_results_to_file, predict_no_ui_but_counting_down
fast_debug = False
def readPdf(pdfPath):
"""
读取pdf文件返回文本内容
"""
import pdfminer
from pdfminer.pdfparser import PDFParser
from pdfminer.pdfdocument import PDFDocument
from pdfminer.pdfpage import PDFPage, PDFTextExtractionNotAllowed
from pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreter
from pdfminer.pdfdevice import PDFDevice
from pdfminer.layout import LAParams
from pdfminer.converter import PDFPageAggregator
fp = open(pdfPath, 'rb')
# Create a PDF parser object associated with the file object
parser = PDFParser(fp)
# Create a PDF document object that stores the document structure.
# Password for initialization as 2nd parameter
document = PDFDocument(parser)
# Check if the document allows text extraction. If not, abort.
if not document.is_extractable:
raise PDFTextExtractionNotAllowed
# Create a PDF resource manager object that stores shared resources.
rsrcmgr = PDFResourceManager()
# Create a PDF device object.
# device = PDFDevice(rsrcmgr)
# BEGIN LAYOUT ANALYSIS.
# Set parameters for analysis.
laparams = LAParams(
char_margin=10.0,
line_margin=0.2,
boxes_flow=0.2,
all_texts=False,
)
# Create a PDF page aggregator object.
device = PDFPageAggregator(rsrcmgr, laparams=laparams)
# Create a PDF interpreter object.
interpreter = PDFPageInterpreter(rsrcmgr, device)
# loop over all pages in the document
outTextList = []
for page in PDFPage.create_pages(document):
# read the page into a layout object
interpreter.process_page(page)
layout = device.get_result()
for obj in layout._objs:
if isinstance(obj, pdfminer.layout.LTTextBoxHorizontal):
# print(obj.get_text())
outTextList.append(obj.get_text())
return outTextList
def 解析Paper(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt):
import time, glob, os
from bs4 import BeautifulSoup
print('begin analysis on:', file_manifest)
for index, fp in enumerate(file_manifest):
if ".tex" in fp:
with open(fp, 'r', encoding='utf-8') as f:
file_content = f.read()
if ".pdf" in fp.lower():
file_content = readPdf(fp)
file_content = BeautifulSoup(''.join(file_content), features="lxml").body.text.encode('gbk', 'ignore').decode('gbk')
prefix = "接下来请你逐文件分析下面的论文文件,概括其内容" if index==0 else ""
i_say = prefix + f'请对下面的文章片段用中文做一个概述,文件名是{os.path.relpath(fp, project_folder)},文章内容是 ```{file_content}```'
i_say_show_user = prefix + f'[{index}/{len(file_manifest)}] 请对下面的文章片段做一个概述: {os.path.abspath(fp)}'
chatbot.append((i_say_show_user, "[Local Message] waiting gpt response."))
print('[1] yield chatbot, history')
yield chatbot, history, '正常'
if not fast_debug:
msg = '正常'
# ** gpt request **
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature, history=[]) # 带超时倒计时
print('[2] end gpt req')
chatbot[-1] = (i_say_show_user, gpt_say)
history.append(i_say_show_user); history.append(gpt_say)
print('[3] yield chatbot, history')
yield chatbot, history, msg
print('[4] next')
if not fast_debug: time.sleep(2)
all_file = ', '.join([os.path.relpath(fp, project_folder) for index, fp in enumerate(file_manifest)])
i_say = f'根据以上你自己的分析,对全文进行概括,用学术性语言写一段中文摘要,然后再写一段英文摘要(包括{all_file})。'
chatbot.append((i_say, "[Local Message] waiting gpt response."))
yield chatbot, history, '正常'
if not fast_debug:
msg = '正常'
# ** gpt request **
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say, chatbot, top_p, temperature, history=history) # 带超时倒计时
chatbot[-1] = (i_say, gpt_say)
history.append(i_say); history.append(gpt_say)
yield chatbot, history, msg
res = write_results_to_file(history)
chatbot.append(("完成了吗?", res))
yield chatbot, history, msg
@CatchException
def 批量总结PDF文档pdfminer(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT):
history = [] # 清空历史,以免输入溢出
import glob, os
# 基本信息:功能、贡献者
chatbot.append([
"函数插件功能?",
"批量总结PDF文档,此版本使用pdfminer插件,带token约简功能。函数插件贡献者: Euclid-Jie。"])
yield chatbot, history, '正常'
# 尝试导入依赖,如果缺少依赖,则给出安装建议
try:
import pdfminer, bs4
except:
report_execption(chatbot, history,
a = f"解析项目: {txt}",
b = f"导入软件依赖失败。使用该模块需要额外依赖,安装方法```pip install --upgrade pdfminer beautifulsoup4```。")
yield chatbot, history, '正常'
return
if os.path.exists(txt):
project_folder = txt
else:
if txt == "": txt = '空空如也的输入栏'
report_execption(chatbot, history, a = f"解析项目: {txt}", b = f"找不到本地项目或无权访问: {txt}")
yield chatbot, history, '正常'
return
file_manifest = [f for f in glob.glob(f'{project_folder}/**/*.tex', recursive=True)] + \
[f for f in glob.glob(f'{project_folder}/**/*.pdf', recursive=True)] # + \
# [f for f in glob.glob(f'{project_folder}/**/*.cpp', recursive=True)] + \
# [f for f in glob.glob(f'{project_folder}/**/*.c', recursive=True)]
if len(file_manifest) == 0:
report_execption(chatbot, history, a = f"解析项目: {txt}", b = f"找不到任何.tex或pdf文件: {txt}")
yield chatbot, history, '正常'
return
yield from 解析Paper(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt)
+4 -3
View File
@@ -50,7 +50,8 @@ def 解析源代码(file_manifest, project_folder, top_p, temperature, chatbot,
def 解析项目本身(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT):
history = [] # 清空历史,以免输入溢出
import time, glob, os
file_manifest = [f for f in glob.glob('*.py')]
file_manifest = [f for f in glob.glob('./*.py') if ('test_project' not in f) and ('gpt_log' not in f)] + \
[f for f in glob.glob('./crazy_functions/*.py') if ('test_project' not in f) and ('gpt_log' not in f)]
for index, fp in enumerate(file_manifest):
# if 'test_project' in fp: continue
with open(fp, 'r', encoding='utf-8') as f:
@@ -65,7 +66,7 @@ def 解析项目本身(txt, top_p, temperature, chatbot, history, systemPromptTx
if not fast_debug:
# ** gpt request **
# gpt_say = predict_no_ui(inputs=i_say, top_p=top_p, temperature=temperature)
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature, history=[]) # 带超时倒计时
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature, history=[], long_connection=True) # 带超时倒计时
chatbot[-1] = (i_say_show_user, gpt_say)
history.append(i_say_show_user); history.append(gpt_say)
@@ -79,7 +80,7 @@ def 解析项目本身(txt, top_p, temperature, chatbot, history, systemPromptTx
if not fast_debug:
# ** gpt request **
# gpt_say = predict_no_ui(inputs=i_say, top_p=top_p, temperature=temperature, history=history)
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say, chatbot, top_p, temperature, history=history) # 带超时倒计时
gpt_say = yield from predict_no_ui_but_counting_down(i_say, i_say, chatbot, top_p, temperature, history=history, long_connection=True) # 带超时倒计时
chatbot[-1] = (i_say, gpt_say)
history.append(i_say); history.append(gpt_say)
+13 -5
View File
@@ -1,16 +1,24 @@
from predict import predict_no_ui
from toolbox import CatchException, report_execption, write_results_to_file, predict_no_ui_but_counting_down
fast_debug = False
from predict import predict_no_ui_long_connection
from toolbox import CatchException, report_execption, write_results_to_file
import datetime
@CatchException
def 高阶功能模板函数(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT):
history = [] # 清空历史,以免输入溢出
chatbot.append(("这是什么功能?", "[Local Message] 请注意,您正在调用一个[函数插件]的模板,该函数面向希望实现更多有趣功能的开发者,它可以作为创建新功能函数的模板。为了做到简单易读,该函数只有25行代码,所以不会实时反馈文字流或心跳,请耐心等待程序输出完成。此外我们也提供可同步处理大量文件的多线程Demo供您参考。您若希望分享新的功能模组,请不吝PR!"))
yield chatbot, history, '正常' # 由于请求gpt需要一段时间,我们先及时地做一次状态显示
for i in range(5):
i_say = f'我给出一个数字,你给出该数字的平方。我给出数字:{i}'
currentMonth = (datetime.date.today() + datetime.timedelta(days=i)).month
currentDay = (datetime.date.today() + datetime.timedelta(days=i)).day
i_say = f'历史中哪些事件发生在{currentMonth}{currentDay}日?列举两条并发送相关图片。发送图片时,请使用Markdown,将Unsplash API中的PUT_YOUR_QUERY_HERE替换成描述该事件的一个最重要的单词。'
chatbot.append((i_say, "[Local Message] waiting gpt response."))
yield chatbot, history, '正常' # 由于请求gpt需要一段时间,我们先及时地做一次状态显示
gpt_say = predict_no_ui(inputs=i_say, top_p=top_p, temperature=temperature) # 请求gpt,需要一段时间
# history = [] 每次询问不携带之前的询问历史
gpt_say = predict_no_ui_long_connection(
inputs=i_say, top_p=top_p, temperature=temperature, history=[],
sys_prompt="当你想发送一张照片时,请使用Markdown, 并且不要有反斜线, 不要用代码块。使用 Unsplash API (https://source.unsplash.com/1280x720/? < PUT_YOUR_QUERY_HERE >)。") # 请求gpt,需要一段时间
chatbot[-1] = (i_say, gpt_say)
history.append(i_say);history.append(gpt_say)
+55 -44
View File
@@ -1,59 +1,70 @@
# """
# 'primary' for main call-to-action,
# 'secondary' for a more subdued style,
# 'stop' for a stop button.
# """
# 'primary' 颜色对应 theme.py 中的 primary_hue
# 'secondary' 颜色对应 theme.py 中的 neutral_hue
# 'stop' 颜色对应 theme.py 中的 color_er
# 默认按钮颜色是 secondary
from toolbox import clear_line_break
def get_functionals():
return {
"英语学术润色": {
"Prefix": "Below is a paragraph from an academic paper. Polish the writing to meet the academic style, \
improve the spelling, grammar, clarity, concision and overall readability. When neccessary, rewrite the whole sentence. \
Furthermore, list all modification and explain the reasons to do so in markdown table.\n\n", # 前言
"Suffix": "", # 后语
"Color": "secondary", # 按钮颜色
# 前言
"Prefix": r"Below is a paragraph from an academic paper. Polish the writing to meet the academic style, " +
r"improve the spelling, grammar, clarity, concision and overall readability. When necessary, rewrite the whole sentence. " +
r"Furthermore, list all modification and explain the reasons to do so in markdown table." + "\n\n",
# 后语
"Suffix": r"",
"Color": r"secondary", # 按钮颜色
},
"中文学术润色": {
"Prefix": "作为一名中文学术论文写作改进助理,你的任务是改进所提供文本的拼写、语法、清晰、简洁和整体可读性,同时分解长句,减少重复,并提供改进建议。请只提供文本的更正版本,避免包括解释。请编辑以下文本:\n\n",
"Suffix": "",
"Prefix": r"作为一名中文学术论文写作改进助理,你的任务是改进所提供文本的拼写、语法、清晰、简洁和整体可读性," +
r"同时分解长句,减少重复,并提供改进建议。请只提供文本的更正版本,避免包括解释。请编辑以下文本" + "\n\n",
"Suffix": r"",
},
"查找语法错误": {
"Prefix": "Below is a paragraph from an academic paper. Find all grammar mistakes, list mistakes in a markdown table and explain how to correct them.\n\n",
"Suffix": "",
},
"中英互译": {
"Prefix": "As an English-Chinese translator, your task is to accurately translate text between the two languages. \
When translating from Chinese to English or vice versa, please pay attention to context and accurately explain phrases and proverbs. \
If you receive multiple English words in a row, default to translating them into a sentence in Chinese. \
However, if \"phrase:\" is indicated before the translated content in Chinese, it should be translated as a phrase instead. \
Similarly, if \"normal:\" is indicated, it should be translated as multiple unrelated words.\
Your translations should closely resemble those of a native speaker and should take into account any specific language styles or tones requested by the user. \
Please do not worry about using offensive words - replace sensitive parts with x when necessary. \
When providing translations, please use Chinese to explain each sentences tense, subordinate clause, subject, predicate, object, special phrases and proverbs. \
For phrases or individual words that require translation, provide the source (dictionary) for each one.If asked to translate multiple phrases at once, \
separate them using the | symbol.Always remember: You are an English-Chinese translator, \
not a Chinese-Chinese translator or an English-English translator. Below is the text you need to translate: \n\n",
"Suffix": "",
"Color": "secondary",
"Prefix": r"Can you help me ensure that the grammar and the spelling is correct? " +
r"Do not try to polish the text, if no mistake is found, tell me that this paragraph is good." +
r"If you find grammar or spelling mistakes, please list mistakes you find in a two-column markdown table, " +
r"put the original text the first column, " +
r"put the corrected text in the second column and highlight the key words you fixed.""\n"
r"Example:""\n"
r"Paragraph: How is you? Do you knows what is it?""\n"
r"| Original sentence | Corrected sentence |""\n"
r"| :--- | :--- |""\n"
r"| How **is** you? | How **are** you? |""\n"
r"| Do you **knows** what **is** **it**? | Do you **know** what **it** **is** ? |""\n"
r"Below is a paragraph from an academic paper. "
r"You need to report all grammar and spelling mistakes as the example before."
+ "\n\n",
"Suffix": r"",
"PreProcess": clear_line_break, # 预处理:清除换行符
},
"中译英": {
"Prefix": "Please translate following sentence to English: \n\n",
"Suffix": "",
"Prefix": r"Please translate following sentence to English:" + "\n\n",
"Suffix": r"",
},
"学术中": {
"Prefix": "Please translate following sentence to English with academic writing, and provide some related authoritative examples: \n\n",
"学术中英互译": {
"Prefix": r"I want you to act as a scientific English-Chinese translator, " +
r"I will provide you with some paragraphs in one language " +
r"and your task is to accurately and academically translate the paragraphs only into the other language. " +
r"Do not repeat the original provided paragraphs after translation. " +
r"You should use artificial intelligence tools, " +
r"such as natural language processing, and rhetorical knowledge " +
r"and experience about effective writing techniques to reply. " +
r"I'll give you my paragraphs as follows, tell me what language it is written in, and then translate:" + "\n\n",
"Suffix": "",
},
"英译中": {
"Prefix": "请翻译成中文:\n\n",
"Suffix": "",
},
"解释代码": {
"Prefix": "请解释以下代码:\n```\n",
"Suffix": "\n```\n",
"Color": "secondary",
},
"英译中": {
"Prefix": r"请翻译成中文:" + "\n\n",
"Suffix": r"",
},
"找图片": {
"Prefix": r"我需要你找一张网络图片。使用Unsplash API(https://source.unsplash.com/960x640/?<英语关键词>)获取图片URL" +
r"然后请使用Markdown格式封装,并且不要有反斜线,不要用代码块。现在,请按以下描述给我发送图片:" + "\n\n",
"Suffix": r"",
},
"解释代码": {
"Prefix": r"请解释以下代码:" + "\n```\n",
"Suffix": "\n```\n",
},
}
+52 -36
View File
@@ -1,3 +1,11 @@
from toolbox import HotReload # HotReload 的意思是热更新,修改函数插件后,不需要重启程序,代码直接生效
# UserVisibleLevel是过滤器参数。
# 由于UI界面空间有限,所以通过这种方式决定UI界面中显示哪些插件
# 默认函数插件 VisibleLevel 是 0
# 当 UserVisibleLevel >= 函数插件的 VisibleLevel 时,该函数插件才会被显示出来
UserVisibleLevel = 1
def get_crazy_functionals():
from crazy_functions.读文章写摘要 import 读文章写摘要
@@ -7,62 +15,70 @@ def get_crazy_functionals():
from crazy_functions.解析项目源代码 import 解析一个C项目的头文件
from crazy_functions.解析项目源代码 import 解析一个C项目
from crazy_functions.高级功能函数模板 import 高阶功能模板函数
from crazy_functions.代码重写为全英文_多线程 import 全项目切换英文
return {
"[实验] 请解析并解构此项目本身": {
function_plugins = {
"请解析并解构此项目本身(源码自译解)": {
"AsButton": False, # 加入下拉菜单中
"Function": 解析项目本身
},
"[实验] 解析整个py项目(配合input输入框)": {
"解析整个Py项目": {
"Color": "stop", # 按钮颜色
"Function": 解析一个Python项目
},
"[实验] 解析整个C++项目头文件(配合input输入框)": {
"解析整个C++项目头文件": {
"Color": "stop", # 按钮颜色
"Function": 解析一个C项目的头文件
},
"[实验] 解析整个C++项目(配合input输入框": {
"解析整个C++项目(.cpp/.h": {
"Color": "stop", # 按钮颜色
"AsButton": False, # 加入下拉菜单中
"Function": 解析一个C项目
},
"[实验] 读tex论文写摘要(配合input输入框)": {
"读Tex论文写摘要": {
"Color": "stop", # 按钮颜色
"Function": 读文章写摘要
},
"[实验] 批量生成函数注释(配合input输入框)": {
"批量生成函数注释": {
"Color": "stop", # 按钮颜色
"Function": 批量生成函数注释
},
"[实验] 实验功能函数模板": {
"Color": "stop", # 按钮颜色
"Function": 高阶功能模板函数
"[多线程demo] 把本项目源代码切换成全英文": {
# HotReload 的意思是热更新,修改函数插件代码后,不需要重启程序,代码直接生效
"Function": HotReload(全项目切换英文)
},
"[函数插件模板demo] 历史上的今天": {
# HotReload 的意思是热更新,修改函数插件代码后,不需要重启程序,代码直接生效
"Function": HotReload(高阶功能模板函数)
},
}
def on_file_uploaded(files, chatbot, txt):
if len(files) == 0: return chatbot, txt
import shutil, os, time, glob
from toolbox import extract_archive
try: shutil.rmtree('./private_upload/')
except: pass
time_tag = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
os.makedirs(f'private_upload/{time_tag}', exist_ok=True)
for file in files:
file_origin_name = os.path.basename(file.orig_name)
shutil.copy(file.name, f'private_upload/{time_tag}/{file_origin_name}')
extract_archive(f'private_upload/{time_tag}/{file_origin_name}',
dest_dir=f'private_upload/{time_tag}/{file_origin_name}.extract')
moved_files = [fp for fp in glob.glob('private_upload/**/*', recursive=True)]
txt = f'private_upload/{time_tag}'
moved_files_str = '\t\n\n'.join(moved_files)
chatbot.append(['我上传了文件,请查收',
f'[Local Message] 收到以下文件: \n\n{moved_files_str}\n\n调用路径参数已自动修正到: \n\n{txt}\n\n现在您可以直接选择任意实现性功能'])
return chatbot, txt
# VisibleLevel=1 经过测试,但功能上距离达到完美状态还差一点点
if UserVisibleLevel >= 1:
from crazy_functions.批量总结PDF文档 import 批量总结PDF文档
from crazy_functions.批量总结PDF文档pdfminer import 批量总结PDF文档pdfminer
from crazy_functions.总结word文档 import 总结word文档
function_plugins.update({
"[仅供开发调试] 批量总结PDF文档": {
"Color": "stop",
"Function": HotReload(批量总结PDF文档) # HotReload 的意思是热更新,修改函数插件代码后,不需要重启程序,代码直接生效
},
"[仅供开发调试] 批量总结PDF文档pdfminer": {
"Color": "stop",
"AsButton": False, # 加入下拉菜单中
"Function": HotReload(批量总结PDF文档pdfminer)
},
"[仅供开发调试] 批量总结Word文档": {
"Color": "stop",
"Function": HotReload(总结word文档)
},
})
# VisibleLevel=2 尚未充分测试的函数插件,放在这里
if UserVisibleLevel >= 2:
function_plugins.update({
})
return function_plugins
def on_report_generated(files, chatbot):
from toolbox import find_recent_files
report_files = find_recent_files('gpt_log')
if len(report_files) == 0: return report_files, chatbot
# files.extend(report_files)
chatbot.append(['汇总报告如何远程获取?', '汇总报告已经添加到右侧文件上传区,请查收。'])
return report_files, chatbot
+90 -49
View File
@@ -1,103 +1,144 @@
import os; os.environ['no_proxy'] = '*' # 避免代理网络产生意外污染
import gradio as gr
from predict import predict
from toolbox import format_io, find_free_port
from toolbox import format_io, find_free_port, on_file_uploaded, on_report_generated, get_conf
# 建议您复制一个config_private.py放自己的秘密, 如API和代理网址, 避免不小心传github被别人看到
try: from config_private import proxies, WEB_PORT, LLM_MODEL
except: from config import proxies, WEB_PORT, LLM_MODEL
proxies, WEB_PORT, LLM_MODEL, CONCURRENT_COUNT, AUTHENTICATION, CHATBOT_HEIGHT = \
get_conf('proxies', 'WEB_PORT', 'LLM_MODEL', 'CONCURRENT_COUNT', 'AUTHENTICATION', 'CHATBOT_HEIGHT')
# 如果WEB_PORT是-1, 则随机选取WEB端口
PORT = find_free_port() if WEB_PORT <= 0 else WEB_PORT
if not AUTHENTICATION: AUTHENTICATION = None
initial_prompt = "Serve me as a writing and programming assistant."
title_html = """<h1 align="center">ChatGPT 学术优化</h1>"""
# 问询记录, python 版本建议3.9+(越新越好)
import logging
os.makedirs('gpt_log', exist_ok=True)
try:logging.basicConfig(filename='gpt_log/chat_secrets.log', level=logging.INFO, encoding='utf-8')
except:logging.basicConfig(filename='gpt_log/chat_secrets.log', level=logging.INFO)
print('所有问询记录将自动保存在本地目录./gpt_log/chat_secrets.log, 请注意自我隐私保护哦!')
os.makedirs("gpt_log", exist_ok=True)
try:logging.basicConfig(filename="gpt_log/chat_secrets.log", level=logging.INFO, encoding="utf-8")
except:logging.basicConfig(filename="gpt_log/chat_secrets.log", level=logging.INFO)
print("所有问询记录将自动保存在本地目录./gpt_log/chat_secrets.log, 请注意自我隐私保护哦!")
# 一些普通功能模块
from functional import get_functionals
functional = get_functionals()
# 对一些丧心病狂的实验性功能模块进行测试
from functional_crazy import get_crazy_functionals, on_file_uploaded, on_report_generated
crazy_functional = get_crazy_functionals()
# 高级函数插件
from functional_crazy import get_crazy_functionals
crazy_fns = get_crazy_functionals()
# 处理markdown文本格式的转变
gr.Chatbot.postprocess = format_io
# 做一些外观色彩上的调整
from theme import adjust_theme
from theme import adjust_theme, advanced_css
set_theme = adjust_theme()
with gr.Blocks(theme=set_theme, analytics_enabled=False) as demo:
cancel_handles = []
with gr.Blocks(theme=set_theme, analytics_enabled=False, css=advanced_css) as demo:
gr.HTML(title_html)
with gr.Row():
with gr.Row().style(equal_height=True):
with gr.Column(scale=2):
chatbot = gr.Chatbot()
chatbot.style(height=1000)
chatbot.style()
chatbot.style(height=CHATBOT_HEIGHT)
history = gr.State([])
TRUE = gr.State(True)
FALSE = gr.State(False)
with gr.Column(scale=1):
with gr.Row():
with gr.Column(scale=12):
txt = gr.Textbox(show_label=False, placeholder="Input question here.").style(container=False)
with gr.Column(scale=1):
with gr.Row():
submitBtn = gr.Button("提交", variant="primary")
with gr.Row():
resetBtn = gr.Button("重置", variant="secondary"); resetBtn.style(size="sm")
stopBtn = gr.Button("停止", variant="secondary"); stopBtn.style(size="sm")
with gr.Row():
from check_proxy import check_proxy
statusDisplay = gr.Markdown(f"Tip: 按Enter提交, 按Shift+Enter换行. \nNetwork: {check_proxy(proxies)}\nModel: {LLM_MODEL}")
status = gr.Markdown(f"Tip: 按Enter提交, 按Shift+Enter换行。当前模型: {LLM_MODEL} \n {check_proxy(proxies)}")
with gr.Accordion("基础功能区", open=True) as area_basic_fn:
with gr.Row():
for k in functional:
variant = functional[k]["Color"] if "Color" in functional[k] else "secondary"
functional[k]["Button"] = gr.Button(k, variant=variant)
with gr.Accordion("函数插件区", open=True) as area_crazy_fn:
with gr.Row():
gr.Markdown("以下部分实验性功能需从input读取路径.")
gr.Markdown("注意:以下“红颜色”标识的函数插件需从input读取路径作为参数.")
with gr.Row():
for k in crazy_functional:
variant = crazy_functional[k]["Color"] if "Color" in crazy_functional[k] else "secondary"
crazy_functional[k]["Button"] = gr.Button(k, variant=variant)
for k in crazy_fns:
if not crazy_fns[k].get("AsButton", True): continue
variant = crazy_fns[k]["Color"] if "Color" in crazy_fns[k] else "secondary"
crazy_fns[k]["Button"] = gr.Button(k, variant=variant)
with gr.Row():
gr.Markdown("上传本地文件供上面的实验性功能调用.")
with gr.Accordion("更多函数插件", open=True):
dropdown_fn_list = [k for k in crazy_fns.keys() if not crazy_fns[k].get("AsButton", True)]
with gr.Column(scale=1):
dropdown = gr.Dropdown(dropdown_fn_list, value=r"打开插件列表", label="").style(container=False)
with gr.Column(scale=1):
switchy_bt = gr.Button(r"请先从插件列表中选择", variant="secondary")
with gr.Row():
file_upload = gr.Files(label='任何文件,但推荐上传压缩文件(zip, tar)', file_count="multiple")
systemPromptTxt = gr.Textbox(show_label=True, placeholder=f"System Prompt", label="System prompt", value=initial_prompt).style(container=True)
#inputs, top_p, temperature, top_k, repetition_penalty
with gr.Accordion("arguments", open=False):
with gr.Accordion("点击展开“文件上传区”。上传本地文件可供红色函数插件调用。", open=False) as area_file_up:
file_upload = gr.Files(label="任何文件, 但推荐上传压缩文件(zip, tar)", file_count="multiple")
with gr.Accordion("展开SysPrompt & GPT参数 & 交互界面布局", open=False):
system_prompt = gr.Textbox(show_label=True, placeholder=f"System Prompt", label="System prompt", value=initial_prompt)
top_p = gr.Slider(minimum=-0, maximum=1.0, value=1.0, step=0.01,interactive=True, label="Top-p (nucleus sampling)",)
temperature = gr.Slider(minimum=-0, maximum=5.0, value=1.0, step=0.01, interactive=True, label="Temperature",)
temperature = gr.Slider(minimum=-0, maximum=2.0, value=1.0, step=0.01, interactive=True, label="Temperature",)
checkboxes = gr.CheckboxGroup(["基础功能区", "函数插件区"], value=["基础功能区", "函数插件区"], label="显示/隐藏功能区")
txt.submit(predict, [txt, top_p, temperature, chatbot, history, systemPromptTxt], [chatbot, history, statusDisplay])
submitBtn.click(predict, [txt, top_p, temperature, chatbot, history, systemPromptTxt], [chatbot, history, statusDisplay], show_progress=True)
# 功能区显示开关与功能区的互动
def fn_area_visibility(a):
ret = {}
ret.update({area_basic_fn: gr.update(visible=("基础功能区" in a))})
ret.update({area_crazy_fn: gr.update(visible=("函数插件区" in a))})
return ret
checkboxes.select(fn_area_visibility, [checkboxes], [area_basic_fn, area_crazy_fn] )
# 整理反复出现的控件句柄组合
input_combo = [txt, top_p, temperature, chatbot, history, system_prompt]
output_combo = [chatbot, history, status]
predict_args = dict(fn=predict, inputs=input_combo, outputs=output_combo)
empty_txt_args = dict(fn=lambda: "", inputs=[], outputs=[txt]) # 用于在提交后清空输入栏
# 提交按钮、重置按钮
cancel_handles.append(txt.submit(**predict_args)) #; txt.submit(**empty_txt_args) 在提交后清空输入栏
cancel_handles.append(submitBtn.click(**predict_args)) #; submitBtn.click(**empty_txt_args) 在提交后清空输入栏
resetBtn.click(lambda: ([], [], "已重置"), None, output_combo)
# 基础功能区的回调函数注册
for k in functional:
functional[k]["Button"].click(predict,
[txt, top_p, temperature, chatbot, history, systemPromptTxt, TRUE, gr.State(k)], [chatbot, history, statusDisplay], show_progress=True)
click_handle = functional[k]["Button"].click(predict, [*input_combo, gr.State(True), gr.State(k)], output_combo)
cancel_handles.append(click_handle)
# 文件上传区,接收文件后与chatbot的互动
file_upload.upload(on_file_uploaded, [file_upload, chatbot, txt], [chatbot, txt])
for k in crazy_functional:
click_handle = crazy_functional[k]["Button"].click(crazy_functional[k]["Function"],
[txt, top_p, temperature, chatbot, history, systemPromptTxt, gr.State(PORT)], [chatbot, history, statusDisplay]
)
try: click_handle.then(on_report_generated, [file_upload, chatbot], [file_upload, chatbot])
except: pass
# 函数插件-固定按钮区
for k in crazy_fns:
if not crazy_fns[k].get("AsButton", True): continue
click_handle = crazy_fns[k]["Button"].click(crazy_fns[k]["Function"], [*input_combo, gr.State(PORT)], output_combo)
click_handle.then(on_report_generated, [file_upload, chatbot], [file_upload, chatbot])
cancel_handles.append(click_handle)
# 函数插件-下拉菜单与随变按钮的互动
def on_dropdown_changed(k):
variant = crazy_fns[k]["Color"] if "Color" in crazy_fns[k] else "secondary"
return {switchy_bt: gr.update(value=k, variant=variant)}
dropdown.select(on_dropdown_changed, [dropdown], [switchy_bt] )
# 随变按钮的回调函数注册
def route(k, *args, **kwargs):
if k in [r"打开插件列表", r"先从插件列表中选择"]: return
yield from crazy_fns[k]["Function"](*args, **kwargs)
click_handle = switchy_bt.click(route,[switchy_bt, *input_combo, gr.State(PORT)], output_combo)
click_handle.then(on_report_generated, [file_upload, chatbot], [file_upload, chatbot])
# def expand_file_area(file_upload, area_file_up):
# if len(file_upload)>0: return {area_file_up: gr.update(open=True)}
# click_handle.then(expand_file_area, [file_upload, area_file_up], [area_file_up])
cancel_handles.append(click_handle)
# 终止按钮的回调函数注册
stopBtn.click(fn=None, inputs=None, outputs=None, cancels=cancel_handles)
# 延迟函数, 做一些准备工作, 最后尝试打开浏览器
# gradio的inbrowser触发不太稳定,回滚代码到原始的浏览器打开函数
def auto_opentab_delay():
import threading, webbrowser, time
print(f"URL http://localhost:{PORT}")
def open(): time.sleep(2)
webbrowser.open_new_tab(f'http://localhost:{PORT}')
t = threading.Thread(target=open)
t.daemon = True; t.start()
print(f"如果浏览器没有自动打开,请复制并转到以下URL: http://localhost:{PORT}")
def open():
time.sleep(2)
webbrowser.open_new_tab(f"http://localhost:{PORT}")
threading.Thread(target=open, name="open-browser", daemon=True).start()
auto_opentab_delay()
demo.title = "ChatGPT 学术优化"
demo.queue().launch(server_name="0.0.0.0", share=True, server_port=PORT)
demo.queue(concurrency_count=CONCURRENT_COUNT).launch(server_name="0.0.0.0", share=True, server_port=PORT, auth=AUTHENTICATION)
+63 -8
View File
@@ -1,5 +1,16 @@
# 借鉴了 https://github.com/GaiZhenbiao/ChuanhuChatGPT 项目
"""
该文件中主要包含三个函数
不具备多线程能力的函数
1. predict: 正常对话时使用具备完备的交互功能不可多线程
具备多线程调用能力的函数
2. predict_no_ui高级实验性功能模块调用不会实时显示在界面上参数简单可以多线程并行方便实现复杂的功能逻辑
3. predict_no_ui_long_connection在实验过程中发现调用predict_no_ui处理长文档时和openai的连接容易断掉这个函数用stream的方式解决这个问题同样支持多线程
"""
import json
import gradio as gr
import logging
@@ -9,10 +20,12 @@ import importlib
# config_private.py放自己的秘密如API和代理网址
# 读取时首先看是否存在私密的config_private配置文件(不受git管控),如果有,则覆盖原config文件
try: from config_private import proxies, API_URL, API_KEY, TIMEOUT_SECONDS, MAX_RETRY, LLM_MODEL
except: from config import proxies, API_URL, API_KEY, TIMEOUT_SECONDS, MAX_RETRY, LLM_MODEL
from toolbox import get_conf
proxies, API_URL, API_KEY, TIMEOUT_SECONDS, MAX_RETRY, LLM_MODEL = \
get_conf('proxies', 'API_URL', 'API_KEY', 'TIMEOUT_SECONDS', 'MAX_RETRY', 'LLM_MODEL')
timeout_bot_msg = '[local] Request timeout, network error. please check proxy settings in config.py.'
timeout_bot_msg = '[Local Message] Request timeout. Network error. Please check proxy settings in config.py.' + \
'网络错误,检查代理服务器是否可用,以及代理设置的格式是否正确,格式须是[协议]://[地址]:[端口],缺一不可。'
def get_full_error(chunk, stream_response):
"""
@@ -25,7 +38,7 @@ def get_full_error(chunk, stream_response):
break
return chunk
def predict_no_ui(inputs, top_p, temperature, history=[]):
def predict_no_ui(inputs, top_p, temperature, history=[], sys_prompt=""):
"""
发送至chatGPT等待回复一次性完成不显示中间过程
predict函数的简化版
@@ -36,7 +49,7 @@ def predict_no_ui(inputs, top_p, temperature, history=[]):
history 是之前的对话列表
注意无论是inputs还是history内容太长了都会触发token数量溢出的错误然后raise ConnectionAbortedError
"""
headers, payload = generate_payload(inputs, top_p, temperature, history, system_prompt="", stream=False)
headers, payload = generate_payload(inputs, top_p, temperature, history, system_prompt=sys_prompt, stream=False)
retry = 0
while True:
@@ -47,8 +60,8 @@ def predict_no_ui(inputs, top_p, temperature, history=[]):
except requests.exceptions.ReadTimeout as e:
retry += 1
traceback.print_exc()
if MAX_RETRY!=0: print(f'请求超时,正在重试 ({retry}/{MAX_RETRY}) ……')
if retry > MAX_RETRY: raise TimeoutError
if MAX_RETRY!=0: print(f'请求超时,正在重试 ({retry}/{MAX_RETRY}) ……')
try:
result = json.loads(response.text)["choices"][0]["message"]["content"]
@@ -58,6 +71,47 @@ def predict_no_ui(inputs, top_p, temperature, history=[]):
raise ConnectionAbortedError("Json解析不合常规,可能是文本过长" + response.text)
def predict_no_ui_long_connection(inputs, top_p, temperature, history=[], sys_prompt=""):
"""
发送至chatGPT等待回复一次性完成不显示中间过程但内部用stream的方法避免有人中途掐网线
"""
headers, payload = generate_payload(inputs, top_p, temperature, history, system_prompt=sys_prompt, stream=True)
retry = 0
while True:
try:
# make a POST request to the API endpoint, stream=False
response = requests.post(API_URL, headers=headers, proxies=proxies,
json=payload, stream=True, timeout=TIMEOUT_SECONDS); break
except requests.exceptions.ReadTimeout as e:
retry += 1
traceback.print_exc()
if retry > MAX_RETRY: raise TimeoutError
if MAX_RETRY!=0: print(f'请求超时,正在重试 ({retry}/{MAX_RETRY}) ……')
stream_response = response.iter_lines()
result = ''
while True:
try: chunk = next(stream_response).decode()
except StopIteration: break
if len(chunk)==0: continue
if not chunk.startswith('data:'):
error_msg = get_full_error(chunk.encode('utf8'), stream_response).decode()
if "reduce the length" in error_msg:
raise ConnectionAbortedError("OpenAI拒绝了请求:" + error_msg)
else:
raise RuntimeError("OpenAI拒绝了请求:" + error_msg)
json_data = json.loads(chunk.lstrip('data:'))['choices'][0]
delta = json_data["delta"]
if len(delta) == 0: break
if "role" in delta: continue
if "content" in delta: result += delta["content"]; print(delta["content"], end='')
else: raise RuntimeError("意外Json结构:"+delta)
if json_data['finish_reason'] == 'length':
raise ConnectionAbortedError("正常结束,但显示Token不足。")
return result
def predict(inputs, top_p, temperature, chatbot=[], history=[], system_prompt='',
stream = True, additional_fn=None):
"""
@@ -71,8 +125,9 @@ def predict(inputs, top_p, temperature, chatbot=[], history=[], system_prompt=''
"""
if additional_fn is not None:
import functional
importlib.reload(functional)
importlib.reload(functional) # 热更新prompt
functional = functional.get_functionals()
if "PreProcess" in functional[additional_fn]: inputs = functional[additional_fn]["PreProcess"](inputs) # 获取预处理函数(如果有的话)
inputs = functional[additional_fn]["Prefix"] + inputs + functional[additional_fn]["Suffix"]
if stream:
@@ -130,7 +185,7 @@ def predict(inputs, top_p, temperature, chatbot=[], history=[], system_prompt=''
chunk = get_full_error(chunk, stream_response)
error_msg = chunk.decode()
if "reduce the length" in error_msg:
chatbot[-1] = (chatbot[-1][0], "[Local Message] Input (or history) is too long, please reduce input or clear history by refleshing this page.")
chatbot[-1] = (chatbot[-1][0], "[Local Message] Input (or history) is too long, please reduce input or clear history by refreshing this page.")
history = []
elif "Incorrect API key" in error_msg:
chatbot[-1] = (chatbot[-1][0], "[Local Message] Incorrect API key provided.")
+175
View File
@@ -0,0 +1,175 @@
# chatgpt-academic项目自译解报告
(Author补充:以下分析均由本项目调用ChatGPT一键生成,如果有不准确的地方,全怪GPT😄)
## [0/18] 程序摘要: functional_crazy.py
这是一个功能扩展的程序,文件名为 `functional_crazy.py`。代码的主要功能是通过提供一系列函数插件,增强程序的功能,让用户可以通过界面中的按钮,快速调用对应的函数插件实现相应的操作。代码中使用了 `HotReload` 函数插件,可以在不重启程序的情况下更新函数插件的代码,让其生效。同时,通过 `UserVisibleLevel` 变量的设置,可以控制哪些插件会在UI界面显示出来。函数插件列表包括了以下功能:解析项目本身、解析一个Python项目、解析一个C++项目头文件、解析一个C++项目、读取文章并生成摘要、批量生成函数注释、全项目切换成英文、批量总结PDF文档、批量总结PDF文档pdfminer、批量总结Word文档、高阶功能模板函数、以及其他未经充分测试的函数插件。
## [1/18] 程序摘要: main.py
该程序是一个基于Gradio构建的对话生成模型的Web界面示例,包含了以下主要功能:
1.加载模型并对用户输入进行响应;
2.通过调用外部函数库来获取用户的输入,并在模型生成的过程中进行处理;
3.支持用户上传本地文件,供外部函数库调用;
4.支持停止当前的生成过程;
5.保存用户的历史记录,并将其记录在本地日志文件中,以供后续分析和使用。
该程序需要依赖于一些外部库和软件包,如Gradio、torch等。用户需要确保这些依赖项已经安装,并且在运行该程序前对config_private.py配置文件进行相应的修改。
## [2/18] 程序摘要: functional.py
该文件定义了一个名为“functional”的函数,函数的作用是返回一个包含多个字典(键值对)的字典,每个键值对表示一种功能。该字典的键值由功能名称和对应的数据组成。其中的每个字典都包含4个键值对,分别为“Prefix”、“Suffix”、“Color”和“PreProcess”,分别表示前缀、后缀、按钮颜色和预处理函数。如果某些键值对没有给出,那么程序中默认相应的值,如按钮颜色默认为“secondary”等。每个功能描述了不同的学术润色/翻译/其他服务,如“英语学术润色”、“中文学术润色”、“查找语法错误”等。函数还引用了一个名为“clear_line_break”的函数,用于预处理修改前的文本。
## [3/18] 程序摘要: show_math.py
该程序文件名为show_math.py,主要用途是将Markdown和LaTeX混合格式转换成带有MathML的HTML格式。该程序通过递归地处理LaTeX和Markdown混合段落逐一转换成HTML/MathML标记出来,并在LaTeX公式创建中进行错误处理。在程序文件中定义了3个变量,分别是incompleteconvError和convert,其中convert函数是用来执行转换的主要函数。程序使用正则表达式进行LaTeX格式和Markdown段落的分割,从而实现转换。如果在Latex转换过程中发生错误,程序将输出相应的错误信息。
## [4/18] 程序摘要: predict.py
本程序文件的文件名为"./predict.py",主要包含三个函数:
1. predict:正常对话时使用,具备完备的交互功能,不可多线程;
2. predict_no_ui:高级实验性功能模块调用,不会实时显示在界面上,参数简单,可以多线程并行,方便实现复杂的功能逻辑;
3. predict_no_ui_long_connection:在实验过程中发现调用predict_no_ui处理长文档时,和openai的连接容易断掉,这个函数用stream的方式解决这个问题,同样支持多线程。
其中,predict函数用于基础的对话功能,发送至chatGPT,流式获取输出,根据点击的哪个按钮,进行对话预处理等额外操作;predict_no_ui函数用于payload比较大的情况,或者用于实现多线、带嵌套的复杂功能;predict_no_ui_long_connection实现调用predict_no_ui处理长文档时,避免连接断掉的情况,支持多线程。
## [5/18] 程序摘要: check_proxy.py
该程序文件名为check_proxy.py,主要功能是检查代理服务器的可用性并返回代理服务器的地理位置信息或错误提示。具体实现方式如下:
首先使用requests模块向指定网站(https://ipapi.co/json/)发送GET请求,请求结果以JSON格式返回。如果代理服务器参数(proxies)是有效的且没有指明'https'代理,则用默认字典值'无'替代。
然后,程序会解析返回的JSON数据,并根据数据中是否包含国家名字字段来判断代理服务器的地理位置。如果有国家名字字段,则将其打印出来并返回代理服务器的相关信息。如果没有国家名字字段,但有错误信息字段,则返回其他错误提示信息。
在程序执行前,程序会先设置环境变量no_proxy,并使用toolbox模块中的get_conf函数从配置文件中读取代理参数。
最后,检测程序会输出检查结果并返回对应的结果字符串。
## [6/18] 程序摘要: config_private.py
本程序文件名为`config_private.py`,其功能为配置私有信息以便在主程序中使用。主要功能包括:
- 配置OpenAI API的密钥和API URL
- 配置是否使用代理,如果使用代理配置代理地址和端口
- 配置发送请求的超时时间和失败重试次数的限制
- 配置并行使用线程数和用户名密码
- 提供检查功能以确保API密钥已经正确设置
其中,需要特别注意的是:最后一个检查功能要求在运行之前必须将API密钥正确设置,否则程序会直接退出。
## [7/18] 程序摘要: config.py
该程序文件是一个配置文件,用于配置OpenAI的API参数和优化体验的相关参数,具体包括以下几个步骤:
1.设置OpenAI的API密钥。
2.选择是否使用代理,如果使用则需要设置代理地址和端口等参数。
3.设置请求OpenAI后的超时时间、网页的端口、重试次数、选择的OpenAI模型、API的网址等。
4.设置并行使用的线程数和用户名密码。
该程序文件的作用为在使用OpenAI API时进行相关参数的配置,以保证请求的正确性和速度,并且优化使用体验。
## [8/18] 程序摘要: theme.py
该程序是一个自定义Gradio主题的Python模块。主题文件名为"./theme.py"。程序引入了Gradio模块,并定义了一个名为"adjust_theme()"的函数。该函数根据输入值调整Gradio的默认主题,返回一个包含所需自定义属性的主题对象。主题属性包括颜色、字体、过渡、阴影、按钮边框和渐变等。主题颜色列表包括石板色、灰色、锌色、中性色、石头色、红色、橙色、琥珀色、黄色、酸橙色、绿色、祖母绿、青蓝色、青色、天蓝色、蓝色、靛蓝色、紫罗兰色、紫色、洋红色、粉红色和玫瑰色。如果Gradio版本较旧,则不能自定义字体和颜色。
## [9/18] 程序摘要: toolbox.py
该程序文件包含了一系列函数,用于实现聊天程序所需的各种功能,如预测对话、将对话记录写入文件、将普通文本转换为Markdown格式文本、装饰器函数CatchException和HotReload等。其中一些函数用到了第三方库,如Python-Markdown、mdtex2html、zipfile、tarfile、rarfile和py7zr。除此之外,还有一些辅助函数,如get_conf、clear_line_break和extract_archive等。主要功能包括:
1. 导入markdown、mdtex2html、threading、functools等模块。
2. 定义函数predict_no_ui_but_counting_down,用于生成对话。
3. 定义函数write_results_to_file,用于将对话记录生成Markdown文件。
4. 定义函数regular_txt_to_markdown,将普通文本转换为Markdown格式的文本。
5. 定义装饰器函数CatchException,用于捕获函数执行异常并返回生成器。
6. 定义函数report_execption,用于向chatbot中添加错误信息。
7. 定义函数text_divide_paragraph,用于将文本按照段落分隔符分割开,生成带有段落标签的HTML代码。
8. 定义函数markdown_convertion,用于将Markdown格式的文本转换为HTML格式。
9. 定义函数format_io,用于将输入和输出解析为HTML格式。
10. 定义函数find_free_port,用于返回当前系统中可用的未使用端口。
11. 定义函数extract_archive,用于解压归档文件。
12. 定义函数find_recent_files,用于查找最近创建的文件。
13. 定义函数on_file_uploaded,用于处理上传文件的操作。
14. 定义函数on_report_generated,用于处理生成报告文件的操作。
## [10/18] 程序摘要: crazy_functions/生成函数注释.py
该程序文件是一个Python脚本,文件名为“生成函数注释.py”,位于“./crazy_functions/”目录下。该程序实现了一个批量生成函数注释的功能,可以对指定文件夹下的所有Python和C++源代码文件中的所有函数进行注释,使用Markdown表格输出注释结果。
该程序引用了predict.py和toolbox.py两个模块,其中predict.py实现了一个基于GPT模型的文本生成功能,用于生成函数注释,而toolbox.py实现了一些工具函数,包括异常处理函数、文本写入函数等。另外,该程序还定义了两个函数,一个是“生成函数注释”函数,用于处理单个文件的注释生成;另一个是“批量生成函数注释”函数,用于批量处理多个文件的注释生成。
## [11/18] 程序摘要: crazy_functions/读文章写摘要.py
这个程序文件是一个名为“读文章写摘要”的函数。该函数的输入包括文章的文本内容、top_p(生成文本时选择最可能的词语的概率阈值)、temperature(控制生成文本的随机性的因子)、对话历史等参数,以及一个聊天机器人和一个系统提示的文本。该函数的主要工作是解析一组.tex文件,然后生成一段学术性语言的中文和英文摘要。在解析过程中,该函数使用一个名为“toolbox”的模块中的辅助函数和一个名为“predict”的模块中的函数来执行GPT-2模型的推理工作,然后将结果返回给聊天机器人。另外,该程序还包括一个名为“fast_debug”的bool型变量,用于调试和测试。
## [12/18] 程序摘要: crazy_functions/代码重写为全英文_多线程.py
该程序文件实现了一个多线程操作,用于将指定目录下的所有 Python 文件中的中文转化为英文,并将转化后的文件存入另一个目录中。具体实现过程如下:
1. 集合目标文件路径并清空历史记录。
2. 循环目标文件,对每个文件启动一个线程进行任务操作。
3. 各个线程同时开始执行任务函数,并在任务完成后将转化后的文件写入指定目录,最终生成一份任务执行报告。
## [13/18] 程序摘要: crazy_functions/高级功能函数模板.py
该程序文件名为高级功能函数模板.py,它包含了一个名为“高阶功能模板函数”的函数,这个函数可以作为开发新功能函数的模板。该函数引用了predict.py和toolbox.py文件中的函数。在该函数内部,它首先清空了历史记录,然后对于今天和今天以后的四天,它问用户历史中哪些事件发生在这些日期,并列举两条事件并发送相关的图片。在向用户询问问题时,使用了GPT进行响应。由于请求GPT需要一定的时间,所以函数会在重新显示状态之前等待一段时间。在每次与用户的互动中,使用yield关键字生成器函数来输出聊天机器人的当前状态,包括聊天消息、历史记录和状态('正常')。最后,程序调用write_results_to_file函数将聊天的结果写入文件,以供后续的评估和分析。
## [14/18] 程序摘要: crazy_functions/总结word文档.py
该程序文件名为总结word文档.py,主要功能是批量总结Word文档。具体实现过程是解析docx格式和doc格式文件,生成文件内容,然后使用自然语言处理工具对文章内容做中英文概述,最后给出建议。该程序需要依赖python-docx和pywin32,如果没有安装,会给出安装建议。
## [15/18] 程序摘要: crazy_functions/批量总结PDF文档pdfminer.py
该程序文件名为pdfminer.py,位于./crazy_functions/目录下。程序实现了批量读取PDF文件,并使用pdfminer解析PDF文件内容。此外,程序还根据解析得到的文本内容,调用机器学习模型生成对每篇文章的概述,最终生成全文摘要。程序中还对模块依赖进行了导入检查,若缺少依赖,则会提供安装建议。
## [16/18] 程序摘要: crazy_functions/解析项目源代码.py
这个程序文件中包含了几个函数,分别是:
1. `解析源代码(file_manifest, project_folder, top_p, temperature, chatbot, history, systemPromptTxt)`:通过输入文件路径列表对程序文件进行逐文件分析,根据分析结果做出整体功能和构架的概括,并生成包括每个文件功能的markdown表格。
2. `解析项目本身(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT)`:对当前文件夹下的所有Python文件及其子文件夹进行逐文件分析,并生成markdown表格。
3. `解析一个Python项目(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT)`:对指定路径下的所有Python文件及其子文件夹进行逐文件分析,并生成markdown表格。
4. `解析一个C项目的头文件(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT)`:对指定路径下的所有头文件进行逐文件分析,并生成markdown表格。
5. `解析一个C项目(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT)`:对指定路径下的所有.h、.cpp、.c文件及其子文件夹进行逐文件分析,并生成markdown表格。
程序中还包含了一些辅助函数和变量,如CatchException装饰器函数,report_execption函数、write_results_to_file函数等。在执行过程中还会调用其他模块中的函数,如toolbox模块的函数和predict模块的函数。
## [17/18] 程序摘要: crazy_functions/批量总结PDF文档.py
这个程序文件是一个名为“批量总结PDF文档”的函数插件。它导入了predict和toolbox模块,并定义了一些函数,包括is_paragraph_breaknormalize_text和clean_text。这些函数是对输入文本进行预处理和清洗的功能函数。主要的功能函数是解析PDF,它打开每个PDF文件并将其内容存储在file_content变量中,然后传递给聊天机器人,以产生一句话的概括。在解析PDF文件之后,该函数连接了所有文件的摘要,以产生一段学术语言和英文摘要。最后,函数批量处理目标文件夹中的所有PDF文件,并输出结果。
## 根据以上你自己的分析,对程序的整体功能和构架做出概括。然后用一张markdown表格整理每个文件的功能。
该程序是一个聊天机器人,使用了OpenAI的GPT语言模型以及一些特殊的辅助功能去处理各种学术写作和科研润色任务。整个程序由一些函数组成,每个函数都代表了不同的学术润色/翻译/其他服务。
下面是程序中每个文件的功能列表:
| 文件名 | 功能 |
|--------|--------|
| functional_crazy.py | 实现高级功能函数模板和其他一些辅助功能函数 |
| main.py | 程序的主要入口,负责程序的启动和UI的展示 |
| functional.py | 定义各种功能按钮的颜色和响应函数 |
| show_math.py | 解析LaTeX文本,将其转换为Markdown格式 |
| predict.py | 基础的对话功能,用于与chatGPT进行交互 |
| check_proxy.py | 检查代理设置的正确性 |
| config_private.py | 配置程序的API密钥和其他私有信息 |
| config.py | 配置OpenAI的API参数和程序的其他属性 |
| theme.py | 设置程序主题样式 |
| toolbox.py | 存放一些辅助函数供程序使用 |
| crazy_functions/生成函数注释.py | 生成Python文件中所有函数的注释 |
| crazy_functions/读文章写摘要.py | 解析文章文本,生成中英文摘要 |
| crazy_functions/代码重写为全英文_多线程.py | 将中文代码内容转化为英文 |
| crazy_functions/高级功能函数模板.py | 实现高级功能函数模板 |
| crazy_functions/总结word文档.py | 解析Word文件,生成文章内容的概要 |
| crazy_functions/批量总结PDF文档pdfminer.py | 解析PDF文件,生成文章内容的概要(使用pdfminer库) |
| crazy_functions/批量总结PDF文档.py | 解析PDF文件,生成文章内容的概要(使用PyMuPDF库) |
| crazy_functions/解析项目源代码.py | 解析C/C++源代码,生成markdown表格 |
| crazy_functions/批量总结PDF文档.py | 对PDF文件进行批量摘要生成 |
总的来说,该程序提供了一系列的学术润色和翻译的工具,支持对各种类型的文件进行分析和处理。同时也提供了对话式用户界面,便于用户使用和交互。
+2
View File
@@ -1,3 +1,5 @@
gradio>=3.23
requests[socks]
mdtex2html
Markdown
latex2mathml
+34 -23
View File
@@ -1,29 +1,28 @@
import gradio as gr
# gradio可用颜色列表
# slate
# gray
# zinc
# neutral
# stone
# red
# orange
# amber
# yellow
# lime
# green
# emerald
# teal
# cyan
# sky
# blue
# indigo
# violet
# purple
# fuchsia
# pink
# rose
# gr.themes.utils.colors.slate (石板色)
# gr.themes.utils.colors.gray (灰色)
# gr.themes.utils.colors.zinc (锌色)
# gr.themes.utils.colors.neutral (中性色)
# gr.themes.utils.colors.stone (石头色)
# gr.themes.utils.colors.red (红色)
# gr.themes.utils.colors.orange (橙色)
# gr.themes.utils.colors.amber (琥珀色)
# gr.themes.utils.colors.yellow (黄色)
# gr.themes.utils.colors.lime (酸橙色)
# gr.themes.utils.colors.green (绿色)
# gr.themes.utils.colors.emerald (祖母绿)
# gr.themes.utils.colors.teal (青蓝色)
# gr.themes.utils.colors.cyan (青色)
# gr.themes.utils.colors.sky (天蓝色)
# gr.themes.utils.colors.blue (蓝色)
# gr.themes.utils.colors.indigo (靛蓝色)
# gr.themes.utils.colors.violet (紫罗兰色)
# gr.themes.utils.colors.purple (紫色)
# gr.themes.utils.colors.fuchsia (洋红色)
# gr.themes.utils.colors.pink (粉红色)
# gr.themes.utils.colors.rose (玫瑰色)
def adjust_theme():
try:
@@ -81,3 +80,15 @@ def adjust_theme():
except:
set_theme = None; print('gradio版本较旧, 不能自定义字体和颜色')
return set_theme
advanced_css = """
.markdown-body table {
border: 1px solid #ddd;
border-collapse: collapse;
}
.markdown-body th, .markdown-body td {
border: 1px solid #ddd;
padding: 5px;
}
"""
+166 -27
View File
@@ -1,15 +1,36 @@
import markdown, mdtex2html, threading
import markdown, mdtex2html, threading, importlib, traceback, importlib, inspect, re
from show_math import convert as convert_math
from functools import wraps
from functools import wraps, lru_cache
def predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature, history=[]):
def get_reduce_token_percent(text):
try:
# text = "maximum context length is 4097 tokens. However, your messages resulted in 4870 tokens"
pattern = r"(\d+)\s+tokens\b"
match = re.findall(pattern, text)
EXCEED_ALLO = 500 # 稍微留一点余地,否则在回复时会因余量太少出问题
max_limit = float(match[0]) - EXCEED_ALLO
current_tokens = float(match[1])
ratio = max_limit/current_tokens
assert ratio > 0 and ratio < 1
return ratio, str(int(current_tokens-max_limit))
except:
return 0.5, '不详'
def predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temperature, history=[], sys_prompt='', long_connection=True):
"""
调用简单的predict_no_ui接口但是依然保留了些许界面心跳功能当对话太长时会自动采用二分法截断
i_say: 当前输入
i_say_show_user: 显示到对话界面上的当前输入例如输入整个文件时你绝对不想把文件的内容都糊到对话界面上
chatbot: 对话界面句柄
top_p, temperature: gpt参数
history: gpt参数 对话历史
sys_prompt: gpt参数 sys_prompt
long_connection: 是否采用更稳定的连接方式推荐
"""
import time
try: from config_private import TIMEOUT_SECONDS, MAX_RETRY
except: from config import TIMEOUT_SECONDS, MAX_RETRY
from predict import predict_no_ui
from predict import predict_no_ui, predict_no_ui_long_connection
from toolbox import get_conf
TIMEOUT_SECONDS, MAX_RETRY = get_conf('TIMEOUT_SECONDS', 'MAX_RETRY')
# 多线程的时候,需要一个mutable结构在不同线程之间传递信息
# list就是最简单的mutable结构,我们第一个位置放gpt输出,第二个位置传递报错信息
mutable = [None, '']
@@ -17,17 +38,25 @@ def predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temp
def mt(i_say, history):
while True:
try:
mutable[0] = predict_no_ui(inputs=i_say, top_p=top_p, temperature=temperature, history=history)
break
except ConnectionAbortedError as e:
if len(history) > 0:
history = [his[len(his)//2:] for his in history if his is not None]
mutable[1] = 'Warning! History conversation is too long, cut into half. '
if long_connection:
mutable[0] = predict_no_ui_long_connection(inputs=i_say, top_p=top_p, temperature=temperature, history=history, sys_prompt=sys_prompt)
else:
i_say = i_say[:len(i_say)//2]
mutable[1] = 'Warning! Input file is too long, cut into half. '
mutable[0] = predict_no_ui(inputs=i_say, top_p=top_p, temperature=temperature, history=history, sys_prompt=sys_prompt)
break
except ConnectionAbortedError as token_exceeded_error:
# 尝试计算比例,尽可能多地保留文本
p_ratio, n_exceed = get_reduce_token_percent(str(token_exceeded_error))
if len(history) > 0:
history = [his[ int(len(his) *p_ratio): ] for his in history if his is not None]
else:
i_say = i_say[: int(len(i_say) *p_ratio) ]
mutable[1] = f'警告,文本过长将进行截断,Token溢出数:{n_exceed},截断比例:{(1-p_ratio):.0%}'
except TimeoutError as e:
mutable[0] = '[Local Message] Failed with timeout'
mutable[0] = '[Local Message] 请求超时。'
raise TimeoutError
except Exception as e:
mutable[0] = f'[Local Message] 异常:{str(e)}.'
raise RuntimeError(f'[Local Message] 异常:{str(e)}.')
# 创建新线程发出http请求
thread_name = threading.Thread(target=mt, args=(i_say, history)); thread_name.start()
# 原来的线程则负责持续更新UI,实现一个超时倒计时,并等待新线程的任务完成
@@ -39,6 +68,7 @@ def predict_no_ui_but_counting_down(i_say, i_say_show_user, chatbot, top_p, temp
time.sleep(1)
# 把gpt的输出从mutable中取出来
gpt_say = mutable[0]
if gpt_say=='[Local Message] Failed with timeout.': raise TimeoutError
return gpt_say
def write_results_to_file(history, file_name=None):
@@ -47,11 +77,16 @@ def write_results_to_file(history, file_name=None):
"""
import os, time
if file_name is None:
file_name = time.strftime("chatGPT分析报告%Y-%m-%d-%H-%M-%S", time.localtime()) + '.md'
# file_name = time.strftime("chatGPT分析报告%Y-%m-%d-%H-%M-%S", time.localtime()) + '.md'
file_name = 'chatGPT分析报告' + time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime()) + '.md'
os.makedirs('./gpt_log/', exist_ok=True)
with open(f'./gpt_log/{file_name}', 'w') as f:
with open(f'./gpt_log/{file_name}', 'w', encoding = 'utf8') as f:
f.write('# chatGPT 分析报告\n')
for i, content in enumerate(history):
try: # 这个bug没找到触发条件,暂时先这样顶一下
if type(content) != str: content = str(content)
except:
continue
if i%2==0: f.write('## ')
f.write(content)
f.write('\n\n')
@@ -77,15 +112,25 @@ def CatchException(f):
try:
yield from f(txt, top_p, temperature, chatbot, history, systemPromptTxt, WEB_PORT)
except Exception as e:
import traceback
from check_proxy import check_proxy
try: from config_private import proxies
except: from config import proxies
from toolbox import get_conf
proxies, = get_conf('proxies')
tb_str = regular_txt_to_markdown(traceback.format_exc())
chatbot[-1] = (chatbot[-1][0], f"[Local Message] 实验性函数调用出错: \n\n {tb_str} \n\n 当前代理可用性: \n\n {check_proxy(proxies)}")
yield chatbot, history, f'异常 {e}'
return decorated
def HotReload(f):
"""
装饰器函数实现函数插件热更新
"""
@wraps(f)
def decorated(*args, **kwargs):
fn_name = f.__name__
f_hot_reload = getattr(importlib.reload(inspect.getmodule(f)), fn_name)
yield from f_hot_reload(*args, **kwargs)
return decorated
def report_execption(chatbot, history, a, b):
"""
向chatbot中添加错误信息
@@ -104,26 +149,27 @@ def text_divide_paragraph(text):
# wtf input
lines = text.split("\n")
for i, line in enumerate(lines):
if i!=0: lines[i] = "<p>"+lines[i].replace(" ", "&nbsp;")+"</p>"
text = "".join(lines)
lines[i] = lines[i].replace(" ", "&nbsp;")
text = "</br>".join(lines)
return text
def markdown_convertion(txt):
"""
将Markdown格式的文本转换为HTML格式如果包含数学公式则先将公式转换为HTML格式
"""
pre = '<div class="markdown-body">'
suf = '</div>'
if ('$' in txt) and ('```' not in txt):
return markdown.markdown(txt,extensions=['fenced_code','tables']) + '<br><br>' + \
markdown.markdown(convert_math(txt, splitParagraphs=False),extensions=['fenced_code','tables'])
return pre + markdown.markdown(txt,extensions=['fenced_code','tables']) + '<br><br>' + markdown.markdown(convert_math(txt, splitParagraphs=False),extensions=['fenced_code','tables']) + suf
else:
return markdown.markdown(txt,extensions=['fenced_code','tables'])
return pre + markdown.markdown(txt,extensions=['fenced_code','tables']) + suf
def format_io(self, y):
"""
将输入和输出解析为HTML格式将y中最后一项的输入部分段落化并将输出部分的Markdown和数学公式转换为HTML格式
"""
if y is None: return []
if y is None or y == []: return []
i_ask, gpt_reply = y[-1]
i_ask = text_divide_paragraph(i_ask) # 输入部分太自由,预处理一波
y[-1] = (
@@ -162,8 +208,32 @@ def extract_archive(file_path, dest_dir):
with tarfile.open(file_path, 'r:*') as tarobj:
tarobj.extractall(path=dest_dir)
print("Successfully extracted tar archive to {}".format(dest_dir))
# 第三方库,需要预先pip install rarfile
# 此外,Windows上还需要安装winrar软件,配置其Path环境变量,如"C:\Program Files\WinRAR"才可以
elif file_extension == '.rar':
try:
import rarfile
with rarfile.RarFile(file_path) as rf:
rf.extractall(path=dest_dir)
print("Successfully extracted rar archive to {}".format(dest_dir))
except:
print("Rar format requires additional dependencies to install")
return '\n\n需要安装pip install rarfile来解压rar文件'
# 第三方库,需要预先pip install py7zr
elif file_extension == '.7z':
try:
import py7zr
with py7zr.SevenZipFile(file_path, mode='r') as f:
f.extractall(path=dest_dir)
print("Successfully extracted 7z archive to {}".format(dest_dir))
except:
print("7z format requires additional dependencies to install")
return '\n\n需要安装pip install py7zr来解压7z文件'
else:
return
return ''
return ''
def find_recent_files(directory):
"""
@@ -181,6 +251,75 @@ def find_recent_files(directory):
if file_path.endswith('.log'): continue
created_time = os.path.getctime(file_path)
if created_time >= one_minute_ago:
if os.path.isdir(file_path): continue
recent_files.append(file_path)
return recent_files
def on_file_uploaded(files, chatbot, txt):
if len(files) == 0: return chatbot, txt
import shutil, os, time, glob
from toolbox import extract_archive
try: shutil.rmtree('./private_upload/')
except: pass
time_tag = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
os.makedirs(f'private_upload/{time_tag}', exist_ok=True)
err_msg = ''
for file in files:
file_origin_name = os.path.basename(file.orig_name)
shutil.copy(file.name, f'private_upload/{time_tag}/{file_origin_name}')
err_msg += extract_archive(f'private_upload/{time_tag}/{file_origin_name}',
dest_dir=f'private_upload/{time_tag}/{file_origin_name}.extract')
moved_files = [fp for fp in glob.glob('private_upload/**/*', recursive=True)]
txt = f'private_upload/{time_tag}'
moved_files_str = '\t\n\n'.join(moved_files)
chatbot.append(['我上传了文件,请查收',
f'[Local Message] 收到以下文件: \n\n{moved_files_str}'+
f'\n\n调用路径参数已自动修正到: \n\n{txt}'+
f'\n\n现在您点击任意实验功能时,以上文件将被作为输入参数'+err_msg])
return chatbot, txt
def on_report_generated(files, chatbot):
from toolbox import find_recent_files
report_files = find_recent_files('gpt_log')
if len(report_files) == 0: return report_files, chatbot
# files.extend(report_files)
chatbot.append(['汇总报告如何远程获取?', '汇总报告已经添加到右侧“文件上传区”(可能处于折叠状态),请查收。'])
return report_files, chatbot
@lru_cache(maxsize=128)
def read_single_conf_with_lru_cache(arg):
try: r = getattr(importlib.import_module('config_private'), arg)
except: r = getattr(importlib.import_module('config'), arg)
# 在读取API_KEY时,检查一下是不是忘了改config
if arg=='API_KEY':
# 正确的 API_KEY 是 "sk-" + 48 位大小写字母数字的组合
API_MATCH = re.match(r"sk-[a-zA-Z0-9]{48}$", r)
if API_MATCH:
print(f"[API_KEY] 您的 API_KEY 是: {r[:15]}*** API_KEY 导入成功")
else:
assert False, "正确的 API_KEY 是 'sk-' + '48 位大小写字母数字' 的组合,请在config文件中修改API密钥, 添加海外代理之后再运行。" + \
"(如果您刚更新过代码,请确保旧版config_private文件中没有遗留任何新增键值)"
if arg=='proxies':
if r is None:
print('[PROXY] 网络代理状态:未配置。无代理状态下很可能无法访问。建议:检查USE_PROXY选项是否修改。')
else:
print('[PROXY] 网络代理状态:已配置。配置信息如下:', r)
assert isinstance(r, dict), 'proxies格式错误,请注意proxies选项的格式,不要遗漏括号。'
return r
def get_conf(*args):
# 建议您复制一个config_private.py放自己的秘密, 如API和代理网址, 避免不小心传github被别人看到
res = []
for arg in args:
r = read_single_conf_with_lru_cache(arg)
res.append(r)
return res
def clear_line_break(txt):
txt = txt.replace('\n', ' ')
txt = txt.replace(' ', ' ')
txt = txt.replace(' ', ' ')
return txt