Compare commits

...

190 Commits

Author SHA1 Message Date
968a616595 [Version] 更新版本(2.5.2-20200618.1-SNAPSHOT -> 2.5.2-20200630.1-SNAPSHOT); 2020-06-30 01:23:04 +08:00
6443ba68ab [Fix] AutoCleanTimer 修复自动清除定时器没有重复工作;
[Fix] LocalHashCacheStore 修复clean方法抛出'ConcurrentModificationException'异常的问题;
[Change] CacheStoreCentral 调整PreLoadData的热点缓存时间;
[Change] HotDataCacheStore 将最近获取的Key重置其过期时间, 调整clean过程;
2020-06-30 01:21:17 +08:00
cbe0a38f59 Merge branch 'master' of github.com:LamGC/ContentGrabbingJi 2020-06-29 23:33:41 +08:00
eadfacb7d0 [Clear] PixivSearchLinkBuilder 清理代码; 2020-06-23 13:53:54 +08:00
572189d906 [Change] Main 增加'buildPassword'命令用于构造加密后的密码, 移除对PixivLoginProxyServer的嵌入支持;
[Delete] pom.xml 移除依赖项'net.lamgc:PixivLoginProxyServer';
2020-06-23 10:02:47 +08:00
811694587d [Version] 更新版本(2.5.2-20200617.1-SNAPSHOT -> 2.5.2-20200618.1-SNAPSHOT); 2020-06-18 19:27:11 +08:00
419f2de055 [Add] BotEventHandler 设计在消息正在发送,被时限线程池中断时, 重新投递事件以重新发送消息; 2020-06-18 10:11:08 +08:00
368c78e171 [Add] CacheStoreCentral, BotCommandProcess Search命令增加对pageIndex参数使用(之前竟然是个摆设!?); 2020-06-18 10:05:34 +08:00
bccf47db6e [Change] MessageEvent toString方法增加hashCode输出; 2020-06-18 09:35:37 +08:00
cf351074cc [Delete] BufferMessageEvent 移除缓冲消息事件;
[Add] BufferedMessageSender 增加缓冲消息发送器, 用于替代BufferMessageEvent;
[Change] BotCommandProcess 将Random命令中所使用的BufferMessageEvent替换成BufferedMessageSender;
2020-06-18 09:35:25 +08:00
c4ce18d37a [Fix] log4j2.xml 修复HttpClient日志输出过多的问题;
[Change] PixivURL 将PixivURL设计为不可继承且不可实例化;
2020-06-18 09:01:16 +08:00
26fd18917d [Delete] PixivAccessProxyServer 移除弃用的类;
[Change] Locker 删除不必要的代码;
[Change] PixivDownload 删除多余的Javadoc内容;
[Change] TimeLimitThreadPoolExecutorTest 调整日志输出方向, 补充测试细节;
2020-06-17 19:49:36 +08:00
acbd990181 [Version] 更新版本(2.5.2-20200611.1-SNAPSHOT -> 2.5.2-20200617.1-SNAPSHOT); 2020-06-17 16:48:47 +08:00
6db9cda08a Merge branch 'optimize-memory-cache' 2020-06-17 16:47:08 +08:00
1c742bfb6f [Fix] TimeLimitThreadPoolExecutor 修复了超时提醒遭到时限线程池中断的问题, 调整线程池仅发起一次中断; 2020-06-17 16:14:32 +08:00
2f30fe1696 [Change] CacheStoreCentral 调整PreLoadData的本地缓存时间, 添加注意事项; 2020-06-17 10:51:06 +08:00
32db952e63 Merge branch 'master' into optimize-memory-cache 2020-06-15 17:42:51 +08:00
bd6b825704 [Fix] pom.xml 排除slf4j-log4j12依赖项以修复package后应用启动报Slf4j实现冲突的问题; 2020-06-15 17:42:31 +08:00
3943963505 [Fix] BotEventHandler 修复线程池最大线程数与最低线程数错误, 导致无法启动应用的问题; 2020-06-15 17:40:23 +08:00
08822f68eb [Change] Locker 增加清理日志;
[Add] LockerMapTest 增加单元测试;
2020-06-15 17:18:28 +08:00
2f647ee9fa [Add] LockerMap, Locker 增加Locker锁对象和LockerMap锁对象存储;
[Change] CacheStoreCentral 将synchronized所使用的的锁对象由String(常量池)转换成Locker<K>以尝试减少内存占用;
2020-06-15 16:22:37 +08:00
44a7f49510 [Change] BotGlobal 补充IllegalStateException错误信息; 2020-06-15 15:48:07 +08:00
85088e1b2c [Change] RandomIntervalSendTimer 调整日志输出内容; 2020-06-13 23:10:01 +08:00
8dfb858b9f [Change] RedisPoolCacheStore 优化内部方法实现; 2020-06-13 17:24:06 +08:00
0e76cebc31 [Fix] PreLoadDataAttributeComparator 修复获取PreLoadData抛出的异常信息未记录到日志的问题; 2020-06-13 16:51:15 +08:00
c64320ad78 [Change] HttpRequestException 统一接口异常, 调整类所在包路径; 2020-06-13 16:44:52 +08:00
91e065f657 [Change] ImageCacheStore 优化中断处理, 自动取消任务, 补充TaskState设置; 2020-06-13 12:54:17 +08:00
91f8b0070f [Change] PreLoadDataAttributeComparator 更改类名; 2020-06-13 00:28:52 +08:00
ad54dbfbf3 [Change] PreLoadDataComparator, PreLoadDataAttribute 将PreLoadData的属性Enum迁移到单独的类;
[Change] BotCommandProcess 适配更改;
2020-06-12 22:28:14 +08:00
ebb3dea99e [Change] ConsoleMessageEvent, ConsoleMessageSender 将消息传递给Sender发送, 设置消息前缀格式;
[Change] ConsoleMain 调整私聊模式下的前缀;
[Change] MessageSource 规范化命名;
2020-06-12 20:13:36 +08:00
abcd26f21b [Fix] BotEventHandler 修复事件处理线程非预期设置的问题;
[Change] BotEventHandler 设置线程超时时间;
[Fix] CacheStoreCentral 整理'InterruptedException'在'getImageById'的传递路径;
2020-06-12 20:00:37 +08:00
d9b08f8ad9 [Change] BotCommandProcess, CacheStoreCentral 调整个别日志的输出级别; 2020-06-12 19:28:59 +08:00
5e030c12b2 [Change] BotCommandProcess 将各命令中'page'参数更名为'p'以简化命令;
[Change] PixivSearchLinkBuilder 更改类名(PixivSearchBuilder -> PixivSearchLinkBuilder);
[Add] PixivURL 增加接口常量;
[Change] Main, CacheStoreCentral, PixivSearchLinkBuilderTest 适配PixivSearchLinkBuilder的更改;
2020-06-12 17:06:04 +08:00
62eabce8f6 [Clear] PixivDownload, RandomIntervalSendTimer 整理代码; 2020-06-12 10:31:16 +08:00
64bca3c8f7 [Change] RandomIntervalSendTimer 更改Timer线程名; 2020-06-12 10:20:56 +08:00
3b3f97e638 [Fix] CacheStoreCentral 修复Search命令中'option'参数区分大小写的问题;
[Fix] BotCommandProcess 修复Search命令中遇到不存在作品时会中断处理的问题;
[Change] BotCommandProcess 调整Ranking命令的'type'参数默认值(ILLUST -> ALL);
2020-06-12 10:12:29 +08:00
951824cbe2 [Version] 更新版本(2.5.2-20200610.4-SNAPSHOT -> 2.5.2-20200611.1-SNAPSHOT); 2020-06-11 16:47:35 +08:00
e104abedeb [Add] MiraiMain 增加原始消息事件日志信息; 2020-06-11 16:46:32 +08:00
c3967d214d [Change] Dockerfile.sample 提升镜像使用的Jdk版本(8-jre -> 14 jdk), 增加Arthas诊断工具, 优化构建指令顺序; 2020-06-11 10:57:40 +08:00
26e377a2c7 [Change] log4j2.xml 调整日志输出到文件的限制;
[Change] 将内容较长的日志设为TRACE级别, 以减少日志占用;
[CLear] 整理代码;
2020-06-11 09:49:19 +08:00
87f2535b48 [Change] BotCommandProcess 优化Tag过滤表达式;
[Change] ImageCacheHandler 调整日志输出级别;
2020-06-11 09:08:58 +08:00
97d06c4fc3 [Change] CacheStoreCentral 调整代码以为后续更改做准备;
[Change] BotCommandProcess, ImageCacheHandler, PreLoadDataComparator, RandomRankingArtworksSender 适配CacheStoreCentral的更改;
2020-06-10 20:21:23 +08:00
bfe25c2012 [Version] 更新版本(2.5.2-20200610.3-SNAPSHOT -> 2.5.2-20200610.4-SNAPSHOT); 2020-06-10 16:39:01 +08:00
036f3eaf4a [Delete] simple.properties 删除弃用的Properties模板文件; 2020-06-10 16:27:13 +08:00
f07c8d0b76 [Add] MiraiMain 增加Bot网络重连相关设置; 2020-06-10 16:26:20 +08:00
d0cb4417b1 [Version] 更新版本(2.5.2-20200610.2-SNAPSHOT -> 2.5.2-20200610.3-SNAPSHOT); 2020-06-10 15:47:30 +08:00
3f256c5a0a [Fix] log4j2.xml 修复日志配置错误的问题;
[Change] BotGlobal, Main 调整日志输出内容和级别;
2020-06-10 15:46:33 +08:00
b9180b9651 [Change] MiraiMain 显性设置DeviceInfo为'randomDeviceInfo'; 2020-06-10 15:26:26 +08:00
7ccb306ca1 Merge pull request #15 from LamGC/change-license-AGPLv3
更改开源许可证(LGPLv3 -> AGPLv3)
2020-06-10 14:57:05 +08:00
5dd13ce088 [Change] pom.xml 增加'licenses'; 2020-06-10 14:49:40 +08:00
e9b33938fa [Add] README.md 增加'LICENSE'部分; 2020-06-10 14:41:12 +08:00
cc040e6ec9 [Change] 更改开源许可证; 2020-06-10 10:44:53 +08:00
04724168aa [Version] 更新版本(2.5.2-20200610.1-SNAPSHOT -> 2.5.2-20200610.2-SNAPSHOT); 2020-06-10 10:20:20 +08:00
89ded96336 [Change] 更改部分组件所使用的的工具类; 2020-06-10 10:17:05 +08:00
af0346e952 [Delete] net.sourceforge.htmlunit:htmlunit 删除未使用的依赖项;
[Delete] com.github.rholder:guava-retrying 删除未使用的依赖项;
[Delete] mysql:mysql-connector-java 删除未使用的依赖项;
2020-06-10 09:39:10 +08:00
4ccf2fafbc [Version] 更新版本(2.5.2-20200609.2-SNAPSHOT -> 2.5.2-20200610.1-SNAPSHOT); 2020-06-10 09:21:27 +08:00
f1e58d72ac [Change] log4j2.xml 调整mirai日志在STANDARD_STD*的输出级别; 2020-06-10 09:20:35 +08:00
9242a1d474 [Change] MiraiToSlf4jLoggerAdapter 调整类命名(MiraiToSlf4jLogger -> MiraiToSlf4jLoggerAdapter);
[Change] MiraiMain 适配调整;
2020-06-10 09:13:12 +08:00
065d21c4e4 [Update] MiraiToSlf4jLogger 补充类Javadoc; 2020-06-10 09:10:17 +08:00
5eab94c429 [Change] Console* 调整包路径; 2020-06-10 09:09:54 +08:00
2dd62bb6c8 [Version] 更新版本(2.5.2-20200609.1-SNAPSHOT -> 2.5.2-20200609.2-SNAPSHOT); 2020-06-09 16:02:03 +08:00
21613fe3c0 [Update] net.mamoe:mirai-core 更新依赖项版本(1.0-RC2-1 -> 1.0.2);
[Update] net.mamoe:mirai-core-qqandroid 更新依赖项版本(1.0-RC2-1 -> 1.0.2);
2020-06-09 15:51:33 +08:00
ca56b2c9ba [Change] MiraiToSlf4jLogger, log4j2.xml, log4j2-test.xml, MiraiMain 将Mirai框架的日志接入Slf4j(Log4j2);
[Change] BotEventHandler 调整日志使用;
2020-06-09 15:35:43 +08:00
e0f773639f [Change] ConsoleMain 调整Terminal获取方式; 2020-06-09 14:16:26 +08:00
75aa78a3d7 [Version] 更新版本(2.5.2-20200606.1-SNAPSHOT -> 2.5.2-20200609.1-SNAPSHOT); 2020-06-09 10:12:02 +08:00
18a8ad95a1 [Change] PixivUgoiraBuilderTest 调整测试细节; 2020-06-09 10:07:54 +08:00
438d0a95d3 [Add] pom.xml 增加JLine库;
[Update] ConsoleMain 增强Cli使用体验;
[Add] BotEventHandler 增加同步执行命令的方法'executeMessageEvent(MessageEvent, boolean)';
[Change] Main 适配ConsoleMain的更改;
2020-06-09 09:50:16 +08:00
6789b5b7c5 [Change] CacheStoreCentral 调整'getImageById'方法中对'pageIndex'的参数值检查时机;
[Change] MiraiMessageSender 增加警告忽略注释;
2020-06-08 19:13:51 +08:00
ad289f952f [Fix] log4j2.xml 修复SYSTEM_OUT级别限定错误的问题; 2020-06-08 18:47:53 +08:00
3ae0e4cd8d [Change] HotDataCacheStore 调整日志输出级别(DEBUG -> TRACE); 2020-06-08 16:08:13 +08:00
5550c7aef1 [Clear] CacheStoreCentral 删除无用的运行器代码; 2020-06-08 15:57:38 +08:00
d4d3432c76 [Add] PixivUgoiraBuilder 增加方法'buildUgoira(OutputStream, boolean)'以提供输出流给Builder输出已构建的动图数据;
[Add] PixivUgoiraBuilder 增加方法'getUgoiraMeta():JsonObject'方法, 可通过该方法获取动图元数据;
2020-06-08 15:30:22 +08:00
d1aeda012e [Fix] BotCommandProcess 修复因'getImageById'迁移导致'image'命令不可用的问题;
[Clear] CacheStoreCentral 清除无用代码;
2020-06-08 09:37:06 +08:00
683a38bc17 [Add] BotGlobal BotGlobal在初始化时将检查Redis连通性;
[Change] RandomRankingArtworksSender 支持外部设置groupId, 以使用群组配置;
[Change] BotCommandProcess, BotAdminCommandProcess 适配RandomRankingArtworksSender更改;
2020-06-08 09:24:17 +08:00
188309509b [Change] BotCommandProcess 简化Search参数'ContentOption'的名称('contentOption' -> 'option'); 2020-06-06 19:27:55 +08:00
3915712337 [Delete] search.txt 删除旧文档; 2020-06-06 17:57:24 +08:00
1e88ba70dd [Version] 更新版本(2.5.2-20200604.3-SNAPSHOT -> 2.5.2-20200606.1-SNAPSHOT); 2020-06-06 17:54:09 +08:00
e6b2544998 [Update] 更新接口文档; 2020-06-06 17:53:37 +08:00
a2f6f1d140 [Clear] BotCommandProcess 整理代码; 2020-06-06 17:07:40 +08:00
f54ed35a09 [Change] SettingProperties 增加群号检查, 防止出现非法群号; 2020-06-06 15:42:35 +08:00
a426f80ec5 [Fix] BotCommandProcess 修复Search命令对作品限制的管理不受'image.allowR18'选项控制的问题; 2020-06-06 11:50:29 +08:00
c1a21d1065 [Change] ConsoleMain, ConsoleMessageEvent 支持启动时设置会话群组Id和QQId; 2020-06-06 10:52:12 +08:00
e570ddbb53 [Change] PreLoadDataComparator 增加对JsonElement是否为JsonObject的检查; 2020-06-06 10:51:24 +08:00
223d78dbd6 [Change #5] 优化'getImageById'从文件读入缓存时, 检查图片完整性的速度; 2020-06-05 17:03:41 +08:00
ef5651be47 [Fix] CacheStore 修正Javadoc中的错误描述; 2020-06-05 16:29:16 +08:00
9a8aac1960 [Fix] BotGlobal 修复BotGlobal初始化失败的问题;
[Fix] BotCommandProcess 修复Search命令未找到相关作品时提示语无法触发的问题;
[Change] BotCommandProcess 调整'isNoSafe(int, Properties, boolean)'方法中对于作品限制的判断方式;
2020-06-05 16:23:13 +08:00
4bbed5fd55 [Change] BotCommandProcess, CacheStoreCentral 将ImageFile缓存管理转移到CacheStoreCentral;
[Change] RandomRankingArtworksSender 适配修改;
[Change] BotCommandProcess, BotGlobal 将'imageStoreDir'由BotCommandProcess转移到BotGlobal;
2020-06-05 10:07:13 +08:00
bcc21149b9 [Change #10] 将缓存存取部分从BotCommandProcess分离;
[Change] 整理代码;
2020-06-05 09:53:30 +08:00
e93c322c02 [Change] BotGlobal, BotCommandProcess 调整PixivDownload的初始化过程;
[Change] BotCommandProcess 将'Search'所属缓存部分抽出到单独的方法('getSearchBody');
2020-06-04 20:14:57 +08:00
feb51b8534 [Change] BotGlobal, BotCommandProcess 将'Gson'和'PixivDownload'对象纳入BotGlobal; 2020-06-04 19:47:20 +08:00
273dbd45b0 [Version] 更新版本(2.5.2-20200604.2-SNAPSHOT -> 2.5.2-20200604.3-SNAPSHOT); 2020-06-04 16:16:53 +08:00
b39a82b936 [Fix] BotGlobal 修复BotGlobal初始化失败的问题; 2020-06-04 16:15:27 +08:00
4589624b9b [Version] 更新版本(2.5.2-20200604.1-SNAPSHOT -> 2.5.2-20200604.2-SNAPSHOT); 2020-06-04 15:58:56 +08:00
033fd0188b [Change] 调整Logger创建方式(getLogger(String) -> getLogger(Class)); 2020-06-04 15:58:10 +08:00
4318869846 [Change] 将'Proxy', 'CookieStore'转移到BotGlobal类; 2020-06-04 15:54:52 +08:00
749b89e668 [Version] 更新版本(2.5.2-20200522.1-SNAPSHOT -> 2.5.2-20200604.1-SNAPSHOT); 2020-06-04 11:26:26 +08:00
9ce18469e6 [Add] 增加控制台框架以进行本地测试, 或非聊天机器人使用; 2020-06-04 11:25:08 +08:00
e803d2161b [Add] PropertiesUtils 增加一个工具类; 2020-06-04 11:24:40 +08:00
3fbf80f233 Merge remote-tracking branch 'origin/master' 2020-06-04 11:20:22 +08:00
8b8ef7e744 [Change] Issue #12 调整框架启动机器人应用的方式, 增加一个用于机器人应用内部共享数据的类;
[Fix] BotEventHandler, ImageCacheStore 增加ShutdownHook用于关闭线程池, 解决线程池阻塞关闭过程的问题;
[Change] BotEventHandler 调整'match(String)'方法;
[Change] BotAdminCommandProcess 调整'savePushList()'方法对文件创建失败的行为;
2020-06-04 11:19:18 +08:00
7fbdf28ec8 [Update] Bug_Report.md 增加默认标签; 2020-06-04 10:50:37 +08:00
b29482927c [Add] Bug_Report.md 添加Bug反馈模板; 2020-06-04 10:47:28 +08:00
69da2b02ac [Fix #11] 修复在图片缓存失效的情况下, 'getImageToBotCode'依然会尝试从缓存获取图片File对象导致NPE;
[Change] BotCommandProcess 调整字符串拼接形式, 统一错误提示语的格式;
2020-06-04 09:29:53 +08:00
eb2de09859 [Change] RandomIntervalSendTimer 将随机间隔发送器的Timer设为daemon thread; 2020-06-03 19:37:45 +08:00
ef70ac77cb [Update] HotDataCacheStore, Cleanable 更新Javadoc; 2020-06-03 17:03:54 +08:00
b4e9fdab7d [Change] TimeLimitThreadPoolExecutor 整理代码, 清除无用参数;
[Update] Pixiv排行榜接口.md 更新文档内容, 补充返回数据信息;
2020-06-03 17:02:46 +08:00
5c5827123d [Update] net.lamgc:java-utils 更新依赖项版本(1.2.0_20200514.1-SNAPSHOT -> 1.2.0_20200517.1-SNAPSHOT), 以修复作品无效检测不生效的问题; 2020-06-03 16:51:26 +08:00
2bfb78304c [Change] HotDataCacheStore, RandomIntervalSendTimer 支持浮动时间参数为0; 2020-06-03 14:50:45 +08:00
637ea46b9a [Fix] MiraiMessageEvent 修复群消息事件错误设置私聊Sender的问题; 2020-06-01 21:02:46 +08:00
f5140a7a1e [Change] SpringCQMessageSenderFactory 尝试通过ThreadLocal获取当前处理的CoolQ对象, 目前EventExecutor暂不支持, 仅预留;
[Change] CQPluginMain 调整'processMessage'方法修饰符, 补充Javadoc;
2020-06-01 20:15:59 +08:00
0fafddc715 [Change] AdultContentDetector, PixivURL 整理代码, 补充Javadoc; 2020-06-01 16:12:33 +08:00
4c03a0f7d9 [Change] HotDataCacheStore 调整'supportedList'的条件, HotDataCacheStore将支持使用ListCacheStore;
[Change] AutoSender 将AutoSender的messageSender增加final属性;
2020-06-01 16:01:17 +08:00
ee02072b2d [Deprecated] PixivURL 将旧版Search接口设为弃用; 2020-06-01 15:58:54 +08:00
3f6c87da3c [Add] MiraiMain 增加对临时消息的支持;
[Change] MiraiMessageEvent 调整消息对象转换的方式, 通过静态方法转换Mirai的MessageEvent;
2020-06-01 15:55:31 +08:00
d45cd8aab5 [Update] 更新文档, 增加排行榜接口文档, 待补充返回数据; 2020-05-28 16:20:16 +08:00
598d6ef205 [Fix] README.md 修复格式错误;
[Update] README.md 补充参数细节;
2020-05-27 19:30:04 +08:00
2190ef1b59 [Change] README.md 调整Readme内容以减少对用户选择带来的影响; 2020-05-27 19:27:29 +08:00
fcc522c057 Merge branch 'master' of github.com:LamGC/ContentGrabbingJi 2020-05-26 14:45:01 +08:00
5a81604d94 [Change] BotCommandProcess 调整'色图'命令处理方法的命名; 2020-05-26 14:31:12 +08:00
376306e404 [Update] README.md 更新文档说明, 提供详细命令说明和注意事项; 2020-05-26 10:37:40 +08:00
0fe69253ed [Change] .gitignore 将Idea项目文件由指定文件名改为匹配后缀名; 2020-05-26 10:28:46 +08:00
fe8bf16d3a [Clean] RandomRankingArtworksSender 清理无用导入;
[Change] AutoCleanTimer 统一LoggerName名称;
2020-05-25 20:23:14 +08:00
99c66babec [Add] BotEventHandler 增加对'Sender.sendMessage(String)'返回值的处理; 2020-05-25 09:45:09 +08:00
cb8b01fd74 Merge branch 'master' of github.com:LamGC/ContentGrabbingJi 2020-05-25 09:25:49 +08:00
1f48b3ffdc [Change] MessageSender, MessageEvent 允许'senderMessage(String)'抛出异常;
[Change] RandomRankingArtworksSender 调整异常捕获日志输出;
[Change] MiraiMessageEvent, MiraiMessageSender, SpringCQMessageEvent 适配更改;
2020-05-25 09:21:46 +08:00
68feed8d3c Merge remote-tracking branch 'origin/master' 2020-05-22 23:09:39 +08:00
d89711b0b9 [Version] 更新版本(2.5.2-20200520.1-SNAPSHOT -> 2.5.2-20200522.1-SNAPSHOT); 2020-05-22 20:42:29 +08:00
a3376e96ee [Add] BotCommandProcess, BotEventHandler, PixivDownload 对作品Id不存在的情况做出反应(而不是作为内部异常反馈); 2020-05-22 20:39:49 +08:00
33d18cef6b Merge remote-tracking branch 'origin/master' 2020-05-22 20:34:32 +08:00
7aa00ff98b [Fix] BotCommandProcess 修复ImageStore不允许使用软链接的问题; 2020-05-22 20:34:15 +08:00
e956e36584 [Fix] RankingUpdateTimer 修复首次执行时间计算错误的问题; 2020-05-22 20:23:46 +08:00
8e27221457 [Change] BufferMessageEvent 调整成员变量修饰符, 补充新的构造方法'<init>(int, int, String)'; 2020-05-22 08:49:48 +08:00
a5fca68ef5 [Change] MessageEventExcutionDebugger 在配置文件无法转换的情况下不再忽略'NumberFormatException'异常; 2020-05-20 15:31:43 +08:00
8edb728fe9 [Update] README.md 补充Readme内容 2020-05-20 14:34:55 +08:00
cc05af8a24 Create README.md 2020-05-20 11:00:25 +08:00
60fa641962 [Version] 更新版本(2.5.2-20200517.1-SNAPSHOT -> 2.5.2-20200520.1-SNAPSHOT); 2020-05-20 10:36:30 +08:00
3ab373cc70 [Update] .gitignore 补充VSCode忽略项; 2020-05-20 10:34:55 +08:00
461cd246d8 [Add] BotCommandProcess 新增'色图'功能以随机从排行榜中获取一张图片;
[Add] BufferMessageEvent 增加用于获取Sender发送内容的Event实现;
[Change] BotEventHandler Handler现在允许命令处理方法不返回消息(返回'null');
2020-05-20 09:55:50 +08:00
f040f130d7 [Update] Main, AutoArtworksSender, ImageCacheStore, PixivAccessProxyServer, PagesQualityParser 整理代码问题, 删除无用类; 2020-05-20 09:05:50 +08:00
1f402fbbac [Change] LocalHashCacheStore 实现'Cleanable'接口, 增加'<init>(int, int, boolean)'构造函数以允许自动清理
[Update] LocalHashCacheStore 补充Javadoc内容;
[Update] .gitignore 补充并整理忽略项;
2020-05-20 08:56:27 +08:00
f7f3c3beaf [Fix] BotCommandProcess 修复help命令中帮助信息的错误; 2020-05-17 19:38:17 +08:00
04c1753c22 [Version] 更新版本(2.5.1 -> 2.5.2-20200517.1-SNAPSHOT); 2020-05-17 17:45:06 +08:00
d993e9d719 [Add] MiraiMessageSender 增加对默认表情的支持; 2020-05-12 15:25:29 +08:00
60e91987d1 Merge branch 'mirai' 2020-05-12 15:14:58 +08:00
65392fc2fe [Fix] net.mamoe:mirai-core, net.mamoe:mirai-core-qqandroid 更新Mirai-Core版本号以修复VerifyError问题(1.0-RC2 -> 1.0-RC2-1);
[Fix] MiraiMessageSender 调整日志输出, 以修复日志打印处理后消息顺序混乱的问题;
2020-05-12 15:14:28 +08:00
d38934a0f4 [Change] RankingUpdateTimer 调整时间补齐的阀值(12:00 -> 11:30);
[Add] documents/interfaces/* 整理部分Pixiv接口的文档;
2020-05-12 15:03:08 +08:00
7943357d96 [Fix] MiraiMain 修复未登录时异常退出, 抛出NPE的问题; 2020-05-11 12:16:35 +08:00
a170ff040b [Update] net.mamoe:mirai-core, net.mamoe:mirai-core-qqandroid 更新依赖项版本(0.39.4 -> 1.0-RC2);
[Update] MiraiMain, MiraiMessageEvent 适配mirai新版本;
[Change] MiraiMessageSenderFactory 增加参数检查并优化过程;
[Change] MiraiMain 显性指定Mirai所用协议为'BotConfiguration.MiraiProtocol.ANDROID_PAD';
2020-05-11 11:36:37 +08:00
cb2ebfdb73 [Change] MessageEvent 对消息内容删除首尾空格以提高用户体验性; 2020-05-09 23:13:53 +08:00
66b22c543a [Fix] SettingProperties 修复了配置文件无法创建, 导致无法保存配置的问题;
[Fix] Main 修复了潜在的文件创建失败的问题;
2020-05-09 22:54:32 +08:00
6bace4b048 [Version] 更新版本(2.5.0 -> 2.5.1); 2020-05-09 22:35:46 +08:00
597aac4e95 [Fix] BotCommandProcess 修复部分指令无法使用群组配置的问题; 2020-05-09 22:35:21 +08:00
181d60285b [Version] 更新版本(2.5.0-20200509.1-SNAPSHOT -> 2.5.0); 2020-05-09 22:20:07 +08:00
c06b8717c3 [Add] CacheStoreUtils 增加用于缓存库的工具类;
[Update] BotCommandProcess 补充注释, 优化"info"命令输出内容的格式;
2020-05-09 22:19:29 +08:00
b4a28b6735 [Version] 更新版本(2.5.0-20200506.1-SNAPSHOT -> 2.5.0-20200509.1-SNAPSHOT); 2020-05-09 18:16:54 +08:00
40057c3683 [Fix] Issue #1 修复了ImageCache对重复缓存请求的策略(忽略并返回)导致的后续处理发生NPE的问题;
[Update] ImageCache 更新了ImageCache, 将该部分单独分离到 ImageCacheStore;
[Update] BotCommandProcess 适配新的ImageCache, 调整内容检查的时机以减少不必要的API访问;
2020-05-09 18:16:16 +08:00
4beb4d78fb [Fix] BotCommandProcess 修复Search命令格式不正确的问题; 2020-05-09 12:52:15 +08:00
d9edaa681f [Add] BotAdminCommandProcess 为 addPushGroup 增加参数检查; 2020-05-09 10:10:48 +08:00
7b52abde60 [Change] BotCommandProcess 增加Search命令的反馈内容(可能必须要QQ支持长文本才能使用); 2020-05-08 23:31:24 +08:00
2cb4fe1dbc [Fix] SettingProperties 修复无法设置全局配置项的问题; 2020-05-08 12:23:43 +08:00
5830327dad [Add] BotEventHandler 增加禁言记录, 防止应用因过多的在禁言状态反馈导致被封号;
[Update] MiraiMain 适配禁言记录功能;
2020-05-07 19:46:54 +08:00
cbd10ff281 [Fix] Issue #4 修复因Mirai API变动导致应用无法从消息获取ImageId的问题; 2020-05-07 18:24:09 +08:00
84b8006f89 [Change] RankingUpdateTimer 将定时更新时间调整为"+8 11:30", 并增加日期指定; 2020-05-07 15:21:05 +08:00
d32d891ad7 [Update] BotCommandProcess 补充Javadoc内容;
[Add] BotCommandProcess 在Ranking命令增加force参数以允许强制查询预计不在排行榜范围的日期;
2020-05-07 11:29:31 +08:00
aabd35ce71 [Add] AutoCleanTimer, Cleanable 增加缓存库自动清理;
[Update] HotDataCacheStore 增加对自动清理的支持;
[Change] RedisPoolCacheStore 将Redis缓存库抽象类设为default(原public);
[Change] MiraiMessageSender, BotCommandProcess 适配HotDataCacheStore的修改;
[Change] BotCommandProcess 调整ranking命令的输出格式;
2020-05-07 10:26:11 +08:00
a36eb713d7 [Version] 更新版本(2.5.0-20200505.1-SNAPSHOT -> 2.5.0-20200506.1-SNAPSHOT); 2020-05-06 11:09:18 +08:00
f452c1c128 Merge branch 'add-MultiProperties' 2020-05-06 11:08:16 +08:00
cb42aadb15 [Fix] SettingProperties 修复在初次设置群组配置时未创建该群组所属Properties导致NPE的问题;
[Update] BotAdminCommandProcess 更新日志内容及命令返回信息;
[Change] BotCommandProcess 调整配置项key格式;
2020-05-06 10:55:32 +08:00
b969bb29b2 Merge branch 'master' into add-MultiProperties 2020-05-06 10:18:00 +08:00
0d74007a98 [Change] search.txt 整理接口相关文档的存储目录; 2020-05-06 09:42:34 +08:00
edade24883 [Change] Issue #2 调整Logger Name以统一Logger Name格式; 2020-05-06 09:34:28 +08:00
34f57404ca [Update] 重构Setting Properties, 使其支持分群配置; 2020-05-06 00:57:48 +08:00
99b6e14ff7 [Change] RandomIntervalSendTimer 调整日志输出内容; 2020-05-05 17:09:45 +08:00
c8fe2c3fdd [Version] 更新版本(2.5.0-20200504.2-SNAPSHOT -> 2.5.0-20200505.1-SNAPSHOT);
[Update] net.lamgc:java-utils 更新依赖版本(1.1.0 -> 1.2.0_20200505.1-SNAPSHOT);
2020-05-05 01:29:11 +08:00
49a33d4078 [Add] MessageEventExecutionDebugger 添加对消息处理的调试器Enum;
[Add] BotEventHandler 添加对 MessageEventExecutionDebugger 的支持;
[Add] VirtualLoadMessageEvent 增加 toVirtualLoadMessageEvent(MessageEvent) 方法;
2020-05-05 01:27:10 +08:00
cd1d2316ee [Add] BotEventHandler 添加executeMessageEvent(MessageEvent)方法;
[Change] BotEventHandler 将BotEventHandler.executor设为private;
[Change] MiraiMain, CQPluginMain, RankingUpdateTimer 适配BotEventHandler的调整
2020-05-04 23:06:07 +08:00
cf08353ed9 [Change] BotEventHandler 调整事件处理线程池的线程数(min: 4, Max: 32); 2020-05-04 22:52:47 +08:00
bef6a684b9 [Change] BotEventHandler 调整事件处理线程池的线程数; 2020-05-04 22:46:06 +08:00
04960889b4 [Version] 更新版本(2.5.0-20200504.01-SNAPSHOT -> 2.5.0-20200504.2-SNAPSHOT);
[Update] net.lamgc:java-utils 更新依赖版本(1.1.0_5-SNAPSHOT -> 1.1.0);
[Update] net.lamgc:PixivLoginProxyServer 更新依赖版本(1.1.0 -> 1.1.1);
2020-05-04 19:44:11 +08:00
0dc8fc78b4 [Version] 更新版本(2.4.0 -> 2.5.0-20200504.01-SNAPSHOT); 2020-05-04 17:07:47 +08:00
40c5284be2 [Change] Dockerfile.sample 调整构建指令顺序以充分利用构建缓存;
[Fix] Main 修复SystemProperties设置null抛出NPE的问题;
2020-05-04 17:07:39 +08:00
46cb47c7d1 [Add] LICENSE 添加开源许可证(LGPLv3); 2020-05-04 16:55:16 +08:00
fe213deecb [Change] Main 调整参数的接收形式;
[Fix] Dockerfile.sample 修复Java应用无法接收stop命令发送信号的问题;
[Change] RankingUpdateTimer 调整更新投递形式, 将同步更新调整为异步更新;
[Change] BotEventHandler 实装TimeLimitThreadPoolExecutor;
2020-05-04 02:07:46 +08:00
f279d99fda [Add] TimeLimitThreadPoolExecutor 增加一个带有执行时间限制的线程池及对应单元测试类; 2020-05-04 02:04:59 +08:00
a09aef5be2 [Delete] 删除已完成或已经没有存在意义的TODO; 2020-05-02 10:40:11 +08:00
766411fa09 [Fix] Dockerfile.sample 修复应用无法接收容器退出信号的问题(现在应用将可以接受信号主动退出, 即使ExitCode不为0); 2020-05-02 00:56:05 +08:00
84b544cea9 [Change] Main 调整日志输出级别, 补充日志输出内容;
[Change] Dockerfile.sample 调整镜像内应用运行路径;
2020-05-01 22:28:15 +08:00
2f492f5b03 [Add] log4j2.xml, log4j2-test.xml 添加允许指定日志存储目录的功能(sys:cgj.logsPath); 2020-05-01 13:19:08 +08:00
15af939c3f [Fix] PixivDownload 修复因排行榜总量的不确定导致排行榜获取异常的问题;
[Change] BotEventHandler 调整事件处理完成后的日志输出形式;
2020-04-30 19:08:34 +08:00
a28cb142b4 [Change] 调整数据存储的路径设置及存储目录参数名; 2020-04-30 17:51:57 +08:00
80 changed files with 5061 additions and 1433 deletions

26
.github/ISSUE_TEMPLATE/Bug_Report.md vendored Normal file
View File

@ -0,0 +1,26 @@
---
name: Bug Report
about: Use this template to feedback bugs.
title: ''
labels: bug
assignees: ''
---
## Environmental information ##
OS(e.g: Windows 10 1909):
Java(e.g: Oracle Jdk 8.242):
Issue version(versionTag or commitId):
## Problem description ##
// Describe the problem in as much detail as possible here
## Expected behavior ##
// What will this function do under normal circumstances?
## Actual behavior ##
// But what does this feature actually look like?
## Recurrence steps ##
// What can we do to recreate this situation?
1.

14
.gitignore vendored
View File

@ -1,6 +1,16 @@
# Ignore test date folder
/pluginData/
/logs/
/.idea/
/CGJ_2.iml
/cookies.store
/target/
# Ignore Idea files
/.idea/
*.iml
# Ignore Visual Studio Code files
.classpath
.factorypath
.project
/.settings/
/.vscode/

View File

@ -1,8 +1,10 @@
FROM openjdk:8u252-jre
FROM openjdk:14
COPY --from=hengyunabc/arthas:latest /opt/arthas /opt/arthas
ENV jarFileName=ContentGrabbingJi-exec.jar
COPY ${jarFileName} /program/CGJ.jar
RUN mkdir /program/data
ENV CGJ_REDIS_URI="127.0.0.1:6379"
ENV CGJ_PROXY=""
RUN mkdir /data/
CMD [java, "-Dcgj.logsPath=/data/logs", "-jar", "/CGJ.jar", "botMode", "-botDataDir=/data"]
WORKDIR /program/data
CMD java -jar /program/CGJ.jar botMode
COPY ${jarFileName} /CGJ.jar

661
LICENSE Normal file
View File

@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

61
README.md Normal file
View File

@ -0,0 +1,61 @@
# ContentGrabbingJi #
Pixiv爬虫一只同时也是一个机器人/插件!
## 支持的机器人平台 ##
- [Mirai](https://github.com/mamoe/mirai)
- [CoolQ](https://cqp.cc)(基于[`SpringCQ`](https://github.com/lz1998/spring-cq), 不支持多账号使用, 需要使用[`CQHttp`](https://cqhttp.cc/)插件)
## Usage ##
> 注意: 运行色图姬前, 你需要准备一个Pixiv账号的会话Cookie存储文件, 否则色图姬将无法运行.
> 详见[PixivLoginProxyServer](https://github.com/LamGC/PixivLoginProxyServer)项目的[Readme](https://github.com/LamGC/PixivLoginProxyServer/blob/master/README.md).
### Arguments ###
> ENV参数名为环境变量名, 用于给Docker容器提供设置方式.
- 通用参数
- `-proxy` / `ENV: CGJ_PROXY`: (**可选**) 设置代理
- 格式: `协议://地址:端口`
- 示例: `socks5://127.0.0.1:1080`
- 机器人参数
- `-botDataDir` / `ENV: CGJ_BOT_DATA_DIR`: (**可选**) 设置`botMode`运行模式下机器人数据存储目录
- 格式: `路径`
- 示例: `./data`
- 默认: `./`
- `-redisAddress` / `ENV: CGJ_REDIS_URI`: (**必填, 计划支持可选**) Redis服务器地址
- 格式: `地址:端口`
- 示例: `127.0.0.1:6379`
> 例如以BotMode启动应用: `java -jar CGJ.jar botMode -proxy "socks5://127.0.0.1:1080 -redisAddress 127.0.0.1:6379`
### Commands ###
- `pluginMode`: CoolQ插件模式(依赖[酷Q机器人](https://cqp.cc/), 支持与CoolQ其他插件共存, 性能耗损大)
- `botMode`: Mirai独立模式(机器人独立运行, 不支持与其他插件共存, 性能耗损小)
- `collectionDownload`: 收藏下载, 以原图画质下载Cookie所属账号的所有收藏作品
- `getRecommends`: 将访问主页获得的推荐作品全部以原图画质下载
- `getRankingIllust`: 以原图画质下载指定排行榜类型的全部作品
- `search`: 搜索指定内容并获取相关作品信息(不下载)
> 注意: 除去机器人模式外, 其他功能后续可能会出现改动.
## LICENSE ##
本项目基于 `GNU Affero General Public License 3.0` 开源许可协议开源,
你可以在本项目目录下的 `LICENSE` 文件查阅协议副本,
或浏览 [https://www.gnu.org/licenses/agpl-3.0.html](https://www.gnu.org/licenses/agpl-3.0.html) 查阅协议副本。
```
ContentGrabbingJi - A pixiv robot used in chat
Copyright (C) 2020 LamGC (lam827@lamgc.net)
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/agpl-3.0.html>.
```
尤其注意:根据协议,如果你基于本项目开发本项目的衍生版本并通过网络为他人提供服务,则必须遵守该协议,向 *服务的使用者* 提供 **为其服务的衍生版本****完整源代码**

View File

@ -0,0 +1,108 @@
## Pixiv作品信息批量获取接口 ##
### 说明 ###
接口可一次获取多个作品的基础信息
### 接口地址 ###
```
GET https://www.pixiv.net/ajax/illust/recommend/illusts
```
- 需要登录: `否`
- 是否为Pixiv接口标准返回格式: `是`
### 参数 ###
- `illust_ids[]`: 作品Id, 可重复添加该参数
### 请求示例 ###
```
GET https://www.pixiv.net/ajax/illust/recommend/illusts?illust_ids[]=82030844&illust_ids[]=82029098&illust_ids[]=82028913
```
### 返回数据 ###
#### 数据示例 ####
```json
{
"error":false,
"message":"",
"body":{
"illusts":[
{
"illustId":"82030844",
"illustTitle":"3rd anniversary",
"id":"82030844",
"title":"3rd anniversary",
"illustType":0,
"xRestrict":0,
"restrict":0,
"sl":2,
"url":"https:\/\/i.pximg.net\/c\/360x360_70\/custom-thumb\/img\/2020\/06\/02\/11\/24\/49\/82030844_p0_custom1200.jpg",
"description":"",
"tags":[
"\u30a2\u30ba\u30fc\u30eb\u30ec\u30fc\u30f3",
"\u6bd4\u53e1(\u30a2\u30ba\u30fc\u30eb\u30ec\u30fc\u30f3)",
"\u6c34\u7740",
"\u8db3\u88cf",
"\u88f8\u8db3",
"\u30a2\u30ba\u30fc\u30eb\u30ec\u30fc\u30f35000users\u5165\u308a",
"\u30a2\u30ba\u30fc\u30eb\u30ec\u30fc\u30f310000users\u5165\u308a",
"\u30dd\u30cb\u30fc\u30c6\u30fc\u30eb",
"\u30db\u30eb\u30bf\u30fc\u30cd\u30c3\u30af"],
"userId":"6662895",
"userName":"ATDAN-",
"width":1500,
"height":844,
"pageCount":1,
"isBookmarkable":true,
"bookmarkData":null,
"alt":"#\u30a2\u30ba\u30fc\u30eb\u30ec\u30fc\u30f3 3rd anniversary - ATDAN-\u7684\u63d2\u753b",
"isAdContainer":false,
"titleCaptionTranslation":{
"workTitle":null,
"workCaption":null
},
"createDate":"2020-06-02T01:29:40+09:00",
"updateDate":"2020-06-02T11:24:49+09:00",
"profileImageUrl":"https:\/\/i.pximg.net\/user-profile\/img\/2016\/01\/11\/21\/46\/50\/10371466_80f6ad67eab3b8abd44a2fb74ddd1ba1_50.jpg",
"type":"illust"
}, // ...
]
}
}
```
#### 参数详解 ####
- `illusts`: (`Object[]`) 存储查询作品信息的数组
- `illustId`: (`string` -> `number`) 作品Id
- `illustTitle`: (`string`) 作品标题
- `id`: (`string` -> `number`) 与`illustId`一致, 猜测是以兼容旧版本为目录而存在
- `title`: (`string`) 与`illustTitle`一致, 猜测是以兼容旧版本为目录而存在
- `illustType`: (`number`) 作品类型
- `0`: 插画作品
- `1`: 漫画作品
- `2`: 动图作品
- `xRestrict`: (`number`) 作品是否为限制级, 基本准确, 少部分不一定(看Pixiv审核怎么理解了)
- `0`: 非限制级内容(即非R18作品)
- `1`: 限制级内容(即R18作品)
- `restrict`: (`number`) 作品限制级(意义不明, 可能是兼容性问题?)?
- `sl`: (`number`) 不明?
- `url`: (`string`) 作品预览图链接, 需要`Referer`请求头
- `description`: (`string`) 作品说明
- `tags`: (`string[]`) 作品标签数组
- `userId`: (`string` -> `number`) 作者用户Id
- `userName`: (`string`) 作者用户名
- `width`: (`number`) 作品长度
- `height`: (`number`) 作品高度
- `pageCount`: (`number`) 作品页数
- `isBookmarkable`: (`boolean`) 不明?
- `alt`: (`string`) 简略介绍信息(在图片加载失败时可提供给`img`标签使用)
- `isAdContainer`: (`boolean`) 不明?
- `titleCaptionTranslation`: (`Object`) 不明?
- `workTitle`: (`Unknown`) 不明?
- `workCaption`: (`Unknown`) 不明?
- `createDate`: (`string`) 作品创建时间(或者是完成时间?)
- `updateDate`: (`string`) 作品上传时间
- `profileImageUrl`: (`string`) 作者用户头像图片链接
- `type`: (`string`) 作品类型名

View File

@ -0,0 +1,166 @@
## Pixiv 排行榜获取接口 ##
### 接口地址 ###
```
GET https://www.pixiv.net/ranking.php
```
- 是否需要登录: `是`
- 是否为Pixiv标准接口返回格式: `否`
- 是否需要Referer请求头: `否`
### 参数 ###
> 提示: 该接口参数较为复杂,请结合表格查看
- `date`: 排行榜时间与Mode有关 (格式: yyyy-MM-dd)
- `mode`: 排行榜模式
- `daily`: 每天
- `weekly`: 每周
- `monthly`: 每月
- `rookie`: 新人
- `original`: 原创
- `male`: 男性偏好
- `female`: 女性偏好
- `daily_r18`: 每天 - 仅成人内容
- `weekly_r18`: 每周 - 仅成人内容
- `male_r18`: 男性偏好 - 仅成人内容
- `female_r18`: 女性偏好 - 仅成人内容
- `content`: 排行榜内容类型
- `all`: 全部内容 (实际使用请直接省略`content`参数)
- `illust`: 插画
- `ugoira`: 动图
- `manga`: 漫画
- `p`: 排行榜页数 (如超出范围,则返回错误信息)
- `format`: 格式
- `json`: 以Json返回数据
- (留空): 返回完整的排行榜网页
#### 参数关系表 ####
`mode`参数与`content`参数有一些支持关系,并不是所有的`mode`参数都能被所有的`content`参数支持,故附下表。
参数 |all|illust|ugoira|manga
:-: |:-:| :-: | :-: | :-:
daily |`√`|`√`|`√`|`√`
weekly |`√`|`√`|`√`|`√`
monthly |`√`|`√`|×|`√`
rookie |`√`|`√`|×|`√`
original |`√`|×|×|×
male |`√`|×|×|×
female |`√`|×|×|×
daily_r18 |`√`|`√`|`√`|`√`
weekly_r18|`√`|`√`|`√`|`√`
male_r18 |`√`|×|×|×
female_r18|`√`|×|×|×
### 返回数据 ###
#### 数据示例 ####
```json
{
"contents":[
{
"title":"【伊アオ】髪結い。",
"date":"2020年05月31日 14:26",
"tags":[
"鬼滅の刃",
"伊アオ",
"嘴平伊之助",
"神崎アオイ",
"鬼滅の刃1000users入り"
],
"url":"https:\/\/i.pximg.net\/c\/240x480\/img-master\/img\/2020\/05\/31\/14\/26\/41\/81987309_p0_master1200.jpg",
"illust_type":"0",
"illust_book_style":"0",
"illust_page_count":"1",
"user_name":"シロウ",
"profile_img":"https:\/\/i.pximg.net\/user-profile\/img\/2020\/05\/01\/02\/18\/18\/18450100_ac34872504959f8cc26f086248066b39_50.png",
"illust_content_type":{
"sexual":0,
"lo":false,
"grotesque":false,
"violent":false,
"homosexual":false,
"drug":false,
"thoughts":false,
"antisocial":false,
"religion":false,
"original":false,
"furry":false,
"bl":false,
"yuri":false
},
"illust_series":false,
"illust_id":81987309,
"width":600,
"height":2226,
"user_id":174995,
"rank":51,
"yes_rank":83,
"rating_count":707,
"view_count":19759,
"illust_upload_timestamp":1590902801,
"attr":"",
"is_bookmarked":false,
"bookmarkable":true
}, // ....
],
"mode":"daily",
"content":"all",
"page":2,
"prev":1,
"next":3,
"date":"20200601",
"prev_date":"20200531",
"next_date":false,
"rank_total":500
}
```
#### 字段说明 ####
- `contents`: (`Object[]`) 排行榜数组, 最多50行排行榜信息
- `illust_id`: (`number`) 作品Id
- `title`: (`string`) 作品标题
- `attr`: (`string`) 不明?
- `tags`: (`string[]`) 原始标签数组
- `url`: (`string`) 预览画质的原始尺寸图下载链接(存在防盗链)
- `illust_type`: (`string` -> `number`) 作品类型
- `illust_book_style`: (`string` -> `number`) 不明?
- `illust_page_count`: (`string` -> `number`) 作品页数
- `user_name`: (`string`) 画师用户名
- `user_id`: (`number`) 画师用户Id
- `profile_img`: (`string`) 画师用户头像
- `illust_content_type`: (`Object`) 作品内容信息(警告: 文档内容仅作为开发参考, 并不传播相关内容!!!)
- `sexual`: (`number`) 作品内容分级
- `0`: 全年龄
- `1`: 青少年
- `2`: 成人级
- `lo`: (`boolean`) 是否为loli作品
- `grotesque`: (`boolean`) 是否为怪诞作品
- `violent`: (`boolean`) 作品是否含有暴力/强暴相关元素
- `homosexual`: (`boolean`) 作品是否含有同性恋相关元素
- `drug`: (`boolean`) 作品是否含有药物相关元素
- `thoughts`: (`boolean`) 作品是否含有思维/记忆相关元素(这个属性翻译起来有些问题, 待纠正)?
- `antisocial`: (`boolean`) 作品是否含有反社会, 令人厌恶的相关元素
- `religion`: (`boolean`) 作品是否含有宗教, 信仰相关元素
- `original`: (`boolean`) 作品是否为原创作品
- `furry`: (`boolean`) 作品是否有兽人相关元素
- `bl`: (`boolean`) 作品是否有耽美相关元素
- `yuri`: (`boolean`) 作品是否有百合相关元素
- `illust_series`: (`boolean`) 是否为系列作品
- `width`: (`number`) 作品宽度(建议以原图为准)
- `height`: (`number`) 作品高度(建议以原图为准)
- `rank`: (`number`) 本期排行榜排名
- `yes_rank`: (`number`) 上期同排行榜排名
- `rating_count`:
- `view_count`: (`number`) 浏览量
- `illust_upload_timestamp`: (`number`) 作品上传时间戳(10位)
- `is_bookmarked`: (`boolean`) 不明?
- `bookmarkable`: (`boolean`) 不明?
- `mode`: (`string`) 请求的排行榜模式字段
- `content`: (`string`) 请求的内容类型
- `page`: (`number`) 当前排行榜页数
- `prev`: (`string` / `boolean`) 上一页排行榜页数, 如果该请求的页数为首页, 则为`false`
- `next`: (`string` / `boolean`) 下一页排行榜页数, 如果该请求的页数为页尾, 则为`false`
- `date`: (`string`) 排行榜日期(格式:`yyyyMMdd`)
- `prev_date`: (`string` / `boolean`) 如果存在上一期排行榜, 则该属性为上期排行榜日期字符串, 否则为`false`
- `next_date`: (`string` / `boolean`) 如果存在下一期排行榜, 则该属性为下期排行榜日期字符串, 否则为`false`
- `rank_total`: (`number`) 该排行榜的总排行数

View File

@ -0,0 +1,32 @@
## 接口名 ##
### 说明 ###
### 接口地址 ###
```
GET/POST https://www.pixiv.net/...
```
- 是否需要登录: `是/否`
- 是否为Pixiv标准接口返回格式: `是/否`
- 是否需要Referer请求头: `是/否`
### 参数 ###
### 请求示例 ###
```
GET/POST
---- Request Body ---- // 如果没有, 可以不写, 如没有记得删除
```
### 返回数据 ###
#### 数据示例 ####
```json
// Object[] 中只留一个, 其他删除后保留逗号, 增加 '// ...' 注释
```
#### 字段说明 ####
- `属性名`: (`JS类型`) 属性说明
- `对象属性名`: (`原始JS类型` -> `可转换JS类型`) 属性说明
- `属性名`: (`JS类型1` / `JS类型2` / ...) 属性说明, 需要清晰说明在什么情况下类型为`JS类型1`, 什么情况下是`JS类型2`.
- `不明属性`: (`Unknown`) 如果属性用途不明, 则在说明后面加上`?`符号, 类型不明则填`Unknown`.

View File

@ -0,0 +1,18 @@
## Pixiv接口标准返回格式 ##
Pixiv大部分接口在返回数据时都会遵循以下格式
```json
{
"error": false,
"message": "",
"body": {
}
}
```
大部分是如此(部分接口比较特别, 不是这个格式)
属性|类型|说明
---|---|---
error|Boolean|如果接口返回错误信息, 该属性为`true`
message|String|如果`error`属性为`true`, 则该属性不为空字符串
body|Object/Array|如果`error`不为`true`, 该属性有数据, 否则属性可能不存在

View File

@ -0,0 +1,365 @@
## Pixiv内容搜索接口 ##
### 说明 ###
> 注意: 本接口可能会影响Pixiv对账号的行为判断猜测不一定会
该接口用于在Pixiv搜索内容。
### 接口地址 ###
```
GET https://www.pixiv.net/ajax/search/{Type}/{Content}?{Param...}
```
- 是否需要登录: `是/否`
- 是否为Pixiv标准接口返回格式: `是/否`
- 是否需要Referer请求头: `否`
### 参数 ###
#### Url参数 ####
- `Type`: 内容类型
- illustrations(插画)
- top(推荐)
- manga(漫画)
- novels(小说)
- `Content`: 搜索内容
#### GET参数 ####
##### 必填 #####
- `word`: 与搜索内容一致 (经测试似乎可以省略)
- `s_mode`: 匹配模式
- `s_tag`: 标签,部分一致
- `s_tag_full`: 标签,完全一致
- `s_tc`: 标签和说明文字
- `type`: 作品类型
- `all`: 插画、漫画和动图
- `illust_and_ugoira`: 插画和动图
- `illust`: 仅插画
- `manga`: 仅漫画
- `ugoira`: 仅动图
- `p`: 指定页数 (当指定页数超出范围后,`body.illust.data`为空数组)
- `order`: 排序设置
- `date`: 按时间从旧到新
- `date_d`: 按时间从新到旧
- `(Unknown)`: 未知, 猜测是会员功能的热门搜索
- `mode`: 内容分级设置
- `all`: 全部内容
- `safe`: 排除成人内容
- `r18`: 仅成人内容
##### 选填 #####
- `wlt`: 作品最低宽度(px)
- `wgt`: 作品最高宽度(px)
- `hlt`: 作品最低高度(px)
- `hgt`: 作品最高高度(px)
- `ratio`: 作品横宽比过滤 (初步测试表明,该参数无法指定横宽比,可能暂不支持该功能)
- `0.5`: 仅横图
- `-0.5`: 仅纵图
- `0`: 仅正方形图
- `tool`: 限定作品绘制工具
- `scd`: 过滤作品发布时间 - 开始时间(yyyy-MM-dd)
- `scd`: 过滤作品发布时间 - 结束时间(yyyy-MM-dd)
- `(Unknown)`: 最小收藏数 (该参数为会员限定功能,后续补充)
### 返回数据 ###
#### 数据示例 ####
```json
{
"error":false,
"body":{
"illustManga":{
"data":[
{
"illustId":"82130571",
"illustTitle":"空の絵",
"id":"82130571",
"title":"空の絵",
"illustType":0,
"xRestrict":0,
"restrict":0,
"sl":2,
"url":"https:\/\/i.pximg.net\/c\/250x250_80_a2\/img-master\/img\/2020\/06\/06\/17\/51\/14\/82130571_p0_square1200.jpg",
"description":"",
"tags":[
"風景",
"空",
"草",
"雲"],
"userId":"31507675",
"userName":"昏omeme",
"width":1600,
"height":1600,
"pageCount":2,
"isBookmarkable":true,
"bookmarkData":null,
"alt":"#風景 空の絵 - 昏omeme的插画",
"isAdContainer":false,
"titleCaptionTranslation":{
"workTitle":null,
"workCaption":null
},
"createDate":"2020-06-06T17:51:14+09:00",
"updateDate":"2020-06-06T17:51:14+09:00",
"profileImageUrl":"https:\/\/i.pximg.net\/user-profile\/img\/2020\/05\/06\/19\/21\/04\/18509741_e3166e69809c44d6926454ecaac89590_50.png"
}, // ...
],
"total":165875,
"bookmarkRanges":[
{
"min":null,
"max":null
},
{
"min":10000,
"max":null
},
{
"min":5000,
"max":null
},
{
"min":1000,
"max":null
},
{
"min":500,
"max":null
},
{
"min":300,
"max":null
},
{
"min":100,
"max":null
},
{
"min":50,
"max":null
}
]
},
"popular":{
"recent":[
{
"illustId":"82062770",
"illustTitle":"Still you remember",
"id":"82062770",
"title":"Still you remember",
"illustType":0,
"xRestrict":0,
"restrict":0,
"sl":2,
"url":"https:\/\/i.pximg.net\/c\/250x250_80_a2\/img-master\/img\/2020\/06\/03\/18\/02\/15\/82062770_p0_square1200.jpg",
"description":"",
"tags":[
"オリジナル",
"女の子",
"カラス",
"风景",
"線路"],
"userId":"1069005",
"userName":"へちま",
"width":2000,
"height":1415,
"pageCount":1,
"isBookmarkable":true,
"bookmarkData":null,
"alt":"#オリジナル Still you remember - へちま的插画",
"isAdContainer":false,
"titleCaptionTranslation":{
"workTitle":null,
"workCaption":null
},
"createDate":"2020-06-03T18:02:15+09:00",
"updateDate":"2020-06-03T18:02:15+09:00",
"profileImageUrl":"https:\/\/i.pximg.net\/user-profile\/img\/2013\/05\/10\/00\/38\/05\/6213543_c94edc0d13776214467bd0c47ee6491a_50.jpg"
}, // ...
],
"permanent":[
{
"illustId":"60993044",
"illustTitle":"無題",
"id":"60993044",
"title":"無題",
"illustType":0,
"xRestrict":0,
"restrict":0,
"sl":2,
"url":"https:\/\/i.pximg.net\/c\/250x250_80_a2\/img-master\/img\/2017\/01\/18\/13\/07\/46\/60993044_p0_square1200.jpg",
"description":"",
"tags":[
"少女",
"女の子",
"原创",
"オリジナル",
"场景",
"落書き",
"創作",
"背景",
"风景",
"オリジナル7500users入り"],
"userId":"18811972",
"userName":"淅洵",
"width":3507,
"height":2480,
"pageCount":1,
"isBookmarkable":true,
"bookmarkData":null,
"alt":"#少女 無題 - 淅洵的插画",
"isAdContainer":false,
"titleCaptionTranslation":{
"workTitle":null,
"workCaption":null
},
"createDate":"2017-01-18T13:07:46+09:00",
"updateDate":"2017-01-18T13:07:46+09:00",
"profileImageUrl":"https:\/\/i.pximg.net\/user-profile\/img\/2017\/05\/29\/17\/17\/49\/12623968_6cf3f1979e10643425972ae205a7920d_50.jpg"
}, // ...
]
},
"relatedTags":[
"風景",
"背景",
"風景画",
"空",
"雲",
"創作",
"ファンタジー",
"夏",
"青",
"建物",
"青空",
"少女",
"東京",
"抽象画",
"男の子",
"透明水彩"
],
"tagTranslation":{
"風景":{
"zh":"风景"
},
"背景":{
"zh":"background"
},
"風景画":{
"zh":"landscape painting"
},
"空":{
"zh":"sky"
},
"雲":{
"zh":"云"
},
"創作":{
"zh":"原创"
},
"ファンタジー":{
"zh":"奇幻"
},
"夏":{
"zh":"夏天"
},
"青":{
"zh":"蓝"
},
"建物":{
"zh":"building"
},
"青空":{
"zh":"蓝天"
},
"少女":{
"zh":"young girl"
},
"東京":{
"zh":"tokyo"
},
"抽象画":{
"zh":"abstract art"
},
"男の子":{
"zh":"男孩子"
},
"透明水彩":{
"zh":"transparent watercolor"
}
},
"zoneConfig":{
"header":{
"url":"https:\/\/pixon.ads-pixiv.net\/show?zone_id=header&format=js&s=1&up=0&a=22&ng=g&l=zh&uri=%2Fajax%2Fsearch%2Fartworks%2F_PARAM_&is_spa=1&K=59bba275c645c&ab_test_digits_first=20&yuid=FwdzEnA&suid=Pgfip96ymw5tvu9l9&num=5edb6277927"
},
"footer":{
"url":"https:\/\/pixon.ads-pixiv.net\/show?zone_id=footer&format=js&s=1&up=0&a=22&ng=g&l=zh&uri=%2Fajax%2Fsearch%2Fartworks%2F_PARAM_&is_spa=1&K=59bba275c645c&ab_test_digits_first=20&yuid=FwdzEnA&suid=Pgfip96yn1fgocj2&num=5edb6277775"
},
"infeed":{
"url":"https:\/\/pixon.ads-pixiv.net\/show?zone_id=illust_search_grid&format=js&s=1&up=0&a=22&ng=g&l=zh&uri=%2Fajax%2Fsearch%2Fartworks%2F_PARAM_&is_spa=1&K=59bba275c645c&ab_test_digits_first=20&yuid=FwdzEnA&suid=Pgfip96yn4t7cho88&num=5edb6277137"
}
},
"extraData":{
"meta":{
"title":"#风景のイラスト・マンガ作品(投稿超过10万件 - pixiv",
"description":"pixiv",
"canonical":"https:\/\/www.pixiv.net\/tags\/%E9%A3%8E%E6%99%AF",
"alternateLanguages":{
"ja":"https:\/\/www.pixiv.net\/tags\/%E9%A3%8E%E6%99%AF",
"en":"https:\/\/www.pixiv.net\/en\/tags\/%E9%A3%8E%E6%99%AF"
},
"descriptionHeader":"pixiv"
}
}
}
}
```
#### 字段说明 ####
- `novel`: (`Object`) 小说搜索结果
- `data`: (`Object`) 搜索结果(仅当前页数)
- (待补充)
- `total`: (`number`) 搜索结果总量
- `popular`: (`Object`) 受欢迎的搜索结果
- `relatedTags`: (`string[]`) 与搜索结果相关的标签
- `tagTranslation`: (`Object`) 相关标签的翻译信息
- `{Attr: 标签名}`: 标签名为属性名
- `语言名(例如 中文是 zh)`: (`string`) 标签翻译名
- `zoneConfig`: (`Object`) 猜测是广告相关信息?
- `extraData`: (`Object`) 扩展信息
- `meta`: (`Object`) 网页元数据
- `title`: (`string`) 网页标题
- `description`: 搜索结果说明内容
- `descriptionHeader`: (`string`) 说明内容的Html代码
- `alternateLanguages`: (`Object`) 不明链接?
- `{语言名}`: 对应语言的链接
- `illustManga`: (`Object`) 漫画和插画的搜索结果
- `total`: (`number`) 搜索结果总量
- `data`: (`Object[]`) 搜索结果(仅当前页数)
- `illustId`: (`string` -> `number`) 作品Id
- `illustTitle`: (`string`) 作品标题
- `id`: (`string` -> `number`) 与`illustId`一致, 猜测是以兼容旧版本为目录而存在
- `title`: (`string`) 与`illustTitle`一致, 猜测是以兼容旧版本为目录而存在
- `illustType`: (`number`) 作品类型
- `0`: 插画作品
- `1`: 漫画作品
- `2`: 动图作品
- `xRestrict`: (`number`) 作品是否为限制级, 基本准确, 少部分不一定(看Pixiv审核怎么理解了)
- `0`: 非限制级内容(即非R18作品)
- `1`: 限制级内容(即R18作品)
- `restrict`: (`number`) 作品限制级(意义不明, 可能是兼容性问题?)?
- `sl`: (`number`) 不明?
- `url`: (`string`) 作品预览图链接, 需要`Referer`请求头
- `description`: (`string`) 作品说明
- `tags`: (`string[]`) 作品标签数组
- `userId`: (`string` -> `number`) 作者用户Id
- `userName`: (`string`) 作者用户名
- `width`: (`number`) 作品长度
- `height`: (`number`) 作品高度
- `pageCount`: (`number`) 作品页数
- `isBookmarkable`: (`boolean`) 不明?
- `bookmarkData`: (`Unknown`) 不明?
- `alt`: (`string`) 简略介绍信息(在图片加载失败时可提供给`img`标签使用)
- `isAdContainer`: (`boolean`) 不明?
- `titleCaptionTranslation`: (`Object`) 不明?
- `workTitle`: (`Unknown`) 不明?
- `workCaption`: (`Unknown`) 不明?
- `createDate`: (`string`) 作品创建时间(或者是完成时间?)
- `updateDate`: (`string`) 作品上传时间
- `profileImageUrl`: (`string`) 作者用户头像图片链接

View File

@ -0,0 +1,129 @@
## Pixiv首页数据接口 ##
### 说明 ###
> 注意: 该接口涉及用户账户隐私, 不要尝试对该接口返回数据做不安全云端存储, 或未经用户允许的发送出去.
该接口用于获取Pixiv推荐给账号的首页作品信息每次调用都会有不同结果。
### 接口地址 ###
```
GET https://www.pixiv.net/ajax/top/{type}?mode={mode}&lang={lang}
```
### 参数 ###
- `type`: 首页类型
- `illust`: 插画
- `manga`: 漫画
- `novel`: 小说
- `mode`: 内容类型
- `all`: 不限类型
- `r18`: 成人内容
- `lang`: 语言(只写几个)
- `zh`: 中文
- 是否需要登录: `是`
- 是否为Pixiv接口标准返回格式: `是`
- 是否需要Referer请求头: `是`
### 请求示例 ###
```
GET https://www.pixiv.net/ajax/top/illust?mode=all&lang=zh
```
### 返回数据 ###
#### 数据示例 ####
```
(内容过长, 略)
```
#### 字段说明 ####
- `page`: 网页相关内容
- `tags`: (`Object[]`) 热门标签
- `tag`: (`String`) 标签名
- `count`: (`Number`) 作品数量?
- `lev`: (`Number`) 不明?
- `ids`: (`Number[]`) 作品数组
- `follow`: (`Number[]`) 已关注作者的作品推荐
- `mypixiv`: (`?[]`) 不明
- `recommend`: (`String[]` -> `Number[]`) 推荐作品的Id?
- `recommendByTag`: (`Object[]`) 标签的推荐作品
- `tag`: (`String`) 标签名
- `ids`: ((`String[]` -> `Number[]`) 作品Id数组
- `ranking`: (`Object`) 排行榜前100名数据
- `date`: (`String`) 排行榜日期(`yyyyMMdd`)
- `items`: (`Object[]`) 排行榜简略数据
- `rank`: (`String` -> `Number`) 排行榜名次
- `id`: (`String` -> `Number`) 作品Id
- `pixivision`: (`Object[]`) Pixiv的推荐文章
- `id`: (`String` -> `Number`) 文章Id
- `title`: (`String`) 文章标题
- `thumbnailUrl`: (`String`) 文章封面图的Url地址
- `abtestId`: (`String` -> `Number`) pixivision文章地址的参数`utm_content`的值
- `recommendUser`: (`Object[]`) 推荐用户及其作品
- `id`: (`Number`) 用户Id
- `illustIds`: (`String[]` -> `Number[]`) 插画作品Id
- `novelIds`: (`String[]` -> `Number[]`) 小说作品Id
- `contestOngoing`: (`Object[]`) 进行中的比赛活动(没找到更多信息了, 先这么说着)
- (待完善)
- `contestResult`: (`Object[]`) 比赛结果信息
- `slug`: (`String`) 比赛代号?
- `type`: (`String`) 比赛作品类型?
- `name`: (`String`) 比赛名
- `url`: (`String`) 结果公布链接
- `iconUrl`: (`String`) 图标链接(不明意义的图标)
- `editorRecommend`: (`Object[]`) 编辑推荐(小说), 后续可能会删除, 这个应该是配合活动出的数据
- (待完善)
- `boothFollowItemIds`: (`String[]` -> `Number[]`) 已关注用户的最新商品
- `sketchLiveFollowIds`: (`?[]`) 不明?
- `sketchLivePopularIds`: (`String[]` -> `Number[]`) 不明?
- `myFavoriteTags`: (`?[]`) 关注的标签
- `newPost`: (`String[]` -> `Number[]`) 不明?
- `thumbnails`: (`Object`) 已关注用户的作品
- `illust`: (`Object[]`) 插画作品
- `illustId`: (`String` -> `Number`) 作品Id(或者准确来讲是 插画Id?)
- `illustTitle`: (`String`) 插画标题
- `id`: (`String` -> `Number`) 作品Id
- `title`: (`String`) 作品标题
- `illustType`: (`Number`) 作品类型
- `xRestrict`: (`Number`) 不明?
- `restrict`: (`Number`) 不明?
- `sl`: (`Number`) 不明?
- `url`: (`String`) 作品在主页的封面图Url
- `description`: (`String`) 作品说明?
- `tags`: (`String[]`) 标签原始名数组(不带翻译的原始名称)
- `userId`: (`String` -> `Number`) 用户Id
- `userName`: (`String`) 用户名
- `width`: (`Number`) 作品宽度
- `height`: (`Number`) 作品高度
- `pageCount`: (`Number`) 作品页数
- `isBookmarkable`: (`Boolean`) 不明?
- `bookmarkData`: (`?`) 不明?
- `alt`: (`String`) 简略信息
- `isAdContainer`: (`Boolean`) 广告标识?
- `titleCaptionTranslation`: (`Object`) 不明?
- (待完善)
- `urls`: (`Object`) 其他封面图尺寸的链接
- (略)
- `seriesId`: (`?`) 不明?
- `seriesTitle`: (`?`) 不明?
- `profileImageUrl`: (`String`) 用户头像图链接
- `users`: (`Object[]`) 用户信?
- `userId`: (`String` -> `Number`) 用户Id
- `name`: (`String`) 用户名
- `image`: (`String`) 用户头像图链接
- `imageBig`: (`String`) 用户大尺寸头像图链接
- `premium`: (`Boolean`) 是否为Pixiv高级会员
- `isFollowed`: (`Boolean`) 是否关注
- `isMypixiv`: (`Boolean`) 不明?
- `isBlocking`: (`Boolean`) 是否为黑名单?
- `background`: (`?`) 不明?
- `partial`: (`Number`) 不明?
- `tagTranslation`: 标签翻译名
- `key=${标签原始名}`: (`Object`) 标签翻译信息
- `${语言代码}`: (`String`) 对应语言代码的翻译, 不一定有
- `boothItems`: (`Object[]`) 商品信息
- (待完善)
- `sketchLives`: (`Object[]`) 不明?
- (待完善)
- `zoneConfig`: (`Object`) 不明?
- (待完善)

45
pom.xml
View File

@ -6,7 +6,15 @@
<groupId>net.lamgc</groupId>
<artifactId>ContentGrabbingJi</artifactId>
<version>2.4.0</version>
<version>2.5.2-20200630.1-SNAPSHOT</version>
<licenses>
<license>
<name>GNU Affero General Public License 3.0</name>
<url>https://www.gnu.org/licenses/agpl-3.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<repositories>
<repository>
@ -19,7 +27,7 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<maven.compiler.encoding>UTF-8</maven.compiler.encoding>
<mirai.CoreVersion>0.39.4</mirai.CoreVersion>
<mirai.CoreVersion>1.0.2</mirai.CoreVersion>
<mirai.JaptVersion>1.1.1</mirai.JaptVersion>
<kotlin.version>1.3.71</kotlin.version>
<ktor.version>1.3.2</ktor.version>
@ -86,7 +94,7 @@
<dependency>
<groupId>net.lamgc</groupId>
<artifactId>java-utils</artifactId>
<version>1.1.0_5-SNAPSHOT</version>
<version>1.2.0_20200517.1-SNAPSHOT</version>
</dependency>
<dependency>
@ -110,21 +118,6 @@
<artifactId>gson</artifactId>
<version>2.8.5</version>
</dependency>
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.29</version>
</dependency>
<dependency>
<groupId>com.github.rholder</groupId>
<artifactId>guava-retrying</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.19</version>
</dependency>
<dependency>
<groupId>net.lz1998</groupId>
<artifactId>spring-cq</artifactId>
@ -169,16 +162,22 @@
<artifactId>ktor-server-core</artifactId>
<version>${ktor.version}</version>
</dependency>
<dependency>
<groupId>net.lamgc</groupId>
<artifactId>PixivLoginProxyServer</artifactId>
<version>1.1.0</version>
</dependency>
<dependency>
<groupId>com.squareup</groupId>
<artifactId>gifencoder</artifactId>
<version>0.10.1</version>
</dependency>
<dependency>
<groupId>org.jline</groupId>
<artifactId>jline</artifactId>
<version>3.15.0</version>
</dependency>
<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
<version>4.2.3</version>
</dependency>
</dependencies>
</project>

View File

@ -1,24 +0,0 @@
搜索标签信息https://www.pixiv.net/ajax/search/tags/标签名
搜索接口:
https://www.pixiv.net/ajax/search/{Type}/搜索内容
Type = illustrations(插画) / top(顶部?) / manga(漫画) / novels(小说)
word=搜索内容 [参数可能不是必须的]
s_mode=s_tag(标签-部分一致) / s_tag_full(标签-完全一致) / s_tc(标题、说明文字)
type= all(插画、漫画、动图_动态插图) / illust_and_ugoira(插画、动图) / illust(插画) / manga(漫画) / ugoira(动图)
p=页数 [超出页数的情况下将获取不到数据(即"body.illust.data"是空数组)]
order=date(按旧排序) / date_d(按新排序) / Unknown(按热门度排序, 需要会员)
mode= all(全部) / safe(全年龄) / r18(咳咳)
可选参数:
wlt=最小宽度像素
wgt=最高宽度像素
hlt=最小高度像素
hgt=最高高度像素
ratio=0.5(横图) / -0.5(纵图) / 0(正方形) [可能不能改变参数, 三个值是固定的]
tool=使用工具, 不是很重要晚些再加
scd=开始时间(yyyy-MM-dd)
ecd=结束时间(yyyy-MM-dd)
最小收藏数 = 收藏数限定参数为会员功能, 无法获取

View File

@ -1,35 +1,34 @@
package net.lamgc.cgj;
import com.github.monkeywie.proxyee.proxy.ProxyConfig;
import com.github.monkeywie.proxyee.proxy.ProxyType;
import com.google.common.base.Strings;
import com.google.gson.Gson;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import net.lamgc.cgj.bot.boot.ApplicationBoot;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.bot.framework.cli.ConsoleMain;
import net.lamgc.cgj.bot.framework.coolq.CQConfig;
import net.lamgc.cgj.bot.framework.mirai.MiraiMain;
import net.lamgc.cgj.pixiv.PixivDownload;
import net.lamgc.cgj.pixiv.PixivSearchBuilder;
import net.lamgc.cgj.pixiv.PixivSearchLinkBuilder;
import net.lamgc.cgj.pixiv.PixivURL;
import net.lamgc.plps.PixivLoginProxyServer;
import net.lamgc.utils.base.ArgumentsProperties;
import net.lamgc.utils.base.runner.Argument;
import net.lamgc.utils.base.runner.ArgumentsRunner;
import net.lamgc.utils.base.runner.Command;
import org.apache.commons.io.IOUtils;
import net.lamgc.utils.encrypt.MessageDigestUtils;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.client.CookieStore;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
import org.apache.tomcat.util.http.fileupload.util.Streams;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import java.io.*;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.text.ParseException;
import java.text.SimpleDateFormat;
@ -42,79 +41,71 @@ import java.util.zip.ZipOutputStream;
@SpringBootApplication
public class Main {
private final static Logger log = LoggerFactory.getLogger("Main");
private final static Logger log = LoggerFactory.getLogger(Main.class);
private final static File storeDir = new File("store/");
public static CookieStore cookieStore;
private static CookieStore cookieStore;
public static HttpHost proxy;
private static HttpHost proxy;
public static void main(String[] args) throws IOException, ClassNotFoundException {
log.trace("ContentGrabbingJi 正在启动...");
if(args.length != 0 && args[0].equalsIgnoreCase("buildpassword")) {
ArgumentsRunner.run(Main.class, args);
} else {
standardStart(args);
}
}
private static void standardStart(String[] args) throws IOException, ClassNotFoundException {
log.info("ContentGrabbingJi 正在启动...");
log.debug("Args: {}, LogsPath: {}", Arrays.toString(args), System.getProperty("cgj.logsPath"));
log.debug("运行目录: {}", System.getProperty("user.dir"));
ArgumentsProperties argsProp = new ArgumentsProperties(args);
if(argsProp.containsKey("proxy")) {
URL proxyUrl = new URL(argsProp.getValue("proxy"));
proxy = new HttpHost(proxyUrl.getHost(), proxyUrl.getPort());
log.info("已启用Http协议代理{}", proxy.toHostString());
} else {
proxy = null;
}
if(!storeDir.exists() && !storeDir.mkdirs()) {
log.error("创建文件夹失败!");
}
ApplicationBoot.initialApplication(args);
log.debug("botDataDir: {}", System.getProperty("cgj.botDataDir"));
// TODO: 需要修改参数名了, 大概改成类似于 workerDir这样的吧
if(argsProp.containsKey("cqRootDir")) {
log.info("cqRootDir: {}", argsProp.getValue("cqRootDir"));
System.setProperty("cgj.cqRootDir", argsProp.getValue("cqRootDir"));
} else {
log.warn("未设置cqRootDir, 当前运行目录将作为酷Q机器人所在目录.");
System.setProperty("cgj.cqRootDir", "./");
}
if(argsProp.containsKey("redisAddr")) {
log.info("redisAddress: {}", argsProp.getValue("redisAddr"));
System.setProperty("cgj.redisAddress", argsProp.getValue("redisAddr"));
} else {
log.info("未设置RedisAddress, 将使用默认值连接Redis服务器(127.0.0.1:6379)");
System.setProperty("cgj.redisAddress", "127.0.0.1");
}
File cookieStoreFile = new File("cookies.store");
proxy = BotGlobal.getGlobal().getProxy();
File cookieStoreFile = new File(BotGlobal.getGlobal().getDataStoreDir(), "cookies.store");
if(!cookieStoreFile.exists()) {
log.error("未找到cookies.store文件, 是否启动PixivLoginProxyServer? (yes/no)");
Scanner scanner = new Scanner(System.in);
if(scanner.nextLine().equalsIgnoreCase("yes")) {
startPixivLoginProxyServer();
} else {
System.exit(1);
return;
}
log.warn("未找到cookies.store文件, 请检查数据存放目录下是否存在'cookies.store'文件!");
System.exit(1);
return;
}
ObjectInputStream ois = new ObjectInputStream(new FileInputStream(cookieStoreFile));
cookieStore = (CookieStore) ois.readObject();
BotGlobal.getGlobal().setCookieStore(cookieStore);
ois.close();
log.info("已载入CookieStore");
log.debug("传入参数: {}", Arrays.toString(args));
ArgumentsRunner.run(Main.class, args);
System.exit(0);
}
@Command
public static void buildPassword(@Argument(name = "password") String password) {
System.out.println("Password: " +
Base64.getEncoder().encodeToString(MessageDigestUtils.encrypt(password.getBytes(),
MessageDigestUtils.Algorithm.MD5)));
}
@Command
public static void botMode(@Argument(name = "args", force = false) String argsStr) {
new MiraiMain().init();
MiraiMain main = new MiraiMain();
main.init();
main.close();
}
@Command
public static void consoleMode() throws IOException {
ConsoleMain.start();
}
@Command
public static void pluginMode(@Argument(name = "args", force = false) String argsStr) {
if(!System.getProperty("cgj.cqRootDir").endsWith("\\") && !System.getProperty("cgj.cqRootDir").endsWith("/")) {
System.setProperty("cgj.cqRootDir", System.getProperty("cgj.cqRootDir") + "/");
}
log.info("酷Q机器人根目录: {}", System.getProperty("cgj.cqRootDir"));
log.info("酷Q机器人根目录: {}", BotGlobal.getGlobal().getDataStoreDir().getPath());
CQConfig.init();
Pattern pattern = Pattern.compile("/\\s*(\".+?\"|[^:\\s])+((\\s*:\\s*(\".+?\"|[^\\s])+)|)|(\".+?\"|[^\"\\s])+");
Matcher matcher = pattern.matcher(Strings.nullToEmpty(argsStr));
@ -130,7 +121,7 @@ public class Main {
@Command
public static void collectionDownload() throws IOException {
PixivDownload pixivDownload = new PixivDownload(Objects.requireNonNull(cookieStore), proxy);
File outputFile = new File(storeDir, "collection.zip");
File outputFile = new File(getStoreDir(), "collection.zip");
if(!outputFile.exists() && !outputFile.createNewFile()) {
log.error("文件创建失败: " + outputFile.getAbsolutePath());
}
@ -142,7 +133,7 @@ public class Main {
ZipEntry entry = new ZipEntry(link.substring(link.lastIndexOf("/") + 1));
log.info("正在写入: " + entry.getName());
zos.putNextEntry(entry);
IOUtils.copy(inputStream, zos);
Streams.copy(inputStream, zos, false);
zos.flush();
} catch (IOException e) {
log.error("写入文件项时发生异常", e);
@ -157,10 +148,10 @@ public class Main {
PixivDownload pixivDownload = new PixivDownload(Objects.requireNonNull(cookieStore), proxy);
String date = new SimpleDateFormat("yyyyMMdd").format(new Date());
int id = 1;
File outputFile = new File(storeDir, "recommends-" + date + "-" + id + ".zip");
File outputFile = new File(getStoreDir(), "recommends-" + date + "-" + id + ".zip");
while(outputFile.exists()) {
id++;
outputFile = new File(storeDir, "recommends-" + date + "-" + id + ".zip");
outputFile = new File(getStoreDir(), "recommends-" + date + "-" + id + ".zip");
}
if(!outputFile.createNewFile()) {
@ -174,7 +165,7 @@ public class Main {
ZipEntry entry = new ZipEntry(link.substring(link.lastIndexOf("/") + 1));
log.info("正在写入: " + entry.getName());
zos.putNextEntry(entry);
IOUtils.copy(inputStream, zos);
Streams.copy(inputStream, zos, false);
zos.flush();
log.info("已成功写入 {}", entry.getName());
} catch (IOException e) {
@ -224,10 +215,10 @@ public class Main {
}
int id = 1;
File outputFile = new File(storeDir, "ranking" + rankingMode.modeParam + "-" + date + "-" + id + ".zip");
File outputFile = new File(getStoreDir(), "ranking" + rankingMode.modeParam + "-" + date + "-" + id + ".zip");
while(outputFile.exists()) {
id++;
outputFile = new File(storeDir, "ranking" + rankingMode.modeParam + "-" + date + "-" + id + ".zip");
outputFile = new File(getStoreDir(), "ranking" + rankingMode.modeParam + "-" + date + "-" + id + ".zip");
}
if(!outputFile.createNewFile()) {
@ -245,7 +236,7 @@ public class Main {
entry.setComment(rankInfo.toString());
log.info("正在写入: " + entry.getName());
zos.putNextEntry(entry);
IOUtils.copy(inputStream, zos);
Streams.copy(inputStream, zos, false);
zos.flush();
log.info("已成功写入 {}", entry.getName());
} catch (IOException e) {
@ -269,24 +260,24 @@ public class Main {
@Argument(name = "excludeKeywords", force = false) String excludeKeywords,
@Argument(name = "contentOption", force = false) String contentOption
) throws IOException {
PixivSearchBuilder searchBuilder = new PixivSearchBuilder(Strings.isNullOrEmpty(content) ? "" : content);
PixivSearchLinkBuilder searchBuilder = new PixivSearchLinkBuilder(Strings.isNullOrEmpty(content) ? "" : content);
if (type != null) {
try {
searchBuilder.setSearchType(PixivSearchBuilder.SearchType.valueOf(type.toUpperCase()));
searchBuilder.setSearchType(PixivSearchLinkBuilder.SearchType.valueOf(type.toUpperCase()));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchType: {}", type);
}
}
if(area != null) {
try {
searchBuilder.setSearchArea(PixivSearchBuilder.SearchArea.valueOf(area));
searchBuilder.setSearchArea(PixivSearchLinkBuilder.SearchArea.valueOf(area));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchArea: {}", area);
}
}
if(contentOption != null) {
try {
searchBuilder.setSearchContentOption(PixivSearchBuilder.SearchContentOption.valueOf(contentOption));
searchBuilder.setSearchContentOption(PixivSearchLinkBuilder.SearchContentOption.valueOf(contentOption));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchContentOption: {}", contentOption);
}
@ -325,7 +316,7 @@ public class Main {
JsonObject resultBody = jsonObject.getAsJsonObject("body");
for(PixivSearchBuilder.SearchArea searchArea : PixivSearchBuilder.SearchArea.values()) {
for(PixivSearchLinkBuilder.SearchArea searchArea : PixivSearchLinkBuilder.SearchArea.values()) {
if(!resultBody.has(searchArea.jsonKey) || resultBody.getAsJsonObject(searchArea.jsonKey).getAsJsonArray("data").size() == 0) {
//log.info("返回数据不包含 {}", searchArea.jsonKey);
continue;
@ -364,53 +355,11 @@ public class Main {
log.info("这里啥都没有哟w");
}
private static void saveCookieStoreToFile() throws IOException {
log.info("正在保存CookieStore...");
File outputFile = new File("./cookies.store");
if(!outputFile.exists() && !outputFile.delete() && !outputFile.createNewFile()){
log.error("保存CookieStore失败.");
return;
private static File getStoreDir() {
if(!storeDir.exists() && !storeDir.mkdirs()) {
log.error("创建文件夹失败!");
}
ObjectOutputStream stream = new ObjectOutputStream(new FileOutputStream(outputFile));
stream.writeObject(cookieStore);
stream.flush();
stream.close();
log.info("CookieStore保存成功.");
return storeDir;
}
private static void startPixivLoginProxyServer(){
ProxyConfig proxyConfig = null;
if(proxy != null) {
proxyConfig = new ProxyConfig(ProxyType.HTTP, proxy.getHostName(), proxy.getPort());
}
PixivLoginProxyServer proxyServer = new PixivLoginProxyServer(proxyConfig);
Thread proxyServerStartThread = new Thread(() -> {
log.info("启动代理服务器...");
proxyServer.start(1006);
log.info("代理服务器已关闭.");
});
proxyServerStartThread.setName("LoginProxyServerThread");
proxyServerStartThread.start();
//System.console().readLine();
Scanner scanner = new Scanner(System.in);
log.info("登录完成后, 使用\"done\"命令结束登录过程.");
while(true) {
if (scanner.nextLine().equalsIgnoreCase("done")) {
log.info("关闭PLPS服务器...");
proxyServer.close();
cookieStore = proxyServer.getCookieStore();
try {
log.info("正在保存CookieStore...");
saveCookieStoreToFile();
log.info("CookieStore保存完成.");
} catch (IOException e) {
log.error("CookieStore保存时发生异常, 本次CookieStore仅可在本次运行使用.", e);
}
break;
} else {
log.warn("要结束登录过程, 请使用\"done\"命令.");
}
}
}
}

View File

@ -1,57 +0,0 @@
package net.lamgc.cgj.bot;
import net.lz1998.cq.robot.CoolQ;
import org.apache.http.client.methods.HttpGet;
import java.util.Random;
import java.util.Timer;
import java.util.TimerTask;
public class AutoArtworksSender {
private final CoolQ CQ;
private final ReceiveType receiveType;
private final long targetReceiveId;
private Timer timer = new Timer();
private TimerTask task = new TimerTask() {
@Override
public void run() {
HttpGet request = new HttpGet();
// https://api.imjad.cn/pixiv/v2/?type=tags
}
};
public AutoArtworksSender(CoolQ cq, ReceiveType receiveType, long receiveId) {
this.CQ = cq;
this.receiveType = receiveType;
this.targetReceiveId = receiveId;
}
public void reset(long time) {
if(time <= 0) {
timer.schedule(task, new Random().nextInt(10 * 60 * 60 * 1000) + 7200000L); //2H ~ 12H
} else {
timer.schedule(task, time);
}
}
public void sendMessage(String message, boolean auto_escape) {
switch (receiveType) {
case GROUP:
CQ.sendGroupMsg(targetReceiveId, message, auto_escape);
break;
case Discuss:
CQ.sendDiscussMsg(targetReceiveId, message, auto_escape);
break;
case PRIVATE:
CQ.sendPrivateMsg(targetReceiveId, message, auto_escape);
break;
}
}
public enum ReceiveType {
PRIVATE, GROUP, Discuss
}
}

View File

@ -9,7 +9,7 @@ import java.util.Objects;
*/
public abstract class AutoSender {
private MessageSender messageSender;
private final MessageSender messageSender;
/**
* 构造一个自动发送器

View File

@ -5,6 +5,7 @@ import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonObject;
import com.google.gson.reflect.TypeToken;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.bot.message.MessageSenderBuilder;
import net.lamgc.cgj.bot.message.MessageSource;
import net.lamgc.cgj.pixiv.PixivDownload;
@ -20,11 +21,9 @@ import java.util.*;
public class BotAdminCommandProcess {
private final static Logger log = LoggerFactory.getLogger(BotAdminCommandProcess.class.getSimpleName());
private final static Logger log = LoggerFactory.getLogger(BotAdminCommandProcess.class);
private final static File globalPropFile = new File("global.properties");
private final static File pushListFile = new File("pushList.json");
private final static File pushListFile = new File(BotGlobal.getGlobal().getDataStoreDir(), "pushList.json");
private final static Hashtable<Long, JsonObject> pushInfoMap = new Hashtable<>();
@ -39,59 +38,47 @@ public class BotAdminCommandProcess {
}
@Command
public static String setGlobalProperty(@Argument(name = "key") String key, @Argument(name = "value") String value, @Argument(name = "save", force = false) boolean saveNow) {
String lastValue = BotCommandProcess.globalProp.getProperty(key);
BotCommandProcess.globalProp.setProperty(key, Strings.nullToEmpty(value));
if(saveNow) {
saveGlobalProperties();
public static String setProperty(
@Argument(name = "group", force = false) long groupId,
@Argument(name = "key") String key,
@Argument(name = "value") String value
) {
if(Strings.isNullOrEmpty(key)) {
return "未选择配置项key.";
}
return "全局配置项 " + key + " 现已设置为: " + value + " (设置前的值: " + lastValue + ")";
String lastValue = SettingProperties.setProperty(groupId, key, value.equals("null") ? null : value);
return (groupId <= 0 ? "已更改全局配置 " : "已更改群组 " + groupId + " 配置 ") +
key + " 的值: '" + value + "' (原配置值: '" + lastValue + "')";
}
@Command
public static String getGlobalProperty(@Argument(name = "key") String key) {
return "全局配置项 " + key + " 当前值: " + BotCommandProcess.globalProp.getProperty(key, "(Empty)");
public static String getProperty(
@Argument(name = "group", force = false) long groupId,
@Argument(name = "key") String key
) {
if(Strings.isNullOrEmpty(key)) {
return "未选择配置项key.";
}
return (groupId <= 0 ? "全局配置 " : "群组 " + groupId + " 配置 ") +
key + " 设定值: '" + SettingProperties.getProperty(groupId, key, "(empty)") + "'";
}
@Command
public static String saveGlobalProperties() {
log.info("正在保存全局配置文件...");
try {
if(!globalPropFile.exists()) {
if(!globalPropFile.createNewFile()) {
log.error("全局配置项文件保存失败!({})", "文件创建失败");
return "全局配置项文件保存失败!";
}
}
BotCommandProcess.globalProp.store(new FileOutputStream(globalPropFile), "");
log.info("全局配置文件保存成功!");
return "保存全局配置文件 - 操作已完成.";
} catch (IOException e) {
log.error("全局配置项文件保存失败!", e);
return "全局配置项文件保存失败!";
}
public static String saveProperties() {
log.info("正在保存配置文件...");
SettingProperties.saveProperties();
log.info("配置文件保存操作已完成.");
return "保存配置 - 操作已完成.";
}
@Command
public static String loadGlobalProperties(@Argument(name = "reload", force = false) boolean reload) {
Properties cache = new Properties();
if(!globalPropFile.exists()) {
return "未找到全局配置文件, 无法重载";
}
try(Reader reader = new BufferedReader(new FileReader(globalPropFile))) {
cache.load(reader);
} catch (IOException e) {
log.error("重载全局配置文件时发生异常", e);
return "加载全局配置文件时发生错误!";
}
public static String loadProperties(@Argument(name = "reload", force = false) boolean reload) {
if(reload) {
BotCommandProcess.globalProp.clear();
SettingProperties.clearProperties();
}
BotCommandProcess.globalProp.putAll(cache);
return "全局配置文件重载完成.";
SettingProperties.loadProperties();
return "操作已完成.";
}
@Command
@ -125,6 +112,30 @@ public class BotAdminCommandProcess {
@Argument(name = "type", force = false, defaultValue = "ILLUST") String rankingContentType,
@Argument(name = "original", force = false, defaultValue = "false") boolean original
) {
if(minTime <= 0 || floatTime <= 0) {
return "时间不能为0或负数";
} else if(rankingStart <= 0 || rankingStop - rankingStart <= 0) {
return "排行榜范围选取错误!";
}
PixivURL.RankingContentType type;
PixivURL.RankingMode mode;
try {
type = PixivURL.RankingContentType.valueOf("TYPE_" + rankingContentType.toUpperCase());
} catch(IllegalArgumentException e) {
return "无效的排行榜类型参数!";
}
try {
mode = PixivURL.RankingMode.valueOf("MODE_" + rankingMode.toUpperCase());
} catch(IllegalArgumentException e) {
return "无效的排行榜模式参数!";
}
if(!type.isSupportedMode(mode)) {
return "不兼容的排行榜模式与类型!";
}
long group = groupId <= 0 ? fromGroup : groupId;
JsonObject setting = new JsonObject();
setting.addProperty(RANKING_SETTING_TIME_MIN, minTime);
@ -141,7 +152,8 @@ public class BotAdminCommandProcess {
removePushGroup(fromGroup, groupId);
}
log.info("正在增加Timer...(Setting: {})", setting);
log.info("群组 {} 新推送配置: {}", group, setting);
log.info("正在增加Timer...");
pushInfoMap.put(group, setting);
addPushTimer(group, setting);
return "已在 " + group + " 开启定时推送功能。";
@ -171,8 +183,8 @@ public class BotAdminCommandProcess {
@Command
public static String savePushList() {
try {
if(!pushListFile.exists()) {
pushListFile.createNewFile();
if(!pushListFile.exists() && !pushListFile.createNewFile()) {
throw new IOException("文件夹创建失败!(Path: " + pushListFile.getPath() + ")");
}
} catch (IOException e) {
log.error("PushList.json文件创建失败", e);
@ -258,7 +270,8 @@ public class BotAdminCommandProcess {
}
AutoSender sender = new RandomRankingArtworksSender(
MessageSenderBuilder.getMessageSender(MessageSource.Group, id),
MessageSenderBuilder.getMessageSender(MessageSource.GROUP, id),
id,
rankingStart,
rankingEnd,
rankingMode, rankingContentType,

View File

@ -64,7 +64,7 @@ public class BotCode {
private String platformName;
private String functionName;
private Hashtable<String, String> parameter = new Hashtable<>();
private final Hashtable<String, String> parameter = new Hashtable<>();
/**
* 构造一个机器功能码

View File

@ -1,99 +1,49 @@
package net.lamgc.cgj.bot;
import com.google.common.base.Strings;
import com.google.common.base.Throwables;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import com.google.gson.*;
import io.netty.handler.codec.http.HttpHeaderNames;
import net.lamgc.cgj.Main;
import net.lamgc.cgj.bot.cache.*;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.bot.sort.PreLoadDataComparator;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.bot.cache.CacheStore;
import net.lamgc.cgj.bot.cache.CacheStoreCentral;
import net.lamgc.cgj.bot.cache.JsonRedisCacheStore;
import net.lamgc.cgj.bot.event.BufferedMessageSender;
import net.lamgc.cgj.bot.sort.PreLoadDataAttribute;
import net.lamgc.cgj.bot.sort.PreLoadDataAttributeComparator;
import net.lamgc.cgj.pixiv.PixivDownload;
import net.lamgc.cgj.pixiv.PixivSearchBuilder;
import net.lamgc.cgj.pixiv.PixivDownload.PageQuality;
import net.lamgc.cgj.pixiv.PixivSearchLinkBuilder;
import net.lamgc.cgj.pixiv.PixivURL;
import net.lamgc.cgj.util.URLs;
import net.lamgc.utils.base.runner.Argument;
import net.lamgc.utils.base.runner.Command;
import net.lamgc.utils.event.EventExecutor;
import net.lz1998.cq.utils.CQCode;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpHead;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.regex.Pattern;
@SuppressWarnings({"SameParameterValue"})
public class BotCommandProcess {
private final static PixivDownload pixivDownload = new PixivDownload(Main.cookieStore, Main.proxy);
private final static Logger log = LoggerFactory.getLogger(BotCommandProcess.class.getSimpleName());
private final static File imageStoreDir = new File(System.getProperty("cgj.cqRootDir"), "data/image/cgj/");
public final static Properties globalProp = new Properties();
private final static Gson gson = new GsonBuilder()
.serializeNulls()
.create();
private final static Hashtable<String, File> imageCache = new Hashtable<>();
private final static CacheStore<JsonElement> illustInfoCache = new JsonRedisCacheStore(BotEventHandler.redisServer, "illustInfo", gson);
private final static CacheStore<JsonElement> illustPreLoadDataCache = new HotDataCacheStore<>(
new JsonRedisCacheStore(BotEventHandler.redisServer, "illustPreLoadData", gson),
new LocalHashCacheStore<>(), 3600000, 900000);
private final static CacheStore<JsonElement> searchBodyCache = new JsonRedisCacheStore(BotEventHandler.redisServer, "searchBody", gson);
private final static CacheStore<List<JsonObject>> rankingCache = new JsonObjectRedisListCacheStore(BotEventHandler.redisServer, "ranking", gson);
private final static CacheStore<List<String>> pagesCache = new StringListRedisCacheStore(BotEventHandler.redisServer, "imagePages");
public final static CacheStore<JsonElement> reportStore = new JsonRedisCacheStore(BotEventHandler.redisServer, "report", gson);
private final static Logger log = LoggerFactory.getLogger(BotCommandProcess.class);
/**
* 图片异步缓存执行器
* 作品报告存储 - 不过期
*/
private final static EventExecutor imageCacheExecutor = new EventExecutor(new ThreadPoolExecutor(
Runtime.getRuntime().availableProcessors() >= 2 ? 2 : 1,
(int) Math.ceil(Runtime.getRuntime().availableProcessors() / 2F),
5L, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(128),
new ThreadFactoryBuilder()
.setNameFormat("imageCacheThread-%d")
.build(),
new ThreadPoolExecutor.DiscardOldestPolicy()
));
public final static CacheStore<JsonElement> reportStore =
new JsonRedisCacheStore(BotGlobal.getGlobal().getRedisServer(),
"report", BotGlobal.getGlobal().getGson());
private final static RankingUpdateTimer updateTimer = new RankingUpdateTimer();
public static void initialize() {
log.info("正在初始化...");
File globalPropFile = new File("./global.properties");
if(globalPropFile.exists() && globalPropFile.isFile()) {
log.info("正在加载全局配置文件...");
try {
globalProp.load(new FileInputStream(globalPropFile));
log.info("全局配置文件加载完成.");
} catch (IOException e) {
log.error("加载全局配置文件时发生异常", e);
}
} else {
log.info("未找到全局配置文件,跳过加载.");
}
try {
imageCacheExecutor.addHandler(new ImageCacheHandler());
} catch (IllegalAccessException e) {
e.printStackTrace();
}
SettingProperties.loadProperties();
updateTimer.schedule(null);
log.info("初始化完成.");
@ -111,77 +61,90 @@ public class BotCommandProcess {
@Command(defaultCommand = true)
public static String help() {
StringBuilder helpStrBuilder = new StringBuilder();
helpStrBuilder.append("CGJ Bot使用指南").append("\n");
helpStrBuilder.append("使用方法:.cgj <命令> [参数...]").append("\n");
helpStrBuilder.append("例如查询作品信息功能:").append("\n");
helpStrBuilder.append(".cgj artwork -id 80846159").append("\n");
helpStrBuilder.append("目前可用的命令:").append("\n");
helpStrBuilder.append("\t").append("ranking - 获取今天或指定日期排行榜的前10名作品").append("\n");
helpStrBuilder.append("\t\t").append("-date - 指定查询日期(年-月-日)").append("\n");
helpStrBuilder.append("\t\t").append("-type - 排行榜类型(illust/插画, ugoira/动图, manga/漫画)").append("\n");
helpStrBuilder.append("\t").append("search - 搜索指定关键词并显示前10个作品").append("\n");
helpStrBuilder.append("\t\t").append("-content - 搜索内容").append("\n");
helpStrBuilder.append("\t").append("link - 获取作品的Pixiv页面").append("\n");
helpStrBuilder.append("\t\t").append("-id - 作品id").append("\n");
helpStrBuilder.append("\t").append("info - 获取Pixiv作品信息").append("\n");
helpStrBuilder.append("\t\t").append("-id - 作品id").append("\n");
helpStrBuilder.append("\t").append("image - 获取指定作品的图片").append("\n");
helpStrBuilder.append("\t\t").append("-id - 作品id").append("\n");
helpStrBuilder.append("\t\t").append("-quality - 图片质量(original/原图 regular/预览图)").append("\n");
helpStrBuilder.append("\t\t").append("-page - 页数").append("\n");
helpStrBuilder.append("\t").append("report - 报告不当作品").append("\n");
helpStrBuilder.append("\t\t").append("-id - 作品Id").append("\n");
helpStrBuilder.append("\t\t").append("-msg - 报告原因").append("\n");
return helpStrBuilder.toString();
return "CGJ Bot使用指南" + "\n" +
"使用方法:.cgj <命令> [参数...]" + "\n" +
"例如查询作品信息功能:" + "\n" +
".cgj info -id 80846159" + "\n" +
"目前可用的命令:" + "\n" +
"\t" + "ranking - 获取今天或指定日期排行榜的前10名作品" + "\n" +
"\t\t" + "-date - 指定查询日期(年-月-日)" + "\n" +
"\t\t" + "-type - 排行榜类型(illust/插画, ugoira/动图, manga/漫画)" + "\n" +
"\t" + "search - 搜索指定关键词并显示前10个作品" + "\n" +
"\t\t" + "-content - 搜索内容" + "\n" +
"\t" + "link - 获取作品的Pixiv页面" + "\n" +
"\t\t" + "-id - 作品id" + "\n" +
"\t" + "info - 获取Pixiv作品信息" + "\n" +
"\t\t" + "-id - 作品id" + "\n" +
"\t" + "image - 获取指定作品的图片" + "\n" +
"\t\t" + "-id - 作品id" + "\n" +
"\t\t" + "-quality - 图片质量(original/原图 regular/预览图)" + "\n" +
"\t\t" + "-page - 页数" + "\n" +
"\t" + "report - 报告不当作品" + "\n" +
"\t\t" + "-id - 作品Id" + "\n" +
"\t\t" + "-msg - 报告原因" + "\n";
}
/**
* 作品信息查询
* @param fromGroup 来源群(系统提供)
* @param illustId 作品Id
* @return 返回作品信息
*/
@Command(commandName = "info")
public static String artworkInfo(@Argument(name = "id") int illustId) {
public static String artworkInfo(@Argument(name = "$fromGroup") long fromGroup,
@Argument(name = "id") int illustId)
throws InterruptedException {
if(illustId <= 0) {
return "错误的作品id";
return "这个作品Id是错误的!";
}
try {
if(isNoSafe(illustId, globalProp, false) || isReported(illustId)) {
if(isNoSafe(illustId, SettingProperties.getProperties(fromGroup), false) || isReported(illustId)) {
return "阅览禁止:该作品已被封印!!";
}
JsonObject illustPreLoadData = getIllustPreLoadData(illustId, false);
StringBuilder builder = new StringBuilder("---------------- 作品信息 ----------------\n");
builder.append("作品Id: ").append(illustId).append("\n");
builder.append("作品标题:").append(illustPreLoadData.get("illustTitle").getAsString()).append("\n");
builder.append("作者(作者Id)").append(illustPreLoadData.get("userName").getAsString())
.append("(").append(illustPreLoadData.get("userId").getAsInt()).append(")\n");
builder.append("点赞数:").append(illustPreLoadData.get(PreLoadDataComparator.Attribute.LIKE.attrName).getAsInt()).append("\n");
builder.append("收藏数:").append(illustPreLoadData.get(PreLoadDataComparator.Attribute.BOOKMARK.attrName).getAsInt()).append("\n");
builder.append("围观数:").append(illustPreLoadData.get(PreLoadDataComparator.Attribute.VIEW.attrName).getAsInt()).append("\n");
builder.append("评论数:").append(illustPreLoadData.get(PreLoadDataComparator.Attribute.COMMENT.attrName).getAsInt()).append("\n");
builder.append("数:").append(illustPreLoadData.get(PreLoadDataComparator.Attribute.PAGE.attrName).getAsInt()).append("\n");
builder.append("---------------- 作品图片 ----------------\n");
builder.append(getImageById(illustId, PixivDownload.PageQuality.REGULAR, 1)).append("\n");
builder.append("使用 \".cgj image -id ")
.append(illustId)
.append("\" 获取原图。\n如有不当作品使用\".cgj report -id ")
.append(illustId).append("\"向色图姬反馈。");
return builder.toString();
JsonObject illustPreLoadData = CacheStoreCentral.getCentral().getIllustPreLoadData(illustId, false);
// 在 Java 6 开始, 编译器会将用'+'进行的字符串拼接将自动转换成StringBuilder拼接
return "色图姬帮你了解了这个作品的信息!\n" + "---------------- 作品信息 ----------------" +
"\n作品Id: " + illustId +
"\n作品标题" + illustPreLoadData.get("illustTitle").getAsString() +
"\n作者(作者Id)" + illustPreLoadData.get("userName").getAsString() +
"(" + illustPreLoadData.get("userId").getAsInt() + ")" +
"\n点赞数:" + illustPreLoadData.get(PreLoadDataAttribute.LIKE.attrName).getAsInt() +
"\n收藏数" + illustPreLoadData.get(PreLoadDataAttribute.BOOKMARK.attrName).getAsInt() +
"\n围观数" + illustPreLoadData.get(PreLoadDataAttribute.VIEW.attrName).getAsInt() +
"\n评论数:" + illustPreLoadData.get(PreLoadDataAttribute.COMMENT.attrName).getAsInt() +
"\n页数" + illustPreLoadData.get(PreLoadDataAttribute.PAGE.attrName).getAsInt() + "" +
"\n作品链接" + artworksLink(fromGroup, illustId) + "\n" +
"---------------- 作品图片 ----------------\n" +
CacheStoreCentral.getCentral().getImageById(fromGroup, illustId, PageQuality.REGULAR, 1) + "\n" +
"使用 \".cgj image -id " +
illustId +
"\" 获取原图。\n如有不当作品可使用\".cgj report -id " +
illustId + "\"向色图姬反馈。";
} catch (IOException e) {
e.printStackTrace();
}
return "尚未支持";
}
/**
* 排行榜命令
* @param fromGroup 来源群(系统提供)
* @param queryTime 查询时间, 格式: 年-月-日
* @param force 是否强制查询, 当主动提供的时间不在查询范围时, 是否强制查询, 仅系统可用
* @param contentMode 内容模式
* @param contentType 排行榜类型
* @return 返回排行榜信息
*/
@Command
public static String ranking(
@Argument(name = "$fromGroup") long fromGroup,
@Argument(force = false, name = "date") Date queryTime,
@Argument(force = false, name = "force") boolean force,
@Argument(force = false, name = "mode", defaultValue = "DAILY") String contentMode,
@Argument(force = false, name = "type", defaultValue = "ILLUST") String contentType
) {
@Argument(force = false, name = "type", defaultValue = "ALL") String contentType
) throws InterruptedException {
Date queryDate = queryTime;
if (queryDate == null) {
queryDate = new Date();
@ -195,7 +158,7 @@ public class BotCommandProcess {
}
queryDate = gregorianCalendar.getTime();
} else {
if(new Date().before(queryDate)) {
if(new Date().before(queryDate) && !force) {
log.warn("查询的日期过早, 无法查询排行榜.");
return "查询日期过早, 暂未更新指定日期的排行榜!";
}
@ -204,7 +167,8 @@ public class BotCommandProcess {
PixivURL.RankingMode mode;
try {
String rankingModeValue = contentMode.toUpperCase();
mode = PixivURL.RankingMode.valueOf(rankingModeValue.startsWith("MODE_") ? rankingModeValue : "MODE_" + rankingModeValue);
mode = PixivURL.RankingMode.valueOf(rankingModeValue.startsWith("MODE_") ?
rankingModeValue : "MODE_" + rankingModeValue);
} catch (IllegalArgumentException e) {
log.warn("无效的RankingMode值: {}", contentMode);
return "参数无效, 请查看帮助信息";
@ -213,7 +177,8 @@ public class BotCommandProcess {
PixivURL.RankingContentType type;
try {
String contentTypeValue = contentType.toUpperCase();
type = PixivURL.RankingContentType.valueOf(contentTypeValue.startsWith("TYPE_") ? contentTypeValue : "TYPE_" + contentTypeValue);
type = PixivURL.RankingContentType.valueOf(
contentTypeValue.startsWith("TYPE_") ? contentTypeValue : "TYPE_" + contentTypeValue);
} catch (IllegalArgumentException e) {
log.warn("无效的RankingContentType值: {}", contentType);
return "参数无效, 请查看帮助信息";
@ -225,13 +190,15 @@ public class BotCommandProcess {
return "不支持的内容类型或模式!";
}
StringBuilder resultBuilder = new StringBuilder(mode.name() + " - 以下是 ").append(new SimpleDateFormat("yyyy-MM-dd").format(queryDate)).append(" 的Pixiv插画排名榜前十名\n");
StringBuilder resultBuilder = new StringBuilder(mode.name() + " - 以下是 ")
.append(new SimpleDateFormat("yyyy-MM-dd").format(queryDate)).append(" 的Pixiv插画排名榜前十名\n");
try {
int index = 0;
int itemLimit = 10;
String itemLimitPropertyKey = "ranking.ItemCountLimit";
String itemLimitPropertyKey = "ranking.itemCountLimit";
try {
itemLimit = Integer.parseInt(globalProp.getProperty(itemLimitPropertyKey, "10"));
itemLimit = Integer.parseInt(SettingProperties
.getProperty(fromGroup, itemLimitPropertyKey, "10"));
} catch(NumberFormatException e) {
log.warn("配置项 {} 的参数值格式有误!", itemLimitPropertyKey);
}
@ -239,13 +206,14 @@ public class BotCommandProcess {
int imageLimit = 3;
String imageLimitPropertyKey = "ranking.imageCountLimit";
try {
imageLimit = Integer.parseInt(globalProp.getProperty(imageLimitPropertyKey, "3"));
imageLimit = Integer.parseInt(
SettingProperties.getProperty(fromGroup, imageLimitPropertyKey, "3"));
} catch(NumberFormatException e) {
log.warn("配置项 {} 的参数值格式有误!", imageLimitPropertyKey);
}
//TODO(LamGC, 2020.4.11): 将JsonRedisCacheStore更改为使用Redis的List集合, 以提高性能
List<JsonObject> rankingInfoList = getRankingInfoByCache(type, mode, queryDate, 1, Math.max(0, itemLimit), false);
List<JsonObject> rankingInfoList = CacheStoreCentral.getCentral()
.getRankingInfoByCache(type, mode, queryDate, 1, Math.max(0, itemLimit), false);
if(rankingInfoList.isEmpty()) {
return "无法查询排行榜,可能排行榜尚未更新。";
}
@ -259,26 +227,72 @@ public class BotCommandProcess {
String authorName = rankInfo.get("user_name").getAsString();
String title = rankInfo.get("title").getAsString();
resultBuilder.append(rank).append(". (id: ").append(illustId).append(") ").append(title)
.append("(Author: ").append(authorName).append(",").append(authorId).append(") ").append(pagesCount).append("p.\n");
.append("(Author: ").append(authorName).append(",").append(authorId).append(") ")
.append(pagesCount).append("p.\n");
if (index <= imageLimit) {
resultBuilder.append(getImageById(illustId, PixivDownload.PageQuality.REGULAR, 1)).append("\n");
resultBuilder
.append(CacheStoreCentral.getCentral()
.getImageById(fromGroup, illustId, PixivDownload.PageQuality.REGULAR, 1))
.append("\n");
}
}
} catch (IOException e) {
log.error("消息处理异常", e);
return "排名榜获取失败!详情请查看机器人控制台。";
}
return resultBuilder.append("如查询当前时间获取到昨天时间,则今日排名榜未更新。\n如有不当作品,可使用\".cgj report -id 作品id\"向色图姬反馈。").toString();
return resultBuilder.append("如查询当前时间获取到昨天时间,则今日排名榜未更新。\n" +
"如有不当作品,可使用\".cgj report -id 作品id\"向色图姬反馈。").toString();
}
/**
* 查询指定作者的作品(尚未完成)
* @return 返回作者信息和部分作品
*/
@Command(commandName = "userArt")
public static String userArtworks() {
return "功能未完成";
}
/**
* 随机获取一副作品
*/
@Command(commandName = "random")
public static String randomImage(
@Argument(name = "$fromGroup") long fromGroup,
@Argument(force = false, name = "mode", defaultValue = "DAILY") String contentMode,
@Argument(force = false, name = "type", defaultValue = "ILLUST") String contentType) {
PixivURL.RankingMode mode;
try {
String rankingModeValue = contentMode.toUpperCase();
mode = PixivURL.RankingMode.valueOf(rankingModeValue.startsWith("MODE_") ?
rankingModeValue : "MODE_" + rankingModeValue);
} catch (IllegalArgumentException e) {
log.warn("无效的RankingMode值: {}", contentMode);
return "参数无效, 请查看帮助信息";
}
PixivURL.RankingContentType type;
try {
String contentTypeValue = contentType.toUpperCase();
type = PixivURL.RankingContentType.valueOf(
contentTypeValue.startsWith("TYPE_") ? contentTypeValue : "TYPE_" + contentTypeValue);
} catch (IllegalArgumentException e) {
log.warn("无效的RankingContentType值: {}", contentType);
return "参数无效, 请查看帮助信息";
}
BufferedMessageSender bufferedSender = new BufferedMessageSender();
RandomRankingArtworksSender artworksSender =
new RandomRankingArtworksSender(bufferedSender, fromGroup, 1, 200, mode, type,
PageQuality.ORIGINAL);
artworksSender.send();
return bufferedSender.getBufferContent();
}
/**
* 搜索命令
* @param fromGroup 来源群(系统提供)
* @param content 搜索内容
* @param type 搜索类型
* @param area 搜索区域
@ -290,107 +304,33 @@ public class BotCommandProcess {
* @throws IOException 当搜索发生异常时抛出
*/
@Command
public static String search(@Argument(name = "content") String content,
@Argument(name = "type", force = false) String type,
@Argument(name = "area", force = false) String area,
@Argument(name = "in", force = false) String includeKeywords,
@Argument(name = "ex", force = false) String excludeKeywords,
@Argument(name = "contentOption", force = false) String contentOption,
@Argument(name = "page", force = false, defaultValue = "1") int pagesIndex
) throws IOException {
log.info("正在执行搜索...");
PixivSearchBuilder searchBuilder = new PixivSearchBuilder(Strings.isNullOrEmpty(content) ? "" : content);
if (type != null) {
try {
searchBuilder.setSearchType(PixivSearchBuilder.SearchType.valueOf(type.toUpperCase()));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchType: {}", type);
}
}
if (area != null) {
try {
searchBuilder.setSearchArea(PixivSearchBuilder.SearchArea.valueOf(area));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchArea: {}", area);
}
}
if (contentOption != null) {
try {
searchBuilder.setSearchContentOption(PixivSearchBuilder.SearchContentOption.valueOf(contentOption));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchContentOption: {}", contentOption);
}
}
if (!Strings.isNullOrEmpty(includeKeywords)) {
for (String keyword : includeKeywords.split(";")) {
searchBuilder.removeExcludeKeyword(keyword.trim());
searchBuilder.addIncludeKeyword(keyword.trim());
log.debug("已添加关键字: {}", keyword);
}
}
if (!Strings.isNullOrEmpty(excludeKeywords)) {
for (String keyword : excludeKeywords.split(";")) {
searchBuilder.removeIncludeKeyword(keyword.trim());
searchBuilder.addExcludeKeyword(keyword.trim());
log.debug("已添加排除关键字: {}", keyword);
}
}
log.info("正在搜索作品, 条件: {}", searchBuilder.getSearchCondition());
String requestUrl = searchBuilder.buildURL().intern();
log.debug("RequestUrl: {}", requestUrl);
JsonObject resultBody = null;
if(!searchBodyCache.exists(requestUrl)) {
synchronized (requestUrl) {
if (!searchBodyCache.exists(requestUrl)) {
log.debug("searchBody缓存失效, 正在更新...");
JsonObject jsonObject;
HttpGet httpGetRequest = pixivDownload.createHttpGetRequest(requestUrl);
HttpResponse response = pixivDownload.getHttpClient().execute(httpGetRequest);
String responseBody = EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
log.debug("ResponseBody: {}", responseBody);
jsonObject = gson.fromJson(responseBody, JsonObject.class);
if (jsonObject.get("error").getAsBoolean()) {
log.error("接口请求错误, 错误信息: {}", jsonObject.get("message").getAsString());
return "处理命令时发生错误!";
}
long expire = 7200 * 1000;
String propValue = globalProp.getProperty("cache.searchBody.expire", "7200000");
try {
expire = Long.parseLong(propValue);
} catch (Exception e) {
log.warn("全局配置项 \"{}\" 值非法, 已使用默认值: {}", propValue, expire);
}
resultBody = jsonObject.getAsJsonObject().getAsJsonObject("body");
searchBodyCache.update(requestUrl, jsonObject, expire);
log.debug("searchBody缓存已更新(有效时间: {})", expire);
} else {
log.debug("搜索缓存命中.");
}
}
} else {
log.debug("搜索缓存命中.");
}
if(Objects.isNull(resultBody)) {
resultBody = searchBodyCache.getCache(requestUrl).getAsJsonObject().getAsJsonObject("body");
}
public static String search(
@Argument(name = "$fromGroup") long fromGroup,
@Argument(name = "content") String content,
@Argument(name = "type", force = false) String type,
@Argument(name = "area", force = false) String area,
@Argument(name = "in", force = false) String includeKeywords,
@Argument(name = "ex", force = false) String excludeKeywords,
@Argument(name = "option", force = false) String contentOption,
@Argument(name = "p", force = false, defaultValue = "1") int pagesIndex
) throws IOException, InterruptedException {
log.debug("正在执行搜索...");
JsonObject resultBody = CacheStoreCentral.getCentral()
.getSearchBody(content, type, area, includeKeywords, excludeKeywords, contentOption, pagesIndex);
StringBuilder result = new StringBuilder("内容 " + content + " 的搜索结果:\n");
log.debug("正在处理信息...");
int limit = 8;
try {
limit = Integer.parseInt(globalProp.getProperty("search.ItemCountLimit", "8"));
limit = Integer.parseInt(SettingProperties.
getProperty(fromGroup, "search.itemCountLimit", "8"));
} catch (Exception e) {
log.warn("参数转换异常!将使用默认值(" + limit + ")", e);
}
for (PixivSearchBuilder.SearchArea searchArea : PixivSearchBuilder.SearchArea.values()) {
if (!resultBody.has(searchArea.jsonKey) || resultBody.getAsJsonObject(searchArea.jsonKey).getAsJsonArray("data").size() == 0) {
int totalCount = 0;
for (PixivSearchLinkBuilder.SearchArea searchArea : PixivSearchLinkBuilder.SearchArea.values()) {
if (!resultBody.has(searchArea.jsonKey) ||
resultBody.getAsJsonObject(searchArea.jsonKey).getAsJsonArray("data").size() == 0) {
log.debug("返回数据不包含 {}", searchArea.jsonKey);
continue;
}
@ -398,9 +338,9 @@ public class BotCommandProcess {
.getAsJsonObject(searchArea.jsonKey).getAsJsonArray("data");
ArrayList<JsonElement> illustsList = new ArrayList<>();
illustsArray.forEach(illustsList::add);
illustsList.sort(new PreLoadDataComparator(PreLoadDataComparator.Attribute.LIKE));
illustsList.sort(new PreLoadDataAttributeComparator(PreLoadDataAttribute.BOOKMARK));
log.info("已找到与 {} 相关插图信息({})", content, searchArea.name().toLowerCase());
log.debug("已找到与 {} 相关插图信息({})", content, searchArea.name().toLowerCase());
int count = 1;
for (JsonElement jsonElement : illustsList) {
if (count > limit) {
@ -414,7 +354,8 @@ public class BotCommandProcess {
StringBuilder builder = new StringBuilder("[");
illustObj.get("tags").getAsJsonArray().forEach(el -> builder.append(el.getAsString()).append(", "));
builder.replace(builder.length() - 2, builder.length(), "]");
log.debug("{} ({} / {})\n\t作品id: {}, \n\t作者名(作者id): {} ({}), \n\t作品标题: {}, \n\t作品Tags: {}, \n\t页数: {}, \n\t作品链接: {}",
log.debug("{} ({} / {})\n\t作品id: {}, \n\t作者名(作者id): {} ({}), \n\t" +
"作品标题: {}, \n\t作品Tags: {}, \n\t页数: {}页, \n\t作品链接: {}",
searchArea.name(),
count,
illustsList.size(),
@ -427,10 +368,18 @@ public class BotCommandProcess {
PixivURL.getPixivRefererLink(illustId)
);
//pageCount
String imageMsg = getImageById(illustId, PixivDownload.PageQuality.REGULAR, 1);
if (isNoSafe(illustId, globalProp, true)) {
String imageMsg;
try {
imageMsg = CacheStoreCentral.getCentral()
.getImageById(fromGroup, illustId, PixivDownload.PageQuality.REGULAR, 1);
} catch (NoSuchElementException e) {
if(e.getMessage().startsWith("No work found: ")) {
log.warn("作品 {} 不存在, 跳过该作品...", illustId);
continue;
}
throw e;
}
if (isNoSafe(illustId, SettingProperties.getProperties(fromGroup), false)) {
log.warn("作品Id {} 为R-18作品, 跳过.", illustId);
continue;
} else if(isReported(illustId)) {
@ -438,25 +387,58 @@ public class BotCommandProcess {
continue;
}
result.append(searchArea.name()).append(" (").append(count).append(" / ").append(limit).append(")\n\t作品id: ").append(illustId)
JsonObject illustPreLoadData = CacheStoreCentral.getCentral()
.getIllustPreLoadData(illustId, false);
result.append(searchArea.name()).append(" (").append(count).append(" / ")
.append(limit).append(")\n\t作品id: ").append(illustId)
.append(", \n\t作者名: ").append(illustObj.get("userName").getAsString())
.append("\n\t作品标题: ").append(illustObj.get("illustTitle").getAsString())
.append("\n\t作品页数: ").append(illustObj.get("pageCount").getAsInt())
.append("\n\t作品页数: ").append(illustObj.get("pageCount").getAsInt()).append("")
.append("\n\t点赞数")
.append(illustPreLoadData.get(PreLoadDataAttribute.LIKE.attrName).getAsInt())
.append("\n\t收藏数")
.append(illustPreLoadData.get(PreLoadDataAttribute.BOOKMARK.attrName).getAsInt())
.append("\n\t围观数")
.append(illustPreLoadData.get(PreLoadDataAttribute.VIEW.attrName).getAsInt())
.append("\n\t评论数")
.append(illustPreLoadData.get(PreLoadDataAttribute.COMMENT.attrName).getAsInt())
.append("\n").append(imageMsg).append("\n");
count++;
totalCount++;
}
if (count > limit) {
break;
}
}
return Strings.nullToEmpty(result.toString()) + "预览图片并非原图,使用“.cgj image -id 作品id”获取原图\n如有不当作品可使用\".cgj report -id 作品id\"向色图姬反馈。";
return totalCount <= 0 ?
"搜索完成,未找到相关作品。" :
Strings.nullToEmpty(result.toString()) + "预览图片并非原图,使用“.cgj image -id 作品id”获取原图\n" +
"如有不当作品,可使用\".cgj report -id 作品id\"向色图姬反馈。";
}
/**
* 获取作品页面的下载链接
* @param illustId 作品Id
* @param quality 画质类型
* @return 返回作品所有页面在Pixiv的下载链接(有防盗链, 考虑要不要设置镜像站)
*/
@Command(commandName = "pages")
public static String getPagesList(@Argument(name = "id") int illustId, @Argument(name = "quality", force = false) PixivDownload.PageQuality quality) {
public static String getPagesList(
@Argument(name = "$fromGroup") long fromGroup,
@Argument(name = "id") int illustId,
@Argument(name = "quality", force = false) PixivDownload.PageQuality quality) {
try {
List<String> pagesList = PixivDownload.getIllustAllPageDownload(pixivDownload.getHttpClient(), pixivDownload.getCookieStore(), illustId, quality);
StringBuilder builder = new StringBuilder("作品ID ").append(illustId).append(" 共有").append(pagesList.size()).append("页:").append("\n");
if(isNoSafe(illustId, SettingProperties.getProperties(fromGroup), false)) {
log.warn("来源群 {} 查询的作品Id {} 为R18作品, 根据配置设定, 屏蔽该作品.", fromGroup, illustId);
return "该作品已被封印!";
}
List<String> pagesList =
PixivDownload.getIllustAllPageDownload(
BotGlobal.getGlobal().getPixivDownload().getHttpClient(),
BotGlobal.getGlobal().getPixivDownload().getCookieStore(),
illustId, quality);
StringBuilder builder = new StringBuilder("作品ID ").append(illustId)
.append(" 共有").append(pagesList.size()).append("页:").append("\n");
int index = 0;
for (String link : pagesList) {
builder.append("Page ").append(++index).append(": ").append(link).append("\n");
@ -468,10 +450,17 @@ public class BotCommandProcess {
}
}
/**
* 获取作品链接
* @param fromGroup 来源群(系统提供)
* @param illustId 作品Id
* @return 返回作品在Pixiv的链接
*/
@Command(commandName = "link")
public static String artworksLink(@Argument(name = "id") int illustId) {
public static String artworksLink(@Argument(name = "$fromGroup") long fromGroup,
@Argument(name = "id") int illustId) {
try {
if (isNoSafe(illustId, globalProp, false)) {
if (isNoSafe(illustId, SettingProperties.getProperties(fromGroup), false)) {
log.warn("作品Id {} 已被屏蔽.", illustId);
return "由于相关设置,该作品已被屏蔽!";
} else if(isReported(illustId)) {
@ -485,112 +474,10 @@ public class BotCommandProcess {
return PixivURL.getPixivRefererLink(illustId);
}
/**
* 通过illustId获取作品图片
* @param illustId 作品Id
* @param quality 图片质量
* @param pageIndex 指定页面索引, 从1开始
* @return 如果成功, 返回BotCode, 否则返回错误信息.
*/
@Command(commandName = "image")
public static String getImageById(@Argument(name = "id") int illustId,
@Argument(name = "quality", force = false) PixivDownload.PageQuality quality,
@Argument(name = "page", force = false, defaultValue = "1") int pageIndex) {
log.debug("IllustId: {}, Quality: {}, PageIndex: {}", illustId, quality.name(), pageIndex);
List<String> pagesList;
try {
pagesList = getIllustPages(illustId, quality, false);
} catch (IOException e) {
log.error("获取下载链接列表时发生异常", e);
return "发生网络异常,无法获取图片!";
}
if(log.isDebugEnabled()) {
StringBuilder logBuilder = new StringBuilder("作品Id {} 所有页面下载链接: \n");
AtomicInteger index = new AtomicInteger();
pagesList.forEach(item -> logBuilder.append(index.incrementAndGet()).append(". ").append(item).append("\n"));
log.debug(logBuilder.toString());
}
if (pagesList.size() < pageIndex || pageIndex <= 0) {
log.warn("指定的页数超出了总页数({} / {})", pageIndex, pagesList.size());
return "指定的页数超出了范围(总共 " + pagesList.size() + " 页)";
}
try {
if (isNoSafe(illustId, globalProp, false)) {
log.warn("作品 {} 存在R-18内容且设置\"image.allowR18\"为false将屏蔽该作品不发送.", illustId);
return "(根据设置,该作品已被屏蔽!)";
} else if(isReported(illustId)) {
log.warn("作品Id {} 被报告, 正在等待审核, 跳过该作品.", illustId);
return "(该作品已被封印)";
}
} catch (IOException e) {
log.warn("作品信息无法获取!", e);
return "发生网络异常,无法获取图片!";
}
String downloadLink = pagesList.get(pageIndex - 1);
String fileName = URLs.getResourceName(Strings.nullToEmpty(downloadLink));
File imageFile = new File(getImageStoreDir(), downloadLink.substring(downloadLink.lastIndexOf("/") + 1));
log.debug("FileName: {}, DownloadLink: {}", fileName, downloadLink);
if(!imageCache.containsKey(fileName)) {
if(imageFile.exists()) {
HttpHead headRequest = new HttpHead(downloadLink);
headRequest.addHeader("Referer", PixivURL.getPixivRefererLink(illustId));
HttpResponse headResponse;
try {
headResponse = pixivDownload.getHttpClient().execute(headRequest);
} catch (IOException e) {
log.error("获取图片大小失败!", e);
return "图片获取失败!";
}
String contentLengthStr = headResponse.getFirstHeader(HttpHeaderNames.CONTENT_LENGTH.toString()).getValue();
log.debug("图片大小: {}B", contentLengthStr);
if (imageFile.length() == Long.parseLong(contentLengthStr)) {
imageCache.put(URLs.getResourceName(downloadLink), imageFile);
log.debug("作品Id {} 第 {} 页缓存已补充.", illustId, pageIndex);
return getImageToBotCode(imageFile, false).toString();
}
}
ImageCacheObject taskObject = new ImageCacheObject(imageCache, illustId, downloadLink, imageFile);
try {
imageCacheExecutor.executorSync(taskObject);
} catch (InterruptedException e) {
log.error("等待图片下载时发生中断", e);
return "图片获取失败!";
}
} else {
log.debug("图片 {} 缓存命中.", fileName);
}
return getImageToBotCode(imageCache.get(fileName), false).toString();
}
/**
* 通过文件获取图片的BotCode代码
* @param targetFile 图片文件
* @param updateCache 是否刷新缓存(只是让机器人重新上传, 如果上传接口有重复检测的话是无法处理的)
* @return 返回设定好参数的BotCode
*/
private static BotCode getImageToBotCode(File targetFile, boolean updateCache) {
String fileName = targetFile.getName();
BotCode code = BotCode.parse(CQCode.image(getImageStoreDir().getName() + "/" + fileName));
code.addParameter("absolutePath", targetFile.getAbsolutePath());
code.addParameter("imageName", fileName.substring(0, fileName.lastIndexOf(".")));
code.addParameter("updateCache", updateCache ? "true" : "false");
return code;
}
static void clearCache() {
log.warn("正在清除所有缓存...");
imageCache.clear();
illustInfoCache.clear();
illustPreLoadDataCache.clear();
pagesCache.clear();
searchBodyCache.clear();
File imageStoreDir = new File(System.getProperty("cgj.cqRootDir") + "data/image/cgj/");
CacheStoreCentral.getCentral().clearCache();
File imageStoreDir = new File(BotGlobal.getGlobal().getDataStoreDir(), "data/image/cgj/");
File[] listFiles = imageStoreDir.listFiles();
if (listFiles == null) {
log.debug("图片缓存目录为空或内部文件获取失败!");
@ -603,14 +490,30 @@ public class BotCommandProcess {
log.warn("缓存删除完成.");
}
@Command(commandName = "image")
public static String getImageById(
@Argument(name = "$fromGroup") long fromGroup,
@Argument(name = "id") int illustId,
@Argument(name = "quality", force = false) PixivDownload.PageQuality quality,
@Argument(name = "p", force = false, defaultValue = "1") int pageIndex
) throws InterruptedException {
return CacheStoreCentral.getCentral().getImageById(fromGroup, illustId, quality, pageIndex);
}
/**
* 举报某一作品
* @param fromGroup 来源群(系统提供)
* @param illustId 需要举报的作品id
* @param reason 举报原因
* @return 返回提示信息
*/
@Command
public static String report(@Argument(name = "$fromGroup") long fromGroup, @Argument(name = "$fromQQ") long fromQQ, @Argument(name = "id") int illustId, @Argument(name = "msg", force = false) String reason) {
public static String report(
@Argument(name = "$fromGroup") long fromGroup,
@Argument(name = "$fromQQ") long fromQQ,
@Argument(name = "id") int illustId,
@Argument(name = "msg", force = false) String reason
) {
log.warn("收到作品反馈(IllustId: {}, 原因: {})", illustId, reason);
JsonObject reportJson = new JsonObject();
reportJson.addProperty("illustId", illustId);
@ -631,13 +534,10 @@ public class BotCommandProcess {
return reportStore.exists(String.valueOf(illustId));
}
/*
下一目标:
添加定时发图
定时发图支持设置关注标签
标签....标签支持搜索吧
/**
* Tag过滤表达式
*/
private final static Pattern tagPattern = Pattern.compile(".*R-*18.*");
/**
* 检查指定作品是否为r18
* @param illustId 作品Id
@ -645,160 +545,25 @@ public class BotCommandProcess {
* @param returnRaw 是否返回原始值
* @return 如果为true, 则不为全年龄
* @throws IOException 获取数据时发生异常时抛出
* @throws NoSuchElementException 当作品不存在时抛出
*/
public static boolean isNoSafe(int illustId, Properties settingProp, boolean returnRaw) throws IOException {
boolean rawValue = getIllustInfo(illustId, false).getAsJsonArray("tags").contains(new JsonPrimitive("R-18"));
return returnRaw || settingProp == null ? rawValue : rawValue && !settingProp.getProperty("image.allowR18", "false").equalsIgnoreCase("true");
}
private static JsonObject getIllustInfo(int illustId, boolean flushCache) throws IOException {
String illustIdStr = buildSyncKey(Integer.toString(illustId));
JsonObject illustInfoObj = null;
if (!illustInfoCache.exists(illustIdStr) || flushCache) {
synchronized (illustIdStr) { // TODO: 这里要不做成HashMap存储key而避免使用常量池?
if (!illustInfoCache.exists(illustIdStr) || flushCache) {
illustInfoObj = pixivDownload.getIllustInfoByIllustId(illustId);
illustInfoCache.update(illustIdStr, illustInfoObj, null);
public static boolean isNoSafe(int illustId, Properties settingProp, boolean returnRaw)
throws IOException, NoSuchElementException {
JsonObject illustInfo = CacheStoreCentral.getCentral().getIllustInfo(illustId, false);
JsonArray tags = illustInfo.getAsJsonArray("tags");
boolean rawValue = illustInfo.get("xRestrict").getAsInt() != 0;
if(!rawValue) {
for(JsonElement tag : tags) {
boolean current = tagPattern.matcher(tag.getAsString()).matches();
if (current) {
rawValue = true;
break;
}
}
}
if(Objects.isNull(illustInfoObj)) {
illustInfoObj = illustInfoCache.getCache(illustIdStr).getAsJsonObject();
log.debug("作品Id {} IllustInfo缓存命中.", illustId);
}
return illustInfoObj;
return returnRaw || settingProp == null ? rawValue :
rawValue && !settingProp.getProperty("image.allowR18", "false")
.equalsIgnoreCase("true");
}
/**
* 获取作品预加载数据.
* 可以获取作品的一些与用户相关的信息
* @param illustId 作品Id
* @param flushCache 是否刷新缓存
* @return 成功返回JsonObject对象
* @throws IOException 当Http请求处理发生异常时抛出
*/
public static JsonObject getIllustPreLoadData(int illustId, boolean flushCache) throws IOException {
String illustIdStr = buildSyncKey(Integer.toString(illustId));
JsonObject result = null;
if (!illustPreLoadDataCache.exists(illustIdStr) || flushCache) {
synchronized (illustIdStr) {
if (!illustPreLoadDataCache.exists(illustIdStr) || flushCache) {
log.debug("IllustId {} 缓存失效, 正在更新...", illustId);
JsonObject preLoadDataObj = pixivDownload.getIllustPreLoadDataById(illustId)
.getAsJsonObject("illust")
.getAsJsonObject(Integer.toString(illustId));
long expire = 7200 * 1000;
String propValue = globalProp.getProperty("cache.illustPreLoadData.expire", "7200000");
log.debug("PreLoadData有效时间设定: {}", propValue);
try {
expire = Long.parseLong(propValue);
} catch (Exception e) {
log.warn("全局配置项 \"{}\" 值非法, 已使用默认值: {}", propValue, expire);
}
result = preLoadDataObj;
illustPreLoadDataCache.update(illustIdStr, preLoadDataObj, expire);
log.debug("作品Id {} preLoadData缓存已更新(有效时间: {})", illustId, expire);
}
}
}
if(Objects.isNull(result)) {
result = illustPreLoadDataCache.getCache(illustIdStr).getAsJsonObject();
log.debug("作品Id {} PreLoadData缓存命中.", illustId);
}
return result;
}
public static List<String> getIllustPages(int illustId, PixivDownload.PageQuality quality, boolean flushCache) throws IOException {
String pagesSign = buildSyncKey(Integer.toString(illustId), ".", quality.name());
List<String> result = null;
if (!pagesCache.exists(pagesSign) || flushCache) {
synchronized (pagesSign) {
if (!pagesCache.exists(pagesSign) || flushCache) {
List<String> linkList = PixivDownload.getIllustAllPageDownload(pixivDownload.getHttpClient(), pixivDownload.getCookieStore(), illustId, quality);
result = linkList;
pagesCache.update(pagesSign, linkList, null);
}
}
}
if(Objects.isNull(result)) {
result = pagesCache.getCache(pagesSign);
log.debug("作品Id {} Pages缓存命中.", illustId);
}
return result;
}
private static File getImageStoreDir() {
if(!imageStoreDir.exists() && !imageStoreDir.mkdirs()) {
log.warn("酷Q图片缓存目录失效(Path: {} )", imageStoreDir.getAbsolutePath());
throw new RuntimeException(new IOException("文件夹创建失败!"));
}
return imageStoreDir;
}
private final static Random expireTimeFloatRandom = new Random();
/**
* 获取排行榜
* @param contentType 排行榜类型
* @param mode 排行榜模式
* @param queryDate 查询时间
* @param start 开始排名, 从1开始
* @param range 取范围
* @param flushCache 是否强制刷新缓存
* @return 成功返回有值List, 失败且无异常返回空
* @throws IOException 获取异常时抛出
*/
public static List<JsonObject> getRankingInfoByCache(PixivURL.RankingContentType contentType, PixivURL.RankingMode mode, Date queryDate, int start, int range, boolean flushCache) throws IOException {
if(!contentType.isSupportedMode(mode)) {
log.warn("试图获取不支持的排行榜类型已拒绝.(ContentType: {}, RankingMode: {})", contentType.name(), mode.name());
if(log.isDebugEnabled()) {
try {
Thread.dumpStack();
} catch(Exception e) {
log.debug("本次非法请求的堆栈信息如下: \n{}", Throwables.getStackTraceAsString(e));
}
}
return new ArrayList<>(0);
}
String date = new SimpleDateFormat("yyyyMMdd").format(queryDate);
String requestSign = buildSyncKey(contentType.name(), ".", mode.name(), ".", date);
List<JsonObject> result = null;
if(!rankingCache.exists(requestSign) || flushCache) {
synchronized(requestSign) {
if(!rankingCache.exists(requestSign) || flushCache) {
log.debug("Ranking缓存失效, 正在更新...(RequestSign: {})", requestSign);
List<JsonObject> rankingResult = pixivDownload.getRanking(contentType, mode, queryDate, 1, 500);
long expireTime = 0;
if(rankingResult.size() == 0) {
expireTime = 5400000 + expireTimeFloatRandom.nextInt(1800000);
log.warn("数据获取失败, 将设置浮动有效时间以准备下次更新. (ExpireTime: {}ms)", expireTime);
}
result = new ArrayList<>(rankingResult).subList(start - 1, start + range - 1);
rankingCache.update(requestSign, rankingResult, expireTime);
log.debug("Ranking缓存更新完成.(RequestSign: {})", requestSign);
}
}
}
if (Objects.isNull(result)) {
result = rankingCache.getCache(requestSign, start - 1, range);
log.debug("RequestSign [{}] 缓存命中.", requestSign);
}
log.debug("Result-Length: {}", result.size());
return PixivDownload.getRanking(result, start - 1, range);
}
private static String buildSyncKey(String... keys) {
StringBuilder sb = new StringBuilder();
for (String string : keys) {
sb.append(string);
}
return sb.toString().intern();
}
}

View File

@ -0,0 +1,83 @@
package net.lamgc.cgj.bot;
import net.lamgc.cgj.bot.event.MessageEvent;
import net.lamgc.cgj.bot.event.VirtualLoadMessageEvent;
import net.lamgc.utils.event.EventExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Properties;
/**
* 消息事件处理调试器.
* <p>当启用了消息事件处理调试后, 将会根据调试器代号调用指定调试器</p>
*/
@SuppressWarnings("unused")
public enum MessageEventExecutionDebugger {
/**
* PM - 压力测试
*/
PM ((executor, event, properties, log) -> {
MessageEvent virtualLoadEvent = VirtualLoadMessageEvent.toVirtualLoadMessageEvent(event, false);
int rotation = 5;
int number = 50;
int interval = 2500;
try {
rotation = Integer.parseInt(properties.getProperty("debug.pm.rotation", "5"));
} catch(NumberFormatException e) {
log.warn("配置项 {} 值无效, 将使用默认值.({})", "debug.pm.rotation", rotation);
}
try {
number = Integer.parseInt(properties.getProperty("debug.pm.number", "50"));
} catch(NumberFormatException e) {
log.warn("配置项 {} 值无效, 将使用默认值.({})", "debug.pm.number", number);
}
try {
interval = Integer.parseInt(properties.getProperty("debug.pm.interval", "2500"));
} catch(NumberFormatException e) {
log.warn("配置项 {} 值无效, 将使用默认值.({})", "debug.pm.interval", interval);
}
boolean interrupted = false;
Thread currentThread = Thread.currentThread();
for(int rotationCount = 0; rotationCount < rotation && !interrupted; rotationCount++) {
for(int sendCount = 0; sendCount < number; sendCount++) {
if(currentThread.isInterrupted()) {
interrupted = true;
break;
}
executor.executor(virtualLoadEvent);
}
try {
Thread.sleep(interval);
} catch (InterruptedException e) {
break;
}
}
});
public final MessageExecuteDebugger debugger;
MessageEventExecutionDebugger(MessageExecuteDebugger debugger) {
this.debugger = debugger;
}
public static Logger getDebuggerLogger(MessageEventExecutionDebugger debugger) {
return LoggerFactory.getLogger(MessageEventExecutionDebugger.class.getName() + "." + debugger.name());
}
@FunctionalInterface
public interface MessageExecuteDebugger {
/**
* 接收事件并根据指定需求处理
* @param executor 事件执行器
* @param event 消息事件对象
* @param properties 配置项, 调试器应按'debug.[debuggerName].'为前缀存储相应调试信息
* @throws Exception 当抛出异常则打断调试, 并输出至日志
*/
void accept(EventExecutor executor, MessageEvent event, Properties properties, Logger logger) throws Exception;
}
}

View File

@ -5,8 +5,6 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.lang.reflect.Field;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.atomic.AtomicBoolean;
@ -15,8 +13,8 @@ import java.util.concurrent.atomic.AtomicBoolean;
*/
public class RandomIntervalSendTimer extends TimerTask {
private final static Timer timer = new Timer("Thread-RIST");
private final static Logger log = LoggerFactory.getLogger("RandomIntervalSendTimer");
private final static Timer timer = new Timer("Thread-RandomIntervalSendTimer", true);
private final static Logger log = LoggerFactory.getLogger(RandomIntervalSendTimer.class);
private final static Map<Long, RandomIntervalSendTimer> timerMap = new HashMap<>();
private final long timerId;
@ -24,8 +22,10 @@ public class RandomIntervalSendTimer extends TimerTask {
private final AutoSender sender;
private final long time;
private final int floatTime;
private AtomicBoolean loop = new AtomicBoolean();
private final AtomicBoolean loop = new AtomicBoolean();
private final AtomicBoolean start = new AtomicBoolean();
private final String hashId = Integer.toHexString(this.hashCode());
/**
* 创建一个随机延迟发送器
@ -77,7 +77,13 @@ public class RandomIntervalSendTimer extends TimerTask {
* @param startNow 现在开始
* @param loop 是否循环
*/
private RandomIntervalSendTimer(long timerId, AutoSender sender, long time, int floatTime, boolean startNow, boolean loop) {
private RandomIntervalSendTimer(
long timerId,
AutoSender sender,
long time,
int floatTime,
boolean startNow,
boolean loop) {
this.timerId = timerId;
this.sender = sender;
this.time = time;
@ -92,18 +98,16 @@ public class RandomIntervalSendTimer extends TimerTask {
start(this.loop.get());
}
private final static DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-ss HH:mm:ss");
/**
* 启动定时器
* @param loop 是否循环, 如果为true, 则任务完成后会自动调用start方法继续循环, 直到被调用{@code #}或总定时器被销毁;
*/
public void start(boolean loop) {
this.loop.set(loop);
long nextDelay = time + timeRandom.nextInt(floatTime);
long nextDelay = time + (floatTime <= 0 ? 0 : timeRandom.nextInt(floatTime));
Date nextDate = new Date();
nextDate.setTime(nextDate.getTime() + nextDelay);
log.info("定时器 {} 下一延迟: {}ms ({})", Integer.toHexString(this.hashCode()), nextDelay, dateFormat.format(nextDate));
log.info("定时器 {} 下一延迟: {}ms ({})", hashId, nextDelay, nextDate);
if(start.get()) {
try {
Field state = this.getClass().getSuperclass().getDeclaredField("state");
@ -121,11 +125,15 @@ public class RandomIntervalSendTimer extends TimerTask {
@Override
public void run() {
log.info("定时器 {} 开始执行...(Sender: {}@{})", this.hashId, sender.getClass().getSimpleName(), sender.hashCode());
try {
sender.send();
} catch (Exception e) {
log.error("定时器 {} 执行时发生异常:\n{}", Integer.toHexString(this.hashCode()), Throwables.getStackTraceAsString(e));
log.error("定时器 {} 执行时发生异常:\n{}",
Integer.toHexString(this.hashCode()),
Throwables.getStackTraceAsString(e));
}
log.info("定时器 {} 执行结束.", this.hashId);
if (this.loop.get()) {
start();
}

View File

@ -1,13 +1,13 @@
package net.lamgc.cgj.bot;
import com.google.gson.JsonObject;
import net.lamgc.cgj.bot.cache.CacheStoreCentral;
import net.lamgc.cgj.bot.message.MessageSender;
import net.lamgc.cgj.pixiv.PixivDownload;
import net.lamgc.cgj.pixiv.PixivURL;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Calendar;
import java.util.Date;
import java.util.List;
@ -19,6 +19,7 @@ import java.util.Random;
public class RandomRankingArtworksSender extends AutoSender {
private final Logger log;
private final long groupId;
private final int rankingStart;
private final int rankingStop;
private final PixivURL.RankingMode mode;
@ -35,11 +36,40 @@ public class RandomRankingArtworksSender extends AutoSender {
* @param quality 图片质量, 详见{@link PixivDownload.PageQuality}
* @throws IndexOutOfBoundsException 当 rankingStart > rankingStop时抛出
*/
public RandomRankingArtworksSender(MessageSender messageSender, int rankingStart, int rankingStop, PixivURL.RankingMode mode, PixivURL.RankingContentType contentType, PixivDownload.PageQuality quality) {
public RandomRankingArtworksSender(
MessageSender messageSender,
int rankingStart,
int rankingStop,
PixivURL.RankingMode mode,
PixivURL.RankingContentType contentType,
PixivDownload.PageQuality quality) {
this(messageSender, 0, rankingStart, rankingStop, mode, contentType, quality);
}
/**
* 构造一个推荐作品发送器
* @param messageSender 消息发送器
* @param groupId 群组Id, 如果发送目标为群组, 则可设置群组Id, 以使用群组配置.
* @param rankingStart 排行榜开始范围(从1开始, 名次)如传入0或负数则为默认值默认为1
* @param rankingStop 排名榜结束范围(包括该名次)如传入0或负数则为默认值默认为150
* @param mode 排行榜模式
* @param contentType 排行榜内容类型
* @param quality 图片质量, 详见{@link PixivDownload.PageQuality}
* @throws IndexOutOfBoundsException 当 rankingStart > rankingStop时抛出
*/
public RandomRankingArtworksSender(
MessageSender messageSender,
long groupId,
int rankingStart,
int rankingStop,
PixivURL.RankingMode mode,
PixivURL.RankingContentType contentType,
PixivDownload.PageQuality quality) {
super(messageSender);
this.groupId = groupId;
this.mode = mode;
this.contentType = contentType;
log = LoggerFactory.getLogger("RecommendArtworksSender@" + Integer.toHexString(this.hashCode()));
log = LoggerFactory.getLogger(this.toString());
this.rankingStart = rankingStart > 0 ? rankingStart : 1;
this.rankingStop = rankingStop > 0 ? rankingStop : 150;
if(this.rankingStart > this.rankingStop) {
@ -62,7 +92,7 @@ public class RandomRankingArtworksSender extends AutoSender {
int selectRanking = rankingStart + new Random().nextInt(rankingStop - rankingStart + 1);
try {
List<JsonObject> rankingList = BotCommandProcess.getRankingInfoByCache(
List<JsonObject> rankingList = CacheStoreCentral.getCentral().getRankingInfoByCache(
contentType,
mode,
queryDate,
@ -77,7 +107,8 @@ public class RandomRankingArtworksSender extends AutoSender {
JsonObject rankingInfo = rankingList.get(0);
int illustId = rankingInfo.get("illust_id").getAsInt();
if(BotCommandProcess.isNoSafe(illustId, BotCommandProcess.globalProp, false)) {
if(BotCommandProcess.isNoSafe(illustId,
SettingProperties.getProperties(groupId), false)) {
log.warn("作品为r18作品, 取消本次发送.");
return;
} else if(BotCommandProcess.isReported(illustId)) {
@ -85,15 +116,14 @@ public class RandomRankingArtworksSender extends AutoSender {
return;
}
StringBuilder message = new StringBuilder();
message.append("#美图推送 - 今日排行榜 第 ").append(rankingInfo.get("rank").getAsInt()).append("\n");
message.append("标题").append(rankingInfo.get("title").getAsString()).append("(").append(illustId).append(")\n");
message.append("作者:").append(rankingInfo.get("user_name").getAsString()).append("\n");
message.append(BotCommandProcess.getImageById(illustId, quality, 1));
message.append("\n如有不当作品可使用\".cgj report -id ").append(illustId).append("\"向色图姬反馈。");
getMessageSender().sendMessage(message.toString());
} catch (IOException e) {
e.printStackTrace();
String message = "#美图推送 - 今日排行榜 第 " + rankingInfo.get("rank").getAsInt() + "\n" +
"标题:" + rankingInfo.get("title").getAsString() + "(" + illustId + ")\n" +
"作者" + rankingInfo.get("user_name").getAsString() + "\n" +
CacheStoreCentral.getCentral().getImageById(0, illustId, quality, 1) +
"\n如有不当作品可使用\".cgj report -id " + illustId + "\"向色图姬反馈。";
getMessageSender().sendMessage(message);
} catch (Exception e) {
log.error("发送随机作品时发生异常", e);
}
}
}

View File

@ -1,12 +1,12 @@
package net.lamgc.cgj.bot;
import com.google.common.base.Throwables;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.bot.event.VirtualLoadMessageEvent;
import net.lamgc.cgj.pixiv.PixivURL;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.text.SimpleDateFormat;
import java.time.LocalDate;
import java.util.Calendar;
import java.util.Date;
@ -27,21 +27,24 @@ public class RankingUpdateTimer {
Calendar cal = Calendar.getInstance();
cal.setTime(firstRunDate == null ? new Date() : firstRunDate);
LocalDate currentLocalDate = LocalDate.now();
if(cal.get(Calendar.DAY_OF_YEAR) <= currentLocalDate.getDayOfYear() && cal.get(Calendar.HOUR_OF_DAY) >= 12) {
cal.set(Calendar.DAY_OF_YEAR, cal.get(Calendar.DAY_OF_YEAR) + 1);
if(cal.get(Calendar.DAY_OF_YEAR) < currentLocalDate.getDayOfYear() || (
cal.get(Calendar.DAY_OF_YEAR) == currentLocalDate.getDayOfYear() &&
(cal.get(Calendar.HOUR_OF_DAY) * 60 + cal.get(Calendar.MINUTE) >= 690))) {
cal.set(Calendar.DAY_OF_YEAR, currentLocalDate.getDayOfYear() + 1);
}
cal.set(Calendar.HOUR_OF_DAY, 12);
cal.set(Calendar.HOUR_OF_DAY, 11);
cal.set(Calendar.MINUTE, 30);
cal.set(Calendar.SECOND, 0);
cal.set(Calendar.MILLISECOND, 0);
log.warn("已设置排行榜定时更新, 首次运行时间: {}", cal.getTime());
long delay = cal.getTime().getTime() - System.currentTimeMillis();
log.warn("已设置排行榜定时更新, 首次运行时间: {} ({}min)", cal.getTime(), delay / 1000 / 60);
timer.schedule(new TimerTask() {
@Override
public void run() {
now(null);
}
}, cal.getTime(), 86400000); // 1 Day
}, delay, 86400000); // 1 Day
}
public void now(Date queryDate) {
@ -52,13 +55,14 @@ public class RankingUpdateTimer {
LocalDate currentLocalDate = LocalDate.now();
if(calendar.get(Calendar.DAY_OF_YEAR) == currentLocalDate.getDayOfYear() ||
calendar.get(Calendar.DAY_OF_YEAR) == currentLocalDate.getDayOfYear() - 1) {
if(calendar.get(Calendar.HOUR_OF_DAY) < 12) {
if(calendar.get(Calendar.HOUR_OF_DAY) < 11) {
calendar.add(Calendar.DAY_OF_YEAR, -2);
} else {
calendar.add(Calendar.DAY_OF_YEAR, -1);
}
}
String dateStr = new SimpleDateFormat("yyyy-MM-dd").format(calendar.getTime());
log.info("正在获取 {} 期排行榜数据...", calendar.getTime());
for (PixivURL.RankingMode rankingMode : PixivURL.RankingMode.values()) {
for (PixivURL.RankingContentType contentType : PixivURL.RankingContentType.values()) {
@ -66,15 +70,9 @@ public class RankingUpdateTimer {
log.debug("不支持的类型, 填空值跳过...(类型: {}.{})", rankingMode.name(), contentType.name());
}
log.info("当前排行榜类型: {}.{}, 正在更新...", rankingMode.name(), contentType.name());
try {
//BotCommandProcess.getRankingInfoByCache(contentType, rankingMode, calendar.getTime(), 1, 0, true);
BotEventHandler.executor.executorSync(
new VirtualLoadMessageEvent(0,0,
".cgj ranking -type=" + contentType.name() + " -mode=" + rankingMode.name()));
log.info("排行榜 {}.{} 更新完成.", rankingMode.name(), contentType.name());
} catch (InterruptedException e) {
log.error("排行榜 {}.{} 更新时发生异常. \n{}", rankingMode.name(), contentType.name(), Throwables.getStackTraceAsString(e));
}
BotEventHandler.executeMessageEvent(new VirtualLoadMessageEvent(0,0,
".cgj ranking -type=" + contentType.name() + " -mode=" + rankingMode.name() + " -force -date " + dateStr));
log.info("排行榜 {}.{} 负载指令已投递.", rankingMode.name(), contentType.name());
}
}
log.warn("定时任务更新完成.");

View File

@ -0,0 +1,246 @@
package net.lamgc.cgj.bot;
import com.google.common.base.Throwables;
import net.lamgc.cgj.bot.boot.BotGlobal;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.util.*;
public final class SettingProperties {
private final static Logger log = LoggerFactory.getLogger(SettingProperties.class);
private final static File globalPropFile = new File(getPropertiesDir(), "global.properties");
private final static Properties globalProp = new Properties();
private final static Map<Long, Properties> groupPropMap = new HashMap<>();
private final static Set<Long> changeList = Collections.synchronizedSet(new HashSet<>());
/**
* 全局配置项
*/
public final static long GLOBAL = 0;
/**
* 清空所有Properties.
*/
public static void clearProperties() {
groupPropMap.clear();
globalProp.clear();
}
/**
* 加载配置文件
*/
public static void loadProperties() {
loadGlobalProperties();
File[] files = getPropertiesDir()
.listFiles((dir, fileName) -> fileName.startsWith("group.") && fileName.endsWith(".properties"));
if(files == null) {
log.error("检索群组配置文件失败, 可能是被拒绝访问.");
return;
}
for (File file : files) {
String name = file.getName();
long groupId;
try {
groupId = Long.parseLong(name.substring(name.indexOf("group.") + 6, name.lastIndexOf(".properties")));
if(groupId <= 0) {
log.warn("无效的群配置文件: {}", groupId);
continue;
}
} catch (NumberFormatException e) {
log.error("非法的配置文件名: {}", name);
continue;
}
if(!groupPropMap.containsKey(groupId)) {
groupPropMap.put(groupId, new Properties(globalProp));
}
loadGroupProperties(groupId, groupPropMap.get(groupId));
}
}
/**
* 保存配置项
*/
public static void saveProperties() {
log.info("正在保存所有配置...");
saveGlobalProperties();
for (Long groupId : groupPropMap.keySet()) {
if(!changeList.contains(groupId)) {
log.debug("群组 {} 配置无改动, 忽略保存.", groupId);
return;
}
log.debug("正在保存群组 {} 配置文件...", groupId);
saveGroupProperties(groupId);
}
log.info("配置保存完成.");
}
/**
* 保存指定群组的配置文件
* @param groupId 要保存配置的群组Id
*/
private static void saveGroupProperties(long groupId) {
try {
saveGroupProperties(groupId, getGroupProperties(groupId));
} catch (IOException e) {
log.error("群组 {} 配置保存失败\n{}", groupId, Throwables.getStackTraceAsString(e));
}
}
private static void saveGroupProperties(Long groupId, Properties properties) throws IOException {
File groupPropFile = new File(getPropertiesDir(), "group." + groupId + ".properties");
if(!groupPropFile.exists() && !groupPropFile.createNewFile()) {
log.error("群组 {} 配置文件创建失败!", groupId);
return;
}
saveProperties(properties, new FileOutputStream(groupPropFile));
}
private static void loadGlobalProperties() {
if(globalPropFile.exists() && globalPropFile.isFile()) {
log.info("正在加载全局配置文件...");
try (Reader reader = new InputStreamReader(new FileInputStream(globalPropFile), StandardCharsets.UTF_8)) {
globalProp.load(reader);
log.info("全局配置文件加载完成.");
} catch (IOException e) {
log.error("加载全局配置文件时发生异常", e);
}
} else {
log.info("未找到全局配置文件,跳过加载.");
}
}
/**
* 保存全局配置项
*/
private static void saveGlobalProperties() {
try {
if(!globalPropFile.exists() && !globalPropFile.createNewFile()) {
log.error("创建全局配置文件失败.");
return;
}
saveProperties(globalProp, new FileOutputStream(globalPropFile));
} catch (IOException e) {
log.error("全局配置文件保存时发生异常", e);
}
}
private static void loadGroupProperties(long groupId, Properties properties) {
File propFile = new File(getPropertiesDir(), "group." + groupId + ".properties");
Properties groupProp = Objects.requireNonNull(properties);
if(!propFile.exists() || !propFile.isFile()) {
log.warn("群组 {} 配置文件不存在, 或不是一个文件.({})", groupId, propFile.getAbsolutePath());
return;
}
try (Reader reader = new InputStreamReader(new FileInputStream(propFile), StandardCharsets.UTF_8)) {
groupProp.load(reader);
} catch (IOException e) {
log.error("读取群组 {} 群配置文件时发生异常:\n{}", groupId, Throwables.getStackTraceAsString(e));
}
}
private static void saveProperties(Properties properties, OutputStream stream) throws IOException {
properties.store(new OutputStreamWriter(stream, StandardCharsets.UTF_8), null);
}
/**
* 获取配置文件目录
* @return 返回目录File对象.
*/
private static File getPropertiesDir() {
File propDir = new File(BotGlobal.getGlobal().getDataStoreDir(), "/setting/");
if(!propDir.exists() && !propDir.mkdirs()) {
log.warn("Setting文件夹创建失败!");
}
return propDir;
}
public static String getProperty(long groupId, String key) {
return getProperty(groupId, key, null);
}
public static String getProperty(long groupId, String key, String defaultValue) {
if(groupId <= 0) {
return globalProp.getProperty(key, defaultValue);
} else {
Properties properties = groupPropMap.get(groupId);
return properties == null ? defaultValue : properties.getProperty(key, defaultValue);
}
}
/**
* 设置配置项
* @param groupId 群组Id, 如为0或负数则为全局配置
* @param key 配置项key名
* @param value 欲设置的新值, 如为null则删除该配置项
* @return 返回上一次设定值
*/
public static String setProperty(long groupId, String key, String value) {
Objects.requireNonNull(key);
Properties targetProperties;
if(groupId <= 0) {
targetProperties = globalProp;
} else {
changeList.add(groupId);
targetProperties = getGroupProperties(groupId);
}
String lastValue = targetProperties.getProperty(key);
if(value != null) {
targetProperties.setProperty(key, value);
} else {
targetProperties.remove(key);
}
return lastValue;
}
/**
* 获取GlobalProperties
* @return 全局Properties
*/
private static Properties getGlobalProperties() {
return globalProp;
}
/**
* 获取群组Properties
* @param groupId 群组Id
* @return 如果存在, 返回Properties, 不存在返回null.
* @throws IllegalArgumentException 当群组Id 小于或等于0 时抛出.
*/
private static Properties getGroupProperties(long groupId) {
if (groupId <= 0) {
throw new IllegalArgumentException("Group number cannot be 0 or negative: " + groupId);
}
if(!groupPropMap.containsKey(groupId)) {
groupPropMap.put(groupId, new Properties(globalProp));
}
return groupPropMap.get(groupId);
}
/**
* 获取群组 Properties, 如果指定群组没有 Properties, 则使用GlobalProperties.
* @param groupId 指定群组Id
* @return 如果群组存在所属Properties, 则返回群组Properties, 否则返回GlobalProperties.
*/
public static Properties getProperties(long groupId) {
if(groupId > 0 && groupPropMap.containsKey(groupId)) {
return groupPropMap.get(groupId);
}
return getGlobalProperties();
}
}

View File

@ -0,0 +1,46 @@
package net.lamgc.cgj.bot.boot;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.util.PropertiesUtils;
import net.lamgc.utils.base.ArgumentsProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public final class ApplicationBoot {
private final static Logger log = LoggerFactory.getLogger(ApplicationBoot.class);
private ApplicationBoot() {}
/**
* 初始化应用.
* <p>该方法不会初始化机器人, 仅初始化应用所需的配置信息.</p>
*/
public static void initialApplication(String[] args) {
ArgumentsProperties argsProp = new ArgumentsProperties(args);
if(!PropertiesUtils.getSettingToSysProp(argsProp, "proxy", null)) {
PropertiesUtils.getEnvSettingToSysProp("CGJ_PROXY", "proxy", null);
}
if(!PropertiesUtils.getSettingToSysProp(argsProp, "botDataDir", "./") &&
!PropertiesUtils.getEnvSettingToSysProp("CGJ_BOT_DATA_DIR", "botDataDir", "./")) {
log.warn("未设置botDataDir, 当前运行目录将作为酷Q机器人所在目录.");
}
if(!PropertiesUtils.getSettingToSysProp(argsProp, "redisAddress", "127.0.0.1") &&
!PropertiesUtils.getEnvSettingToSysProp("CGJ_REDIS_URI", "redisAddress", "127.0.0.1")) {
log.warn("未设置RedisAddress, 将使用默认值连接Redis服务器(127.0.0.1:6379)");
}
// 初始化 BotGlobal
//noinspection ResultOfMethodCallIgnored 这里仅仅是加载BotGlobal而已
BotGlobal.getGlobal();
}
/**
* 初始化机器人.
* <p>本方法由框架调用.</p>
*/
public static void initialBot() {
BotEventHandler.initial();
}
}

View File

@ -0,0 +1,132 @@
package net.lamgc.cgj.bot.boot;
import com.google.common.base.Strings;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import net.lamgc.cgj.pixiv.PixivDownload;
import org.apache.http.HttpHost;
import org.apache.http.client.CookieStore;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.exceptions.JedisConnectionException;
import java.io.File;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URI;
import java.net.URL;
import java.nio.file.Files;
public final class BotGlobal {
private final static Logger log = LoggerFactory.getLogger(BotGlobal.class);
private final static BotGlobal instance = new BotGlobal();
public static BotGlobal getGlobal() {
if(instance == null) {
throw new IllegalStateException("BotGlobal has not been initialized");
}
return instance;
}
private final URI redisUri;
/**
* 所有缓存共用的JedisPool
*/
private final JedisPool redisServer;
private final File dataStoreDir;
private final HttpHost proxy;
private CookieStore cookieStore;
private final Gson gson = new GsonBuilder()
.serializeNulls()
.create();
private PixivDownload pixivDownload;
private final File imageStoreDir;
private BotGlobal() {
this.redisUri = URI.create("redis://" + System.getProperty("cgj.redisAddress"));
this.redisServer = new JedisPool(
getRedisUri().getHost(),
getRedisUri().getPort() == -1 ? 6379 : getRedisUri().getPort());
try (Jedis jedis = this.redisServer.getResource()) {
log.warn("Redis连接状态(Ping): {}", jedis.ping().equalsIgnoreCase("pong"));
} catch(JedisConnectionException e) {
log.warn("Redis连接失败, 将会影响到后续功能运行.({})", e.getCause().getMessage());
}
String dataStoreDirPath = System.getProperty("cgj.botDataDir");
this.dataStoreDir = new File((!dataStoreDirPath.endsWith("/") || !dataStoreDirPath.endsWith("\\")) ?
dataStoreDirPath + System.getProperty("file.separator") : dataStoreDirPath);
this.imageStoreDir = new File(getDataStoreDir(), "data/image/cgj/");
String proxyAddress = System.getProperty("cgj.proxy");
HttpHost temp = null;
if(!Strings.isNullOrEmpty(proxyAddress)) {
try {
URL proxyUrl = new URL(proxyAddress);
temp = new HttpHost(proxyUrl.getHost(), proxyUrl.getPort());
log.info("已启用代理:{}", temp.toHostString());
} catch (MalformedURLException e) {
log.error("Proxy地址解析失败, 代理将不会启用.", e);
}
}
this.proxy = temp;
}
public URI getRedisUri() {
return redisUri;
}
public File getDataStoreDir() {
if(!dataStoreDir.exists() && !dataStoreDir.mkdirs()) {
log.error("DataStoreDir 创建失败, 数据存储可能失效!");
}
return dataStoreDir;
}
public JedisPool getRedisServer() {
return redisServer;
}
public HttpHost getProxy() {
return proxy;
}
public void setCookieStore(CookieStore cookieStore) {
if(this.cookieStore != null) {
throw new IllegalStateException("CookieStore set");
}
this.cookieStore = cookieStore;
this.pixivDownload =
new PixivDownload(cookieStore, proxy);
}
public Gson getGson() {
return gson;
}
public PixivDownload getPixivDownload() {
return pixivDownload;
}
public File getImageStoreDir() {
if(!imageStoreDir.exists() && !Files.isSymbolicLink(imageStoreDir.toPath())) {
if(!imageStoreDir.mkdirs()) {
log.warn("酷Q图片缓存目录失效(Path: {} )", imageStoreDir.getAbsolutePath());
throw new RuntimeException(new IOException("文件夹创建失败!"));
}
}
return imageStoreDir;
}
}

View File

@ -0,0 +1,52 @@
package net.lamgc.cgj.bot.cache;
import com.google.common.base.Throwables;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Set;
import java.util.Timer;
import java.util.TimerTask;
import java.util.concurrent.CopyOnWriteArraySet;
public class AutoCleanTimer extends TimerTask {
private final static Set<Cleanable> cleanSet = new CopyOnWriteArraySet<>();
private final static Timer cleanTimer = new Timer("Thread-AutoClean", true);
private final static Logger log = LoggerFactory.getLogger(AutoCleanTimer.class);
static {
cleanTimer.schedule(new AutoCleanTimer(), 100L, 100L);
}
/**
* 增加需要定时执行清理的缓存库
* @param store 已实现Cleanable的对象
*/
public static void add(Cleanable store) {
cleanSet.add(store);
}
/**
* 移除已添加的缓存库
* @param store 需要从AutoCleanTimer移除的对象
*/
public static void remove(Cleanable store) {
cleanSet.remove(store);
}
private AutoCleanTimer() {}
@Override
public void run() {
cleanSet.forEach(cleanable -> {
try {
cleanable.clean();
} catch (Exception e) {
log.error("{} 执行清理动作时发生异常:\n{}", cleanable.toString(), Throwables.getStackTraceAsString(e));
}
});
}
}

View File

@ -13,7 +13,7 @@ public interface CacheStore<T> {
* 更新或添加缓存项
* @param key 缓存键名
* @param value 缓存值
* @param expire 有效期, 单位为ms(毫秒), 如不过期传入0或赋值
* @param expire 有效期, 单位为ms(毫秒), 如不过期传入0或负数
*/
void update(String key, T value, long expire);

View File

@ -0,0 +1,701 @@
package net.lamgc.cgj.bot.cache;
import com.google.common.base.Strings;
import com.google.common.base.Throwables;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import net.lamgc.cgj.bot.BotCode;
import net.lamgc.cgj.bot.BotCommandProcess;
import net.lamgc.cgj.bot.SettingProperties;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.exception.HttpRequestException;
import net.lamgc.cgj.pixiv.PixivDownload;
import net.lamgc.cgj.pixiv.PixivSearchLinkBuilder;
import net.lamgc.cgj.pixiv.PixivURL;
import net.lamgc.cgj.util.Locker;
import net.lamgc.cgj.util.LockerMap;
import net.lamgc.cgj.util.URLs;
import net.lamgc.utils.encrypt.MessageDigestUtils;
import net.lz1998.cq.utils.CQCode;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
@SuppressWarnings("SynchronizationOnLocalVariableOrMethodParameter")
public final class CacheStoreCentral {
private final static Logger log = LoggerFactory.getLogger(CacheStoreCentral.class);
private static CacheStoreCentral central = new CacheStoreCentral();
public static CacheStoreCentral getCentral() {
if(central == null) {
initialCentral();
}
return central;
}
private synchronized static void initialCentral() {
if(central != null) {
return;
}
central = new CacheStoreCentral();
}
private final LockerMap<String> lockerMap = new LockerMap<>();
private CacheStoreCentral() {}
private final Hashtable<String, File> imageCache = new Hashtable<>();
private final CacheStore<JsonElement> imageChecksumCache =
new JsonRedisCacheStore(BotGlobal.getGlobal().getRedisServer(),
"imageChecksum", BotGlobal.getGlobal().getGson());
/*
* 注意:
* 在启用了远端缓存的情况下, 不允许滥用本地缓存
* 只有在处理命令中需要短时间大量存取的缓存项才能进行本地缓存(例如PreLoadData需要在排序中大量获取);
* 如果没有短时间大量存取的需要, 切勿使用本地缓存
*/
/**
* 作品信息缓存 - 不过期
*/
private final CacheStore<JsonElement> illustInfoCache =
new JsonRedisCacheStore(BotGlobal.getGlobal().getRedisServer(),
"illustInfo", BotGlobal.getGlobal().getGson());
/**
* 作品信息预加载数据 - 有效期 2 小时, 本地缓存有效期 0.5 ± 0.25 小时
*/
private final CacheStore<JsonElement> illustPreLoadDataCache =
CacheStoreUtils.hashLocalHotDataStore(
new JsonRedisCacheStore(BotGlobal.getGlobal().getRedisServer(),
"illustPreLoadData", BotGlobal.getGlobal().getGson()),
// 600000, 120000);
60000, 1);
/**
* 搜索内容缓存, 有效期 2 小时
*/
private final CacheStore<JsonElement> searchBodyCache =
new JsonRedisCacheStore(BotGlobal.getGlobal().getRedisServer(),
"searchBody", BotGlobal.getGlobal().getGson());
/**
* 排行榜缓存, 不过期
*/
private final CacheStore<List<JsonObject>> rankingCache =
new JsonObjectRedisListCacheStore(BotGlobal.getGlobal().getRedisServer(),
"ranking", BotGlobal.getGlobal().getGson());
/**
* 作品页面下载链接缓存 - 不过期
*/
private final CacheStore<List<String>> pagesCache =
new StringListRedisCacheStore(BotGlobal.getGlobal().getRedisServer(), "imagePages");
/**
* 清空所有缓存
*/
public void clearCache() {
imageCache.clear();
illustInfoCache.clear();
illustPreLoadDataCache.clear();
searchBodyCache.clear();
rankingCache.clear();
pagesCache.clear();
}
/**
* 通过illustId获取作品图片
* @param fromGroup 来源群(系统提供)
* @param illustId 作品Id
* @param quality 图片质量
* @param pageIndex 指定页面索引, 从1开始
* @return 如果成功, 返回BotCode, 否则返回错误信息.
*/
public String getImageById(long fromGroup, int illustId, PixivDownload.PageQuality quality, int pageIndex) throws InterruptedException {
log.debug("IllustId: {}, Quality: {}, PageIndex: {}", illustId, quality.name(), pageIndex);
if(pageIndex <= 0) {
log.warn("指定的页数不能小于或等于0: {}", pageIndex);
return "指定的页数不能小于或等于0";
}
try {
if (BotCommandProcess.isNoSafe(illustId, SettingProperties.getProperties(fromGroup), false)) {
log.warn("作品 {} 存在R-18内容且设置\"image.allowR18\"为false将屏蔽该作品不发送.", illustId);
return "(根据设置,该作品已被屏蔽!)";
} else if(BotCommandProcess.isReported(illustId)) {
log.warn("作品Id {} 被报告, 正在等待审核, 跳过该作品.", illustId);
return "(该作品已被封印)";
}
} catch (IOException e) {
log.warn("作品信息无法获取!", e);
return "(发生网络异常,无法获取图片!)";
}
List<String> pagesList;
try {
pagesList = getIllustPages(illustId, quality, false);
} catch (IOException e) {
log.error("获取下载链接列表时发生异常", e);
return "发生网络异常,无法获取图片!";
}
if(log.isDebugEnabled()) {
StringBuilder logBuilder = new StringBuilder("作品Id " + illustId + " 所有页面下载链接: \n");
AtomicInteger index = new AtomicInteger();
pagesList.forEach(item ->
logBuilder.append(index.incrementAndGet()).append(". ").append(item).append("\n"));
log.debug(logBuilder.toString());
}
if (pagesList.size() < pageIndex) {
log.warn("指定的页数超出了总页数({} / {})", pageIndex, pagesList.size());
return "指定的页数超出了范围(总共 " + pagesList.size() + " 页)";
}
String downloadLink = pagesList.get(pageIndex - 1);
String fileName = URLs.getResourceName(Strings.nullToEmpty(downloadLink));
File imageFile = new File(BotGlobal.getGlobal().getImageStoreDir(),
downloadLink.substring(downloadLink.lastIndexOf("/") + 1));
log.debug("FileName: {}, DownloadLink: {}", fileName, downloadLink);
if(!imageCache.containsKey(fileName)) {
if(imageFile.exists() && imageFile.isFile()) {
ImageChecksum imageChecksum = getImageChecksum(illustId, pageIndex);
if(imageChecksum != null) {
try {
log.trace("正在检查作品Id {} 第 {} 页图片文件 {} ...", illustId, pageIndex, imageFile.getName());
if (ImageChecksum.checkFile(imageChecksum, Files.readAllBytes(imageFile.toPath()))) {
imageCache.put(URLs.getResourceName(downloadLink), imageFile);
log.trace("作品Id {} 第 {} 页缓存已补充.", illustId, pageIndex);
return getImageToBotCode(imageFile, false).toString();
} else {
log.warn("图片文件 {} 校验失败, 重新下载图片...", imageFile.getName());
}
} catch(IOException e) {
log.error("文件检验时读取失败, 重新下载文件...(file: {})", imageFile.getPath());
}
} else {
log.warn("图片存在但校验不存在, 重新下载图片...");
}
}
try {
Throwable throwable = ImageCacheStore.executeCacheRequest(
new ImageCacheObject(imageCache, illustId, pageIndex, downloadLink, imageFile));
if(throwable != null) {
throw throwable;
}
} catch (InterruptedException e) {
log.warn("图片缓存被中断", e);
throw e;
} catch (Throwable e) {
log.error("图片 {} 获取失败:\n{}", illustId + "p" + pageIndex, Throwables.getStackTraceAsString(e));
return "(错误: 图片获取出错)";
}
} else {
log.trace("图片 {} 缓存命中.", fileName);
}
return getImageToBotCode(imageCache.get(fileName), false).toString();
}
/**
* 通过文件获取图片的BotCode代码
* @param targetFile 图片文件
* @param updateCache 是否刷新缓存(只是让机器人重新上传, 如果上传接口有重复检测的话是无法处理的)
* @return 返回设定好参数的BotCode
*/
@SuppressWarnings("SameParameterValue")
private BotCode getImageToBotCode(File targetFile, boolean updateCache) {
String fileName = Objects.requireNonNull(targetFile, "targetFile is null").getName();
BotCode code = BotCode.parse(
CQCode.image(BotGlobal.getGlobal().getImageStoreDir().getName() + "/" + fileName));
code.addParameter("absolutePath", targetFile.getAbsolutePath());
code.addParameter("imageName", fileName.substring(0, fileName.lastIndexOf(".")));
code.addParameter("updateCache", updateCache ? "true" : "false");
return code;
}
/**
* 获取作品信息
* @param illustId 作品Id
* @param flushCache 强制刷新缓存
* @return 返回作品信息
* @throws IOException 当Http请求发生异常时抛出
* @throws NoSuchElementException 当作品未找到时抛出
*/
public JsonObject getIllustInfo(int illustId, boolean flushCache)
throws IOException, NoSuchElementException {
Locker<String> locker = buildSyncKey(Integer.toString(illustId));
String illustIdStr = locker.getKey();
JsonObject illustInfoObj = null;
if (!illustInfoCache.exists(illustIdStr) || flushCache) {
try {
locker.lock();
synchronized (locker) {
if (!illustInfoCache.exists(illustIdStr) || flushCache) {
illustInfoObj = BotGlobal.getGlobal().getPixivDownload().getIllustInfoByIllustId(illustId);
illustInfoCache.update(illustIdStr, illustInfoObj, null);
}
}
} finally {
locker.unlock();
}
}
if(Objects.isNull(illustInfoObj)) {
illustInfoObj = illustInfoCache.getCache(illustIdStr).getAsJsonObject();
log.trace("作品Id {} IllustInfo缓存命中.", illustId);
}
return illustInfoObj;
}
/**
* 获取作品预加载数据.
* 可以获取作品的一些与用户相关的信息
* @param illustId 作品Id
* @param flushCache 是否刷新缓存
* @return 成功返回JsonObject对象
* @throws IOException 当Http请求处理发生异常时抛出
*/
public JsonObject getIllustPreLoadData(int illustId, boolean flushCache) throws IOException {
Locker<String> locker = buildSyncKey(Integer.toString(illustId));
String illustIdStr = locker.getKey();
JsonObject result = null;
if (!illustPreLoadDataCache.exists(illustIdStr) || flushCache) {
try {
locker.lock();
synchronized (locker) {
if (!illustPreLoadDataCache.exists(illustIdStr) || flushCache) {
log.trace("IllustId {} 缓存失效, 正在更新...", illustId);
JsonObject preLoadDataObj = BotGlobal.getGlobal().getPixivDownload()
.getIllustPreLoadDataById(illustId)
.getAsJsonObject("illust")
.getAsJsonObject(Integer.toString(illustId));
long expire = 7200 * 1000;
String propValue = SettingProperties.
getProperty(SettingProperties.GLOBAL, "cache.illustPreLoadData.expire", "7200000");
log.debug("PreLoadData有效时间设定: {}", propValue);
try {
expire = Long.parseLong(propValue);
} catch (Exception e) {
log.warn("全局配置项 \"{}\" 值非法, 已使用默认值: {}", propValue, expire);
}
result = preLoadDataObj;
illustPreLoadDataCache.update(illustIdStr, preLoadDataObj, expire);
log.trace("作品Id {} preLoadData缓存已更新(有效时间: {})", illustId, expire);
}
}
} finally {
locker.unlock();
}
}
if(Objects.isNull(result)) {
result = illustPreLoadDataCache.getCache(illustIdStr).getAsJsonObject();
log.trace("作品Id {} PreLoadData缓存命中.", illustId);
}
return result;
}
public List<String> getIllustPages(int illustId, PixivDownload.PageQuality quality, boolean flushCache)
throws IOException {
Locker<String> locker
= buildSyncKey(Integer.toString(illustId), ".", quality.name());
String pagesSign = locker.getKey();
List<String> result = null;
if (!pagesCache.exists(pagesSign) || flushCache) {
try {
locker.lock();
synchronized (locker) {
if (!pagesCache.exists(pagesSign) || flushCache) {
List<String> linkList = PixivDownload
.getIllustAllPageDownload(BotGlobal.getGlobal().getPixivDownload().getHttpClient(),
BotGlobal.getGlobal().getPixivDownload().getCookieStore(), illustId, quality);
result = linkList;
pagesCache.update(pagesSign, linkList, null);
}
}
} finally {
locker.unlock();
}
}
if(Objects.isNull(result)) {
result = pagesCache.getCache(pagesSign);
log.trace("作品Id {} Pages缓存命中.", illustId);
}
return result;
}
private final Random expireTimeFloatRandom = new Random();
/**
* 获取排行榜
* @param contentType 排行榜类型
* @param mode 排行榜模式
* @param queryDate 查询时间
* @param start 开始排名, 从1开始
* @param range 取范围
* @param flushCache 是否强制刷新缓存
* @return 成功返回有值List, 失败且无异常返回空
* @throws IOException 获取异常时抛出
*/
public List<JsonObject> getRankingInfoByCache(PixivURL.RankingContentType contentType,
PixivURL.RankingMode mode,
Date queryDate, int start, int range, boolean flushCache)
throws IOException {
if(!contentType.isSupportedMode(mode)) {
log.warn("试图获取不支持的排行榜类型已拒绝.(ContentType: {}, RankingMode: {})", contentType.name(), mode.name());
if(log.isDebugEnabled()) {
try {
Thread.dumpStack();
} catch(Exception e) {
log.debug("本次非法请求的堆栈信息如下: \n{}", Throwables.getStackTraceAsString(e));
}
}
return new ArrayList<>(0);
}
String date = new SimpleDateFormat("yyyyMMdd").format(queryDate);
Locker<String> locker
= buildSyncKey(contentType.name(), ".", mode.name(), ".", date);
String requestSign = locker.getKey();
List<JsonObject> result = null;
if(!rankingCache.exists(requestSign) || flushCache) {
try {
locker.lock();
synchronized (locker) {
if (!rankingCache.exists(requestSign) || flushCache) {
log.trace("Ranking缓存失效, 正在更新...(RequestSign: {})", requestSign);
List<JsonObject> rankingResult = BotGlobal.getGlobal().getPixivDownload()
.getRanking(contentType, mode, queryDate, 1, 500);
long expireTime = 0;
if (rankingResult.size() == 0) {
expireTime = 5400000 + expireTimeFloatRandom.nextInt(1800000);
log.warn("数据获取失败, 将设置浮动有效时间以准备下次更新. (ExpireTime: {}ms)", expireTime);
}
result = new ArrayList<>(rankingResult).subList(start - 1, start + range - 1);
rankingCache.update(requestSign, rankingResult, expireTime);
log.trace("Ranking缓存更新完成.(RequestSign: {})", requestSign);
}
}
} finally {
locker.unlock();
}
}
if (Objects.isNull(result)) {
result = rankingCache.getCache(requestSign, start - 1, range);
log.trace("RequestSign [{}] 缓存命中.", requestSign);
}
return PixivDownload.getRanking(result, start - 1, range);
}
/**
* 获取搜索结果
* @param content 搜索内容
* @param type 类型
* @param area 范围
* @param includeKeywords 包含关键词
* @param excludeKeywords 排除关键词
* @param contentOption 内容类型
* @return 返回完整搜索结果
* @throws IOException 当请求发生异常, 或接口返回异常信息时抛出.
*/
public JsonObject getSearchBody(
String content,
String type,
String area,
String includeKeywords,
String excludeKeywords,
String contentOption,
int pageIndex
) throws IOException {
PixivSearchLinkBuilder searchBuilder = new PixivSearchLinkBuilder(Strings.isNullOrEmpty(content) ? "" : content);
if (type != null) {
try {
searchBuilder.setSearchType(PixivSearchLinkBuilder.SearchType.valueOf(type.toUpperCase()));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchType: {}", type);
}
}
if (area != null) {
try {
searchBuilder.setSearchArea(PixivSearchLinkBuilder.SearchArea.valueOf(area));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchArea: {}", area);
}
}
if (contentOption != null) {
try {
searchBuilder.setSearchContentOption(
PixivSearchLinkBuilder.SearchContentOption.valueOf(contentOption.trim().toUpperCase()));
} catch (IllegalArgumentException e) {
log.warn("不支持的SearchContentOption: {}", contentOption);
}
}
if (!Strings.isNullOrEmpty(includeKeywords)) {
for (String keyword : includeKeywords.split(";")) {
searchBuilder.removeExcludeKeyword(keyword.trim());
searchBuilder.addIncludeKeyword(keyword.trim());
log.trace("已添加关键字: {}", keyword);
}
}
if (!Strings.isNullOrEmpty(excludeKeywords)) {
for (String keyword : excludeKeywords.split(";")) {
searchBuilder.removeIncludeKeyword(keyword.trim());
searchBuilder.addExcludeKeyword(keyword.trim());
log.trace("已添加排除关键字: {}", keyword);
}
}
if(pageIndex > 0) {
searchBuilder.setPage(pageIndex);
}
log.debug("正在搜索作品, 条件: {}", searchBuilder.getSearchCondition());
Locker<String> locker
= buildSyncKey(searchBuilder.buildURL());
String requestUrl = locker.getKey();
log.debug("RequestUrl: {}", requestUrl);
JsonObject resultBody = null;
if(!searchBodyCache.exists(requestUrl)) {
try {
locker.lock();
synchronized (locker) {
if (!searchBodyCache.exists(requestUrl)) {
log.trace("searchBody缓存失效, 正在更新...");
JsonObject jsonObject;
HttpGet httpGetRequest = BotGlobal.getGlobal().getPixivDownload().
createHttpGetRequest(requestUrl);
HttpResponse response = BotGlobal.getGlobal().getPixivDownload().
getHttpClient().execute(httpGetRequest);
String responseBody = EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
log.trace("ResponseBody: {}", responseBody);
jsonObject = BotGlobal.getGlobal().getGson().fromJson(responseBody, JsonObject.class);
if (jsonObject.get("error").getAsBoolean()) {
log.error("接口请求错误, 错误信息: {}", jsonObject.get("message").getAsString());
throw new HttpRequestException(response.getStatusLine(), responseBody);
}
long expire = 7200 * 1000;
String propValue = SettingProperties
.getProperty(SettingProperties.GLOBAL, "cache.searchBody.expire", "7200000");
try {
expire = Long.parseLong(propValue);
} catch (Exception e) {
log.warn("全局配置项 \"{}\" 值非法, 已使用默认值: {}", propValue, expire);
}
resultBody = jsonObject.getAsJsonObject().getAsJsonObject("body");
searchBodyCache.update(requestUrl, jsonObject, expire);
log.trace("searchBody缓存已更新(有效时间: {})", expire);
} else {
log.trace("搜索缓存命中.");
}
}
} finally {
locker.unlock();
}
} else {
log.trace("搜索缓存命中.");
}
if(Objects.isNull(resultBody)) {
resultBody = searchBodyCache.getCache(requestUrl).getAsJsonObject().getAsJsonObject("body");
}
return resultBody;
}
protected ImageChecksum getImageChecksum(int illustId, int pageIndex) {
String cacheKey = illustId + ":" + pageIndex;
if(!imageChecksumCache.exists(cacheKey)) {
return null;
} else {
return ImageChecksum.fromJsonObject(imageChecksumCache.getCache(cacheKey).getAsJsonObject());
}
}
protected void setImageChecksum(ImageChecksum checksum) {
String cacheKey = checksum.getIllustId() + ":" + checksum.getPage();
imageChecksumCache.update(cacheKey, ImageChecksum.toJsonObject(checksum), 0);
}
/**
* 合并String并存取到常量池, 以保证对象一致
* @param keys String对象
* @return 合并后, 如果常量池存在合并后的结果, 则返回常量池中的对象, 否则存入常量池后返回.
*/
private Locker<String> buildSyncKey(String... keys) {
StringBuilder sb = new StringBuilder();
for (String string : keys) {
sb.append(string);
}
return lockerMap.createLocker(sb.toString(), true);
}
/**
* 图片检验信息
*/
public static class ImageChecksum implements Serializable {
private final static MessageDigestUtils.Algorithm ALGORITHM = MessageDigestUtils.Algorithm.SHA256;
private ImageChecksum() {}
private int illustId;
private int page;
private String fileName;
private long size;
private byte[] checksum;
public long getSize() {
return size;
}
public void setSize(long size) {
this.size = size;
}
public byte[] getChecksum() {
return checksum;
}
public void setChecksum(byte[] checksum) {
this.checksum = checksum;
}
public int getIllustId() {
return illustId;
}
public void setIllustId(int illustId) {
this.illustId = illustId;
}
public String getFileName() {
return fileName;
}
public void setFileName(String fileName) {
this.fileName = fileName;
}
public int getPage() {
return page;
}
public void setPage(int page) {
this.page = page;
}
public static ImageChecksum buildImageChecksumFromStream(
int illustId, int pageIndex,
String fileName, InputStream imageStream) throws IOException {
ImageChecksum checksum = new ImageChecksum();
checksum.setIllustId(illustId);
checksum.setPage(pageIndex);
checksum.setFileName(fileName);
ByteArrayOutputStream bufferStream = new ByteArrayOutputStream();
checksum.setSize(IOUtils.copyLarge(imageStream, bufferStream));
checksum.setChecksum(
MessageDigestUtils.encrypt(bufferStream.toByteArray(), ALGORITHM));
return checksum;
}
/**
* 将图片检验信息转换成JsonObject
* @param checksum 检验信息对象
* @return 转换后的JsonObject对象
*/
public static JsonObject toJsonObject(ImageChecksum checksum) {
JsonObject result = new JsonObject();
result.addProperty("illustId", checksum.getIllustId());
result.addProperty("page", checksum.getPage());
result.addProperty("fileName", checksum.getFileName());
result.addProperty("size", checksum.getSize());
result.addProperty("checksum", Base64.getEncoder().encodeToString(checksum.getChecksum()));
return result;
}
/**
* 从JsonObject转换到图片检验信息
* @param checksumObject JsonObject对象
* @return 转换后的图片检验信息对象
*/
public static ImageChecksum fromJsonObject(JsonObject checksumObject) {
ImageChecksum checksum = new ImageChecksum();
checksum.setIllustId(checksumObject.get("illustId").getAsInt());
checksum.setPage(checksumObject.get("page").getAsInt());
checksum.setFileName(checksumObject.get("fileName").getAsString());
checksum.setSize(checksumObject.get("size").getAsLong());
checksum.setChecksum(Base64.getDecoder().decode(checksumObject.get("checksum").getAsString()));
return checksum;
}
/**
* 比对图片文件是否完整.
* @param checksum 图片检验信息
* @param imageData 图片数据
* @return 如果检验成功, 则返回true
*/
public static boolean checkFile(ImageChecksum checksum, byte[] imageData) {
byte[] sha256Checksum = MessageDigestUtils.encrypt(imageData, ALGORITHM);
return checksum.getSize() == imageData.length &&
Arrays.equals(checksum.getChecksum(), sha256Checksum);
}
@Override
public String toString() {
return "ImageChecksum{" +
"illustId=" + illustId +
", page=" + page +
", fileName='" + fileName + '\'' +
", size=" + size +
", checksum=" + Base64.getEncoder().encodeToString(getChecksum()) +
'}';
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ImageChecksum checksum1 = (ImageChecksum) o;
return illustId == checksum1.illustId &&
page == checksum1.page &&
size == checksum1.size &&
Objects.equals(fileName, checksum1.fileName) &&
Arrays.equals(checksum, checksum1.checksum);
}
@Override
public int hashCode() {
int result = Objects.hash(illustId, page, fileName, size);
result = 31 * result + Arrays.hashCode(checksum);
return result;
}
}
}

View File

@ -0,0 +1,35 @@
package net.lamgc.cgj.bot.cache;
public final class CacheStoreUtils {
private CacheStoreUtils() {}
/**
* 将 CacheStore 转换成 基于 {@link LocalHashCacheStore} 的 {@link HotDataCacheStore}
* <p>通过该方法转换, 会自动启用 自动清理</p>
* @param cacheStore 上游缓存库
* @param expireTime 热点缓存最小有效期
* @param floatRange 缓存浮动最大范围
* @param <T> 缓存库数据类型
* @return 返回 {@link HotDataCacheStore}
*/
public static <T> CacheStore<T> hashLocalHotDataStore(CacheStore<T> cacheStore, long expireTime, int floatRange) {
return hashLocalHotDataStore(cacheStore, expireTime, floatRange, true);
}
/**
* 将 CacheStore 转换成 基于 {@link LocalHashCacheStore} 的 {@link HotDataCacheStore}
* @param cacheStore 上游缓存库
* @param expireTime 热点缓存最小有效期
* @param floatRange 缓存浮动最大范围
* @param autoClean 是否启用自动清理
* @param <T> 缓存库数据类型
* @return 返回 {@link HotDataCacheStore}
*/
public static <T> CacheStore<T> hashLocalHotDataStore(CacheStore<T> cacheStore,
long expireTime, int floatRange, boolean autoClean) {
return new HotDataCacheStore<>(cacheStore, new LocalHashCacheStore<>(), expireTime, floatRange, autoClean);
}
}

View File

@ -0,0 +1,13 @@
package net.lamgc.cgj.bot.cache;
/**
* 可清理接口, 实现该接口代表该类具有清理动作.
*/
public interface Cleanable {
/**
* 该方法需要CacheStore完成对过期Entry的清除.
*/
void clean() throws Exception;
}

View File

@ -10,31 +10,38 @@ import java.util.*;
* @param <T> 存储类型
* @author LamGC
*/
public class HotDataCacheStore<T> implements CacheStore<T> {
public class HotDataCacheStore<T> implements CacheStore<T>, Cleanable {
private final CacheStore<T> parent;
private final CacheStore<T> current;
private final long expireTime;
private final int expireFloatRange;
private final Random random = new Random();
private final Logger log = LoggerFactory.getLogger(HotDataCacheStore.class.getSimpleName() + "@" + Integer.toHexString(this.hashCode()));
private final Logger log = LoggerFactory.getLogger(this.toString());
/**
* 构造热点缓存存储对象
* @param parent 上级缓存存储库
* @param current 热点缓存存储库, 最好使用本地缓存(例如 {@linkplain LocalHashCacheStore LocalHashCacheStore})
* @param expireTime 本地缓存库的缓存项过期时间, 单位毫秒;
* 该时间并不是所有缓存项的最终过期时间, 还需要根据expireFloatRange的设定随机设置, 公式:
* {@code expireTime + new Random().nextInt(expireFloatRange)}
* @param expireFloatRange 过期时间的浮动范围(单位毫秒), 用于防止短时间内大量缓存项失效导致的缓存雪崩
* 该时间并不是所有缓存项的最终过期时间, 还需要根据expireFloatRange的设定随机设置, 公式:
* {@code expireTime + new Random().nextInt(expireFloatRange)}
* @param expireFloatRange 过期时间的浮动范围(单位毫秒), 用于防止短时间内大量缓存项失效导致的缓存雪崩,
* 如设置为0或负数, 则不启用浮动范围.
* @param autoClean 是否交由{@link AutoCleanTimer}自动执行清理, 启用后, AutoCleanTimer会自动检查过期Key并进行删除.
*/
public HotDataCacheStore(CacheStore<T> parent, CacheStore<T> current, long expireTime, int expireFloatRange) {
public HotDataCacheStore(CacheStore<T> parent, CacheStore<T> current, long expireTime, int expireFloatRange, boolean autoClean) {
this.parent = parent;
this.current = current;
this.expireTime = expireTime;
this.expireFloatRange = expireFloatRange;
log.debug("HotDataCacheStore初始化完成. (Parent: {}, Current: {}, expireTime: {}, expireFloatRange: {})",
parent, current, expireTime, expireFloatRange);
if(autoClean) {
AutoCleanTimer.add(this);
}
log.trace("HotDataCacheStore初始化完成. " +
"(Parent: {}, Current: {}, expireTime: {}, expireFloatRange: {}, autoClean: {})",
parent, current, expireTime, expireFloatRange, autoClean);
}
@Override
@ -51,23 +58,27 @@ public class HotDataCacheStore<T> implements CacheStore<T> {
@Override
public T getCache(String key) {
if(!exists(key)) {
log.debug("查询缓存键名不存在, 直接返回null.");
log.trace("查询缓存键名不存在, 直接返回null.");
return null;
}
T result = current.getCache(key);
if(Objects.isNull(result)) {
log.debug("Current缓存库未命中, 查询Parent缓存库");
log.trace("Current缓存库未命中, 查询Parent缓存库");
T parentResult = parent.getCache(key);
if(Objects.isNull(parentResult)) {
log.debug("Parent缓存库未命中, 缓存不存在");
log.trace("Parent缓存库未命中, 缓存不存在");
return null;
}
log.debug("Parent缓存命中, 正在更新Current缓存库...");
current.update(key, parentResult, expireTime + random.nextInt(expireFloatRange));
log.debug("Current缓存库更新完成.");
log.trace("Parent缓存命中, 正在更新Current缓存库...");
current.update(key, parentResult,
expireTime + (expireFloatRange <= 0 ? 0 : random.nextInt(expireFloatRange)));
log.trace("Current缓存库更新完成.");
result = parentResult;
} else {
log.debug("Current缓存库缓存命中.");
// 更新该Key的过期时间
current.update(key, result,
expireTime + (expireFloatRange <= 0 ? 0 : random.nextInt(expireFloatRange)));
log.trace("Current缓存库缓存命中.");
}
return result;
}
@ -114,11 +125,31 @@ public class HotDataCacheStore<T> implements CacheStore<T> {
@Override
public boolean supportedPersistence() {
// 由于Current的缓存数据会更新到Parent上,
// 所以只要任意一边支持持久化, 那么该缓存库就支持持久化
return current.supportedPersistence() || parent.supportedPersistence();
}
@Override
public boolean supportedList() {
return false;
// 只有两边都支持List, 该缓存库才会支持持久化
return current.supportedList() && parent.supportedList();
}
/**
* 检查并清理已过期的Entry.
* <p>该方法仅清理Current缓存库, 不会对上游缓存库造成影响.</p>
*/
@Override
public void clean() throws Exception {
if(current instanceof Cleanable) {
((Cleanable) current).clean();
} else {
for(String key : this.current.keys()) {
if (!current.exists(key)) {
current.remove(key);
}
}
}
}
}

View File

@ -1,53 +1,54 @@
package net.lamgc.cgj.bot.cache;
import net.lamgc.cgj.Main;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.exception.HttpRequestException;
import net.lamgc.cgj.pixiv.PixivURL;
import net.lamgc.cgj.util.URLs;
import net.lamgc.utils.event.EventHandler;
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.tomcat.util.http.fileupload.util.Streams;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.*;
import java.util.Collections;
import java.util.HashSet;
import java.util.Set;
public class ImageCacheHandler implements EventHandler {
private final static Logger log = LoggerFactory.getLogger("ImageCacheHandler");
private final static Logger log = LoggerFactory.getLogger(ImageCacheHandler.class);
private final static HttpClient httpClient = HttpClientBuilder.create().setProxy(Main.proxy).build();
private final static HttpClient httpClient = HttpClientBuilder.create()
.setProxy(BotGlobal.getGlobal().getProxy())
.build();
private final static Set<ImageCacheObject> cacheQueue = Collections.synchronizedSet(new HashSet<>());
@SuppressWarnings("unused")
public void getImageToCache(ImageCacheObject event) {
public void getImageToCache(ImageCacheObject event) throws Exception {
if(cacheQueue.contains(event)) {
log.debug("图片 {} 已存在相同缓存任务, 跳过.", event.getStoreFile().getName());
log.warn("图片 {} 已存在相同缓存任务, 跳过.", event.getStoreFile().getName());
return;
} else {
cacheQueue.add(event);
}
try {
log.info("图片 {} Event正在进行...({})", event.getStoreFile().getName(), Integer.toHexString(event.hashCode()));
log.debug("图片 {} Event正在进行...({})", event.getStoreFile().getName(), Integer.toHexString(event.hashCode()));
File storeFile = event.getStoreFile();
log.debug("正在缓存图片 {} (Path: {})", storeFile.getName(), storeFile.getAbsolutePath());
try {
if(!storeFile.exists() && !storeFile.createNewFile()) {
log.error("无法创建文件(Path: {})", storeFile.getAbsolutePath());
return;
throw new IOException("Failed to create file");
}
} catch (IOException e) {
log.error("无法创建文件(Path: {})", storeFile.getAbsolutePath());
e.printStackTrace();
throw e;
}
HttpGet request = new HttpGet(event.getDownloadLink());
@ -57,23 +58,36 @@ public class ImageCacheHandler implements EventHandler {
response = httpClient.execute(request);
} catch (IOException e) {
log.error("Http请求时发生异常", e);
return;
throw e;
}
if(response.getStatusLine().getStatusCode() != 200) {
log.warn("Http请求异常{}", response.getStatusLine());
return;
HttpRequestException requestException = new HttpRequestException(response);
log.warn("Http请求异常{}", requestException.getStatusLine());
throw requestException;
}
log.debug("正在下载...(Content-Length: {}KB)", response.getEntity().getContentLength() / 1024);
try(FileOutputStream fos = new FileOutputStream(storeFile)) {
IOUtils.copy(response.getEntity().getContent(), fos);
log.trace("正在下载...(Content-Length: {}KB)", response.getEntity().getContentLength() / 1024);
ByteArrayOutputStream bufferOutputStream = new ByteArrayOutputStream();
try(FileOutputStream fileOutputStream = new FileOutputStream(storeFile)) {
Streams.copy(response.getEntity().getContent(), bufferOutputStream, false);
ByteArrayInputStream bufferInputStream = new ByteArrayInputStream(bufferOutputStream.toByteArray());
CacheStoreCentral.ImageChecksum imageChecksum = CacheStoreCentral.ImageChecksum
.buildImageChecksumFromStream(
event.getIllustId(),
event.getPageIndex(),
event.getStoreFile().getName(),
bufferInputStream
);
bufferInputStream.reset();
Streams.copy(bufferInputStream, fileOutputStream, false);
CacheStoreCentral.getCentral().setImageChecksum(imageChecksum);
} catch (IOException e) {
log.error("下载图片时发生异常", e);
return;
throw e;
}
event.getImageCache().put(URLs.getResourceName(event.getDownloadLink()), storeFile);
} finally {
log.info("图片 {} Event结束({})", event.getStoreFile().getName(), Integer.toHexString(event.hashCode()));
log.debug("图片 {} Event结束({})", event.getStoreFile().getName(), Integer.toHexString(event.hashCode()));
cacheQueue.remove(event);
}
}

View File

@ -12,13 +12,16 @@ public class ImageCacheObject implements EventObject {
private final int illustId;
private final int pageIndex;
private final String downloadLink;
private final File storeFile;
public ImageCacheObject(Map<String, File> imageCache, int illustId, String downloadLink, File storeFile) {
public ImageCacheObject(Map<String, File> imageCache, int illustId, int pageIndex, String downloadLink, File storeFile) {
this.imageCache = imageCache;
this.illustId = illustId;
this.pageIndex = pageIndex;
this.downloadLink = downloadLink;
this.storeFile = storeFile;
}
@ -39,12 +42,17 @@ public class ImageCacheObject implements EventObject {
return illustId;
}
public int getPageIndex() {
return pageIndex;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
ImageCacheObject that = (ImageCacheObject) o;
return illustId == that.illustId &&
pageIndex == that.pageIndex &&
Objects.equals(imageCache, that.imageCache) &&
Objects.equals(downloadLink, that.downloadLink) &&
Objects.equals(storeFile, that.storeFile);
@ -52,13 +60,15 @@ public class ImageCacheObject implements EventObject {
@Override
public int hashCode() {
return Objects.hash(imageCache, illustId, downloadLink, storeFile);
return Objects.hash(imageCache, illustId, pageIndex, downloadLink, storeFile);
}
@Override
public String toString() {
return "ImageCacheObject@" + Integer.toHexString(hashCode()) + "{" +
"illustId=" + illustId +
return "ImageCacheObject{" +
"imageCache=" + imageCache +
", illustId=" + illustId +
", pageIndex=" + pageIndex +
", downloadLink='" + downloadLink + '\'' +
", storeFile=" + storeFile +
'}';

View File

@ -0,0 +1,142 @@
package net.lamgc.cgj.bot.cache;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import net.lamgc.cgj.exception.HttpRequestException;
import java.util.Hashtable;
import java.util.Map;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.ReentrantLock;
public final class ImageCacheStore {
private final static Logger log = LoggerFactory.getLogger(ImageCacheStore.class);
private final static Map<ImageCacheObject, Task> cacheMap = new Hashtable<>();
private final static ThreadPoolExecutor imageCacheExecutor = new ThreadPoolExecutor(
4, 6,
30L,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>(100),
new ThreadFactoryBuilder()
.setNameFormat("ImageCacheThread-%d")
.build()
);
private final static ImageCacheHandler handler = new ImageCacheHandler();
static {
Thread shutdownThread = new Thread(imageCacheExecutor::shutdownNow);
shutdownThread.setName("Thread-ImageCacheShutdown");
Runtime.getRuntime().addShutdownHook(shutdownThread);
}
private ImageCacheStore() {}
/**
* 传递图片缓存任务, 并等待缓存完成.
* @param cacheObject 缓存任务组
*/
public static Throwable executeCacheRequest(ImageCacheObject cacheObject) throws InterruptedException {
Task task = getTaskState(cacheObject);
if(task.taskState.get() == TaskState.COMPLETE) {
return null;
}
boolean locked = false;
try {
if(task.taskState.get() == TaskState.COMPLETE) {
return null;
}
task.lock.lock();
locked = true;
// 双重检查
if(task.taskState.get() == TaskState.COMPLETE) {
return null;
}
// 置任务状态
task.taskState.set(TaskState.RUNNING);
Future<Throwable> future = imageCacheExecutor.submit(() -> {
try {
handler.getImageToCache(cacheObject);
} catch (Throwable e) {
return e;
}
return null;
});
Throwable throwable;
try {
throwable = future.get();
if(throwable == null) {
task.taskState.set(TaskState.COMPLETE);
} else {
task.taskState.set(TaskState.ERROR);
}
} catch (ExecutionException e) {
log.error("执行图片缓存任务时发生异常", e);
task.taskState.set(TaskState.ERROR);
return e.getCause();
} catch (InterruptedException e) {
future.cancel(true);
task.taskState.set(TaskState.ERROR);
throw e;
}
return throwable;
} finally {
if(locked) {
task.lock.unlock();
}
}
}
private static Task getTaskState(ImageCacheObject cacheObject) {
if(!cacheMap.containsKey(cacheObject)) {
cacheMap.put(cacheObject, new Task());
}
return cacheMap.get(cacheObject);
}
/**
* 获取错误信息
*/
public static String getErrorMessageFromThrowable(Throwable throwable, boolean onlyRequestException) {
if(throwable == null) {
return "";
} else if(!(throwable instanceof HttpRequestException)) {
if(onlyRequestException) {
return "";
}
return throwable.getMessage();
}
JsonObject result = new Gson()
.fromJson(((HttpRequestException) throwable).getContent(), JsonObject.class);
return result.get("msg").getAsString();
}
/**
* 任务状态
*/
private enum TaskState {
READY, RUNNING, COMPLETE, ERROR
}
private static class Task {
public final ReentrantLock lock = new ReentrantLock(true);
public final AtomicReference<TaskState> taskState = new AtomicReference<>(TaskState.READY);
}
}

View File

@ -2,29 +2,52 @@ package net.lamgc.cgj.bot.cache;
import org.jetbrains.annotations.NotNull;
import java.util.Date;
import java.util.Hashtable;
import java.util.Objects;
import java.util.Set;
import java.util.*;
import java.util.concurrent.atomic.AtomicReference;
/**
* 基于Hashtable的本地缓存库
* @param <T> 缓存类型
*/
public class LocalHashCacheStore<T> implements CacheStore<T> {
public class LocalHashCacheStore<T> implements CacheStore<T>, Cleanable {
private final Hashtable<String, CacheObject<T>> cache;
/**
* 构造一个基于Hashtable的本地缓存库
* @see Hashtable
*/
public LocalHashCacheStore() {
this(0);
}
/**
* 构造一个基于Hashtable的本地缓存库
* @param initialCapacity 初始容量
* @see Hashtable
*/
public LocalHashCacheStore(int initialCapacity) {
this(initialCapacity, 0F);
}
/**
* 构造一个基于Hashtable的本地缓存库
* @param initialCapacity 初始容量
* @param loadFactor 重载因子
* @see Hashtable
*/
public LocalHashCacheStore(int initialCapacity, float loadFactor) {
this(initialCapacity, loadFactor, false);
}
/**
* 构造一个基于Hashtable的本地缓存库
* @param initialCapacity 初始容量
* @param loadFactor 重载因子
* @param autoClean 是否自动清理
* @see Hashtable
*/
public LocalHashCacheStore(int initialCapacity, float loadFactor, boolean autoClean) {
if(initialCapacity != 0) {
if(loadFactor <= 0F) {
cache = new Hashtable<>(initialCapacity);
@ -34,6 +57,10 @@ public class LocalHashCacheStore<T> implements CacheStore<T> {
} else {
cache = new Hashtable<>();
}
if(autoClean) {
AutoCleanTimer.add(this);
}
}
@Override
@ -118,11 +145,23 @@ public class LocalHashCacheStore<T> implements CacheStore<T> {
return false;
}
@Override
public void clean() {
Date currentDate = new Date();
Set<String> expireKeySet = new HashSet<>();
cache.forEach((key, value) -> {
if(value.isExpire(currentDate)) {
expireKeySet.add(key);
}
});
expireKeySet.forEach(cache::remove);
}
public static class CacheObject<T> implements Comparable<CacheObject<T>> {
private AtomicReference<T> value;
private AtomicReference<Date> expire;
private final AtomicReference<T> value;
private final AtomicReference<Date> expire;
public CacheObject(T value, Date expire) {
this.value = new AtomicReference<>(value);

View File

@ -13,7 +13,7 @@ import java.util.Set;
import java.util.function.Consumer;
import java.util.function.Function;
public abstract class RedisPoolCacheStore<T> implements CacheStore<T> {
abstract class RedisPoolCacheStore<T> implements CacheStore<T> {
private final JedisPool jedisPool;
private final String keyPrefix;
@ -36,7 +36,7 @@ public abstract class RedisPoolCacheStore<T> implements CacheStore<T> {
if(jedisPool.isClosed()) {
throw new IllegalStateException("JedisPool is closed");
}
log = LoggerFactory.getLogger(this.getClass().getSimpleName() + "@" + Integer.toHexString(jedisPool.hashCode()));
log = LoggerFactory.getLogger(this.getClass().getName() + "@" + Integer.toHexString(jedisPool.hashCode()));
if(!Strings.isNullOrEmpty(keyPrefix)) {
this.keyPrefix = keyPrefix.endsWith(".") ? keyPrefix : keyPrefix + ".";
} else {
@ -51,27 +51,27 @@ public abstract class RedisPoolCacheStore<T> implements CacheStore<T> {
@Override
public void update(String key, T value, Date expire) {
try (Jedis jedis = jedisPool.getResource()) {
executeJedisCommand(jedis -> {
jedis.set(keyPrefix + key, parse(value));
if(expire != null) {
jedis.pexpireAt(keyPrefix + key, expire.getTime());
log.debug("已设置Key {} 的过期时间(Expire: {})", key, expire.getTime());
}
}
});
}
@Override
public T getCache(String key) {
try (Jedis jedis = jedisPool.getResource()) {
return executeJedisCommand(jedis -> {
return analysis(jedis.get(keyPrefix + key));
}
});
}
@Override
public boolean exists(String key) {
try (Jedis jedis = jedisPool.getResource()) {
return executeJedisCommand(jedis -> {
return jedis.exists(keyPrefix + key);
}
});
}
@Override
@ -81,23 +81,21 @@ public abstract class RedisPoolCacheStore<T> implements CacheStore<T> {
@Override
public boolean clear() {
try (Jedis jedis = jedisPool.getResource()) {
return executeJedisCommand(jedis -> {
return jedis.flushDB().equalsIgnoreCase("ok");
}
});
}
@Override
public Set<String> keys() {
try (Jedis jedis = jedisPool.getResource()) {
return executeJedisCommand(jedis -> {
return jedis.keys(keyPrefix + "*");
}
});
}
@Override
public boolean remove(String key) {
try (Jedis jedis = jedisPool.getResource()) {
return jedis.del(keyPrefix + key) == 1;
}
return executeJedisCommand(jedis -> jedis.del(keyPrefix + key) == 1);
}
/**

View File

@ -5,27 +5,26 @@ import com.google.common.base.Throwables;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import net.lamgc.cgj.bot.BotAdminCommandProcess;
import net.lamgc.cgj.bot.BotCommandProcess;
import net.lamgc.cgj.bot.MessageEventExecutionDebugger;
import net.lamgc.cgj.bot.SettingProperties;
import net.lamgc.cgj.util.DateParser;
import net.lamgc.cgj.util.PagesQualityParser;
import net.lamgc.cgj.util.TimeLimitThreadPoolExecutor;
import net.lamgc.utils.base.runner.ArgumentsRunner;
import net.lamgc.utils.base.runner.ArgumentsRunnerConfig;
import net.lamgc.utils.base.runner.exception.DeveloperRunnerException;
import net.lamgc.utils.base.runner.exception.NoSuchCommandException;
import net.lamgc.utils.base.runner.exception.ParameterNoFoundException;
import net.lamgc.utils.event.EventExecutor;
import net.lamgc.utils.event.EventHandler;
import net.lamgc.utils.event.*;
import net.lamgc.utils.event.EventObject;
import net.lamgc.utils.event.EventUncaughtExceptionHandler;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import redis.clients.jedis.JedisPool;
import java.lang.reflect.Method;
import java.net.URI;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.regex.Matcher;
@ -39,22 +38,19 @@ public class BotEventHandler implements EventHandler {
private final ArgumentsRunner processRunner;
private final ArgumentsRunner adminRunner;
private final static Logger log = LoggerFactory.getLogger("BotEventHandler");
private final static Logger log = LoggerFactory.getLogger(BotEventHandler.class);
/**
* 所有缓存共用的JedisPool
*/
private final static URI redisServerUri = URI.create("redis://" + System.getProperty("cgj.redisAddress"));
public final static JedisPool redisServer = new JedisPool(redisServerUri.getHost(), redisServerUri.getPort() == -1 ? 6379 : redisServerUri.getPort());
private final static Map<Long, AtomicBoolean> muteStateMap = new Hashtable<>();
/**
* 消息事件执行器
*/
public final static EventExecutor executor = new EventExecutor(new ThreadPoolExecutor(
(int) Math.ceil(Runtime.getRuntime().availableProcessors() / 2F),
Runtime.getRuntime().availableProcessors(),
private final static EventExecutor executor = new EventExecutor(new TimeLimitThreadPoolExecutor(
180000, // 3minThr
Math.max(Runtime.getRuntime().availableProcessors(), 4),
Math.min(Math.max(Runtime.getRuntime().availableProcessors(), 4), 32),
30L,
TimeUnit.SECONDS,
TimeUnit.SECONDS,
new LinkedBlockingQueue<>(1536),
new ThreadFactoryBuilder()
.setNameFormat("CommandProcess-%d")
@ -62,23 +58,26 @@ public class BotEventHandler implements EventHandler {
));
private static boolean initialled = false;
static {
initial();
}
/**
* 初始化BotEventHandler
*/
public synchronized static void initial() {
if(initialled) {
Logger logger = LoggerFactory.getLogger("BotEventHandler@<init>");
logger.warn("BotEventHandler已经执行过初始化方法, 可能存在多次执行的问题, 堆栈信息: \n {}",
log.warn("BotEventHandler已经执行过初始化方法, 可能存在多次执行的问题, 堆栈信息: \n {}",
Throwables.getStackTraceAsString(new Exception()));
return;
}
executor.setEnableEventResend(true);
executor.setEventUncaughtExceptionHandler(new EventUncaughtExceptionHandler() {
private final Logger log = LoggerFactory.getLogger("EventUncaughtExceptionHandler");
private final Logger log = LoggerFactory.getLogger(this.getClass());
@Override
public void exceptionHandler(Thread executeThread, EventHandler handler, Method handlerMethod, EventObject event, Throwable cause) {
log.error("发生未捕获异常:\nThread:{}, EventHandler: {}, HandlerMethod: {}, EventObject: {}\n{}",
log.error("EventExecutor@{} 发生未捕获异常:\n\t" +
"Thread:{}\n\tEventHandler: {}\n\tHandlerMethod: {}\n\tEventObject: {}\n" +
"------------------ Stack Trace ------------------\n{}",
executor.hashCode(),
executeThread.getName(),
handler.toString(),
handlerMethod.getName(),
@ -88,25 +87,19 @@ public class BotEventHandler implements EventHandler {
});
try {
executor.addHandler(new BotEventHandler());
Thread shutdownThread = new Thread(() -> executor.shutdown(true));
shutdownThread.setName("Thread-EventHandlerShutdown");
Runtime.getRuntime().addShutdownHook(shutdownThread);
} catch (IllegalAccessException e) {
LoggerFactory.getLogger("BotEventHandler@Static").error("添加Handler时发生异常", e);
log.error("添加Handler时发生异常", e);
}
initialled = true;
}
private final static AtomicBoolean preLoaded = new AtomicBoolean();
/**
* 预加载
*/
public synchronized static void preLoad() {
if(preLoaded.get()) {
return;
}
try {
BotAdminCommandProcess.loadPushList();
} finally {
preLoaded.set(true);
} catch(Throwable e) {
log.error("加载推送列表失败", e);
}
initialled = true;
}
private BotEventHandler() {
@ -116,14 +109,54 @@ public class BotEventHandler implements EventHandler {
runnerConfig.addStringParameterParser(new DateParser(new SimpleDateFormat("yyyy-MM-dd")));
runnerConfig.addStringParameterParser(new PagesQualityParser());
log.debug("DateParser添加情况: {}", runnerConfig.hasStringParameterParser(Date.class));
processRunner = new ArgumentsRunner(BotCommandProcess.class, runnerConfig);
adminRunner = new ArgumentsRunner(BotAdminCommandProcess.class, runnerConfig);
BotCommandProcess.initialize();
}
/**
* 投递消息事件
* @param event 事件对象
*/
@NotAccepted
public static void executeMessageEvent(MessageEvent event) {
try {
executeMessageEvent(event, false);
} catch (InterruptedException e) {
log.error("执行时发生异常", e);
throw new RuntimeException(e);
}
}
/**
* 投递消息事件
* @param event 事件对象
* @param sync 是否同步执行事件
*/
@NotAccepted
public static void executeMessageEvent(MessageEvent event, boolean sync) throws InterruptedException {
String debuggerName = SettingProperties.getProperty(0, "debug.debugger");
if(!event.getMessage().startsWith(ADMIN_COMMAND_PREFIX) &&
!Strings.isNullOrEmpty(debuggerName)) {
try {
MessageEventExecutionDebugger debugger = MessageEventExecutionDebugger.valueOf(debuggerName.toUpperCase());
debugger.debugger.accept(executor, event, SettingProperties.getProperties(SettingProperties.GLOBAL),
MessageEventExecutionDebugger.getDebuggerLogger(debugger));
} catch(IllegalArgumentException e) {
log.warn("未找到指定调试器: '{}'", debuggerName);
} catch (Exception e) {
log.error("事件调试处理时发生异常", e);
}
} else {
if(sync) {
BotEventHandler.executor.executorSync(event);
} else {
BotEventHandler.executor.executor(event);
}
}
}
/**
* 以事件形式处理消息事件
* @param event 消息事件对象
@ -132,7 +165,10 @@ public class BotEventHandler implements EventHandler {
public void processMessage(MessageEvent event) {
String msg = event.getMessage();
log.debug(event.toString());
if(!match(msg)) {
if(mismatch(msg)) {
return;
} else if(isMute(event.getFromGroup())) {
log.debug("机器人已被禁言, 忽略请求.");
return;
}
@ -172,7 +208,8 @@ public class BotEventHandler implements EventHandler {
Object result;
try {
if(msg.toLowerCase().startsWith(ADMIN_COMMAND_PREFIX)) {
if(!String.valueOf(event.getFromQQ()).equals(BotCommandProcess.globalProp.getProperty("admin.adminId"))) {
if(!String.valueOf(event.getFromQQ())
.equals(SettingProperties.getProperty(0, "admin.adminId"))) {
result = "你没有执行该命令的权限!";
} else {
result = adminRunner.run(args.length <= 1 ? new String[0] : Arrays.copyOfRange(args, 1, args.length));
@ -185,18 +222,39 @@ public class BotEventHandler implements EventHandler {
} catch(ParameterNoFoundException e) {
result = "命令缺少参数: " + e.getParameterName();
} catch(DeveloperRunnerException e) {
log.error("执行命令时发生异常", e);
result = "命令执行时发生错误,无法完成!";
Throwable cause = e.getCause();
if (cause instanceof InterruptedException) {
log.error("命令执行超时, 终止执行.", cause);
result = "色图姬查阅图库太久,被赶出来了!";
} else if(cause instanceof NoSuchElementException && cause.getMessage().startsWith("No work found: ")) {
String message = cause.getMessage();
log.error("指定作品不存在.(Id: {})", message.substring(message.lastIndexOf(": ") + 2));
result = "色图姬找不到这个作品!";
} else {
log.error("执行命令时发生异常", e);
result = "色图姬在执行命令时遇到了一个错误!";
}
}
log.info("命令处理完成.(耗时: {}ms)", System.currentTimeMillis() - time);
if(Objects.requireNonNull(result) instanceof String) {
long processTime = System.currentTimeMillis() - time;
if(!Objects.isNull(result) && result instanceof String && !isMute(event.getFromGroup())) {
try {
event.sendMessage((String) result);
int sendResult = event.sendMessage((String) result);
if (sendResult < 0) {
log.warn("消息发送失败, Sender {} 返回错误代码: {}", event.getClass().getName(), sendResult);
}
} catch(InterruptedException e) {
log.info("事件在发送消息时超时, 重新投递该事件.(Event: {})", event);
EventExecutor.resendCurrentEvent();
} catch (Exception e) {
log.error("发送消息时发生异常", e);
}
} else if(isMute(event.getFromGroup())) {
log.warn("命令反馈时机器人已被禁言, 跳过反馈.");
}
log.info("命令反馈完成.(耗时: {}ms)", System.currentTimeMillis() - time);
long totalTime = System.currentTimeMillis() - time;
log.info("命令反馈完成.(事件耗时: {}ms, P: {}%({}ms), R: {}%({}ms))", totalTime,
String.format("%.3f", ((double) processTime / (double)totalTime) * 100F), processTime,
String.format("%.3f", ((double) (totalTime - processTime) / (double)totalTime) * 100F), totalTime - processTime);
}
/**
@ -204,8 +262,49 @@ public class BotEventHandler implements EventHandler {
* @param message 要检查的消息
* @return 如果为true则提交
*/
public static boolean match(String message) {
return message.startsWith(COMMAND_PREFIX) || message.startsWith(ADMIN_COMMAND_PREFIX);
public static boolean mismatch(String message) {
return !message.startsWith(COMMAND_PREFIX) && !message.startsWith(ADMIN_COMMAND_PREFIX);
}
/**
* 查询某群组中Bot是否被禁言
* @param groupId 待查询的群组号
* @return 如果被禁言, 返回true, 如果未被禁言或禁言情况未知, 返回false
*/
private static boolean isMute(long groupId) {
Boolean mute = isMute(groupId, false);
return mute != null && mute;
}
/**
* 查询某群是否被禁言.
* @param groupId 群组Id
* @param rawValue 是否返回原始值(当没有该群状态, 且本参数为true时, 将返回null)
* @return 返回状态值, 如无该群禁言记录且rawValue = true, 则返回null
*/
public static Boolean isMute(long groupId, boolean rawValue) {
if(groupId <= 0) {
return false;
}
AtomicBoolean state = muteStateMap.get(groupId);
if(state == null && rawValue) {
return null;
}
return state != null && state.get();
}
/**
* 设置机器人禁言状态.
* <p>设置该项可防止因机器人在禁言期间反馈请求导致被封号.</p>
* @param mute 如果被禁言, 传入true
*/
public static void setMuteState(long groupId, boolean mute) {
if(!muteStateMap.containsKey(groupId)) {
muteStateMap.put(groupId, new AtomicBoolean(mute));
} else {
muteStateMap.get(groupId).set(mute);
}
log.warn("群组 {} 机器人禁言状态已变更: {}", groupId, mute ? "已禁言" : "已解除");
}
}

View File

@ -0,0 +1,23 @@
package net.lamgc.cgj.bot.event;
import net.lamgc.cgj.bot.message.MessageSender;
public class BufferedMessageSender implements MessageSender {
private final StringBuffer buffer = new StringBuffer();
@Override
public int sendMessage(String message) {
buffer.append(message);
return 0;
}
/**
* 从缓冲区中取出消息内容.
* @return 返回事件发送的消息内容.
*/
public String getBufferContent() {
return buffer.toString();
}
}

View File

@ -12,16 +12,17 @@ public abstract class MessageEvent implements EventObject, MessageSender {
public MessageEvent(long fromGroup, long fromQQ, String message) {
this.fromGroup = fromGroup;
this.fromQQ = fromQQ;
this.message = message;
this.message = message.trim();
}
/**
* 发送消息
* @param message 消息内容
* @return 成功返回MessageId, 如没有MessageId则返回0, 失败返回负数错误码
* @throws Exception 该方法根据不同实现, 可能会抛出不同异常, 详见实现所标识的文档内容.
*/
@Override
public abstract int sendMessage(final String message);
public abstract int sendMessage(final String message) throws Exception;
/**
* 获取图片下载地址.
@ -56,7 +57,7 @@ public abstract class MessageEvent implements EventObject, MessageSender {
@Override
public String toString() {
return this.getClass().getSimpleName() + "{" +
return this.getClass().getSimpleName() + "@" + this.hashCode() + "{" +
"fromGroup=" + getFromGroup() +
", fromQQ=" + getFromQQ() +
", message='" + getMessage() + '\'' +

View File

@ -5,6 +5,28 @@ package net.lamgc.cgj.bot.event;
*/
public class VirtualLoadMessageEvent extends MessageEvent {
/**
* 将任意消息事件转换为假负载消息事件.
* <p>转换之后, 除了fromGroup, fromQQ, message外其他信息不会保留</p>
* @param event 待转换的消息事件
* @param inheritImpl 是否继承除 sendMessage 外的其他 MessageEvent 实现
* @return 转换后的消息事件
*/
public static VirtualLoadMessageEvent toVirtualLoadMessageEvent(MessageEvent event, boolean inheritImpl) {
if(event instanceof VirtualLoadMessageEvent) {
return (VirtualLoadMessageEvent) event;
} else if(!inheritImpl) {
return new VirtualLoadMessageEvent(event.getFromGroup(), event.getFromQQ(), event.getMessage());
} else {
return new VirtualLoadMessageEvent(event.getFromGroup(), event.getFromQQ(), event.getMessage()) {
@Override
public String getImageUrl(String image) {
return event.getImageUrl(image);
}
};
}
}
public VirtualLoadMessageEvent(long fromGroup, long fromQQ, String message) {
super(fromGroup, fromQQ, message);
}

View File

@ -0,0 +1,51 @@
package net.lamgc.cgj.bot.framework.cli;
import net.lamgc.cgj.bot.boot.ApplicationBoot;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.bot.framework.cli.message.ConsoleMessageEvent;
import net.lamgc.cgj.bot.framework.cli.message.ConsoleMessageSenderFactory;
import net.lamgc.cgj.bot.message.MessageSenderBuilder;
import org.jline.reader.LineReader;
import org.jline.reader.LineReaderBuilder;
import org.jline.reader.impl.history.DefaultHistory;
import org.jline.terminal.TerminalBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
public class ConsoleMain {
private final static Logger log = LoggerFactory.getLogger(ConsoleMain.class);
public static void start() throws IOException {
MessageSenderBuilder.setCurrentMessageSenderFactory(new ConsoleMessageSenderFactory());
ApplicationBoot.initialBot();
LineReader lineReader = LineReaderBuilder.builder()
.appName("CGJ")
.history(new DefaultHistory())
.terminal(TerminalBuilder.terminal())
.build();
long qqId = Long.parseLong(lineReader.readLine("会话QQ: "));
long groupId = Long.parseLong(lineReader.readLine("会话群组号:"));
boolean isGroup = false;
do {
String input = lineReader.readLine("App " + qqId + (isGroup ? "@" + groupId : "#private") + " >");
if(input.equalsIgnoreCase("#exit")) {
System.out.println("退出应用...");
break;
} else if(input.equalsIgnoreCase("#setgroup")) {
isGroup = !isGroup;
System.out.println("System: 群模式状态已变更: " + isGroup);
continue;
}
try {
BotEventHandler.executeMessageEvent(new ConsoleMessageEvent(isGroup ? groupId : 0, qqId, input), true);
} catch (InterruptedException e) {
log.error("执行时发生中断", e);
}
} while(true);
}
}

View File

@ -0,0 +1,29 @@
package net.lamgc.cgj.bot.framework.cli.message;
import net.lamgc.cgj.bot.event.MessageEvent;
import net.lamgc.cgj.bot.message.MessageSenderBuilder;
import net.lamgc.cgj.bot.message.MessageSource;
public class ConsoleMessageEvent extends MessageEvent {
public ConsoleMessageEvent(long groupId, long qqId, String message) {
super(groupId, qqId, message);
}
@Override
public int sendMessage(String message) throws Exception {
if(getFromGroup() <= 0) {
return MessageSenderBuilder
.getMessageSender(MessageSource.PRIVATE, getFromQQ()).sendMessage(message);
} else {
return MessageSenderBuilder
.getMessageSender(MessageSource.GROUP, getFromQQ()).sendMessage(message);
}
}
@Override
public String getImageUrl(String image) {
return null;
}
}

View File

@ -0,0 +1,26 @@
package net.lamgc.cgj.bot.framework.cli.message;
import net.lamgc.cgj.bot.message.MessageSender;
import net.lamgc.cgj.bot.message.MessageSource;
import java.text.SimpleDateFormat;
import java.util.Date;
public class ConsoleMessageSender implements MessageSender {
private final static SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS");
private final MessageSource source;
private final long id;
ConsoleMessageSender(MessageSource source, long id) {
this.source = source;
this.id = id;
}
@Override
public synchronized int sendMessage(String message) {
System.out.println(dateFormat.format(new Date()) + " Bot -> " +
(source == MessageSource.PRIVATE ? "#" : "@") + id + ": " + message);
return 0;
}
}

View File

@ -0,0 +1,13 @@
package net.lamgc.cgj.bot.framework.cli.message;
import net.lamgc.cgj.bot.message.MessageSender;
import net.lamgc.cgj.bot.message.MessageSenderFactory;
import net.lamgc.cgj.bot.message.MessageSource;
public class ConsoleMessageSenderFactory implements MessageSenderFactory {
@Override
public MessageSender createMessageSender(MessageSource source, long id) {
return new ConsoleMessageSender(source, id);
}
}

View File

@ -1,5 +1,6 @@
package net.lamgc.cgj.bot.framework.coolq;
import net.lamgc.cgj.bot.boot.ApplicationBoot;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.bot.framework.coolq.message.SpringCQMessageEvent;
import net.lamgc.utils.event.EventHandler;
@ -17,8 +18,9 @@ public class CQPluginMain extends CQPlugin implements EventHandler {
public CQPluginMain() {
// TODO(LamGC, 2020.04.21): SpringCQ无法适配MessageSenderBuilder
BotEventHandler.preLoad();
LoggerFactory.getLogger(this.toString())
// MessageSenderBuilder.setCurrentMessageSenderFactory(new SpringCQMessageSenderFactory());
ApplicationBoot.initialBot();
LoggerFactory.getLogger(CQPluginMain.class)
.info("BotEventHandler.COMMAND_PREFIX = {}", BotEventHandler.COMMAND_PREFIX);
}
@ -37,11 +39,17 @@ public class CQPluginMain extends CQPlugin implements EventHandler {
return processMessage(cq, event);
}
public int processMessage(CoolQ cq, CQMessageEvent event) {
if(!BotEventHandler.match(event.getMessage())) {
/**
* 处理消息
* @param cq CoolQ机器人对象
* @param event 消息事件
* @return 是否拦截消息
*/
private static int processMessage(CoolQ cq, CQMessageEvent event) {
if(BotEventHandler.mismatch(event.getMessage())) {
return MESSAGE_IGNORE;
}
BotEventHandler.executor.executor(new SpringCQMessageEvent(cq, event));
BotEventHandler.executeMessageEvent(new SpringCQMessageEvent(cq, event));
return MESSAGE_BLOCK;
}

View File

@ -26,17 +26,17 @@ public class SpringCQMessageEvent extends MessageEvent {
this.cq = Objects.requireNonNull(cq);
MessageSource source;
if(messageEvent instanceof CQGroupMessageEvent) {
source = MessageSource.Group;
source = MessageSource.GROUP;
} else if (messageEvent instanceof CQDiscussMessageEvent) {
source = MessageSource.Discuss;
source = MessageSource.DISCUSS;
} else {
source = MessageSource.Private;
source = MessageSource.PRIVATE;
}
messageSender = new SpringCQMessageSender(cq, source, source == MessageSource.Private ? getFromQQ() : getFromGroup());
messageSender = new SpringCQMessageSender(cq, source, source == MessageSource.PRIVATE ? getFromQQ() : getFromGroup());
}
@Override
public int sendMessage(final String message) {
public int sendMessage(final String message) throws Exception {
return messageSender.sendMessage(message);
}

View File

@ -6,9 +6,9 @@ import net.lz1998.cq.robot.CoolQ;
public class SpringCQMessageSender implements MessageSender {
private CoolQ coolQ;
private MessageSource source;
private long target;
private final CoolQ coolQ;
private final MessageSource source;
private final long target;
public SpringCQMessageSender(CoolQ coolQ, MessageSource source, long target) {
this.coolQ = coolQ;
@ -19,11 +19,11 @@ public class SpringCQMessageSender implements MessageSender {
@Override
public int sendMessage(String message) {
switch (source) {
case Private:
case PRIVATE:
return coolQ.sendPrivateMsg(target, message, false).getData().getMessageId();
case Group:
case GROUP:
return coolQ.sendGroupMsg(target, message, false).getData().getMessageId();
case Discuss:
case DISCUSS:
return coolQ.sendDiscussMsg(target, message, false).getData().getMessageId();
default:
return -1;

View File

@ -5,16 +5,14 @@ import net.lamgc.cgj.bot.message.MessageSenderFactory;
import net.lamgc.cgj.bot.message.MessageSource;
import net.lz1998.cq.robot.CoolQ;
import java.util.Objects;
public class SpringCQMessageSenderFactory implements MessageSenderFactory {
private final CoolQ coolQ;
public SpringCQMessageSenderFactory(CoolQ coolQ) {
this.coolQ = coolQ;
}
private final static ThreadLocal<CoolQ> threadCoolQ = new ThreadLocal<>();
@Override
public MessageSender createMessageSender(MessageSource source, long id) {
return new SpringCQMessageSender(coolQ, source, id);
return new SpringCQMessageSender(
Objects.requireNonNull(threadCoolQ.get(), "CoolQ object is not included in ThreadLocal"), source, id);
}
}

View File

@ -1,25 +1,32 @@
package net.lamgc.cgj.bot.framework.mirai;
import net.lamgc.cgj.bot.boot.ApplicationBoot;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.bot.framework.mirai.message.MiraiMessageEvent;
import net.lamgc.cgj.bot.message.MessageSenderBuilder;
import net.lamgc.cgj.bot.framework.mirai.message.MiraiMessageSenderFactory;
import net.lamgc.cgj.bot.message.MessageSenderBuilder;
import net.mamoe.mirai.Bot;
import net.mamoe.mirai.BotFactoryJvm;
import net.mamoe.mirai.event.events.BotMuteEvent;
import net.mamoe.mirai.event.events.BotUnmuteEvent;
import net.mamoe.mirai.japt.Events;
import net.mamoe.mirai.message.FriendMessage;
import net.mamoe.mirai.message.GroupMessage;
import net.mamoe.mirai.message.FriendMessageEvent;
import net.mamoe.mirai.message.GroupMessageEvent;
import net.mamoe.mirai.message.MessageEvent;
import net.mamoe.mirai.message.TempMessageEvent;
import net.mamoe.mirai.utils.BotConfiguration;
import org.apache.commons.net.util.Base64;
import net.mamoe.mirai.utils.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.*;
import java.util.Base64;
import java.util.Properties;
public class MiraiMain implements Closeable {
private final Logger log = LoggerFactory.getLogger(this.toString());
private final Logger log = LoggerFactory.getLogger(MiraiMain.class);
private Bot bot;
@ -34,7 +41,7 @@ public class MiraiMain implements Closeable {
return;
}
File botPropFile = new File("./bot.properties");
File botPropFile = new File(BotGlobal.getGlobal().getDataStoreDir(), "./bot.properties");
try (Reader reader = new BufferedReader(new FileReader(botPropFile))) {
botProperties.load(reader);
} catch (IOException e) {
@ -42,18 +49,65 @@ public class MiraiMain implements Closeable {
return;
}
bot = BotFactoryJvm.newBot(Long.parseLong(botProperties.getProperty("bot.qq", "0")), Base64.decodeBase64(botProperties.getProperty("bot.password", "")), new BotConfiguration());
Events.subscribeAlways(GroupMessage.class, (msg) -> BotEventHandler.executor.executor(new MiraiMessageEvent(msg)));
Events.subscribeAlways(FriendMessage.class, (msg) -> BotEventHandler.executor.executor(new MiraiMessageEvent(msg)));
Utils.setDefaultLogger(MiraiToSlf4jLoggerAdapter::new);
BotConfiguration configuration = new BotConfiguration();
configuration.randomDeviceInfo();
configuration.setProtocol(BotConfiguration.MiraiProtocol.ANDROID_PAD);
// 心跳包周期间隔 (ms)
configuration.setHeartbeatPeriodMillis(
Long.parseLong(botProperties.getProperty("network.heartbeatPeriodMillis", "60000")));
// 心跳包超时时间 (ms)
configuration.setHeartbeatTimeoutMillis(
Long.parseLong(botProperties.getProperty("network.heartbeatTimeoutMillis", "5000")));
// 重连间隔时间
configuration.setReconnectPeriodMillis(
Integer.parseInt(botProperties.getProperty("network.reconnectPeriodMillis", "5")));
// 重连最大次数
configuration.setReconnectionRetryTimes(
Integer.parseInt(botProperties.getProperty("network.reconnectionRetryTimes", "10")));
bot = BotFactoryJvm.newBot(Long.parseLong(botProperties.getProperty("bot.qq", "0")),
Base64.getDecoder().decode(botProperties.getProperty("bot.password", "")), configuration);
Events.subscribeAlways(GroupMessageEvent.class, this::executeMessageEvent);
Events.subscribeAlways(FriendMessageEvent.class, this::executeMessageEvent);
Events.subscribeAlways(TempMessageEvent.class, this::executeMessageEvent);
Events.subscribeAlways(BotMuteEvent.class,
event -> BotEventHandler.setMuteState(event.getGroup().getId(), true));
Events.subscribeAlways(BotUnmuteEvent.class,
event -> BotEventHandler.setMuteState(event.getGroup().getId(), false));
bot.login();
MessageSenderBuilder.setCurrentMessageSenderFactory(new MiraiMessageSenderFactory(bot));
BotEventHandler.preLoad();
ApplicationBoot.initialBot();
bot.join();
}
public void close() {
/**
* 处理消息事件
* @param message 消息事件对象
*/
private void executeMessageEvent(MessageEvent message) {
log.debug("Mirai Message: {}", message);
if(message instanceof GroupMessageEvent) {
GroupMessageEvent GroupMessageEvent = (GroupMessageEvent) message;
if(BotEventHandler.isMute(GroupMessageEvent.getGroup().getId(), true) == null) {
BotEventHandler.setMuteState(GroupMessageEvent.getGroup().getId(),
((GroupMessageEvent) message).getGroup().getBotMuteRemaining() != 0);
}
}
BotEventHandler.executeMessageEvent(MiraiMessageEvent.covertEventObject(message));
}
/**
* 关闭机器人
*/
public synchronized void close() {
if(bot == null) {
return;
}
log.warn("正在关闭机器人...");
bot.close(null);
bot = null;
log.warn("机器人已关闭.");
}

View File

@ -0,0 +1,89 @@
package net.lamgc.cgj.bot.framework.mirai;
import net.mamoe.mirai.utils.MiraiLogger;
import net.mamoe.mirai.utils.MiraiLoggerPlatformBase;
import org.jetbrains.annotations.Nullable;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;
/**
* MiraiLoggerToSlf4jLogger适配器
* <p>该Logger通过Slf4j的Marker进行标识, loggerName为{@code mirai.[identity]}</p>
* <p>由于适配器适配方式的原因, 日志输出的调用信息将不可用(调用指向了适配器内的方法);</p>
*/
public class MiraiToSlf4jLoggerAdapter extends MiraiLoggerPlatformBase {
private final static Marker marker = MarkerFactory.getMarker("mirai");
private final Logger logger;
private final String identity;
public MiraiToSlf4jLoggerAdapter(String identity) {
this.identity = identity;
this.logger = LoggerFactory.getLogger("mirai." + identity);
}
@Override
protected void debug0(@Nullable String s) {
logger.debug(marker, s);
}
@Override
protected void debug0(@Nullable String s, @Nullable Throwable throwable) {
logger.debug(marker, s, throwable);
}
@Override
protected void error0(@Nullable String s) {
logger.error(marker, s);
}
@Override
protected void error0(@Nullable String s, @Nullable Throwable throwable) {
logger.error(marker, s, throwable);
}
@Override
protected void info0(@Nullable String s) {
logger.info(marker, s);
}
@Override
protected void info0(@Nullable String s, @Nullable Throwable throwable) {
logger.info(marker, s, throwable);
}
@Override
protected void verbose0(@Nullable String s) {
logger.trace(marker, s);
}
@Override
protected void verbose0(@Nullable String s, @Nullable Throwable throwable) {
logger.trace(marker, s, throwable);
}
@Override
protected void warning0(@Nullable String s) {
logger.warn(marker, s);
}
@Override
protected void warning0(@Nullable String s, @Nullable Throwable throwable) {
logger.warn(marker, s, throwable);
}
@Nullable
@Override
public String getIdentity() {
if(identity == null) {
MiraiLogger followerLogger = getFollower();
return followerLogger == null ? null : followerLogger.getIdentity();
} else {
return identity;
}
}
}

View File

@ -1,33 +1,92 @@
package net.lamgc.cgj.bot.framework.mirai.message;
import net.lamgc.cgj.bot.event.MessageEvent;
import net.lamgc.cgj.bot.message.MessageSender;
import net.lamgc.cgj.bot.message.MessageSource;
import net.mamoe.mirai.message.ContactMessage;
import net.mamoe.mirai.message.GroupMessage;
import net.mamoe.mirai.message.FriendMessageEvent;
import net.mamoe.mirai.message.GroupMessageEvent;
import net.mamoe.mirai.message.MessageEvent;
import net.mamoe.mirai.message.TempMessageEvent;
import net.mamoe.mirai.message.data.MessageChain;
import net.mamoe.mirai.message.data.MessageUtils;
import java.util.Objects;
public class MiraiMessageEvent extends MessageEvent {
public class MiraiMessageEvent extends net.lamgc.cgj.bot.event.MessageEvent {
private final ContactMessage messageObject;
private final MessageEvent messageObject;
private final MessageSender messageSender;
public MiraiMessageEvent(ContactMessage message) {
super(message instanceof GroupMessage ? ((GroupMessage) message).getGroup().getId() : 0,
message.getSender().getId(), message.getMessage().contentToString());
/**
* 通过Mirai的MessageEvent转换成应用支持的MessageEvent.
* @deprecated 请使用 {@link #covertEventObject(MessageEvent)}方法转换.
* @param message 消息对象
* @see #covertEventObject(MessageEvent)
*/
@Deprecated
public MiraiMessageEvent(MessageEvent message) {
super(message instanceof GroupMessageEvent ? ((GroupMessageEvent) message).getGroup().getId() : 0,
message.getSender().getId(), getMessageBodyWithoutSource(message.getMessage().toString()));
this.messageObject = Objects.requireNonNull(message);
if(message instanceof GroupMessage) {
messageSender = new MiraiMessageSender(((GroupMessage) message).getGroup(), MessageSource.Group);
if(message instanceof GroupMessageEvent) {
messageSender = new MiraiMessageSender(((GroupMessageEvent) message).getGroup(), MessageSource.GROUP);
} else {
messageSender = new MiraiMessageSender(message.getSender(), MessageSource.Private);
messageSender = new MiraiMessageSender(message.getSender(), MessageSource.PRIVATE);
}
}
/**
* 通过解析好的信息构造MessageEvent
* @param messageObject 消息原始对象
* @param groupId 群组Id, 非群聊或无需使用群聊时, 该参数为0
* @param qqId 发送者Id, 不能为0
* @param message 原始消息内容对象, 由构造方法内部解析
*/
private MiraiMessageEvent(MessageEvent messageObject, long groupId, long qqId, MessageChain message) {
super(groupId, qqId, getMessageBodyWithoutSource(message.toString()));
this.messageObject = Objects.requireNonNull(messageObject, "messageObject is null");
if(groupId != 0) {
this.messageSender = new MiraiMessageSender(((GroupMessageEvent) messageObject).getGroup(), MessageSource.GROUP);
} else {
this.messageSender = new MiraiMessageSender(messageObject.getSender(), MessageSource.GROUP);
}
}
/**
* 将Mirai原始MessageEvent转换成应用支持的MessageEvent对象
* @param event 原始消息对象
* @return 原始消息对象所对应的应用MessageEvent对象.
* @throws IllegalArgumentException 当出现不支持的Mirai {@link MessageEvent}实现时将抛出异常.
* @see MessageEvent 原始消息对象
* @see net.lamgc.cgj.bot.event.MessageEvent 应用消息对象
*/
public static MiraiMessageEvent covertEventObject(MessageEvent event) throws IllegalArgumentException {
if(event instanceof GroupMessageEvent) {
return new MiraiMessageEvent(event,
((GroupMessageEvent) event).getGroup().getId(), event.getSender().getId(), event.getMessage());
} else if(event instanceof FriendMessageEvent) {
return new MiraiMessageEvent(event, 0, event.getSender().getId(), event.getMessage());
} else if(event instanceof TempMessageEvent) {
return new MiraiMessageEvent(event, 0, event.getSender().getId(), event.getMessage());
} else {
throw new IllegalArgumentException("Unsupported event type: " + event.toString());
}
}
/**
* 将ContactMessage获得的消息内容删除 Mirai:source 并返回.
* <p>该做法比较保守, 防止Mirai:source位置出现变动.</p>
* @param message ContactMessage的消息内容;
* @return 返回删除了Mirai:source的消息
*/
private static String getMessageBodyWithoutSource(String message) {
StringBuilder builder = new StringBuilder(message);
int startIndex = builder.indexOf("[mirai:source:");
int endIndex = builder.indexOf("]", startIndex) + 1;
return builder.delete(startIndex, endIndex).toString();
}
@Override
public int sendMessage(final String message) {
public int sendMessage(final String message) throws Exception {
return messageSender.sendMessage(message);
}

View File

@ -2,19 +2,16 @@ package net.lamgc.cgj.bot.framework.mirai.message;
import com.google.common.base.Strings;
import net.lamgc.cgj.bot.BotCode;
import net.lamgc.cgj.bot.boot.BotGlobal;
import net.lamgc.cgj.bot.cache.CacheStore;
import net.lamgc.cgj.bot.cache.HotDataCacheStore;
import net.lamgc.cgj.bot.cache.LocalHashCacheStore;
import net.lamgc.cgj.bot.cache.StringRedisCacheStore;
import net.lamgc.cgj.bot.event.BotEventHandler;
import net.lamgc.cgj.bot.message.MessageSender;
import net.lamgc.cgj.bot.message.MessageSource;
import net.mamoe.mirai.Bot;
import net.mamoe.mirai.contact.Contact;
import net.mamoe.mirai.message.data.Image;
import net.mamoe.mirai.message.data.Message;
import net.mamoe.mirai.message.data.MessageChain;
import net.mamoe.mirai.message.data.MessageUtils;
import net.mamoe.mirai.message.data.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -29,11 +26,11 @@ public class MiraiMessageSender implements MessageSender {
private final Contact member;
private final MessageSource source;
private final static Logger log = LoggerFactory.getLogger("MiraiMessageSender");
private final static Logger log = LoggerFactory.getLogger(MiraiMessageSender.class);
private final static CacheStore<String> imageIdCache = new HotDataCacheStore<>(
new StringRedisCacheStore(BotEventHandler.redisServer, "mirai.imageId"),
new StringRedisCacheStore(BotGlobal.getGlobal().getRedisServer(), "mirai.imageId"),
new LocalHashCacheStore<>(),
5400000, 1800000);
5400000, 1800000, true);
/**
* 使用id构造发送器
@ -43,7 +40,7 @@ public class MiraiMessageSender implements MessageSender {
* @throws NoSuchElementException 当在机器人好友列表或群列表里没有这个好友或群的时候抛出
*/
public MiraiMessageSender(Bot bot, MessageSource source, long id) {
this(source == MessageSource.Private ? bot.getFriend(id) : bot.getGroup(id), source);
this(source == MessageSource.PRIVATE ? bot.getFriend(id) : bot.getGroup(id), source);
}
/**
@ -60,7 +57,7 @@ public class MiraiMessageSender implements MessageSender {
public int sendMessage(final String message) {
log.debug("处理前的消息内容:\n{}", message);
Message msgBody = processMessage(Objects.requireNonNull(message));
log.debug("处理后的消息内容(可能出现乱序的情况, 但实际上顺序是没问题的):\n{}", msgBody);
log.debug("处理后的消息内容(可能出现乱序的情况, 但实际上顺序是没问题的):\n{}", msgBody.contentToString());
member.sendMessage(msgBody);
return 0;
}
@ -80,7 +77,7 @@ public class MiraiMessageSender implements MessageSender {
.replaceAll("&38", "&")
.split("\\|");
MessageChain messages = MessageUtils.newChain().plus("");
MessageChain messages = MessageUtils.newChain();
int codeIndex = 0;
for(String text : texts) {
if(text.equals("{BotCode}")) {
@ -117,6 +114,15 @@ public class MiraiMessageSender implements MessageSender {
} else {
return img;
}
case "face":
if(!code.containsParameter("id")) {
return MessageUtils.newChain("(无效的表情Id)");
}
int faceId = Integer.parseInt(code.getParameter("id"));
if(faceId <= 0) {
return MessageUtils.newChain("(无效的表情Id)");
}
return new Face(faceId);
default:
log.warn("解析到不支持的BotCode: {}", code);
return MessageUtils.newChain("(不支持的BotCode)");
@ -128,6 +134,7 @@ public class MiraiMessageSender implements MessageSender {
* @param code 图片BotCode
* @return Image对象
*/
@SuppressWarnings("SynchronizationOnLocalVariableOrMethodParameter")
public Image uploadImage(BotCode code) {
log.debug("传入BotCode信息:\n{}", code);
String absolutePath = code.getParameter("absolutePath");
@ -144,7 +151,7 @@ public class MiraiMessageSender implements MessageSender {
synchronized (imageName) {
if(!imageIdCache.exists(imageName) ||
Strings.nullToEmpty(code.getParameter("updateCache")) .equalsIgnoreCase("true")) {
log.debug("imageName [{}] 缓存失效或强制更新, 正在更新缓存...", imageName);
log.trace("imageName [{}] 缓存失效或强制更新, 正在更新缓存...", imageName);
image = uploadImage0(new File(absolutePath));
String cacheExpireAt;
long expireTime = 864000000; // 10d
@ -156,13 +163,13 @@ public class MiraiMessageSender implements MessageSender {
}
}
imageIdCache.update(imageName, image.getImageId(), expireTime);
log.debug("imageName [{}] 缓存更新完成.(有效时间: {})", imageName, expireTime);
log.trace("imageName [{}] 缓存更新完成.(有效时间: {})", imageName, expireTime);
} else {
log.debug("ImageName: [{}] 缓存命中.", imageName);
log.trace("ImageName: [{}] 缓存命中.", imageName);
}
}
} else {
log.debug("ImageName: [{}] 缓存命中.", imageName);
log.trace("ImageName: [{}] 缓存命中.", imageName);
}
if(image == null) {

View File

@ -5,6 +5,8 @@ import net.lamgc.cgj.bot.message.MessageSenderFactory;
import net.lamgc.cgj.bot.message.MessageSource;
import net.mamoe.mirai.Bot;
import java.util.Objects;
public class MiraiMessageSenderFactory implements MessageSenderFactory {
private final Bot bot;
@ -14,15 +16,11 @@ public class MiraiMessageSenderFactory implements MessageSenderFactory {
}
@Override
public MessageSender createMessageSender(MessageSource source, long id) throws Exception {
switch(source) {
case Group:
case Discuss:
return new MiraiMessageSender(bot.getGroup(id), source);
case Private:
return new MiraiMessageSender(bot.getFriend(id), source);
default:
throw new NoSuchFieldException(source.toString());
public MessageSender createMessageSender(MessageSource source, long id) {
Objects.requireNonNull(source);
if(id <= 0) {
throw new IllegalArgumentException("id cannot be 0 or negative: " + id);
}
return new MiraiMessageSender(bot, source, id);
}
}

View File

@ -9,7 +9,8 @@ public interface MessageSender {
* 返回0则发送器不支持消息Id,
* 返回非0正整数则为消息Id,
* 返回负数则为错误.
* @throws Exception 该方法根据不同实现, 可能会抛出不同异常, 详见实现所标识的文档内容.
*/
int sendMessage(final String message);
int sendMessage(final String message) throws Exception;
}

View File

@ -7,17 +7,17 @@ public enum MessageSource {
/**
* 私聊消息
*/
Private,
PRIVATE,
/**
* 群组消息
*/
Group,
GROUP,
/**
* 讨论组消息
*/
Discuss,
DISCUSS,
/**
* 未知来源
*/
Unknown
UNKNOWN
}

View File

@ -0,0 +1,41 @@
package net.lamgc.cgj.bot.sort;
@SuppressWarnings("unused")
public enum PreLoadDataAttribute {
/**
* 按点赞数排序
*/
LIKE("likeCount"),
/**
* 按页面数排序
*/
PAGE("pageCount"),
/**
* 按收藏数排序
*/
BOOKMARK("bookmarkCount"),
/**
* 按评论数排序
*/
COMMENT("commentCount"),
/**
* 不明
*/
RESPONSE("responseCount"),
/**
* 按查看次数排序
*/
VIEW("viewCount"),
;
public final String attrName;
PreLoadDataAttribute(String attrName) {
this.attrName = attrName;
}
}

View File

@ -0,0 +1,59 @@
package net.lamgc.cgj.bot.sort;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import net.lamgc.cgj.bot.cache.CacheStoreCentral;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Comparator;
/**
* 收藏数比较器
*/
public class PreLoadDataAttributeComparator implements Comparator<JsonElement> {
private final static Logger log = LoggerFactory.getLogger(PreLoadDataAttributeComparator.class);
private final PreLoadDataAttribute attribute;
public PreLoadDataAttributeComparator(PreLoadDataAttribute attribute) {
this.attribute = attribute;
}
@Override
public int compare(JsonElement o1, JsonElement o2) {
if(!o1.isJsonObject() || !o2.isJsonObject()) {
if(o1.isJsonObject()) {
return 1;
} else if(o2.isJsonObject()) {
return -1;
} else {
return 0;
}
}
if(!o1.getAsJsonObject().has("illustId") || !o2.getAsJsonObject().has("illustId")) {
if(o1.getAsJsonObject().has("illustId")) {
return 1;
} else if(o2.getAsJsonObject().has("illustId")) {
return -1;
} else {
return 0;
}
}
try {
JsonObject illustPreLoadData1 = CacheStoreCentral.getCentral()
.getIllustPreLoadData(o1.getAsJsonObject().get("illustId").getAsInt(), false);
JsonObject illustPreLoadData2 = CacheStoreCentral.getCentral()
.getIllustPreLoadData(o2.getAsJsonObject().get("illustId").getAsInt(), false);
return Integer.compare(
illustPreLoadData2.get(attribute.attrName).getAsInt(),
illustPreLoadData1.get(attribute.attrName).getAsInt());
} catch (IOException e) {
log.error("获取预加载数据失败", e);
return 0;
}
}
}

View File

@ -1,82 +0,0 @@
package net.lamgc.cgj.bot.sort;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import net.lamgc.cgj.bot.BotCommandProcess;
import java.io.IOException;
import java.util.Comparator;
/**
* 收藏数比较器
*/
public class PreLoadDataComparator implements Comparator<JsonElement> {
private final Attribute attribute;
public PreLoadDataComparator(Attribute attribute) {
this.attribute = attribute;
}
@Override
public int compare(JsonElement o1, JsonElement o2) {
if(!o1.getAsJsonObject().has("illustId") || !o2.getAsJsonObject().has("illustId")) {
if(o1.getAsJsonObject().has("illustId")) {
return 1;
} else if(o2.getAsJsonObject().has("illustId")) {
return -1;
} else {
return 0;
}
}
try {
JsonObject illustPreLoadData1 = BotCommandProcess.getIllustPreLoadData(o1.getAsJsonObject().get("illustId").getAsInt(), false);
JsonObject illustPreLoadData2 = BotCommandProcess.getIllustPreLoadData(o2.getAsJsonObject().get("illustId").getAsInt(), false);
return Integer.compare(illustPreLoadData2.get(attribute.attrName).getAsInt(), illustPreLoadData1.get(attribute.attrName).getAsInt());
} catch (IOException e) {
e.printStackTrace();
return 0;
}
}
public enum Attribute {
/**
* 按点赞数排序
*/
LIKE("likeCount"),
/**
* 按页面数排序
*/
PAGE("pageCount"),
/**
* 按收藏数排序
*/
BOOKMARK("bookmarkCount"),
/**
* 按评论数排序
*/
COMMENT("commentCount"),
/**
* 不明
*/
RESPONSE("responseCount"),
/**
* 按查看次数排序
*/
VIEW("viewCount"),
;
public final String attrName;
Attribute(String attrName) {
this.attrName = attrName;
}
}
}

View File

@ -0,0 +1,44 @@
package net.lamgc.cgj.exception;
import java.io.IOException;
import java.util.Objects;
import org.apache.http.HttpResponse;
import org.apache.http.StatusLine;
import org.apache.http.util.EntityUtils;
public class HttpRequestException extends IOException {
private static final long serialVersionUID = -2229221075943552798L;
private final StatusLine statusLine;
private final String content;
public HttpRequestException(HttpResponse response) throws IOException {
this(response.getStatusLine(), EntityUtils.toString(response.getEntity()));
}
public HttpRequestException(StatusLine statusLine, String content) {
super("Http Response Error: " + Objects.requireNonNull(statusLine, "statusLine is null") +
", Response Content: " + (content == null ? "null" : '\'' + content + '\''));
this.statusLine = statusLine;
this.content = content;
}
/**
* 获取Http状态行
*/
public StatusLine getStatusLine() {
return statusLine;
}
/**
* 获取Response内容
* @return 如果没有返回, 则返回null
*/
public String getContent() {
return content;
}
}

View File

@ -18,7 +18,7 @@ public interface AdultContentDetector {
* @param pageIndex 指定页数, 设为0或负数则视为单页面作品
* @return 如果为true则为成人作品, 该方法将由检测器决定如何为成人作品.
*/
boolean isAdultContent(int illustId, boolean isUgoira, int pageIndex) throws Exception;
boolean isAdultContent(int illustId, boolean isUgoira, int pageIndex) throws Exception;
/**
* 检查某一作品是否为成人内容
@ -26,8 +26,8 @@ public interface AdultContentDetector {
* @param isUgoira 是否为动图
* @param pageIndex 指定页数, 设为0或负数则视为单页面作品
* @param threshold 指数阀值, 当等于或大于该阀值时返回true.
* @return 如果为true则为成人作品, 该方法将由 threshold 参数决定是否为成人作品.
* @return 如果为true则为成人作品, 该方法将由 threshold 参数决定是否为成人作品(如果超过阈值, 则为true).
*/
boolean isAdultContent(int illustId, boolean isUgoira, int pageIndex, double threshold) throws Exception;
boolean isAdultContent(int illustId, boolean isUgoira, int pageIndex, double threshold) throws Exception;
}

View File

@ -6,6 +6,7 @@ import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.reflect.TypeToken;
import io.netty.handler.codec.http.HttpHeaderNames;
import net.lamgc.cgj.exception.HttpRequestException;
import org.apache.http.Header;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
@ -31,9 +32,10 @@ import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.BiConsumer;
@SuppressWarnings("ALL")
public class PixivDownload {
private final static Logger log = LoggerFactory.getLogger("PixivDownload");
private final static Logger log = LoggerFactory.getLogger(PixivDownload.class);
private final HttpClient httpClient;
@ -115,10 +117,10 @@ public class PixivDownload {
}
}
} while(!document.select(".pager-container>.next").isEmpty());
log.debug("获取完成.");
log.trace("获取完成.");
AtomicInteger count = new AtomicInteger(1);
linkList.forEach(link -> {
log.debug("Next Link [{}]: {}", count.getAndIncrement(), link);
log.trace("Next Link [{}]: {}", count.getAndIncrement(), link);
InputStream imageInputStream = null;
int tryCount = 0;
do {
@ -133,9 +135,9 @@ public class PixivDownload {
} while(imageInputStream == null);
try(InputStream imageInput = new BufferedInputStream(imageInputStream, 256 * 1024)) {
log.debug("调用回调方法...");
log.trace("调用回调方法...");
fn.accept(link, imageInput);
log.debug("回调方法调用完成.");
log.trace("回调方法调用完成.");
} catch (IOException e) {
log.error("图片获取失败", e);
}
@ -219,8 +221,8 @@ public class PixivDownload {
int authorId = rankInfo.get("user_id").getAsInt();
String authorName = rankInfo.get("user_name").getAsString();
String title = rankInfo.get("title").getAsString();
log.debug("当前到第 {}/{} 名(总共 {} 名), IllustID: {}, Author: ({}) {}, Title: {}", rank, rankStart + range - 1, range, illustId, authorId, authorName, title);
log.debug("正在获取PagesLink...");
log.trace("当前到第 {}/{} 名(总共 {} 名), IllustID: {}, Author: ({}) {}, Title: {}", rank, rankStart + range - 1, range, illustId, authorId, authorName, title);
log.trace("正在获取PagesLink...");
List<String> linkList;
try {
linkList = getIllustAllPageDownload(httpClient, this.cookieStore, illustId, quality);
@ -235,14 +237,14 @@ public class PixivDownload {
log.debug("PagesLink 获取完成, 总数: {}", linkList.size());
for (int pageIndex = 0; pageIndex < linkList.size(); pageIndex++) {
String downloadLink = linkList.get(pageIndex);
log.debug("当前Page: {}/{}", pageIndex + 1, linkList.size());
log.trace("当前Page: {}/{}", pageIndex + 1, linkList.size());
try(InputStream imageInputStream = new BufferedInputStream(getImageAsInputStream(HttpClientBuilder.create().build(), downloadLink), 256 * 1024)) {
fn.download(rank, downloadLink, rankInfo.deepCopy(), imageInputStream);
} catch(IOException e) {
log.error("下载插画时发生异常", e);
return;
}
log.debug("完成.");
log.trace("完成.");
}
});
}
@ -265,7 +267,7 @@ public class PixivDownload {
int authorId = rankInfo.get("user_id").getAsInt();
String authorName = rankInfo.get("user_name").getAsString();
String title = rankInfo.get("title").getAsString();
log.debug("Array-当前到第 {}/{} 名(总共 {} 名), IllustID: {}, Author: ({}) {}, Title: {}", rank, rankStart + range, range, illustId, authorId, authorName, title);
log.trace("Array-当前到第 {}/{} 名(总共 {} 名), IllustID: {}, Author: ({}) {}, Title: {}", rank, rankStart + range, range, illustId, authorId, authorName, title);
results.add(rankInfo);
}
log.debug("JsonArray读取完成.");
@ -285,7 +287,8 @@ public class PixivDownload {
}
/**
* 获取排行榜
* 获取排行榜.
* <p>注意: 如果范围实际上没超出, 但返回排行榜不足, 会导致与实际请求的数量不符, 需要检查</p>
* @param contentType 排行榜类型
* @param mode 排行榜模式
* @param time 查询时间
@ -316,20 +319,24 @@ public class PixivDownload {
int count = 0;
Gson gson = new Gson();
ArrayList<JsonObject> results = new ArrayList<>(range);
for (int pageIndex = startPages; pageIndex <= endPages && count < range; pageIndex++) {
boolean canNext = true;
for (int pageIndex = startPages; canNext && pageIndex <= endPages && count < range; pageIndex++) {
HttpGet request = createHttpGetRequest(PixivURL.getRankingLink(contentType, mode, time, pageIndex, true));
log.debug("RequestUri: {}", request.getURI());
log.trace("RequestUri: {}", request.getURI());
HttpResponse response = httpClient.execute(request);
String responseBody = EntityUtils.toString(response.getEntity());
log.debug("ResponseBody: {}", responseBody);
log.trace("ResponseBody: {}", responseBody);
if(response.getStatusLine().getStatusCode() != 200) {
throw new IOException("Http Response Error: '" + response.getStatusLine() + "', ResponseBody: '" + responseBody + '\'');
throw new HttpRequestException(response.getStatusLine(), responseBody);
}
JsonArray resultArray = gson.fromJson(responseBody, JsonObject.class).getAsJsonArray("contents");
JsonObject resultObject = gson.fromJson(responseBody, JsonObject.class);
canNext = resultObject.get("next").getAsJsonPrimitive().isNumber();
JsonArray resultArray = resultObject.getAsJsonArray("contents");
for (int resultIndex = startIndex; resultIndex < resultArray.size() && count < range; resultIndex++, count++) {
results.add(resultArray.get(resultIndex).getAsJsonObject());
}
// 重置索引
startIndex = 0;
}
@ -347,7 +354,7 @@ public class PixivDownload {
HttpResponse response = httpClient.execute(request);
if(response.getStatusLine().getStatusCode() != 200) {
throw new IOException("Http响应码非200: " + response.getStatusLine());
throw new HttpRequestException(response);
}
Document document = Jsoup.parse(EntityUtils.toString(response.getEntity()));
@ -394,8 +401,8 @@ public class PixivDownload {
if(resultObject.get("error").getAsBoolean()) {
String message = resultObject.get("message").getAsString();
log.debug("请求错误, 错误信息: {}", message);
throw new IOException(message);
log.warn("作品页面接口请求错误, 错误信息: {}", message);
throw new HttpRequestException(response);
}
JsonArray linkArray = resultObject.getAsJsonArray("body");
@ -468,9 +475,11 @@ public class PixivDownload {
request.addHeader(HttpHeaderNames.REFERER.toString(), referer);
HttpResponse response = httpClient.execute(request);
log.debug("response: {}", response);
log.debug("Content Length: {}KB", Float.parseFloat(response.getFirstHeader(HttpHeaderNames.CONTENT_LENGTH.toString()).getValue()) / 1024F);
log.debug("{}", response.getFirstHeader(HttpHeaderNames.CONTENT_TYPE.toString()));
log.trace("response: {}", response);
log.trace("Content Length: {}KB",
Float.parseFloat(response.getFirstHeader(HttpHeaderNames.CONTENT_LENGTH.toString()).getValue()) / 1024F
);
log.trace(response.getFirstHeader(HttpHeaderNames.CONTENT_TYPE.toString()).toString());
return response.getEntity().getContent();
}
@ -491,69 +500,31 @@ public class PixivDownload {
} else {
return false;
}
}
/**
* 获取作品信息
* @param illustId 作品ID
* @return 成功获取返回JsonObject, 失败返回null, <br/>
* Json示例: <br/>
* <pre>
* {
* "illustId": "79584670",
* "illustTitle": "このヤンキーはウブすぎる",
* "id": "79584670",
* "title": "このヤンキーはウブすぎる",
* "illustType": 1,
* "xRestrict": 0,
* "restrict": 0,
* "sl": 2,
* "url": "https://i.pximg.net/c/360x360_70/img-master/img/2020/02/19/00/38/23/79584670_p0_square1200.jpg",
* "description": "",
* "tags": [
* "漫画",
* "オリジナル",
* "創作",
* "創作男女",
* "コロさん、ポリさん此方です!",
* "恋の予感",
* "あまずっぺー",
* "交換日記",
* "続編希望!!",
* "オリジナル10000users入り"
* ],
* "userId": "4778293",
* "userName": "隈浪さえ",
* "width": 3288,
* "height": 4564,
* "pageCount": 4,
* "isBookmarkable": true,
* "bookmarkData": null,
* "alt": "#オリジナル このヤンキーはウブすぎる - 隈浪さえ的漫画",
* "isAdContainer": false,
* "profileImageUrl": "https://i.pximg.net/user-profile/img/2019/12/04/18/56/19/16639046_fea29ce38ea89b0cb2313b40b3a72f9a_50.jpg",
* "type": "illust"
* }
* </pre>
* @return 成功获取返回JsonObject, 失败返回null.
* @throws IOException 当请求发生异常, 或接口返回错误信息时抛出.
* @throws NoSuchElementException 当该作品不存在时抛出异常
*/
public JsonObject getIllustInfoByIllustId(int illustId) throws IOException {
public JsonObject getIllustInfoByIllustId(int illustId) throws IOException, NoSuchElementException {
HttpGet request = createHttpGetRequest(PixivURL.getPixivIllustInfoAPI(illustId));
HttpResponse response = httpClient.execute(request);
String responseStr = EntityUtils.toString(response.getEntity());
log.debug("Response Content: {}", responseStr);
log.trace("Response Content: {}", responseStr);
JsonObject responseObj = new Gson().fromJson(responseStr, JsonObject.class);
if(responseObj.get("error").getAsBoolean()) {
throw new IOException(responseObj.get("message").getAsString());
throw new HttpRequestException(response.getStatusLine(), responseStr);
}
JsonArray illustsArray = responseObj.getAsJsonObject("body").getAsJsonArray("illusts");
if(illustsArray.size() == 1) {
return illustsArray.get(0).getAsJsonObject();
} else {
return null;
throw new NoSuchElementException("No work found: " + illustId);
}
}

View File

@ -15,8 +15,8 @@ import java.util.Objects;
* @author LamGC
* @see PixivURL#PIXIV_SEARCH_CONTENT_URL
*/
@SuppressWarnings("ALL")
public class PixivSearchBuilder {
@SuppressWarnings("unused")
public class PixivSearchLinkBuilder {
private final String content;
@ -26,8 +26,8 @@ public class PixivSearchBuilder {
private SearchOrder searchOrder = SearchOrder.DATE_D;
private SearchContentOption searchContentOption = SearchContentOption.ALL;
private HashSet<String> includeKeywords = new HashSet<>(0);
private HashSet<String> excludeKeywords = new HashSet<>(0);
private final HashSet<String> includeKeywords = new HashSet<>(0);
private final HashSet<String> excludeKeywords = new HashSet<>(0);
private int page = 1;
@ -42,7 +42,7 @@ public class PixivSearchBuilder {
private Date startDate = null;
private Date endDate = null;
public PixivSearchBuilder(String searchContent) {
public PixivSearchLinkBuilder(String searchContent) {
this.content = Objects.requireNonNull(searchContent);
}
@ -99,7 +99,7 @@ public class PixivSearchBuilder {
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
PixivSearchBuilder that = (PixivSearchBuilder) o;
PixivSearchLinkBuilder that = (PixivSearchLinkBuilder) o;
return page == that.page &&
wgt == that.wgt &&
hgt == that.hgt &&
@ -141,7 +141,7 @@ public class PixivSearchBuilder {
@Override
public String toString() {
return "PixivSearchBuilder{" +
return "PixivSearchLinkBuilder{" +
"content='" + content + '\'' +
", searchArea=" + searchArea +
", searchMode=" + searchMode +
@ -161,19 +161,11 @@ public class PixivSearchBuilder {
'}';
}
public PixivSearchBuilder setSearchArea(SearchArea searchArea) {
public PixivSearchLinkBuilder setSearchArea(SearchArea searchArea) {
this.searchArea = Objects.requireNonNull(searchArea);
return this;
}
/**
* 获取搜索区域
* @return 返回搜索区域对象
*/
public SearchArea getSearchArea() {
return searchArea;
}
/**
* 获取搜索条件.
* @return 搜索条件内容
@ -202,50 +194,50 @@ public class PixivSearchBuilder {
return searchContent.toString();
}
public PixivSearchBuilder setSearchMode(SearchMode searchMode) {
public PixivSearchLinkBuilder setSearchMode(SearchMode searchMode) {
this.searchMode = Objects.requireNonNull(searchMode);
return this;
}
public PixivSearchBuilder setSearchType(SearchType searchType) {
public PixivSearchLinkBuilder setSearchType(SearchType searchType) {
this.searchType = Objects.requireNonNull(searchType);
return this;
}
public PixivSearchBuilder setSearchOrder(SearchOrder searchOrder) {
public PixivSearchLinkBuilder setSearchOrder(SearchOrder searchOrder) {
this.searchOrder = Objects.requireNonNull(searchOrder);
return this;
}
public PixivSearchBuilder setSearchContentOption(SearchContentOption searchContentOption) {
public PixivSearchLinkBuilder setSearchContentOption(SearchContentOption searchContentOption) {
this.searchContentOption = Objects.requireNonNull(searchContentOption);
return this;
}
public PixivSearchBuilder setRatioOption(RatioOption ratioOption) {
public PixivSearchLinkBuilder setRatioOption(RatioOption ratioOption) {
this.ratioOption = Objects.requireNonNull(ratioOption);
return this;
}
public PixivSearchBuilder setDateRange(Date startDate, Date endDate) {
public PixivSearchLinkBuilder setDateRange(Date startDate, Date endDate) {
this.startDate = startDate;
this.endDate = endDate;
return this;
}
public PixivSearchBuilder setMaxSize(int width, int height) {
public PixivSearchLinkBuilder setMaxSize(int width, int height) {
this.wgt = width;
this.hgt = height;
return this;
}
public PixivSearchBuilder setMinSize(int width, int height) {
public PixivSearchLinkBuilder setMinSize(int width, int height) {
this.wlt = width;
this.hlt = height;
return this;
}
public PixivSearchBuilder setPage(int pageIndex) {
public PixivSearchLinkBuilder setPage(int pageIndex) {
if (pageIndex <= 0) {
throw new IllegalArgumentException("Invalid pageIndex: " + pageIndex);
}
@ -253,22 +245,22 @@ public class PixivSearchBuilder {
return this;
}
public PixivSearchBuilder addExcludeKeyword(String keyword) {
public PixivSearchLinkBuilder addExcludeKeyword(String keyword) {
excludeKeywords.add(keyword);
return this;
}
public PixivSearchBuilder removeExcludeKeyword(String keyword) {
public PixivSearchLinkBuilder removeExcludeKeyword(String keyword) {
excludeKeywords.remove(keyword);
return this;
}
public PixivSearchBuilder addIncludeKeyword(String keyword) {
public PixivSearchLinkBuilder addIncludeKeyword(String keyword) {
includeKeywords.add(keyword);
return this;
}
public PixivSearchBuilder removeIncludeKeyword(String keyword) {
public PixivSearchLinkBuilder removeIncludeKeyword(String keyword) {
includeKeywords.remove(keyword);
return this;
}

View File

@ -10,90 +10,100 @@ import java.util.GregorianCalendar;
* 目前已整理的一些Pixiv接口列表
*/
@SuppressWarnings("unused")
public class PixivURL {
public final class PixivURL {
private PixivURL() {}
public static final String PIXIV_INDEX_URL = "https://www.pixiv.net";
public final static String PIXIV_INDEX_URL = "https://www.pixiv.net";
/**
* P站预登陆url
*/
public static final String PIXIV_LOGIN_PAGE_URL = "https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index";
public final static String PIXIV_LOGIN_PAGE_URL = "https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index";
/**
* P站登录请求url
*/
public static final String PIXIV_LOGIN_URL = "https://accounts.pixiv.net/api/login?lang=zh";
public final static String PIXIV_LOGIN_URL = "https://accounts.pixiv.net/api/login?lang=zh";
/**
* P站搜索请求url
* @deprecated 该接口已被替换, 请使用{@link PixivSearchLinkBuilder}构造搜索Url
* @see PixivSearchLinkBuilder
*/
private static final String PIXIV_SEARCH_URL = "https://www.pixiv.net/search.php";
@Deprecated
private final static String PIXIV_SEARCH_URL = "https://www.pixiv.net/search.php";
/**
* P站搜索用户url
* 需要替换的参数:
* {nick} - 用户昵称、部分名称
* @deprecated 该接口已被替换, 请使用{@link PixivSearchLinkBuilder}构造搜索Url
* @see PixivSearchLinkBuilder
*/
public static final String PIXIV_SEARCH_USER_URL = PIXIV_SEARCH_URL + "?s_mode=s_usr&nick={nick}";
@Deprecated
public final static String PIXIV_SEARCH_USER_URL = PIXIV_SEARCH_URL + "?s_mode=s_usr&nick={nick}";
/**
* P站搜索插画url
* 需要替换的参数:
* {word} - 插画相关文本
* @deprecated 该接口已被替换, 请使用{@link PixivSearchLinkBuilder}构造搜索Url
* @see PixivSearchLinkBuilder
*/
public static final String PIXIV_SEARCH_TAG_URL = PIXIV_SEARCH_URL + "?s_mode=s_tag&word={word}";
@Deprecated
public final static String PIXIV_SEARCH_TAG_URL = PIXIV_SEARCH_URL + "?s_mode=s_tag&word={word}";
/**
* P站插图下载链接获取url
* 需要替换的文本:
* {illustId} - 插画ID
*/
public static final String PIXIV_ILLUST_API_URL = "https://www.pixiv.net/ajax/illust/{illustId}/pages";
public final static String PIXIV_ILLUST_API_URL = "https://www.pixiv.net/ajax/illust/{illustId}/pages";
/**
* P站用户插图列表获取API
* <p>所需数据在 body属性内的 illusts(属性名,属性值不重要), manga(多图) pickup(精选)</p>
* 需要替换的文本:
* {userId} - 用户ID
*/
//TODO: 所需数据在 body属性内的 illusts(属性名,属性值不重要), manga(多图) pickup(精选)
//{"error":false,"message":"","body":{"illusts":{"74369837":null,"70990542":null,"70608653":null,"69755191":null,"69729450":null,"69729416":null,"69503608":null,"69288766":null,"69083882":null,"69051458":null,"68484200":null,"68216927":null,"68216866":null,"68192333":null,"67915106":null,"67914932":null,"67854803":null,"67854745":null,"67854670":null,"67787211":null,"67772199":null,"67770637":null,"67754861":null,"67754804":null,"67754726":null,"67740486":null,"67740480":null,"67740450":null,"67740434":null,"67726337":null,"67499196":null,"67499163":null,"67499145":null,"67499111":null,"67499085":null,"67499038":null,"67498987":null,"67473178":null,"66271465":null,"63682753":null,"63682697":null,"59385148":null,"59383265":null,"59383240":null,"59383227":null,"59383173":null},"manga":[],"novels":[],"mangaSeries":[],"novelSeries":[],"pickup":[],"bookmarkCount":{"public":{"illust":1,"novel":0},"private":{"illust":0,"novel":0}}}}
public static final String PIXIV_USER_ILLUST_LIST_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/all";
public final static String PIXIV_USER_ILLUST_LIST_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/all";
/**
* 能够同时获取插图信息的用户插图列表获取API
* 需要替换的文本:
* {userId} - 用户ID
*/
public static final String PIXIV_USER_TOP_ILLUST_LIST_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/top";
public final static String PIXIV_USER_TOP_ILLUST_LIST_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/top";
/**
* P站单图详情页url
* 需要替换的文本:
* {illustId} - 插画ID
*/
public static final String PIXIV_ILLUST_MEDIUM_URL = "https://www.pixiv.net/member_illust.php?mode=medium&illust_id={illustId}";
public final static String PIXIV_ILLUST_MEDIUM_URL = "https://www.pixiv.net/member_illust.php?mode=medium&illust_id={illustId}";
/**
* P站多图详情页url
* 需要替换的文本:
* {illustId} - 插画ID
*/
public static final String PIXIV_ILLUST_MANGA_URL = "https://www.pixiv.net/member_illust.php?mode=manga&illust_id={illustId}";
public final static String PIXIV_ILLUST_MANGA_URL = "https://www.pixiv.net/member_illust.php?mode=manga&illust_id={illustId}";
/**
* P站用户页面url
* 需要替换的文本:
* {userId} - 用户ID
*/
public static final String PIXIV_USER_URL = "https://www.pixiv.net/member.php?id={userId}";
public final static String PIXIV_USER_URL = "https://www.pixiv.net/member.php?id={userId}";
/**
* P站插图信息获取API
* 这个API能获取插图基本信息但不能获取大小
* 请使用{@link #getPixivIllustInfoAPI(int[])}获取URL
*/
private static final String PIXIV_GET_ILLUST_INFO_URL = "https://www.pixiv.net/ajax/illust/recommend/illusts?";
private final static String PIXIV_GET_ILLUST_INFO_URL = "https://www.pixiv.net/ajax/illust/recommend/illusts?";
/**
* P站获取用户所有插图ID的Api
@ -101,7 +111,15 @@ public class PixivURL {
* 需要替换的文本:
* {userId} - 用户ID
*/
public static final String PIXIV_GET_USER_ALL_ILLUST_ID_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/all";
public final static String PIXIV_GET_USER_ALL_ILLUST_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/all";
/**
* P站获取用户推荐插画及用户基本数据
* 这个API能获得作者的部分(推荐)作品, 每个作品有详细数据, 还能获取作者主页信息(比如主页说明, 看板图)
* 需要替换的文本:
* {userId} - 用户ID
*/
public final static String PIXIV_GET_USER_TOP_ILLUST_URL = "https://www.pixiv.net/ajax/user/{userId}/profile/top";
/**
* P站标签搜索URL
@ -109,7 +127,7 @@ public class PixivURL {
* 需要替换的文本:
* {content} - 大致tag内容
*/
public static final String PIXIV_TAG_SEARCH_URL = "https://www.pixiv.net/ajax/search/tags/{content}";
public final static String PIXIV_TAG_SEARCH_URL = "https://www.pixiv.net/ajax/search/tags/{content}";
/**
* 获取动图下载链接和拼接数据.
@ -119,11 +137,17 @@ public class PixivURL {
*/
public final static String PIXIV_GET_UGOIRA_META_URL = "https://www.pixiv.net/ajax/illust/{illustId}/ugoira_meta";
/**
* 获取自己帐号的部分数据(目前仅能获取: 关注数, 粉丝数和看板图)
* 需要登录.
*/
public final static String PIXIV_GET_USER_EXTRA_URL = "https://www.pixiv.net/ajax/user/extra";
/**
* 请求时带上需要退出的Cookies
* 无论成功与否都会返回302重定向到{@linkplain #PIXIV_LOGIN_PAGE_URL 登录页面}
*/
public static final String PIXIV_LOGOUT_URL = "https://www.pixiv.net/logout.php";
public final static String PIXIV_LOGOUT_URL = "https://www.pixiv.net/logout.php";
/**
* 构造P站获取插图信息的Api Url
@ -164,6 +188,7 @@ public class PixivURL {
/**
* 查询用户收藏.<br/>
* 该URL返回HTML页面需要进行解析.<br/>
* <p>注意: 该接口需要登陆</p>
* 需要替换的文本:<br/>
* {pageIndex} - 页数, 超出了则结果为空<br/>
*/
@ -269,7 +294,7 @@ public class PixivURL {
/**
* Pixiv搜索接口.<br/>
* 要使用该链接请使用{@link PixivSearchBuilder}构造链接.<br/>
* 要使用该链接请使用{@link PixivSearchLinkBuilder}构造链接.<br/>
* 需要替换的参数: <br/>
* content - 搜索内容
*/
@ -336,6 +361,7 @@ public class PixivURL {
* @param mode 要检查的RankingMode项
* @return 如果支持返回true
*/
@SuppressWarnings("BooleanMethodIsAlwaysInverted")
public boolean isSupportedMode(RankingMode mode) {
return Arrays.binarySearch(supportedMode, mode) >= 0;
}

View File

@ -7,11 +7,12 @@ import com.squareup.gifencoder.GifEncoder;
import com.squareup.gifencoder.Image;
import com.squareup.gifencoder.ImageOptions;
import io.netty.handler.codec.http.HttpHeaderNames;
import org.apache.commons.io.IOUtils;
import net.lamgc.cgj.exception.HttpRequestException;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
import org.apache.tomcat.util.http.fileupload.util.Streams;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@ -30,7 +31,7 @@ import java.util.zip.ZipInputStream;
*/
public final class PixivUgoiraBuilder {
private final Logger log = LoggerFactory.getLogger(PixivUgoiraBuilder.class.getSimpleName() + "@" + Integer.toHexString(this.hashCode()));
private final Logger log = LoggerFactory.getLogger(this.toString());
private final HttpClient httpClient;
private final JsonObject ugoiraMeta;
@ -46,12 +47,12 @@ public final class PixivUgoiraBuilder {
log.debug("Request Url: {}", request.getURI());
HttpResponse response = httpClient.execute(request);
String bodyStr = EntityUtils.toString(response.getEntity());
log.debug("JsonBodyStr: {}", bodyStr);
log.trace("JsonBodyStr: {}", bodyStr);
JsonObject resultObject = new Gson().fromJson(bodyStr, JsonObject.class);
if(resultObject.get("error").getAsBoolean()) {
String message = resultObject.get("message").getAsString();
log.error("获取动图元数据失败!(接口报错: {})", message);
throw new IOException(message);
throw new HttpRequestException(response.getStatusLine(), bodyStr);
} else if(!resultObject.has("body")) {
String message = "接口返回数据不存在body属性, 可能接口发生改变!";
log.error(message);
@ -82,7 +83,33 @@ public final class PixivUgoiraBuilder {
log.debug("IllustId: {}, UgoiraMeta: {}", this.illustId, this.ugoiraMeta);
}
/**
* 获取动图元数据
* @return 动图元数据, 返回的对象不影响Builder中的meta对象
*/
public JsonObject getUgoiraMeta() {
return this.ugoiraMeta.deepCopy();
}
/**
* 构建动图
* @param original 是否为原图画质
* @return 返回动图数据输入流
* @throws IOException 当获取数据发生异常时抛出
*/
public InputStream buildUgoira(boolean original) throws IOException {
ByteArrayOutputStream bufferOutput = new ByteArrayOutputStream();
buildUgoira(bufferOutput, original);
return new ByteArrayInputStream(bufferOutput.toByteArray());
}
/**
* 构建动图
* @param outputStream 动图输出流
* @param original 是否为原图画质
* @throws IOException 当获取数据发生异常时抛出
*/
public void buildUgoira(OutputStream outputStream, boolean original) throws IOException {
getUgoiraImageSize();
log.debug("动图尺寸信息: Height: {}, Width: {}", height, width);
@ -95,13 +122,12 @@ public final class PixivUgoiraBuilder {
HttpResponse response = httpClient.execute(request);
log.trace("请求已发送, 正在处理响应...");
ZipInputStream zipInputStream = new ZipInputStream(new BufferedInputStream(response.getEntity().getContent(), 64 * 1024));
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
ZipEntry entry;
ByteArrayOutputStream cacheOutputStream = new ByteArrayOutputStream(512);
HashMap<String, InputStream> frameMap = new HashMap<>(frames.size());
while((entry = zipInputStream.getNextEntry()) != null) {
log.trace("ZipEntry {} 正在接收...", entry);
IOUtils.copy(zipInputStream, cacheOutputStream);
Streams.copy(zipInputStream, cacheOutputStream, false);
frameMap.put(entry.getName(), new ByteArrayInputStream(cacheOutputStream.toByteArray()));
log.trace("ZipEntry {} 已接收完成.", entry);
cacheOutputStream.reset();
@ -140,7 +166,6 @@ public final class PixivUgoiraBuilder {
}
});
encoder.finishEncoding();
return new ByteArrayInputStream(outputStream.toByteArray());
}
/**
@ -149,15 +174,15 @@ public final class PixivUgoiraBuilder {
private void getUgoiraImageSize() throws IOException {
log.debug("正在从Pixiv获取动图尺寸...");
HttpGet request = new HttpGet(PixivURL.getPixivIllustInfoAPI(illustId));
log.debug("Request Url: {}", request.getURI());
log.trace("Request Url: {}", request.getURI());
HttpResponse response = httpClient.execute(request);
String responseBody = EntityUtils.toString(response.getEntity(), StandardCharsets.UTF_8);
log.debug("ResponseBody: {}", responseBody);
log.trace("ResponseBody: {}", responseBody);
JsonObject resultObject = new Gson().fromJson(responseBody, JsonObject.class);
if(resultObject.get("error").getAsBoolean()) {
String message = resultObject.get("message").getAsString();
log.error("接口返回错误: {}", message);
throw new IOException(message);
throw new HttpRequestException(response.getStatusLine(), responseBody);
}
JsonArray illustsArray = resultObject.getAsJsonObject("body").getAsJsonArray("illusts");

View File

@ -1,149 +0,0 @@
package net.lamgc.cgj.proxy;
import com.github.monkeywie.proxyee.intercept.HttpProxyIntercept;
import com.github.monkeywie.proxyee.intercept.HttpProxyInterceptInitializer;
import com.github.monkeywie.proxyee.intercept.HttpProxyInterceptPipeline;
import com.github.monkeywie.proxyee.intercept.common.CertDownIntercept;
import com.github.monkeywie.proxyee.proxy.ProxyConfig;
import com.github.monkeywie.proxyee.server.HttpProxyServer;
import com.github.monkeywie.proxyee.server.HttpProxyServerConfig;
import io.netty.channel.Channel;
import io.netty.handler.codec.http.HttpHeaderNames;
import io.netty.handler.codec.http.HttpHeaders;
import io.netty.handler.codec.http.HttpRequest;
import io.netty.handler.codec.http.HttpResponse;
import org.apache.http.client.CookieStore;
import org.apache.http.impl.cookie.BasicClientCookie;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.net.HttpCookie;
import java.util.Date;
import java.util.List;
/**
* 登录成功后提供CookieStore, 然后由程序自动登录Pixiv
* @author LamGC
*/
public class PixivAccessProxyServer {
private final Logger log = LoggerFactory.getLogger(this.getClass().getSimpleName());
private final HttpProxyServer proxyServer;
private final CookieStore cookieStore;
public PixivAccessProxyServer(CookieStore cookieStore){
this(cookieStore, null);
}
public PixivAccessProxyServer(CookieStore cookieStore, ProxyConfig proxyConfig){
HttpProxyServerConfig config = new HttpProxyServerConfig();
this.cookieStore = cookieStore;
config.setHandleSsl(true);
this.proxyServer = new HttpProxyServer();
this.proxyServer
.serverConfig(config)
.proxyConfig(proxyConfig)
.proxyInterceptInitializer(new HttpProxyInterceptInitializer(){
@Override
public void init(HttpProxyInterceptPipeline pipeline) {
pipeline.addLast(new CertDownIntercept());
pipeline.addLast(new HttpProxyIntercept(){
private boolean match(HttpRequest request){
String host = request.headers().get(HttpHeaderNames.HOST);
return host.equalsIgnoreCase("pixiv.net") || host.contains(".pixiv.net");
}
@Override
public void beforeRequest(Channel clientChannel, HttpRequest httpRequest, HttpProxyInterceptPipeline pipeline) throws Exception {
log.info("URL: " + httpRequest.headers().get(HttpHeaderNames.HOST) + httpRequest.uri());
if(!match(httpRequest)){
super.beforeRequest(clientChannel, httpRequest, pipeline);
return;
}
log.info("正在注入Cookies...");
HttpHeaders requestHeaders = httpRequest.headers();
if(requestHeaders.contains(HttpHeaderNames.COOKIE)){
log.info("原请求存在自带Cookies, 正在清除Cookies...");
log.debug("原Cookies: {}", requestHeaders.getAsString(HttpHeaderNames.COOKIE));
requestHeaders.remove(HttpHeaderNames.COOKIE);
}
StringBuilder cookieBuilder = new StringBuilder();
cookieStore.getCookies().forEach(cookie -> {
if(cookie.isExpired(new Date())){
return;
}
cookieBuilder.append(cookie.getName()).append("=").append(cookie.getValue()).append("; ");
});
log.info("Cookies构造完成, 结果: " + cookieBuilder.toString());
requestHeaders.add(HttpHeaderNames.COOKIE, cookieBuilder.toString());
log.info("Cookies注入完成.");
super.beforeRequest(clientChannel, httpRequest, pipeline);
}
@Override
public void afterResponse(Channel clientChannel, Channel proxyChannel, HttpResponse httpResponse, HttpProxyInterceptPipeline pipeline) throws Exception {
if(!match(pipeline.getHttpRequest())){
super.afterResponse(clientChannel, proxyChannel, httpResponse, pipeline);
return;
}
log.info("正在更新Response Cookie...(Header Name: " + HttpHeaderNames.SET_COOKIE + ")");
List<String> responseCookies = httpResponse.headers().getAll(HttpHeaderNames.SET_COOKIE);
responseCookies.forEach(value -> {
/*if(check(value)){
log.info("黑名单Cookie, 已忽略: " + value);
return;
}*/
log.info("Response Cookie: " + value);
BasicClientCookie cookie = parseRawCookie(value);
cookieStore.addCookie(null);
});
httpResponse.headers().remove(HttpHeaderNames.SET_COOKIE);
super.afterResponse(clientChannel, proxyChannel, httpResponse, pipeline);
}
protected BasicClientCookie parseRawCookie(String rawCookie) {
List<HttpCookie> cookies = HttpCookie.parse(rawCookie);
if (cookies.size() < 1)
return null;
HttpCookie httpCookie = cookies.get(0);
BasicClientCookie cookie = new BasicClientCookie(httpCookie.getName(), httpCookie.getValue());
if (httpCookie.getMaxAge() >= 0) {
Date expiryDate = new Date(System.currentTimeMillis() + httpCookie.getMaxAge() * 1000);
cookie.setExpiryDate(expiryDate);
}
if (httpCookie.getDomain() != null)
cookie.setDomain(httpCookie.getDomain());
if (httpCookie.getPath() != null)
cookie.setPath(httpCookie.getPath());
if (httpCookie.getComment() != null)
cookie.setComment(httpCookie.getComment());
cookie.setSecure(httpCookie.getSecure());
return cookie;
}
});
}
});
}
public void start(int port){
this.proxyServer.start(port);
}
public void close(){
this.proxyServer.close();
}
/**
* 导出CookieStore.
* 注意!该方法导出的CookieStore不适用于ApacheHttpClient, 如需使用则需要进行转换.
* @return CookieStore对象
*/
public CookieStore getCookieStore(){
return this.cookieStore;
}
}

View File

@ -0,0 +1,65 @@
package net.lamgc.cgj.util;
import java.util.concurrent.atomic.AtomicInteger;
public final class Locker<K> {
private final LockerMap<K> fromMap;
private final K key;
private final boolean autoDestroy;
private final AtomicInteger lockCount = new AtomicInteger(0);
/**
* 构造一个锁对象
* @param map 所属LockerMap
* @param key 所属Key
*/
Locker(LockerMap<K> map, K key, boolean autoDestroy) {
this.fromMap = map;
this.key = key;
this.autoDestroy = autoDestroy;
}
/**
* 上锁
*/
public void lock() {
lockCount.incrementAndGet();
}
/**
* 解锁
*/
public void unlock() {
int newValue = lockCount.decrementAndGet();
if(newValue <= 0 && autoDestroy) {
destroy();
}
}
/**
* 获取锁对象所属Key
*/
public K getKey() {
return key;
}
/**
* 销毁锁对象
*/
public void destroy() {
fromMap.destroyLocker(this);
}
@Override
public String toString() {
return "Locker@" + this.hashCode() + "{" +
"fromMap=" + fromMap +
", key=" + key +
'}';
}
}

View File

@ -0,0 +1,31 @@
package net.lamgc.cgj.util;
import java.util.HashMap;
public class LockerMap<K> {
private final HashMap<K, Locker<K>> lockerHashMap = new HashMap<>();
/**
* 创建锁
* @param key Key
* @return 如果Key所属锁存在, 则返回对应锁, 否则返回新锁
*/
public Locker<K> createLocker(K key, boolean autoDestroy) {
if(lockerHashMap.containsKey(key)) {
return lockerHashMap.get(key);
}
Locker<K> newLocker = new Locker<>(this, key, autoDestroy);
lockerHashMap.put(key, newLocker);
return newLocker;
}
/**
* 销毁锁
* @param locker 锁对象
*/
public void destroyLocker(Locker<K> locker) {
lockerHashMap.remove(locker.getKey());
}
}

View File

@ -1,7 +1,6 @@
package net.lamgc.cgj.util;
import net.lamgc.cgj.pixiv.PixivDownload;
import net.lamgc.cgj.pixiv.PixivURL;
import net.lamgc.utils.base.runner.StringParameterParser;
public class PagesQualityParser implements StringParameterParser<PixivDownload.PageQuality> {

View File

@ -0,0 +1,51 @@
package net.lamgc.cgj.util;
import net.lamgc.utils.base.ArgumentsProperties;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public final class PropertiesUtils {
private final static Logger log = LoggerFactory.getLogger(PropertiesUtils.class);
private PropertiesUtils() {}
/**
* 从ArgumentsProperties获取设置项到System Properties
* @param prop ArgumentsProperties对象
* @param key 设置项key
* @param defaultValue 默认值
* @return 如果成功从ArgumentsProperties获得设置项, 返回true, 如未找到(使用了defaultValue或null), 返回false;
*/
public static boolean getSettingToSysProp(ArgumentsProperties prop, String key, String defaultValue) {
if(prop.containsKey(key)) {
log.info("{}: {}", key, prop.getValue(key));
System.setProperty("cgj." + key, prop.getValue(key));
return true;
} else {
if(defaultValue != null) {
System.setProperty("cgj." + key, defaultValue);
}
return false;
}
}
/**
* 将环境变量的值读取并存入System Properties.
* @param envKey 待获取的环境变量Key
* @param sysPropKey 要设置的System Properties Key值
* @param defaultValue 默认值, 可选
* @return 如果设置成功, 返回true, 否则返回false
*/
public static boolean getEnvSettingToSysProp(String envKey, String sysPropKey, String defaultValue) {
String env = System.getenv(envKey);
if(env != null) {
System.setProperty("cgj." + sysPropKey, env);
return true;
} else if(defaultValue != null) {
System.setProperty("cgj." + sysPropKey, defaultValue);
}
return false;
}
}

View File

@ -0,0 +1,192 @@
package net.lamgc.cgj.util;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
/**
* 带有时间限制的线程池.
* 当线程超出了限制时间时, 将会对该线程发出中断.
*/
public class TimeLimitThreadPoolExecutor extends ThreadPoolExecutor {
/**
* 执行时间限制, 单位毫秒.
* 默认30s.
*/
private final AtomicLong executeTimeLimit = new AtomicLong();
/**
* 检查间隔时间.
* 默认100ms.
*/
private final AtomicLong timeoutCheckInterval = new AtomicLong(100);
private final Map<Thread, MonitorInfo> workerThreadMap = new Hashtable<>();
private final Thread timeoutCheckThread = createTimeoutCheckThread();
public TimeLimitThreadPoolExecutor(long executeLimitTime,
int corePoolSize,
int maximumPoolSize,
long keepAliveTime,
TimeUnit unit,
BlockingQueue<Runnable> workQueue) {
super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue);
setInitialTime(executeLimitTime);
timeoutCheckThread.start();
}
public TimeLimitThreadPoolExecutor(long executeLimitTime,
int corePoolSize,
int maximumPoolSize,
long keepAliveTime,
TimeUnit unit,
BlockingQueue<Runnable> workQueue,
ThreadFactory threadFactory) {
super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory);
setInitialTime(executeLimitTime);
timeoutCheckThread.start();
}
public TimeLimitThreadPoolExecutor(long executeLimitTime,
int corePoolSize,
int maximumPoolSize,
long keepAliveTime,
TimeUnit unit,
BlockingQueue<Runnable> workQueue,
RejectedExecutionHandler handler) {
super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, handler);
setInitialTime(executeLimitTime);
timeoutCheckThread.start();
}
public TimeLimitThreadPoolExecutor(long executeLimitTime,
int corePoolSize,
int maximumPoolSize,
long keepAliveTime,
TimeUnit unit,
BlockingQueue<Runnable> workQueue,
ThreadFactory threadFactory,
RejectedExecutionHandler handler) {
super(corePoolSize, maximumPoolSize, keepAliveTime, unit, workQueue, threadFactory, handler);
setInitialTime(executeLimitTime);
timeoutCheckThread.start();
}
private void setInitialTime(long executeLimitTime) {
if(executeLimitTime > 0) {
executeTimeLimit.set(executeLimitTime);
}
}
/**
* 设置执行时间.
* <p>注意: 该修改仅在线程池完全停止后才有效</p>
* @see #isTerminated()
* @param time 新的限制时间(ms)
*/
public void setExecuteTimeLimit(long time) {
if(time <= 0) {
throw new IllegalArgumentException("Time is not allowed to be set to 0 or less");
}
if(this.isTerminated()) {
executeTimeLimit.set(time);
}
}
/**
* 设置超时检查间隔.
* <p>该方法仅会在当前检查后生效.</p>
* @param time 新的检查间隔(ms)
*/
public void setTimeoutCheckInterval(long time) {
if(time <= 0) {
throw new IllegalArgumentException("Time is not allowed to be set to 0 or less");
}
timeoutCheckInterval.set(time);
}
/**
* 获取当前设置的执行时间限制.
* @return 执行时间限制(ms).
*/
public long getExecuteTimeLimit() {
return executeTimeLimit.get();
}
/**
* 获取当前设定的超时检查间隔
* @return 间隔时间(ms).
*/
public long getTimeoutCheckInterval() {
return timeoutCheckInterval.get();
}
private Thread createTimeoutCheckThread() {
Thread checkThread = new Thread(() -> {
if(executeTimeLimit.get() <= 0) {
return;
}
while (true) {
try {
long interval = this.timeoutCheckInterval.get();
//noinspection BusyWait 用于等待超时
Thread.sleep(interval);
// 检查是否存在超时的任务
final long executeTimeLimit = this.executeTimeLimit.get();
workerThreadMap.forEach((thread, info) -> {
long currentTime = info.getTimeRemaining().getAndAdd(interval);
if(currentTime > executeTimeLimit) {
if(!info.getNotifyInterrupted().get() && !thread.isInterrupted()) {
thread.interrupt();
info.getNotifyInterrupted().set(true);
}
}
});
} catch(InterruptedException ignored) {
break;
}
}
});
checkThread.setName("ThreadPool-" + Integer.toHexString(this.hashCode()) +"-TimeoutCheck");
return checkThread;
}
@Override
protected void beforeExecute(Thread t, Runnable r) {
workerThreadMap.put(t, new MonitorInfo());
super.beforeExecute(t, r);
}
@Override
protected void afterExecute(Runnable r, Throwable t) {
workerThreadMap.remove(Thread.currentThread());
super.afterExecute(r, t);
}
@Override
protected void terminated() {
this.timeoutCheckThread.interrupt();
super.terminated();
}
private static class MonitorInfo {
private final AtomicLong timeRemaining = new AtomicLong();
private final AtomicBoolean notifyInterrupted = new AtomicBoolean(false);
public AtomicBoolean getNotifyInterrupted() {
return notifyInterrupted;
}
public AtomicLong getTimeRemaining() {
return timeRemaining;
}
}
}

View File

@ -3,25 +3,39 @@
<properties>
<property name="logStorePath">./logs</property>
<property name="charset">UTF-8</property>
<property name="pattern">[%-d{HH:mm:ss.SSS} %5level][%logger.%method():%-3L][%thread]: %msg%n</property>
<property name="standard_pattern">[%-d{HH:mm:ss.SSS} %5level][%logger.%method():%-3L][%thread]: %msg%n</property>
<property name="mirai_pattern">[%-d{HH:mm:ss.SSS} %5level][%logger]: %msg%n</property>
<property name="logsDir">${sys:cgj.logsPath:-logs}</property>
</properties>
<Appenders>
<Console name="CONSOLE_STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="${pattern}" charset="${charset}"/>
<Console name="STANDARD_STDOUT" target="SYSTEM_OUT">
<PatternLayout charset="${charset}">
<MarkerPatternSelector defaultPattern="${standard_pattern}">
<PatternMatch key="mirai" pattern="${mirai_pattern}" />
</MarkerPatternSelector>
</PatternLayout>
<Filters>
<ThresholdFilter level="WARN" onMatch="DENY" onMismatch="ACCEPT"/>
</Filters>
</Console>
<Console name="CONSOLE_STDERR" target="SYSTEM_ERR">
<PatternLayout pattern="${pattern}" charset="${charset}"/>
<Console name="STANDARD_STDERR" target="SYSTEM_ERR">
<PatternLayout charset="${charset}">
<MarkerPatternSelector defaultPattern="${standard_pattern}">
<PatternMatch key="mirai" pattern="${mirai_pattern}" />
</MarkerPatternSelector>
</PatternLayout>
<Filters>
<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>
</Filters>
</Console>
<RollingFile name="rollingFile" fileName="logs/latest.log" filePattern="logs/running.%-d{yyyy-MM-dd_HH-mm-ss}.log.gz">
<PatternLayout pattern="${pattern}" charset="${charset}"/>
<RollingFile name="rollingFile" fileName="${logsDir}/latest.log" filePattern="${logsDir}/running.%-d{yyyy-MM-dd_HH-mm-ss}.log.gz">
<PatternLayout charset="${charset}">
<MarkerPatternSelector defaultPattern="${standard_pattern}">
<PatternMatch key="mirai" pattern="${mirai_pattern}" />
</MarkerPatternSelector>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
</Policies>
@ -31,8 +45,8 @@
<Loggers>
<Logger level="INFO" name="org.apache.http"/>
<Root level="TRACE">
<AppenderRef ref="CONSOLE_STDOUT"/>
<AppenderRef ref="CONSOLE_STDERR"/>
<AppenderRef ref="STANDARD_STDOUT"/>
<AppenderRef ref="STANDARD_STDERR"/>
<AppenderRef ref="rollingFile"/>
</Root>
</Loggers>

View File

@ -1,27 +1,47 @@
<?xml version="1.0" encoding="UTF-8"?>
<configuration status="WARN">
<!--
测试版跟发布版在日志配置文件上的区别仅仅只有'Loggers'的不同, 'properties'和'Appenders'是一致的.
-->
<properties>
<property name="logStorePath">./logs</property>
<property name="charset">UTF-8</property>
<property name="pattern">[%-d{HH:mm:ss.SSS} %5level][%logger.%method():%-3L][%thread]: %msg%n</property>
<property name="standard_pattern">[%-d{HH:mm:ss.SSS} %5level][%logger.%method():%-3L][%thread]: %msg%n</property>
<property name="mirai_pattern">[%-d{HH:mm:ss.SSS} %5level][%logger]: %msg%n</property>
<property name="logsDir">${sys:cgj.logsPath:-logs}</property>
</properties>
<Appenders>
<Console name="CONSOLE_STDOUT" target="SYSTEM_OUT">
<PatternLayout pattern="${pattern}" charset="${charset}"/>
<Console name="STANDARD_STDOUT" target="SYSTEM_OUT">
<PatternLayout charset="${charset}">
<MarkerPatternSelector defaultPattern="${standard_pattern}">
<PatternMatch key="mirai" pattern="${mirai_pattern}" />
</MarkerPatternSelector>
</PatternLayout>
<Filters>
<LevelRangeFilter minLevel="INFO" maxLevel="INFO" />
<LevelRangeFilter minLevel="INFO" maxLevel="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
</Filters>
</Console>
<Console name="CONSOLE_STDERR" target="SYSTEM_ERR">
<PatternLayout pattern="${pattern}" charset="${charset}"/>
<Console name="STANDARD_STDERR" target="SYSTEM_ERR">
<PatternLayout charset="${charset}">
<MarkerPatternSelector defaultPattern="${standard_pattern}">
<PatternMatch key="mirai" pattern="${mirai_pattern}" />
</MarkerPatternSelector>
</PatternLayout>
<Filters>
<ThresholdFilter level="WARN" onMatch="ACCEPT" onMismatch="DENY"/>
</Filters>
</Console>
<RollingFile name="rollingFile" fileName="logs/latest.log" filePattern="logs/running.%-d{yyyy-MM-dd_HH-mm-ss}.log.gz">
<PatternLayout pattern="${pattern}" charset="${charset}"/>
<RollingFile name="rollingFile" fileName="${logsDir}/latest.log" filePattern="${logsDir}/running.%-d{yyyy-MM-dd_HH-mm-ss}.log.gz">
<Filters>
<ThresholdFilter level="DEBUG" onMatch="ACCEPT" onMismatch="DENY"/>
</Filters>
<PatternLayout charset="${charset}">
<MarkerPatternSelector defaultPattern="${standard_pattern}">
<PatternMatch key="mirai" pattern="${mirai_pattern}" />
</MarkerPatternSelector>
</PatternLayout>
<Policies>
<OnStartupTriggeringPolicy />
</Policies>
@ -30,9 +50,9 @@
<Loggers>
<Logger level="INFO" name="org.apache.http"/>
<Root level="TRACE">
<AppenderRef ref="CONSOLE_STDOUT"/>
<AppenderRef ref="CONSOLE_STDERR"/>
<Root level="DEBUG">
<AppenderRef ref="STANDARD_STDOUT"/>
<AppenderRef ref="STANDARD_STDERR"/>
<AppenderRef ref="rollingFile"/>
</Root>
</Loggers>

View File

@ -1,9 +0,0 @@
#用于访问Pixiv的代理服务器
reptile.proxy.type=socks5/socks4/http
reptile.proxy.host=127.0.0.1
reptile.proxy.port=1080
reptile.proxy.username=
reptile.proxy.password=
#登录用代理, 需要让浏览器使用该代理, 访问Pixiv并登录
login.proxy.host=127.0.0.1
login.proxy.port=1080

View File

@ -2,7 +2,6 @@ package net.lamgc.cgj.pixiv;
import com.google.gson.Gson;
import com.google.gson.JsonObject;
import org.apache.commons.io.IOUtils;
import org.apache.http.HttpHost;
import org.apache.http.client.CookieStore;
import org.apache.http.client.methods.CloseableHttpResponse;
@ -10,6 +9,7 @@ import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.util.EntityUtils;
import org.apache.tomcat.util.http.fileupload.util.Streams;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.Ignore;
@ -31,9 +31,9 @@ public class PixivDownloadTest {
private static CookieStore cookieStore;
private final static Logger log = LoggerFactory.getLogger("PixivDownloadTest");
private final static Logger log = LoggerFactory.getLogger(PixivDownloadTest.class);
private static HttpHost proxy = new HttpHost("127.0.0.1", 1001);
private static final HttpHost proxy = new HttpHost("127.0.0.1", 1001);
@BeforeClass
public static void before() throws IOException, ClassNotFoundException {
@ -64,7 +64,7 @@ public class PixivDownloadTest {
ZipEntry entry = new ZipEntry(link.substring(link.lastIndexOf("/") + 1));
log.info("正在写入: " + entry.getName());
zos.putNextEntry(entry);
IOUtils.copy(inputStream, zos);
Streams.copy(inputStream, zos, false);
zos.flush();
} catch (IOException e) {
log.error("写入文件项时发生异常", e);
@ -96,7 +96,7 @@ public class PixivDownloadTest {
ZipEntry entry = new ZipEntry(link.substring(link.lastIndexOf("/") + 1));
log.info("正在写入: " + entry.getName());
zos.putNextEntry(entry);
IOUtils.copy(inputStream, zos);
Streams.copy(inputStream, zos, false);
zos.flush();
log.info("已成功写入 {}", entry.getName());
} catch (IOException e) {
@ -139,7 +139,7 @@ public class PixivDownloadTest {
entry.setComment(rankInfo.toString());
log.info("正在写入: " + entry.getName());
zos.putNextEntry(entry);
IOUtils.copy(inputStream, zos);
Streams.copy(inputStream, zos, false);
zos.flush();
log.info("已成功写入 {}", entry.getName());
} catch (IOException e) {
@ -180,9 +180,8 @@ public class PixivDownloadTest {
log.info("正在调用方法...");
try {
pixivDownload.getRankingAsInputStream(null, null, queryDate, 5, 50, PixivDownload.PageQuality.ORIGINAL, (rank, link, rankInfo, inputStream) -> {
log.info("空操作");
});
pixivDownload.getRankingAsInputStream(null, null, queryDate, 5, 50,
PixivDownload.PageQuality.ORIGINAL, (rank, link, rankInfo, inputStream) -> log.info("空操作"));
} finally {
zos.finish();
zos.flush();

View File

@ -1,26 +0,0 @@
package net.lamgc.cgj.pixiv;
import org.junit.Assert;
import org.junit.Test;
public class PixivSearchBuilderTest {
@Test
public void buildTest() {
PixivSearchBuilder builder = new PixivSearchBuilder("hololive");
//builder.addExcludeKeyword("fubuki").addExcludeKeyword("minato");
builder.addIncludeKeyword("35").addIncludeKeyword("okayu").addIncludeKeyword("百鬼あやめ");
System.out.println(builder.buildURL());
}
@Test
public void equalsTest() {
Assert.assertEquals(new PixivSearchBuilder("风景"), new PixivSearchBuilder("风景"));
}
@Test
public void hashCodeTest() {
Assert.assertEquals(new PixivSearchBuilder("风景").hashCode(), new PixivSearchBuilder("风景").hashCode());
}
}

View File

@ -0,0 +1,25 @@
package net.lamgc.cgj.pixiv;
import org.junit.Assert;
import org.junit.Test;
public class PixivSearchLinkBuilderTest {
@Test
public void buildTest() {
PixivSearchLinkBuilder builder = new PixivSearchLinkBuilder("hololive");
builder.addIncludeKeyword("35").addIncludeKeyword("okayu").addIncludeKeyword("百鬼あやめ");
System.out.println(builder.buildURL());
}
@Test
public void equalsTest() {
Assert.assertEquals(new PixivSearchLinkBuilder("风景"), new PixivSearchLinkBuilder("风景"));
}
@Test
public void hashCodeTest() {
Assert.assertEquals(new PixivSearchLinkBuilder("风景").hashCode(), new PixivSearchLinkBuilder("风景").hashCode());
}
}

View File

@ -1,11 +1,12 @@
package net.lamgc.cgj.pixiv;
import org.apache.http.HttpHost;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.client.HttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.bouncycastle.util.io.Streams;
import org.junit.Ignore;
import org.junit.Test;
import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.IOException;
@ -18,8 +19,9 @@ public class PixivUgoiraBuilderTest {
@Test
public void buildTest() throws IOException {
File outputFile = new File("./output2.gif");
CloseableHttpClient httpClient = HttpClientBuilder.create().setProxy(new HttpHost("127.0.0.1", 1001)).build();
PixivUgoiraBuilder builder = new PixivUgoiraBuilder(httpClient, 80766493);
HttpClient httpClient = HttpClientBuilder.create().setProxy(new HttpHost("127.0.0.1", 1080)).build();
PixivUgoiraBuilder builder = new PixivUgoiraBuilder(httpClient, 81163967);
LoggerFactory.getLogger(PixivUgoiraBuilderTest.class).info("UgoiraMeta: {}", builder.getUgoiraMeta());
InputStream inputStream = builder.buildUgoira(true);
Files.write(outputFile.toPath(), Streams.readAll(inputStream));
}

View File

@ -0,0 +1,18 @@
package net.lamgc.cgj.util;
import org.junit.Assert;
import org.junit.Test;
public class LockerMapTest {
@Test
public void createAndFinalizeTest() {
LockerMap<String> map = new LockerMap<>();
Locker<String> locker = map.createLocker("Test", true);
Assert.assertEquals(locker, map.createLocker("Test", true));
locker.lock();
locker.unlock();
System.gc();
}
}

View File

@ -0,0 +1,41 @@
package net.lamgc.cgj.util;
import org.junit.Assert;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.TimeUnit;
public class TimeLimitThreadPoolExecutorTest {
private final static Logger log = LoggerFactory.getLogger(TimeLimitThreadPoolExecutorTest.class);
@Test
public void timeoutTest() throws InterruptedException {
TimeLimitThreadPoolExecutor executor = new TimeLimitThreadPoolExecutor(1000, 1, 1, 30, TimeUnit.SECONDS, new ArrayBlockingQueue<>(50));
log.info("ThreadPoolExecutor.isTerminated: {}", executor.isTerminated());
log.info("ThreadPoolExecutor.isShutdown: {}", executor.isShutdown());
executor.setTimeoutCheckInterval(150);
log.info("当前设定: ExecuteTimeLimit: {}ms, CheckInterval: {}ms", executor.getExecuteTimeLimit(),
executor.getTimeoutCheckInterval());
executor.execute(() -> {
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("线程 " + Thread.currentThread().getName() + " 被中断");
}
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
Assert.fail("Multiple interrupts occurred");
}
});
executor.shutdown();
Assert.assertTrue(executor.awaitTermination(5 * 1000, TimeUnit.MILLISECONDS));
}
}