diff --git a/research/TrackingRecoveryCandidates.md b/research/TrackingRecoveryCandidates.md index e0dd8d5..e7626cb 100644 --- a/research/TrackingRecoveryCandidates.md +++ b/research/TrackingRecoveryCandidates.md @@ -1,17 +1,40 @@ -# 转身与短时遮挡候选(开发中) +# 转身与短时遮挡:成熟跟踪器比较 -2026-11-06,接续正式 1.18.1(106)。用户要求稳定马赛克、可根据丢失前后运动补齐轨迹,并要求比较 GitHub 成熟算法。正式 APP 尚未改动。 +2026-11-06,接续正式 1.18.1(106)。1.19.0(107)源代码已接入,正式安装与宿主验收待完成。用户素材和逐帧位置只留在本机缓存,不上传仓库。 -现有 OpenCV LK 负责局部光流与仿射拟合,没有完整对象外观模型;转身时背景特征可能仍给出高置信度。不能仅沿错误光流继续外推。 +原 OpenCV LK 只有局部光流与仿射拟合,没有完整对象外观模型;转身时背景特征仍可能给出高内点率。仅沿此结果继续外推会延续偏移。 -已建立独立、无界面的本地基准:640×360,25fps,285 帧原实拍;另在第 130—139 帧人为遮住头部。素材和逐帧位置不上传仓库。 +## 同素材实测 -- Apple Vision accurate 对照:转身全段跟踪完成;原始连续更新版本在遮挡中会学习遮挡物。候选在失去可信观测时冻结最后可信图像,只从冻结图像试探重新定位。当前候选全段完成,短遮挡测试预测 11 帧后重新确认,热运行约 1ms/帧。此为局部基准,未接入正式宿主。 -- 候选运动保护:预测最多 0.8 秒,重新定位需连续两帧确认;限制跳跃和尺寸突变。计划在重新定位后用两端可信位置补齐中间键,尾部未确认预测不得作为成功轨迹提交。此部分尚未接入提交路径。 -- [OpenCV CSRT](https://github.com/opencv/opencv_contrib/tree/4.x/modules/tracking):完整相关滤波对象追踪,测试官方 OpenCV 4.11 wheel,后续比较同片段。 -- [OpenCV Zoo VitTrack](https://github.com/opencv/opencv_zoo/tree/main/models/object_tracking_vittrack):轻量 Transformer 单目标跟踪,有追踪分数,代码/模型目录 Apache-2.0;官方模型 714726 字节,SHA-256 `2990f0b7cd44d92afa48cd97db6de7be113fc1d9594fddb74e2725c10478e91d`,已核对 LFS 指针与完整下载。后续比较同片段。 -- [NanoTrack](https://github.com/HonglinChu/NanoTrack):提供原生 Mac/ncnn 示例;OpenCV Zoo 的对比说明指出其分数不适合判断丢失,暂列次选,不用 README 的速度当本机实测。 -- [OSTrack](https://github.com/botaoye/OSTrack):完整 Transformer 方案,原示例以 PyTorch/CUDA 环境为主;不把研究榜单直接当作 Mac 插件速度。 -- [ByteTrack](https://github.com/FoundationVision/ByteTrack):关联检测框与多目标身份,需要额外检测器,单独引入不能解决背头没有人脸框的问题。 +Mac mini M4,640×360 BGRA、25fps、285 帧同一实拍。开源候选使用官方 OpenCV 4.11.0.86 wheel / cv2 4.11、2 个线程;计时包含初始化和逐帧跟踪,不含视频解码、宿主供图、预览和渲染。此为一段素材的比较,不代表通用排名。合成遮挡只用于可重复回归,不能替代真实复杂遮挡。 -在同素材准确度、短遮挡恢复、背景误锁、计算开销和宿主交互验证通过前,不替换正式版。 +| 候选 | 原片总计算时间 | 转身表现 | 10 帧人工遮挡 | +| --- | ---: | --- | --- | +| [CSRT](https://github.com/opencv/opencv_contrib/tree/4.x/modules/tracking) | 1622.38 ms | 框向额头缩小,下巴覆盖不足 | 返回成功但最终跟到衣服 | +| [VitTrack](https://github.com/opencv/opencv_zoo/tree/main/models/object_tracking_vittrack) | 779.83 ms | 框位置、大小仍偏移 | 能报丢失及恢复,但恢复后仍有上述偏移 | +| Apple Vision + 本次恢复保护 | 398.43 ms | 285 帧完整,视觉检查框保持在头部 | 325.51 ms,11 帧暂时预测后重新确认 | + +[Apple Vision VNTrackingRequest](https://developer.apple.com/documentation/vision/vntrackingrequest) 使用 accurate 模式。原始连续更新版本会学习遮挡物;本次在丢失时冻结最后可信图像,从该图像重建请求试探恢复,需连续两次可信观测。没有声称 Vision 内部一定采用 Metal;具体设备调度由系统框架决定。 + +- 长遮挡(55 帧)及突切随机画面:尝试到第 151 帧,最多产生 20 个暂时预测,第 21 个缺失停止。宿主提交前删除全部未确认尾部,不把预测当作成功跟踪。 +- 反向跟踪:285 帧完整、无预测,319.08 ms。双向回归还由原生宿主回调检查覆盖。 +- 重新定位后以丢失前后可信端点补齐短段,中心线性插值、尺寸对数插值;补齐键置信度标为 0.35,状态提示复核。保留精确帧时间、手工键和撤销流程。 +- 预测最多 0.8 秒;不处理长期离开画面、相似人物交叉后的全局身份确认。马赛克涉及隐私时仍必须播放复核遮盖范围。 + +## 选择与集成边界 + +先采用实测更稳的系统对象跟踪,应用于被局部马赛克选中的矩形/椭圆蒙版。保留作者宽高比和角度,采用保守统一缩放;旋转蒙版先转换到真实外接矩形。普通调色蒙版、钢笔和图章保留现有 OpenCV 多边形光流/仿射路径,避免用矩形对象框改变其精确形状行为。切换后需重新运行跟踪,已有工程轨迹不会静默改写。 + +计算在宿主分析回调,界面预览最多 15Hz,只保留最新待显示帧;不逐帧重建参数面板。需正式宿主验证回调线程和实际操作响应,不能用算法耗时替代 UI 延迟。 + +## 其他 GitHub 候选 + +- [NanoTrack](https://github.com/HonglinChu/NanoTrack):Apache-2.0,提供 Mac / ncnn 示例;未本机实测,不能使用 README FPS 作为本机结论。 +- [OSTrack](https://github.com/botaoye/OSTrack):MIT,研究工程主要面向 PyTorch / CUDA;未本机实测,未来可评估 Core ML 转换后的速度与恢复能力。 +- [ByteTrack](https://github.com/FoundationVision/ByteTrack):MIT,多目标检测框关联;还需要检测器,单独引入不能解决转身后没有人脸检测框的问题。 + +VitTrack 模型来自官方 OpenCV Zoo,714726 字节,SHA-256 `2990f0b7cd44d92afa48cd97db6de7be113fc1d9594fddb74e2725c10478e91d`;目录声明 Apache-2.0。仅在缓存中用于比较,未加入 APP 或 Git。现有 APP OpenCV 4.5.1 依赖不因基准 wheel 变化而升级。 + +## 复测入口 + +`tools/tracking/benchmark_object.mm`、`benchmark_vision.mm`、`benchmark_opensource.py`。编译原生候选:`xcrun clang++ -std=c++17 -O3 -fno-objc-arc -framework Foundation -framework Vision -framework CoreVideo -framework CoreGraphics tools/tracking/benchmark_object.mm -o <缓存>/benchmark-object`。输入使用本地 640×360 BGRA 连续帧;不将用户素材纳入仓库。聚合数据见 `verification/ObjectTrackingBenchmarks1.19.0.json`。 diff --git a/verification/ObjectTrackingBenchmarks1.19.0.json b/verification/ObjectTrackingBenchmarks1.19.0.json new file mode 100644 index 0000000..0e0ff6d --- /dev/null +++ b/verification/ObjectTrackingBenchmarks1.19.0.json @@ -0,0 +1,92 @@ +{ + "date": "2026-11-06", + "productCandidate": "1.19.0 (107)", + "input": { + "size": [ + 640, + 360 + ], + "fps": 25, + "frames": 285, + "media": "local-only; not uploaded" + }, + "scope": "Algorithm-only CLI, includes initialization, excludes decoding, host/UI/rendering; one real clip and synthetic occlusions; not a universal accuracy ranking", + "results": { + "object": { + "maximumMs": 143.29075, + "milliseconds": 398.43138, + "attemptedFrames": 285, + "predictedFrames": 0, + "invalidFrames": 0 + }, + "csrt": { + "algorithm": "csrt", + "opencv": "4.11.0", + "threads": 2, + "milliseconds": 1622.3796229999848, + "attemptedFrames": 285, + "predictedFrames": 0, + "invalidFrames": 0 + }, + "vit": { + "algorithm": "vit", + "opencv": "4.11.0", + "threads": 2, + "milliseconds": 779.8291599999997, + "attemptedFrames": 285, + "predictedFrames": 0, + "invalidFrames": 0 + }, + "object-occlusion": { + "maximumMs": 53.167667, + "milliseconds": 325.51462799999996, + "attemptedFrames": 285, + "predictedFrames": 11, + "invalidFrames": 0 + }, + "csrt-occlusion": { + "algorithm": "csrt", + "opencv": "4.11.0", + "threads": 2, + "milliseconds": 1669.4421300000015, + "attemptedFrames": 285, + "predictedFrames": 0, + "invalidFrames": 0 + }, + "vit-occlusion": { + "algorithm": "vit", + "opencv": "4.11.0", + "threads": 2, + "milliseconds": 783.9419580000007, + "attemptedFrames": 285, + "predictedFrames": 0, + "invalidFrames": 10 + }, + "object-long": { + "maximumMs": 95.583792, + "milliseconds": 263.44187500000004, + "attemptedFrames": 151, + "predictedFrames": 20, + "invalidFrames": 1 + }, + "object-cut": { + "maximumMs": 56.445167, + "milliseconds": 225.25990700000003, + "attemptedFrames": 151, + "predictedFrames": 20, + "invalidFrames": 1 + }, + "object-reverse": { + "maximumMs": 50.584, + "milliseconds": 319.07913499999984, + "attemptedFrames": 285, + "predictedFrames": 0, + "invalidFrames": 0 + } + }, + "visualConclusion": { + "csrt": "turn shrinks toward forehead; after artificial occlusion locks clothing", + "vit": "position and scale drift despite loss score", + "object": "turn stays on head; short gap reacquired, long gap/cut stop" + } +} diff --git a/大鱼调色/Plugin/DYPlugIn.mm b/大鱼调色/Plugin/DYPlugIn.mm index 2f7c5ef..4aa7755 100644 --- a/大鱼调色/Plugin/DYPlugIn.mm +++ b/大鱼调色/Plugin/DYPlugIn.mm @@ -14,6 +14,7 @@ #import #include "MaskCore/DYTracker.hpp" #include "MaskCore/DYOpenCVTracker.hpp" +#include "MaskCore/DYObjectTracker.hpp" #include #include "DYViewerParameterBridge.inc.h" #include "DYToolHostDeclarations.inc.h" diff --git a/大鱼调色/Plugin/DYToolHostIvars.inc.h b/大鱼调色/Plugin/DYToolHostIvars.inc.h index 0903df5..e21740d 100644 --- a/大鱼调色/Plugin/DYToolHostIvars.inc.h +++ b/大鱼调色/Plugin/DYToolHostIvars.inc.h @@ -23,4 +23,7 @@ DYStampMotion _trackingStampMotion; dy::TrackerROI _trackingStampSourceSeed; dy::OpenCVTracker _toolTracker,_stampSourceTracker; + dy::ObjectTracker _objectTracker; + BOOL _trackingObjectMode; + NSUInteger _trackingObjectPending,_trackingObjectRepaired; NSMutableDictionary *_toolCaptureDevices; diff --git a/大鱼调色/Plugin/DYToolHostMethods.inc.h b/大鱼调色/Plugin/DYToolHostMethods.inc.h index e647504..0c19bf2 100644 --- a/大鱼调色/Plugin/DYToolHostMethods.inc.h +++ b/大鱼调色/Plugin/DYToolHostMethods.inc.h @@ -35,6 +35,7 @@ NSString *direction=[frame[@"backward"]boolValue]?@"反向":@"正向"; NSString *message=total?[NSString stringWithFormat:@"%@ · %lu / %lu 帧 · %.0f%%",direction,(unsigned long)done,(unsigned long)total,100.*done/total] :[NSString stringWithFormat:@"%@ · 已分析 %lu 帧",direction,(unsigned long)done]; + if([frame[@"predicted"]boolValue])message=[message stringByAppendingString:@" · 暂时丢失,预测并寻找目标"]; [self postTrackingPreview:frame active:YES message:message]; } - (void)queueTrackingPreview:(NSDictionary*)frame { @@ -46,7 +47,7 @@ } // Keep one coherent packet pending. A slow UI may skip intermediate updates // without ever pairing geometry from one frame with pixels from another. - dispatch_after(dispatch_time(DISPATCH_TIME_NOW,NSEC_PER_SEC/30),dispatch_get_main_queue(),^{ + dispatch_after(dispatch_time(DISPATCH_TIME_NOW,NSEC_PER_SEC/15),dispatch_get_main_queue(),^{ if(self->_trackingGeneration==generation)[self publishTrackingPreview]; }); } @@ -163,6 +164,7 @@ [_trackingNode release];_trackingNode=[node copy];[_trackingMaskID release];_trackingMaskID=[notification.userInfo[@"mask"]copy];[_trackingFrozenMask release];_trackingFrozenMask=[mask copy];[_trackingCandidate release];_trackingCandidate=[DYMutablePlist(mask)retain];[_trackingSamples release];_trackingSamples=[NSMutableArray new];[_trackingMessage release];_trackingMessage=nil; NSArray *source=DYStampSourceAtTime(mask,time); _trackingStampMotion=DYStampMotionAtTime(mask,time); + _trackingObjectMode=DYMosaicTracksMask(doc,mask[@"id"]);_trackingObjectPending=_trackingObjectRepaired=0; if(source){ _trackingStampMotion.active=true; _trackingStampSourceSeed={[source[0] floatValue],[source[1] floatValue], @@ -204,6 +206,7 @@ } - (BOOL)setupAnalysisForTimeRange:(CMTimeRange)range frameDuration:(CMTime)duration error:(NSError**)error { if(!CMTIME_IS_NUMERIC(duration)||duration.value<=0||duration.epoch!=0)return [self stopToolAnalysis:@"宿主分析帧长无效" error:error];_trackingFrameDuration=duration;_trackingPreviousTime=kCMTimeInvalid;_trackingFlip=NO;[_trackingSamples removeAllObjects]; + _objectTracker.stop();_trackingObjectPending=0; if(!_trackingSingle&&CMTIMERANGE_IS_VALID(_trackingPreviewInputRange)){ double step=CMTimeGetSeconds(duration); double forward=CMTimeGetSeconds(CMTimeSubtract(CMTimeRangeGetEnd(_trackingPreviewInputRange),_trackingReferenceTime))/step; @@ -236,13 +239,32 @@ if(_trackingCancelled.load())return [self finishToolAnalysis:@"已取消;保留最后连续可信轨迹" error:error];if(_trackingSingle&&_trackingSamples.count>=2)return [self finishToolAnalysis:nil error:error];if(_trackingSamples.count>=DYToolKeyLimit)return [self finishToolAnalysis:@"轨迹已到数量上限,保留已完成部分" error:error]; CMTime expected=CMTIME_IS_NUMERIC(_trackingPreviousTime)?(_trackingBackward?CMTimeSubtract(_trackingPreviousTime,_trackingFrameDuration):CMTimeAdd(_trackingPreviousTime,_trackingFrameDuration)):_trackingReferenceTime;if(!CMTIME_IS_NUMERIC(time)||time.epoch!=expected.epoch||CMTimeCompare(time,expected)!=0)return [self stopToolAnalysis:@"宿主帧序列不连续,停在最后可信帧" error:error]; NSDictionary *preview=nil; - dy::GrayFrame frame=[self toolGrayFrame:tile preview:_trackingSingle?nullptr:&preview error:error]; + dy::GrayFrame frame=[self toolGrayFrame:tile preview:(_trackingSingle&&!_trackingObjectMode)?nullptr:&preview error:error]; if(!frame.valid())return [self stopToolAnalysis:@"宿主未返回完整源画面" error:error]; if(!_trackingSamples.count){if(_trackingReferenceFrame.valid()&&frame.width==_trackingReferenceFrame.width&&frame.height==_trackingReferenceFrame.height){double same=0,flip=0;size_t count=0;for(int y=0;y640||height>640||rgba.length!=size_t(width)*height*4)return [self finishToolAnalysis:@"目标跟踪未取得完整分析画面,原轨迹已保留" error:error]; + if(!_trackingSamples.count){ + // Vision consumes an axis-aligned object rectangle. Track the true bounds + // of a rotated authored shape, then apply only its uniform scale/position. + float aspect=float(width)/height,ca=std::cos(_trackingSeed.angle),sa=std::sin(_trackingSeed.angle); + float w=_trackingSeed.width*aspect,h=_trackingSeed.height; + BOOL ellipse=_trackingSeed.kind==DYMaskEllipse; + float bw=ellipse?std::hypot(w*ca,h*sa):fabs(w*ca)+fabs(h*sa); + float bh=ellipse?std::hypot(w*sa,h*ca):fabs(w*sa)+fabs(h*ca); + dy::TrackerROI roi={_trackingSeed.centerX,_trackingSeed.centerY,bw/aspect,bh,0}; + os_log(OS_LOG_DEFAULT,"dayu_object_track_worker main=%{public}d",(int)NSThread.isMainThread); + if(!_objectTracker.reset((const uint8_t*)rgba.bytes,width,height,_trackingFlip,roi,1./CMTimeGetSeconds(_trackingFrameDuration)))return [self finishToolAnalysis:@"无法建立目标外观,请调整蒙版后重试" error:error]; + objectStep={{roi.cx,roi.cy,roi.width,roi.height,1},true,false,false,0}; + }else objectStep=_objectTracker.track((const uint8_t*)rgba.bytes,width,height,_trackingFlip); + target=_objectTracker.result(objectStep); + if(!target.valid)return [self finishToolAnalysis:@"目标未能重新确认,已停止;请调整蒙版后继续" error:error]; + }else if(!_trackingSamples.count){ dy::TrackerROI targetROI={_trackingSeed.centerX,_trackingSeed.centerY,_trackingSeed.width,_trackingSeed.height,_trackingSeed.angle}; if(!_toolTracker.reset(frame,targetROI)) return [self finishToolAnalysis:stamp?@"覆盖区域内纹理不足,原图章已保留":@"蒙版内纹理不足,原蒙版已保留" error:error]; @@ -277,6 +299,13 @@ p.angle=_trackingSeed.angle+target.rotation; } if(!dyValidMaskParameters(p))return [self stopToolAnalysis:@"跟踪变换无效,未写入错误结果" error:error]; + if(_trackingObjectMode){ + if(objectStep.recovered){ + if(!DYRepairObjectTrackingGap(_trackingSamples,p,objectStep.gap))return [self finishToolAnalysis:@"遮挡段无法安全补齐,已保留确认轨迹" error:error]; + _trackingObjectRepaired+=objectStep.gap; + } + _trackingObjectPending=objectStep.predicted?objectStep.gap:0; + } [_trackingSamples addObject:DYStampKey(time,p,NO,confidence,motion)];_trackingPreviousTime=time; if(!_trackingSingle){ if(!(_trackingBoth&&_trackingBackward&&_trackingSamples.count==1&&_trackingPreviewCompleted))++_trackingPreviewCompleted; @@ -290,7 +319,7 @@ sampled.height=_trackingStampSourceSeed.height*source.scaleY; packet[@"sourceShape"]=[NSData dataWithBytes:&sampled length:sizeof(sampled)]; } - packet[@"shape"]=[NSData dataWithBytes:&covered length:sizeof(covered)]; + packet[@"shape"]=[NSData dataWithBytes:&covered length:sizeof(covered)];packet[@"predicted"]=@(objectStep.predicted); packet[@"nodes"]=_trackingFrozenMask[@"nodes"]; packet[@"flip"]=@(_trackingFlip);packet[@"time"]=@(CMTimeGetSeconds(time)); packet[@"completed"]=@(_trackingPreviewCompleted); @@ -305,7 +334,13 @@ }@finally{DYBetaSpanEnd(trackingSpan,nil);_trackingInFrame.store(false);} } - (BOOL)cleanupAnalysis:(NSError**)error { - if(!_trackingActive.load()||_trackingFinishing.exchange(true))return YES;BOOL changed=_trackingSamples.count>1;if(changed&&!DYMergeTracking(_trackingCandidate,_trackingSamples,_trackingFrameDuration)){[_trackingMessage release];_trackingMessage=[@"轨迹合并失败,原蒙版已保留" copy];_trackingDiscard=YES;} + if(!_trackingActive.load()||_trackingFinishing.exchange(true))return YES; + if(_trackingObjectPending){ + NSUInteger remove=MIN(_trackingObjectPending,_trackingSamples.count);[_trackingSamples removeObjectsInRange:NSMakeRange(_trackingSamples.count-remove,remove)]; + NSString *message=[NSString stringWithFormat:@"%@;已排除末尾 %lu 帧未确认预测",_trackingMessage?:@"目标丢失,请检查并从最后确认位置继续",(unsigned long)remove];[_trackingMessage release];_trackingMessage=[message copy];_trackingObjectPending=0; + } + BOOL changed=_trackingSamples.count>1;if(changed&&!DYMergeTracking(_trackingCandidate,_trackingSamples,_trackingFrameDuration)){[_trackingMessage release];_trackingMessage=[@"轨迹合并失败,原蒙版已保留" copy];_trackingDiscard=YES;} + _objectTracker.stop(); _toolTracker.stop(); if(_trackingStampMotion.active)_stampSourceTracker.stop(); uint64_t generation=_trackingGeneration;BOOL second=_trackingBoth&&!_trackingBackward&&!_trackingCancelled.load()&&!_trackingDiscard; @@ -313,6 +348,7 @@ if(!self->_trackingSingle)[self publishTrackingPreview]; if(second){self->_trackingBackward=YES;self->_trackingFinishing.store(false);self->_trackingStarted=NSProcessInfo.processInfo.systemUptime;ida=[self->_apiManager apiForProtocol:@protocol(FxCustomParameterActionAPI_v4)];BOOL started=NO;if(a){[a startAction:self];@try{started=[[self->_apiManager apiForProtocol:@protocol(FxAnalysisAPI)]startBackwardAnalysis:kFxAnalysisLocation_GPU error:nil];}@finally{[a endAction:self];}}if(started){[self toolStatus:@"正向完成,正在从同一起点反向跟踪…"];return;}self->_trackingFinishing.store(true);} BOOL hasResults=![self->_trackingCandidate[@"keys"]isEqual:self->_trackingFrozenMask[@"keys"]];NSString *message=self->_trackingMessage?:[NSString stringWithFormat:@"%@跟踪完成 · %lu 个轨迹键",self->_trackingBoth?@"双向":(self->_trackingBackward?@"反向":@"正向"),(unsigned long)[self->_trackingCandidate[@"keys"]count]]; + if(self->_trackingObjectRepaired)message=[message stringByAppendingFormat:@" · 补齐 %lu 帧短暂遮挡,请复核",(unsigned long)self->_trackingObjectRepaired]; if(hasResults&&!self->_trackingDiscard){ida=[self->_apiManager apiForProtocol:@protocol(FxCustomParameterActionAPI_v4)];if(a){[a startAction:self];@try{NSMutableDictionary *doc=DYMutablePlist(DYReadTools(self->_apiManager,a.currentTime).document);NSString *node=DYSharedMaskWithID(doc,self->_trackingMaskID)?@"shared":self->_trackingNode;NSMutableDictionary *mask=[self toolMask:self->_trackingMaskID node:node document:doc];if(mask&&[mask isEqual:self->_trackingFrozenMask]){self->_trackingCandidate[@"status"]=message;[mask setDictionary:self->_trackingCandidate];idundo=[self->_apiManager apiForProtocol:@protocol(FxUndoAPI)];[undo startUndoGroup:mask[@"stampSource"]?@"仿制图章跟踪":@"蒙版区域跟踪"];BOOL written=DYWriteTools(self->_apiManager,doc,a.currentTime);[undo endUndoGroup];if(!written)message=@"宿主未接受轨迹,原蒙版已保留";}else message=@"目标工具或蒙版已更改,已丢弃过期跟踪结果";}@finally{[a endAction:self];}}} CMTime stepTime=self->_trackingPreviousTime;BOOL advance=self->_trackingSingle&&changed&&!self->_trackingDiscard&&!self->_trackingCancelled.load()&&CMTIME_IS_NUMERIC(stepTime); @synchronized(self->_toolCaptureDevices){for(NSMutableDictionary *entry in self->_toolCaptureDevices.allValues)[entry removeObjectsForKeys:@[@"encoded",@"bytes",@"buffer",@"previewBuffer"]];} diff --git a/大鱼调色/Plugin/DYToolState.h b/大鱼调色/Plugin/DYToolState.h index e02d059..82fafb5 100644 --- a/大鱼调色/Plugin/DYToolState.h +++ b/大鱼调色/Plugin/DYToolState.h @@ -57,6 +57,8 @@ NSArray *DYStampSourceAtTime(NSDictionary *stamp,CMTime time); NSArray *DYStampSamplingScaleAtTime(NSDictionary *stamp,CMTime time); NSDictionary *DYStampKey(CMTime time,DYMaskParameters parameters,BOOL manual,float confidence,DYStampMotion motion); BOOL DYMergeTracking(NSMutableDictionary *mask,NSArray *samples,CMTime frameDuration=kCMTimeInvalid); +BOOL DYMosaicTracksMask(NSDictionary *document,NSString *maskID); +BOOL DYRepairObjectTrackingGap(NSMutableArray *samples,DYMaskParameters next,NSUInteger gap); NSDictionary *DYToolRenderSnapshot(NSDictionary *document,CMTime time); DYUniforms DYToolUniforms(NSDictionary *node); BOOL DYToolIsNeutral(NSDictionary *node); diff --git a/大鱼调色/Plugin/DYToolState.mm b/大鱼调色/Plugin/DYToolState.mm index 94f30d9..3e75474 100644 --- a/大鱼调色/Plugin/DYToolState.mm +++ b/大鱼调色/Plugin/DYToolState.mm @@ -316,6 +316,30 @@ static NSArray *smoothStampTrackingSamples(NSArray *samples){ [result addObject:samples.lastObject]; return result; } +BOOL DYMosaicTracksMask(NSDictionary *doc,NSString *maskID){ + NSDictionary *mosaic=doc[@"mosaic"],*mask=DYSharedMaskWithID(doc,maskID); + NSArray *ids=mosaic[@"maskIDs"]?: (mosaic[@"maskID"]?@[mosaic[@"maskID"]]:@[]); + DYMaskParameters p=DYMaskAtTime(mask,kCMTimeInvalid); + return mask&&(p.kind==DYMaskEllipse||p.kind==DYMaskRectangle)&&!mask[@"stampSource"]&&[mask[@"enabled"]boolValue]&&[mosaic[@"enabled"]boolValue]&&[mosaic[@"source"]isEqual:@"mask"]&&[ids containsObject:maskID]; +} +BOOL DYRepairObjectTrackingGap(NSMutableArray *samples,DYMaskParameters next,NSUInteger gap){ + if(!gap)return YES;if(samples.count<=gap||!dyValidMaskParameters(next))return NO; + NSUInteger anchor=samples.count-gap-1;DYMaskParameters left; + if(!readParameters(samples[anchor][@"parameters"],&left)||left.kind!=next.kind)return NO; + NSMutableArray *repaired=[NSMutableArray arrayWithCapacity:gap]; + for(NSUInteger i=1;i<=gap;i++){ + NSDictionary *original=samples[anchor+i];if([original[@"manual"]boolValue]||original[@"stampMotion"])return NO; + float t=float(i)/float(gap+1);DYMaskParameters p=left; + p.centerX=left.centerX+(next.centerX-left.centerX)*t;p.centerY=left.centerY+(next.centerY-left.centerY)*t; + p.width=std::exp(std::log(left.width)+(std::log(next.width)-std::log(left.width))*t); + p.height=std::exp(std::log(left.height)+(std::log(next.height)-std::log(left.height))*t); + p.angle=left.angle+std::remainder(next.angle-left.angle,float(2*M_PI))*t; + if(!dyValidMaskParameters(p))return NO; + NSMutableDictionary *key=[[original mutableCopy]autorelease];key[@"parameters"]=parameterData(p);key[@"confidence"]=@.35; + [repaired addObject:key]; + } + [samples replaceObjectsInRange:NSMakeRange(anchor+1,gap) withObjectsFromArray:repaired];return YES; +} BOOL DYMergeTracking(NSMutableDictionary *m,NSArray *samples,CMTime frameDuration){ if(!samples.count)return YES; NSMutableArray *all=[NSMutableArray array]; diff --git a/大鱼调色/Plugin/DYToolTrackingDiagnostics.inc.h b/大鱼调色/Plugin/DYToolTrackingDiagnostics.inc.h index 10088dc..38e7f82 100644 --- a/大鱼调色/Plugin/DYToolTrackingDiagnostics.inc.h +++ b/大鱼调色/Plugin/DYToolTrackingDiagnostics.inc.h @@ -172,6 +172,43 @@ paired&&ended&&[saved[@"keys"] count]==3&&[sourceLoss.domain isEqual:FxPlugErrorDomain]&& sourceLoss.code==kFxError_UserCancelled&&[sourceLoss.localizedDescription containsString:@"取样区域"]); } + { + NSMutableDictionary *objectDoc=DYMutablePlist([[[DYToolDocument alloc]init]autorelease].document),*objectMask=(id)DYAddSharedMask(objectDoc,DYMaskRectangle); + NSString *ID=objectMask[@"id"];DYMaskParameters shape=DYMaskAtTime(objectMask,kCMTimeZero);shape.width=shape.height=.45;shape.feather=0;objectMask[@"parameters"]=[NSData dataWithBytes:&shape length:sizeof(shape)]; + objectDoc[@"mosaic"]=[NSMutableDictionary dictionaryWithDictionary:@{@"enabled":@YES,@"source":@"mask",@"maskIDs":@[ID],@"size":@16}]; + run.record(@"mosaic-selected-mask-uses-object-tracking",DYMosaicTracksMask(objectDoc,ID)&&!DYMosaicTracksMask(objectDoc,@"missing")); + shape.kind=DYMaskBezier;objectMask[@"parameters"]=[NSData dataWithBytes:&shape length:sizeof(shape)]; + run.record(@"bezier-mosaic-preserves-polygon-flow-tracking",!DYMosaicTracksMask(objectDoc,ID)); + shape.kind=DYMaskRectangle;objectMask[@"parameters"]=[NSData dataWithBytes:&shape length:sizeof(shape)]; + objectDoc[@"mosaic"][@"enabled"]=@NO;run.record(@"disabled-mosaic-keeps-precise-flow-path",!DYMosaicTracksMask(objectDoc,ID));objectDoc[@"mosaic"][@"enabled"]=@YES; + DYWriteTools(base,objectDoc,kCMTimeZero);capture(0);host->_state=kFxAnalysisState_NotAnalyzing; + [plugin toolTrackRequested:[NSNotification notificationWithName:@"DYToolTrack" object:host userInfo:@{@"node":@"shared",@"mask":ID,@"direction":@1}]]; + BOOL ok=[plugin desiredAnalysisTimeRange:&desired forInputWithTimeRange:inputRange error:nil]&&[plugin setupAnalysisForTimeRange:desired frameDuration:timeAt(1) error:nil]; + for(int i=0;i<6&&ok;i++){setFrame(i);ok&=[plugin analyzeFrame:frame atTime:timeAt(i) error:nil];} + run.record(@"mosaic-object-host-callbacks-produce-six-frames",ok&&[[(NSObject*)plugin valueForKey:@"trackingSamples"]count]==6); + // Simulate cancellation while two provisional predictions are awaiting a + // confirmed endpoint. Those keys must never reach the host document. + [(NSObject*)plugin setValue:@2 forKey:@"trackingObjectPending"];[plugin cleanupAnalysis:nil];drain(); + NSDictionary *savedMask=DYSharedMaskWithID(DYReadTools(base,kCMTimeZero).document,ID); + run.record(@"object-tracking-discards-unconfirmed-tail-before-commit",[savedMask[@"keys"]count]==4&&[savedMask[@"status"]containsString:@"未确认预测"]); + } [NSNotificationCenter.defaultCenter removeObserver:observer]; [plugin release]; } +{ + dy::ObjectMotion motion;dy::ObjectBox box{.3,.4,.1,.15,.95};motion.reset(box,25);bool ok=true; + for(int i=1;i<=10;i++){box.cx=.3f+.002f*i;ok&=motion.advance(box,true).valid;} + auto predicted=motion.advance({},false);run.record(@"object-short-loss-predicts-prior-motion",ok&&predicted.valid&&predicted.predicted&&predicted.box.cx>box.cx&&predicted.box.confidence<.3); + motion.advance({},false);box.cx=.326;auto first=motion.advance(box,true);box.cx=.328;auto second=motion.advance(box,true); + run.record(@"object-reacquisition-requires-two-consistent-observations",first.predicted&&second.recovered&&second.gap==3&&!second.predicted); + auto teleport=box;teleport.cx=.9;run.record(@"object-motion-rejects-distant-distractor",motion.advance(teleport,true).predicted); + dy::ObjectStep last;for(int i=0;i<25;i++)last=motion.advance({},false);run.record(@"object-long-loss-stops-bounded-prediction",!last.valid&&!motion.active()); + motion.reset({.5,.5,.1,.1,.9},std::numeric_limits::quiet_NaN()); + run.record(@"object-invalid-fps-does-not-enable-prediction",!motion.active()); + DYMaskParameters a=dyDefaultMaskParameters(),b=a;a.centerX=.2;b.centerX=.8;a.width=.1;b.width=.4; + NSMutableArray *keys=[NSMutableArray arrayWithObjects:DYMaskKey(CMTimeMake(3,25),a,NO,1),DYMaskKey(CMTimeMake(2,25),a,NO,.1),DYMaskKey(CMTimeMake(1,25),a,NO,.1),nil]; + BOOL repaired=DYRepairObjectTrackingGap(keys,b,2);DYMaskParameters x,y;[keys[1][@"parameters"]getBytes:&x length:sizeof(x)];[keys[2][@"parameters"]getBytes:&y length:sizeof(y)]; + run.record(@"object-gap-repair-uses-both-endpoints-in-reverse-order",repaired&&fabs(x.centerX-.4)<1e-6&&fabs(y.centerX-.6)<1e-6&&[keys[1][@"value"]intValue]==2&&[keys[1][@"confidence"]floatValue]<.5); + NSArray *before=[[keys copy]autorelease];run.record(@"object-gap-repair-rejects-missing-confirmed-anchor",!DYRepairObjectTrackingGap(keys,b,3)&&[keys isEqual:before]); + keys[1]=DYMaskKey(CMTimeMake(2,25),a,YES,1);before=[[keys copy]autorelease];run.record(@"object-gap-repair-never-overwrites-manual-key",!DYRepairObjectTrackingGap(keys,b,2)&&[keys isEqual:before]); +} diff --git a/大鱼调色/Plugin/DYToolTrackingProbe.inc.h b/大鱼调色/Plugin/DYToolTrackingProbe.inc.h index 356d80a..0f7a44e 100644 --- a/大鱼调色/Plugin/DYToolTrackingProbe.inc.h +++ b/大鱼调色/Plugin/DYToolTrackingProbe.inc.h @@ -1,4 +1,5 @@ #include "MaskCore/DYTracker.hpp" +#include "MaskCore/DYObjectMotion.hpp" @interface DYToolTrackingProbe : NSObject { @public id _base;CMTime _time,_timelineOffset,_seekTime;int _seekCount;FxAnalysisState _state;int _forwardStarts,_backwardStarts; diff --git a/大鱼调色/Plugin/DYViewerParameterBridge.inc.h b/大鱼调色/Plugin/DYViewerParameterBridge.inc.h index 49fa58b..39e5462 100644 --- a/大鱼调色/Plugin/DYViewerParameterBridge.inc.h +++ b/大鱼调色/Plugin/DYViewerParameterBridge.inc.h @@ -162,7 +162,7 @@ static BOOL DYViewerSanitizeValue(NSDictionary *d,double input,double *output) { if(current&&[faces[@"boxes"]count])add([NSString stringWithFormat:@"将本帧 %lu 张人脸转为可选跟踪蒙版",(unsigned long)[faces[@"boxes"]count]],@{@"importFaces":faces,@"actionTime":[(NSDictionary*)CMTimeCopyAsDictionary(actionTime,kCFAllocatorDefault)autorelease]},NO); else {NSMenuItem *hint=[menu addItemWithTitle:current?@"本帧未识别到人脸":@"先开启全部人脸,暂停后再次打开以指定人脸" action:nil keyEquivalent:@""];hint.enabled=NO;} [menu addItem:NSMenuItem.separatorItem]; - NSMenuItem *hint=[menu addItemWithTitle:@"勾选要遮挡的蒙版(可多选);在控制面板逐个跟踪" action:nil keyEquivalent:@""];hint.enabled=NO; + NSMenuItem *hint=[menu addItemWithTitle:@"勾选后逐个跟踪;矩形/椭圆支持短时预测与重新定位" action:nil keyEquivalent:@""];hint.enabled=NO; NSArray *IDs=settings[@"maskIDs"]?: (settings[@"maskID"]?@[settings[@"maskID"]]:@[]);NSInteger count=0; for(NSDictionary *mask in DYSharedMasks(doc)){if(mask[@"stampSource"])continue;count++;DYMaskParameters p=DYMaskAtTime(mask,kCMTimeInvalid);NSString *shape=mask[@"label"]?:[NSString stringWithFormat:@"蒙版 %ld · %@",(long)count,@[@"椭圆",@"矩形",@"钢笔",@"渐变",@"HSL",@"亮度"][p.kind]]; add([shape stringByAppendingString:[mask[@"keys"]count]?@" · 已跟踪":@" · 静态"],@{@"toggleMask":mask[@"id"]},[settings[@"enabled"]boolValue]&&[settings[@"source"]isEqual:@"mask"]&&[IDs containsObject:mask[@"id"]]); diff --git a/大鱼调色/Plugin/Info.plist b/大鱼调色/Plugin/Info.plist index 1bafe12..55d92e4 100644 --- a/大鱼调色/Plugin/Info.plist +++ b/大鱼调色/Plugin/Info.plist @@ -17,9 +17,9 @@ CFBundlePackageType XPC! CFBundleShortVersionString - 1.18.1 + 1.19.0 CFBundleVersion - 106 + 107 NSHumanReadableCopyright PlugInKit diff --git a/大鱼调色/Plugin/MaskCore/DYObjectMotion.hpp b/大鱼调色/Plugin/MaskCore/DYObjectMotion.hpp index 0c8da2f..a12f6a6 100644 --- a/大鱼调色/Plugin/MaskCore/DYObjectMotion.hpp +++ b/大鱼调色/Plugin/MaskCore/DYObjectMotion.hpp @@ -15,7 +15,7 @@ class ObjectMotion { bool active_=false; static bool finite(ObjectBox b){return std::isfinite(b.cx)&&std::isfinite(b.cy)&&std::isfinite(b.width)&&std::isfinite(b.height)&&std::isfinite(b.confidence)&&b.width>.003f&&b.height>.003f&&b.width<=1&&b.height<=1&&b.cx>=0&&b.cx<=1&&b.cy>=0&&b.cy<=1;} public: - void reset(ObjectBox b,double fps){trusted_=b;vx_=vy_=0;missing_=candidateCount_=0;limit_=std::clamp(int(std::ceil(std::clamp(fps,1.,240.)*.8)),2,192);active_=finite(b);} + void reset(ObjectBox b,double fps){trusted_=b;vx_=vy_=0;missing_=candidateCount_=0;active_=finite(b)&&std::isfinite(fps)&&fps>0;limit_=active_?std::clamp(int(std::ceil(std::clamp(fps,1.,240.)*.8)),2,192):2;} int missing()const{return missing_;} bool active()const{return active_;} ObjectBox prediction(int ahead=1)const{auto b=trusted_;b.cx+=vx_*(missing_+ahead);b.cy+=vy_*(missing_+ahead);return b;} diff --git a/大鱼调色/Wrapper Application/Info.plist b/大鱼调色/Wrapper Application/Info.plist index 900b3da..8d3e562 100644 --- a/大鱼调色/Wrapper Application/Info.plist +++ b/大鱼调色/Wrapper Application/Info.plist @@ -17,9 +17,9 @@ CFBundlePackageType APPL CFBundleShortVersionString - 1.18.1 + 1.19.0 CFBundleVersion - 106 + 107 LSMinimumSystemVersion $(MACOSX_DEPLOYMENT_TARGET) NSHumanReadableCopyright diff --git a/开发交接.md b/开发交接.md index 3b3f308..4844633 100644 --- a/开发交接.md +++ b/开发交接.md @@ -4,7 +4,7 @@ ## 当前基线 -2026-11-06 后续开发中:用户要求继续解决转身偏移,短时预测/重新定位后补齐,并比较 GitHub 成熟算法。当前独立候选已验证 Vision 冻结可信模板能跨过人工短遮挡;尚未接入 APP,正在与 OpenCV CSRT/VitTrack 做同片段比较。[候选与接手入口](research/TrackingRecoveryCandidates.md)。正式安装与线上仍为下述 1.18.1;不要把独立基准当作正式交付。 +2026-11-06 后续开发中:1.19.0(107)已接入马赛克矩形/椭圆对象追踪、冻结可信模板、0.8 秒内预测、两次确认及前后端点补齐,尾部未确认键不提交。CSRT、VitTrack 同片段已实测,当前 Vision 组合更稳。长遮挡/突切停止与反向独立基准通过,正在正式构建和宿主验收。[比较与限制](research/TrackingRecoveryCandidates.md)。正式安装与线上暂仍为 1.18.1。 **1.18.1(106)已正式安装、发布 Git Release,并更新文章 15792 与主 APP 公测通道。** 实际编译提交 `1cc7f8edb2acb3cfc62bd72447c2956ad7a01601`;发行标签 `v1.18.1-tracking-ui-20261106` 指向 `5fe4b1933e015a906fab36b65dcbc45a0d6bee49`。APP 公证 `2c4e6cba-8178-4a50-9198-83e66be37add` Accepted。安装后 964 项原生检查通过,FCP 独立副本完成两轮 285 帧跟踪及模块/参数交互。