fix(camera): harden + correct the AR/depth real-world-scale module#226
Open
Eliaaazzz wants to merge 5 commits into
Open
fix(camera): harden + correct the AR/depth real-world-scale module#226Eliaaazzz wants to merge 5 commits into
Eliaaazzz wants to merge 5 commits into
Conversation
The repo shipped jest-expo + @testing-library/react-native as devDeps and
several RN-importing test files, but no jest config to tie them together, so
`npm test` failed for every test that imported react-native
("Cannot use import statement outside a module"). It also pinned
react@19.1.0 while react-test-renderer floated to ^19.2.3 (no exact match),
which @testing-library/react-native rejects.
- Add frontend/jest.config.js (preset: jest-expo) + jest.setup.js (official
AsyncStorage mock).
- Pin react-test-renderer to 19.1.0 (frontend devDep + root overrides) to match
the pinned react.
Result: npm test runs — 102/103 tests pass (the 1 failure is a pre-existing
auth-store issue: a dynamic import needing --experimental-vm-modules, unrelated).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses confirmed findings from the AR-module review:
useDeviceCapabilities:
- Remove iPhone 13 / 13 mini (iPhone14,4/14,5) from the LiDAR list — they have
no LiDAR, so these very common phones were mis-tiered as 'lidar' (<1% accuracy).
- Replace the `modelId.includes('pro')` heuristic with exact model-id membership:
it never matched a real Device.modelId ('iPhone15,2'...) and would have falsely
flagged the LiDAR-less iPhone 11 Pro / 2018 iPad Pro if fed a model name.
- Relabel the depth tier "Pro Accuracy (LiDAR)" -> "Pro Accuracy (Depth Sensor)"
since it also covers Android ToF devices (a Galaxy S23 Ultra is not LiDAR).
useARScale:
- Reset started.current in the effect cleanup so a later start() can actually
restart after an `enabled` toggle (previously the session stopped but the guard
stayed set, so start() early-returned forever).
- Add a 12-90cm plausibility gate on measured img_w_cm: an off-table raycast
(e.g. a far wall) now degrades to the plate fallback instead of sending a
fabricated scale to the backend.
Adds contract tests pinning the "never fabricate scale / degrade to plate
fallback" guarantee (native present, native absent, plausibility gate, clean
restart) and the standard_ar tier label.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
iOS GetDepthAtCenterPlugin (reference template): - Remove the CMGetAttachment(sampleBuffer, "AVDepthData") path — AVFoundation never attaches AVDepthData to a video CMSampleBuffer, so as written the plugin could never return a distance on any device. Replace with a timestamp-keyed DepthStore that the app's AVCaptureDataOutputSynchronizer delegate feeds (the iOS analogue of the Android DepthBridge), with a staleness guard so a stale depth map is never paired with a newer video frame. - Never emit fx:0 with hasDepth:true: when cameraCalibrationData is absent the pinhole conversion would divide by zero, so report hasDepth:false instead. Android GetDepthAtCenterPlugin (reference template): - Decode the real DEPTH16 confidence bits (high 3 bits, 0 => 100%, else (code-1)/7) and report the mean confidence instead of a hardcoded 0.9 — so the JS minConfidence filter is meaningful. Also guard fx<=0 => hasDepth:false. These remain device-only templates (cannot be compiled here), but are now correct starting points rather than dead/divide-by-zero paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README described a scale pipeline that does not exist. Corrected to reality:
- Remove all references to src/utils/scale.ts / estimateImageWidthCm() (no such
file or function exists). The pinhole helper is now shown as code-to-add under
"Completing the LiDAR path", not as existing/wired.
- Stop claiming "the TypeScript side is fully wired". State clearly what IS wired
(the standard_ar AR path: useARScale -> route.params.imgWcm -> backend) vs what
is NOT (the LiDAR depth -> img_w_cm conversion: DepthResult has no fx, there is
no conversion, CaptureMetadata has no imgWcm).
- Fix the DepthResult contract block to match the real interface (drop the
fx/horizontalFovDeg fields it never declared; note fx is returned by native but
not yet consumed; the FOV fallback is not implemented).
- Fix the verify step: CaptureMetadata carries { distanceCm, confidence, accuracy,
deviceTier } — there is no metadata.imgWcm. The value that reaches the backend
is route.params.imgWcm.
- Document the new iOS DepthStore / synchronizer wiring; note img_w_cm is gated
(not clamped) to 12-90cm in useARScale.
- Drop the stale estimateImageWidthCm reference in useDistanceEstimation's docstring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
From an adversarial verification of the previous commits: - README (the big one): stop claiming the standard_ar path is "wired in JS, one device-side step remaining". Nothing calls useARScale and nothing sets route.params.imgWcm — only the SINK (ReviewMealScreen -> img_w_cm, 35cm default) is wired. Reword "what is wired today" + the diagram to say the producer-side JS glue is also unwritten for both paths. - README: the verify step claimed GeminiMealAnalysisService logs img_w_cm; it does not log the request body. Point to the outbound nutritionApi request / gateway logs instead. - iOS Swift: header comment said DepthStore is "keyed by presentation timestamp" (implying a map); it keeps only the latest sample matched within a staleness window. Clarified. - useDeviceCapabilities: add the iPad Pro 12.9" 5th-gen (2021, M1) LiDAR ids (iPad13,8-13,11), which exact-match detection would otherwise miss. - Add useDeviceCapabilities.test.tsx pinning the tier fixes (iPhone 13 -> standard_ar, 15 Pro -> lidar, iPad 5th-gen -> lidar, ToF -> not labeled LiDAR) — the one coverage gap the verification flagged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
aurafit | 287b233 | Jun 28 2026, 01:14 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
aurafitness | 287b233 | Jun 28 2026, 01:15 PM |
Deploying aurafitness-2 with
|
| Latest commit: |
287b233
|
| Status: | 🚫 Build failed. |
Deploying aurafitness with
|
| Latest commit: |
287b233
|
| Status: | 🚫 Build failed. |
Deploying aurafitness-1 with
|
| Latest commit: |
287b233
|
| Status: | 🚫 Build failed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Follow-up fixes after pulling the freshly-merged ARKit / depth real-world scale module
(
a0d1283, thefeat(camera): ARKit plane-raycast scale modulework) and reviewing it. Thenative iOS/Android files are reference templates that can't be compiled outside Xcode/Android
Studio — so this PR verifies and hardens everything testable (TypeScript bridge, device tiering,
test infra) and corrects the native templates + docs to be honest, correct starting points.
Verified locally:
tsc --noEmitclean (only 4 pre-existing, unrelated errors) and 110/111jest tests pass (the 1 failure is the pre-existing
useAuthStoredynamic-import issue).Fixes
Test infrastructure —
npm testwas broken for every RN-importing test (no jest config wiredjest-expo;react-test-rendererfloated off the pinnedreact). Addedjest.config.js(
jest-expopreset) +jest.setup.js(AsyncStorage mock), pinnedreact-test-renderer@19.1.0.JS bridge bugs
useDeviceCapabilities: removed iPhone 13 / 13 mini from the LiDAR list (they have none),replaced the dead/over-broad
'pro'substring with exact model-id matching, added the iPad Pro12.9" 5th-gen LiDAR IDs, and relabeled the depth tier so Android ToF devices aren't called "LiDAR".
useARScale: cleanup now resets the start-guard so it can restart after anenabledtoggle;added a 12–90cm plausibility gate so an off-table raycast degrades to the plate fallback instead
of sending a fabricated scale.
and the device-tier classification.
Native templates (still device-only, now correct)
GetDepthAtCenterPlugin: removed theCMGetAttachment(sampleBuffer, "AVDepthData")path —AVFoundation never attaches depth to a video sample buffer, so as written it could never return a
distance. Replaced with a timestamp-matched
DepthStorefed by theAVCaptureDataOutputSynchronizerdelegate (the iOS analogue of the Android
DepthBridge). Never emitfx:0withhasDepth:true.GetDepthAtCenterPlugin: decode the real DEPTH16 confidence bits instead of hardcoding0.9(so the JSminConfidencefilter is meaningful); samefx<=0guard.Docs — corrected
native-depth-plugin/README.md: removed references to a non-existentsrc/utils/scale.ts/estimateImageWidthCm(), fixed theDepthResultcontract block and theverify steps, and replaced the "TS side is fully wired" claim with an accurate "what's wired (the
route.params.imgWcmsink) vs what's still a template / unwritten glue" section.Honest residual
Building the native modules and writing the capture-screen JS glue (calling
useARScale().measure()and pushing
imgWcmintoroute.params) remain device-side work — now documented accurately ratherthan overstated as done.
🤖 Generated with Claude Code