diff --git a/IOS-PRIORITIES.md b/IOS-PRIORITIES.md new file mode 100644 index 00000000000..5c4c8e9e275 --- /dev/null +++ b/IOS-PRIORITIES.md @@ -0,0 +1,100 @@ +# iOS App Priorities (OpenClaw / Moltbot) + +This report is based on repo code + docs in `/Users/mariano/Coding/openclaw`, with focus on: +- iOS Swift sources under `apps/ios/Sources` +- Shared Swift packages under `apps/shared/OpenClawKit` +- Gateway protocol + node docs in `docs/` +- macOS node implementation under `apps/macos/Sources/OpenClaw/NodeMode` + +## Current iOS state (what works today) + +**Gateway connectivity + pairing** +- Uses the unified Gateway WebSocket protocol with device identity + challenge signing (via `GatewayChannel` in OpenClawKit). +- Discovery via Bonjour (`NWBrowser`) for `_openclaw-gw._tcp` plus manual host/port fallback and TLS pinning support (`apps/ios/Sources/Gateway/*`). +- Stores gateway token/password in Keychain (`GatewaySettingsStore.swift`). + +**Node command handling** (implemented in `NodeAppModel.handleInvoke`) +- Canvas: `canvas.present`, `canvas.hide`, `canvas.navigate`, `canvas.eval`, `canvas.snapshot`. +- A2UI: `canvas.a2ui.reset`, `canvas.a2ui.push`, `canvas.a2ui.pushJsonl`. +- Camera: `camera.list`, `camera.snap`, `camera.clip`. +- Screen: `screen.record` (ReplayKit-based screen recording). +- Location: `location.get` (CoreLocation-based). +- Foreground gating: returns `NODE_BACKGROUND_UNAVAILABLE` for canvas/camera/screen when backgrounded. + +**Voice features** +- Voice Wake: continuous speech recognition with wake-word gating and gateway sync (`VoiceWakeManager.swift`). +- Talk Mode: speech-to-text + chat.send + ElevenLabs streaming TTS + system voice fallback (`TalkModeManager.swift`). + +**Chat UI** +- Uses shared SwiftUI chat client (`OpenClawChatUI`) and Gateway chat APIs (`IOSGatewayChatTransport.swift`). + +**UI surface** +- Full-screen canvas with overlay controls for chat, settings, and Talk orb (`RootCanvas.swift`). +- Settings for gateway selection, voice, camera, location, screen prevent-sleep, and debug flags (`SettingsTab.swift`). + +## Protocol requirements the iOS app must honor + +From `docs/gateway/protocol.md` + `docs/nodes/index.md` + OpenClawKit: +- WebSocket `connect` handshake with `role: "node"`, `caps`, `commands`, and `permissions` claims. +- Device identity + challenge signing on connect; device token persistence. +- Respond to `node.invoke.request` with `node.invoke.result`. +- Emit node events (`node.event`) for voice transcripts and agent requests. +- Use gateway RPCs needed by the iOS UI: `config.get`, `voicewake.get/set`, `chat.*`, `sessions.list`. + +## Gaps / incomplete or mismatched behavior + +**1) Declared commands exceed iOS implementation** +`GatewayConnectionController.currentCommands()` includes: +- `system.run`, `system.which`, `system.notify`, `system.execApprovals.get`, `system.execApprovals.set` + +…but `NodeAppModel.handleInvoke` does not implement any `system.*` commands and will return `INVALID_REQUEST: unknown command` for them. This is a protocol-level mismatch: the gateway will believe iOS supports system execution + notifications, but the node cannot fulfill those requests. + +**2) Permissions map is always empty** +iOS sends `permissions: [:]` in its connect options, while macOS node reports real permission states via `PermissionManager`. This means the gateway cannot reason about iOS permission availability even though camera/mic/location/screen limitations materially affect command success. + +**3) Canvas parity gaps** +- `canvas.hide` is currently a no-op on iOS (returns ok but doesn’t change UI). +- `canvas.present` ignores placement params (macOS supports window placement). + +These may be acceptable platform limitations, but they should be explicitly handled/documented so the node surface is consistent and predictable. + +## iOS vs. macOS node feature parity + +macOS node mode (`apps/macos/Sources/OpenClaw/NodeMode/*`) supports: +- `system.run`, `system.which`, `system.notify`, `system.execApprovals.get/set`. +- Permission reporting in `connect.permissions`. +- Canvas window placement + hide. + +iOS currently implements the shared node surface (canvas/camera/screen/location + voice) but does **not** match macOS on the system/exec side and permission reporting. + +## Prioritized work items (ordered by importance) + +1) **Fix the command/implementation mismatch for `system.*`** + - Either remove `system.*` from iOS `currentCommands()` **or** implement iOS equivalents (at minimum `system.notify` via local notifications) with clear error semantics for unsupported actions. + - This is the highest risk mismatch because it misleads the gateway and any operator about what the iOS node can actually do. + +2) **Report real iOS permission state in `connect.permissions`** + - Mirror macOS behavior by sending camera/microphone/location/screen-recording permission flags. + - This enables the gateway to make better decisions and reduces “it failed because permissions” surprises. + +3) **Clarify/normalize iOS canvas behaviors** + - Decide how `canvas.hide` should behave on iOS (e.g., return to the local scaffold) and implement it. + - Document that `canvas.present` ignores placement on iOS, or add a platform-specific best effort. + +4) **Explicitly document platform deltas vs. macOS node** + - The docs currently describe `system.*` under “Nodes” and cite macOS/headless node support. iOS should be clearly marked as not supporting system exec to avoid incorrect user expectations. + +5) **Release readiness (if the goal is to move beyond internal preview)** + - Docs state the iOS app is “internal preview” (`docs/platforms/ios.md`). + - If public distribution is desired, build out TestFlight/App Store release steps (fastlane exists in `apps/ios/fastlane/`). + +## Files referenced (key evidence) + +- iOS node behavior: `apps/ios/Sources/Model/NodeAppModel.swift` +- iOS command declarations: `apps/ios/Sources/Gateway/GatewayConnectionController.swift` +- iOS discovery + TLS: `apps/ios/Sources/Gateway/*` +- iOS voice: `apps/ios/Sources/Voice/*` +- iOS screen/camera/location: `apps/ios/Sources/Screen/*`, `apps/ios/Sources/Camera/*`, `apps/ios/Sources/Location/*` +- Shared protocol + commands: `apps/shared/OpenClawKit/Sources/OpenClawKit/*` +- macOS node runtime: `apps/macos/Sources/OpenClaw/NodeMode/*` +- Node + protocol docs: `docs/nodes/index.md`, `docs/gateway/protocol.md`, `docs/platforms/ios.md` diff --git a/apps/ios/Sources/Calendar/CalendarService.swift b/apps/ios/Sources/Calendar/CalendarService.swift new file mode 100644 index 00000000000..0ba8e8417a8 --- /dev/null +++ b/apps/ios/Sources/Calendar/CalendarService.swift @@ -0,0 +1,66 @@ +import EventKit +import Foundation +import OpenClawKit + +final class CalendarService: CalendarServicing { + func events(params: OpenClawCalendarEventsParams) async throws -> OpenClawCalendarEventsPayload { + let store = EKEventStore() + let status = EKEventStore.authorizationStatus(for: .event) + let authorized = await Self.ensureAuthorization(store: store, status: status) + guard authorized else { + throw NSError(domain: "Calendar", code: 1, userInfo: [ + NSLocalizedDescriptionKey: "CALENDAR_PERMISSION_REQUIRED: grant Calendar permission", + ]) + } + + let (start, end) = Self.resolveRange( + startISO: params.startISO, + endISO: params.endISO) + let predicate = store.predicateForEvents(withStart: start, end: end, calendars: nil) + let events = store.events(matching: predicate) + let limit = max(1, min(params.limit ?? 50, 500)) + let selected = Array(events.prefix(limit)) + + let formatter = ISO8601DateFormatter() + let payload = selected.map { event in + OpenClawCalendarEventPayload( + identifier: event.eventIdentifier ?? UUID().uuidString, + title: event.title ?? "(untitled)", + startISO: formatter.string(from: event.startDate), + endISO: formatter.string(from: event.endDate), + isAllDay: event.isAllDay, + location: event.location, + calendarTitle: event.calendar.title) + } + + return OpenClawCalendarEventsPayload(events: payload) + } + + private static func ensureAuthorization(store: EKEventStore, status: EKAuthorizationStatus) async -> Bool { + switch status { + case .authorized: + return true + case .notDetermined: + return await withCheckedContinuation { cont in + store.requestAccess(to: .event) { granted, _ in + cont.resume(returning: granted) + } + } + case .restricted, .denied: + return false + case .fullAccess: + return true + case .writeOnly: + return false + @unknown default: + return false + } + } + + private static func resolveRange(startISO: String?, endISO: String?) -> (Date, Date) { + let formatter = ISO8601DateFormatter() + let start = startISO.flatMap { formatter.date(from: $0) } ?? Date() + let end = endISO.flatMap { formatter.date(from: $0) } ?? start.addingTimeInterval(7 * 24 * 3600) + return (start, end) + } +} diff --git a/apps/ios/Sources/Contacts/ContactsService.swift b/apps/ios/Sources/Contacts/ContactsService.swift new file mode 100644 index 00000000000..560302cb9d4 --- /dev/null +++ b/apps/ios/Sources/Contacts/ContactsService.swift @@ -0,0 +1,72 @@ +import Contacts +import Foundation +import OpenClawKit + +final class ContactsService: ContactsServicing { + func search(params: OpenClawContactsSearchParams) async throws -> OpenClawContactsSearchPayload { + let store = CNContactStore() + let status = CNContactStore.authorizationStatus(for: .contacts) + let authorized = await Self.ensureAuthorization(store: store, status: status) + guard authorized else { + throw NSError(domain: "Contacts", code: 1, userInfo: [ + NSLocalizedDescriptionKey: "CONTACTS_PERMISSION_REQUIRED: grant Contacts permission", + ]) + } + + let limit = max(1, min(params.limit ?? 25, 200)) + let keys: [CNKeyDescriptor] = [ + CNContactIdentifierKey as CNKeyDescriptor, + CNContactGivenNameKey as CNKeyDescriptor, + CNContactFamilyNameKey as CNKeyDescriptor, + CNContactOrganizationNameKey as CNKeyDescriptor, + CNContactPhoneNumbersKey as CNKeyDescriptor, + CNContactEmailAddressesKey as CNKeyDescriptor, + ] + + var contacts: [CNContact] = [] + if let query = params.query?.trimmingCharacters(in: .whitespacesAndNewlines), !query.isEmpty { + let predicate = CNContact.predicateForContacts(matchingName: query) + contacts = try store.unifiedContacts(matching: predicate, keysToFetch: keys) + } else { + let request = CNContactFetchRequest(keysToFetch: keys) + try store.enumerateContacts(with: request) { contact, stop in + contacts.append(contact) + if contacts.count >= limit { + stop.pointee = true + } + } + } + + let sliced = Array(contacts.prefix(limit)) + let payload = sliced.map { contact in + OpenClawContactPayload( + identifier: contact.identifier, + displayName: CNContactFormatter.string(from: contact, style: .fullName) + ?? "\(contact.givenName) \(contact.familyName)".trimmingCharacters(in: .whitespacesAndNewlines), + givenName: contact.givenName, + familyName: contact.familyName, + organizationName: contact.organizationName, + phoneNumbers: contact.phoneNumbers.map { $0.value.stringValue }, + emails: contact.emailAddresses.map { String($0.value) }) + } + + return OpenClawContactsSearchPayload(contacts: payload) + } + + private static func ensureAuthorization(store: CNContactStore, status: CNAuthorizationStatus) async -> Bool { + switch status { + case .authorized, .limited: + return true + case .notDetermined: + return await withCheckedContinuation { cont in + store.requestAccess(for: .contacts) { granted, _ in + cont.resume(returning: granted) + } + } + case .restricted, .denied: + return false + @unknown default: + return false + } + } +} diff --git a/apps/ios/Sources/Device/DeviceStatusService.swift b/apps/ios/Sources/Device/DeviceStatusService.swift new file mode 100644 index 00000000000..fed2716b5b8 --- /dev/null +++ b/apps/ios/Sources/Device/DeviceStatusService.swift @@ -0,0 +1,87 @@ +import Foundation +import OpenClawKit +import UIKit + +final class DeviceStatusService: DeviceStatusServicing { + private let networkStatus: NetworkStatusService + + init(networkStatus: NetworkStatusService = NetworkStatusService()) { + self.networkStatus = networkStatus + } + + func status() async throws -> OpenClawDeviceStatusPayload { + let battery = self.batteryStatus() + let thermal = self.thermalStatus() + let storage = self.storageStatus() + let network = await self.networkStatus.currentStatus() + let uptime = ProcessInfo.processInfo.systemUptime + + return OpenClawDeviceStatusPayload( + battery: battery, + thermal: thermal, + storage: storage, + network: network, + uptimeSeconds: uptime) + } + + func info() -> OpenClawDeviceInfoPayload { + let device = UIDevice.current + let appVersion = Bundle.main.infoDictionary?["CFBundleShortVersionString"] as? String ?? "dev" + let appBuild = Bundle.main.infoDictionary?["CFBundleVersion"] as? String ?? "0" + let locale = Locale.preferredLanguages.first ?? Locale.current.identifier + return OpenClawDeviceInfoPayload( + deviceName: device.name, + modelIdentifier: Self.modelIdentifier(), + systemName: device.systemName, + systemVersion: device.systemVersion, + appVersion: appVersion, + appBuild: appBuild, + locale: locale) + } + + private func batteryStatus() -> OpenClawBatteryStatusPayload { + let device = UIDevice.current + device.isBatteryMonitoringEnabled = true + let level = device.batteryLevel >= 0 ? Double(device.batteryLevel) : nil + let state: OpenClawBatteryState = switch device.batteryState { + case .charging: .charging + case .full: .full + case .unplugged: .unplugged + case .unknown: .unknown + @unknown default: .unknown + } + return OpenClawBatteryStatusPayload( + level: level, + state: state, + lowPowerModeEnabled: ProcessInfo.processInfo.isLowPowerModeEnabled) + } + + private func thermalStatus() -> OpenClawThermalStatusPayload { + let state: OpenClawThermalState = switch ProcessInfo.processInfo.thermalState { + case .nominal: .nominal + case .fair: .fair + case .serious: .serious + case .critical: .critical + @unknown default: .nominal + } + return OpenClawThermalStatusPayload(state: state) + } + + private func storageStatus() -> OpenClawStorageStatusPayload { + let attrs = (try? FileManager.default.attributesOfFileSystem(forPath: NSHomeDirectory())) ?? [:] + let total = (attrs[.systemSize] as? NSNumber)?.int64Value ?? 0 + let free = (attrs[.systemFreeSize] as? NSNumber)?.int64Value ?? 0 + let used = max(0, total - free) + return OpenClawStorageStatusPayload(totalBytes: total, freeBytes: free, usedBytes: used) + } + + private static func modelIdentifier() -> String { + var systemInfo = utsname() + uname(&systemInfo) + let machine = withUnsafeBytes(of: &systemInfo.machine) { ptr in + String(bytes: ptr.prefix { $0 != 0 }, encoding: .utf8) + } + let trimmed = machine?.trimmingCharacters(in: .whitespacesAndNewlines) ?? "" + return trimmed.isEmpty ? "unknown" : trimmed + } +} diff --git a/apps/ios/Sources/Device/NetworkStatusService.swift b/apps/ios/Sources/Device/NetworkStatusService.swift new file mode 100644 index 00000000000..7d92d1cc1ca --- /dev/null +++ b/apps/ios/Sources/Device/NetworkStatusService.swift @@ -0,0 +1,69 @@ +import Foundation +import Network +import OpenClawKit + +final class NetworkStatusService: @unchecked Sendable { + func currentStatus(timeoutMs: Int = 1500) async -> OpenClawNetworkStatusPayload { + await withCheckedContinuation { cont in + let monitor = NWPathMonitor() + let queue = DispatchQueue(label: "bot.molt.ios.network-status") + let state = NetworkStatusState() + + monitor.pathUpdateHandler = { path in + guard state.markCompleted() else { return } + monitor.cancel() + cont.resume(returning: Self.payload(from: path)) + } + + monitor.start(queue: queue) + + queue.asyncAfter(deadline: .now() + .milliseconds(timeoutMs)) { + guard state.markCompleted() else { return } + monitor.cancel() + cont.resume(returning: Self.fallbackPayload()) + } + } + } + + private static func payload(from path: NWPath) -> OpenClawNetworkStatusPayload { + let status: OpenClawNetworkPathStatus = switch path.status { + case .satisfied: .satisfied + case .requiresConnection: .requiresConnection + case .unsatisfied: .unsatisfied + @unknown default: .unsatisfied + } + + var interfaces: [OpenClawNetworkInterfaceType] = [] + if path.usesInterfaceType(.wifi) { interfaces.append(.wifi) } + if path.usesInterfaceType(.cellular) { interfaces.append(.cellular) } + if path.usesInterfaceType(.wiredEthernet) { interfaces.append(.wired) } + if interfaces.isEmpty { interfaces.append(.other) } + + return OpenClawNetworkStatusPayload( + status: status, + isExpensive: path.isExpensive, + isConstrained: path.isConstrained, + interfaces: interfaces) + } + + private static func fallbackPayload() -> OpenClawNetworkStatusPayload { + OpenClawNetworkStatusPayload( + status: .unsatisfied, + isExpensive: false, + isConstrained: false, + interfaces: [.other]) + } +} + +private final class NetworkStatusState: @unchecked Sendable { + private let lock = NSLock() + private var completed = false + + func markCompleted() -> Bool { + self.lock.lock() + defer { self.lock.unlock() } + if self.completed { return false } + self.completed = true + return true + } +} diff --git a/apps/ios/Sources/Gateway/GatewayConnectionController.swift b/apps/ios/Sources/Gateway/GatewayConnectionController.swift index 65d099c0106..71686e00070 100644 --- a/apps/ios/Sources/Gateway/GatewayConnectionController.swift +++ b/apps/ios/Sources/Gateway/GatewayConnectionController.swift @@ -1,8 +1,16 @@ -import OpenClawKit +import AVFoundation +import Contacts +import CoreLocation +import CoreMotion import Darwin +import EventKit import Foundation +import OpenClawKit import Network import Observation +import Photos +import ReplayKit +import Speech import SwiftUI import UIKit @@ -282,7 +290,7 @@ final class GatewayConnectionController { scopes: [], caps: self.currentCaps(), commands: self.currentCommands(), - permissions: [:], + permissions: self.currentPermissions(), clientId: "openclaw-ios", clientMode: "node", clientDisplayName: displayName) @@ -320,6 +328,15 @@ final class GatewayConnectionController { let locationMode = OpenClawLocationMode(rawValue: locationModeRaw) ?? .off if locationMode != .off { caps.append(OpenClawCapability.location.rawValue) } + caps.append(OpenClawCapability.device.rawValue) + caps.append(OpenClawCapability.photos.rawValue) + caps.append(OpenClawCapability.contacts.rawValue) + caps.append(OpenClawCapability.calendar.rawValue) + caps.append(OpenClawCapability.reminders.rawValue) + if Self.motionAvailable() { + caps.append(OpenClawCapability.motion.rawValue) + } + return caps } @@ -335,10 +352,6 @@ final class GatewayConnectionController { OpenClawCanvasA2UICommand.reset.rawValue, OpenClawScreenCommand.record.rawValue, OpenClawSystemCommand.notify.rawValue, - OpenClawSystemCommand.which.rawValue, - OpenClawSystemCommand.run.rawValue, - OpenClawSystemCommand.execApprovalsGet.rawValue, - OpenClawSystemCommand.execApprovalsSet.rawValue, ] let caps = Set(self.currentCaps()) @@ -350,10 +363,70 @@ final class GatewayConnectionController { if caps.contains(OpenClawCapability.location.rawValue) { commands.append(OpenClawLocationCommand.get.rawValue) } + if caps.contains(OpenClawCapability.device.rawValue) { + commands.append(OpenClawDeviceCommand.status.rawValue) + commands.append(OpenClawDeviceCommand.info.rawValue) + } + if caps.contains(OpenClawCapability.photos.rawValue) { + commands.append(OpenClawPhotosCommand.latest.rawValue) + } + if caps.contains(OpenClawCapability.contacts.rawValue) { + commands.append(OpenClawContactsCommand.search.rawValue) + } + if caps.contains(OpenClawCapability.calendar.rawValue) { + commands.append(OpenClawCalendarCommand.events.rawValue) + } + if caps.contains(OpenClawCapability.reminders.rawValue) { + commands.append(OpenClawRemindersCommand.list.rawValue) + } + if caps.contains(OpenClawCapability.motion.rawValue) { + commands.append(OpenClawMotionCommand.activity.rawValue) + commands.append(OpenClawMotionCommand.pedometer.rawValue) + } return commands } + private func currentPermissions() -> [String: Bool] { + var permissions: [String: Bool] = [:] + permissions["camera"] = AVCaptureDevice.authorizationStatus(for: .video) == .authorized + permissions["microphone"] = AVCaptureDevice.authorizationStatus(for: .audio) == .authorized + permissions["speechRecognition"] = SFSpeechRecognizer.authorizationStatus() == .authorized + permissions["location"] = Self.isLocationAuthorized( + status: CLLocationManager().authorizationStatus) + && CLLocationManager.locationServicesEnabled() + permissions["screenRecording"] = RPScreenRecorder.shared().isAvailable + + let photoStatus = PHPhotoLibrary.authorizationStatus(for: .readWrite) + permissions["photos"] = photoStatus == .authorized || photoStatus == .limited + permissions["contacts"] = CNContactStore.authorizationStatus(for: .contacts) == .authorized + + let calendarStatus = EKEventStore.authorizationStatus(for: .event) + permissions["calendar"] = calendarStatus == .authorized || calendarStatus == .fullAccess + let remindersStatus = EKEventStore.authorizationStatus(for: .reminder) + permissions["reminders"] = remindersStatus == .authorized || remindersStatus == .fullAccess + + let motionStatus = CMMotionActivityManager.authorizationStatus() + let pedometerStatus = CMPedometer.authorizationStatus() + permissions["motion"] = + motionStatus == .authorized || pedometerStatus == .authorized + + return permissions + } + + private static func isLocationAuthorized(status: CLAuthorizationStatus) -> Bool { + switch status { + case .authorizedAlways, .authorizedWhenInUse, .authorized: + return true + default: + return false + } + } + + private static func motionAvailable() -> Bool { + CMMotionActivityManager.isActivityAvailable() || CMPedometer.isStepCountingAvailable() + } + private func platformString() -> String { let v = ProcessInfo.processInfo.operatingSystemVersion let name = switch UIDevice.current.userInterfaceIdiom { @@ -407,6 +480,10 @@ extension GatewayConnectionController { self.currentCommands() } + func _test_currentPermissions() -> [String: Bool] { + self.currentPermissions() + } + func _test_platformString() -> String { self.platformString() } diff --git a/apps/ios/Sources/Info.plist b/apps/ios/Sources/Info.plist index 00c26b9e9a2..7fe8f61bf90 100644 --- a/apps/ios/Sources/Info.plist +++ b/apps/ios/Sources/Info.plist @@ -41,6 +41,16 @@ OpenClaw uses your location when you allow location sharing. NSMicrophoneUsageDescription OpenClaw needs microphone access for voice wake. + NSPhotoLibraryUsageDescription + OpenClaw can read recent photos when requested via the gateway. + NSContactsUsageDescription + OpenClaw can read your contacts when requested via the gateway. + NSCalendarsUsageDescription + OpenClaw can read your calendar events when requested via the gateway. + NSRemindersUsageDescription + OpenClaw can read your reminders when requested via the gateway. + NSMotionUsageDescription + OpenClaw can read motion activity and pedometer data when requested via the gateway. NSSpeechRecognitionUsageDescription OpenClaw uses on-device speech recognition for voice wake. UIApplicationSceneManifest diff --git a/apps/ios/Sources/Media/PhotoLibraryService.swift b/apps/ios/Sources/Media/PhotoLibraryService.swift new file mode 100644 index 00000000000..c8655c11c22 --- /dev/null +++ b/apps/ios/Sources/Media/PhotoLibraryService.swift @@ -0,0 +1,103 @@ +import Foundation +import Photos +import OpenClawKit +import UIKit + +final class PhotoLibraryService: PhotosServicing { + func latest(params: OpenClawPhotosLatestParams) async throws -> OpenClawPhotosLatestPayload { + let status = await Self.ensureAuthorization() + guard status == .authorized || status == .limited else { + throw NSError(domain: "Photos", code: 1, userInfo: [ + NSLocalizedDescriptionKey: "PHOTOS_PERMISSION_REQUIRED: grant Photos permission", + ]) + } + + let limit = max(1, min(params.limit ?? 1, 20)) + let fetchOptions = PHFetchOptions() + fetchOptions.fetchLimit = limit + fetchOptions.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)] + let assets = PHAsset.fetchAssets(with: .image, options: fetchOptions) + + var results: [OpenClawPhotoPayload] = [] + let maxWidth = params.maxWidth.flatMap { $0 > 0 ? $0 : nil } ?? 1600 + let quality = params.quality.map { max(0.1, min(1.0, $0)) } ?? 0.85 + let formatter = ISO8601DateFormatter() + + assets.enumerateObjects { asset, _, stop in + if results.count >= limit { stop.pointee = true; return } + if let payload = try? Self.renderAsset( + asset, + maxWidth: maxWidth, + quality: quality, + formatter: formatter) + { + results.append(payload) + } + } + + return OpenClawPhotosLatestPayload(photos: results) + } + + private static func ensureAuthorization() async -> PHAuthorizationStatus { + let current = PHPhotoLibrary.authorizationStatus(for: .readWrite) + if current == .notDetermined { + return await withCheckedContinuation { cont in + PHPhotoLibrary.requestAuthorization(for: .readWrite) { status in + cont.resume(returning: status) + } + } + } + return current + } + + private static func renderAsset( + _ asset: PHAsset, + maxWidth: Int, + quality: Double, + formatter: ISO8601DateFormatter) throws -> OpenClawPhotoPayload + { + let manager = PHImageManager.default() + let options = PHImageRequestOptions() + options.isSynchronous = true + options.isNetworkAccessAllowed = true + options.deliveryMode = .highQualityFormat + + let targetSize: CGSize = { + guard maxWidth > 0 else { return PHImageManagerMaximumSize } + let aspect = CGFloat(asset.pixelHeight) / CGFloat(max(1, asset.pixelWidth)) + let width = CGFloat(maxWidth) + return CGSize(width: width, height: width * aspect) + }() + + var image: UIImage? + manager.requestImage( + for: asset, + targetSize: targetSize, + contentMode: .aspectFit, + options: options) + { result, _ in + image = result + } + + guard let image else { + throw NSError(domain: "Photos", code: 2, userInfo: [ + NSLocalizedDescriptionKey: "photo load failed", + ]) + } + + let jpeg = image.jpegData(compressionQuality: quality) + guard let data = jpeg else { + throw NSError(domain: "Photos", code: 3, userInfo: [ + NSLocalizedDescriptionKey: "photo encode failed", + ]) + } + + let created = asset.creationDate.map { formatter.string(from: $0) } + return OpenClawPhotoPayload( + format: "jpeg", + base64: data.base64EncodedString(), + width: Int(image.size.width), + height: Int(image.size.height), + createdAt: created) + } +} diff --git a/apps/ios/Sources/Model/NodeAppModel.swift b/apps/ios/Sources/Model/NodeAppModel.swift index 963318a8a2d..ef51f58790b 100644 --- a/apps/ios/Sources/Model/NodeAppModel.swift +++ b/apps/ios/Sources/Model/NodeAppModel.swift @@ -3,6 +3,7 @@ import Network import Observation import SwiftUI import UIKit +import UserNotifications @MainActor @Observable @@ -15,9 +16,9 @@ final class NodeAppModel { } var isBackgrounded: Bool = false - let screen = ScreenController() - let camera = CameraController() - private let screenRecorder = ScreenRecordService() + let screen: ScreenController + private let camera: any CameraServicing + private let screenRecorder: any ScreenRecordingServicing var gatewayStatusText: String = "Offline" var gatewayServerName: String? var gatewayRemoteAddress: String? @@ -29,9 +30,16 @@ final class NodeAppModel { private var gatewayTask: Task? private var voiceWakeSyncTask: Task? @ObservationIgnored private var cameraHUDDismissTask: Task? + private let notificationCenter: NotificationCentering let voiceWake = VoiceWakeManager() let talkMode = TalkModeManager() - private let locationService = LocationService() + private let locationService: any LocationServicing + private let deviceStatusService: any DeviceStatusServicing + private let photosService: any PhotosServicing + private let contactsService: any ContactsServicing + private let calendarService: any CalendarServicing + private let remindersService: any RemindersServicing + private let motionService: any MotionServicing private var lastAutoA2uiURL: String? private var gatewayConnected = false @@ -42,7 +50,31 @@ final class NodeAppModel { var cameraFlashNonce: Int = 0 var screenRecordActive: Bool = false - init() { + init( + screen: ScreenController = ScreenController(), + camera: any CameraServicing = CameraController(), + screenRecorder: any ScreenRecordingServicing = ScreenRecordService(), + locationService: any LocationServicing = LocationService(), + notificationCenter: NotificationCentering = LiveNotificationCenter(), + deviceStatusService: any DeviceStatusServicing = DeviceStatusService(), + photosService: any PhotosServicing = PhotoLibraryService(), + contactsService: any ContactsServicing = ContactsService(), + calendarService: any CalendarServicing = CalendarService(), + remindersService: any RemindersServicing = RemindersService(), + motionService: any MotionServicing = MotionService()) + { + self.screen = screen + self.camera = camera + self.screenRecorder = screenRecorder + self.locationService = locationService + self.notificationCenter = notificationCenter + self.deviceStatusService = deviceStatusService + self.photosService = photosService + self.contactsService = contactsService + self.calendarService = calendarService + self.remindersService = remindersService + self.motionService = motionService + self.voiceWake.configure { [weak self] cmd in guard let self else { return } let sessionKey = await MainActor.run { self.mainSessionKey } @@ -542,6 +574,22 @@ final class NodeAppModel { return try await self.handleCameraInvoke(req) case OpenClawScreenCommand.record.rawValue: return try await self.handleScreenRecordInvoke(req) + case OpenClawSystemCommand.notify.rawValue: + return try await self.handleSystemNotify(req) + case OpenClawDeviceCommand.status.rawValue, + OpenClawDeviceCommand.info.rawValue: + return try await self.handleDeviceInvoke(req) + case OpenClawPhotosCommand.latest.rawValue: + return try await self.handlePhotosInvoke(req) + case OpenClawContactsCommand.search.rawValue: + return try await self.handleContactsInvoke(req) + case OpenClawCalendarCommand.events.rawValue: + return try await self.handleCalendarInvoke(req) + case OpenClawRemindersCommand.list.rawValue: + return try await self.handleRemindersInvoke(req) + case OpenClawMotionCommand.activity.rawValue, + OpenClawMotionCommand.pedometer.rawValue: + return try await self.handleMotionInvoke(req) default: return BridgeInvokeResponse( id: req.id, @@ -626,6 +674,7 @@ final class NodeAppModel { private func handleCanvasInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { switch req.command { case OpenClawCanvasCommand.present.rawValue: + // iOS ignores placement hints; canvas always fills the screen. let params = (try? Self.decodeParams(OpenClawCanvasPresentParams.self, from: req.paramsJSON)) ?? OpenClawCanvasPresentParams() let url = params.url?.trimmingCharacters(in: .whitespacesAndNewlines) ?? "" @@ -636,6 +685,7 @@ final class NodeAppModel { } return BridgeInvokeResponse(id: req.id, ok: true) case OpenClawCanvasCommand.hide.rawValue: + self.screen.showDefaultCanvas() return BridgeInvokeResponse(id: req.id, ok: true) case OpenClawCanvasCommand.navigate.rawValue: let params = try Self.decodeParams(OpenClawCanvasNavigateParams.self, from: req.paramsJSON) @@ -859,6 +909,112 @@ final class NodeAppModel { return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: payload) } + private func handleSystemNotify(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + let params = try Self.decodeParams(OpenClawSystemNotifyParams.self, from: req.paramsJSON) + let title = params.title.trimmingCharacters(in: .whitespacesAndNewlines) + let body = params.body.trimmingCharacters(in: .whitespacesAndNewlines) + if title.isEmpty, body.isEmpty { + return BridgeInvokeResponse( + id: req.id, + ok: false, + error: OpenClawNodeError(code: .invalidRequest, message: "INVALID_REQUEST: empty notification")) + } + + let status = await self.notificationCenter.authorizationStatus() + if status == .notDetermined { + _ = try await self.notificationCenter.requestAuthorization(options: [.alert, .sound, .badge]) + } + let finalStatus = await self.notificationCenter.authorizationStatus() + guard finalStatus == .authorized || finalStatus == .provisional || finalStatus == .ephemeral else { + return BridgeInvokeResponse( + id: req.id, + ok: false, + error: OpenClawNodeError(code: .unavailable, message: "NOT_AUTHORIZED: notifications")) + } + + let content = UNMutableNotificationContent() + content.title = title + content.body = body + let request = UNNotificationRequest( + identifier: UUID().uuidString, + content: content, + trigger: nil) + try await self.notificationCenter.add(request) + return BridgeInvokeResponse(id: req.id, ok: true) + } + + private func handleDeviceInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + switch req.command { + case OpenClawDeviceCommand.status.rawValue: + let payload = try await self.deviceStatusService.status() + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + case OpenClawDeviceCommand.info.rawValue: + let payload = self.deviceStatusService.info() + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + default: + return BridgeInvokeResponse( + id: req.id, + ok: false, + error: OpenClawNodeError(code: .invalidRequest, message: "INVALID_REQUEST: unknown command")) + } + } + + private func handlePhotosInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + let params = (try? Self.decodeParams(OpenClawPhotosLatestParams.self, from: req.paramsJSON)) ?? + OpenClawPhotosLatestParams() + let payload = try await self.photosService.latest(params: params) + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + } + + private func handleContactsInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + let params = (try? Self.decodeParams(OpenClawContactsSearchParams.self, from: req.paramsJSON)) ?? + OpenClawContactsSearchParams() + let payload = try await self.contactsService.search(params: params) + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + } + + private func handleCalendarInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + let params = (try? Self.decodeParams(OpenClawCalendarEventsParams.self, from: req.paramsJSON)) ?? + OpenClawCalendarEventsParams() + let payload = try await self.calendarService.events(params: params) + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + } + + private func handleRemindersInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + let params = (try? Self.decodeParams(OpenClawRemindersListParams.self, from: req.paramsJSON)) ?? + OpenClawRemindersListParams() + let payload = try await self.remindersService.list(params: params) + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + } + + private func handleMotionInvoke(_ req: BridgeInvokeRequest) async throws -> BridgeInvokeResponse { + switch req.command { + case OpenClawMotionCommand.activity.rawValue: + let params = (try? Self.decodeParams(OpenClawMotionActivityParams.self, from: req.paramsJSON)) ?? + OpenClawMotionActivityParams() + let payload = try await self.motionService.activities(params: params) + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + case OpenClawMotionCommand.pedometer.rawValue: + let params = (try? Self.decodeParams(OpenClawPedometerParams.self, from: req.paramsJSON)) ?? + OpenClawPedometerParams() + let payload = try await self.motionService.pedometer(params: params) + let json = try Self.encodePayload(payload) + return BridgeInvokeResponse(id: req.id, ok: true, payloadJSON: json) + default: + return BridgeInvokeResponse( + id: req.id, + ok: false, + error: OpenClawNodeError(code: .invalidRequest, message: "INVALID_REQUEST: unknown command")) + } + } + } private extension NodeAppModel { diff --git a/apps/ios/Sources/Motion/MotionService.swift b/apps/ios/Sources/Motion/MotionService.swift new file mode 100644 index 00000000000..c5ac7aaf953 --- /dev/null +++ b/apps/ios/Sources/Motion/MotionService.swift @@ -0,0 +1,88 @@ +import CoreMotion +import Foundation +import OpenClawKit + +final class MotionService: MotionServicing { + func activities(params: OpenClawMotionActivityParams) async throws -> OpenClawMotionActivityPayload { + guard CMMotionActivityManager.isActivityAvailable() else { + throw NSError(domain: "Motion", code: 1, userInfo: [ + NSLocalizedDescriptionKey: "MOTION_UNAVAILABLE: activity not supported on this device", + ]) + } + + let (start, end) = Self.resolveRange(startISO: params.startISO, endISO: params.endISO) + let limit = max(1, min(params.limit ?? 200, 1000)) + + let manager = CMMotionActivityManager() + let mapped = try await withCheckedThrowingContinuation { (cont: CheckedContinuation<[OpenClawMotionActivityEntry], Error>) in + manager.queryActivityStarting(from: start, to: end, to: OperationQueue()) { activity, error in + if let error { + cont.resume(throwing: error) + } else { + let formatter = ISO8601DateFormatter() + let sliced = Array((activity ?? []).suffix(limit)) + let entries = sliced.map { entry in + OpenClawMotionActivityEntry( + startISO: formatter.string(from: entry.startDate), + endISO: formatter.string(from: end), + confidence: Self.confidenceString(entry.confidence), + isWalking: entry.walking, + isRunning: entry.running, + isCycling: entry.cycling, + isAutomotive: entry.automotive, + isStationary: entry.stationary, + isUnknown: entry.unknown) + } + cont.resume(returning: entries) + } + } + } + + return OpenClawMotionActivityPayload(activities: mapped) + } + + func pedometer(params: OpenClawPedometerParams) async throws -> OpenClawPedometerPayload { + guard CMPedometer.isStepCountingAvailable() else { + throw NSError(domain: "Motion", code: 2, userInfo: [ + NSLocalizedDescriptionKey: "PEDOMETER_UNAVAILABLE: step counting not supported", + ]) + } + + let (start, end) = Self.resolveRange(startISO: params.startISO, endISO: params.endISO) + let pedometer = CMPedometer() + let payload = try await withCheckedThrowingContinuation { (cont: CheckedContinuation) in + pedometer.queryPedometerData(from: start, to: end) { data, error in + if let error { + cont.resume(throwing: error) + } else { + let formatter = ISO8601DateFormatter() + let payload = OpenClawPedometerPayload( + startISO: formatter.string(from: start), + endISO: formatter.string(from: end), + steps: data?.numberOfSteps.intValue, + distanceMeters: data?.distance?.doubleValue, + floorsAscended: data?.floorsAscended?.intValue, + floorsDescended: data?.floorsDescended?.intValue) + cont.resume(returning: payload) + } + } + } + return payload + } + + private static func resolveRange(startISO: String?, endISO: String?) -> (Date, Date) { + let formatter = ISO8601DateFormatter() + let start = startISO.flatMap { formatter.date(from: $0) } ?? Calendar.current.startOfDay(for: Date()) + let end = endISO.flatMap { formatter.date(from: $0) } ?? Date() + return (start, end) + } + + private static func confidenceString(_ confidence: CMMotionActivityConfidence) -> String { + switch confidence { + case .low: "low" + case .medium: "medium" + case .high: "high" + @unknown default: "unknown" + } + } +} diff --git a/apps/ios/Sources/Reminders/RemindersService.swift b/apps/ios/Sources/Reminders/RemindersService.swift new file mode 100644 index 00000000000..803c0dbac2e --- /dev/null +++ b/apps/ios/Sources/Reminders/RemindersService.swift @@ -0,0 +1,70 @@ +import EventKit +import Foundation +import OpenClawKit + +final class RemindersService: RemindersServicing { + func list(params: OpenClawRemindersListParams) async throws -> OpenClawRemindersListPayload { + let store = EKEventStore() + let status = EKEventStore.authorizationStatus(for: .reminder) + let authorized = await Self.ensureAuthorization(store: store, status: status) + guard authorized else { + throw NSError(domain: "Reminders", code: 1, userInfo: [ + NSLocalizedDescriptionKey: "REMINDERS_PERMISSION_REQUIRED: grant Reminders permission", + ]) + } + + let limit = max(1, min(params.limit ?? 50, 500)) + let statusFilter = params.status ?? .incomplete + + let predicate = store.predicateForReminders(in: nil) + let payload = try await withCheckedThrowingContinuation { (cont: CheckedContinuation<[OpenClawReminderPayload], Error>) in + store.fetchReminders(matching: predicate) { items in + let formatter = ISO8601DateFormatter() + let filtered = (items ?? []).filter { reminder in + switch statusFilter { + case .all: + return true + case .completed: + return reminder.isCompleted + case .incomplete: + return !reminder.isCompleted + } + } + let selected = Array(filtered.prefix(limit)) + let payload = selected.map { reminder in + let due = reminder.dueDateComponents.flatMap { Calendar.current.date(from: $0) } + return OpenClawReminderPayload( + identifier: reminder.calendarItemIdentifier, + title: reminder.title, + dueISO: due.map { formatter.string(from: $0) }, + completed: reminder.isCompleted, + listName: reminder.calendar.title) + } + cont.resume(returning: payload) + } + } + + return OpenClawRemindersListPayload(reminders: payload) + } + + private static func ensureAuthorization(store: EKEventStore, status: EKAuthorizationStatus) async -> Bool { + switch status { + case .authorized: + return true + case .notDetermined: + return await withCheckedContinuation { cont in + store.requestAccess(to: .reminder) { granted, _ in + cont.resume(returning: granted) + } + } + case .restricted, .denied: + return false + case .fullAccess: + return true + case .writeOnly: + return false + @unknown default: + return false + } + } +} diff --git a/apps/ios/Sources/Services/NodeServiceProtocols.swift b/apps/ios/Sources/Services/NodeServiceProtocols.swift new file mode 100644 index 00000000000..728bca9553a --- /dev/null +++ b/apps/ios/Sources/Services/NodeServiceProtocols.swift @@ -0,0 +1,61 @@ +import CoreLocation +import Foundation +import OpenClawKit +import UIKit + +protocol CameraServicing: Sendable { + func listDevices() async -> [CameraController.CameraDeviceInfo] + func snap(params: OpenClawCameraSnapParams) async throws -> (format: String, base64: String, width: Int, height: Int) + func clip(params: OpenClawCameraClipParams) async throws -> (format: String, base64: String, durationMs: Int, hasAudio: Bool) +} + +protocol ScreenRecordingServicing: Sendable { + func record( + screenIndex: Int?, + durationMs: Int?, + fps: Double?, + includeAudio: Bool?, + outPath: String?) async throws -> String +} + +@MainActor +protocol LocationServicing: Sendable { + func authorizationStatus() -> CLAuthorizationStatus + func accuracyAuthorization() -> CLAccuracyAuthorization + func ensureAuthorization(mode: OpenClawLocationMode) async -> CLAuthorizationStatus + func currentLocation( + params: OpenClawLocationGetParams, + desiredAccuracy: OpenClawLocationAccuracy, + maxAgeMs: Int?, + timeoutMs: Int?) async throws -> CLLocation +} + +protocol DeviceStatusServicing: Sendable { + func status() async throws -> OpenClawDeviceStatusPayload + func info() -> OpenClawDeviceInfoPayload +} + +protocol PhotosServicing: Sendable { + func latest(params: OpenClawPhotosLatestParams) async throws -> OpenClawPhotosLatestPayload +} + +protocol ContactsServicing: Sendable { + func search(params: OpenClawContactsSearchParams) async throws -> OpenClawContactsSearchPayload +} + +protocol CalendarServicing: Sendable { + func events(params: OpenClawCalendarEventsParams) async throws -> OpenClawCalendarEventsPayload +} + +protocol RemindersServicing: Sendable { + func list(params: OpenClawRemindersListParams) async throws -> OpenClawRemindersListPayload +} + +protocol MotionServicing: Sendable { + func activities(params: OpenClawMotionActivityParams) async throws -> OpenClawMotionActivityPayload + func pedometer(params: OpenClawPedometerParams) async throws -> OpenClawPedometerPayload +} + +extension CameraController: CameraServicing {} +extension ScreenRecordService: ScreenRecordingServicing {} +extension LocationService: LocationServicing {} diff --git a/apps/ios/Sources/Services/NotificationService.swift b/apps/ios/Sources/Services/NotificationService.swift new file mode 100644 index 00000000000..348e93edc61 --- /dev/null +++ b/apps/ios/Sources/Services/NotificationService.swift @@ -0,0 +1,58 @@ +import Foundation +import UserNotifications + +enum NotificationAuthorizationStatus: Sendable { + case notDetermined + case denied + case authorized + case provisional + case ephemeral +} + +protocol NotificationCentering: Sendable { + func authorizationStatus() async -> NotificationAuthorizationStatus + func requestAuthorization(options: UNAuthorizationOptions) async throws -> Bool + func add(_ request: UNNotificationRequest) async throws +} + +struct LiveNotificationCenter: NotificationCentering, @unchecked Sendable { + private let center: UNUserNotificationCenter + + init(center: UNUserNotificationCenter = .current()) { + self.center = center + } + + func authorizationStatus() async -> NotificationAuthorizationStatus { + let settings = await self.center.notificationSettings() + return switch settings.authorizationStatus { + case .authorized: + .authorized + case .provisional: + .provisional + case .ephemeral: + .ephemeral + case .denied: + .denied + case .notDetermined: + .notDetermined + @unknown default: + .denied + } + } + + func requestAuthorization(options: UNAuthorizationOptions) async throws -> Bool { + try await self.center.requestAuthorization(options: options) + } + + func add(_ request: UNNotificationRequest) async throws { + try await withCheckedThrowingContinuation { (cont: CheckedContinuation) in + self.center.add(request) { error in + if let error { + cont.resume(throwing: error) + } else { + cont.resume(returning: ()) + } + } + } + } +} diff --git a/apps/ios/SwiftSources.input.xcfilelist b/apps/ios/SwiftSources.input.xcfilelist index 4952019c773..81e869e2c64 100644 --- a/apps/ios/SwiftSources.input.xcfilelist +++ b/apps/ios/SwiftSources.input.xcfilelist @@ -6,16 +6,25 @@ Sources/Gateway/KeychainStore.swift Sources/Camera/CameraController.swift Sources/Chat/ChatSheet.swift Sources/Chat/IOSGatewayChatTransport.swift +Sources/Contacts/ContactsService.swift +Sources/Device/DeviceStatusService.swift +Sources/Device/NetworkStatusService.swift Sources/OpenClawApp.swift Sources/Location/LocationService.swift +Sources/Media/PhotoLibraryService.swift +Sources/Motion/MotionService.swift Sources/Model/NodeAppModel.swift Sources/RootCanvas.swift Sources/RootTabs.swift +Sources/Services/NodeServiceProtocols.swift +Sources/Services/NotificationService.swift Sources/Screen/ScreenController.swift Sources/Screen/ScreenRecordService.swift Sources/Screen/ScreenTab.swift Sources/Screen/ScreenWebView.swift Sources/SessionKey.swift +Sources/Calendar/CalendarService.swift +Sources/Reminders/RemindersService.swift Sources/Settings/SettingsNetworkingHelpers.swift Sources/Settings/SettingsTab.swift Sources/Settings/VoiceWakeWordsSettingsView.swift @@ -40,6 +49,7 @@ Sources/Voice/VoiceWakePreferences.swift ../shared/OpenClawKit/Sources/OpenClawKit/BonjourEscapes.swift ../shared/OpenClawKit/Sources/OpenClawKit/BonjourTypes.swift ../shared/OpenClawKit/Sources/OpenClawKit/BridgeFrames.swift +../shared/OpenClawKit/Sources/OpenClawKit/CalendarCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/CameraCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/CanvasA2UIAction.swift ../shared/OpenClawKit/Sources/OpenClawKit/CanvasA2UICommands.swift @@ -47,10 +57,15 @@ Sources/Voice/VoiceWakePreferences.swift ../shared/OpenClawKit/Sources/OpenClawKit/CanvasCommandParams.swift ../shared/OpenClawKit/Sources/OpenClawKit/CanvasCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/Capabilities.swift +../shared/OpenClawKit/Sources/OpenClawKit/ContactsCommands.swift +../shared/OpenClawKit/Sources/OpenClawKit/DeviceCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/OpenClawKitResources.swift ../shared/OpenClawKit/Sources/OpenClawKit/DeepLinks.swift ../shared/OpenClawKit/Sources/OpenClawKit/JPEGTranscoder.swift +../shared/OpenClawKit/Sources/OpenClawKit/MotionCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/NodeError.swift +../shared/OpenClawKit/Sources/OpenClawKit/PhotosCommands.swift +../shared/OpenClawKit/Sources/OpenClawKit/RemindersCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/ScreenCommands.swift ../shared/OpenClawKit/Sources/OpenClawKit/StoragePaths.swift ../shared/OpenClawKit/Sources/OpenClawKit/SystemCommands.swift diff --git a/apps/ios/Tests/GatewayConnectionControllerTests.swift b/apps/ios/Tests/GatewayConnectionControllerTests.swift index 0d3bdbba0ee..3941d581984 100644 --- a/apps/ios/Tests/GatewayConnectionControllerTests.swift +++ b/apps/ios/Tests/GatewayConnectionControllerTests.swift @@ -61,6 +61,11 @@ private func withUserDefaults(_ updates: [String: Any?], _ body: () throws -> #expect(caps.contains(OpenClawCapability.camera.rawValue)) #expect(caps.contains(OpenClawCapability.location.rawValue)) #expect(caps.contains(OpenClawCapability.voiceWake.rawValue)) + #expect(caps.contains(OpenClawCapability.device.rawValue)) + #expect(caps.contains(OpenClawCapability.photos.rawValue)) + #expect(caps.contains(OpenClawCapability.contacts.rawValue)) + #expect(caps.contains(OpenClawCapability.calendar.rawValue)) + #expect(caps.contains(OpenClawCapability.reminders.rawValue)) } } @@ -76,4 +81,40 @@ private func withUserDefaults(_ updates: [String: Any?], _ body: () throws -> #expect(commands.contains(OpenClawLocationCommand.get.rawValue)) } } + + @Test @MainActor func currentCommandsExcludeShellAndIncludeNotifyAndDevice() { + withUserDefaults([ + "node.instanceId": "ios-test", + ]) { + let appModel = NodeAppModel() + let controller = GatewayConnectionController(appModel: appModel, startDiscovery: false) + let commands = Set(controller._test_currentCommands()) + + #expect(commands.contains(OpenClawSystemCommand.notify.rawValue)) + #expect(!commands.contains(OpenClawSystemCommand.run.rawValue)) + #expect(!commands.contains(OpenClawSystemCommand.which.rawValue)) + #expect(!commands.contains(OpenClawSystemCommand.execApprovalsGet.rawValue)) + #expect(!commands.contains(OpenClawSystemCommand.execApprovalsSet.rawValue)) + + #expect(commands.contains(OpenClawDeviceCommand.status.rawValue)) + #expect(commands.contains(OpenClawDeviceCommand.info.rawValue)) + } + } + + @Test @MainActor func currentPermissionsIncludeExpectedKeys() { + let appModel = NodeAppModel() + let controller = GatewayConnectionController(appModel: appModel, startDiscovery: false) + let permissions = controller._test_currentPermissions() + let keys = Set(permissions.keys) + + #expect(keys.contains("camera")) + #expect(keys.contains("microphone")) + #expect(keys.contains("location")) + #expect(keys.contains("screenRecording")) + #expect(keys.contains("photos")) + #expect(keys.contains("contacts")) + #expect(keys.contains("calendar")) + #expect(keys.contains("reminders")) + #expect(keys.contains("motion")) + } } diff --git a/apps/ios/Tests/NodeAppModelInvokeTests.swift b/apps/ios/Tests/NodeAppModelInvokeTests.swift index 124059021d6..30ad207ec88 100644 --- a/apps/ios/Tests/NodeAppModelInvokeTests.swift +++ b/apps/ios/Tests/NodeAppModelInvokeTests.swift @@ -1,7 +1,9 @@ -import OpenClawKit +import CoreLocation import Foundation +import OpenClawKit import Testing import UIKit +import UserNotifications @testable import OpenClaw private func withUserDefaults(_ updates: [String: Any?], _ body: () throws -> T) rethrows -> T { @@ -29,6 +31,139 @@ private func withUserDefaults(_ updates: [String: Any?], _ body: () throws -> return try body() } +private final class TestNotificationCenter: NotificationCentering, @unchecked Sendable { + private(set) var requestAuthorizationCalls = 0 + private(set) var addedRequests: [UNNotificationRequest] = [] + private var status: NotificationAuthorizationStatus + + init(status: NotificationAuthorizationStatus) { + self.status = status + } + + func authorizationStatus() async -> NotificationAuthorizationStatus { + status + } + + func requestAuthorization(options: UNAuthorizationOptions) async throws -> Bool { + requestAuthorizationCalls += 1 + status = .authorized + return true + } + + func add(_ request: UNNotificationRequest) async throws { + addedRequests.append(request) + } +} + +private struct TestCameraService: CameraServicing { + func listDevices() async -> [CameraController.CameraDeviceInfo] { [] } + func snap(params: OpenClawCameraSnapParams) async throws -> (format: String, base64: String, width: Int, height: Int) { + ("jpeg", "dGVzdA==", 1, 1) + } + func clip(params: OpenClawCameraClipParams) async throws -> (format: String, base64: String, durationMs: Int, hasAudio: Bool) { + ("mp4", "dGVzdA==", 1000, true) + } +} + +private struct TestScreenRecorder: ScreenRecordingServicing { + func record( + screenIndex: Int?, + durationMs: Int?, + fps: Double?, + includeAudio: Bool?, + outPath: String?) async throws -> String + { + let url = FileManager.default.temporaryDirectory.appendingPathComponent("openclaw-screen-test.mp4") + FileManager.default.createFile(atPath: url.path, contents: Data()) + return url.path + } +} + +@MainActor +private struct TestLocationService: LocationServicing { + func authorizationStatus() -> CLAuthorizationStatus { .authorizedWhenInUse } + func accuracyAuthorization() -> CLAccuracyAuthorization { .fullAccuracy } + func ensureAuthorization(mode: OpenClawLocationMode) async -> CLAuthorizationStatus { .authorizedWhenInUse } + func currentLocation( + params: OpenClawLocationGetParams, + desiredAccuracy: OpenClawLocationAccuracy, + maxAgeMs: Int?, + timeoutMs: Int?) async throws -> CLLocation + { + CLLocation(latitude: 37.3349, longitude: -122.0090) + } +} + +private struct TestDeviceStatusService: DeviceStatusServicing { + let statusPayload: OpenClawDeviceStatusPayload + let infoPayload: OpenClawDeviceInfoPayload + + func status() async throws -> OpenClawDeviceStatusPayload { statusPayload } + func info() -> OpenClawDeviceInfoPayload { infoPayload } +} + +private struct TestPhotosService: PhotosServicing { + let payload: OpenClawPhotosLatestPayload + func latest(params: OpenClawPhotosLatestParams) async throws -> OpenClawPhotosLatestPayload { payload } +} + +private struct TestContactsService: ContactsServicing { + let payload: OpenClawContactsSearchPayload + func search(params: OpenClawContactsSearchParams) async throws -> OpenClawContactsSearchPayload { payload } +} + +private struct TestCalendarService: CalendarServicing { + let payload: OpenClawCalendarEventsPayload + func events(params: OpenClawCalendarEventsParams) async throws -> OpenClawCalendarEventsPayload { payload } +} + +private struct TestRemindersService: RemindersServicing { + let payload: OpenClawRemindersListPayload + func list(params: OpenClawRemindersListParams) async throws -> OpenClawRemindersListPayload { payload } +} + +private struct TestMotionService: MotionServicing { + let activityPayload: OpenClawMotionActivityPayload + let pedometerPayload: OpenClawPedometerPayload + + func activities(params: OpenClawMotionActivityParams) async throws -> OpenClawMotionActivityPayload { + activityPayload + } + + func pedometer(params: OpenClawPedometerParams) async throws -> OpenClawPedometerPayload { + pedometerPayload + } +} + +@MainActor +private func makeTestAppModel( + notificationCenter: NotificationCentering = TestNotificationCenter(status: .authorized), + deviceStatusService: DeviceStatusServicing, + photosService: PhotosServicing, + contactsService: ContactsServicing, + calendarService: CalendarServicing, + remindersService: RemindersServicing, + motionService: MotionServicing) -> NodeAppModel +{ + NodeAppModel( + screen: ScreenController(), + camera: TestCameraService(), + screenRecorder: TestScreenRecorder(), + locationService: TestLocationService(), + notificationCenter: notificationCenter, + deviceStatusService: deviceStatusService, + photosService: photosService, + contactsService: contactsService, + calendarService: calendarService, + remindersService: remindersService, + motionService: motionService) +} + +private func decodePayload(_ json: String?, as type: T.Type) throws -> T { + let data = try #require(json?.data(using: .utf8)) + return try JSONDecoder().decode(type, from: data) +} + @Suite(.serialized) struct NodeAppModelInvokeTests { @Test @MainActor func decodeParamsFailsWithoutJSON() { #expect(throws: Error.self) { @@ -124,6 +259,11 @@ private func withUserDefaults(_ updates: [String: Any?], _ body: () throws -> let payloadData = try #require(evalRes.payloadJSON?.data(using: .utf8)) let payload = try JSONSerialization.jsonObject(with: payloadData) as? [String: Any] #expect(payload?["result"] as? String == "2") + + let hide = BridgeInvokeRequest(id: "hide", command: OpenClawCanvasCommand.hide.rawValue) + let hideRes = await appModel._test_handleInvoke(hide) + #expect(hideRes.ok == true) + #expect(appModel.screen.urlString.isEmpty) } @Test @MainActor func handleInvokeA2UICommandsFailWhenHostMissing() async throws { @@ -155,6 +295,196 @@ private func withUserDefaults(_ updates: [String: Any?], _ body: () throws -> #expect(res.error?.code == .invalidRequest) } + @Test @MainActor func handleInvokeSystemNotifyCreatesNotificationRequest() async throws { + let notifier = TestNotificationCenter(status: .notDetermined) + let deviceStatus = TestDeviceStatusService( + statusPayload: OpenClawDeviceStatusPayload( + battery: OpenClawBatteryStatusPayload(level: 0.5, state: .charging, lowPowerModeEnabled: false), + thermal: OpenClawThermalStatusPayload(state: .nominal), + storage: OpenClawStorageStatusPayload(totalBytes: 100, freeBytes: 50, usedBytes: 50), + network: OpenClawNetworkStatusPayload( + status: .satisfied, + isExpensive: false, + isConstrained: false, + interfaces: [.wifi]), + uptimeSeconds: 10), + infoPayload: OpenClawDeviceInfoPayload( + deviceName: "Test", + modelIdentifier: "Test1,1", + systemName: "iOS", + systemVersion: "1.0", + appVersion: "dev", + appBuild: "0", + locale: "en-US")) + let appModel = makeTestAppModel( + notificationCenter: notifier, + deviceStatusService: deviceStatus, + photosService: TestPhotosService(payload: OpenClawPhotosLatestPayload(photos: [])), + contactsService: TestContactsService(payload: OpenClawContactsSearchPayload(contacts: [])), + calendarService: TestCalendarService(payload: OpenClawCalendarEventsPayload(events: [])), + remindersService: TestRemindersService(payload: OpenClawRemindersListPayload(reminders: [])), + motionService: TestMotionService( + activityPayload: OpenClawMotionActivityPayload(activities: []), + pedometerPayload: OpenClawPedometerPayload( + startISO: "2024-01-01T00:00:00Z", + endISO: "2024-01-01T01:00:00Z", + steps: nil, + distanceMeters: nil, + floorsAscended: nil, + floorsDescended: nil))) + + let params = OpenClawSystemNotifyParams(title: "Hello", body: "World") + let data = try JSONEncoder().encode(params) + let json = String(decoding: data, as: UTF8.self) + let req = BridgeInvokeRequest( + id: "notify", + command: OpenClawSystemCommand.notify.rawValue, + paramsJSON: json) + let res = await appModel._test_handleInvoke(req) + #expect(res.ok == true) + #expect(notifier.requestAuthorizationCalls == 1) + #expect(notifier.addedRequests.count == 1) + let request = try #require(notifier.addedRequests.first) + #expect(request.content.title == "Hello") + #expect(request.content.body == "World") + } + + @Test @MainActor func handleInvokeDeviceAndDataCommandsReturnPayloads() async throws { + let deviceStatusPayload = OpenClawDeviceStatusPayload( + battery: OpenClawBatteryStatusPayload(level: 0.25, state: .unplugged, lowPowerModeEnabled: false), + thermal: OpenClawThermalStatusPayload(state: .fair), + storage: OpenClawStorageStatusPayload(totalBytes: 200, freeBytes: 80, usedBytes: 120), + network: OpenClawNetworkStatusPayload( + status: .satisfied, + isExpensive: true, + isConstrained: false, + interfaces: [.cellular]), + uptimeSeconds: 42) + let deviceInfoPayload = OpenClawDeviceInfoPayload( + deviceName: "TestPhone", + modelIdentifier: "Test2,1", + systemName: "iOS", + systemVersion: "2.0", + appVersion: "dev", + appBuild: "1", + locale: "en-US") + let photosPayload = OpenClawPhotosLatestPayload( + photos: [ + OpenClawPhotoPayload(format: "jpeg", base64: "dGVzdA==", width: 1, height: 1, createdAt: nil), + ]) + let contactsPayload = OpenClawContactsSearchPayload( + contacts: [ + OpenClawContactPayload( + identifier: "c1", + displayName: "Jane Doe", + givenName: "Jane", + familyName: "Doe", + organizationName: "", + phoneNumbers: ["+1"], + emails: ["jane@example.com"]), + ]) + let calendarPayload = OpenClawCalendarEventsPayload( + events: [ + OpenClawCalendarEventPayload( + identifier: "e1", + title: "Standup", + startISO: "2024-01-01T00:00:00Z", + endISO: "2024-01-01T00:30:00Z", + isAllDay: false, + location: nil, + calendarTitle: "Work"), + ]) + let remindersPayload = OpenClawRemindersListPayload( + reminders: [ + OpenClawReminderPayload( + identifier: "r1", + title: "Ship build", + dueISO: "2024-01-02T00:00:00Z", + completed: false, + listName: "Inbox"), + ]) + let motionPayload = OpenClawMotionActivityPayload( + activities: [ + OpenClawMotionActivityEntry( + startISO: "2024-01-01T00:00:00Z", + endISO: "2024-01-01T00:10:00Z", + confidence: "high", + isWalking: true, + isRunning: false, + isCycling: false, + isAutomotive: false, + isStationary: false, + isUnknown: false), + ]) + let pedometerPayload = OpenClawPedometerPayload( + startISO: "2024-01-01T00:00:00Z", + endISO: "2024-01-01T01:00:00Z", + steps: 123, + distanceMeters: 456, + floorsAscended: 1, + floorsDescended: 2) + + let appModel = makeTestAppModel( + deviceStatusService: TestDeviceStatusService( + statusPayload: deviceStatusPayload, + infoPayload: deviceInfoPayload), + photosService: TestPhotosService(payload: photosPayload), + contactsService: TestContactsService(payload: contactsPayload), + calendarService: TestCalendarService(payload: calendarPayload), + remindersService: TestRemindersService(payload: remindersPayload), + motionService: TestMotionService( + activityPayload: motionPayload, + pedometerPayload: pedometerPayload)) + + let deviceStatusReq = BridgeInvokeRequest(id: "device", command: OpenClawDeviceCommand.status.rawValue) + let deviceStatusRes = await appModel._test_handleInvoke(deviceStatusReq) + #expect(deviceStatusRes.ok == true) + let decodedDeviceStatus = try decodePayload(deviceStatusRes.payloadJSON, as: OpenClawDeviceStatusPayload.self) + #expect(decodedDeviceStatus == deviceStatusPayload) + + let deviceInfoReq = BridgeInvokeRequest(id: "device-info", command: OpenClawDeviceCommand.info.rawValue) + let deviceInfoRes = await appModel._test_handleInvoke(deviceInfoReq) + #expect(deviceInfoRes.ok == true) + let decodedDeviceInfo = try decodePayload(deviceInfoRes.payloadJSON, as: OpenClawDeviceInfoPayload.self) + #expect(decodedDeviceInfo == deviceInfoPayload) + + let photosReq = BridgeInvokeRequest(id: "photos", command: OpenClawPhotosCommand.latest.rawValue) + let photosRes = await appModel._test_handleInvoke(photosReq) + #expect(photosRes.ok == true) + let decodedPhotos = try decodePayload(photosRes.payloadJSON, as: OpenClawPhotosLatestPayload.self) + #expect(decodedPhotos == photosPayload) + + let contactsReq = BridgeInvokeRequest(id: "contacts", command: OpenClawContactsCommand.search.rawValue) + let contactsRes = await appModel._test_handleInvoke(contactsReq) + #expect(contactsRes.ok == true) + let decodedContacts = try decodePayload(contactsRes.payloadJSON, as: OpenClawContactsSearchPayload.self) + #expect(decodedContacts == contactsPayload) + + let calendarReq = BridgeInvokeRequest(id: "calendar", command: OpenClawCalendarCommand.events.rawValue) + let calendarRes = await appModel._test_handleInvoke(calendarReq) + #expect(calendarRes.ok == true) + let decodedCalendar = try decodePayload(calendarRes.payloadJSON, as: OpenClawCalendarEventsPayload.self) + #expect(decodedCalendar == calendarPayload) + + let remindersReq = BridgeInvokeRequest(id: "reminders", command: OpenClawRemindersCommand.list.rawValue) + let remindersRes = await appModel._test_handleInvoke(remindersReq) + #expect(remindersRes.ok == true) + let decodedReminders = try decodePayload(remindersRes.payloadJSON, as: OpenClawRemindersListPayload.self) + #expect(decodedReminders == remindersPayload) + + let motionReq = BridgeInvokeRequest(id: "motion", command: OpenClawMotionCommand.activity.rawValue) + let motionRes = await appModel._test_handleInvoke(motionReq) + #expect(motionRes.ok == true) + let decodedMotion = try decodePayload(motionRes.payloadJSON, as: OpenClawMotionActivityPayload.self) + #expect(decodedMotion == motionPayload) + + let pedometerReq = BridgeInvokeRequest(id: "pedometer", command: OpenClawMotionCommand.pedometer.rawValue) + let pedometerRes = await appModel._test_handleInvoke(pedometerReq) + #expect(pedometerRes.ok == true) + let decodedPedometer = try decodePayload(pedometerRes.payloadJSON, as: OpenClawPedometerPayload.self) + #expect(decodedPedometer == pedometerPayload) + } + @Test @MainActor func handleDeepLinkSetsErrorWhenNotConnected() async { let appModel = NodeAppModel() let url = URL(string: "openclaw://agent?message=hello")! diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/CalendarCommands.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/CalendarCommands.swift new file mode 100644 index 00000000000..8d12e97760a --- /dev/null +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/CalendarCommands.swift @@ -0,0 +1,53 @@ +import Foundation + +public enum OpenClawCalendarCommand: String, Codable, Sendable { + case events = "calendar.events" +} + +public struct OpenClawCalendarEventsParams: Codable, Sendable, Equatable { + public var startISO: String? + public var endISO: String? + public var limit: Int? + + public init(startISO: String? = nil, endISO: String? = nil, limit: Int? = nil) { + self.startISO = startISO + self.endISO = endISO + self.limit = limit + } +} + +public struct OpenClawCalendarEventPayload: Codable, Sendable, Equatable { + public var identifier: String + public var title: String + public var startISO: String + public var endISO: String + public var isAllDay: Bool + public var location: String? + public var calendarTitle: String? + + public init( + identifier: String, + title: String, + startISO: String, + endISO: String, + isAllDay: Bool, + location: String? = nil, + calendarTitle: String? = nil) + { + self.identifier = identifier + self.title = title + self.startISO = startISO + self.endISO = endISO + self.isAllDay = isAllDay + self.location = location + self.calendarTitle = calendarTitle + } +} + +public struct OpenClawCalendarEventsPayload: Codable, Sendable, Equatable { + public var events: [OpenClawCalendarEventPayload] + + public init(events: [OpenClawCalendarEventPayload]) { + self.events = events + } +} diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/Capabilities.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/Capabilities.swift index 1cb820e7322..d5c5e3c439c 100644 --- a/apps/shared/OpenClawKit/Sources/OpenClawKit/Capabilities.swift +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/Capabilities.swift @@ -6,4 +6,10 @@ public enum OpenClawCapability: String, Codable, Sendable { case screen case voiceWake case location + case device + case photos + case contacts + case calendar + case reminders + case motion } diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/ContactsCommands.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/ContactsCommands.swift new file mode 100644 index 00000000000..4163ac12d06 --- /dev/null +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/ContactsCommands.swift @@ -0,0 +1,51 @@ +import Foundation + +public enum OpenClawContactsCommand: String, Codable, Sendable { + case search = "contacts.search" +} + +public struct OpenClawContactsSearchParams: Codable, Sendable, Equatable { + public var query: String? + public var limit: Int? + + public init(query: String? = nil, limit: Int? = nil) { + self.query = query + self.limit = limit + } +} + +public struct OpenClawContactPayload: Codable, Sendable, Equatable { + public var identifier: String + public var displayName: String + public var givenName: String + public var familyName: String + public var organizationName: String + public var phoneNumbers: [String] + public var emails: [String] + + public init( + identifier: String, + displayName: String, + givenName: String, + familyName: String, + organizationName: String, + phoneNumbers: [String], + emails: [String]) + { + self.identifier = identifier + self.displayName = displayName + self.givenName = givenName + self.familyName = familyName + self.organizationName = organizationName + self.phoneNumbers = phoneNumbers + self.emails = emails + } +} + +public struct OpenClawContactsSearchPayload: Codable, Sendable, Equatable { + public var contacts: [OpenClawContactPayload] + + public init(contacts: [OpenClawContactPayload]) { + self.contacts = contacts + } +} diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/DeviceCommands.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/DeviceCommands.swift new file mode 100644 index 00000000000..c58224b3f14 --- /dev/null +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/DeviceCommands.swift @@ -0,0 +1,134 @@ +import Foundation + +public enum OpenClawDeviceCommand: String, Codable, Sendable { + case status = "device.status" + case info = "device.info" +} + +public enum OpenClawBatteryState: String, Codable, Sendable { + case unknown + case unplugged + case charging + case full +} + +public enum OpenClawThermalState: String, Codable, Sendable { + case nominal + case fair + case serious + case critical +} + +public enum OpenClawNetworkPathStatus: String, Codable, Sendable { + case satisfied + case unsatisfied + case requiresConnection +} + +public enum OpenClawNetworkInterfaceType: String, Codable, Sendable { + case wifi + case cellular + case wired + case other +} + +public struct OpenClawBatteryStatusPayload: Codable, Sendable, Equatable { + public var level: Double? + public var state: OpenClawBatteryState + public var lowPowerModeEnabled: Bool + + public init(level: Double?, state: OpenClawBatteryState, lowPowerModeEnabled: Bool) { + self.level = level + self.state = state + self.lowPowerModeEnabled = lowPowerModeEnabled + } +} + +public struct OpenClawThermalStatusPayload: Codable, Sendable, Equatable { + public var state: OpenClawThermalState + + public init(state: OpenClawThermalState) { + self.state = state + } +} + +public struct OpenClawStorageStatusPayload: Codable, Sendable, Equatable { + public var totalBytes: Int64 + public var freeBytes: Int64 + public var usedBytes: Int64 + + public init(totalBytes: Int64, freeBytes: Int64, usedBytes: Int64) { + self.totalBytes = totalBytes + self.freeBytes = freeBytes + self.usedBytes = usedBytes + } +} + +public struct OpenClawNetworkStatusPayload: Codable, Sendable, Equatable { + public var status: OpenClawNetworkPathStatus + public var isExpensive: Bool + public var isConstrained: Bool + public var interfaces: [OpenClawNetworkInterfaceType] + + public init( + status: OpenClawNetworkPathStatus, + isExpensive: Bool, + isConstrained: Bool, + interfaces: [OpenClawNetworkInterfaceType]) + { + self.status = status + self.isExpensive = isExpensive + self.isConstrained = isConstrained + self.interfaces = interfaces + } +} + +public struct OpenClawDeviceStatusPayload: Codable, Sendable, Equatable { + public var battery: OpenClawBatteryStatusPayload + public var thermal: OpenClawThermalStatusPayload + public var storage: OpenClawStorageStatusPayload + public var network: OpenClawNetworkStatusPayload + public var uptimeSeconds: Double + + public init( + battery: OpenClawBatteryStatusPayload, + thermal: OpenClawThermalStatusPayload, + storage: OpenClawStorageStatusPayload, + network: OpenClawNetworkStatusPayload, + uptimeSeconds: Double) + { + self.battery = battery + self.thermal = thermal + self.storage = storage + self.network = network + self.uptimeSeconds = uptimeSeconds + } +} + +public struct OpenClawDeviceInfoPayload: Codable, Sendable, Equatable { + public var deviceName: String + public var modelIdentifier: String + public var systemName: String + public var systemVersion: String + public var appVersion: String + public var appBuild: String + public var locale: String + + public init( + deviceName: String, + modelIdentifier: String, + systemName: String, + systemVersion: String, + appVersion: String, + appBuild: String, + locale: String) + { + self.deviceName = deviceName + self.modelIdentifier = modelIdentifier + self.systemName = systemName + self.systemVersion = systemVersion + self.appVersion = appVersion + self.appBuild = appBuild + self.locale = locale + } +} diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/MotionCommands.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/MotionCommands.swift new file mode 100644 index 00000000000..ab487bfd00a --- /dev/null +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/MotionCommands.swift @@ -0,0 +1,95 @@ +import Foundation + +public enum OpenClawMotionCommand: String, Codable, Sendable { + case activity = "motion.activity" + case pedometer = "motion.pedometer" +} + +public struct OpenClawMotionActivityParams: Codable, Sendable, Equatable { + public var startISO: String? + public var endISO: String? + public var limit: Int? + + public init(startISO: String? = nil, endISO: String? = nil, limit: Int? = nil) { + self.startISO = startISO + self.endISO = endISO + self.limit = limit + } +} + +public struct OpenClawMotionActivityEntry: Codable, Sendable, Equatable { + public var startISO: String + public var endISO: String + public var confidence: String + public var isWalking: Bool + public var isRunning: Bool + public var isCycling: Bool + public var isAutomotive: Bool + public var isStationary: Bool + public var isUnknown: Bool + + public init( + startISO: String, + endISO: String, + confidence: String, + isWalking: Bool, + isRunning: Bool, + isCycling: Bool, + isAutomotive: Bool, + isStationary: Bool, + isUnknown: Bool) + { + self.startISO = startISO + self.endISO = endISO + self.confidence = confidence + self.isWalking = isWalking + self.isRunning = isRunning + self.isCycling = isCycling + self.isAutomotive = isAutomotive + self.isStationary = isStationary + self.isUnknown = isUnknown + } +} + +public struct OpenClawMotionActivityPayload: Codable, Sendable, Equatable { + public var activities: [OpenClawMotionActivityEntry] + + public init(activities: [OpenClawMotionActivityEntry]) { + self.activities = activities + } +} + +public struct OpenClawPedometerParams: Codable, Sendable, Equatable { + public var startISO: String? + public var endISO: String? + + public init(startISO: String? = nil, endISO: String? = nil) { + self.startISO = startISO + self.endISO = endISO + } +} + +public struct OpenClawPedometerPayload: Codable, Sendable, Equatable { + public var startISO: String + public var endISO: String + public var steps: Int? + public var distanceMeters: Double? + public var floorsAscended: Int? + public var floorsDescended: Int? + + public init( + startISO: String, + endISO: String, + steps: Int?, + distanceMeters: Double?, + floorsAscended: Int?, + floorsDescended: Int?) + { + self.startISO = startISO + self.endISO = endISO + self.steps = steps + self.distanceMeters = distanceMeters + self.floorsAscended = floorsAscended + self.floorsDescended = floorsDescended + } +} diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/PhotosCommands.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/PhotosCommands.swift new file mode 100644 index 00000000000..8d22f5d2791 --- /dev/null +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/PhotosCommands.swift @@ -0,0 +1,41 @@ +import Foundation + +public enum OpenClawPhotosCommand: String, Codable, Sendable { + case latest = "photos.latest" +} + +public struct OpenClawPhotosLatestParams: Codable, Sendable, Equatable { + public var limit: Int? + public var maxWidth: Int? + public var quality: Double? + + public init(limit: Int? = nil, maxWidth: Int? = nil, quality: Double? = nil) { + self.limit = limit + self.maxWidth = maxWidth + self.quality = quality + } +} + +public struct OpenClawPhotoPayload: Codable, Sendable, Equatable { + public var format: String + public var base64: String + public var width: Int + public var height: Int + public var createdAt: String? + + public init(format: String, base64: String, width: Int, height: Int, createdAt: String? = nil) { + self.format = format + self.base64 = base64 + self.width = width + self.height = height + self.createdAt = createdAt + } +} + +public struct OpenClawPhotosLatestPayload: Codable, Sendable, Equatable { + public var photos: [OpenClawPhotoPayload] + + public init(photos: [OpenClawPhotoPayload]) { + self.photos = photos + } +} diff --git a/apps/shared/OpenClawKit/Sources/OpenClawKit/RemindersCommands.swift b/apps/shared/OpenClawKit/Sources/OpenClawKit/RemindersCommands.swift new file mode 100644 index 00000000000..a909f5df1b1 --- /dev/null +++ b/apps/shared/OpenClawKit/Sources/OpenClawKit/RemindersCommands.swift @@ -0,0 +1,51 @@ +import Foundation + +public enum OpenClawRemindersCommand: String, Codable, Sendable { + case list = "reminders.list" +} + +public enum OpenClawReminderStatusFilter: String, Codable, Sendable { + case incomplete + case completed + case all +} + +public struct OpenClawRemindersListParams: Codable, Sendable, Equatable { + public var status: OpenClawReminderStatusFilter? + public var limit: Int? + + public init(status: OpenClawReminderStatusFilter? = nil, limit: Int? = nil) { + self.status = status + self.limit = limit + } +} + +public struct OpenClawReminderPayload: Codable, Sendable, Equatable { + public var identifier: String + public var title: String + public var dueISO: String? + public var completed: Bool + public var listName: String? + + public init( + identifier: String, + title: String, + dueISO: String? = nil, + completed: Bool, + listName: String? = nil) + { + self.identifier = identifier + self.title = title + self.dueISO = dueISO + self.completed = completed + self.listName = listName + } +} + +public struct OpenClawRemindersListPayload: Codable, Sendable, Equatable { + public var reminders: [OpenClawReminderPayload] + + public init(reminders: [OpenClawReminderPayload]) { + self.reminders = reminders + } +}