Feature request: Recording path fail over

caseyd

I recently found out that my SS instance was balking on startup due to the NAS being cranky.
This was causing a fair bit of difficulty on the SS host.
It would be great to have a timeout and fail over to an alternate path, with alert, when this occurs.

Ben

SecuritySpy already has this functionality: when a custom capture destination is not available, SecuritySpy will switch its recording to the Mac's system drive. Subsequently it will switch back to the custom capture destination if it becomes available again (though this requires the camera to go to Passive and then Active mode again). Additionally, if you have set an email address in the preferences to be notified of any errors, you should get an email informing you of this.

caseyd

so what I see is SSpy locking up under the condition of the _drive_ being mounted but the volumes unavailable. it needs to be force-quit. It does not recover.

This state is occurring when I added nightly scheduled restarts of the SSmac.

yes I am perusing the issue with the drive ( it's a raid ) manufacturer.

part of the thread dump
================

Process: SecuritySpy [48852]
Path: /Applications/SecuritySpy.app/Contents/MacOS/SecuritySpy
Architecture: i386
Parent: launchd [290]
UID: 501
Task size: 2650 pages
CPU Time: 0.007s

Thread 0x198296 priority 46
51 start + 53 (SecuritySpy) [0x24f5]
51 main + 69 (SecuritySpy) [0x2575]
51 Initialise + 4210 (SecuritySpy) [0x36c2]
51 SetUpCaptureDestination + 282 (SecuritySpy) [0xde3a]
51 UpdateCaptureDestinationFromAliasBackup + 336 (SecuritySpy) [0x56810]
51 ResolveAliasWithMountFlags + 37 (CarbonCore) [0x9140c5d2]
51 _ResolveAliasWithMountFlags + 103 (CarbonCore) [0x9140c641]
51 FSResolveAliasWithMountFlagsInternal + 133 (CarbonCore) [0x9140b3c2]
51 FSMatchAliasInternal + 2228 (CarbonCore) [0x913d22ea]
51 FindTarget + 266 (CarbonCore) [0x913d385c]
51 _FSGetCatalogInfoByName + 282 (CarbonCore) [0x913f4919]
51 FSMount::getattrs(unsigned long, char const*, unsigned long, unsigned long, FSAttributeInfo*, unsigned long, unsigned char*) + 229 (CarbonCore) [0x9142378f]
51 FSMount::_getattrs(unsigned long, char const*, unsigned long, unsigned long, FSAttributeInfo*, unsigned long, unsigned char*) + 308 (CarbonCore) [0x914a9db8]
51 GetVolFSAttributes(FSMount*, unsigned long, char const*, unsigned long, unsigned long, FSAttributeInfo*, unsigned long, unsigned long, FSVolAttributeInfo*, unsigned char*) + 619 (CarbonCore) [0x91438d53]
51 __getattrlist + 10 (libsystem_kernel.dylib) [0x9027e1ae]
*51 ??? (mach_kernel + 849280) [0xffffff80002cf580]
*51 unix_syscall + 467 (mach_kernel) [0xffffff80005e9573]
*51 getattrlist + 144 (mach_kernel) [0xffffff80002e1240]
*51 namei + 1454 (mach_kernel) [0xffffff80002f25fe]
*51 lookup + 556 (mach_kernel) [0xffffff80002f2c1c]
*51 VNOP_LOOKUP + 52 (mach_kernel) [0xffffff8000318d04]
*51 hfs_vnop_lookup + 1144 (mach_kernel) [0xffffff8000501138]
*51 cat_lookup + 152 (mach_kernel) [0xffffff80004f1358]
*51 ??? (mach_kernel + 3085909) [0xffffff80004f1655]
*51 BTSearchRecord + 179 (mach_kernel) [0xffffff8000525293]
*51 GetNode + 44 (mach_kernel) [0xffffff80005282bc]
*51 GetBTreeBlock + 182 (mach_kernel) [0xffffff80004ef756]
*51 buf_biowait + 121 (mach_kernel) [0xffffff80002e4bc9]
*51 msleep + 116 (mach_kernel) [0xffffff80005722f4]
*51 ??? (mach_kernel + 3612422) [0xffffff8000571f06]
*51 lck_mtx_sleep + 78 (mach_kernel) [0xffffff800022670e]
*51 thread_block_reason + 332 (mach_kernel) [0xffffff800022dcac]
*51 ??? (mach_kernel + 190945) [0xffffff800022e9e1]
*51 machine_switch_context + 366 (mach_kernel) [0xffffff80002b4bce]

Ben

The next version of SecuritySpy uses a different mechanism to find drives, so hopefully this particular issue won't be a problem in the future. For now though, the only mechanism that SecuritySpy can use is the "ResolveAliasWithMountFlags" function that you see in the trace above, which is an Apple function that is supposed to resolve an alias, and if it needs to it should mount any unmounted volumes before delivering to the caller the file system details. I've never seen it totally hang like this.

Have you tried adding the drive to the list of login items, so that it auto-mounts upon startup? See this guide: How to Automatically Connect to a Network Drive at Login in OS X.

caseyd

I automount some AFS mount-points from a NAS, I'll double check to see if this poor thing is on the list.
However it failed on a firmware update :P so I'll only be doing this to complete this thread ... to help the Future!

caseyd

OK I am at this again.

( yes the volume is listed in the Login Items, as is SSpy )

But I am not getting any fail-over from SSpy 4.2.9. Infact SS locks up on start and requires force quits. ( I can append a dump again )

specs os x 10.12.6 / Mini 16G 2.6Ghz i7 -> QNAP TS-670 Pro 4.3.5.0760
on that machine I am also running the SS webserver, and am monitoring that from a kiosk-configured RPi / HP Slate 21 combo.

Soo anyway this was running fine a month or so after the QNAP upgrade, but started getting cranky 12-12-2018, with repeated logging of forced wakeups ( wakupes_resource.diag) leading to the current condition ( last 2 days ) of .hang logs.

from the casual user the forced wakeup looks like SSpy crashes and auto restarts. The crash alert remains on the screen, but SSpy is running.

In trying to work this out SSPY locks up restart even after manually mounting the NAS volumes - so it should be finding the proper place to store.

As a detail it locks up displaying 'Initialising...' and some times at whatever 'looking for storage location' dialogs.

So I have a couple of questions.

1) can I manually edit the SSpy preferences file in ~/Library/Preferences/Security Spy Preferences /vNN ( where NN in my case is == 75 ) Why, to change the volume points to local disk so I can boot SSpy to continue to debug. I might need a hex editor but I don't know if you're checksumming etc.

2) Can I use NFS mounts SSpy? I am considering hard-mounting NFS mounts for my /Volumes/Cameras and leaving AFP.

caseyd

more on just what I want to know about text editing the SSpy preferences file.

in the following entry

== start ( probably mangled in the Forum's display.
L\A���$A����/Volumes/Cameras LEFT BRACKET afp://baduhq@BaduStorage(AFP)._afpovertcp._tcp.local/Cameras Server H �=A���$D7EB1663-831F-3A78-A6D1-52D9689E341F�/T����� � \ � p � ��DP\P�4f63b6d18e8dbb9c14294c74700de50b0d4b5ab6;00000000;0000000000000020;com.apple.app-sandbox.read-write;8|�@� h � � �P �����00000000015;/volumes/cameras/securityspy������
=== end

I am considering changing the

LEFT BRACKET afp://baduhq@BaduStorage(AFP)._afpovertcp._tcp.local/Cameras

to
LEFT BRACKET smb://baduhq@BaduStorage._smb._tcp.local/Cameras

after doing a remount via SMB. My axis cameras are having no problem managing their rule-based image uploads via SMB to this NAS, so ... it seems more reliable than AFP.

anyhoo this would allow me to get past the blocking start.

( edits for characters )

caseyd

( plowing along. )

Applying the Hex Fiend to the Properties file worked. I was able to restart SSpy. There was a quick dialog for accepting the license agreement.

I am still curious about NFS mounts. I am going to see ( in ust a moment I suppose ) how well SMB holds up.

Ben

I don't think you will be able to successfully edit the Preferences file. The fact that you got a new dialog that asked you to accept the license agreement indicates that your settings have all been lost (due to a corrupt Preferences file in this case) so I hope you have a backup of the Preferences file.

When SecuritySpy looks for a capture destination that it can't find, it should give up after some time and continue loading. If you hold the escape key while it's looking, it should give up sooner (or immediately).

However, we have seen some cases of hangs within a system call that is used by SecuritySpy to find/mount network drives. We haven't found a good solution for this yet, because the problem isn't in our code. The workaround for this seems to be the following:

- Turn off all network interfaces via the Network system preference (i.e. turn off WiFi and unplug any Ethernet cable connected to your Mac).

- Load SecuritySpy. With no network connection, it shouldn't hang.

- Once SecuritySpy has opened, go to the Preferences window and reset all custom capture destinations to their default value on the system drive.

- Then quit SecuritySpy and reinstate the network.

You should now be able to open SecuritySpy and set up the custom capture destination again to point to the NAS. I would strongly recommend you use AFP to mount the NAS rather than SMB.

caseyd

actually editing the prefs file as I outlined restored my SSPy.

It booted well, and I was able to double check that switching to SMB worked fin. All masks, for example, were retained. all accounts for the Axis pool. etc.

perhaps I'm just good? :P anyway I am old and have been doing stuff for a long time

I have a ticket in with QNAP and I'll continue to update this thread.

(the machine is racked & headless so it's a bit of a 'scene' to get to it and attach gear. Which is why I didn't simply do that. I will use that technique as a later resort.

Would you like the hang & restart dumps from Console? no biggie to send )

caseyd

also interested in your perspective on AFP vrs NFS vrs SMB.

AFAIK AFP is deprecated, however useful it is.

Ben

Yes, if you can send us the hang and restart logs to support@bensoftware.com that would be useful.

If you have retained your settings after manually editing the Preferences file that definitely means you have skill!

The issue with the protocol for network volumes is with SecuritySpy's auto-deletion feature. For this to work reliably, SecuritySpy needs to read certain information from the files such as their creation date and size, as well as the amount of free bytes remaining on the volume. Across all system versions that SecuritySpy currently supports (macOS 10.7 - 10.14), AFP Is the only one that works reliably for this.

However, if you are running a newer macOS version, it really shouldn't matter: AFP, NFS and SMB should all work fine. In the next major update to SecuritySpy we will have to drop support for older macOS system versions for a variety of reasons, so at that time we will update our advice about this.