Tricaster Hang Up on Live Remote - And Now No Video

[donald.williams]

So last night I shot a video for the Department of Defense Education Activity Europe. Everything was going fine - as it had for the past 2 nights - when suddenly during the last line of the production (Which was a culminating event of 150 students weeks worth of work from all over Europe) all the camera screens went black - I was thinking - no problem its the "thank you's" here at the end - I will just switch off record - (I figured we had popped a circuit breaker that the cameras were on - at least we made it to the end, right? - WRONG!!!!) In fact the tri-caster had locked up - because I wasnt working with my usual tricaster and I was at a remote where the equipment was supplied - I had no backup system in place - surely the file would be there when i restarted the computer - (Total system hang here - the VU Level meters were hung) Ctrl+Alt+Del wouldn't work either. Much to my dismay I called Newtek today for tech support - thinking they would be able to turn the cache file into a .avi file... WRONG! My head is going to roll for something that totally isnt my fault - The tricaster is billed as a one-stop shop - but when I talked to tech support I was told I should have been prudent and should have had a backup. He said he gets calls like this often - and the engineering department has no solution - You tell me how I explain to parents in Iraq that the machine we were counting on (that we paid $5000) for broke down on the last line and we lost everything... Heres the kicker the space is missing from the drive - so surely the RAW data is there in a cache... 2 things - I would like a way to recover what we had recorded to that point in the concert. I would also like for Newtek to warn people in their literature that their tech department does not recommend using the tricaster as a stand alone device - because in the event of a power failure or system freeze - at this point everything is lost - Maybe you should move to a different file format that writes while recording and doesnt have to be "stopped" to write the file. Does anyone have any idea how I can fix this? Once again this was a high-profile event with people from the DODEA HQ unit and they are already asking when the video will be ready so they can send it out to parents in Iraq... Please help me - surely there is a way to convert that cache file into video... Newtek you have really let me down on this one... No warning that this can happen - its a known issue - and you have no solution... other than to write the forums - Hopefully someone can help me - Sorry if I seem upset - but I am...:help:

[Streamking]

That sucks, sorry I only have hindsight advise of always going to tape and separate audio record(X5). I know this doesn't help now.
Have you contacted any of the specialized HHD data recovery companies? Apparently they can pull of any data that hasn't been copied over.
The DOD must have contacts to these people, or internal resources. It would be worth fessing up and having a chat to them, explaining exactly what happened... :)

[CreatvGnius]

...you have really let me down on this one... No warning that this can happen - its a known issue - and you have no solution... other than to write the forums - Hopefully someone can help me - Sorry if I seem upset - but I am...:help:

Naw, you've got that wrong, Donald. There's no such "known issue" that's being down-played here. I appeal to you to think level-headed here. As with any other scenario when a user doesn't have his computer connected to an UPS (Uninterruptible Power Supply) system to manage power spikes, surges and outages, I must suggest the failure has to lie with those responsible for the production (ya' gotta' bring a functioning UPS/line conditioner to the job).

If the A.C. line power goes out on one's unprotected home PC, and the data gets destroyed, that data loss can't be said to be the fault of the manufacturer of the PC, or that of the software that's running on it. PC.

Search in computer-centric forums if you must, for some help in data recovery would be my suggestion, to augment the suggestion someone posted just before me.

After applying some sober-minded effort toward your resolving your issue, here's hoping things make for a near miraculous turn-around for you.
-PeterG

[donald.williams]

Peter, please understand that the tech guy I talked to on the phone - did say that he gets calls about this.... So this is a known issue - the engineering department at Newtek should work on a solution for data recovery, (He indicated that his tech dept has stated there is no solution to this problem), especially if it is known that a whole production can be lost - because the "record" button can't be pushed at the end of the production...

As far as power - this was not a power loss issue - - no power was lost - All cameras were still on... The tricaster froze up... it totally locked up... We initially thought there was a power loss - but in fact there was not... It appeared as if the cameras had lost power - but in actuality the tricaster had just frozen...

I take offense at the fact that you think this is operator error - We used the system as configured for the 2 previous nights with no faults...

We did nothing different - and now we have no video... The Newtek support guy I spoke to indicated that they don't know of any data recovery service that can take a tricaster cache file and wrap it into an .avi file...

Also does the manual indicate anywhere that a UPS should be used with the tricaster? It is billed as a standalone device...

My boss is already aware of the situation... More than anything there is just massive disappointment...

We too are hoping for a miraculous turn around - if anyone can point us in the direction of a good HDD recover place that has maybe dealt with this before we would be grateful.

[SBowie]

Peter, please understand that the tech guy I talked to on the phone - did say that he gets calls about this.... So this is a known issue - the engineering department at Newtek should work on a solution for data recovery, (He indicated that his tech dept has stated there is no solution to this problem), especially if it is known that a whole production can be lost - because the "record" button can't be pushed at the end of the production...

I take offense at the fact that you think this is operator error - We used the system as configured for the 2 previous nights with no faults...I really sympathize with your pain, but:

1) I also wish there was a way to recover the cache, and I have seen this before - but if Engineering tells TS it can't be done, it's likely true.

2) TC is pretty darn stable, but any device can fail (witness the billions spent on space ventures that frequently teeter on the failure of parts that have been engineered up the wazzoo and passed endless QC controls)

3) given number 2, in any production environment where catastrophic loss is unacceptable, redundancy is a given.

On that last point, from wedding videographers on through the entire production world, professionals who really care about the result leave no room for catastrophe ("Gee, I'm sorry but this was a one camera shoot and the tape for the ceremony has a scratch on it from one end to the other .. could you get everyone togather again next weeknd? What? Oh, Grandma died? Wow, that's tough.") Murphy's Law is the first rule of professional planning.

To be a bit more forthright than I'd like to be given your situation, I've still got to say I'd have thought redundancy was a word the military of all people would understand. Maybe some really creative solution could be found by Engineering for heroic rescues - but on the other end all you'd have had to do to ensure you had something at the end was plug in a deck during the production. So I feel your pain, really I do - I had something similar occur many years ago when I was shooting weddings in medium format - but this is a lesson in planning for problems, so that when they occur (as they will sooner or later), you can save the day.

[Gordon]

Hi Donald;
I know your pain first hand. I even knew better than to trust a hard drive recording device but it wasn't a paid gig (I was volunteering my time and equipment) so I cut corners and didn't bring a UPS nor a backup record device. At the very end, after the 2 hour presentation was completely done, the sound guy accidently unplugged the main power. I actually had my mouse on the stop record button but didn't hit it in time. When the TriCaster started back up there was nothing on the hard drive and all the space was still available.

In your case, if the space is not available and you think a file can still be recovered, try R-Studio (Recovery Studio), from www.r-studio.com. It will recover files from a formatted hard drive, provided they haven't been overwritten. It is a lot more thorough than a simple undelete utility. Expensive, but worth it. One caveat; even if the cache file can be recovered there may not be an easy way to convert it to an AVI. I seem to remembering trying and not having any success - but I'm pretty sure this was with a VT[4] and not the TriCaster.

... I had no backup system in place ... Ouch, that is the school of hard knocks. Everyone knows to always have some backup in place, even if it's running tapes in a couple camera, when it is an important gig. I find it easiest to bring a DVD recorder but even they will not update the menu if you lose power - with no way to recovere the DVD file. So in this case you would still have lost it all, (also learned that the hard way). The only solution is a UPS + backup recorder. Make sure you test the UPS with everything plugged in so you know that it will handle the TriCaster, one monitor and the DVD recorder.
Maybe you should move to a different file format that writes while recording and doesnt have to be "stopped" to write the file. It wouldn't matter what software or what file you are using in Windows OS, the file allocation table doesn't get written until the file is closed. I suppose NewTek could use a direct access file type on a new partition, like many video companies did 15 years ago, but then they would lose all compatibility other software like DVD burning. Furthermore, there are good reasons why proprietary file systems/types were abandoned 10-15 years ago, they were NOT perfectly reliable neither. (DPS Perception comes to mind).

[CreatvGnius]

Peter, please understand that the tech guy I talked to on the phone - did say that he gets calls about this.... So this is a known issue...

You're kidding me, right? The fact that many people call the electric company to lament about their having experienced the shocking reality of having lost their TV, cable box, modem, computer or other appliances isn't to be frivolously characterized as "a known issue" warranting intervention by said power company.

People need to protect their appliances from the uncertainties inherent in AC power. It would appear you're not aware of such risk, or that you're deliberately minimizing it, which I doubt -- (although under your current distress, it's certainly possible that you could be tempted to do so). :devil:

Be well, Donald, and remember: protect valuable appliances that depend on A.C. power -- not only against outages (which you've indicated wasn't your experience there) but also against voltage irregularities like voltage drops, rises, spikes and surges; line noise, and the like.

Therefore, it is prudent to make certain your equipment is protected with a device that performs multifaceted protection services on that lame A.C. power available at the venue. Fact of life.

-PeterG

As far as power - this was not a power loss issue - - no power was lost - All cameras were still on... The tricaster froze up... it totally locked up...
[/Power]

TriCaster is a glorified computer system. Any computer needs to be protected from the ravages of the power company's raw AC power. I note you didn't mention what kind of protection you were using.:hey:
See discourse above.

We initially thought there was a power loss - but in fact there was not... It appeared as if the cameras had lost power - but in actuality the tricaster had just frozen...

I take offense at the fact that you think this is operator error - We used the system as configured for the 2 previous nights with no faults...

We did nothing different - and now we have no video... The Newtek support guy I spoke to indicated that they don't know of any data recovery service that can take a tricaster cache file and wrap it into an .avi file...

Also does the manual indicate anywhere that a UPS should be used with the tricaster? It is billed as a standalone device...

My boss is already aware of the situation.

It is my hope that he's informed about the vulnerability of unprotected A.C. appliances -- especially those that are microprocessor-based -- to data loss, if not total destruction. If so, he won't be quick to place blame on those not directly connected with the production set-up crew or equipment maintenance staff, or what-have-they.

Painful as these observations may be, your interaction here gives rise to the need for what's been communicated to you about the seriousness of such a habitual oversight. I promise you I am not misleading you on this. :hey:
-PeterG

[CreatvGnius]

Couple of other thoughts: Here's hoping you've informed your boss about the need for protected power on computer devices.

As for TriCaster being a standalone product -- it is. But that must be understood within the context of truth. Does TriCaster™ work its magic without need for the use of a display monitor? Without 110 Volts AC power (i.e., providing TriCaster™ or any other computer clean AC power, would be sensible practice) -- and without input from its "user"?

Naw!
-PeterG

[D3Cast]

See this thread.

http://www.newtek.com/forums/showthread.php?t=57039

No promises, but best of luck.

I remain amazed that, with all of their available knowledge about computer-based video files, NewTek has been unable to deliver a tool that fixes 'unfinished' AVI files, a situation that occurs in a variety of ways, including EXITING TRICASTER BEFORE UNCLICKING 'RECORD OUTPUT'.

Sorry to yell. The inability to recover these files is, in fact, a known issue at NewTek, and when it happens to you - fancy UPS system or no - a big deal.

-steve

[SBowie]

I remain amazed that, with all of their available knowledge about computer-based video files, NewTek has been unable to deliver a tool that fixes 'unfinished' AVI files, a situation that occurs in a variety of ways, including EXITING TRICASTER BEFORE UNCLICKING 'RECORD OUTPUT'.
I agree that the software should warn you when you attempt to Exit without stopping recording first.

On the other point though, 'corrupt file salvage' of the type you have described previously may well be a 'hit and miss' affair at best. NewTek may feel that providing a utility that 'works some of the time' would be more problematic for all concerned, leading people to depend on it ... with a good deal of unpleasantness when it frequently fails.

OTOH, plugging in a backup VCR will virtually never fail. For critical work there's just no substitute for this course, and it is not an expensive solution. I can't imagine not including that in the planning. It would be like taking a single camera to a remote venue and hoping for the best.

The inability to recover these files is, in fact, a known issue at NewTek, and when it happens to you - fancy UPS system or no - a big deal. The need for reliable power is a given, but supplying it is not really a panacea.

[mahasf]

How do you integrate a backup solution into your recordings? Currently I have a DV deck being fed from a DA, which gets its audio and video feed via the TC. But if the TC goes down, so does my deck feed.

Do you totally bypass the TC, and send an aux audio feed directly to the camera also?

[donald.williams]

Ok so went to

See this thread.

http://www.newtek.com/forums/showthread.php?t=57039

No promises, but best of luck.

Thanks for the tip D3Cast -

I am having the same problem you had in your post though...

The VT Temp File is 24+- GB - I feed it into ASF-AVI-RM-WMV Repair - chug chug chug and out comes a 45 minute file instead of the 1 hour 49 minute file...

If I try to open the VT Temp File under newtek info - it comes up with Windows media player - and shows the time (1:49:56) So it seems to me the other blocks are there - the program just can't reconstruct them...

At this point I have 45 minutes of the footage recovered -

Thanks to all who have helped so far~ Newtek should include this program on their machines with the warning that it shouldn't be used as a primary backup solution... as results may vary -

The VT Temp file from the tricaster is called VT Temp File [226756112,0].avi

Anybody have suggestions on recovering the other hour of the show...

Thanks to everyone so far - event the comments that I should have been proactive with a b/u solution - (Problem was that I was at a remote - and not working with my usual equipment set - this equipment was brought in and provided for me)

Any other solutions???

Thanks

[SBowie]

Nice gracious response - not easy in the current predicament, so kudos for keeping a calm head.

[CreatvGnius]

Nice gracious response - not easy in the current predicament, so kudos for keeping a calm head.
:agree:
Very gracious of you, Donald! Hang in there.
-PeterG

[doc debolt]

this is just my normal plan of attack for any shoot -

All cams run tape (unless its just not economical to do so)
output from cams into TCP
output from TCP into DVD recorder
TCP recording to HD

this way , i have raw footage - in case of later necessary edits - i have the raw tapes, the TC recorded file, the DVD in case the TCP lost power/crashed, etc...

I came to this workflow because of the loss of power that happened 4 times in 3 days - and finding out that the AVI file gets tanked if you dont click the on/off record button - was a true bummer.....thus the need for the DVD recorder as backup - 99% of the time i have no issues - the DVD recorder is there for that 1% of the time when you are glad you had it.

doc

[CreatvGnius]

...I came to this workflow because of the loss of power that happened 4 times in 3 days - and finding out that the AVI file gets tanked if you dont click the on/off record button - was a true bummer.....thus the need for the DVD recorder as backup - 99% of the time i have no issues - the DVD recorder is there for that 1% of the time when you are glad you had it.doc Better than nothing.

But truth be told, that DVD recorder is providing you a grossly compressed record of what you switched. You'd be better off going to a digital video tape deck, providing only 5:1 compression, rather than DVD's up to 20:1 ratio. That dichotomy is not what I'd call a "backup", in the pure sense of the word.

One other thing. :foreheads That apparently missing U.P.S. would have provided the needed safety net, giving you time to safely power your system down, during a power outage.

[Why, Oh why don't people make it a priority to provide a working U.P.S./Line Conditioner for every computer they subject to the ravages of the local power company's A.C. line?] :stumped:
-PeterG

[billmi]

I agree that the software should warn you when you attempt to Exit without stopping recording first.


I'm not sure it even needs to warn you.

Obviously, if you are quitting the software, you have no intention to keep recording. The simple addition of a stop recording command into the exit script would suffice, IMHO.

[CreatvGnius]

I'm not sure it even needs to warn you.

Obviously, if you are quitting the software, you have no intention to keep recording. The simple addition of a stop recording command into the exit script would suffice, IMHO. No doubt.
-PeterG

[SBowie]

Obviously, if you are quitting the software, you have no intention to keep recording. The simple addition of a stop recording command into the exit script would suffice, IMHO.There are probably several ways to go, but it seems to me someone may wish to Exit without saving the current session (yes, you'd wonder why they were recording in the first place then, but people are funny). I think - to cover all bases - a simple "Exit without saving current recording, or Cancel?" dialog would be ok. Maybe I'm trying to be overly considerate ....

[mahasf]

Has anyone submitted this idea as a future feature request?

[D3Cast]

Yes, a while back - it even got a "that's a good idea" from Paul...

-steve

[billmi]

I think - to cover all bases - a simple "Exit without saving current recording, or Cancel?" dialog would be ok. Maybe I'm trying to be overly considerate ....

You're just too polite, Steve.

So maybe a menu like this:

WARNING: The Tricaster is still recording. Would you like to:
1 - Exit and continue recording while the Tricaster is turned off.
2 - Exit without stopping recording and loose your work.
3 - Stop recording and exit.

Then if you choose 1, you should get a dialog box that says "Sorry, the Tricaster does a lot of things, but it can't keep recording when it is off, that would go against the very concept of being off." and then returns the user to the 3 choice menu.

If you choose 2, the dialog box could say "In light of how frustrated customers can become if work is lost, this option is no longer available." and then returns the user to the 3-choice menu.

Item 3 would simply do as it states.

:D

On the more serious side, Donald, it sounds like a rough go, and I feel for you. Even with everything in place and repetitive redundancies to act as backups, l things like this can still go wrong, and that's not a fun place to be. I've seen hardware switchers crash/lock before as well. The reality is these days, dedicated "hardware" still has software running inside of it.

I record the Tricaster's output straight to a DVD recorder simultaneously to the hard drive recording, as an easy, reasonably stable and inexpensive backup.

A UPS, I would also consider a must.

Several years ago, when working as a computer support tech, I was faced with a lab full of computers that seemingly reset/crashed at odd times, yet no one card in the machine seemed to be at fault.

There were no visible signs of brown-outs or anything that I would have thought indicated a power problem - but when we plugged in a charting voltmeter, we saw plenty of power dips down to 100 or even as low as 90 volts through the day. Called in an electrician to adjust the building's transformer, and viola, everything was fine. Since it's constantly writing to the hard drive while recording, the Tricaster, I would expect will need more consistent power than a computer that's only hitting the drives every now and again.

[SBowie]

Maybe so - but I think a simple dialog would have avoided a lot of these problems. Maybe it could open a browser to dictionary.com for a full discussion of the meaning of the word "off"? :)

[CreatvGnius]

You're just too polite, Steve.

So maybe a menu like this:

WARNING: The Tricaster is still recording. Would you like to:
1 - Exit and continue recording while the Tricaster is turned off.
2 - Exit without stopping recording and loose your work.
3 - Stop recording and exit.

4- Test system setup by entering AC power outage simulation? :D

[CreatvGnius]

Maybe so - but I think a simple dialog would have avoided a lot of these problems. Maybe it could open a browser to dictionary.com for a full discussion of the meaning of the word "off"? :)

For sure, "Attempt at Shutdown" behavior is very much dependent upon one's interpretation of what the meaning of "off" is.
-PeterG

[CreatvGnius]

:confused:

[donald.williams]

You guys should check this out - We might have a solution


Guys, I have wrote a program to scan and fix the un-closed NewTek AVIs (for NT25 and NTHQ only) from TriCaster/VT capture. The beta version was tested fine and I planed to release it but I found some more variation file format of the NT codec last month. The final program might be announce around 15-30 days. (My company is going to move to new place the week and it seems I don't have time to finish it in following two weeks.)

If anyone of you have interested in my program, please drop a mail (animlab@ms1.hinet.net) to me. This program will be priced at USD$99.00. For anyone post before the final release will get the price at USD$ 49.00.

[Crom]

My Tricaster has also stopped recording to disk during a live session. I was contracted for the particular job, on my own events I run tape in the cameras (60 minutes worth) and record the live switched video out to an external MiniDV tape deck. At the time I was using an external Firewire drive that belonged to the client.

For the The following weeks event the client brought a DV deck to record main output.

The kicker was that the little Knight Rider horizontal pulsating light froze, I was afraid that the whole thing would crash, luckily I pressed record again, went to the record panel and set the path to the D: Media folder on the tricaster. Was able to contiune the show, but lost 5 minutes of the show.

We never lost the live stream. We had to do a pickup shot after the live switch. The presenter was kind enough to redo his opening statement.

I have to agree with the rest of the guys here. I have 2 Tricasters for this reason alone... It's just good business and show's that you're serious.

Having done these events there's just too much that can go wrong no to have a backup.

[sbrandt]

"...this way , i have raw footage - in case of later necessary edits - i have the raw tapes, the TC recorded file, the DVD in case the TCP lost power/crashed, etc..."



What DVD recorder do you use?

[donald.williams]

Well I am happy to report that I got the program mentioned earlier - and it does work - I was able to restore my whole video. NEWTEK - you should purchase a copy of this and place it on every tricaster you ship out~

It took 0 effort on my part - and today I have the complete and final performance~

Would like to thank Chen Pan and AnimLab for a great little piece of software!

[donald.williams]

You can get the contact information for Chen Pan through the forum below:

See this thread.

http://www.newtek.com/forums/showthread.php?t=57039

No promises, but best of luck.

I remain amazed that, with all of their available knowledge about computer-based video files, NewTek has been unable to deliver a tool that fixes 'unfinished' AVI files, a situation that occurs in a variety of ways, including EXITING TRICASTER BEFORE UNCLICKING 'RECORD OUTPUT'.

Sorry to yell. The inability to recover these files is, in fact, a known issue at NewTek, and when it happens to you - fancy UPS system or no - a big deal.

-steve

[animlab]

Thank you for the recommand, Williams.

For anyone interested to restore his unclosed (unfinished) avi files, my FixNTavi can help. FixNTavi price at USD$99.00. Just drop me an email.

Chen Pan
AnimLab
animlab@ms1.hinet.net