How to capture audio/video streams

[Fred]

With quite a few concerts on the 2013 tour either being streamed or likely to be streamed, I thought we could gather some useful information on how exactly to capture an audio/video stream without having to resort to "primitive" methods, like screen recording or recording audio via your sound card's Stereo Mix input. So, anyone who has experience in this field want to kick off proceedings? This rookie could use a lesson.

(I realise this thread is in the completely wrong place, but it's three days to Lolla and my assessment is that it'd be a good idea to kick off this thread somewhere where people can actually see it and might be looking for it, just so we don't run into a "shit, I hope someone's recording this"-situation again.)

[Erik]

I cannot say that this still works, but back in April, here was the method a lot of folks used for Coachella.

http://stefansundin.com/blog/452

[ninjaw]

I cannot say that this still works, but back in April, here was the method a lot of folks used for Coachella.

This produced the glitchy one ? the only working one was made with HiDownload Platinum, you can still read this info here (http://www.dimeadozen.org/torrents-details.php?id=446799). There was only one good rip.

The most known live lossless capture tools are TubeMaster++ (http://www.tubemaster.net/) and Replay Media Catcher (http://applian.com/download-videos/).
The most known non-live youtube capture tool would be JDownloader (http://jdownloader.org/)
I'm still love using the french tool captvty (http://captvty.fr/) it got a very powerful and updated misc link downloader (this is a tv tool) this tool already know lot's of video links, and it helped me one time when TubeMaster++ was fooled refusing to download revealed links.

After capturing you need to change format, mux, cut, tag (but never convert)
mp3directcut (http://mpesch3.de1.cc/) is the most know tool it can cut MP2, MP3, and MP4 (require plugins) without loss, you can find mp2 in all mpeg2 videos, and MP4 in all youtube vids, a mp4 is like a video but with only an AAC file inside. This tool is also known to cut mp3+fucking CUE files
To lossless cut full hd videos, i've been amazed by SmartCutter (http://www.fame-ring.com/smart_cutter.html), even better than Tmpgenc MPEG Smart Renderer 4 (http://tmpgenc.pegasys-inc.com/en/product/tmsr4.html)
MkvToolNix (http://www.bunkus.org/videotools/mkvtoolnix/)'s MKVmerge is very powerful to (de)mux MKV
YAMB (http://yamb.unite-video.com/) can help you (de)mux MP4

[DigitalChaos]

paging @TheBang (http://www.echoingthesound.org/community/member.php?u=176) and probably a shitload of others that I can't think of.

Thanks for getting this thread started!

[DigitalChaos]

It's probably best if there is a diversity of tools used across people when capturing. Tool used, geographic location, etc all matter.
We will want to mention the use of TunnelBear and other proxy tools when it comes to dealing with region locked streams.

[Leviathant]

rtmpdump is my friend, and could be yours too, depending on how you wield your google-fu.

[cynicmuse]

This produced the glitchy one ? the only working one was made with HiDownload Platinum, you can still read this info here (http://www.dimeadozen.org/torrents-details.php?id=446799). There was only one good rip.
That was the second method that I tried after I had some glitches in the live rip. That method didn't work at all... there were visible and audible glitches at every join of the files. Both the audio and the video were variable frame rate. If you had a continuous capture, it wasn't an issue. Otherwise, no dice.
Edit: I'm a video novice (and I was doing all of the editing on a linux machine, which means extra fun). At this point, I don't remember if I saved the files as .ts or as .mp4. I know that weird things happened when I tried to convert from one file format to another.

I used VLC to save the glitchy stream; I think that the glitches happened because I was running two copies of the 1080p stream and I ran into buffering issues. Next time I'll run the monitoring stream on a different machine. I hadn't had any glitches in the earlier recordings when I hadn't been running two copies. I'll give rtmpdump a shot and see what happens.

[ninjaw]

That was the second method that I tried after I had some glitches in the live rip. That method didn't work at all... there were visible and audible glitches at every join of the files. Both the audio and the video were variable frame rate. If you had a continuous capture, it wasn't an issue. Otherwise, no dice.
Edit: I'm a video novice (and I was doing all of the editing on a linux machine, which means extra fun). At this point, I don't remember if I saved the files as .ts or as .mp4. I know that weird things happened when I tried to convert from one file format to another.

I used VLC to save the glitchy stream; I think that the glitches happened because I was running two copies of the 1080p stream and I ran into buffering issues. Next time I'll run the monitoring stream on a different machine. I hadn't had any glitches in the earlier recordings when I hadn't been running two copies. I'll give rtmpdump a shot and see what happens.

I read some details on the hidownload thing, i even tried myself on the stream, it worked ! the thing was saving numberous files and i guess ppl were having buffer issues past 1h, the guy who succeded was given a list of revealed urls from another tool.
Please remember that stream was using an unknown new youtube protection at the time, since then, i saw same thing on regular vids, and some classsic tools already updated.

[cynicmuse]

I read some details on the hidownload thing, i even tried myself on the stream, it worked ! the thing was saving numberous files and i guess ppl were having buffer issues past 1h, the guy who succeded was given a list of revealed urls from another tool.
Please remember that stream was using an unknown new youtube protection at the time, since then, i saw same thing on regular vids, and some classsic tools already updated.

Thanks for the info on what hidownload saved. I think that I goofed when I saved the files with vlc. I should have just used mplayer with the dumpstream option; the dumpstream option allows mplayer to just dump the stream with no re-encoding or re-encapsulation while vlc requires re-encapsulation when saving a stream.

As far as the protection on the stream goes, it was relatively easy to determine the url of the playlist (m3u8 file). I used the web console of my browser to track network requests as I switched the quality of the stream and got it that way.

[TheBang]

There are two types of Flash streaming: RTMP and HTTP Live Streaming

RTMP is the traditional Flash streaming protocol which was developed by Adobe. This is what you'd use rtmpdump or Replay Media Catch (which uses rtmpdump) to capture. Frankly, for YouTube, I haven't seen any RTMP streams since last year. YouTube appears to have transitioned entirely to HTTP Live Streaming. If you're watching a Flash stream from someone besides YouTube then it might still be RTMP.

HTTP Live Streaming (HLS) is a new(er) streaming protocol developed by Apple, initially for iOS streaming, but ultimately replacing the traditional QuickTime RTSP streaming (Darwin Streaming Server, etc.). This is the one that uses the .m3u8 files, and chunks the video stream as discreet HTTP downloads. According to the specification (http://developer.apple.com/library/ios/#documentation/networkinginternet/conceptual/streamingmediaguide/HTTPStreamingArchitecture/HTTPStreamingArchitecture.html), the encapsulation is MPEG-2 Transport Stream, so .ts is probably the correct encapsulation to choose if you're using VLC to capture the stream.

I'm actually not sure of the best way to capture HTTP Live Streaming streams. I failed on both the HTDA and Fuij Rock streams (which were both HLS). The nice thing about HLS is that if you can get the correct m3u8, I think you can kind of start and stop anywhere you want in the stream, since it's just chunked HTTP requests. This isn't really possible with live RTMP and RTSP streams.

I'll have to check out HiDownload Platinum. ffmpeg and VLC are also supposed to be able to do it. I had a hard time getting ffmpeg to work on the HDTA stream though. If anyone has any tips, please contribute.

Fortunately, there's plenty of time to practice on the streams before NIN takes the stage tomorrow!

[TheBang]

The Lollapalooza stream is live. Start practicing your captures!

[ninjaw]

I confirm Hidownload working : launch youtube, start sniffer, go to option, add .ts to the download automatically list, here you go ! it will record a long list of 5sec chunks.
After that ? rename them in TS, i use total commander for that, join them together, i use MKVmerge it will produce a final MKV file

[[parasite]]

any mac recommendations?

[ecirb]

If you download all the files to a folder with nothing else in it, you can open notepad and copy and paste the following into it:

copy /b * merged.ts

Save that as a .bat file in the folder you are downloading to. Go to the folder and double click the .bat file. It will take all the pieces and merge them into one file named merged.ts. Obviously you can replace merged with whatever you want to name the file.

[cynicmuse]

The stream only works live, unlike the Coachella stream where you could scan through it. The m3u8 file only has a couple of chunks in it instead of the whole stream so the Coachella method won't work (unless you'd like to download about a hundered m3u8 files and then a few hundred more ts files). I've gotten chunks with vlc somewhat successfully because it will regenerate the m3u8 file; I'll have to try the vlc method on the faster machine to see if it's a viable method. Otherwise, I'll boot up the windows partition and use hidownload.

[ninjaw]

Hidowloading keeps bugging about memory as reported before, however, Replay Media Catcher is working and is far easier: it does record automaticaly chunks and even the whole video at the same time, however he's naming them mp3 and mpeg, they are obviously TS files, I recommand removing the annoying capture sound, and the youtube renaming, so you can keep those numerical type files, maybe we can even exchange missing one ?

[Skeptikk]

So I have no idea what I've done but I think Ive got Replay Media Catcher recording so I'm not going to touch the laptop until NIN's over then cut out all this Killers stuff hahaha.

My screen has 20 odd thumbnails some failed and complete and then one is counting upwards of 300,000% downloading. No I have no idea what the other ones are but this one with the mega % is actually recording and playing me back video.

[finckscl]

Errr. hi :)
I downloaded ReplayMediaCatcher from torrent, followed all instructions, and no luck.
So, as Skeptikk I won't touch anything until NIN's over.

PS: FUCK The Killers.

[TheBang]

For Replay Media Catcher 5, I think what you have to do is go to Internet > HTTP > Content Type / File Extensions, and Delete the first 5 or so entries you see for File Extension "mp3". This will keep it from initiating all those extraneous MP3 downloads, and after that you should only see a single mpg download, which should hopefully be the stream.

For HiDownload, you want to go to System > Options > Sniffer, and add ".ts" to end of the long list of file extensions. You probably also want to go to General and check "Disable balloon notifications" or else it will get annoying. You should then see a bunch of numbered files being downloaded in sequence. For a 1080p stream, each chunk would be about 2-3 MB. At the end of it, you'll have to stitch all the chunks together. That's the part I'm not sure about yet.

[gstead]

I'm on my Mac using ffmpeg on the command line.


export URL720p="<get_url_from_firebug>"
ffmpeg -re -i $URL720p -codec copy -y "lollapalooza.m3u8"

This creates the 5-second .ts files mentioned above. I have a script to 'cat' those together and voila! Super nice video.

Caveat: I've never done this before but it seems to work.

[cynicmuse]

I'm on my Mac using ffmpeg on the command line.

This creates the 5-second .ts files mentioned above. I have a script to 'cat' those together and voila! Super nice video.

Caveat: I've never done this before but it seems to work.
Good to know. I got annoyed enough to boot up the windows partition and have Replay Media Catcher working there. I hadn't thought about using firebug to grab the m3u8 files.

[finckscl]

I downloaded ReplayMediaCatcher 5 (the previous was the 4), followed yr instructions. Made some tests, went good and freed 10 GB on HDD.
Let's see what happens!

[ItsJustDave]

I was trying to capture at 1080p but I've dropped to 720p in the hopes that will work out better. I'm using hidownload.

Here we go!

[finckscl]

I went straight for the 1080p stream. Some glitches at youtube, I hope they won't reflect on the final output.

PS: It's just me or this Sanctified is way more evil than the original?
;)

[ItsJustDave]

I was trying to capture at 1080p but I've dropped to 720p in the hopes that will work out better. I'm using hidownload.

Here we go!

hidownload out of memory. Continuing to queue, but stopped downloading. Any tips?

[finckscl]

Close any window/tab/program unneeded at this time.
Hint: Using fb reduces free memory.

[cynicmuse]

I'm seeing a few glitches on the live YouTube feed as well with the 1080 stream. I'm waiting until the stream is done and then I'll watch it in full screen and see what it looks like.

[TheBang]

hidownload out of memory. Continuing to queue, but stopped downloading. Any tips?
I ended up quitting and relaunching it. Got a gap in my files. I'll have to check next time it runs out of memory and see if it's a system memory problem? I highly doubt it, as i have 16GB of RAM. So I suspect it's just a HiDownload issue.

[ItsJustDave]

Close any window/tab/program unneeded at this time.
Hint: Using fb reduces free memory.

I understand how memory consumption works, hidownload indicates out of memory when I'm only using 31% of memory. Just wondering if I can keep things in queue for later download (after I close and reopen)?

292

[TheBang]

I'm missing the following 1080p chunks from tonight's broadcast. If anyone has them, please PM me:

8541-8570
8858-8862
9156-9160
9463-9488

[finckscl]

I understand how memory consumption works, hidownload indicates out of memory when I'm only using 31% of memory. Just wondering if I can keep things in queue for later download (after I close and reopen)?

292

It's clear that the problem doesn't reside on RAM resources. Mine crashed several times :/

[Skeptikk]

Thankyou for letting us know about Replay Video Catcher, I now have myself a fucking kickass copy of tonights show :D

[ninjaw]

I'm missing the following 1080p chunks from tonight's broadcast. If anyone has them, please PM me:

8541-8570
8858-8862
9156-9160
9463-9488

i do have some of these, yes

[ItsJustDave]

It looks like we're getting another go at this. Stream has started back up at the beginning again. I'm using Jaksta on my mac with success at the moment.

[ninjaw]

i do have some of these, yes

Currently upping 8858-9488 right here https://www.dropbox.com/sh/o78dicw8sqsp899/blTBl2VYqO

[finckscl]

It looks like we're getting another go at this. Stream has started back up at the beginning again. I'm using Jaksta on my mac with success at the moment.

I hope that works, man! Good luck! (but are you gonna go through it all of the streaming?)
Gotta pack my things for a flight, otherwise I'd stay in here :/

[ninjaw]

Does everybody knows QuickPar ? it's a VERY easy freeware that just test create and repair large numbers of files, it works like SFV with additional files to repair/recreate missing files, you see where i'm going there ? when anyone got the full list of chunks he can share PAR files to help any one else automatically.
Note: this amazing tool is capable to check even files with other names, and rename them

[ninjaw]

Currently upping 8858-9488 right here https://www.dropbox.com/sh/o78dicw8sqsp899/blTBl2VYqO
@TheBang (http://www.echoingthesound.org/community/member.php?u=176) your files are up, sorry my fucking smartphone didn't work so I miss the first 30mn of the show, and so your first request, but i did filled your three last request, please confirm us, that indeed numbers are the same for everyone, and we can actually share missing chunks.

[cynicmuse]

I think that I have the whole file, but I need to watch it. I'll let you know in 1.5 hours.

[ninlive]

If anyone wants to upload their 1080 version to the RITC server, let me know via email and ill send you the FTP info. ryanj at reflectinginthechrome dot com.

Sorry I didn't capture it. I was there. ;)

[ninjaw]

It may be useful to create a show with the numerical files, because that way we have a very easy way to check for missing parts, (not even usefull to watch it all), also the join job won't be live, it may produce a better and safe file. Unfortunaly i miss a large part, mainly i miss 8615-8763 if someone like to help.
The show is going to air again in 3 or 4 hours, be warned all chunck numbers will be different, and probably not even cut the same way : also, I won't have the same speed in france than in plain night, don't count on me this time.

[cynicmuse]

I was watching the file because replay media catcher just dumped everything in one file, so I don't have section numbers. I have 99.99% of the show. There's less than a 1 second skip around 4:33 into Find My Way if you start counting at the end "play something new". Is that something that can be fixed with QuickPar? I'm not familiar with file fixing utilities.

I'll give it another shot at 5:30am PDT; QOTSA went on at 1:54am and NIN started roughly 3 hours 45 minutes after that.

[ninjaw]

I was watching the file because replay media catcher just dumped everything in one file, so I don't have section numbers. I have 99.99% of the show. There's less than a 1 second skip around 4:33 into Find My Way if you start counting at the end "play something new". Is that something that can be fixed with QuickPar? I'm not familiar with file fixing utilities.
I'll give it another shot at 5:30am PDT; QOTSA went on at 1:54am and NIN started roughly 3 hours 45 minutes after that.

No it cannot, this solution cannot be used, apparently internal header are different, however, files seems to be exchangable. However I may have Find my way perfect, and there's some tool who can easily cut and join a file, i'm in the process of watching my own large file (my replay media catcher saved both) it take time just to remux the file, files doesnt' seems to be Mpeg2 as pretend replay media catcher :)

[ninjaw]

Wait a minute, there's something very odd : looks like the main mpeg file is actually a real mpeg2+mp2 file ! however all chunks are real AVC+AAC file ??? obviously the MP4 version should be better ?

[cynicmuse]

Hmm. Now that I'm looking at my file, the video is 1080p, but the audio is 128 kbps mp3... and I don't know why.

[ninjaw]

Hmm. Now that I'm looking at my file, the video is 1080p, but the audio is 128 kbps mp3... and I don't know why.

Right, it's 128kbps, mp2 is really close to mp3, so vlc is saying MPega layer 1,2,3 but it's mp2 according to other tools

My "Find my way" version is perfect at the spot you mentionned, i have the tools able to mix both files lossless, but ... why work in mpeg2 when we have a mpeg4 ?

[cynicmuse]

Right, it's 128kbps, mp2 is really close to mp3, so vlc is saying MPega layer 1,2,3 but it's mp2 according to other tools

My "Find my way" version is perfect at the spot you mentionned, i have the tools able to mix both files lossless, but ... why work in mpeg2 when we have a mpeg4 ?
Now I'm really confused. What is the mpeg4 file? The ts sections?

[ninjaw]

Now I'm really confused. What is the mpeg4 file? The ts sections?

Yes they are in MPEG4, i'd like to have an Answer from TheBang it should have it full now

[ninjaw]

I'm really not sure about this, looks at what is showing Tmpgenc
http://img841.imageshack.us/img841/8666/thn4.jpg
Yes i (perfectly) captured two different stream in full hd. MP4 should be better than MP2 BUT 1) There is less bitrate 2) It looks like a usual USA TV format, what if the mp4 is just a dirty converted live video ?
I really can't say wich version is better.

[TheBang]

Thank you ninjaw, your chunks slot in perfectly, so it's good to know that recorders can swap chunks like that. I found a few more corrupt ones (HiDownload can apparently corrupt a few of the chunks just prior to its Out of Memory issues). Can I please get the following from you, ninjaw?

9153-9155
9388

I'm also still looking for the following chunks:

8536 - 8556

Just for information, the full NIN show is chunks 8330 - 9508.

[TheBang]

So, regarding Replay Media Catcher, I'm pretty sure that it's transcoding. I got the same thing from it, MPEG-2 video with MP2 audio. Having MPEG-2 elementary streams is not part of the HLS specification (http://developer.apple.com/library/ios/#documentation/networkinginternet/conceptual/streamingmediaguide/HTTPStreamingArchitecture/HTTPStreamingArchitecture.html), it's not compatible with other HLS clients, and it's terribly inefficient for streaming, compared to MPEG-4. The only conclusion I can come to is that Replay Media Catcher is transcoding it for some reason, and that makes it worthless.

The chunked TS files are what we want. The 1080p stream has an H.264 video stream that runs at about 3-5 Mbps, and an AAC audio stream at about 256 Kbps. If anything is released that is other than these specifications, then it's been transcoded, and is inferior to the direct stream dump.

TS is MPEG-2 Transport Stream (see above HLS specification), but the elementary streams it contains are H.264 video and AAC audio. To assemble the TS files, it's really as easy as simply concatenating all the files together. Once you have the full file, you can rewrapper it into something more standard like an MPEG-4 container or MKV.

[ninjaw]

Thank you @ninjaw (http://www.echoingthesound.org/community/member.php?u=746) , your chunks slot in perfectly, so it's good to know that recorders can swap chunks like that. I found a few more corrupt ones (HiDownload can apparently corrupt a few of the chunks just prior to its Out of Memory issues). Can I please get the following from you, ninjaw?

9153-9155
9388

I'm also still looking for the following chunks:

8536 - 8556

Just for information, the full NIN show is chunks 8330 - 9508.

Actually there is full credits in 9508-9543 they can be nice too
YOUR FILES ARE UP !

I did receive 500Mo of chunks from the start, unfortunally this doesn't include 8536 - 8556

I myself need : 8615-8763 9088 9388 9389 9427 (and lot more from the start)

[ninjaw]

So, regarding Replay Media Catcher, I'm pretty sure that it's transcoding. I got the same thing from it, MPEG-2 video with MP2 audio. Having MPEG-2 elementary streams is not part of the HLS specification (http://developer.apple.com/library/ios/#documentation/networkinginternet/conceptual/streamingmediaguide/HTTPStreamingArchitecture/HTTPStreamingArchitecture.html), it's not compatible with other HLS clients, and it's terribly inefficient for streaming, compared to MPEG-4. The only conclusion I can come to is that Replay Media Catcher is transcoding it for some reason, and that makes it worthless.

The chunked TS files are what we want. The 1080p stream has an H.264 video stream that runs at about 3-5 Mbps, and an AAC audio stream at about 256 Kbps. If anything is released that is other than these specifications, then it's been transcoded, and is inferior to the direct stream dump.

TS is MPEG-2 Transport Stream (see above HLS specification), but the elementary streams it contains are H.264 video and AAC audio. To assemble the TS files, it's really as easy as simply concatenating all the files together. Once you have the full file, you can rewrapper it into something more standard like an MPEG-4 container or MKV.

You may be right, i'm voting for recombine those chunks, mainly because this new way of rip allow to be sure of the result quality

[cynicmuse]

I figured out how to rip the stream with vlc from the command line; it will regenerate the m3u8 automatically. I'll give that a shot for the next round since Replay Media Catcher was transcoding, which pisses me off. Why transcode instead of just dumping the file? Especially because mp2 is a larger file size.

[ninjaw]

Shit my 9388 looks bad too, and btw : i cannot do anything with my mpeg2 file it seems to be some corruption at some point

[TheBang]

Actually there is full credits in 9508-9543 they can be nice too
I stopped before that, because I had to catch my bus. I'll see if I can get it on the rebroadcast.


I myself need : 8615-8763 9088 9388 9389 9427 (and lot more from the start)
Your 9388 is corrupt also? Sounds like it might have been an encoder drop at the Lollapalooza source then.

[ninjaw]

I figured out how to rip the stream with vlc from the command line; it will regenerate the m3u8 automatically. I'll give that a shot for the next round since Replay Media Catcher was transcoding, which pisses me off. Why transcode instead of just dumping the file? Especially because mp2 is a larger file size.

actually there's no reason why your version would not save "mp3/TS" chunks ? even replay media catcher 4 was doing that without mp2, are you sure ti have activated everything needed ? Also : I still have for now a good connection, i'll try to capture again

[TheBang]

I figured out how to rip the stream with vlc from the command line; it will regenerate the m3u8 automatically.
Can you give the step-by-step instructions for that, please? I had trouble figuring out how to do that with VLC.

[ninjaw]

I stopped before that, because I had to catch my bus. I'll see if I can get it on the rebroadcast.

Oh, I'm upping them with your other request so.

[ItsJustDave]

Just a reminder. Stream still going on its second run through and Band of Horses just finished up. That puts NIN set about 1 and a half hours from now. Jaksta crashed while I was sleeping and as a result I lost some chunks (regrettably while QOTSA was on), but I'll shut it down and start fresh before the big show. I'm grabbing 1080p chunks.

[TheBang]

It looks like we're getting another go at this. Stream has started back up at the beginning again. I'm using Jaksta on my mac with success at the moment.
Did you have to do anything special to get Jaksta to work? It's doesn't seem to be doing anything for me when I go to the Lolla stream.

BTW, I believe Jaksta is made by the same people who made Replay Media Catcher.

[cynicmuse]

actually there's no reason why your version would not save "mp3/TS" chunks ? even replay media catcher 4 was doing that without mp2, are you sure ti have activated everything needed ? Also : I still have for now a good connection, i'll try to capture again
I'm using version 5. Do I need to add a ts entry to replay media catcher? What would be the proper format for that?


Can you give the step-by-step instructions for that, please? I had trouble figuring out how to do that with VLC.
Re vlc: Here are the options that I'm using:
vlc -I dummy -vvv "insert url for latest m3u8 playlist here" --sout-file-append --sout=file/ts:foobar.ts
-I dummy: no interface
-vvv "insert url for latest m3u8 playlist here": sets the url to download
--sout-file-append: need this option because it will otherwise just save the file specified in sout over and over again, overwriting what we want; this way, it appends it to foobar.ts which is a ts file that I've already created
--sout=file/ts:foobar.ts: we want to output to a file, encapsulating it as a ts, with a file name of foobar.ts

However, this only seems to work on my linux machine and not the mac. I'm not sure if this is some weird linux quirk or what.
Edit: both are running VLC 2.0.7.

Edit2: Nevermind. VLC doesn't work because the m3u8 playlist isn't regenerating totally new chunks... it's recording 1 2 3 2 3 4 3 5 etc. I guess if I figured out how to name the chunks based on time stamp instead of appending them this could work, though it'd be a pain to stitch together.

[ninjaw]

I'm using version 5. Do I need to add a ts entry to replay media catcher? What would be the proper format for that?

No, not really, it should do it as it does for us, it does save them as mp3
Quick note: TheBang my upload is finnished, my download too, we definitly should have the same thing now, this chunk exchange is really an excellent way to capture streams !

[TheBang]

I'm using version 5. Do I need to add a ts entry to replay media catcher? What would be the proper format for that?
Content type: video/MP2T
File Extension: ts

Thanks for the vlc instructions. I'll check them out

[cynicmuse]

Thanks y'all. I hadn't run mediainfo on what Replay Media Catcher was calling .mp3 files before I'd killed all of the mp3 options.

[ninjaw]

I just had a look a full MKV of what we have, it's quite good ! I called people with details for chunk downloading, we should be ok !

[ItsJustDave]

Did you have to do anything special to get Jaksta to work? It's doesn't seem to be doing anything for me when I go to the Lolla stream.

BTW, I believe Jaksta is made by the same people who made Replay Media Catcher.

I had to play around with the settings a bit before I got it to capture the chunks as numbered files. I've got the following set in preferences:

293294

I start the youtube stream in my browser then I start a fresh instance of jaksta (already in monitor mode). Hope this helps.

[ItsJustDave]

Here we go (again)!

[ninjaw]

For now it's perfect for me ! I have news trop live recording, i clicked 'erase' on the fucking MPEG main file ans now i dont See it anymore ! (Easier from the small interface) i can confirm this shitty file is encoder live ! When you click live info on it you can see the ffmpeg command... Fucking useless !!

Envoyé depuis mon GT-I9300 en utilisant Tapatalk 4 Beta

[cynicmuse]

I'm out. I walked away for a minute and replay media catcher died. I'll blame it on that 2gb of RAM. I'll upgrade to 64 bit windows before the next live stream. Thanks for all of the help. Good luck.

[ItsJustDave]

I'm out. I walked away for a minute and replay media catcher died. I'll blame it on that 2gb of RAM. I'll upgrade to 64 bit windows before the next live stream. Thanks for all of the help. Good luck.

Keep in mind we just need to get all the pieces collectively. If you've got/getting numbered chunks we're in good shape.

[cynicmuse]

Sorry. RMC 5 is seriously borked when I restarted it. I now have 3 copies of the mpg all re-encoding with ffmpeg simultaneously while not getting any of the ts files, at which point it crashed :( If anyone needs the first 250 files, I can help.

[ninjaw]

Exactly : To avoid a lot or crash i ask To erase the MPEG from the tool, every song, each time it's crashin i have To force close chat awful ffmpeg. Or course the fucking tool recreate a ffmpeg. Plz hang on ! De may need missing chucks

Envoyé depuis mon GT-I9300 en utilisant Tapatalk 4 Beta

[ninjaw]

Carefull if you dont forceclose thar ffmpeg shit it will continue working and create many fullhd stream !

Envoyé depuis mon GT-I9300 en utilisant Tapatalk 4 Beta

[ItsJustDave]

It looks like I've got a contiguous run of files from the beginning to end (including the credits). About to merge them and see how it looks.

Replay of NIN set is 1080 chunks (15304-16383)

Directly followed by credit roll 34 chunks (16384-16417)

VLC details (from first chunk):
295

[ninjaw]

Nin including fulll intro and credits are 15303-16417 (16418?) checking my stuff : At first look chunks may be exactly cut the same way as first time

[ninjaw]

Here are chunks i miss 15492 15773 16071 16073 16077-16100 16193 16197 16397-16402 16404-16407 16410

[ninjaw]

chunks aren't cut the same way between stream & replay...

[ItsJustDave]

During 'Copy Of A' 15366 appears to be a copy of 15365.

:)

[Moebius]

try to regenerate the segment URLs, it's possible to do it, you have to guess at the clen (use a binary search). also some other important detail I'm forgetting but maybe you can figure it out.

the urls themselves are live for at least a few hours after the stream ends.

[ninjaw]

During 'Copy Of A' 15366 appears to be a copy of 15365.

:)

I confirm that

[ItsJustDave]

Here are chunks i miss 15492 15773 16071 16073 16077-16100 16193 16197 16397-16402 16404-16407 16410

Putting these to a dropbox public share now.

EDIT: Also, I'd suggest whoever puts the final product out watches through beginning to end in case there's other duplicated chunks.


md5s generated from my chunks http://www.sendspace.com/file/xkwz4z

Note that the duplicated chunk has 2 different md5s, the first chunk was slightly larger. I haven't determined if the first or second is the better one to use.

[Moebius]

During 'Copy Of A' 15366 appears to be a copy of 15365.

:)

Are you saying this is a bug in the stream or just a strange coincidence? i.e. are you going to editing out one of them?

[ItsJustDave]

Are you saying this is a bug in the stream or just a strange coincidence? i.e. are you going to editing out one of them?

It was a bug in the stream. I've compared sequences (3 chunks before, 3 after) with one or the other in the middle and they are indiscernible to me.

[ItsJustDave]

Here are chunks i miss 15492 15773 16071 16073 16077-16100 16193 16197 16397-16402 16404-16407 16410
https://www.dropbox.com/sh/28i13ynzcz7v8ux/rgrxRsfYx1/ninjaw%20missing

[ninjaw]

Putting these to a dropbox public share now.

EDIT: Also, I'd suggest whoever puts the final product out watches through beginning to end in case there's other duplicated chunks.


md5s generated from my chunks http://www.sendspace.com/file/xkwz4z

Note that the duplicated chunk has 2 different md5s, the first chunk was slightly larger. I haven't determined if the first or second is the better one to use.

Only 23 files passed your md5 test :)

[ninjaw]

https://www.dropbox.com/sh/28i13ynzcz7v8ux/rgrxRsfYx1/ninjaw missing

Fantastic ! my list looks complete however there could be dead chunk, we experienced that before

[ItsJustDave]

It was a bug in the stream. I've compared sequences (3 chunks before, 3 after) with one or the other in the middle and they are indiscernible to me.

I've packed up the chunks in question: http://www.sendspace.com/file/9z55ni

[ItsJustDave]

Fantastic ! my list looks complete however there could be dead chunk, we experienced that before

With the chunks you lost and the non-matching md5s, I'm thinking the data I've got is more likely to be accurate. Hopefully someone else who pulled chunks (even for a brief time) can run md5s as well?

[ninjaw]

With the chunks you lost and the non-matching md5s, I'm thinking the data I've got is more likely to be accurate. Hopefully someone else who pulled chunks (even for a brief time) can run md5s as well?

No, i was already said by others that files aren't compatibles, must be teh TS header, Actually someone just PMed me the missing chunks from the first stream ! (exept the dead chunk)

[ItsJustDave]

I'd like to hand my rip off to a few people for QC before wide distribution. Thoughts?

[ItsJustDave]

No, i was already said by others that files aren't compatibles, must be teh TS header, Actually someone just PMed me the missing chunks from the first stream ! (exept the dead chunk)

fair enough

[Moebius]

I have a script working to pull old segment chunks straight from youtube, I can provide additional md5s of questionable segments (it'll take a while to pull EVERY segment though)

[ninlive]

How large is everyone's files? Curious. I've got a full 1080 recording that was 50GB then converted to a ts file to about 8.4GB

[ninjaw]

2.52Go did you use digital or analog ?

[Moebius]

Sanity check, does this look correct to others (from the replay stream):



ab879e3b073bd930e3a8e224b2d49b4a /media/Elements/nin/good/15492.ts


filesize=1461136

[ItsJustDave]

I've been running through playback (stopping and starting here and there) and it appears that they cut frail/wretched from the replay. Unless it was also missing the first time around? Chunk 15661 cuts from the end of piggy to terrible lie. Was frail/wretched in the original stream?

[ninjaw]

Sanity check, does this look correct to others (from the replay stream):
filesize=1461136

Yes itr does look correct

[ItsJustDave]

How large is everyone's files? Curious. I've got a full 1080 recording that was 50GB then converted to a ts file to about 8.4GB

That seems wrong.

[ItsJustDave]

Sanity check, does this look correct to others (from the replay stream):



filesize=1461136

matches my md5 (which is posted in this thread)

[ninjaw]

I've been running through playback (stopping and starting here and there) and it appears that they cut frail/wretched from the replay. Unless it was also missing the first time around? Chunk 15661 cuts from the end of piggy to terrible lie. Was frail/wretched in the original stream?

WTF ! yes it was lucky us we found last chunks

[Moebius]

matches my md5 (which is posted in this thread)

Your sendspace link for the md5s gives me a "Download.exe" executable...

EDIT: nm, deceptive ads...

[ItsJustDave]

nm, deceptive ads...

I hate those things.

BTW, are you able to pull chunks from the original stream? Or just the replay?

[Moebius]

I hate those things.

BTW, are you able to pull chunks from the original stream? Or just the replay?

That's what I'm wondering, do you have the original stream url?

EDIT: this is the URL I had for the replay: /watch?v=QK6zhpuBZVo&feature=player_embedded
I think the live stream was at the same id?

[ItsJustDave]

That's what I'm wondering, do you have the original stream url?

EDIT: this is the URL I had for the replay: /watch?v=QK6zhpuBZVo&feature=player_embedded (https://www.youtube.com/watch?v=QK6zhpuBZVo&feature=player_embedded)
I think the live stream was at the same URL?

The URL never changed, you'd just be looking for earlier chunks.

[Moebius]

The URL never changed, you'd just be looking for earlier chunks.

I'll start searching...

[ninjaw]

shit we are still missing 8541-8556 also 9388 looks dead

[ninjaw]

LAST MINUTE someone is uploading me 8541-8556

[Moebius]

0 -title card-
3200 Smith Westerns
3800 -intermission-
4000 Icona Pop
4500 Crystal Castles
5200 -intermission-
5300 ? some band
5900 Queens of the Stone Age
6600 Band of Horses
7200 -intermission-
7300 ??
7700 The Killers

[ItsJustDave]

Just for information, the full NIN show is chunks 8330 - 9508.

Hope this helps moebius

[Moebius]

Yep, I see it, thanks. How many segments in is the cut in the replay?

[ItsJustDave]

Yep, I see it, thanks. How many segments in is the cut in the replay?

Replay was 15304-16383. Ugly Piggy/Terrible Lie quick cut was 15661.

P.S. You are kicking ass

[ninjaw]

It's official I got all chunks from the original stream, as will @TheBang (http://www.echoingthesound.org/community/member.php?u=176), 1215 chunks, 2.73Go

[Moebius]

It's official I got all chunks from the original stream, as will @TheBang (http://www.echoingthesound.org/community/member.php?u=176), 1215 chunks, 2.73Go

Can you post segment md5s?

[Moebius]

It does seem like the Frail is in the original playback

EDIT: see segment 8695 for example. lots of video glitch though.

[ninjaw]

Can you post segment md5s?

What for it doesnt work ?

[ninjaw]

actually trying my magic to lossless fix dead 9388 glitch

[TheBang]

How large is everyone's files? Curious. I've got a full 1080 recording that was 50GB then converted to a ts file to about 8.4GB
This is transcoded garbage. If someone says, "and then I converted to" that means it's going to be inferior quality to a dump of the original stream.


It's official I got all chunks from the original stream, as will @TheBang (http://www.echoingthesound.org/community/member.php?u=176), 1215 chunks, 2.73Go
I'll take a look later today and start auditing.

[gstead]

I was watching the file because replay media catcher just dumped everything in one file, so I don't have section numbers. I have 99.99% of the show. There's less than a 1 second skip around 4:33 into Find My Way if you start counting at the end "play something new".

My 720p ffmpeg dump glitched at exactly this point. So perhaps it was upstream of us.

[Moebius]

What for it doesnt work ?

Can you generate md5sums for the individual segment ts files you have, so others can cross-check and confirm correctness?

[ItsJustDave]

It does seem like the Frail is in the original playback

EDIT: see segment 8695 for example. lots of video glitch though.

Keep in mind, you will observe more artifacting in any single chunk when separated from the leading chunks.

[ninjaw]

Can you generate md5sums for the individual segment ts files you have, so others can cross-check and confirm correctness?

Same question, we saw several times it didn't work

[ninjaw]

Nice ! @Erik (http://www.echoingthesound.org/community/member.php?u=627) (and @meimnot) found the good 9388 chunk ! now the stream looks complete and glitch free ! thanks also to @Vler Schinicius (http://www.echoingthesound.org/community/members/1558-Vler-Schinicius) who filled large missing parts

[Moebius]

Same question, we saw several times it didn't work

That's not true, ItsJustDave posted md5s for the replay segments and they match segments I have from that portion.

[ninjaw]

That's not true, ItsJustDave posted md5s for the replay segments and they match segments I have from that portion.

Yeah but not me. Anyway here's a par2 http://www.sendspace.com/file/3zelbx

[Moebius]

Yeah but not me. Anyway here's a par2 http://www.sendspace.com/file/3zelbx


Hmm, I'm not very familiar with par2 but it doesn't seem to like my files. I downloaded a few of the end blocks from the original range, 9540-9543. This is what par2verify gives me (I'm on Linux):


Target: "9538.ts" - missing.Target: "9539.ts" - missing.
File: "9540.ts" - no data found.
File: "9541.ts" - no data found.
File: "9542.ts" - no data found.
File: "9543.ts" - no data found.


Can you possible give me a plain md5sum output? Maybe you have cygwin?

[ninjaw]

Hmm, I'm not very familiar with par2 but it doesn't seem to like my files. I downloaded a few of the end blocks from the original range, 9540-9543. This is what par2verify gives me (I'm on Linux):

can you possible give me a plain md5sum output? Maybe you have cygwin?

Well not rigtht now but par2 is the best of those (because it fixes files)

[Moebius]

Okay, I seem to be having a better result with the first few segments. It looks like par2 perhaps can't verify files in the middle of the sequence if the beginning is missing.

[ninjaw]

I am upping a m4a album i really need a break... https://www.dropbox.com/sh/3z0dcv1yrwtfw46/LvDdWDm68m

[kargath]

This is a known problem with the bootlegged copies of hidownload... I bought the program and the memory error hasn't happened to my since. I wasn't capturing last night though since I was at the show.

*edit* damnit.. forgot to quote... but this is in response to the people getting memory problems where it tanks.

[ItsJustDave]

This is a known problem with the bootlegged copies of hidownload... I bought the program and the memory error hasn't happened to my since. I wasn't capturing last night though since I was at the show.

*edit* damnit.. forgot to quote... but this is in response to the people getting memory problems where it tanks.

I was using their demo version from their site. So apparently there's a memory leak in the demo that also affects cracked versions

[Moebius]

Well not rigtht now but par2 is the best of those (because it fixes files)

It can fix them, assuming the original files it was generated from are correct. But that's what we're trying to determine in the first place...

I've found a few blocks that par2 reports as damage; could be either your version or mine that is corrupt.

[Moebius]

I've found a few blocks that par2 reports as damage; could be either your version or mine that is corrupt.

So far these two show up as damaged, out of approximately the first ~320 segments:


Target: "8536.ts" - damaged. Found 1 of 3 data blocks.
Target: "8540.ts" - damaged. Found 1 of 2 data blocks.


I've re-downloaded them twice and I get the same result.


0327c557068e78debca59d3716c7cea7 ./8540.ts
0327c557068e78debca59d3716c7cea7 ./8540.ts

[ninjaw]

Yes or it could be what we already talked

Envoyé depuis mon GT-I9300 en utilisant Tapatalk 4 Beta

[ninjaw]

I dont know what you're trying to verify. Actually par2 can fix even if all files are damaged, kinda magic. Not that it'll be usefull on chat kind of files. What if you'll do a hex compare ? I may be right from the start

Envoyé depuis mon GT-I9300 en utilisant Tapatalk 4 Beta

[ItsJustDave]

I dont know what you're trying to verify. Actually par2 can fix even if all files are damaged, kinda magic. Not that it'll be usefull on chat kind of files. What if you'll do a hex compare ? I may be right from the start

Envoyé depuis mon GT-I9300 en utilisant Tapatalk 4 Beta

Your par2 files were not generated from the files as they existed on you tubes server. As a result they just verify the similarity between your versions and everyone else's. 2 people who uniquely pulled data from YouTube getting the same md5 is pretty strong evidence that the md5 can be trusted.

[jmtd]

I am upping a m4a album i really need a break... https://www.dropbox.com/sh/3z0dcv1yrwtfw46/LvDdWDm68m

Thanks (and to all) for putting in all the effort... Take a well deserved break! Ill start a listen test this evening I think.

Looking fwd to an archive of the video too :)

[Moebius]

I dont know what you're trying to verify. Actually par2 can fix even if all files are damaged, kinda magic. Not that it'll be usefull on chat kind of files. What if you'll do a hex compare ? I may be right from the start

I'm trying to verify that we've captured all the segments 100% correctly from what was streamed from youtube.com. Your par2 can only repair my files to look like your files, which doesn't help if your files have errors compared to the original stream.

For the files where our checksums don't match, it would help if someone had a third to check against. But we could hex compare as well to see if we can figure out why it's different.

[Moebius]

I put some files online @ http://nunatak.sizone.org/~moebius/nin/

It includes my md5, Dave's md5's, as well my copies of 8536 and 8540 which fail the par2 check.

I will also upload ninjaw's par2 file and the full par2verify output (takes a couple minutes to run).

I don't have all the segments downloaded yet (of either the original broadcast or the replay) but I'm getting a ton of par2 errors from 8764 onward, in addition to the individual failures on 8536 and 8540.

EDIT: Here's the par2verify output, starts out well, then lots of errors 8764-9400 (http://nunatak.sizone.org/~moebius/nin/par2.out.txt).

[ninjaw]

Thanks (and to all) for putting in all the effort... Take a well deserved break! Ill start a listen test this evening I think.

Looking fwd to an archive of the video too :)

i forgot to separate song 4&5 this is now done and the files are actually updating

[Moebius]

For posterity, here is basically what I'm doing.

When you start the stream in your browser, if you look at the network connections being made, you will see the URLs for the video segments themselves look like this:


http://www.youtube.com/videoplayback/id/QK6zhpuBZVo.1/itag/96/source/yt_live_broadcast/sq/16178/file/seg.ts?clen=2374440&cmbypass=yes&cp=U0hWSlVRTl9FTkNONl9NS1JIOjRROWhONnoyamJz&cpn=IDN8XKXzApugOWiJ&dg_shard=QK6zhpuBZVo.1_96&dur=4.000&expire=1375560704&fexp=909546%2C906397%2C929117%2C929121%2C929906%2C 929907%2C925720%2C925722%2C925718%2C925714%2C92991 7%2C929919%2C912521%2C904830%2C919373%2C904122%2C9 11423%2C909549%2C900816%2C936301%2C912711%2C935802&gir=yes&hls_chunk_host=www.youtube.com&ip=27.11.23.134&ipbits=8&keepalive=yes&key=dg_yt0&live=1&lmt=1375538334074523&maudio=1&newshard=yes&playlist_type=LIVE&pmbypass=yes&ratebypass=yes&signature=89E3996F481C5A340D46A3EB439EA7EDFA445500 .2D72C767856743E9E8F2BE990EFD7C6AE016EDA2&sparams=ip%2Cipbits%2Cexpire%2Cid%2Citag%2Csource% 2Cratebypass%2Clive%2Ccmbypass%2Cnewshard%2Chls_ch unk_host%2Cgir%2Cdg_shard%2Cmaudio%2Ccp%2Cplaylist _type%2Cpmbypass&sver=3&upn=WuLyorMATMY

Comparing one segment URL to another shows that only a few parts change:
* /sq/16178/file: the segment to retrieve
* clen=2374440: length of the segment, in bytes
* dur=4.000: duration of the segment, in seconds
* lmt=1375538334074523: some sort of timestamp

dur varies between 4 and 6 seconds, but it turns out not to matter, you can remove it and still get the file.
lmt somehow controls how long you can get the file; if this is too high, you can't get the segment. Removing this give an error, but you can just set it to 1 and it will still work.

The tough part is clen, which is required. Set it too low, and you won't get the whole file. Set it too high, and you might get a small portion of the file, but you'll get an error before it completes.

So basically you have to brute force it. Keep trying to get the file with different clen; if it works, increase and try again. Stop when it fails. You can do a binary search to make it more efficient, but it'll still be kinda slow.

You can still grab the individual segments for at least a few hours after the stream finishes, but eventually it'll stop responding entirely.

[ninjaw]

I put some files online @ http://nunatak.sizone.org/~moebius/nin/

It includes my md5, Dave's md5's, as well my copies of 8536 and 8540 which fail the par2 check.

I will also upload ninjaw's par2 file and the full par2verify output (takes a couple minutes to run).

I don't have all the segments downloaded yet (of either the original broadcast or the replay) but I'm getting a ton of par2 errors from 8764 onward, in addition to the individual failures on 8536 and 8540.

EDIT: Here's the par2verify output, starts out well, then lots of errors 8764-9400 (http://nunatak.sizone.org/~moebius/nin/par2.out.txt).

Thanks for these tests, even if i keep thinking it's pointless. So i used your files to test what you both refused to do: unmux to test core files only. Here's what i discovered, one of us are using 17ms shorter videos, and it's not me. So i was wrong, it's not the stupid container, but what is it ? It may be due to the fact i'm not living on the same side of the world.

[ninjaw]

Thanks for these tests, even if i keep thinking it's pointless. So i used your files to test what you both refused to do: unmux to test core files only. Here's what i discovered, one of us are using 17ms shorter videos, and it's not me. So i was wrong, it's not the stupid container, but what is it ? It may be due to the fact i'm not living on the same side of the world.

Oh i upped my two files for ya to check what can be wrong https://www.dropbox.com/sh/o78dicw8sqsp899/blTBl2VYqO

[Moebius]

... and speak of the devil. Time's up, can't get any more pieces. I grabbed 8329-9543, except missing 9532 (part of the credits). I also have a bit of the replay, 15300-15491ish.

[ninjaw]

I do start to upload ritcnin, i consider my file working, even if i may have forgot a bad chunk.

[Moebius]

So i used your files to test what you both refused to do: unmux to test core files only. Here's what i discovered, one of us are using 17ms shorter videos, and it's not me. So i was wrong, it's not the stupid container, but what is it ? It may be due to the fact i'm not living on the same side of the world.

Can you clarify what you mean by "unmux to test core files only"? Are you talking about pulling the individual video and audio data streams out of the mpeg transport container files? It's been a long while since I played with any video internals, not sure what tools I might have to do such a thing.

I'm also not really able to perform a visual inspection of the files on playback to see if one looks better than the other; my poor old laptop is far too wimpy to play these files properly at full resolution.

[ninjaw]

Can you clarify what you mean by "unmux to test core files only"? Are you talking about pulling the individual video and audio data streams out of the mpeg transport container files? It's been a long while since I played with any video internals, not sure what tools I might have to do such a thing.

I'm also not really able to perform a visual inspection of the files on playback to see if one looks better than the other; my poor old laptop is far too wimpy to play these files properly at full resolution.

That's it, i used the freeware tsmuxer, after that i discovered that new par2 won't work too, i used mediainfo to check differences

[Moebius]

Updated my files with the final md5 of all portions I have, new par2 output and a listing of filesizes of all segments.

[Moebius]

Oh geez, just found a nice link about an easier way to snag the segment URLs (http://stefansundin.com/blog/452)... wish I had found this a few hours ago...

[ninjaw]

Oh geez, just found a nice link about an easier way to snag the segment URLs (http://stefansundin.com/blog/452)... wish I had found this a few hours ago...

this is in post #2 of this thread

[cynicmuse]

Oh geez, just found a nice link about an easier way to snag the segment URLs (http://stefansundin.com/blog/452)... wish I had found this a few hours ago...
You still would have had to do some brute forcing. Since the stream wasn't searchable like Coachella's, they were regenerating the .m3u8 playlist every couple of seconds so that it contained 4-5 entries. So you would have had to repeatedly pull the file. I believe that @gstead (http://www.echoingthesound.org/community/member.php?u=3184) was using firebug to do this.

[binaryhermit]

I'd like to point out that there's a skip at about 1:09 into track 4 (1,000,000) on ninjaw's mp4 album from the live stream. Just something that may need to be checked out.

[ItsJustDave]

I'd like to point out that there's a skip at about 1:09 into track 4 (1,000,000) on ninjaw's mp4 album from the live stream. Just something that may need to be checked out.

I think Moebius has a full file set, I'm going to follow up with him and see if we can't sort it out (one way or another). It may be a defect in the original stream.

[binaryhermit]

It may be a defect in the original stream.That's also a possibility.

[binaryhermit]

Is it on this file? https://mega.co.nz/#fmaAwz3LRb
That link doesn't link to a file.

[ItsJustDave]

I captured the stream with Audio Hijack and didn't notice a glitch during 1,000,000. Is it on this file? https://mega.co.nz/#fmaAwz3LRb

At which quality level in youtube? That could make a difference here.

[TheBang]

I'd like to point out that there's a skip at about 1:09 into track 4 (1,000,000) on ninjaw's mp4 album from the live stream. Just something that may need to be checked out.

That's because there are defects in 8536 to 8540, which ninjaw got from me. @Moebius (http://www.echoingthesound.org/community/member.php?u=848), could you post your copies of those, please?
I'm also still looking for 9388. The one ninjaw gave me is the same as the one I captured, which is suspect quality. I'm not certain it was captured properly.

Edit: Actually, I think ninjaw got it from someone else, and it matches mine, so maybe it's OK.


I captured the stream with Audio Hijack and didn't notice a glitch during 1,000,000. Is it on this file? https://www.wetransfer.com/downloads/a5650f9c5b5b00c8a76ecbf6de98f9a720130803233108/b6b68214eabaceee4d8275db3a9f535320130803233108/92f0be
Audio Hijack Pro does recording, not bitstream dumps, so it's not usable for this.

[binaryhermit]

That m4a file seems fine, at least in the general vicinity of where ninjaw's file glitched.

[cynicmuse]

I'm also still looking for 9388. The one ninjaw gave me is the same as the one I captured, which is suspect quality. I'm not certain it was captured properly.
During what song is 9388? I had a glitch at roughly 4:30 into Find My Way that gstead saw as well. If it's there, then it's most likely the source.

[TheBang]

During what song is 9388? I had a glitch at roughly 4:30 into Find My Way that gstead saw as well. If it's there, then it's most likely the source.
It's during HLAH.

It turns out, my 9388 appears to be OK. I really need 8536 to 8540 still though.

[ItsJustDave]

It's during HLAH.

It turns out, my 9388 appears to be OK. I really need 8536 to 8540 still though.

Have you checked your md5s against moebius's list yet?

[TheBang]

Have you checked your md5s against moebius's list yet?
Thanks, I must've glossed over that post. 8536 and 8540 were, in fact, the two corrupt ones, and moebius' are correct. I've stitched them in, and they work perfectly. So, I should have a complete set of chunks now. I will go over all the MD5's to compare, and then do a complete run through of the video file.

[ItsJustDave]

Thanks, I must've glossed over that post. 8536 and 8540 were, in fact, the two corrupt ones, and moebius' are correct. I've stitched them in, and they work perfectly. So, I should have a complete set of chunks now. I will go over all the MD5's to compare, and then do a complete run through of the video file.

Sweet! Let us know how the md5 comparison goes. If they all match, we should be golden!

[TheBang]

OK, so very interesting results from the md5's.

Basically, everything that I had captured (that was uncorrupted) matched moebius' md5's exactly. Some of the missing chunks I had that were filled in by other people (I had about 3 different people contribute pieces to me) also matched moebius. Everything that I got from ninjaw's own capture differed, as moebius pointed out, and they differed from my chunks too, and differed from some of the other people's contributions that overlapped with ninjaw's. So, basically, given that everyone else's captures are consistent, and ninjaw's differs, I think, unfortunately, ninjaw's chunks are not usable. Sorry, ninjaw. When slotted in, the packets actually don't look like there's anything wrong, but quite simply, it appears that they are altered from how they were on the server. I'm assuming this was due to the capture tool. What were you using, ninjaw? Was it Replay Media Catcher? If so, then I think Replay Media Catcher should be avoided altogether for HLS stream capture. Either people were getting the MPEG-2 transcodes, or for these .ts file captures, they're being altered in some manner.

I downloaded my chunks using HiDownload. How did you get yours, moebius? ItsJustDave, all those ones you got from the replay were with Jaksta?

The chunks that were gathered from the following people were all consistent with each other, in terms of the MD5 checksums, so I believe they are accurate captures of what the server actually served:

TheBang
moebius
ItsJustDave
Erik
Vler Schinicius

So, after discarding ninjaw's chunks, and regathering sources, I have the following hole:

9479-9487

If I could get those from you, @Moebius (http://www.echoingthesound.org/community/member.php?u=848), or anyone else, I'd appreciate it.

Additionally, I did have one more chunk which differed from moebius':

TheBang: 056b22a9805248486627bd5fc568b18f 9112
moebius: 8cb5f12efbd291ff404fdb6e01b5870e 9112

So, either mine or moebius' 9112 chunk is corrupt. We need a third source for that one, either an md5 or the actual chunk, to figure out which is correct.

[ItsJustDave]

ItsJustDave, all those ones you got from the replay were with Jaksta?

Yeah, beta Jaksta 1.3.0 (129) for Mac downloaded from http://download.shedworx.com/jaksta/beta/

[Moebius]

That's because there are defects in 8536 to 8540, which ninjaw got from me. @Moebius (http://www.echoingthesound.org/community/member.php?u=848), could you post your copies of those, please?
I'm also still looking for 9388. The one ninjaw gave me is the same as the one I captured, which is suspect quality. I'm not certain it was captured properly.

Edit: Actually, I think ninjaw got it from someone else, and it matches mine, so maybe it's OK.


Here are my
8536.ts (http://nunatak.sizone.org/~moebius/nin/8536.ts)
8540.ts (http://nunatak.sizone.org/~moebius/nin/8540.ts)
9388.ts (http://nunatak.sizone.org/~moebius/nin/9388.ts)

[Moebius]

You still would have had to do some brute forcing. Since the stream wasn't searchable like Coachella's, they were regenerating the .m3u8 playlist every couple of seconds so that it contained 4-5 entries. So you would have had to repeatedly pull the file. I believe that @gstead (http://www.echoingthesound.org/community/member.php?u=3184) was using firebug to do this.

Look closely at that blog post and you'll find he has a trick... you can add "?start_seq=###" to the m3u8 playlist URL to bring back the range you want.

[TheBang]

Additionally, I did have one more chunk which differed from moebius':

TheBang: 056b22a9805248486627bd5fc568b18f 9112
moebius: 8cb5f12efbd291ff404fdb6e01b5870e 9112

So, either mine or moebius' 9112 chunk is corrupt. We need a third source for that one, either an md5 or the actual chunk, to figure out which is correct.
I got another copy of 9112 from Erik. It matched my md5, 056b22a9805248486627bd5fc568b18f, so I guess yours is corrupt, moebius. You can download the 9112 here:

http://we.tl/yqT6SHMwob


Here are my
8536.ts (http://nunatak.sizone.org/~moebius/nin/8536.ts)
8540.ts (http://nunatak.sizone.org/~moebius/nin/8540.ts)
9388.ts (http://nunatak.sizone.org/~moebius/nin/9388.ts)
Thanks! I've got 3 copies of 9388 now, all the same checksum, so I think we're good on that one. I still need 9479-9487, please.

[ItsJustDave]

Here are my
8536.ts (http://nunatak.sizone.org/~moebius/nin/8536.ts)
8540.ts (http://nunatak.sizone.org/~moebius/nin/8540.ts)
9388.ts (http://nunatak.sizone.org/~moebius/nin/9388.ts)

I think TheBang was looking for 8537, 8538, and 8539 as well. And it probably wouldn't hurt to have your take on 9112 either. TheBang should be able to place both in a test sequence and see if either is preferred. Thanks to everyone that's contributed in this endeavor. I've learned a few lessons. I'm wondering if there's any simple way to set things up so that we can focus more on capturing and keeping unique URLs when it's live so that grabbing chunks again later is simplified. Also, md5 FTMFW!

[TheBang]

I think TheBang was looking for 8537, 8538, and 8539 as well.
Nah, I'm good on those, thanks. I just put the whole sequence initially, because I was suspicious, but they match moebius' md5's, and 8536 and 8540 were the only one that noticeably had corruption.

[Moebius]

I downloaded my chunks using HiDownload. How did you get yours, moebius? ItsJustDave, all those ones you got from the replay were with Jaksta?


I grabbed mine with some bash scripts, starting off a legitimate segment URL sniffed from the browser session and then bruting the URLs to reconstruct the individual segment URLs. These were then downloaded straight from YouTube's servers with wget.




So, after discarding ninjaw's chunks, and regathering sources, I have the following hole:

9479-9487

If I could get those from you, @Moebius (http://www.echoingthesound.org/community/member.php?u=848), or anyone else, I'd appreciate it.


Uploading now, I'll PM you the link.

[gstead]

That's because there are defects in 8536 to 8540, which ninjaw got from me. @Moebius (http://www.echoingthesound.org/community/member.php?u=848), could you post your copies of those, please?
I'm also still looking for 9388. The one ninjaw gave me is the same as the one I captured, which is suspect quality. I'm not certain it was captured properly.

Edit: Actually, I think ninjaw got it from someone else, and it matches mine, so maybe it's OK.


Audio Hijack Pro does recording, not bitstream dumps, so it's not usable for this.


That m4a file seems fine, at least in the general vicinity of where ninjaw's file glitched.
---
Not sure if you guys finally got the audio you need? I recorded straight-to-disk-no-encoding from the 720 stream. ffmpeg reports the audio as:


Stream #0:1[0x101]: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 256 kb/s

I extracted this audio stream (again w/o encoding) to a 256 kb/s aac file. If you need it just say so. It is pristine except for the glitch in "Find My Way".

[Moebius]

I got another copy of 9112 from Erik. It matched my md5, 056b22a9805248486627bd5fc568b18f, so I guess yours is corrupt, moebius. You can download the 9112 here:

http://we.tl/yqT6SHMwob

Thanks for the 9112. Seems I somehow cut off the last 5522 bytes, wonder how that happened. There's definitely an audio glitch when playing my 9112-9113 merged together that is not there with yours. Looks like you've got the right one.

[Moebius]

BTW, is there a better way to assemble these besides concatenating all the parts? As I understand it, that's basically what happens when you play the stream normally. I do see a bunch of messages from vlc on playback, but they seem to be harmless:


libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 13) for PID 0
libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 13) for PID 4095
libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 10) for PID 17
libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 14) for PID 0
libdvbpsi error (PSI decoder): TS discontinuity (received 0, expected 14) for PID 4095

[Moebius]

Look closely at that blog post and you'll find he has a trick... you can add "?start_seq=###" to the m3u8 playlist URL to bring back the range you want.

Okay, played with this a bit more on the Saturday stream; it's not entirely a silver bullet, as it will give you at most the last 5760 segment URLs. But that's pretty handy...

[cynicmuse]

Okay, played with this a bit more on the Saturday stream; it's not entirely a silver bullet, as it will give you at most the last 5760 segment URLs. But that's pretty handy...
That's good to know. I hadn't tried it on this m3u8 because the base file that I got when I used wget just had 5 entries. The Coachella m3u8 had the whole list of URLS in it.

edit: As far as joining .ts files, I didn't find something that worked under linux. There's a binary for tsmuxer that's linux compatible but it wasn't working on my system due to multi-arch weirdness. Both tssplitter and tsmuxer worked via wine with no issues.

[ItsJustDave]

BTW, is there a better way to assemble these besides concatenating all the parts? As I understand it, that's basically what happens when you play the stream normally. I do see a bunch of messages from vlc on playback, but they seem to be harmless:

There's a tool called tssplitter (Windows only it appears) that appears to have been used to join other ts rips I've just looked at on dime. I fired up parallels and used it to join the ts files I've got and it was bit for bit the same as doing cat src/*.ts > dest/merged.ts I'm not sure if there's any other magic to it in specific scenarios. I'm fairly new to .ts files

[ninjaw]

I'd like to point out that there's a skip at about 1:09 into track 4 (1,000,000) on ninjaw's mp4 album from the live stream. Just something that may need to be checked out.

It appears 8540 were slighty shorter I've been sent a fix

[ninjaw]

I'm trying to get back a matching chunk set too (in another folder just in case, i do like my europpean chunks, am i the only one in europe ? it could explain that) so the moebius md5 is a ref ? Here's an idea : to help anybody, as soon of someone get a full set that fits you could you produce a PAR2 file ? not just a PAR2 file as i did before, but a PAR2 file with 10% repair files, that way it will fix anyone's set (well with less than 10% errors).
The freeware is called Quickpar, the tool is widely used and adopted in NG world, it's made for large number of files like this, but usually larger. Maybe we could reduce the blocs size before creating the par2 set, we are working with tiny errors on tiny files
Edit: if you're in, you can offer more than 10% fix, the tool was made so we don't have to download all the par2 set, but just what we need to fix

[ninjaw]

Uploading now, I'll PM you the link.

Could you send me it too ? so we have to recreate a verify tool for 9112 ?

PS: btw fact: the only glitch found in my M4A set wasn't from my european chunks

[ninjaw]

Actually upping version 2 of the perfectly captured Lollapalooza M4A Album, a very small glitch were found on 1000000 (1'09) it appears I had a very small shorter version on a chunk, so small i couldn't tell myself, when the big video join job tell me a warning at 16% but now i'm very confident of this version as i didn't had any warning, so confident i filled for you the titles and created a very fast cover https://www.dropbox.com/sh/3z0dcv1yrwtfw46/LvDdWDm68m
(It's directly cut from the original stream perfectly captured and kept in the original youtube compression, album was 100% recreated, cut arent the same, trash the other one)

[Moebius]

There's a tool called tssplitter (Windows only it appears) that appears to have been used to join other ts rips I've just looked at on dime. I fired up parallels and used it to join the ts files I've got and it was bit for bit the same as doing cat src/*.ts > dest/merged.ts I'm not sure if there's any other magic to it in specific scenarios. I'm fairly new to .ts files

Yeah, I wasn't sure if the 'proper' ts tools ended up doing anything different. mpeg-ts is made to be streamed, after all.

[ninjaw]

I've been able to download the moebius version, I am actually creating a PAR2 set on moebius set including the 9112 fix, I chose a 740Ko block size, quickpar told me it was the best choice (3 to 4 bloc per file it means it doesnt need to repair 50% of the file for a small glitch but 25%) I'm not sure it will be useful but if anyone else try to collect them all: here you go.
Can i upload to ritc ? I already have a chapter file ready to insert.

[ninjaw]

here is the repair set (I got a lot more if needed) https://www.dropbox.com/sh/o78dicw8sqsp899/blTBl2VYqO
the complete set can repair 10% of corrupted files, that's 204 creation of files, it also can rename files automaticaly

[skullboy0]

Actually upping version 2 of the perfectly captured Lollapalooza M4A Album, a very small glitch were found on 1000000 (1'09) it appears I had a very small shorter version on a chunk, so small i couldn't tell myself, when the big video join job tell me a warning at 16% but now i'm very confident of this version as i didn't had any warning, so confident i filled for you the titles and created a very fast cover https://www.dropbox.com/sh/3z0dcv1yrwtfw46/LvDdWDm68m
(It's directly cut from the original stream perfectly captured and kept in the original youtube compression, album was 100% recreated, cut arent the same, trash the other one)

Just an FYI, but I was going to put this on my iPhone to listen to later, & The Way Out is Through is listed as The way out is out.

[ninjaw]

Just an FYI, but I was going to put this on my iPhone to listen to later, & The Way Out is Through is listed as The way out is out.

Oh shit lol thank you its corrected and uploading

[phix95]

I didn't read thru all these, but I use Flashgot to capture streams. Once the stream is playing you'll see the built in browser icon blink, just click it and select the quality you want it to save.

[theimage13]

I didn't read thru all these, but I use Flashgot to capture streams. Once the stream is playing you'll see the built in browser icon blink, just click it and select the quality you want it to save.

So...did you use it for this? Do you have a rip? Have you made glitch free rips of long broadcasts with this software before? Another option can't hurt!

[phix95]

I'm sorry I didn't. It was the plan. It was all queued in my browser and I closed the window by accident. I tried to find where it would have been saved but couldn't. I was SO pissed I messed it up.
I had Fraps running to to capture screenshots as well but that didn't work either. (Just figured out there's a setting in Fraps to record the desktop (and hence, streams) that I didn't know I had to set, doh)

[ninjaw]

Flashgot would probably not work (who use firefox these days ?), even if it worked, can I remember you that we do have a perfect capture ? it has been sent to ritc, than we found a way to download it chunk by chunk and verify it between several people, and even a way to download bad chunks after the live stream ? This method gave probably best results that any live direct capture. PS: this was not made by me, but others, i calle dthat an ETS collective.

[ninjaw]

To Dave & Moebius: Yes TS is a container made especially for streaming and TV (and bluray's m2ts seems to be the same) so basically the thing is made for easy cut & join ( and even live recompression ) we can find lots of unknown stuff inside, that's why i suspected first the container about my differences. I did join my files with MKVmerge, because I know it produced lots of good results for me.

[phix95]

http://www.spin.com/articles/nine-inch-nails-lollapalooza-100-minute-performance-video/

On Trent's twitter. At least you can see the whole thing. And why the hate for FF?? I'll try to dl it there. But that's great if you already have it.

[gstead]

http://www.spin.com/articles/nine-inch-nails-lollapalooza-100-minute-performance-video/

On Trent's twitter. At least you can see the whole thing. And why the hate for FF?? I'll try to dl it there. But that's great if you already have it.

Same glitch in Find My Way. So that problem was definitely upstream of many of us :)

PS: I use FF all day every day!

[ninjaw]

perfect video is now available on ritc, you can now stop watching crappy trent's captures, and verify your glitch.

[ninlive]

perfect video is now available on ritc, you can now stop watching crappy trent's captures, and verify your glitch.

Again all, thanks a whole bunch for all the time invested in getting a perfect cap of the performance. I truly do appreciate it, as I know nothing about video files and what are the best ways to go about it. So a big THANK YOU! :)

[Moebius]

So the next show at Outside Lands is apparently going to be streamed via Ustream, anyone have experiencing capping from that? I'm assuming it's going to be a completely different protocol than HTTP Live Streaming (like on YouTube).

[ninjaw]

yeah ? why that ? don't everybody use flv/h264 ? I dont know about professional live stream, i tried one of them right now, it simply works by chunks into the freeware Tubemaster+, looks like it should be lot easier

[cynicmuse]

It looks like rtmpdump along with a python script to get the rtmp url will work (http://askubuntu.com/questions/170471/how-do-i-record-the-nasa-hd-tv-feed), at least if the live streaming is like the live streaming on the NASA HD channel. It also looks like "high quality" is 720p.

[ninjaw]

Actually upping to @RITCNIN (http://www.echoingthesound.org/community/member.php?u=1141) Lollapalooza m4a album version 3 : after version 1 having a glitch track 4 1'09 and version 2 missing a chunk track 4 1'04, this version is directly cut from the ritc video version to avoid any chunk issue. My dropbox was deleted

[TheBang]

Sziget Festival has been broadcasting the past few days, and it doesn't appear to be RTMP. I have figured it out yet:

http://www.ustream.tv/simple-sziget2013

And, of course, that doesn't necessarily mean Outside Lands will be the same either. Fortunately, we'll have all day Friday and Saturday to practice on Outside Lands.

[cynicmuse]

I checked some other live channels on ustream and I think that they're using rtmp for communicating which flv chunk to get and then http to serve the chunk. Hopefully this is not the method that the festival is using.

The first chunk looks like this:
hxxp://flash59.ustream.tv/sjc-uhs07/streams/httpflv/ustreamVideo/459804/streams/live_1_1375970762_763844482.flv?start=19558
while subsequent chunks look like this:
hxxp://flash59.ustream.tv/sjc-uhs07/streams/httpflv/ustreamVideo/459804/streams/live_1_1375970763_763844482.flv
though the number that repeats changes after a few times.

The first chunk has the correct header for a .flv file while the subsequent files don't. I have no idea how to recombine these files. The ones that I got with wget I could combine with cat though mplayer is complaining about there being no frame at the joins I think. The files look ok if I just step through frame by frame. I couldn't join the files that I got with jaksta.

[TheBang]

Thanks for the info, cynicmuse. That tidbit about the first chunk containing the FLV headers helped me crack it.

So, they currently have a promo video running on a loop at the official channel:

http://www.ustream.tv/outsidelands

I couldn't find a manifest file, so in order to download the chunks, I had to use something like Replay Media Catcher or HiDownload in order to sniff and capture the HTTP chunk downloads. This means that you have to play the stream in realtime while downloading it with your downloader, so you need twice the amount of bandwidth as the stream requires.

I don't think you'll necessarily be able to download the chunks later. The last string of numbers in the filename appears to change to a randomized number every 10 chunks or so. So if you miss more than 10 chunks in a row, you may not be able to reconstruct the download URL for the missing chunks.

That said, I was successful with downloading the chunks in realtime with the promo stream. The key is finding the first chunk that has the FLV header. You should be able to find it just by looking at the lowest number, but you can find the filename for the first chunk by looking at the HTTP requests (say, in Firefox Web Console) and filtering for "start" (as in cynicmuse's example).

Once you find the first chunk, then you can simply concatenate the rest of the chunks onto it sequentially. This led to some glitches between the chunks sometimes; however, I was able to fix the concatenated FLV file by running it through FlvFixer.php here:

https://github.com/K-S-V/Scripts/blob/master/FlvFixer.php

The resulting fixed FLV was able to be played without error, and I was able to successfully rewrapper it into MPEG-4 also.

The "Best" quality stream (at least on this promo loop) is 1080p30 AVC at about 4 Mbps, and the audio is 44.1 KHz LC-AAC at about 125 Kbps.

So, you'll need about 10 Mbps of Internet bandwidth to download the 1080p version in realtime, unless a manifest can be found.

[cynicmuse]

Thanks for the info on FlvFixer TheBang; that fixed all of my problems. Jaksta wasn't working because the demo only gets half of the file since it's not a youtube video. Replay Media Catcher got the files without a problem.
Also, I have more than 10 Mbps of bandwidth, so I won't have any problems with a realtime download.

[kargath]

Thanks for the info on FlvFixer TheBang; that fixed all of my problems. Jaksta wasn't working because the demo only gets half of the file since it's not a youtube video. Replay Media Catcher got the files without a problem.
Also, I have more than 10 Mbps of bandwidth, so I won't have any problems with a realtime download.

What version of RMC are you using? Just curious. I also have well more than enough bandwidth, so assuming i'm around home at the time, I'll try capturing as well. I'm still using RMC 4 though, so not sure if that'll work. I bought HiDownload though, so maybe when I get home tonight I'll give that one a shot on the looping video to see what happens.

[ItsJustDave]

I'll do what I can to try and capture chunks Saturday night. We'll be on vacation (and hotel wifi), but I'll try.

I'd suggest that we md5 the original flv files for comparison and keep them around at least until we build a quorum on good/bad. I'd expect that we should see consistent md5s from these.

[cynicmuse]

What version of RMC are you using? Just curious. I also have well more than enough bandwidth, so assuming i'm around home at the time, I'll try capturing as well. I'm still using RMC 4 though, so not sure if that'll work. I bought HiDownload though, so maybe when I get home tonight I'll give that one a shot on the looping video to see what happens.
I'm using RMC 5.

re: md5 for sanity checking... weren't there discrepancies between files downloaded with RMC vs files downloaded with HiDownload or Jaksta? I checked the files that I got from RMC with a hex editor and didn't see any weird shit at the beginning of the files. I can't really tell about the end of the file because there doesn't appear to be any common ending. I'll install wget for windows and compare the files from wget and RMC tonight.

[ninjaw]

what about names ? only hidownload is keeping original names, i can't do it on tubemaster or rpc5

[cynicmuse]

what about names ? only hidownload is keeping original names, i can't do it on tubemaster or rpc5
RPC5 will save the files in a format close enough to the original file name. Under the Naming section of the Internet preferences tab (using the advanced settings on startup), there's an area with the following header:
"Rules are applied in the order listed. The first text returned becomes the file name."
I unchecked the following boxes:
Enable YouTube Naming
Enable Playlist Naming
Enable Referrer Naming
Enable HTTP Title Naming
I left the following four boxes checked:
Enable URL Parameter Naming
Enable URL Path Naming
Enable Meta Data/Tags Naming
Automatically Identify and Tag Music

Using these settings, a file returned by RPC5 has the following name: live_2_1376016796_1948999929-flv.flv
The same file returned by wget has this name: live_2_1376016796_1948999929.flv
Then it's just a matter of batch renaming the files. I can't help with tubemaster.

[Moebius]

Play around with the Friday stream. Downloading these flv is easy enough but vlc doesn't seem to want to play them.

[Moebius]

I'm seeing filenames like this:


live_1_1376080428_981725227.flv
live_1_1376080429_981725227.flv
live_1_1376080430_2073116643.flv
live_1_1376080431_2073116643.flv
live_1_1376080432_2073116643.flv
live_1_1376080433_2073116643.flv
live_1_1376080434_2073116643.flv
live_1_1376080435_2073116643.flv
live_1_1376080436_2073116643.flv
live_1_1376080437_2073116643.flv
live_1_1376080438_2073116643.flv
live_1_1376080439_2073116643.flv
live_1_1376080440_1374003941.flv
live_1_1376080441_1374003941.flv

the second number is clear enough, it's a unix timestamp going by one. The last number seems to be random and changes every 10 files. Can't figure out yet how the client is getting that information.

[ninjaw]

Play around with the Friday stream. Downloading these flv is easy enough but vlc doesn't seem to want to play them.

Same thing, cannot do a thing with files

[cynicmuse]

This stream isn't like youtube's stream; youtube served up ts chunks, each of which had the proper header info to be a ts file, as confirmed by mediainfo. If you use mediainfo (or a hex editor) on any chunk that didn't have a start associated with it (aka the first chunk that you acquired), you get nothing. If you look at the files that have a start associated with them, they have the proper FLV header; the first three letters of the binary file are FLV, followed by lots of related info. Here are the full specifications (http://download.macromedia.com/f4v/video_file_format_spec_v10_1.pdf)from Adobe and here is a hexadecimal example (http://livertmpjavapublisher.blogspot.com/2011/06/h264-avcvideopacket.html) with the relevant portions highlighted. You may be able to create some kind of fake header for the non start files, but you'd have to remove the junk that's on the front end first I think. It's rather evident if you look at the file with a hex editor, but I'm not sure where the junk ends and the real data begins.

I'd say that it's easier to just start the stream five minutes before the set starts, acquire the full set of chunks, merge the chunks (note, as long as you have the file with start associated with it and the rest of the files sequentially, this will form a complete stream), fix the flv file with the FlvFixer, and then remove the .mp4 file from the flash encapsulation. The mp4 file can be edited to remove the extra data. Comparing files will be harder this way.

[ninjaw]

This stream isn't like youtube's stream; youtube served up ts chunks, each of which had the proper header info to be a ts file, as confirmed by mediainfo. If you use mediainfo (or a hex editor) on any chunk that didn't have a start associated with it (aka the first chunk that you acquired), you get nothing. If you look at the files that have a start associated with them, they have the proper FLV header; the first three letters of the binary file are FLV, followed by lots of related info. Here are the full specifications (http://download.macromedia.com/f4v/video_file_format_spec_v10_1.pdf)from Adobe and here is a hexadecimal example (http://livertmpjavapublisher.blogspot.com/2011/06/h264-avcvideopacket.html) with the relevant portions highlighted. You may be able to create some kind of fake header for the non start files, but you'd have to remove the junk that's on the front end first I think. It's rather evident if you look at the file with a hex editor, but I'm not sure where the junk ends and the real data begins.

I'd say that it's easier to just start the stream five minutes before the set starts, acquire the full set of chunks, merge the chunks (note, as long as you have the file with start associated with it and the rest of the files sequentially, this will form a complete stream), fix the flv file with the FlvFixer, and then remove the .mp4 file from the flash encapsulation. The mp4 file can be edited to remove the extra data. Comparing files will be harder this way.

Nope, no flv header nowhere, for me, on a concert, i did have that on a non concert

[ninjaw]

Actually playing with GetFlv, quite handy overpriced tool offering a complete collection of tools all inside one app, include, surf with several kernel (IE, chrome..), flv link detection&download, flv fix, flv dirty convert, Flv turbo multithread, any video dirty capture mode, and even mp3 extraction. (warning, cannot find an auto download, this tool may asking for manual download every 4secs)
And i noticed something disturbing : i had a "start" url, and no more since 200 chunks and counting ?

[TheBang]

I was able to successfully capture a 30 minute test run on yesterday's stream, using the procedure outlined above. As cynicmuse says, the first chunk will have "FLV" in the first 3 bytes. So, whatever your start chunk is, it will differ from everyone else's. After that though, I confirmed that the rest of the chunks are all identical. I downloaded the stream on 3 different computers in two different geographic regions, and the chunks all had the same names and same MD5 checksum.

So, as with last time, it could be useful to compare MD5 sums with different people and exchange chunks. With my 3 computers ready to go, I'll probably be able to capture the full show without errors myself though.

As with last time, I tried Replay Media Catcher, and it is mangling the chunks. HiDownload and manual downloading were consistent with the MD5 checksums, and Replay Media Catcher did not match anything else. So anyone using RMC will not be able to contribute chunks to others, and, more importantly, the chunks will not represent the true data as provided by the server.

Confirmed the previous information too. Average video bitrate at about 4 Mbps, though I saw it was variable, running from about 3 to 5 Mbps. Audio bitrate at 126 Kbps.


And i noticed something disturbing : i had a "start" url, and no more since 200 chunks and counting ?Yeah, you only get one "start" URL (that contains the FLV header), and that chunk will be unique to your capture.

[ninjaw]

I agree with you TheBang, Succesfully made a 3minutes perfect non glitchy capture with Hidownload.
However, Hidownload still crashing like hell, do you have a solution about that ?

Can you record right now so we could test MD5 and names ?

[TheBang]

I agree with you TheBang, Succesfully made a 3minutes perfect non glitchy capture with Hidownload.
However, Hidownload still crashing like hell, do you have a solution about that ?
I do not. My solution is recording on 3 different computers. :)


Can you record right now so we could test MD5 and names ?
I'm going to bed very soon, but I can record for you on Channel 1 in a couple minutes.

Edit: Here are a few chunks from just now

MD5 (live_1_1376088056_111666218.flv) = 87f967bb60dece8ae6ba2c3ab48b88d6
MD5 (live_1_1376088057_111666218.flv) = 91a7bfb02461d41608ebbbe6476d06d3
MD5 (live_1_1376088058_111666218.flv) = c435a708121ff1b71e66afc0a026ee4e

[ninjaw]

I own before crash
live_1_1376087983_1805441976 to live_1_1376088158_2115393032
sorry i dunno md5 but here's a par2 (minor my first one) https://www.dropbox.com/sh/n9n75aijs6gth0d/LxBoZgqq4-
note: this time the first chunk sucked
edit: your names and md5 matched

[ninjaw]

PEOPLE I may have good results with HiDownload version 8.0.0 (http://hidownload.fr.uptodown.com/old) (on this shitty site, REFUSE the main forced download and click on Alternative download without the Download Manager)This one isn't crashing, you don't have to register, succesfully recorded (pefectly) 400 chunks and produced a 30min video files, all in one pass, and only one computer (lol)
edit : ShIT by writing this i just crashed like always (hint : stay on the unfinnished download tab, if you see a lot of them, you crashed), however, by restarting HiDownload and restarting all unfinished downloads, I managed to loose only ONE CHUNK on 500.

[ninjaw]

SUPER HINT 2 ? : I just realised I could recreate the url link ? even better : StreamingStar is offering URLHelper tool you can launch it i the same time, it will record all url (it doesn't record actual files so no bandwith) in case of crash: restart all unfinished downloads, miss a chunck ? right click in url helper and resend info directly into hidownload.

[Moebius]

Yeah, I've found once you have the URLs, it is very easy to download them; you don't need any special parameters or cookies and they aren't extremely time-limited or IP-restricted.

[ninjaw]

Actually it's very time-limited. Another hint : you know what's happening when you press play pause very fast ? usually it produces a double chunk, of course the second one will be named (1)
...So what ? So you can almost safely create starting point when you want.

[ninlive]

Good luck to y'all who are a lot smarter than I and understand how to cap these streams! Again, if you want to upload to the server, the FTP login info is still the same. :)

[cynicmuse]

I have a continuous run from live_2_1376095814_1277304216 to live_2_1376096968_1943445083, the whole show. I used Jaksta and compared the output with that from wget; they had the same md5sum.

Edit 1: I've got some video glitching during Closer off and on; the audio keeps playing. I think that this corresponds to 1376096367-1376096437 with some padding.

Edit 2: I've got similar video glitching during Only off and on. I think that this corresponds to 1376096712-1376096783 with some padding.

Edit 3: And again during THTF.

Edit 4: After watching the whole thing, I've got some doubts about the quality, specifically in the songs listed above. What was weird was that ffmpeg converted it from flv to mp4 without complaint, and it usually spews warnings/errors if something is bad. I'll wait and see what TheBang gets.

[kargath]

I have a continuous run from live_2_1376095814_1277304216 to live_2_1376096968_1943445083, the whole show. I've got the md5sums here (https://www.dropbox.com/s/l95tyqgqsyn2w1z/checksums.md5). I used Jaksta and compared the output with that from wget; they had the same md5sum.

I captured to 96548 before it crapped out. I didn't compare all of our md5s, but the 30 or so that I spot checked all matched...

[ninjaw]

Sorry i slept again. If you want help for anything, like muxing, chapters, audio, i'm here. There's a rerun in 4 hours right ?

[TheBang]

I've got 3 sets of chunks, plus the md5's from cynicmuse to compare, so I should easily be able to determine/assemble a full, uncorrupted copy in a couple hours.

[cynicmuse]

Sorry i slept again. If you want help for anything, like muxing, chapters, audio, i'm here. There's a rerun in 4 hours right ?
Gary Clark Jr. went on around 12:46am PDT, so NIN should start at about 4:43am PDT, based upon their quoted set time starts of 4:28pm and 8:25pm.

[ninjaw]

I managed to download a very good version of the replay here's a par2 for the replay chunks (https://www.dropbox.com/sh/n9n75aijs6gth0d/LxBoZgqq4-). All by myself with one computer and the method described.

[kargath]

I managed to download a very good version of the replay here's a par2 for the replay chunks (https://www.dropbox.com/sh/n9n75aijs6gth0d/LxBoZgqq4-). All by myself with one computer and the method described.

I'm stumbling on one part of this overall method... how does flvfixer work?

[TheBang]

Just FYI, the video's done already and uploaded to RITC server. Will be distributed later this evening by Ryan,


I'm stumbling on one part of this overall method... how does flvfixer work?
You need a PHP interpreter installed in order to run the script. If you're on a Mac, it's already installed, just run it from the Terminal. If you're on Linux, it might be installed, or you can install it quickly with your package manager. If you're on Windows, you can download and install it from here: http://windows.php.net/

[ninjaw]

I'm stumbling on one part of this overall method... how does flvfixer work?

I don't know, but you don't need it at all... what chunk do you miss ?

[TheBang]

Good news, Germany's concert is confirmed streaming live here http://liveweb.arte.tv/fr/video/Rock_n_Heim_2013___Nine_Inch_Nails/ this is a free live website made by a national TV you can find both in france & germany, quite good quality but no HD (yes the HD stream is not HD at all) According to what's printed, only interviews will be available several days, not the main stream. Also, this site is known to stream Rock en seine too. Note: This site is well known by CapTvTy and Jdownloader freeware.
Nice! Based on their archived streams, it's a standard RTMP stream, which means it's a pretty straightforward capture. And unlike HLS, it means that if you have any hiccups in your stream, you can't repair it later, unfortunately. And you weren't kidding when you said the "HD" wasn't HD. At 720x408 at 1 Mbps, it's not even full SD. At least the audio is decent at 160 Kbps AAC.

[cynicmuse]

I'm a little annoyed with these festivals that will broadcast the video in 1080p and then only put a ~128 kbps audio stream with it. Coachella at least had a ~251 kbps audio stream with the 1080p stream.

[ninjaw]

here's a recent festival where they archived full concert : http://oyafestival.arte.tv/fr/#/portfolio/beach-house-fr (they clearly stated they won't archive concerts in germany). I have a story about this, I actually recorded a live stream ... where i were, it sounds incredible, but i let a VNC opened, and the first band just posted an instagram from backstage, next to an internal wifi password notice. So i could launch a Tubemaster++ at the same time. However it captured a very damaged FLV file.

[kargath]

Given that NIN seems to be on every webcast available, I'd say there's a good chance they'll be on this one tomorrow http://www.skynet.be/nl/festivals/pukkelpop-2013/live-stream

They haven't listed which artists will be on the stream yet, but it looks like they'll be updating that link with a little icon next to each band as they get confirmation.

[ninjaw]

We know that, it never worked so far, and it's actually printed that you need to have a belgian contract to see high quality.
Rock en seine just announced streaming partners, it looks bad, strangeley arte live web is still not announced, we may have an audio stream on LeMouv. It could be a video stream on Dailymotion, but "culture box" already announced their four streams, there's no nin.

[ninjaw]

So, pukkelpop, starts in 2 hours, a non live video appeared.. it's fucking silverlight

[JessicaSarahS]

So, pukkelpop, starts in 2 hours, a non live video appeared.. it's fucking silverlight

:mad: Uuuugh.

[ninjaw]

i do obtain 1MB MP4 chunks with RMC5, but i can't replay them

[ninjaw]

I've been able to capture perfectly the intro video, using a msdos command+manifest url

http://md5decrypt.altervista.org/ismdownloaderavc1.htm

code I used :

ismdownloaderAVC1.exe http://195.238.15.230/session/6618815141587427004/vod/festivals/FR_midroll_rock_150_MASTER.ism/Manifest GiveMeAVideo.mkv

[ninjaw]

the fucking skynet hi quality streaming logo is now active, it asks for a Belgacom e-Services login/password

[ninjaw]

HOLY CRAP I've been given a belgian login

[JessicaSarahS]

HOLY CRAP I've been given a belgian login

Whew! I hope all goes well.

[TheBang]

Anyone else have a Belgacom e-Services login they can lend for the night? The more people we have capturing, the more likely there is to be a successful and complete capture.

[ninjaw]

Live is ON ! trying both streams... manifest are really easy to get, it's on every chunk, looks like they are sending AAC+AVC chunks UNMUXED they can be downloaded by hidownload but not by default

[ninjaw]

HOLY ShIT

Duration: LIVE
Timeshift: 00:01:00


Stream 1:
AVC1 640x360 (640x360) @ 900 kbps [selected]
AVC1 480x270 (480x270) @ 600 kbps

Stream 2:
AAC 2 channels 48000 Hz @ 129 kbps [selected]


Press any key to stop recording!
Recording duration:
ERROR: Encrypted content (DRM)!

[ninjaw]

It looks like the only difference betwen free and paid stream is the bitrate, I have no ideas how to get id of these drm

[Fred]

Will screen capturing work? At very least, you should be able to get high-bitrate audio, right?

[ninjaw]

I don't know what tool to use for crappy captures, check previous posts, there's no audio difference between free and paid