Perl and Windows 2000 file size limitations

Discussion in 'Archived Threads 2001-2004' started by John_Berger, Oct 3, 2002.

  1. John_Berger

    John_Berger Cinematographer

    Joined:
    Nov 1, 2001
    Messages:
    2,489
    Likes Received:
    0
    I recently created a Perl script that does modifications to MPEG-2 files without re-rendering. It works exactly as expected with filesizes less than 2 GB; however, once it hits that 2 GB point, the destination file is closed and the program stops.

    I have MPEG-2 files that are larger than 2 GB that work without a problem, which surprised me. Some of my MPEG-2 files are over 4 GB and Windows will play them like there is nothing wrong. So, this issue is a bit disconcerting.

    Does anyone have any idea to break the 2 GB barrier through Perl so that I can get this blasted thing to work properly?
     
  2. Steven K

    Steven K Supporting Actor

    Joined:
    Jan 10, 2000
    Messages:
    830
    Likes Received:
    0
    Hi John

    It probably has something to do with the fact that, on Windows, integers are stored as 32-bit signed values. Values can range from -2147483647 to 2147483648... once you hit a file size larger than 2,147,483,648 bytes, some integer value is probably choking because of an overflow.

    At least this is my guess...
     
  3. Wayne Bundrick

    Wayne Bundrick Cinematographer

    Joined:
    May 17, 1999
    Messages:
    2,358
    Likes Received:
    0
    Let me guess... You're twiddling bits in the MPEG sequence headers to change the aspect ratio from 4:3 to 16:9? [​IMG]
    Steven's right. The Perl engine that executes the script is using 32 bits to measure file size and position, and 32 bits will max out at 2 GB if it's signed or 4 GB if it's unsigned.
    If you could split the file into 2 GB chunks and process each one, then merge them back together... but of course you'd have to do that using a compiler or script engine other than Perl, and in that case you might as well use it to do the original task.
     
  4. John_Berger

    John_Berger Cinematographer

    Joined:
    Nov 1, 2001
    Messages:
    2,489
    Likes Received:
    0
     
  5. Steven K

    Steven K Supporting Actor

    Joined:
    Jan 10, 2000
    Messages:
    830
    Likes Received:
    0
    If you have a few hundred bucks lying around, you can go to eBay and buy an older model SPARC Solaris box, such as an Ultra 5. Solaris is a 64-bit OS, but also keep in mind that the Sparc chip used high-order (big-endian) byte ordering, so you would also have to incorporate some kind of byte-swapping routine into your Perl script if your MPEG files contain values larger than 1 byte (which I'm assuming they do, but I'm not very familiar with the file format).
     
  6. Ken Chan

    Ken Chan Producer

    Joined:
    Apr 11, 1999
    Messages:
    3,301
    Likes Received:
    0
    It's not the 32-bit integer size per se, but rather the API used to access the file is expecting it. There is probably a newer API call that takes a 64-bit value, given that Windows supports files bigger than 4GB.

    A newer version of Perl might support it already, or you could switch to some other language that does.

    //Ken
     
  7. Wayne Bundrick

    Wayne Bundrick Cinematographer

    Joined:
    May 17, 1999
    Messages:
    2,358
    Likes Received:
    0
    It's not your processor or Windows 2000 which can't handle the 64-bit values to get a file larger than 4 GB, obviously they can because you already have a gi-normous file on your hard drive. The problem is the Perl engine which is not using the 64-bit capable API calls.
     
  8. Steven K

    Steven K Supporting Actor

    Joined:
    Jan 10, 2000
    Messages:
    830
    Likes Received:
    0
    Ken is correct, its a problem with size of the variable that Perl uses to store file size information. The CC++ Windows API stores these values as a DWORD, which a 32-bit UNsigned variable.
     
  9. John_Berger

    John_Berger Cinematographer

    Joined:
    Nov 1, 2001
    Messages:
    2,489
    Likes Received:
    0
    I wonder if Perl under Solaris x86 would work. (I'm NOT having a 125 MHz Sparc 10 try to modify a 4 GB file. I might as well come back in a week to see if it's done. [​IMG] )
    I could look to other languages, but it's kind of silly to learn a whole new language that I don't want to learn just for this one tool.
    I'll see what options are out there. Thanks, guys!
     

Share This Page