Problem with streaming PGPCompressedData files

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with streaming PGPCompressedData files

John Rousseau
I'm new to the BC code, so I may just not be calling it correctly, but
I'm seeing a problem where the code is reading past the end of the
current packet (and file).

We have code that creates a TAR file and wraps that in a binary
PGPLiteralData we then optionally compress and/or sign that literal data
and stream it out. When we read the stream, the first object appears to
be read correctly off the stream, but the second object can't be read
because we are either in the middle of a subsequent object or at the end
of the stream. It appears that the BCPGInputStream which is wrapped by
an InflaterInputStream which is created by the PGPCompressedData just
reads (via the fill() call) until the current buffer is full with no
regard to the length of the current packet.

We create a new BCPGInputStream for each object read from the stream, so
if the data is cached at some lower layer, we will lose it.

I see the same problem with ZLIB and ZIP.

I've scanned the list archives and looked closely at the
SignedFileProcessor example, but I can't figure out if we're doing
something wrong, or there's a BCPG problem.

If I hand our stream to gpg on my Linux system, it can decode it just fine.

Thanks for any pointers.
-John

--
John Rousseau
Archivas, Inc.
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Problem with streaming PGPCompressedData files

David Hook-4

I think the problem might be the extra BCPGInputStreams that are being
created - it sounds like the code should be using a single
PGPObjectFactory created off the compressed stream and read each object
from that. In some circumstances BCPGInputStreams have to rely on look
ahead, for that reason opening multiple ones on a single stream where
one would do will not always work.

Regards,

David

On Tue, 2005-08-23 at 14:44 -0400, John Rousseau wrote:

> I'm new to the BC code, so I may just not be calling it correctly, but
> I'm seeing a problem where the code is reading past the end of the
> current packet (and file).
>
> We have code that creates a TAR file and wraps that in a binary
> PGPLiteralData we then optionally compress and/or sign that literal data
> and stream it out. When we read the stream, the first object appears to
> be read correctly off the stream, but the second object can't be read
> because we are either in the middle of a subsequent object or at the end
> of the stream. It appears that the BCPGInputStream which is wrapped by
> an InflaterInputStream which is created by the PGPCompressedData just
> reads (via the fill() call) until the current buffer is full with no
> regard to the length of the current packet.
>
> We create a new BCPGInputStream for each object read from the stream, so
> if the data is cached at some lower layer, we will lose it.
>
> I see the same problem with ZLIB and ZIP.
>
> I've scanned the list archives and looked closely at the
> SignedFileProcessor example, but I can't figure out if we're doing
> something wrong, or there's a BCPG problem.
>
> If I hand our stream to gpg on my Linux system, it can decode it just fine.
>
> Thanks for any pointers.
> -John
>


Reply | Threaded
Open this post in threaded view
|

Re: Problem with streaming PGPCompressedData files

John Rousseau
Yes, that's correct. Thanks for the info.

We're now running into an issue that is explained by the "compressed
data generator" thread on this list. We're just reading compressed data
packets until EOF on the stream.

We'll be happy to beta test the new format compressed data changes as
soon as you have them. :-)

Thanks
-John


David Hook wrote:

> I think the problem might be the extra BCPGInputStreams that are being
> created - it sounds like the code should be using a single
> PGPObjectFactory created off the compressed stream and read each object
> from that. In some circumstances BCPGInputStreams have to rely on look
> ahead, for that reason opening multiple ones on a single stream where
> one would do will not always work.
>
> Regards,
>
> David
>
> On Tue, 2005-08-23 at 14:44 -0400, John Rousseau wrote:
>
>>I'm new to the BC code, so I may just not be calling it correctly, but
>>I'm seeing a problem where the code is reading past the end of the
>>current packet (and file).
>>
>>We have code that creates a TAR file and wraps that in a binary
>>PGPLiteralData we then optionally compress and/or sign that literal data
>>and stream it out. When we read the stream, the first object appears to
>>be read correctly off the stream, but the second object can't be read
>>because we are either in the middle of a subsequent object or at the end
>>of the stream. It appears that the BCPGInputStream which is wrapped by
>>an InflaterInputStream which is created by the PGPCompressedData just
>>reads (via the fill() call) until the current buffer is full with no
>>regard to the length of the current packet.
>>
>>We create a new BCPGInputStream for each object read from the stream, so
>>if the data is cached at some lower layer, we will lose it.
>>
>>I see the same problem with ZLIB and ZIP.
>>
>>I've scanned the list archives and looked closely at the
>>SignedFileProcessor example, but I can't figure out if we're doing
>>something wrong, or there's a BCPG problem.
>>
>>If I hand our stream to gpg on my Linux system, it can decode it just fine.
>>
>>Thanks for any pointers.
>>-John
>>
>
>
>

--
John Rousseau
Archivas, Inc.
[hidden email]