[fitsbits] reopening of Public Comment Period on the compression conventions

[Joe Hourcle]

On Thu, 14 Jan 2016, William Thompson wrote:

> Since we've now moved into the petabyte era in many areas of astronomy, and 
> are closing on the exabyte era, I don't think this restriction to only 
> lossless compression is supportable anymore.  It's the responsibility of the 
> data provider to make sure that the data are presented with sufficient 
> quality to achieve the scientific requirements.  As I pointed out before, 
> many space missions already use lossy compression schemes to keep within 
> their telemetry limitations.

I'm going to have to agree with Bill on this one.

Although serving only uncompressed data may be better for archiving and 
ensuring that data can be read by all people, it may result in people 
assuming that there is more precision than is actually available.

Scientists should hopefully read all of the caveats before using the data, 
but the compressed data would give them a better idea of how much signal 
there actually was.

And it gives us something to point people to when they ask us about the 
'UFOs bigger than the Earth orbiting the Sun'.  (ie, typically compression 
artifacts around cosmic ray or solar energetic particle hits).


I believe that we're also required to archive all data as it comes down 
from the satellite for the missions that we support, and it'd be nice to 
use FITS for this, rather than maintaining another set of tools to read an 
alternate file format.

-Joe

-----
Joe Hourcle
Programmer/Analyst
Solar Data Analysis Center
Goddard Space Flight Center


> On 01/14/16 15:32, Demitri Muna wrote:
>> Hi,
>> 
>> I don’t have any opinions on compression algorithms, but I want to note 
>> that I’m
>> opposed to the FITS format supporting any lossy compression as part of the
>> format. FITS should be archival, and I think that any lossy representation 
>> of
>> data (e.g. a thumbnail) should be created outside of the file. Lossy 
>> algorithms
>> can improve dramatically over time with the increase of CPU speed (see the 
>> ~50%
>> space improvement from H.264 to H.265 for the same quality). Thumbnails are
>> extremely cheap to create and cache and become cheaper to do so over time 
>> with
>> faster I/O and CPU. It’s common in archives to have jpg files next to FITS 
>> files
>> for this reason - I don’t see a benefit to building this into the format. 
>> Keep
>> things light.
>> 
>> Cheers,
>> Demitri
>> 
>> _________________________________________
>> Demitri Muna
>> 
>> Department of Astronomy
>> Le Ohio State University
>> 
>> Home page: http://muna.com
>> 
>> My Projects:
>> http://nightlightapp.io
>> http://trillianverse.org
>> http://scicoder.org
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> fitsbits mailing list
>> fitsbits at listmgr.nrao.edu
>> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>> 
>
> -- 
> William Thompson
> NASA Goddard Space Flight Center
> Code 671
> Greenbelt, MD  20771
> USA
>
> 301-286-2040
> William.T.Thompson at nasa.gov
>
> _______________________________________________
> fitsbits mailing list
> fitsbits at listmgr.nrao.edu
> https://listmgr.nrao.edu/mailman/listinfo/fitsbits
>