python - compressing data with HDFStore -


I'm a newbie for pietsables and have a question about storing compressed panda dataframe. My current code is: 

  import pandas # HDF5 File name H5name = "C: \\ MyDir \\ MyHDF.h5" # Make HDF5 file store # pandas.io.pytables.HDFStore (H5name) # myDF.to_hdf (H5name, "myDFname" writes a Panda DataFrame to the created HDF5 file, attached = true) # Panda DataFrame HDF5 file from myDF1 = pandas.io.pytables.read_hdf (H5name, "myDFname created from Read back ") # Close the file store. Close ()

When I checked the size of HDF 5, the size (212 KB) was much larger than the original CSV file (58kb)

HDFStore (H5name, complevel = 1)

and the file size was not changed. I tried to do all completions from 1 to 9 and the size still remained the same.

I tried to create 

  # HDF 5 file store - pandas.io.pytables.HDFStore (H5name, complevel = 1, complib = "zlib")  
 But there was no change in compression.  
 What could be the problem?  
 In addition, ideally I would like to use a compression which R does for my saving function (like in my case 58kb file was saved in RDATA at 27kb size)? Do I need to do an additional numbering in Python to reduce the size?  
  Edit:   
 I am using Python 3.3.3 and Panda 0.13.1  
  Edit Do:  I tried a large file with a 487 MB CSV file, whose RData is  size  (via the RS saved function) is 16 9 MB, for larger files, I have to compress See I Bzip2 gave the best compression of 202MB (level = 9) and the slowest to read / write. Blosc Compression (level = 9) gave the largest size of 276MB, but was too fast to write / read 
 
 Not sure that R does it differently in its  save.  function, but it is both equally faster and more compressed than any of these compression algos.

If you actually have a small file, HDF 5 is basically a part of your data; Normally 64 KB is the minimum size size. Depedening on what data is, then it may not even be sec in that size.

You can try the msgpack for a simple soln for this size data. HDF 5 is quite efficient for large size and will be well-contained.


Comments

Post a Comment

Popular posts from this blog

ios - How do I use CFArrayRef in Swift? -

I am using Objective-class in my Swift Project through a bridging header. The method signature looks like this: - Some cement (some type) some parameters; I started by getting an example of class, calling method, and storing the value: var myInstance = MyClassWithThatMethod (); Var cfArr = myInstance.someMethod (some value); Then try to get the value in the array: var valueInArrayThatIWant = CFArrayGetValueAtIndex (cfArr, 0); However, I get the error unmanaged & lt; Cfarray & gt; Not like 'CFArray' . Unmanaged & lt; Cfarray & gt; also means? I looked, but I do not need to change the array in a fast array (though it would be good). I need to be able to get value from the array. I also tried CFArray method to pass in a function: func doSomeStuffOnArray (myArray: NSArray) {} Although I get the same error when using: doSomeStuffOnArray (cfArr); // unmanaged & lt; CFArray & gt; I am using CFArray because I want to sto...
Read more

eclipse plugin - Run java code error: Workspace is closed -

To create an automated project, I created a plug-in project with the following dependencies: Org.eclipse.core.resources org.eclipse.equinox.registry org.eclipse.core.runtime and the following The Java class is located in the src folder: Package Examiner; Import org.eclipse.core.resources.IProject; Import org.eclipse.core.resources.IWorkspaceRoot; Import org.eclipse.core.resources.ResourcesPlugin; Import org.eclipse.core.runtime.CoreException; Import org.eclipse.core.runtime.IProgressMonitor; Import org.eclipse.core.runtime.NullProgressMonitor; Public class tes {public static zero main (string [] args) {// TODO auto generated method stub IProgressMonitor progress monitor = new NullProgressMonitor (); IWorkspaceRoot root = ResourcesPlugin.getWorkspace (). GetRoot (); Ipoject project = root.jetproject ("desired projectname"); Try {Project.create (progress monitor); Project.open (progressMonitor); } Grip (CoreException E) {// TODO Auto generated blocking block e....
Read more

c - Error on building source code in VC 6 -

When I create my project in VC6, the BScMKE reports the following error Linking .. Creating Library ... .. / ... / PB / Bean / Debub / ICEERP.Lib and Object ........ P.B. / Bean / DBube / ICREPlay.APP Browse information file ... BSMMK: ​​error can not be read from BK 1505: file '../../PB/bin/TmpOut/ICReplay/Debug/ICReplay_PB.bsc' Error executing bscmake.exe Can someone help me? 2 Reason for reference from: "ICReplay_PB .bsc "file will be corrupted The trunk of the file will be happening with the compiler running out of disk space or is being interrupted when creating .sbr file I think this answer will help you!
Read more