{"id":26360,"date":"2024-03-22T00:27:34","date_gmt":"2024-03-21T23:27:34","guid":{"rendered":"https:\/\/www.graviton.at\/letterswaplibrary\/80-million-tiny-images-dataset-image-decoding-problem\/"},"modified":"2024-03-22T00:27:34","modified_gmt":"2024-03-21T23:27:34","slug":"80-million-tiny-images-dataset-image-decoding-problem","status":"publish","type":"post","link":"https:\/\/www.graviton.at\/letterswaplibrary\/80-million-tiny-images-dataset-image-decoding-problem\/","title":{"rendered":"80 Million Tiny Images Dataset Image Decoding Problem"},"content":{"rendered":"

<\/p>\n

\n

I can’t get to visualize correctly the dataset, i’ve tried to convert the matlab script into a python script but this is the result:<\/p>\n

https:\/\/drive.google.com\/file\/d\/1kzA7mNC4th8nbJh4iGoaZJB_xV4HO7r_\/view?usp=sharing<\/a><\/p>\n

and this is the adapted script: <\/p>\n

import numpy as np <\/p>\n

import os import matplotlib.pyplot as plt<\/p>\n

def load_tiny_images(ndx, filename=None): if filename is None: filename = ‘Z:\/Tiny_Images_Dataset\/data\/tiny_images.bin’ # filename = ‘C:\/atb\/Databases\/Tiny Images\/tiny_images.bin’<\/p>\n

sx = 32 #side size Nimages = len(ndx) nbytes_per_image = sx * sx * 3 img = np.zeros((sx * sx * 3, Nimages), dtype=np.uint8) pointer = (np.array(ndx) – 1) * nbytes_per_image # read data with open(filename, ‘rb’) as f: for i in range(Nimages): f.seek(pointer[i]) # moves the pointer to the beginning of the image img[:, i] = np.frombuffer(f.read(nbytes_per_image), dtype=np.uint8) img = img.reshape((sx, sx, 3, Nimages)) return img <\/p>\n

def show_images(images): N = images.shape[3] fig, axes = plt.subplots(1, N, figsize=(N, 1)) if N == 1: axes = [axes] for i, ax in enumerate(axes): ax.imshow(images[:, :, :, i]) ax.axis(‘off’) plt.show()<\/p>\n

load the first 10\/79302017 imgs<\/h1>\n

img = load_tiny_images(list(range(1, 11)))<\/p>\n

show_images(img)<\/p>\n

What am i missing? is anyone able to correctly open it with python?<\/p>\n

\u200b<\/p>\n

just for completeness, this is the original matlab code (i’m a total zero in matlab):<\/p>\n

\u200b<\/p>\n

function img = loadTinyImages(ndx, filename) <\/p>\n

% % Random access into the file of tiny images. % % It goes faster if ndx is a sorted list % % Input: % ndx = vector of indices % filename = full path and filename % Output: % img = tiny images [32x32x3xlength(ndx)]<\/p>\n

if nargin == 1 filename = ‘Z:Tiny_Images_Datasetdatatiny_images.bin’; % filename = ‘C:atbDatabasesTiny Imagestiny_images.bin’; end<\/p>\n

% Images sx = 32; Nimages = length(ndx); nbytesPerImage = sxsx<\/em>3; img = zeros([sxsx<\/em>3 Nimages], ‘uint8’);<\/p>\n

% Pointer pointer = (ndx-1)*nbytesPerImage; offset = pointer; offset(2:end) = offset(2:end)-offset(1:end-1)-nbytesPerImage;<\/p>\n

% Read data [fid, message] = fopen(filename, ‘r’); if fid == -1 error(message); end frewind(fid) for i = 1:Nimages fseek(fid, offset(i), ‘cof’); tmp = fread(fid, nbytesPerImage, ‘uint8’); img(:,i) = tmp; end fclose(fid);<\/p>\n

img = reshape(img, [sx sx 3 Nimages]);<\/p>\n

% load in first 10 images from 79,302,017 images img = loadTinyImages([1:10]);<\/p>\n

useless to say: in matlab nothing is working, it gives me some path error i have no idea how to resolve and it shows no image etc, i can’t learn matlab now so i’d like to read this huge bin file with python, am i that fool?<\/p>\n

\u200b<\/p>\n

Thanks a lot in advance for any help and sorry about my english<\/p>\n<\/div>\n

submitted by \/u\/AstroGippi <\/a>
[link]<\/a><\/span> [comments]<\/a><\/span><\/p>