8000 Microns 1366 -Search for by sandyhider · Pull Request #12 · jhuapl-boss/boss-tools · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Microns 1366 -Search for #12

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 5, 2018
Merged

Microns 1366 -Search for #12

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
afe132b
updated ingest_lambda to get tile size from s3 metadata and use it to…
sandyhider Jul 20, 2018
5abf386
Added print statement to allow one to search multilambda logs for Til…
sandyhider Jul 20, 2018
5cf1f79
Updated print statements to be more informative and make it easier to…
sandyhider Aug 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 28 additions & 2 deletions lambda/ingest_lambda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,18 +130,44 @@
num_z_slices = 0
for tile_key in tile_key_list:
try:
image_data, message_id, receipt_handle, _ = tile_bucket.getObjectByKey(tile_key)
image_data, message_id, receipt_handle, metadata = tile_bucket.getObjectByKey(tile_key)
except KeyError:
print('Key: {} not found in tile bucket, assuming redelivered SQS message and aborting.'.format(
tile_key))
# Remove message so it's not redelivered.
ingest_queue.deleteMessage(msg_id, msg_rx_handle)
sys.exit("Aborting due to missing tile in bucket")

tile_img = np.asarray(Image.open(BytesIO(image_data)), dtype=dtype)
image_bytes = BytesIO(image_data)
image_size = image_bytes.getbuffer().nbytes

# Get tiles size from metadata, need to shape black tile if actual tile is corrupt.
if 'x_size' in metadata:
tile_size_x = metadata['x_size']
else:
print('MetadataMissing: x_size not in tile metadata: using 1024.')
tile_size_x = 1024

if 'y_size' in metadata:
tile_size_y = metadata['y_size']
else:
print('MetadataMissing: y_size not in tile metadata: using 1024.')
tile_size_y = 1024

if image_size == 0:
print('TileError: Zero length tile, using black instead: {}'.format(tile_key))
tile_img = np.zeros((tile_size_x, tile_size_y), dtype=dtype)
else:
try:
tile_img = np.asarray(Image.open(image_bytes), dtype=dtype)
except TypeError as te:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don’t really need to except as te since you’re not using te?

print('TileError: Incomplete tile, using black instead (tile_size_in_bytes, tile_key): {}, {}'
.format(image_size, tile_key))
tile_img = np.zeros((tile_size_x, tile_size_y), dtype=dtype)
data.append(tile_img)
num_z_slices += 1


# Make 3D array of image data. It should be in XYZ at this point
chunk_data = np.array(data)
del data
Expand Down
2 changes: 2 additions & 0 deletions lambda/tile_upload_lambda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@

from botocore.exceptions import ClientError

print("$$$ IN TILE UPLOAD LAMBDA $$$")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really unrelated but I found it helpful to be able to search multilambda logs for specific types of lambas. that is what this is giving us.


# Load settings
SETTINGS = BossSettings.load()

Expand Down
1 change: 1 addition & 0 deletions lambdafcns/ingest_lambda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
../lambda/ingest_lambda.py
0