tykayn/mapillary_download
3
0
Fork 3
Code Issues 2 Pull Requests Actions Projects Releases Wiki Activity

mapillary_download of a sequence creates 0 Bytes files #3

New Issue
Open
opened 2024-10-22 23:26:22 +02:00 by matthias · 4 comments
matthias commented 2024-10-22 23:26:22 +02:00
Contributor
Copy Link

when using the mapillary_download.py to get a sequence, about 30-40% of downloaded pictues are 0 Bytes (tested with multiple sequences, but all from me and the same day)

image

when using the mapillary_download.py to get a sequence, about 30-40% of downloaded pictues are 0 Bytes (tested with multiple sequences, but all from me and the same day) ![image](/attachments/e5faaa69-4702-4d6f-8ac7-9ed798f12e9a)
image.png
492 KiB
tykayn commented 2024-10-24 21:34:51 +02:00
Owner
Copy Link

hello, thanks for this report. other people are having trouble getting some batches of photos, we did not found why it happens, usually it is not as often as what you report, more around 5%.
It suggests more of a problem on the mapillary side, are these pictures of the sequence still visible on mapillary website ?

hello, thanks for this report. other people are having trouble getting some batches of photos, we did not found why it happens, usually it is not as often as what you report, more around 5%. It suggests more of a problem on the mapillary side, are these pictures of the sequence still visible on mapillary website ?
matthias commented 2024-10-24 22:28:20 +02:00
Author
Contributor
Copy Link

some more detail:

  • the number of files matches the number of images shown on the mapillary website with this sequence
  • if all 0 Byte files are deleted (find . -name '*.jpg' -size 0 -print0 | xargs -0 rm), they are re-created with 0 Bytes when running the script again
  • if the complete sequence is downloaded a second time, exactly the same files are 0 Bytes again
  • on mapillary, the pictures are present (see screenshot below)
    Screenshot From 2024-10-24 22-23-50

if you want to compare, the screenshot is from sequence R59T5DwBwa9W8jH6LJs0xQ

From the API response, I could not see anything suspicious:

# this is a working picture
$ curl -X GET https://graph.mapillary.com/304579994375345?access_token=$MAPILLARY_DEV_TOKEN\&fields=creator,thumb_original_url,altitude,make,model,camera_type,captured_at,compass_angle,geometry,exif_orientation,sequence
{"creator":{"username":"mapillarymatt","id":"100162188900422"},"thumb_original_url":"https:\/\/z-p3-scontent-ham3-1.xx.fbcdn.net\/m1\/v\/t6\/An_HvZ8mUHe5XJ8dUBUh-OHz9hberwPgyEWaom92nCZj2cHgVMOHeLR7OS8L1W6UJlYujWlEp0nNvNuhOzAxx23gI6xZLsQggt8cpzS2KnChcYnr4Y65HudO4ZBsqqlcvZSZrHqMDpBKK60zWqddSw?ccb=10-5&oh=00_AYCPB9GPNHUXGhthqIZt_Y4_fS2hNYCzOxYw9voeUHzLTw&oe=6742484A&_nc_sid=201bca&_nc_zt=28","altitude":40.323488666165,"make":"samsung","model":"SM-A320FL","camera_type":"perspective","captured_at":1505040427840,"compass_angle":89.876647949219,"geometry":{"type":"Point","coordinates":[9.9672751641128,53.543595624698]},"exif_orientation":1,"sequence":"R59T5DwBwa9W8jH6LJs0xQ","id":"304579994375345"}

# this is a 0 Bytes picture
$ curl -X GET https://graph.mapillary.com/841967903366275?access_token=$MAPILLARY_DEV_TOKEN\&fields=creator,thumb_original_url,altitude,make,model,camera_type,captured_at,compass_angle,geometry,exif_orientation,sequence
{"creator":{"username":"mapillarymatt","id":"100162188900422"},"thumb_original_url":"https:\/\/z-p3-scontent-ham3-1.xx.fbcdn.net\/m1\/v\/t6\/An_Xk83Aq-T_ob8Oi4ZnHwxOKMWQYDX9YxE763SwmjL6xF1xOAtBfRh_8EK4_u3R5dDoB2etgsFS3uPAkzy8I-RFGj8GcziLmmOeaQpM1c9yFQrw9hoVa1h80DThznqDKfP9NnQYtbvJ-6dZ-4DbsA?ccb=10-5&oh=00_AYBM340xP0sW6P7iWH-vxGhsyjKF1rc_7LG26f9bPxLMgQ&oe=674233BF&_nc_sid=201bca&_nc_zt=28","altitude":40.484861867836,"make":"samsung","model":"SM-A320FL","camera_type":"perspective","captured_at":1505040440889,"compass_angle":86.340423583984,"geometry":{"type":"Point","coordinates":[9.9682877908367,53.543588492032]},"exif_orientation":1,"sequence":"R59T5DwBwa9W8jH6LJs0xQ","id":"841967903366275"}

Download with wget works for both thumb_original_urls, filesize is in the range.

some more detail: - the number of files matches the number of images shown on the mapillary website with this sequence - if all 0 Byte files are deleted (`find . -name '*.jpg' -size 0 -print0 | xargs -0 rm`), they are re-created with 0 Bytes when running the script again - if the complete sequence is downloaded a second time, exactly the same files are 0 Bytes again - on mapillary, the pictures are present (see screenshot below) ![Screenshot From 2024-10-24 22-23-50](/attachments/52e3795a-eb3b-45c0-8c88-5557a33562d6) if you want to compare, the screenshot is from sequence `R59T5DwBwa9W8jH6LJs0xQ` From the API response, I could not see anything suspicious: ``` # this is a working picture $ curl -X GET https://graph.mapillary.com/304579994375345?access_token=$MAPILLARY_DEV_TOKEN\&fields=creator,thumb_original_url,altitude,make,model,camera_type,captured_at,compass_angle,geometry,exif_orientation,sequence {"creator":{"username":"mapillarymatt","id":"100162188900422"},"thumb_original_url":"https:\/\/z-p3-scontent-ham3-1.xx.fbcdn.net\/m1\/v\/t6\/An_HvZ8mUHe5XJ8dUBUh-OHz9hberwPgyEWaom92nCZj2cHgVMOHeLR7OS8L1W6UJlYujWlEp0nNvNuhOzAxx23gI6xZLsQggt8cpzS2KnChcYnr4Y65HudO4ZBsqqlcvZSZrHqMDpBKK60zWqddSw?ccb=10-5&oh=00_AYCPB9GPNHUXGhthqIZt_Y4_fS2hNYCzOxYw9voeUHzLTw&oe=6742484A&_nc_sid=201bca&_nc_zt=28","altitude":40.323488666165,"make":"samsung","model":"SM-A320FL","camera_type":"perspective","captured_at":1505040427840,"compass_angle":89.876647949219,"geometry":{"type":"Point","coordinates":[9.9672751641128,53.543595624698]},"exif_orientation":1,"sequence":"R59T5DwBwa9W8jH6LJs0xQ","id":"304579994375345"} # this is a 0 Bytes picture $ curl -X GET https://graph.mapillary.com/841967903366275?access_token=$MAPILLARY_DEV_TOKEN\&fields=creator,thumb_original_url,altitude,make,model,camera_type,captured_at,compass_angle,geometry,exif_orientation,sequence {"creator":{"username":"mapillarymatt","id":"100162188900422"},"thumb_original_url":"https:\/\/z-p3-scontent-ham3-1.xx.fbcdn.net\/m1\/v\/t6\/An_Xk83Aq-T_ob8Oi4ZnHwxOKMWQYDX9YxE763SwmjL6xF1xOAtBfRh_8EK4_u3R5dDoB2etgsFS3uPAkzy8I-RFGj8GcziLmmOeaQpM1c9yFQrw9hoVa1h80DThznqDKfP9NnQYtbvJ-6dZ-4DbsA?ccb=10-5&oh=00_AYBM340xP0sW6P7iWH-vxGhsyjKF1rc_7LG26f9bPxLMgQ&oe=674233BF&_nc_sid=201bca&_nc_zt=28","altitude":40.484861867836,"make":"samsung","model":"SM-A320FL","camera_type":"perspective","captured_at":1505040440889,"compass_angle":86.340423583984,"geometry":{"type":"Point","coordinates":[9.9682877908367,53.543588492032]},"exif_orientation":1,"sequence":"R59T5DwBwa9W8jH6LJs0xQ","id":"841967903366275"} ``` Download with wget works for both `thumb_original_url`s, filesize is in the range.
Screenshot From 2024-10-24 22-23-50.png
2.6 MiB
matthias commented 2024-10-25 00:14:53 +02:00
Author
Contributor
Copy Link

traced it down to:

mapillary_download.py Line 39 in 7d297ed805
image = write_exif(r.content, metadata)

Error for all 0 Bytes files: Directory Photo: IFD exceeds data buffer, cannot read next pointer.

Did not cross-check, but at least for this picture failing, there is a funny annotation in exif-data:

I have added two manually downloaded pictures, maybe it is easier to debug the exif extraction with these files:

  • OK.jpg: working with the script
  • KO.jpg: failing with the error Directory Photo: IFD exceeds data buffer, cannot read next pointer.
traced it down to: https://forge.chapril.org/tykayn/mapillary_download/src/commit/7d297ed80564bf618fffc78d0f45e5c356195cbe/mapillary_download.py#L39 Error for all 0 Bytes files: `Directory Photo: IFD exceeds data buffer, cannot read next pointer.` Did not cross-check, but at least for this picture failing, there is a funny annotation in exif-data: I have added two manually downloaded pictures, maybe it is easier to debug the exif extraction with these files: - OK.jpg: working with the script - KO.jpg: failing with the error `Directory Photo: IFD exceeds data buffer, cannot read next pointer.`
Screenshot From 2024-10-25 00-19-18.png
92 KiB
ok.jpg
2.4 MiB
ko.jpg
2.4 MiB
Screenshot From 2024-10-25 00-19-18.png ok.jpg ko.jpg
matthias commented 2024-10-26 23:56:01 +02:00
Author
Contributor
Copy Link

so... pyexiv2.ImageData seems to crash if jfif tags are present

writer.py Line 38 in 4258a8b84b
self.image = pyexiv2.ImageData(picture)

I am not very experienced with this, but found a workaround by removing these with pillow.

Better solutions welcome...

so... pyexiv2.ImageData seems to crash if jfif tags are present https://forge.chapril.org/tykayn/mapillary_download/src/commit/4258a8b84bb2d022ef5d622cf765f9924734ef73/writer.py#L38 I am not very experienced with this, but found a workaround by removing these with pillow. Better solutions welcome...
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: tykayn/mapillary_download#3
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?