Compare commits
10 Commits
b410c1704d
...
a70c952840
Author | SHA1 | Date | |
---|---|---|---|
a70c952840 | |||
|
d3c69c2027 | ||
|
c7a2f714bd | ||
|
b7efcd8900 | ||
|
b25c2161b8 | ||
|
8d3fff3ebe | ||
|
98c850ac62 | ||
|
17466a40b2 | ||
|
f77e6c9443 | ||
|
59933a7310 |
1
.gitignore
vendored
1
.gitignore
vendored
@ -3,3 +3,4 @@ __pycache__/
|
|||||||
build/
|
build/
|
||||||
dist/
|
dist/
|
||||||
*.spec
|
*.spec
|
||||||
|
secrets_variables.sh
|
62
README.md
62
README.md
@ -2,12 +2,44 @@
|
|||||||
Simple code to download images in one or several mapillary sequences. The images will be geotagged and oriented.
|
Simple code to download images in one or several mapillary sequences. The images will be geotagged and oriented.
|
||||||
|
|
||||||
## How to use
|
## How to use
|
||||||
change the access token with your access token and the sequence ids with the ids of the sequences you want to download
|
### Setup env variables
|
||||||
|
copy the secret variables files to setup your maipllary dev token, and your panoramax OpenStreetMap token.
|
||||||
|
```Shell
|
||||||
|
cp secrets_variables_examples.sh secrets_variables.sh
|
||||||
|
editor secrets_variables.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
### Get ALL the sequences of SEVERAL users
|
||||||
|
To avoid to download everything at once, this is a 2 steps process:
|
||||||
|
- 1/ We set a list of users for which we want to get the list of sequences to download, and get a bash script for each user.
|
||||||
|
Change the list of users in the batch file `batch_get_username.sh`.
|
||||||
|
change the access token with your access token and the sequence ids with the ids of the sequences you want to download
|
||||||
|
```bash
|
||||||
|
usernames=( "someone_having_nice_pictures" "someone_else" "oh_look_a_these_usernames" ) # use real accounts, it might work better
|
||||||
|
```
|
||||||
|
This will give you a few files that the other scripts will use to generate a script to download all sequences of each user. One user, one bash script generated.
|
||||||
|
|
||||||
|
|
||||||
|
- 2/ We can launch the download of all the sequences of a user for every user specified in the batch script.
|
||||||
|
```bash
|
||||||
|
bash script_bash_get_sequences_for_user_SOMEONE.sh
|
||||||
|
```
|
||||||
|
**Or all of the generated batch scripts at once.**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash script_bash_get_sequences_for_user_*
|
||||||
|
```
|
||||||
|
|
||||||
|
Don't worry if you rerun the same user `script_bash_get_sequences_for_user_someone.sh` twice, the pictures will not be downloaded twice. Every run of mapillary_download.py checks that pictures are not existing before willing to download them.
|
||||||
|
|
||||||
|
### Get only one sequence with its ID
|
||||||
|
To find a sequence id you can use the batch script, or click on a sequence on the Mapillary website, click on a picture, click on advanced data, and copy the sequence ID.
|
||||||
|
|
||||||
```Shell
|
```Shell
|
||||||
python mapillary_download.py "MLY|xxxx|xxxxxxx" --sequence_ids xxxxxxxxxxx xxxxxxxxxxx
|
python mapillary_download.py "MLY|xxxx|xxxxxxx" --sequence_ids xxxxxxxxxxx xxxxxxxxxxx
|
||||||
```
|
```
|
||||||
|
|
||||||
## Available arguments
|
### Available arguments
|
||||||
```Shell
|
```Shell
|
||||||
python mapillary_download.py -h
|
python mapillary_download.py -h
|
||||||
usage: mapillary_download.py [-h] [--sequence_ids [SEQUENCE_IDS ...]] [--image_ids [IMAGE_IDS ...]] [--destination DESTINATION]
|
usage: mapillary_download.py [-h] [--sequence_ids [SEQUENCE_IDS ...]] [--image_ids [IMAGE_IDS ...]] [--destination DESTINATION]
|
||||||
@ -30,8 +62,18 @@ optional arguments:
|
|||||||
--overwrite overwrite existing images
|
--overwrite overwrite existing images
|
||||||
-v, --version show program's version number and exit
|
-v, --version show program's version number and exit
|
||||||
```
|
```
|
||||||
|
## Sort pictures depending of a bounding box in their Exif GPS Data in a separate folder
|
||||||
|
|
||||||
## How to get my access token
|
This is used to restrict zones for national instances of Panoramax to upload to reuse the sequences of multi national users.
|
||||||
|
|
||||||
|
Edit the bounding boxes in `find_france_photos_and_move.py` and change the destination folder.
|
||||||
|
Running the script will check in the /data folder for every user sequences and move the files being in the bouding boxes in the `destionation` folder path of the script.
|
||||||
|
|
||||||
|
For this script to read the exif of pictures, you will need the python package `exifread`
|
||||||
|
It was tested with pictures taken with the App Open Camera.
|
||||||
|
|
||||||
|
|
||||||
|
## How to get my Mapillary access token
|
||||||
- Go to https://www.mapillary.com/dashboard/developers
|
- Go to https://www.mapillary.com/dashboard/developers
|
||||||
- Click on "Registrer Application", enter the needed informations, enable the application to "Read" data, then click on register :
|
- Click on "Registrer Application", enter the needed informations, enable the application to "Read" data, then click on register :
|
||||||
|
|
||||||
@ -63,3 +105,17 @@ Then you can run `python mapillary_download "MLY|xxxx|xxxxxxx" --sequence_ids x
|
|||||||
When you're done with the script, simply run `deactivate` to exit the virtual environnement.
|
When you're done with the script, simply run `deactivate` to exit the virtual environnement.
|
||||||
|
|
||||||
On windows you can create a virtual environnement too, or use the prebuilt `mapillary_download.exe` available on the release page.
|
On windows you can create a virtual environnement too, or use the prebuilt `mapillary_download.exe` available on the release page.
|
||||||
|
|
||||||
|
|
||||||
|
## How to batch the upload to panoramax
|
||||||
|
|
||||||
|
After having downloaded the sequences of a user you will find them in the folder `data`.
|
||||||
|
go in your `data` folder, and to the user subfolder, and assuming you have already sent pictures with geovisio_cli, you can run this command to send all the sequences of the current folder you're in:
|
||||||
|
|
||||||
|
|
||||||
|
```Shell
|
||||||
|
cd data/some_user
|
||||||
|
|
||||||
|
for dir in */; do dir=${dir%?} ; geovisio upload --api-url https://panoramax.openstreetmap.fr "$dir" --token=BLAH_BLAH ; done
|
||||||
|
```
|
||||||
|
Have fun!
|
||||||
|
44
batch_get_username.sh
Normal file
44
batch_get_username.sh
Normal file
@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# récupérer les séquences pour un tas d'utilisateurs
|
||||||
|
|
||||||
|
|
||||||
|
# Liste des usernames
|
||||||
|
# example:
|
||||||
|
# usernames=( "riri" "fifi" "loulou")
|
||||||
|
usernames=( "someone_having_nice_pictures" "someone_else" "oh_look_a_these_usernames" )
|
||||||
|
|
||||||
|
# check env variables are valid
|
||||||
|
if [ -f "secrets_variables.sh" ]; then
|
||||||
|
source "secrets_variables.sh"
|
||||||
|
if [ "$MAPILLARY_DEV_TOKEN" = "MLY|blahblah_replace_it" ]; then
|
||||||
|
echo "Erreur : La variable MAPILLARY_DEV_TOKEN doit être modifiée pour que le script fonctionne."
|
||||||
|
echo "Veuillez remplacer la valeur par défaut \"MLY|blahblah_replace_it\" par votre propre token de développement Mapillary."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo "Erreur : Le fichier secrets_variables.sh n'a pas été trouvé."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Boucle sur la liste des usernames
|
||||||
|
for username in "${usernames[@]}"; do
|
||||||
|
# Lancer la commande pour chaque username
|
||||||
|
|
||||||
|
echo "---------- utilisateur: $username"
|
||||||
|
|
||||||
|
if [ ! -f "out_$username.json" ]; then
|
||||||
|
bash find_user_id.sh $username
|
||||||
|
fi
|
||||||
|
# Vérifier si le fichier sequences_$username.txt existe
|
||||||
|
if [ ! -f "sequences_$username.txt" ]; then
|
||||||
|
python3 get_sequences_of_username.py --username="$username" --dev_token="$MAPILLARY_DEV_TOKEN" --max_sequence=9999
|
||||||
|
else
|
||||||
|
echo "le fichier sequences txt existe pour $username"
|
||||||
|
fi
|
||||||
|
# Lancer la commande pour chaque username
|
||||||
|
if [ ! -f "script_bash_get_sequences_for_user_$username.sh" ]; then
|
||||||
|
python3 text_array_to_download_script.py --username=$username --dev_token="$MAPILLARY_DEV_TOKEN"
|
||||||
|
fi
|
||||||
|
|
||||||
|
done
|
||||||
|
echo "---------- finished getting users ------------"
|
193
find_france_photos_and_move.py
Normal file
193
find_france_photos_and_move.py
Normal file
@ -0,0 +1,193 @@
|
|||||||
|
###
|
||||||
|
# Filtrage de photos situées en France dans un dossier contenant des séquences de photo géolocalisées
|
||||||
|
# Rechercher et déplacer automatiquement les photos géolocalisées dans une certaine bounding box et les déplacer dans le dossier de destination.
|
||||||
|
#
|
||||||
|
# utilisation:
|
||||||
|
# python find_france_photos_and_move.py --source_dir /chemin/du/répertoire/source --destination_dir /chemin/du/répertoire/destination
|
||||||
|
#
|
||||||
|
###
|
||||||
|
import argparse
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import exifread
|
||||||
|
|
||||||
|
# Définition du rectangle entourant la France métropolitaine et un peu autour
|
||||||
|
france_bbox: tuple[float, float, float, float] = (42.0, -5.0, 51.0, 10.0) # (lat_min, lon_min, lat_max, lon_max)
|
||||||
|
# Définition du rectangle entourant la France métropolitaine et un peu autour
|
||||||
|
france_bbox: tuple[float, float, float, float] = (42.0, -5.0, 51.0, 10.0) # (lat_min, lon_min, lat_max, lon_max)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant la Guadeloupe
|
||||||
|
guadeloupe_bbox: tuple[float, float, float, float] = (15.8, -61.8, 17.3, -59.3)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant la Martinique
|
||||||
|
martinique_bbox: tuple[float, float, float, float] = (14.3, -61.3, 15.1, -59.3)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant la Guyane française
|
||||||
|
guyane_bbox: tuple[float, float, float, float] = (2.0, -54.5, 6.5, -51.5)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant La Réunion
|
||||||
|
reunion_bbox: tuple[float, float, float, float] = (-21.3, 55.2, -20.8, 55.8)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant Mayotte
|
||||||
|
mayotte_bbox: tuple[float, float, float, float] = (-13.0, 45.0, -12.5, 45.5)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant Saint-Pierre-et-Miquelon
|
||||||
|
spm_bbox: tuple[float, float, float, float] = (46.7, -56.2, 47.1, -55.6)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant les îles de Saint-Martin et Saint-Barthélemy
|
||||||
|
stm_sbh_bbox: tuple[float, float, float, float] = (18.0, -64.5, 18.5, -62.5)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant Wallis-et-Futuna
|
||||||
|
wf_bbox: tuple[float, float, float, float] = (-13.3, -176.2, -13.1, -175.8)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant la Nouvelle-Calédonie
|
||||||
|
nc_bbox: tuple[float, float, float, float] = (-22.5, 165.5, -18.5, 169.5)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant la Polynésie française
|
||||||
|
pf_bbox: tuple[float, float, float, float] = (-27.5, -140.0, -7.5, -134.0)
|
||||||
|
|
||||||
|
# Définition du rectangle entourant les Terres australes et antarctiques françaises
|
||||||
|
taaf_bbox: tuple[float, float, float, float] = (-49.5, 68.5, -37.5, 77.5)
|
||||||
|
|
||||||
|
# Chemin du répertoire source
|
||||||
|
source_dir: str = '/home/cipherbliss/Téléchargements/FIBRELAND/TEST_IN_FR/'
|
||||||
|
|
||||||
|
# Chemin du répertoire destination
|
||||||
|
destination_dir: str = '/home/cipherbliss/Téléchargements/FIBRELAND/IN_FRANCE/'
|
||||||
|
sequence_folder: str = 'principale_sequence'
|
||||||
|
count_files_all: int = 0
|
||||||
|
count_files_moved: int = 0
|
||||||
|
# Crée le répertoire destination si il n'existe pas
|
||||||
|
if not os.path.exists(destination_dir):
|
||||||
|
os.makedirs(destination_dir)
|
||||||
|
|
||||||
|
|
||||||
|
# Fonction pour déplacer un fichier si il est dans le rectangle de la France
|
||||||
|
def move_file_if_in_france(filepath, sequence_folder):
|
||||||
|
global count_files_all
|
||||||
|
global count_files_moved
|
||||||
|
|
||||||
|
# Ouvre le fichier image et lit les informations EXIF
|
||||||
|
latitude, longitude = get_gps_info(filepath)
|
||||||
|
|
||||||
|
if latitude and longitude:
|
||||||
|
print(f'Latitude: {latitude}, Longitude: {longitude}')
|
||||||
|
if are_lat_lon_in_france(latitude, longitude):
|
||||||
|
move_file_in_destination(filepath, sequence_folder)
|
||||||
|
else:
|
||||||
|
print('Informations GPS non trouvées')
|
||||||
|
|
||||||
|
|
||||||
|
def move_file_in_destination(filepath, sequence_folder):
|
||||||
|
global count_files_moved
|
||||||
|
# Déplace le fichier dans le sous-répertoire "photos_in_france"
|
||||||
|
dest_subdir = os.path.join(destination_dir, sequence_folder,
|
||||||
|
os.path.basename(os.path.dirname(filepath)))
|
||||||
|
if not os.path.exists(dest_subdir):
|
||||||
|
os.makedirs(dest_subdir)
|
||||||
|
shutil.move(filepath, os.path.join(dest_subdir, filepath))
|
||||||
|
count_files_moved += 1
|
||||||
|
print(f"Moved {filepath} to {dest_subdir}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
def are_lat_lon_in_france(gps_lat, gps_lon):
|
||||||
|
"""
|
||||||
|
recherche d'une zone du territoire français
|
||||||
|
|
||||||
|
|
||||||
|
France métropolitaine : 551 695 km²
|
||||||
|
Terres australes et antarctiques françaises : 432 000 km²
|
||||||
|
Guyane française : 83 534 km²
|
||||||
|
Nouvelle-Calédonie : 18 575 km²
|
||||||
|
Polynésie française : 4 167 km²
|
||||||
|
La Réunion : 2 512 km²
|
||||||
|
Martinique : 1 128 km²
|
||||||
|
Guadeloupe : 1 628 km²
|
||||||
|
Mayotte : 374 km²
|
||||||
|
Saint-Pierre-et-Miquelon : 242 km²
|
||||||
|
Wallis-et-Futuna : 142 km²
|
||||||
|
Saint-Martin et Saint-Barthélemy : 53 km²
|
||||||
|
|
||||||
|
|
||||||
|
:param gps_lat:
|
||||||
|
:param gps_lon:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
global france_bbox, guyane_bbox, nc_bbox, pf_bbox, reunion_bbox, guadeloupe_bbox, martinique_bbox, mayotte_bbox, spm_bbox, stm_sbh_bbox, wf_bbox, taaf_bbox
|
||||||
|
|
||||||
|
print("lat lon :", gps_lat, gps_lon)
|
||||||
|
|
||||||
|
if (france_bbox[0] <= gps_lat <= france_bbox[2] and france_bbox[1] <= gps_lon <= france_bbox[3]):
|
||||||
|
return "France métropolitaine"
|
||||||
|
elif (taaf_bbox[0] <= gps_lat <= taaf_bbox[2] and taaf_bbox[1] <= gps_lon <= taaf_bbox[3]):
|
||||||
|
return "Terres australes et antarctiques françaises"
|
||||||
|
elif (guyane_bbox[0] <= gps_lat <= guyane_bbox[2] and guyane_bbox[1] <= gps_lon <= guyane_bbox[3]):
|
||||||
|
return "Guyane française"
|
||||||
|
elif (reunion_bbox[0] <= gps_lat <= reunion_bbox[2] and reunion_bbox[1] <= gps_lon <= reunion_bbox[3]):
|
||||||
|
return "La Réunion"
|
||||||
|
elif (wf_bbox[0] <= gps_lat <= wf_bbox[2] and wf_bbox[1] <= gps_lon <= wf_bbox[3]):
|
||||||
|
return "Wallis-et-Futuna"
|
||||||
|
elif (stm_sbh_bbox[0] <= gps_lat <= stm_sbh_bbox[2] and stm_sbh_bbox[1] <= gps_lon <= stm_sbh_bbox[3]):
|
||||||
|
return "Saint-Martin et Saint-Barthélemy"
|
||||||
|
elif (spm_bbox[0] <= gps_lat <= spm_bbox[2] and spm_bbox[1] <= gps_lon <= spm_bbox[3]):
|
||||||
|
return "Saint-Pierre-et-Miquelon"
|
||||||
|
elif (mayotte_bbox[0] <= gps_lat <= mayotte_bbox[2] and mayotte_bbox[1] <= gps_lon <= mayotte_bbox[3]):
|
||||||
|
return "Mayotte"
|
||||||
|
elif (martinique_bbox[0] <= gps_lat <= martinique_bbox[2] and martinique_bbox[1] <= gps_lon <= martinique_bbox[3]):
|
||||||
|
return "Martinique"
|
||||||
|
elif (guadeloupe_bbox[0] <= gps_lat <= guadeloupe_bbox[2] and guadeloupe_bbox[1] <= gps_lon <= guadeloupe_bbox[3]):
|
||||||
|
return "Guadeloupe"
|
||||||
|
|
||||||
|
elif (pf_bbox[0] <= gps_lat <= pf_bbox[2] and pf_bbox[1] <= gps_lon <= pf_bbox[3]):
|
||||||
|
return "Polynésie française"
|
||||||
|
elif (nc_bbox[0] <= gps_lat <= nc_bbox[2] and nc_bbox[1] <= gps_lon <= nc_bbox[3]):
|
||||||
|
return "Nouvelle-Calédonie"
|
||||||
|
else:
|
||||||
|
return None # "Hors de France"
|
||||||
|
|
||||||
|
|
||||||
|
def get_gps_info(filepath):
|
||||||
|
with open(filepath, 'rb') as f:
|
||||||
|
tags = exifread.process_file(f)
|
||||||
|
gps_info = {}
|
||||||
|
|
||||||
|
# Recherche les informations GPS dans les informations EXIF
|
||||||
|
|
||||||
|
# print("clés exif ", tags.keys())
|
||||||
|
for tag in tags.keys():
|
||||||
|
if tag.startswith('GPS'):
|
||||||
|
gps_info[tag] = tags[tag]
|
||||||
|
|
||||||
|
# Extraction des informations de latitude et de longitude
|
||||||
|
gps_latitude = convert_rational_to_float(gps_info.get('GPS GPSLatitude'))
|
||||||
|
gps_longitude = convert_rational_to_float(gps_info.get('GPS GPSLongitude'))
|
||||||
|
|
||||||
|
if gps_latitude and gps_longitude:
|
||||||
|
return gps_latitude, gps_longitude
|
||||||
|
else:
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
def convert_rational_to_float(rational):
|
||||||
|
return float(rational.values[0].num) / float(rational.values[0].den)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('--source_dir', default='/home/cipherbliss/Téléchargements/FIBRELAND/TEST_IN_FR/', help='Chemin du répertoire source')
|
||||||
|
parser.add_argument('--destination_dir', default='/home/cipherbliss/Téléchargements/FIBRELAND/IN_FRANCE/', help='Chemin du répertoire destination')
|
||||||
|
parser.add_argument('--sequence_folder', default='principale_sequence', help='Nom du dossier de séquence')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
# Parcourt tous les fichiers dans le répertoire source et ses sous-répertoires
|
||||||
|
for root, dirs, files in os.walk(args.source_dir):
|
||||||
|
for filename in files:
|
||||||
|
# Vérifie si le fichier est une image
|
||||||
|
if filename.lower().endswith(('.png', '.jpg', '.jpeg', '.gif', '.bmp', '.tif')):
|
||||||
|
filepath = os.path.join(root, filename)
|
||||||
|
move_file_if_in_france(filepath, sequence_folder)
|
||||||
|
|
||||||
|
print('fichiers se situant en france déplacés: ', count_files_moved, ' / ', count_files_all)
|
89
get_sequences_of_username.py
Normal file
89
get_sequences_of_username.py
Normal file
@ -0,0 +1,89 @@
|
|||||||
|
import json
|
||||||
|
import requests
|
||||||
|
# lit un json listant les id de photo de chaque séquence et va
|
||||||
|
# chercher la séquence par API.
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
def parse_args(argv =None):
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('--username', type=str, help='Username to get the sequences id of')
|
||||||
|
parser.add_argument('--dev_token', type=str, help='Your mapillary developer token')
|
||||||
|
parser.add_argument('--max_sequence', type=str, help='Username to get the sequences id of')
|
||||||
|
|
||||||
|
global args
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
print(args)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Initialisation de la liste pour stocker les réponses
|
||||||
|
responses = []
|
||||||
|
sequences = []
|
||||||
|
|
||||||
|
def get_image_data_from_sequences():
|
||||||
|
username = args.username
|
||||||
|
input_file = "out_"+username+".json"
|
||||||
|
|
||||||
|
|
||||||
|
# Chargement du fichier JSON d'entrée
|
||||||
|
with open(input_file, "r") as file:
|
||||||
|
input_data = json.load(file)
|
||||||
|
|
||||||
|
# Itération sur les noeuds pour collectionner les image_ids
|
||||||
|
nodelist = input_data["data"]["fetch__User"]["feed"]["nodes"]
|
||||||
|
print( 'séquences : ', len(nodelist))
|
||||||
|
image_ids = [node["image_id"] for node in nodelist]
|
||||||
|
print(image_ids)
|
||||||
|
|
||||||
|
dev_token = args.dev_token
|
||||||
|
|
||||||
|
# Préparation de la tête d'autorisation pour toutes les futures requêtes
|
||||||
|
header = {"Access-Token": dev_token}
|
||||||
|
|
||||||
|
ii=0
|
||||||
|
limit_requests = 1000000000
|
||||||
|
# limit_requests = 5 # pour tester
|
||||||
|
# Boucle sur chaque image_id pour interroger l'API Mapillary
|
||||||
|
for image_id in image_ids:
|
||||||
|
ii+=1
|
||||||
|
if limit_requests >= ii and image_id:
|
||||||
|
params = {"id": image_id, "fields": "id,sequence"}
|
||||||
|
request_url = "https://graph.mapillary.com/" + str(image_id)+"?access_token="+dev_token+"&fields=id,sequence"
|
||||||
|
# print("requete: "+request_url)
|
||||||
|
|
||||||
|
response = requests.get(request_url)
|
||||||
|
|
||||||
|
# Analyse de la réponse
|
||||||
|
parsed_response = {}
|
||||||
|
if response.ok and response.status_code == 200:
|
||||||
|
raw_response = response.json()
|
||||||
|
|
||||||
|
parsed_response["id"] = raw_response["id"]
|
||||||
|
parsed_response["sequence"] = raw_response["sequence"]
|
||||||
|
sequences.append(parsed_response["sequence"])
|
||||||
|
|
||||||
|
print("séquence trouvée: "+str(ii)+"/"+args.max_sequence+" : "+raw_response["sequence"])
|
||||||
|
else:
|
||||||
|
print(response)
|
||||||
|
|
||||||
|
responses.append(parsed_response)
|
||||||
|
|
||||||
|
def persist_files():
|
||||||
|
# Sauvegarde des nouveaux résultats dans le fichier output.json
|
||||||
|
output_file = "sequences_"+args.username+".json"
|
||||||
|
|
||||||
|
with open(output_file, "w") as file:
|
||||||
|
json.dump(responses, file)
|
||||||
|
|
||||||
|
sequence_filename = "sequences_"+args.username+".txt"
|
||||||
|
with open(sequence_filename, "w") as file:
|
||||||
|
json.dump(sequences, file)
|
||||||
|
print('fichier sauvegardé: '+sequence_filename)
|
||||||
|
|
||||||
|
|
||||||
|
parse_args()
|
||||||
|
get_image_data_from_sequences()
|
||||||
|
persist_files()
|
||||||
|
|
||||||
|
# si la requete donne moins du max de noeuds on prévoit d'en relancer une nouvelle pour avoir la suite
|
27
get_user.sh
Normal file
27
get_user.sh
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# lancement de la récupération des identifiants de séquences
|
||||||
|
# exemple:
|
||||||
|
# bash get_user.sh binerf 102718865306727
|
||||||
|
|
||||||
|
source secrets_variables.sh
|
||||||
|
|
||||||
|
export username=$1
|
||||||
|
export num_user=$2
|
||||||
|
echo "télécharger la séquence pour l'utilisateur $username, $num_user"
|
||||||
|
bash curl_land.sh "$username" "$num_user" > "out_${username}.json"
|
||||||
|
|
||||||
|
echo "séquences récupérées:"
|
||||||
|
num_sequences=$(grep -o -w 'image_id' "out_${username}.json" | wc -l)
|
||||||
|
#
|
||||||
|
if (( num_sequences > 0 ))
|
||||||
|
then
|
||||||
|
echo "Séquences trouvées: (${num_sequences}). Noice."
|
||||||
|
python3 get_sequences_of_username.py --username="$username" --max_sequence="$num_sequences" --dev_token="$MAPILLARY_DEV_TOKEN"
|
||||||
|
python3 text_array_to_download_script.py --username="$username" --dev_token="$MAPILLARY_DEV_TOKEN"
|
||||||
|
##
|
||||||
|
else
|
||||||
|
echo "Aucune séquence trouvée (${num_sequences}) ! Pas d'autres actions à entreprendre."
|
||||||
|
# cat "out_${username}.json"
|
||||||
|
fi
|
||||||
|
|
||||||
|
|
38
images_par_username.py
Normal file
38
images_par_username.py
Normal file
@ -0,0 +1,38 @@
|
|||||||
|
import requests, json
|
||||||
|
import argparse
|
||||||
|
from urllib.parse import quote
|
||||||
|
|
||||||
|
def parse_args(argv =None):
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('--access_token', type=str, help='Your mapillary access token')
|
||||||
|
parser.add_argument('--username', type=str, help='Username to get the sequences id of')
|
||||||
|
parser.add_argument('--pictures', type=str, help='Limit of pictures to fetch')
|
||||||
|
|
||||||
|
global args
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
parse_args()
|
||||||
|
|
||||||
|
if args.access_token == None:
|
||||||
|
print('please provide the access_token')
|
||||||
|
exit()
|
||||||
|
|
||||||
|
mly_key = args.access_token
|
||||||
|
creator_username = args.username
|
||||||
|
max_img= args.pictures
|
||||||
|
|
||||||
|
url = f'https://graph.mapillary.com/images?access_token={mly_key}&creator_username={creator_username}&limit={max_img}&fields=id,sequence'
|
||||||
|
|
||||||
|
response = requests.get(url)
|
||||||
|
|
||||||
|
if response.status_code == 200:
|
||||||
|
json = response.json()
|
||||||
|
|
||||||
|
# tri des séquences uniques
|
||||||
|
sequences_ids = [obj['sequence'] for obj in json['data']]
|
||||||
|
unique_ids = list(set(sequences_ids))
|
||||||
|
print(unique_ids)
|
||||||
|
else:
|
||||||
|
print(response)
|
@ -26,7 +26,7 @@ def parse_args(argv =None):
|
|||||||
parser.add_argument('--destination', type=str, default='data', help='Path destination for the images')
|
parser.add_argument('--destination', type=str, default='data', help='Path destination for the images')
|
||||||
parser.add_argument('--image_limit', type=int, default=None, help='How many images you want to download')
|
parser.add_argument('--image_limit', type=int, default=None, help='How many images you want to download')
|
||||||
parser.add_argument('--overwrite', default=False, action='store_true', help='overwrite existing images')
|
parser.add_argument('--overwrite', default=False, action='store_true', help='overwrite existing images')
|
||||||
parser.add_argument("-v", "--version", action="version", version="release 1.1")
|
parser.add_argument("-v", "--version", action="version", version="release 1.6")
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
if args.sequence_ids is None and args.image_ids is None:
|
if args.sequence_ids is None and args.image_ids is None:
|
||||||
parser.error("Please enter at least one sequence id or image id")
|
parser.error("Please enter at least one sequence id or image id")
|
||||||
@ -42,9 +42,10 @@ def download(url, filepath, metadata=None):
|
|||||||
|
|
||||||
|
|
||||||
def get_single_image_data(image_id, mly_header):
|
def get_single_image_data(image_id, mly_header):
|
||||||
req_url = 'https://graph.mapillary.com/{}?fields=thumb_original_url,altitude,camera_type,captured_at,compass_angle,geometry,exif_orientation,sequence'.format(image_id)
|
req_url = 'https://graph.mapillary.com/{}?fields=creator,thumb_original_url,altitude,make,model,camera_type,captured_at,compass_angle,geometry,exif_orientation,sequence'.format(image_id)
|
||||||
r = session.get(req_url, headers=mly_header)
|
r = session.get(req_url, headers=mly_header)
|
||||||
data = r.json()
|
data = r.json()
|
||||||
|
print(data)
|
||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
@ -98,6 +99,9 @@ def write_exif(picture, img_metadata):
|
|||||||
# 'captured_at': 1603459736644, 'geometry': {'type': 'Point', 'coordinates': [2.5174596904057, 48.777089857534]}, 'id': '485924785946693'}
|
# 'captured_at': 1603459736644, 'geometry': {'type': 'Point', 'coordinates': [2.5174596904057, 48.777089857534]}, 'id': '485924785946693'}
|
||||||
|
|
||||||
with writer.Writer(picture) as image:
|
with writer.Writer(picture) as image:
|
||||||
|
image.add_artist(img_metadata)
|
||||||
|
image.add_camera_make(img_metadata)
|
||||||
|
image.add_camera_model(img_metadata)
|
||||||
image.add_datetimeoriginal(img_metadata)
|
image.add_datetimeoriginal(img_metadata)
|
||||||
image.add_lat_lon(img_metadata)
|
image.add_lat_lon(img_metadata)
|
||||||
image.add_altitude(img_metadata)
|
image.add_altitude(img_metadata)
|
||||||
@ -150,14 +154,20 @@ if __name__ == '__main__':
|
|||||||
path = os.path.join(path_destination, date_time_image_filename)
|
path = os.path.join(path_destination, date_time_image_filename)
|
||||||
img_metadata = writer.PictureMetadata(
|
img_metadata = writer.PictureMetadata(
|
||||||
capture_time = datetime.utcfromtimestamp(int(image_data['captured_at'])/1000),
|
capture_time = datetime.utcfromtimestamp(int(image_data['captured_at'])/1000),
|
||||||
|
artist = image_data['creator']['username'],
|
||||||
|
camera_make = image_data['make'],
|
||||||
|
camera_model = image_data['model'],
|
||||||
longitude = image_data['geometry']['coordinates'][0],
|
longitude = image_data['geometry']['coordinates'][0],
|
||||||
latitude = image_data['geometry']['coordinates'][1],
|
latitude = image_data['geometry']['coordinates'][1],
|
||||||
picture_type = PictureType("equirectangular") if image_data['camera_type'] == 'spherical' else None,
|
picture_type = PictureType("equirectangular") if image_data.get('camera_type') == 'spherical' or image_data.get('camera_type') == 'equirectangular' else PictureType("flat"),
|
||||||
direction = image_data['compass_angle'],
|
direction = image_data['compass_angle'],
|
||||||
altitude = image_data['altitude'],
|
altitude = image_data['altitude'],
|
||||||
)
|
)
|
||||||
|
#print("metadata: ", img_metadata)
|
||||||
|
#print("path: ", image_data)
|
||||||
image_exists = os.path.exists(path)
|
image_exists = os.path.exists(path)
|
||||||
if not args.overwrite and image_exists:
|
if not args.overwrite and image_exists:
|
||||||
print("{} already exists. Skipping ".format(path))
|
print("{} already exists. Skipping ".format(path))
|
||||||
continue
|
continue
|
||||||
executor.submit(download, url=image_data['thumb_original_url'], filepath=path, metadata=img_metadata)
|
executor.submit(download, url=image_data['thumb_original_url'], filepath=path, metadata=img_metadata)
|
||||||
|
#download(url=image_data['thumb_original_url'], filepath=path, metadata=img_metadata)
|
||||||
|
5
secrets_variables_example.sh
Normal file
5
secrets_variables_example.sh
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
# scripts common shared variables
|
||||||
|
# this is to copy as a file named "secrets_variables.sh" which is ignored in this git repo
|
||||||
|
|
||||||
|
|
||||||
|
export MAPILLARY_DEV_TOKEN="MLY|blahblah_replace_it"
|
53
text_array_to_download_script.py
Normal file
53
text_array_to_download_script.py
Normal file
@ -0,0 +1,53 @@
|
|||||||
|
import os
|
||||||
|
|
||||||
|
input_file = 'input_file'
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
def parse_args(argv =None):
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('--dev_token', type=str, help='Your mapillary access token')
|
||||||
|
parser.add_argument('--username', type=str, help='Username to get the sequences id of')
|
||||||
|
|
||||||
|
global args
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
print("Construction du script bash de récupération des images de chaque séquences pour Mapillary_download (https://github.com/Stefal/mapillary_download.git)")
|
||||||
|
|
||||||
|
parse_args()
|
||||||
|
|
||||||
|
username=args.username
|
||||||
|
input_file = f"sequences_{username}.txt"
|
||||||
|
|
||||||
|
if not args.dev_token:
|
||||||
|
print(f"Erreur : Le token de développeur de mapillary manque, vérifiez le fichier de variables secretes. Arrêt du script.")
|
||||||
|
exit(1)
|
||||||
|
|
||||||
|
if not os.path.exists(input_file) or not os.path.isfile(input_file):
|
||||||
|
print(f"Erreur : Le fichier '{input_file}' n'a pas été trouvé. Arrêt du script.")
|
||||||
|
exit(1)
|
||||||
|
else:
|
||||||
|
print(f"Fichier '{input_file}' trouvé.")
|
||||||
|
|
||||||
|
|
||||||
|
output_file = f"script_bash_get_sequences_for_user_{username}.sh"
|
||||||
|
|
||||||
|
access_token = "--access_token='"+args.dev_token+"' "
|
||||||
|
format_string = "/usr/bin/python3 mapillary_download.py {} --sequence_id={}\n"
|
||||||
|
|
||||||
|
|
||||||
|
with open(output_file, "w") as output:
|
||||||
|
with open(input_file, "r") as input_handle:
|
||||||
|
content = input_handle.read()
|
||||||
|
sequences = eval(content)
|
||||||
|
for seq in sequences:
|
||||||
|
full_cmd = f"/usr/bin/python3 mapillary_download.py {access_token} --sequence_id='{seq}' --username={username}\n"
|
||||||
|
output.write(full_cmd)
|
||||||
|
|
||||||
|
print(output_file)
|
||||||
|
|
||||||
|
print(f"\n Script Bash généré avec succès.")
|
||||||
|
print(f"Lancez le pour récupérer les photos de l'utilisateur {username}: \n bash {output_file}")
|
||||||
|
|
38
writer.py
38
writer.py
@ -20,6 +20,9 @@ tz_finder = timezonefinder.TimezoneFinder()
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class PictureMetadata:
|
class PictureMetadata:
|
||||||
|
artist: Optional[str] = None
|
||||||
|
camera_make: Optional[str] = None
|
||||||
|
camera_model: Optional[str] = None
|
||||||
capture_time: Optional[datetime] = None
|
capture_time: Optional[datetime] = None
|
||||||
longitude: Optional[float] = None
|
longitude: Optional[float] = None
|
||||||
latitude: Optional[float] = None
|
latitude: Optional[float] = None
|
||||||
@ -44,8 +47,13 @@ class Writer():
|
|||||||
self.image.close()
|
self.image.close()
|
||||||
|
|
||||||
def apply(self) -> None:
|
def apply(self) -> None:
|
||||||
|
try:
|
||||||
|
if self.updated_exif:
|
||||||
self.image.modify_exif(self.updated_exif)
|
self.image.modify_exif(self.updated_exif)
|
||||||
|
if self.updated_xmp:
|
||||||
self.image.modify_xmp(self.updated_xmp)
|
self.image.modify_xmp(self.updated_xmp)
|
||||||
|
except Exception as e:
|
||||||
|
print("exception \nexif: {}\nxmp: {}".format(self.updated_exif, self.updated_xmp))
|
||||||
|
|
||||||
def close(self) -> None:
|
def close(self) -> None:
|
||||||
self.image.close()
|
self.image.close()
|
||||||
@ -146,6 +154,32 @@ class Writer():
|
|||||||
self.updated_xmp["Xmp.GPano.ProjectionType"] = metadata.picture_type.value
|
self.updated_xmp["Xmp.GPano.ProjectionType"] = metadata.picture_type.value
|
||||||
self.updated_xmp["Xmp.GPano.UsePanoramaViewer"] = True
|
self.updated_xmp["Xmp.GPano.UsePanoramaViewer"] = True
|
||||||
|
|
||||||
|
def add_artist(self, metadata: PictureMetadata) -> None:
|
||||||
|
"""
|
||||||
|
Add image author in Exif Artist tag
|
||||||
|
"""
|
||||||
|
|
||||||
|
if metadata.artist is not None:
|
||||||
|
self.updated_exif["Exif.Image.Artist"] = ascii(metadata.artist).strip("'")
|
||||||
|
|
||||||
|
|
||||||
|
def add_camera_make(self, metadata: PictureMetadata) -> None:
|
||||||
|
"""
|
||||||
|
Add camera manufacture in Exif Make tag
|
||||||
|
"""
|
||||||
|
|
||||||
|
if metadata.camera_make is not None:
|
||||||
|
self.updated_exif["Exif.Image.Make"] = ascii(metadata.camera_make).strip("'")
|
||||||
|
|
||||||
|
|
||||||
|
def add_camera_model(self, metadata: PictureMetadata) -> None:
|
||||||
|
"""
|
||||||
|
Add camera model in Exif Model tag
|
||||||
|
"""
|
||||||
|
|
||||||
|
if metadata.camera_model is not None:
|
||||||
|
self.updated_exif["Exif.Image.Model"] = ascii(metadata.camera_model).strip("'")
|
||||||
|
|
||||||
def format_offset(self, offset: timedelta) -> str:
|
def format_offset(self, offset: timedelta) -> str:
|
||||||
"""Format offset for OffsetTimeOriginal. Format is like "+02:00" for paris offset
|
"""Format offset for OffsetTimeOriginal. Format is like "+02:00" for paris offset
|
||||||
>>> format_offset(timedelta(hours=5, minutes=45))
|
>>> format_offset(timedelta(hours=5, minutes=45))
|
||||||
@ -156,7 +190,6 @@ class Writer():
|
|||||||
offset_hour, remainer = divmod(offset.total_seconds(), 3600)
|
offset_hour, remainer = divmod(offset.total_seconds(), 3600)
|
||||||
return f"{'+' if offset_hour >= 0 else '-'}{int(abs(offset_hour)):02}:{int(remainer/60):02}"
|
return f"{'+' if offset_hour >= 0 else '-'}{int(abs(offset_hour)):02}:{int(remainer/60):02}"
|
||||||
|
|
||||||
|
|
||||||
def localize(self, naive_dt: datetime, metadata: PictureMetadata) -> datetime:
|
def localize(self, naive_dt: datetime, metadata: PictureMetadata) -> datetime:
|
||||||
"""
|
"""
|
||||||
Localize a datetime in the timezone of the picture
|
Localize a datetime in the timezone of the picture
|
||||||
@ -188,7 +221,6 @@ class Writer():
|
|||||||
|
|
||||||
return tz.localize(naive_dt)
|
return tz.localize(naive_dt)
|
||||||
|
|
||||||
|
|
||||||
def _from_dms(self, val: str) -> float:
|
def _from_dms(self, val: str) -> float:
|
||||||
"""Convert exif lat/lon represented as degre/minute/second into decimal
|
"""Convert exif lat/lon represented as degre/minute/second into decimal
|
||||||
>>> _from_dms("1/1 55/1 123020/13567")
|
>>> _from_dms("1/1 55/1 123020/13567")
|
||||||
@ -206,7 +238,6 @@ class Writer():
|
|||||||
|
|
||||||
return float(deg) + float(min) / 60 + float(sec) / 3600
|
return float(deg) + float(min) / 60 + float(sec) / 3600
|
||||||
|
|
||||||
|
|
||||||
def _to_dms(self, value: float) -> Tuple[int, int, float]:
|
def _to_dms(self, value: float) -> Tuple[int, int, float]:
|
||||||
"""Return degree/minute/seconds for a decimal
|
"""Return degree/minute/seconds for a decimal
|
||||||
>>> _to_dms(38.889469)
|
>>> _to_dms(38.889469)
|
||||||
@ -223,7 +254,6 @@ class Writer():
|
|||||||
|
|
||||||
return deg, int(min), round(sec, 8)
|
return deg, int(min), round(sec, 8)
|
||||||
|
|
||||||
|
|
||||||
def _to_exif_dms(self, value: float) -> str:
|
def _to_exif_dms(self, value: float) -> str:
|
||||||
"""Return degree/minute/seconds string formated for the exif metadata for a decimal
|
"""Return degree/minute/seconds string formated for the exif metadata for a decimal
|
||||||
>>> _to_exif_dms(38.889469)
|
>>> _to_exif_dms(38.889469)
|
||||||
|
Loading…
Reference in New Issue
Block a user