tykayn/wololo
1
0
Fork 0
Code Issues Pull Requests Actions Projects Releases Wiki Activity

réinit

Browse Source
This commit is contained in:
Tykayn 2025-01-15 22:20:14 +01:00 committed by tykayn
commit 996524bc6d
107 changed files with 1295536 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
.gitignore vendored Normal file
View File

@ -0,0 +1,9 @@
**/*/node_modules
/node_modules/
/coverage
.yarn
.yarn/*
venv
etalab_data/**/*.json
etalab_data/**/*.geojson
etalab_data/**/*.csv

17738
.pnp.cjs generated Executable file
View File

File diff suppressed because one or more lines are too long

2040
.pnp.loader.mjs generated Normal file
View File

File diff suppressed because it is too large Load Diff

235
LICENSE.md Normal file
View File

@ -0,0 +1,235 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for software and other kinds of works, specifically designed to ensure cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, our General Public Licenses are intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License which gives you legal permission to copy, distribute and/or modify the software.
A secondary benefit of defending all users' freedom is that improvements made in alternate versions of the program, if they receive widespread use, become available for other developers to incorporate. Many developers of free software are heartened and encouraged by the resulting cooperation. However, in the case of software used on network servers, this result may fail to come about. The GNU General Public License permits making a modified version and letting the public access it on a server without ever releasing its source code to the public.
The GNU Affero General Public License is designed specifically to ensure that, in such cases, the modified source code becomes available to the community. It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version.
An older license, called the Affero General Public License and published by Affero, was designed to accomplish similar goals. This is a different license, not a version of the Affero GPL, but Affero has released a new version of the Affero GPL which permits relicensing under this license.
The precise terms and conditions for copying, distribution and modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based on the Program.
To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work.
A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source.
The Corresponding Source for a work in source code form is that same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures.
When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified it, and giving a relevant date.
b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices".
c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so.
A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways:
a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b.
d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d.
A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product.
"Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made.
If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM).
The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or authors of the material; or
e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors.
All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11).
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party.
If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it.
A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software. This Corresponding Source shall include the Corresponding Source for any work covered by version 3 of the GNU General Public License that is incorporated pursuant to the following paragraph.
Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the work with which it is combined will remain governed by version 3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of the GNU Affero General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU Affero General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU Affero General Public License, you may choose any version ever published by the Free Software Foundation.
If the Program specifies that a proxy can decide which future versions of the GNU Affero General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program.
Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer network, you should also make sure that it provides a way for users to get its source. For example, if your program is a web application, its interface could display a "Source" link that leads users to an archive of the code. There are many ways you could offer source, and different solutions will be better for different programs; see section 13 for the specific requirements.
You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU AGPL, see <http://www.gnu.org/licenses/>.

106
README.md Normal file
View File

@ -0,0 +1,106 @@
# Wololo! Conversion de jeux de données geojson en tags OSM
Ce dépot propose des outils de conversion et de mise en qualité de jeux de données à partir de geojson ou de csv afin de les importer dans OpenStreetMap.
Pour réaliser les conversions de jeux de données ce dépot propose un ensemble de déclarations dans un simple objet JS.
# installation
utiliser npm, yarn ou pnpm
```bash
pnpm i
```
## Convertir le jeu de données des IRVE / Bornes de recharge
Pour récupérer le jeu de données pour les bornes elec et convertir le geojson en fichier osm, lancer le script bash
```bash
make irve
```
Export depuis overpass turbo dans OSM
https://overpass-turbo.eu/s/1yhY
## config
Configurer le filtre de département et les autres options dans convert_to_osm_tags.ts.
## utilisation
Après avoir bien installé les paquets et récupéré un fichier Geojson de données ouvertes, vous devez avoir une configuration de conversion, puis lancer le script principal de conversion.
```bash
ts-node convert_to_osm_tags.ts
```
Plusieurs options sont disponible pour affiner la conversion:
```
--source=chemin/mon_fichier.geojson
--department=12 (aucun filtre de département par défaut)
--engine=true (true par défaut)
--engine-config=mappingIssy2Roues (mappingIRVE par défaut)
--output-file=autre_nom_de_sortie.json
```
La sortie est filtrée selon le mappage donné et la zone de code postal de la config.
Vous obtenez un résultat converti en json dans le dossier /output. Il s'agit toujours d'un Geojson mais nettoyé selon votre convertisseur, il peut ensuite être converti en fichier osm à ajouter dans JOSM.
Pour le convertir, utiliser geojsontoosm, une dépendance du projet.
` geojsontoosm mon_fichier_converti.json > le_fichier_pour_josm.osm`
Ce projet a été réalisé pour l'intégration des bornes de recharge élec dans OSM à partir des données data gouv.
## Exemple parkings 2 roues d'Issy les moulineaux
Fichier de mapping: `mappings/converters/configIssy_OpenData_2roues.ts`
Fichier de données: `etalab_data/cyclabilité/issy_les_mx_cyclabilité.json`
Commande de conversion:
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/cyclabilité/issy_les_mx_cyclabilité.json --o
utput-file=issy_parking_2roues.json --engine-config=mappingIssy2Roues
```
Résultat: `output/my_converted_data_set__mappingIssy2Roues.json`[makefile](makefile)
# Projets d'intégrations de données
## Ajout d'un jeu de données et de sa conversion
[doc sur l'ajout d'un jeu de données](ajout_jeu_de_données.md)
## Tests
[doc sur les tests Jest](testing.md)
## Examen des jeux de données Osmose
Osmose permet la détection et suggestion de corrections de données dans OSM. On peut utiliser un export de ces données à la conversion d'un Mapper.
Pour cela, récupérez un jeu de données et utilisez le convertisseur Osmose de votre choix sur ces données afin d'en faire un nouveau jeu de données.
[doc sur l'ajout d'un jeu de données](ajout_jeu_de_données.md)
## Bornes de recharge - IRVE
Graphique des bornes de recharges à ajouter selon l'analyse Osmose:
[!https://osmose.openstreetmap.fr/en/issues/graph.png?item=8410](https://osmose.openstreetmap.fr/en/issues/graph.png?item=8410)
![graphique des bornes à ajouter](https://osmose.openstreetmap.fr/en/issues/graph.png?item=8410 "graphique")
* Tableau de conversion de l'open data:
https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
# Ressources
Documentation website fait avec mkdocs:
`mkdocs serve` et aller sur http://localhost:8000
# Contactez l'auteur, contribuez
contact at cipherbliss.com
sur Mastodon : mastodon.cipherbliss.com/@tykayn
# Roadmap
## Correctifs sur les données ajoutées
## bornes de recharge
Corrections à mettre dans le mappeur [./mappings/converters/configIRVE.ts]() et à modifier par édition en masse dans JOSM.
* ne pas mettre ref si la valeur vaut "non concerné"
* convertir les "False" et "0" en "no" pour les clés
* authentication:none
* fee (désactivé, l'open data est trop pourrie pour l'utiliser)
* socket:type2_cable
* convertir les numéros de téléphone avec un préfixe international +33 si commençant par 0
Données possible d'intégrer en plus avec le même jeu de données:
* ne pas mettre ref:EU:EVSE si la valeur vaut "non concerné"
* ajouter les références européennes ref:EU:EVSE.
* l'accessibilité => wheelchair pour certaines valeurs limitées.
## projets à venir
### Toilettes publiques
### Défibrillateurs - DAE
### Centres de santé
### Bus

6
babel.config.js Normal file
View File

@ -0,0 +1,6 @@
module.exports = {
presets: [
['@babel/preset-env', {targets: {node: 'current'}}],
'@babel/preset-typescript',
],
};

183
build_departments.ts Normal file
View File

@ -0,0 +1,183 @@
/**
build all files for departements from the dataset and the mapping engine
**/
const {exec} = require('child_process');
const execSync = require("child_process").execSync;
let min_department = 1;
const max_departement = 1
let enable_domtom = true;
// const max_departement = 95
function exec_command(command: string) {
console.log('=> command',
command
)
const result = execSync(command);
// convert and show the output.
console.log(result.toString("utf8"));
}
function build_department_geojson(department: string) {
let command = `ts-node convert_to_osm_tags.ts --department=${department}`;
console.log('command', command)
exec_command(command)
}
function convert_geojson_to_osm_for_department(department: string) {
let command = `geojsontoosm output/my_converted_data_set_filtered_zipcode_${department}.json > osm_output/bornes-irve-filetered-from-etalab-opendata_dep_${department}.osm`;
console.log('command', command)
exec_command(command)
}
for (let ii = min_department; ii <= max_departement; ii++) {
let departement = '' + ii
build_department_geojson(departement)
}
for (let ii = min_department; ii <= max_departement; ii++) {
let departement = '' + ii
convert_geojson_to_osm_for_department(departement)
}
// build_department_geojson('2A')
// convert_geojson_to_osm_for_department('2A')
// build_department_geojson('2B')
// convert_geojson_to_osm_for_department('2B')
if (enable_domtom) {
// départements corse et dom tom
let more_departements = ['2A', '2B'
, '971'
, '972'
, '973'
, '974'
, '975'
, '976']
more_departements.forEach((department => {
console.log(' mapping data for department ' + department)
let departement_string = '' + department
build_department_geojson(departement_string)
}))
more_departements.forEach((department => {
console.log('convert to osm ' + department)
let departement_string = '' + department
convert_geojson_to_osm_for_department(departement_string)
}))
}
/**
Numéro Départements Chef-lieu
01 Ain Bourg-en-Bresse
02 Aisne Laon
03 Allier Moulin
04 Alpes-de-Haute-Provence Digne
05 Hautes-Alpes Gap
06 Alpes-Maritimes Nice
07 Ardèche Privas
08 Ardennes Charleville-Mézières
09 Ariège Foix
10 Aube Troyes
11 Aude Carcassonne
12 Aveyron Rodez
13 Bouches-du-Rhône Marseille
14 Calvados Caen
15 Cantal Aurillac
16 Charente Angoulême
17 Charente-Maritime La Rochelle
18 Cher Bourges
19 Corrèze Tulle
21 Côte-d'Or Dijon
22 Côtes d'Armor Saint-Brieuc
23 Creuse Guéret
24 Dordogne Périgueux
25 Doubs Besançon
26 Drôme Valence
27 Eure Evreux
28 Eure-et-Loir Chartres
29 Finistère Quimper
30 Gard Nîmes
31 Haute-Garonne Toulouse
32 Gers Auch
33 Gironde Bordeaux
34 Hérault Montpellier
35 Ille-et-Vilaine Rennes
36 Indre Châteauroux
37 Indre-et-Loire Tours
38 Isère Grenoble
39 Jura Lons-le-Saunier
40 Landes Mont-de-Marsan
41 Loir-et-Cher Blois
42 Loire St-Étienne
43 Haute-Loire Le Puy
44 Loire-Atlantique Nantes
45 Loiret Orléans
46 Lot Cahors
47 Lot-et-Garonne Agen
48 Lozère Mende
49 Maine-et-Loire Angers
50 Manche St-Lô
51 Marne Châlons-sur-Marne
52 Haute-Marne Chaumont
53 Mayenne Laval
54 Meurthe-et-Moselle Nancy
55 Meuse Bar-le-Duc
56 Morbihan Vannes
57 Moselle Metz
58 Nièvre Nevers
59 Nord Lille
60 Oise Beauvais
61 Orne Alençon
62 Pas-de-Calais Arras
63 Puy-de-Dôme Clermont-Ferrand
64 Pyrénées-Atlantiques Pau
65 Hautes-Pyrénées Tarbes
66 Pyrénées-Orientales Perpignan
67 Bas-Rhin Strasbourg
68 Haut-Rhin Colmar
69 Rhône Lyon
70 Haute-Saône Vesoul
71 Saône-et-Loire Mâcon
72 Sarthe Le Mans
73 Savoie Chambéry
74 Haute-Savoie Annecy
75 Paris Paris
76 Seine-Maritime Rouen
77 Seine-et-Marne Melun
78 Yvelines Versailles
79 Deux-Sèvres Niort
80 Somme Amiens
81 Tarn Albi
82 Tarn-et-Garonne Montauban
83 Var Toulon
84 Vaucluse Avignon
85 Vendée La Roche-sur-Yon
86 Vienne Poitiers
87 Haute-Vienne Limoges
88 Vosges Épinal
89 Yonne Auxerre
90 Territoire de Belfort Belfort
91 Essonne Evry
92 Hauts-de-Seine Nanterre
93 Seine-St-Denis Bobigny
94 Val-de-Marne Créteil
95 Val-D'Oise Pontoise
2A Corse-du-Sud Ajaccio
2B Haute-Corse Bastia
971 Guadeloupe Basse-Terre
972 Martinique Fort-de-France
973 Guyane Cayenne
974 La Réunion Saint-Denis
976 Mayotte Mamoudzou
**/

476
convert_to_osm_tags.ts Normal file
View File

@ -0,0 +1,476 @@
/**
* convertisseur de données de bornes de recharge électrique à partir de données Chargemap et open data Etalab
*/
import * as fs from 'fs'
import mappingConfigIRVE from './mappings/converters/configIRVE'
import ConfigIRVE from './mappings/converters/configIRVE'
import mappingIssy2Roues from './mappings/converters/configIssy_OpenData_2roues'
import mappingConfigIRVEFromOsmose from './mappings/converters/configIRVE_osmose'
import mappingConfigIRVE_simple from './mappings/converters/mappingConfigIRVE_simple'
import mappingTest from './mappings/converters/configTest'
import mapping_engine from './mappings/engine'
import MappingConfigType, {BoundingBoxCoordinatesType, FeatureCollection} from "./mappings/mapping-config.type";
import utils from './mappings/utils'
import mappingRouenParkingVelos from "./mappings/converters/configRouen_OpenData_velo_parkings";
import mappingFINESS from "./mappings/converters/configFINESS";
import MappingArbresIssy from "./mappings/converters/configArbresIssy";
import MappingArbresEchirolles from "./mappings/converters/configArbresEchirolles";
import MappingArbres92 from "./mappings/converters/configArbresHautsDeSeine";
import MappingMuseums from "./mappings/converters/configMuseums";
import MappingRouenPAV from "./mappings/converters/configRouen_PAV";
import MappingAskAngela from "./mappings/converters/configAskAngela";
import MappingPlanningFamlial from "./mappings/converters/configPlanningFamilial";
import MappingSurveillanceRouen from "./mappings/converters/configSurveillance";
const limitWarningPercentageChangeInPoints = 5; // show a warning when more than N percent of the number of points changed
const allowed_configs: any = {
mappingIssy2Roues,
mappingConfigIRVE,
mappingConfigIRVEFromOsmose,
mappingConfigIRVE_simple,
mappingTest,
ConfigIRVE,
mappingRouenParkingVelos,
mappingFINESS,
MappingArbresIssy,
MappingArbresEchirolles,
MappingArbres92,
MappingMuseums,
MappingRouenPAV,
MappingAskAngela,
MappingPlanningFamlial,
MappingSurveillanceRouen
};
const minimist = require('minimist');
const debugLog = utils.debugLog;
let mini_arguments: any = minimist(process.argv.slice(2))
let use_mapping_engine = true
let output_supplement = '';
let sourceFilePathGeoJson = './data_other/' + output_supplement + '.json'
// wip filter
let filterOnBoundingBox = true
filterOnBoundingBox = false
let boundingBoxCoordinates: BoundingBoxCoordinatesType = {
xMin: 1.91,
xMax: 2.38,
yMin: 48.7,
yMax: 48.4,
}
let default_engine_conf_choice: string = 'ConfigIRVE'
let engine_conf_choice: string = 'ConfigIRVE'
let filterCoordinates = true
filterCoordinates = false
let enable_filter_on_department = true
enable_filter_on_department = false
let filterDepartment = 91
if (mini_arguments['department']) {
filterDepartment = mini_arguments['department']
enable_filter_on_department = true
}
let osmoseFormat = false;
if (mini_arguments['osmose']) {
osmoseFormat = mini_arguments['osmose']
}
if (mini_arguments['source']) {
sourceFilePathGeoJson = mini_arguments['source']
}
if (mini_arguments['engine']) {
use_mapping_engine = mini_arguments['engine']
}
if (mini_arguments['engine-config']) {
engine_conf_choice = mini_arguments['engine-config']
output_supplement += "_" + engine_conf_choice
}
let Mapping_engine: any;
let outname = 'converted_'
if (mini_arguments['output-file']) {
outname = mini_arguments['output-file'];
}
if (mini_arguments['outname']) {
outname = mini_arguments['outname'];
}
if (mini_arguments['testingConfig']) {
console.log('testing')
Mapping_engine = new mapping_engine(mappingTest)
} else if (osmoseFormat) {
console.log(' *********** we use osmose converter *********')
Mapping_engine = new mapping_engine(mappingConfigIRVEFromOsmose)
} else {
Mapping_engine = new mapping_engine(mappingConfigIRVE)
}
let filterZipCode = new RegExp(`^${filterDepartment}`)
let filterZipCodeAdresse = new RegExp(` ${filterDepartment}`)
let filteredName = ''
if (enable_filter_on_department) {
filteredName = '_filtered_zipcode_' + filterDepartment
} else if (filterOnBoundingBox) {
filteredName = '_filtered_bbox_' + boundingBoxCoordinates.xMin + '-' + boundingBoxCoordinates.xMax + '_' + boundingBoxCoordinates.yMin + '-' + boundingBoxCoordinates.yMax
}
let pointCounterMax: number = 1000000
let limitConversionToFirstPoint: boolean = false
// limitConversionToFirstPoint = true
if (limitConversionToFirstPoint) {
pointCounterMax = 1
}
let defaultPropertiesOfPoint: any = {
'amenity': 'charging_station'
}
let converted_geo_json: any = {
type: 'FeatureCollection',
features: []
}
let output_folder = 'output';
/**
* crée un fichier dans le dossier par défaut, output
* @param fileName
* @param fileContent
*/
function writeFile(fileName: string, fileContent: any) {
if (outname) {
fileName = '' + outname;
}
let write_path = `./${output_folder}/${fileName}`
debugLog("write file \n", fileName, write_path)
console.log("-------- write file \n", fileName, "\n", write_path)
return fs.writeFile(
write_path,
fileContent,
'utf8',
(err) => {
if (err) {
debugLog(`Error writing file: ${err}`)
} else {
debugLog(`File ${fileName} is written successfully!`)
}
}
)
}
/**
*
* @param sourceFilePath
* @param mapping
* @param pointCounterMax
* @param boundingBoxCoordinates
*/
function convertDataFromSource(sourceFilePath: string, mapping: MappingConfigType, pointCounterMax: number, boundingBoxCoordinates: any) {
debugLog('convert data: source file from :', sourceFilePath)
fs.readFile(sourceFilePath, 'utf8', function (err, data) {
let point_counter = 0
let feature_points_after_filter: any = []
if (err) {
debugLog('erreur à l ouverture du chemin ' + sourceFilePath)
return console.error(err)
}
let data_transformed: FeatureCollection = JSON.parse(data)
if (data_transformed.features) {
console.log('------ data found in source, features:', data_transformed.features.length)
// find interesting list of points to use
let list_of_points: any = data_transformed.features
debugLog('listOfPoints.length', list_of_points.length)
if (limitConversionToFirstPoint) {
debugLog('limitConversionToFirstPoint enabled')
if (mapping?.filters?.offset) {
mapping.filters.offset = 1
}
}
if (mapping?.filters?.offset) {
debugLog('filter offset enabled', mapping.filters.offset)
list_of_points = list_of_points.splice(0, mapping.filters.offset)
}
console.log('features count before:', list_of_points.length)
/**
* filtering
* run filters before mapping OSM tags
*/
if (mapping.filters) {
if (mapping.filters.exclude_point_if_tag_not_empty) {
list_of_points = Mapping_engine.filterListOfPointsByExcludingIfKeyFilled(list_of_points, mapping.filters.exclude_point_if_tag_not_empty)
}
if (mapping.filters.filter_points_lesser_than_NkW) {
list_of_points = Mapping_engine.filterListOfPointsByExcludingIfMaxPowerIsLesserThan(mapping.filters.filter_points_lesser_than_NkW, list_of_points)
}
}
// for each point from the data source, filter if we take it or not
list_of_points.forEach((feature_point: any) => {
let regex_filter_test_result = true
let remove_original_key = false;
// fix des jeux de données qui ne sont pas des geojson
let properties_list: any = []
if (feature_point.geo_point_2d) {
feature_point["type"] = "Feature"
feature_point["geometry"] = {
"coordinates": feature_point.geo_point_2d[0],
"type": "Point"
}
}
if (feature_point['properties']) {
properties_list = feature_point.properties
if (feature_point.properties['tags']) {
properties_list = feature_point.properties.tags
}
}
if (!feature_point["properties"]) {
console.log('jeu de données sans properties à la base. On tente un mapping interne...')
let properties_list_to_move: any = Object.keys(feature_point)
feature_point["properties"] = {}
properties_list_to_move.forEach((tagKey: string, tagValue: any) => {
feature_point["properties"][tagKey] = feature_point[tagKey]
})
}
// debug sur un offset de 1 seul point
// console.log('properties_list', properties_list)
/**
* exclusion des points sur un filtrage exclude_point_if_tag_not_empty
* utile pour les analyses comportant des points déjà présents dans osm qui ont un id_osm par exemple.
*/
let properties_keys = Object.keys(properties_list)
properties_keys.forEach((tagKey: string) => {
let tagValue = properties_list[tagKey]
if (mapping.filters?.exclude_point_if_tag_not_empty?.indexOf(tagKey) !== undefined && tagValue && tagValue !== null) {
// console.log('la clé est elle présente dans les filtres d exclusion?', tagKey, mapping.filters?.exclude_point_if_tag_not_empty?.indexOf(tagKey))
// console.log('tagValue', tagValue)
// console.log('remove original' , tagKey )
remove_original_key = true;
}
})
if (remove_original_key) {
return;
}
/**
* filtre sur les départements selon la propriété consolidated_code_postal ou adresse_station
*/
if (enable_filter_on_department) {
debugLog('filtre sur les départements activé')
regex_filter_test_result = (
filterZipCode.test(feature_point.properties.consolidated_code_postal)
||
filterZipCodeAdresse.test(feature_point.properties.adresse_station)
)
} else {
debugLog('pas de filtre sur les départements')
}
/**
* filtre sur un carré de coordonnées
*/
if (filterOnBoundingBox) {
debugLog('filtre sur les coordonnées bounding box activé')
let x = feature_point.properties.coordonneesXY[0]
let xMin = boundingBoxCoordinates.xMin
let xMax = boundingBoxCoordinates.xMax
let yMin = boundingBoxCoordinates.yMin
let yMax = boundingBoxCoordinates.yMax
let y = feature_point.properties.coordonneesXY[1]
regex_filter_test_result = (
(x >= xMin && x <= xMax)
&&
(y >= yMin && y <= yMax)
)
} else {
debugLog('pas de filtre sur les coordonnées bounding box')
}
// TODO add filter offset max
// filter points depending on zipcode
if (regex_filter_test_result) {
feature_points_after_filter.push(feature_point)
debugLog(' +1 point', point_counter)
point_counter++
}
})
/**
* conversion
*/
debugLog(' after filtering, feature_points_after_filter number of points: ', feature_points_after_filter.length)
feature_points_after_filter.forEach((feature_point: any) => {
debugLog('convert : work on 1 point')
let mapped_point: any = {}
if (use_mapping_engine) {
mapped_point = Mapping_engine.mapElementFromConf(feature_point)
debugLog('mapped_point', mapped_point)
} else {
debugLog('convert :using simple converter on feature point', feature_point)
mapped_point = mapElementFromConfSimple(feature_point, mapping)
}
if (mapped_point) {
converted_geo_json.features.push(mapped_point)
debugLog('convert : added one point to converted_geo_json')
} else {
debugLog('convert : !!! there is no map one point')
}
})
// output new geojson
debugLog('convert : convertedGeoJson.features.length', converted_geo_json.features.length)
console.log('converted_geo_json.features.length', converted_geo_json.features.length)
// write file on disk
if (converted_geo_json.features.length) {
if (output_supplement) {
output_supplement = '_' + output_supplement;
}
let fileNameToWrite = '_' + filteredName + output_supplement + '.geojson'
console.log('converted features:', converted_geo_json.features.length)
console.log('différences nombre de features: ', data_transformed.features.length - converted_geo_json.features.length)
const percentChange = (data_transformed.features.length) / 100 * converted_geo_json.features.length;
console.log('Changement de features', percentChange, '%')
if (percentChange > limitWarningPercentageChangeInPoints) {
console.log(' /!\\ pas mal de points en moins, plus de ' + percentChange + '%')
}
debugLog('convert : write file ', fileNameToWrite)
// console.log('mapping_engine.stats', Mapping_engine.stats)
console.log('mapping config name:', Mapping_engine.getConfig().config_name)
writeFile(fileNameToWrite, JSON.stringify(converted_geo_json, null, 2))
} else {
console.log('convert : no writing of file, because there is no converted feature')
}
return converted_geo_json
}
})
}
/**
* retuns the converted element from mapping config if present, null otherwise
*/
function mapElementFromConfSimple(featurePoint: any, mappingConfig: any) {
let mappingKeys = Object.keys(mappingConfig)
let featurePointPropertiesKeys = Object.keys(featurePoint.properties)
debugLog('keys', mappingKeys, featurePointPropertiesKeys)
let newProperties: any = defaultPropertiesOfPoint
// reinit properties of current point
let basePoint = Object.create(featurePoint)
basePoint.type = featurePoint.type
basePoint.geometry = featurePoint.geometry
basePoint.properties = newProperties
// apply new properties if found in mapping config
featurePointPropertiesKeys.forEach((pointKeyName: string) => {
if (mappingKeys.indexOf(pointKeyName) !== -1) {
// debugLog('found element', pointKeyName, '=>', mappingConfig[pointKeyName], 'value : ', featurePoint.properties[pointKeyName])
let convertedValue: any = ''
if (utils.isBooleanKey(pointKeyName)) {
let copyOfValue: any = '' + featurePoint.properties[pointKeyName]
if (typeof copyOfValue === typeof Object && copyOfValue.key_converted) {
copyOfValue = copyOfValue.key_converted
}
convertedValue = copyOfValue.toLowerCase() == 'true' ? 'yes' : 'no'
} else {
convertedValue = featurePoint.properties[pointKeyName]
}
if (convertedValue) {
let convertedKey: any = mappingConfig[pointKeyName]
newProperties[convertedKey] = convertedValue
}
}
})
debugLog('basePoint', basePoint)
return basePoint
}
function setMappingConfigFromName(engine_conf_choice: string) {
console.log('------- sourceFilePathGeoJson', sourceFilePathGeoJson)
console.log('------- engine_conf_choice', engine_conf_choice)
if (use_mapping_engine) {
debugLog(' - using mapping engine')
debugLog(' - pointCounterMax', pointCounterMax)
if (osmoseFormat) {
Mapping_engine.setConfig(mappingConfigIRVEFromOsmose)
} else {
if (engine_conf_choice !== default_engine_conf_choice && Object.keys(allowed_configs).indexOf("mappingIssy2Roues") !== -1) {
Mapping_engine.setConfig(allowed_configs[engine_conf_choice])
} else {
console.error('mauvais paramètre de Mapping_engine: ' + engine_conf_choice, '.\n Veuillez en sélectionner un parmi ceux autorisés avec l option --engine-config=MaConfigQuiVaBien parmi ceux ci :', Object.keys(allowed_configs))
return
}
}
let currentMappingConfig = Mapping_engine.getConfig();
convertDataFromSource(sourceFilePathGeoJson, currentMappingConfig, pointCounterMax, boundingBoxCoordinates)
} else {
console.log(' ------ on utilise mappingConfigIRVE_simple')
let mappingConfigIRVE = mappingConfigIRVE_simple
convertDataFromSource(sourceFilePathGeoJson, mappingConfigIRVE, pointCounterMax, boundingBoxCoordinates)
}
}
/**
* Launch conversion of dataset
*/
function init() {
setMappingConfigFromName(engine_conf_choice)
}
init()

8
docs/Extracteurs.md Normal file
View File

@ -0,0 +1,8 @@
# Les extracteurs
Ces scripts permettent de récupérer des données sous licence ODBL depuis openstreetmap.
ils se trouvent dans mappings/extractors.
On peut les exécuter tous à la suite avec le script `run_all_extractors.sh`
```shell
bash update_scripts/run_all_extractors.sh
```

43
docs/ajout_jeu_de_données.md Normal file
View File

@ -0,0 +1,43 @@
# Ajout d'un jeu de données
Pour ajouter un jeu de données et sa conversion vous devrez faire un mapper et mettre le jeu de données dans le convertisseur.
Checklist:
- [] avoir un jeu de données de Points en Geojson à convertir, dans `etalab_data`, dans un dossier thématique (arbres, parking vélo, IRVE...).
- [] un convertisseur (Mapper) qui étend le type `MappingConfigType`
- [] une page de documentation markdown liée à ce jeu de données mentionnant une commande ts-node à lancer dans `docs`.
- [] noter l'url du jeu de données source dans `get_datasets.sh`
- [] ajouter l'import de ce Mapper dans `convert_to_osm_tags.ts`
- [] tester le rendu de la conversion en plaçant la sortie output dans JOSM.
## Récupérer un jeu de données
## Créer un Mapper
Créer un fichier ts dans mappings/converters qui expose un objet étendant `MappingConfigType`. Ici on fait un Mappeur qui ne changera qu'une seule clé en gardant toutes ses valeur, nom_amenageur sera converti en name.
```ts
export const mappingName: MappingConfigType = {
config_name: 'testing config mappingName',
config_author: 'tykayn <contact+geojson2osm@cipherbliss.com>',
default_properties_of_point: {},
tags: {
nom_amenageur : 'name'
},
add_not_mapped_tags_too: true,
source: {
geojson_path: '',
url: ''
},
filters: {},
}
```
Le détail complet des configurations de MappingConfigType est à voir dans la page [configuration_de_conversion.md]
## Ajouter le Mapper dans le point d'entrée de conversion
Choisir le jeu de données en changeant la configuration dans `convert_to_osm_tags.ts`.
Importer son Mapper dans `convert_to_osm_tags.ts` et le faire charger dans l'[Engine](engine.md) dans la partie qui réalise le chargement d'un Mapper.
Ajouter le nom du mapper dans le tableau `allowed_configs`, puis remplir une condition pour que l'Engine prenne le bon modèle de mapping.
# Debug
activer les logs dans le fichier des `utils.ts`.

27
docs/arbres.md Normal file
View File

@ -0,0 +1,27 @@
# Arbres
Des jeux de données sont disponibles pour lister les abres, suivre leur évolution, et mettre en avant les arbres monumentaux, dignes d'intérêt touristique.
## Issy les moulineaux
- ID_ARBRE : ref:FR:issy_les_mx:id_arbre
- CADUC_PERS : leaf_cycle=evergreen pour persistant , deciduous pour caduque
- HAUTEUR est_height=* hauteur estimée
- FEUIL_CONI : feuillu leaf_type=broadleaved / conifère leaf_type=needleleaved
- GENESPVAR : nom d'espèce species=*
- AN_PLANT : date_start
- VERNACULAIRE : species:FR=*
- PARTICULAR : Majeur, Remarquable : historic=monument
- FORME: tree_shape = curtain / free / half_free <= Architecturé, rideau / Libre / Semi-libre
# conversion
Abres d'issy les moulineaux
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/arbres/arbres_issy.geojson --output-file=arbres_issy.geojson --engine-config=MappingArbresIssy
```
Cadastre vert:
```bash
ts-node convert_to_osm_t
ags.ts --source=etalab_data/arbres/arbres_92.json --output-file=arbres_92.geojson --engine-config=MappingArbres92
```

23
docs/ask_angela.md Normal file
View File

@ -0,0 +1,23 @@
# Ask Angela, dispositif anti harcèlement
Les commerces et autres lieux adhérant au dispositif
# Conversion
## Conversion des données de Toulouse
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/toulouse_askangela.json --output-file=toulouse_askangela.json --engine-config=MappingAskAngela
```
## Conversion des données de Bordeaux
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/bordeaux_askangela.json --output-file=bordeaux_askangela.json --engine-config=MappingAskAngela
```
## Conversion des données de Poitiers
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/poitiers_askangela.json --output-file=poitiers_askangela.json --engine-config=MappingAskAngela
```
## Conversion des données de Lyon
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/lyon_askangela.json --output-file=lyon_askangela.json --engine-config=MappingAskAngela
```

BIN
docs/carte_numéros_départements_fr.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 90 KiB

133
docs/configuration_de_conversion.md Normal file
View File

@ -0,0 +1,133 @@
# Configuration de conversion
Ce fichier de configuration est utilisé pour convertir un jeu de données ouvertes en des données exploitables par des tags OpenStreetMap. Il décrit les règles de conversion pour chaque propriété des données ouvertes et les valeurs attendues pour chaque tag OpenStreetMap.
Structure du fichier de configuration
Le fichier de configuration est un objet JSON du type MappingConfigType qui contient les propriétés suivantes :
| Clé | Sous-clé | Sous-sous-clé | Description |
|-----------------------------|---------------------------| --- | --- |
| config_name | | | le nom de la configuration |
| config_author | | | le nom de l'auteur de la configuration avec son email entre chevrons |
| source | | | infos sur la provenance des données à convertir |
| source | geojson_path | | lien relatif dans ce dépôt vers le fichier téléchargé |
| source | url | | lien absolu de la source, sur le portail datagouv par exemple |
| add_not_mapped_tags_too | | | un boolean indiquant si les tags non détectés par la configuration doivent être ajoutés également aux points convertis |
| default_properties_of_point | | | un objet qui décrit les clés et valeur par défaut à mettre pour les points |
| filters | | | un objet qui décrit les filtres à appliquer aux données ouvertes |
| filters | enable_coordinates_filter | | booléen, permet d'activer un filtre sur les points se trouvant dans certaines coordonnées |
| filters | properties | | objet listant colonne et expression régulières pour ne garder que ce qui correspond |
| filters | properties | consolidated_code_postal | | exemple de regex pour trouver les points dont le code postal commence par 76 ou 27 : '^[76|27]' |
| filters | bounding_box | | tableau décrivant un rectangle de coordonnées: Xstart Ystart Xend Yend |
| filters | enable_properties_filter | | objet décrivant les regex à chercher sur certaines colonnes pour ne garder que les objets concernés |
| tags | | | un objet qui décrit les règles de conversion pour chaque tag OpenStreetMap |
# Propriétés de conversion des tags
Chaque clé donne une colonne du jeu de données à convertir en tag OSM, soit en conservant la valeur si on lui donne une simple chaîne de caractères en valeur, soit en lui apportant des modifications plus précises avec un objet en valeur.
Par exemple, nous avons un jeu de données sur les bornes de recharge électriques et leur numéro de SIREN est donné dans une colonne "siren_amenageur", dans OSM on utilisera une clé 'owner:ref:FR:SIREN'.
```json
{
"config_name": "IRVE config",
"tags": {
"siren_amenageur": "owner:ref:FR:SIREN"
}
}
```
Ce qui équivaut à une utilisation d'objet ainsi
```json
{
"config_name": "IRVE config",
"tags": {
"siren_amenageur": {
"key_converted":"owner:ref:FR:SIREN"
}
}
}
```
Les propriétés possibles sont:
| Clé | Description | Type | Valeur par défaut |
|------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- | --- |
| conditional_values | Valeurs conditionnelles, détail ci après | objet | null |
| convert_to_boolean_value | Convertit la valeur en booléen | boolean | false |
| convert_to_phone | Convertir en numéro de téléphone | boolean | false |
| ignore_if_falsy | Ignorer cette ligne si elle vaut false ou 0 | boolean | false |
| key_converted | Convertir en clé OSM | string | null |
| remove_original_key | Enlever la ligne | boolean | false |
| socket_output_find_correspondances | Chercher une puissance de socket pour borne de recharge électrique | boolean | false |
Exemple:
```json
{
"tags": {
"puissance_nominale": {
"key_converted": "charging_station:output",
"socket_output_find_correspondances": true
},
"accessibilite_pmr": {
"key_converted": "wheelchair",
"conditional_values": {
"Non accessible": {
"value_converted": "no"
},
"Réservé PMR": {
"value_converted": "yes"
}
}
}
}
}
```
## Valeurs conditionnelles conditional_values
Un objet décrivant par ses clés des comportements à faire selon la valeur donnée.
Propriétés:
| Clé | Description | Type | Valeur par défaut |
| --- |-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| --- | --- |
| ignore_this_data | Ignorer, pour évacuer des valeurs indésirables | | |
| tags_to_add | Tags à ajouter | objet | |
| transform_function | Une fonction à appliquer à la valeur | | |
| convert_to_boolean_value | Convertit en 'yes' si la valeur est Truthy, et en 'no' si elle est Falsy. Et ne change pas la valeur si elle n'est ni Truthy ni Falsy. | | |
| value_converted | valeur à convertir | string ou number | |
| truthy_value | Convertir la valeur dans le cas où elle est Truthy. exemple: le jeu de données dit que la colonne cable_t2_attache vaut "True", mais on veut le convertir en "1". On met donc truthy_value: '1' | boolean | false |
| falsy_value | Convertir la valeur dans le cas où elle est Falsy, comme pour truthy_value. | boolean | false |
| invert_boolean_value | Inverser la valeur booléenne | boolean | false |
### Tags à ajouter
Quand on utilise la clé "tags_to_add", on lui met en valeur un tableau de tags OSM à ajouter.
Ici, ajout de trois tags si la valeur est `yes` pour `station_deux_roues`
```json
{
"tags": {
"station_deux_roues": {
"remove_original_key": true,
"conditional_values": {
"yes": {
"tags_to_add": [
{
"bicycle": "yes"
},
{
"scooter": "yes"
},
{
"motorcar": "no"
}
]
}
}
}
}
}
```

26
docs/deviner_les_correspondances.md Normal file
View File

@ -0,0 +1,26 @@
# Deviner les correspondances
Dans un Geojson on peut deviner à quel tag correspondent les propriétés des points de la Feature collection pour créer ses modèles de Mapper.
Examinez un geojson avec le script `propose_mapping_from_data.py`:
```shell
# exemple avec un jeu de données de musée
python propose_mapping_from_data.py etalab_data/musées/fr.geojson
```
Ce qui donne:
```json
{
"Identifiant Mus\u00e9ofile": "",
"R\u00e9gion administrative": "",
"D\u00e9partement": "",
"Commune": "",
"Nom officiel du mus\u00e9e": "",
"Adresse": "",
"Lieu": "",
"Code Postal": "",
"T\u00e9l\u00e9phone": "",
"URL": "",
"REF_Deps": ""
}
```

32
docs/engine.md Normal file
View File

@ -0,0 +1,32 @@
# Engine
Les conversions de jeux de données Geojson se font via des règles définies par les Mappers, appliqués par l'Engine. C'est là que les fonctions de conversions sont appliquées:
- conversion d'un numéro de téléphone dans un format réglementé comme dans le wiki OSM
- en booléen
- validation de format horaire
- filtrer les données selon des critères géographiques ou d'offset limité
Après initialisation de l'Engine avec [un `Mapper`](configuration_de_conversion.md), on lui passe un jeu de données geojson et on lance la conversion de données avec `mapElementFromConf()`.
Cela est réalisé dans le point d'entrée principal en node typescript: `convert_to_osm_tags.ts`
Les conversions de chaque propriété sont faites par la fonction convertProperty:
```
convertProperty(pointKeyName, mappingKeys, featurePoint, newProperties)
```
| paramètre | description |
|---------------|---------------------------------------|
| pointKeyName | clé de propriété du geojson |
| mappingKeys | liste des clés de propriétés du Point |
| featurePoint | Point geojson |
| newProperties | objet des nouvelles propriétés |
# Jardinage à partir d'OSM
L'option jardinage permet de ne pas convertir les propriétés de Geojson dans le cas où l'on extrait les données d'OSM.
Cela nécessite un mappeur différent pour appliquer des transformations de mise en qualité.
# Stats
L'engine publie des statistiques de conversion de données

8
docs/finess_santé.md Normal file
View File

@ -0,0 +1,8 @@
# fichier FINESS
Le fichier répertoriant les établissements de santé, hôpitaux, maisons médicales, pharmacies, laboratoires d'analyse...
# conversion
Commande de conversion:
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/finess/finess_idf.json --output-file=finess_idf.json --engine-config=mappingFINESS
```

97
docs/index.md Normal file
View File

@ -0,0 +1,97 @@
# Conversion de jeux de données geojson en tags OSM
Ce dépot propose des outils de conversion et de mise en qualité de jeux de données à partir de geojson ou de csv afin de les importer dans OpenStreetMap.
Une présentation de l'intégration des bornes de recharges électiques dans openstreetmap faite grâce à ce dépot est disponible sur le blog Cipher Bliss:
[https://www.cipherbliss.com/integration-des-bornes-de-recharge-de-vehicules-electriques-dans-openstreetmap/]()
# installation
utiliser npm, yarn ou pnpm
```bash
pnpm i
```
## Tout convertir d'un coup
Pour récupérer le jeu de données pour les bornes elec et convertir le geojson en fichier osm, lancer le script bash
```bash
make
```
Export depuis overpass turbo dans OSM
[https://overpass-turbo.eu/s/1yhY]()
## config
Configurer le filtre de département et les autres options dans convert_to_osm_tags.ts.
Vous pouvez voir le détail de la [création d'un nouveau Mapper ou convertisseur](ajout_jeu_de_données.md), les [propriétés possibles d'un convertisseur](configuration_de_conversion.md) en regardant celui des bornes de recharge de véhicule électrique par exemple: [MappingIRVESuisse](mappings/converters/configIRVE.ts) qui est une extension du Type [MappingConfigType](mappings/mapping-config.type.ts).
## utilisation
Après avoir bien installé les paquets et récupéré un fichier Geojson de données ouvertes, vous devez avoir une configuration de conversion, puis lancer le script principal de conversion.
```bash
ts-node convert_to_osm_tags.ts
```
Plusieurs options sont disponible pour affiner la conversion:
```
--source=chemin/mon_fichier.geojson
--department=12 (aucun filtre de département par défaut)
--engine=true (true par défaut)
--output-file=autre_nom_de_sortie.json
```
La sortie est filtrée selon le mappage donné et la zone de code postal de la config.
Vous obtenez un résultat converti en json dans le dossier /output. Il s'agit toujours d'un Geojson mais nettoyé selon votre convertisseur, il peut ensuite être converti en fichier osm à ajouter dans JOSM.
Pour le convertir, utiliser geojsontoosm, une dépendance du projet.
` geojsontoosm mon_fichier_converti.json > le_fichier_pour_josm.osm`
Ce projet a été réalisé pour l'intégration des bornes de recharge élec dans OSM à partir des données data gouv.
# Tests
Lancement de test avec Jest
# Projets d'intégrations de données
## Bornes de recharge - IRVE
Graphique des bornes de recharges à ajouter selon l'analyse Osmose:
[!https://osmose.openstreetmap.fr/en/issues/graph.png?item=8410](https://osmose.openstreetmap.fr/en/issues/graph.png?item=8410)
![graphique des bornes à ajouter](https://osmose.openstreetmap.fr/en/issues/graph.png?item=8410 "graphique")
* Tableau de conversion de l'open data:
https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
# Ressources
# Contactez l'auteur, contribuez
contact at cipherbliss.com
sur Mastodon : mastodon.cipherbliss.com/@tykayn
# Roadmap
## Correctifs sur les données ajoutées
## bornes de recharge
Corrections à mettre dans le mappeur [../mappings/converters/configIRVE.ts]() et à modifier par édition en masse dans JOSM.
* ne pas mettre ref si la valeur vaut "non concerné"
* convertir les "False" et "0" en "no" pour les clés
* authentication:none
* fee (désactivé, l'open data est trop pourrie pour l'utiliser)
* socket:type2_cable
* convertir les numéros de téléphone avec un préfixe international +33 si commençant par 0
Données possible d'intégrer en plus avec le même jeu de données:
* ne pas mettre ref:EU:EVSE si la valeur vaut "non concerné"
* ajouter les références européennes ref:EU:EVSE.
* l'accessibilité => wheelchair pour certaines valeurs limitées.
## projets à venir
- Toilettes publiques
- Défibrillateurs - DAE
- Centres de santé
- Bus
# infos en bonus IRVE
* [Informations du dataset IRVE](irve_dataset_info)
* [Amplitude de puissance dans le jeu de données des IRVE](irve_puissances.md)
* URL du Dépôt: [https://forge.chapril.org/tykayn/mapping-geojson-osm.git]()
# documentation faite avec mkdocs
le site officiel de mkdocs: [mkdocs.org](https://www.mkdocs.org).

309
docs/irve_dataset_info.md Normal file
View File

@ -0,0 +1,309 @@
# info sur les jeux de données
D'après le jeu de données Etalab
## conversion
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/arbres/suisse.geojson --output-file=irve_suisse.geojson --engine-config=configIRVESuisse
```
## liste des 310 opérateurs
- 270 AGENCY
- 2Ed Coutances
- 2F Production
- ABSOLUT CONCEPT
- Acelec Charge
- advenir@zeborne.com
- Aeroports de lyon
- Allego
- ALLENERGIES
- alterna energie
- ALU GRANON
- Ambroise Avocat
- ANYOS
- AQUA LOISIRS
- AREA Tech
- ART DECO FINITION
- arteco44
- ATELIERS PROVENCE LOISIRS
- Atlante | FR*ATL
- ATRACHATA
- Autel Netherlands B.V. | FR*AUT
- Automobile de l'Est
- AUTORECHARGE
- AUTORECHARGE
- AUTORECHARGE SAS
- Avomarks
- Bastide beaudinard & tours (BBT)
- BAUDUCEL
- Bénédictines du Sacré-Coeur de Montmartre
- BH RESTAURATION
- BIARS DISTRIBUTION
- BORNECO
- Borneco | FR*BHM
- Bornevo
- Bornevo Connect
- Bouygues Energies et Services
- BOUYGUES ENERGIES ET SERVICES
- box
- Camping des graniers
- CAP EMPLOI
- CAR2PLUG
- CHAPITEAUX SERVICE
- ChargePoint
- Chargepoint
- CHARGEPOLY
- CHARRIERE DISTRIBUTION
- CHATEAU DE RONZIERE
- COMMUNAUTE DE COMMUNE LE DOURDANNAIS EN HUREPOIX (CCDH)
- contact@autorecharge.fr
- COPRODEX
- Daltoner Avranches
- Daltoner Caen
- DALTONER CHERBOURG
- Daltoner Granville
- Daltoner Vire
- DIDIER CHARTON-VACHET
- Dropnplug
- e-nergyze
- E-TOTEM
- E-totem
- Easycharge services
- Ecophi mobility
- EcoPhi mobility
- EDF
- EDFRAY
- EGE NOEL BERANGER
- EGSM
- EIZMENDI TRAITEUR EVENEMENTS
- Ekoplug
- ELEC eCONNECT
- ELECTRA
- ELECTRIC 55 CHARGING
- Electriox Groupe
- electromaps
- ELECTROMAPS
- Electromaps
- ELEK BORDEAUX
- ENERGEM
- ENERGIE Eure-et-Loir
- ENERGIE EURE-ET-LOIR
- ENNESSER ET FILS
- ENTREPRISE ADAM
- ESPACE AUTO COURNON
- Est Auto
- EURO TAXI LINE
- EV MAP SAS
- EVBOX
- EVERON
- EVzen (SMEG Développement)
- evzen (SMEG Développement)
- evZen | FR*EVZ
- F.B. AUTOMOBILES
- Festilight
- Freshmile
- FRESHMILE
- freshmile
- FReshmile
- Freshmile SAS
- GABMAG17
- GABORIT BOCAGE SERVICES
- Gamba et Rota
- GARAGE HENRY
- garage Henry herve
- Garage lefebvre
- GEDIBOIS BATI COLMAN
- GENE ELEC 35
- GF3e
- Golf de La Wantzenau
- Green Diffusion
- Green Technologie
- GREEN TECHNOLOGIE
- Green To Wheel
- GreenYellow | FR*GYM
- GROFF SAS
- GROUPE COURTOIS SOCOHY
- GROUPE LGE
- GROUPE LGE SOCIETE AUTOPLUG
- Grunzke Beate
- Guerin
- H alu concept
- HABT
- HAEFELI
- HERVE THERMIQUE
- HFOOD PESSAC
- HFOOD VILLENAVE
- Hostellerie
- Hotel Le Moulin
- Hôtel MACCHI
- Hôtel Restaurant Hubert Kieffer
- ICS Scgilthigheim
- IFERRO
- INOUID
- IONITY
- Ispo france
- IXINA Le Cres
- IXINA Saint Jean de Védas
- ixina Villeneuve-lès-Béziers
- Izivia
- JAV INVESTISSEMENT
- jmd
- JONNARD LES ISSAMBRES
- Kallista BDR
- KIEHL
- Kotelon
- LA CONCIERGERIE D'ISA.
- La dilettante
- La jabotte
- La jardinerie d'aveze sarl
- LA PRAIRIE
- Lacotte
- Lacotte Pierre
- LANGON DISTRIBUTION
- Last Mile Solutions
- LE BISTRO D'HUGO
- Le Capelan
- Le Grand Large
- Le Manoir de l'Isle
- le Relais
- LE REVE DE JANINE
- Leclerc Millau
- LEGA
- LEGELEUX
- Leroux Damien
- Les jardins maraichers des bords de rance
- LIDL, Nicolas Barbarin
- LMS Informatique
- LUMI'IN
- M. Yannick PIERRE
- M.A.J.U.
- ma borne auto
- ma Borne Auto
- MABORNEAUTO
- MAS DES OLIVES
- MAS DU TERME
- MAS GALOFFRE
- Masseria
- MEA ENERGIES
- MEDIODENT
- Mickael Auto
- MOBELEC
- Modulo
- MODULO
- Modulo énergies
- MONTA
- MOVIVE_Izivia
- NEXTENEO
- NM SECURELEC
- NON CONCERNE
- Non concerné
- non concerné
- Normatech
- Normatech Lodmi
- NVH
- NW IECharge
- OCR MAINTENANCE ELECTRONIQUE
- PAS DITINERANCE
- Pascal Chene
- Perrin
- perrin
- PHARMACIE DE HUNDLING
- Pilotage Maritime
- PLAGECO DISTRIBUTION
- Polybati
- Prodici
- ProperPhi
- Provibat
- PROVIRIDIS
- Proviridis | FR*PVD
- PTBG et associés
- QoWatt
- R3
- Ramsay Pole Lille métropole
- RechargerMonAuto
- REGIE MUNICIPALE D'ELECTRICITE DE LOOS
- Rencontre-handi
- REVE
- RICOME ET SADOUL AXA FRANCE
- RONALEV
- Royal Champagne
- RSDA mobility
- SA FOOTBALL CLUB DES GIRONDINS DE BORDEAUX
- sa les broyers
- Sanou électricité
- SAP LABS FRANCE
- SARL BEAUDRE BAUDOT
- SARL JUMO
- SARL LAFOURCADE
- SARL LES BAINS DE ROYAN
- SARL VAHE
- SAS CHATEAU DE MEMANAT
- SAS DE L'AILETTE
- sas e-motum
- SAS Lujasy
- SAS Miodis
- SAS Sabo
- SATUJO
- SCI LA COLLINE
- SCI LA GRANGE DESSOUS
- SCI LES RUISSEAUX
- SCI Lounapiou
- SCI NOKI
- SCI OLYMPE
- SCP ACANTHE DRIMARACCI
- SDC CENTRE MEDICAL ARTZAMENDI
- SDC ORDINAL
- SECAL
- Securecharge
- See You Sun
- SELARL PHARMACIE CEVENOLE
- Séolis
- SGA Industries
- SIEGE 27
- SIPECC
- SNAM GROUPE
- SNER RHONE-ALPES
- société La Clérine
- Société LEVAROY, Monsieur LEROY
- Societe moderne d'isolation
- Société Sigma Tec
- Société SIPECC
- Société Y
- Sodetrel
- SOLIDARAUTO 49
- SOREGIES
- SPBR1
- SPIE CITYNETWORKS
- STATIONS-E
- Sud Camargue
- Sud Hotel
- TANAY Electricité
- Technic Elec
- themis
- Thibal Distribution
- TISSERANT
- total énergie
- Total marketing france
- TotalEnergies Charging Services
- TotalEnergies Marketing France
- ubitricity
- UBITRICITY GMBH
- Vegetalis
- Vigot
- Ville de Riquewihr
- Vincent
- Virta
- VIRTA
- Volta Charging
- WAAT
- WAAT SAS
- WattzHub
- web services advenir
- Wedom
- WeDoM
- WeDoM.io
- WeeCharge
- WICKER TP
- ZEBORNE
- ZEborne
- ZEENCO
- Zen Construction

121
docs/irve_puissances.md Normal file
View File

@ -0,0 +1,121 @@
# domaines de données IRVE
## puissance_nominale
quelle pagaille:
- 0.0
- 0
- 1.7
- 2
- 2.3
- 3
- 3.00
- 3.22
- 3.4
- 3.7
- 4
- 4.6
- 5
- 5.5
- 5.9
- 6
- 6.9
- 7
- 7.00
- 7.0
- 7.36
- 7.4
- 9
- 11
- 11.00
- 11.04
- 12
- 12
- 12
- 14
- 15
- 16
- 17
- 18
- 18.00
- 19
- 20
- 21
- 22
- 22.00
- 22.0
- 22
- 22
- 22
- 22.08
- 23
- 24
- 24.00
- 25
- 26
- 27
- 30
- 32
- 36
- 38
- 40
- 42
- 43
- 43.00
- 44
- 45
- 47
- 48
- 50
- 50.00
- 54
- 56
- 60
- 62
- 63
- 64
- 70
- 72
- 75
- 80
- 90
- 100
- 100.00
- 110
- 115
- 120
- 122
- 124
- 125
- 126
- 149
- 150
- 160
- 175
- 175.00
- 180
- 200
- 210
- 225
- 230
- 240
- 300
- 320
- 325
- 350
- 360
- 400
- 3000
- 7000
- 22000
- 22080
- 49521
- 50000
- 60000
- 60973
- 62500
- 63000
- 71708
- 90000
- 131524
- 300000
- 2859660

7
docs/musées.md Normal file
View File

@ -0,0 +1,7 @@
# musées
Jeu de données des musées en île de france.
# conversion
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/arbres/musées/fr.geojson --output-file=musées_idf.geojson --engine-config=MappingMuseums
```

28
docs/parking_vélo.md Normal file
View File

@ -0,0 +1,28 @@
# Parkings vélo et 2 roues
En exemple, l'open data des parkgin 2 roues (vélo et moto) d'Issy les Moulineaux.
Le jeu de données est publié sur la plateforme data de la ville et un convertisseur est proposé ici.
Il pourrait être étendu avec les autres moyens de transport en précisant des tags sur les abris, les trottinnettes, les emplacements de vélo cargo.
Le jeu de données présente environ 350 points géolocalisés avec des informations de situation, de nature, de capacité de parking et d'équipements voisins.
Pour convertir le jeu de données dans un schéma utilisable sur OSM, on écarte les informations de voisinage et on garde les informations de nature, la date de mise en service, le nombre de places dédiées vélo et moto, et l'identifiant donné par Issy dans un référentiel personnel en sous clé de `ref:FR:` inventé à l'occasion.
# procédure
Création d'un mapping, inclusion de la config dans le code de l'Engine afin de pouvoir choisir cette configuration en option de l'appel en ligne de commande à `convert_to_osm_tags.ts`.
Fichier de mapping: `mappings/converters/configIssy_OpenData_2roues.ts`
Fichier de données: `etalab_data/cyclabilité/issy_les_mx_cyclabilité.json`
Commande de conversion:
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/cyclabilité/issy_les_mx_cyclabilité.json --o
utput-file=issy_parking_2roues.json --engine-config=mappingIssy2Roues
```
Résultat: `output/my_converted_data_set__mappingIssy2Roues.json`
Conversion en fichier OSM avec le paquet python pip geojson2osm:
```shell
geojsontoosm output/my_converted_data_set__mappingIssy2Roues.json > osm_output/issy_parkings_2r.osm
```
## Parking cyclables de la métropole de Rouen
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/cyclabilité/rouen_parking_velos.json --output-file=rouen_parking_velos.json --engine-config=mappingRouenParkingVelos
```

8
docs/recyclage.md Normal file
View File

@ -0,0 +1,8 @@
# Recyclage - PAV - points d'apport volontaire
Les boites de tri de déchets public ont leur configuration de conversion.
# Conversion
## Conversion pour les données de Rouen
```bash
ts-node convert_to_osm_tags.ts --source=etalab_data/recycling/rouen_pav.json --output-file=rouen_pav.geojson --engine-config=MappingRouenPAV
```

64
docs/testing.md Normal file
View File

@ -0,0 +1,64 @@
# Tests
Les tests sont lancés avec Jest et sont à écrire dans le dossier "tests", par défaut le runner est en mode auto rechargement des tests à la modification des fichiers.
`npm run test`
Ils fonctionnent avec des jeux de données qui se trouvent dans tests/data et des convertisseurs placés dans mappings/converters.
## Jeu de données
Le json de test disponible par défaut est un Geojson de type FeatureCollection dans testing.json
Les tests consistent à examiner les propriétés de chaque Feature Point et à les convertir selon les règles du Mapper de test.
## Mappers de test
Chaque test a à sa disposition plusieurs Mappers dans `mappings_to_test.ts` qui étendent `MappingConfigType` et peuvent s'appliquer à plusieurs jeux de données pour en tester la sortie.
- mappingPhone
- mappingRemoveAll
- mappingBoolean
- mappingName
- mappingSame
- mappingTruthy
- mappingFalsy
- mappingIgnoreFalsy
- mappingIgnoreTruthy
Leur objectif est de valider la qualité de conversion de données dans de multiples cas, et de vérifier que l'on ajoutera rien qui s'éloigne trop de ce que l'on peut deviner d'une valeur non fiable.
## Exemple de Mapper
Ce mappeur va appliquer la fonction de conversion de numéro de téléphone au format réglementaire pour les valeurs de tag "phone" selon la documentation du wiki OSM. C'est à dire que le numéro doit avoir un préfixe international (+33 pour la France) et avoir des espaces entre groupes de 2 chiffres.
```js
export const mappingPhone: MappingConfigType = {
config_name: 'testing config mappingPhone',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
telephone_operateur: {
key_converted: 'phone',
convert_to_phone: true, // convertit en yes ou no
},
},
add_not_mapped_tags_too: false,
source: {},
filters: {},
}
```
# Écriture de tests
Pour réaliser les tests on importe l'Engine, le jeu de tests, on donne un Mapping à l'Engine, on applique la fonction de mapping, et on compare les propriétés obtenues à ce que l'on attend de la conversion.
```js
import mapping_engine from '../mappings/engine.ts'
import {
mappingPhone,
} from './data/mappings_to_test'
const testingGeoJson = require('./data/testing.json')
describe('mapping properties with rich mapping engine', () => {
test('conditional phone transform', () => {
let Mapping_engine = new mapping_engine(mappingPhone)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
let expected_converted_phone = '+33 1 23 45 67 89'
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
})
})
```

BIN
etalab_data/askangela/lyon_askangela.ods Normal file
View File

Binary file not shown.

29
etalab_data/gynadco/gather_html.py Normal file
View File

@ -0,0 +1,29 @@
import os
import json
# Liste des fichiers JSON à traiter
json_files = [f"out_250_{i}.json" for i in range(1, 16)]
# Ouvre le fichier list.html en écriture
with open("list.html", "w", encoding="utf-8") as f_list:
# Pour chaque fichier JSON
for json_file in json_files:
# Vérifie si le fichier existe
if os.path.exists(json_file):
# Ouvre le fichier JSON en lecture
with open(json_file, "r", encoding="utf-8") as f_json:
# Charge le contenu JSON
data = json.load(f_json)
# Récupère la propriété "html"
html_content = data["html"]
# Écrit le contenu HTML dans le fichier list.html
f_list.write(html_content)
else:
print(f"Le fichier {json_file} n'existe pas.")

1
etalab_data/gynadco/get.sh Normal file
View File

@ -0,0 +1 @@
curl 'https://gynandco.wordpress.com/?infinity=scrolling' --compressed -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:131.0) Gecko/20100101 Firefox/131.0' -H 'Accept: */*' -H 'Accept-Language: fr,en-US;q=0.7,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'X-Requested-With: XMLHttpRequest' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Origin: https://gynandco.wordpress.com' -H 'Sec-GPC: 1' -H 'Alt-Used: gynandco.wordpress.com' -H 'Connection: keep-alive' -H 'Referer: https://gynandco.wordpress.com/trouver-un-e-soignant-e/liste-des-soignantes/page/1/' -H 'Cookie: ccpa_applies=true; usprivacy=1YNN; euconsent-v2=CQG1_8AQG1_8AECACAFRBMEgAAAAAELAAKiQAAAMnABAECoAAKAAICCAAAIAAAAgCAAgAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgACAAAIAAAAAAAAAAAAEAAAA' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -H 'TE: trailers' --data-raw 'action=infinite_scroll&page=10&currentday=17.09.24&order=DESC&scripts[0]=jetpack-mu-wpcom-settings&scripts[1]=rlt-proxy&scripts[2]=media-video-jwt-bridge&scripts[3]=cmp-script-stub&scripts[4]=wpcom-actionbar-placeholder&scripts[5]=grofiles-cards&scripts[6]=wpgroho&scripts[7]=mobile-useragent-info&scripts[8]=wordads-smart&scripts[9]=the-neverending-homepage&scripts[10]=cmp-script&scripts[11]=jetpack-carousel&scripts[12]=tiled-gallery&scripts[13]=carousel-wpcom&styles[0]=wordads-global&styles[1]=free-site-marketing-bar&styles[2]=the-neverending-homepage&styles[3]=wp-block-library&styles[4]=wp-block-library-theme&styles[5]=jetpack-layout-grid&styles[6]=jetpack-ratings&styles[7]=mediaelement&styles[8]=wp-mediaelement&styles[9]=coblocks-frontend&styles[10]=wpcom-core-compat-playlist-styles&styles[11]=wpcom-text-widget-styles&styles[12]=wpcom-bbpress2-staff-css&styles[13]=twentyeleven-block-style&styles[14]=twentyeleven-wpcom-style&styles[15]=reblogging&styles[16]=geo-location-flair&styles[17]=infinity-twentyeleven&styles[18]=h4-global&styles[19]=wp-emoji-styles&styles[20]=videopress-video-style&styles[21]=jetpack-sharing-buttons-style&styles[22]=classic-theme-styles&styles[23]=global-styles&styles[24]=jetpack-global-styles-frontend-style&styles[25]=jetpack-carousel-swiper-css&styles[26]=jetpack-carousel&styles[27]=tiled-gallery&styles[28]=core-block-supports-duotone&query_args[page]=0&query_args[pagename]=trouver-un-e-soignant-e%2Fliste-des-soignantes&query_args[error]=&query_args[m]=&query_args[p]=0&query_args[post_parent]=&query_args[subpost]=&query_args[subpost_id]=&query_args[attachment]=&query_args[attachment_id]=0&query_args[name]=&query_args[page_id]=0&query_args[second]=&query_args[minute]=&query_args[hour]=&query_args[day]=0&query_args[monthnum]=0&query_args[year]=0&query_args[w]=0&query_args[category_name]=&query_args[tag]=&query_args[cat]=&query_args[tag_id]=&query_args[author]=&query_args[author_name]=&query_args[feed]=&query_args[tb]=&query_args[paged]=0&query_args[meta_key]=&query_args[meta_value]=&query_args[preview]=&query_args[s]=&query_args[sentence]=&query_args[title]=&query_args[fields]=&query_args[menu_order]=&query_args[embed]=&query_args[category__in][]=&query_args[category__not_in][]=&query_args[category__and][]=&query_args[post__in][]=&query_args[post__not_in][]=&query_args[post_name__in][]=&query_args[tag__in][]=&query_args[tag__not_in][]=&query_args[tag__and][]=&query_args[tag_slug__in][]=&query_args[tag_slug__and][]=&query_args[post_parent__in][]=&query_args[post_parent__not_in][]=&query_args[author__in][]=&query_args[author__not_in][]=&query_args[search_columns][]=&query_args[lazy_load_term_meta]=false&query_args[posts_per_page]=250&query_args[ignore_sticky_posts]=false&query_args[suppress_filters]=false&query_args[cache_results]=true&query_args[update_post_term_cache]=true&query_args[update_menu_item_cache]=false&query_args[update_post_meta_cache]=true&query_args[post_type]=&query_args[nopaging]=false&query_args[comments_per_page]=50&query_args[no_found_rows]=false&query_args[order]=DESC&query_before=2024-10-21%2022%3A57%3A29&last_post_date=2024-10-15%2018%3A25%3A29'

90597
etalab_data/gynadco/list.html Normal file
View File

File diff suppressed because it is too large Load Diff

231
etalab_data/gynadco/scrap.py Normal file
View File

@ -0,0 +1,231 @@
import json
import geopandas as gpd
from bs4 import BeautifulSoup
from shapely.geometry import Point
import re
import pandas as pd
def extraire_numero_telephone(line):
# Extraction des nombres dans la ligne
numbers = re.findall(r'\d+', line)
if numbers:
numbers = ''.join(numbers)
# print('numbers',numbers)
# Vérification si un numéro de téléphone est présent
if len(numbers) == 10:
# Reconstruction du numéro de téléphone
phone_number = ''.join(numbers)
return phone_number
else:
return None
def extraire_code_postal(line):
# Extraction du code postal dans la ligne
match = re.search(r'\b(\d{5})\b', line)
if match:
# Extraction des nombres correspondant au code postal
code_postal = match.group(1)
return code_postal
else:
return None
# trouver si la ligne est une adresse en cherchant deux numéros distincts et une virgule
def extraire_addr_line(line):
# Extraction des nombres dans la ligne
numbers = re.findall(r'\d+', line)
# Vérification si un numéro de téléphone est présent
if len(numbers) == 2 and ',' in line and len(numbers[1]) == 5:
return line
else:
return None
# Charger le fichier HTML
with open("list.html", "r") as file:
html = file.read()
# Analyser le code HTML avec BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
# Initialiser une liste pour stocker les informations des docteurs
doctors = []
# Trouver toutes les balises <article> sur la page
articles = soup.find_all("article")
# Parcourir chaque <article> pour extraire les informations des docteurs
for article in articles:
# Récupérer le nom du docteur à partir de la balise <h1>
# print(article.find("h1"))
name = article.find("h1").find('a').text.strip()
url = article.find("h1").find('a').get("href")
# Récupérer l'adresse du docteur à partir de la balise <em> dans la classe "entry-content"
address = ''
em = article.find("em")
if em:
address = em.text
# print(address)
# Vérifier si le contenu de l'article contient "Secteur 1"
if "Secteur 1" in article.text:
sector = "1"
elif "Secteur 2" in article.text:
sector = "2"
else:
sector = None
# Recherche d'un numéro de téléphone dans l'article
phone_number = None
gender = 'unknown'
trans_friendly = ''
handles_violence = ''
visio_meeting = 'no'
pseudo_science = ''
premenstrual_syndrome = ''
accessible_cabinet = ''
tatoo = ''
toxico = ''
sterilisation = ''
abortion = ''
endometriosis = ''
bigbody = ''
poil = ''
bi = ''
lesbian = ''
pma = ''
ist = ''
pregnancy = ''
ivg = ''
generaliste = ''
gyneco = ''
sage_femme = ''
auto_prelev = ''
mycoses = ''
dyspareunie = ''
spoken = '' # langues parlées
diu = '' # dispositif intra utérin
puma= '' # PUMA (ex-CMU)
ame= '' # AME (Aide médicale d’état)
code_postal = ''
for line in article.stripped_strings:
found = extraire_code_postal(line)
if found:
code_postal = found
address = line
found = extraire_numero_telephone(line)
if found:
# print(found)
phone_number = found
if 'Rdv en ligne possible' in line:
visio_meeting = 'yes'
if 'femme soignante' in line:
gender = 'women'
if 'homme soignante' in line:
gender = 'women'
if 'Trans friendly' in line:
trans_friendly = 'yes'
if 'Tattoo' in line:
tatoo = 'yes'
if 'Sensibilité violence' in line:
handles_violence = 'yes'
if 'naturelles/alternatives' in line:
pseudo_science = 'yes'
if 'SPM' in line:
premenstrual_syndrome = 'yes'
if 'Poilfriendly' in line:
poil = 'yes'
if 'Bifriendly' in line:
bi = 'yes'
if 'Stérilisation' in line:
sterilisation = 'yes'
if 'DIU' in line:
diu = 'yes'
if 'Accompagnement grossesse' in line:
pregnancy = 'yes'
if 'suivi des IST' in line:
ist = 'yes'
if 'IVG' in line:
ivg = 'yes'
if 'Médecin généraliste' in line:
generaliste = 'yes'
if 'Sage-femme' in line:
sage_femme = 'yes'
if 'auto prélèvement' in line:
auto_prelev = 'yes'
if 'Conseils mycoses' in line:
mycoses = 'yes'
if 'Lesbiennes friendly' in line:
lesbian = 'yes'
if 'PMA' in line:
pma = 'yes'
if 'PUMA' in line:
puma = 'yes'
if 'AME' in line:
ame = 'yes'
if 'français, anglais' in line:
spoken = 'french;english'
if 'espagnol' in line:
spoken = ('french;english;spanish')
if 'Dyspareunie' in line:
dyspareunie = 'yes'
if 'Gynécologue' in line:
gyneco = 'yes'
# chercher une adresse si on en a pas
if not address:
found = extraire_addr_line(line)
if found:
# print(found)
address = found
# Ajouter les informations du docteur à la liste
doctor = {
"name": name,
"address": address,
"address:code_postal": code_postal,
"ref:FR:convention_secteur": sector,
"contact:phone": phone_number,
"contact:website": url,
"gender": gender,
"visioconference_meeting": visio_meeting,
"handles:gender:trans": trans_friendly,
"handles:violence": handles_violence,
"handles:premenstrual_syndrome": premenstrual_syndrome,
"handles:IST": ist,
"accessible_cabinet": accessible_cabinet,
"pseudo_science": pseudo_science,
"speaks": spoken,
"handles:tatoo": tatoo,
"handles:toxico": toxico,
"handles:sterilisation": sterilisation,
"handles:abortion": abortion,
"handles:endometriosis": endometriosis,
"handles:premenstrual_syndrome": premenstrual_syndrome,
"handles:hairy": poil,
"handles:bigbody": bigbody,
"handles:gender:bi": bi,
"handles:diu": diu,
"handles:mycoses": mycoses,
"handles:pregnancy": pregnancy,
"handles:abortion": ivg,
"handles:auto_prelevement": auto_prelev,
"handles:pma": pma,
"handles:dyspareunie": dyspareunie,
"handles:gender:lesbian": lesbian,
"healcare:generaliste": generaliste,
"healcare:sage_femme": sage_femme,
"healcare:gynecologist": gyneco,
}
doctors.append(doctor)
# Enregistrer les informations des docteurs au format JSON dans un fichier
with open("gynandco.json", "w", encoding="utf-8") as f:
json.dump(doctors, f, ensure_ascii=False, indent=2)
with open('gynandco.json', 'r') as f:
data = json.load(f)
df = pd.DataFrame(data)
df.to_csv('gynandco.csv', index=False)

1
etalab_data/irve_bornes_recharge/puissances_repartition.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

2331
etalab_data/planing_familial/liste.html Normal file
View File

File diff suppressed because it is too large Load Diff

63
etalab_data/planing_familial/scrap.py Normal file
View File

@ -0,0 +1,63 @@
import json
import geopandas as gpd
from bs4 import BeautifulSoup
from shapely.geometry import Point
# Charger le fichier HTML
with open("liste.html", "r") as file:
html = file.read()
# Analyser le code HTML avec BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
# Trouver toutes les balises <article> qui contiennent les informations des antennes
articles = soup.find_all("article", class_="node node--type-hp-antenne node--view-mode-map-result")
# Liste pour stocker les données de chaque antenne
antennes = []
# Parcourir chaque article et extraire les informations
for article in articles:
titre = article.find("button", class_="nsb action-title").text.strip()
adresse = article.find("p", class_="address").text.strip()
website = article.find("a", class_="icon arrow").get("href").strip()
violences = article.find("li", class_="icon violences")
sexualities = article.find("li", class_="icon sexualities")
detection = article.find("li", class_="icon detection")
contraception = article.find("li", class_="icon contraception")
abortion = article.find("li", class_="icon abortion")
# Gérer les articles qui n'ont pas de numéro de téléphone
telephone = ""
try:
telephone = article.find("p", class_="tel").text.strip()
except AttributeError:
pass
latitude = float(article["data-latlng"].split(",")[0].replace("[", "").replace('"', ""))
longitude = float(article["data-latlng"].split(",")[1].replace("]", "").replace('"', ""))
geometry = Point(longitude, latitude)
antenne = {
"type": "Feature",
"geometry":geometry,
"nom": titre,
"adresse": adresse,
"contact:phone": telephone,
"contact:website": 'https://www.planning-familial.org'+website,
"family_planning:handles:violences": ('yes' if violences else 'no'),
"family_planning:handles:sexualities": ('yes' if sexualities else 'no'),
"family_planning:handles:detection": ('yes' if detection else 'no'),
"family_planning:handles:abortion": ('yes' if abortion else 'no'),
"family_planning:handles:contraception": ('yes' if contraception else 'no'),
}
# print(antenne)
antennes.append(antenne)
# Convertir la liste des antennes en un GeoDataFrame
gdf = gpd.GeoDataFrame(antennes)
# Exporter le GeoDataFrame au format GeoJSON
gdf.to_file("antennes_planning_familial.json", driver="GeoJSON")

0
etalab_data/toilettes/.gitkeep Normal file
View File

199
jest.config.ts Normal file
View File

@ -0,0 +1,199 @@
/**
* For a detailed explanation regarding each configuration property, visit:
* https://jestjs.io/docs/configuration
*/
import type {Config} from 'jest';
const config: Config = {
// All imported modules in your tests should be mocked automatically
// automock: false,
// Stop running tests after `n` failures
// bail: 0,
// The directory where Jest should store its cached dependency information
// cacheDirectory: "/tmp/jest_rs",
// Automatically clear mock calls, instances, contexts and results before every test
clearMocks: true,
// Indicates whether the coverage information should be collected while executing the test
collectCoverage: true,
// An array of glob patterns indicating a set of files for which coverage information should be collected
// collectCoverageFrom: undefined,
// The directory where Jest should output its coverage files
coverageDirectory: "coverage",
// An array of regexp pattern strings used to skip coverage collection
// coveragePathIgnorePatterns: [
// "/node_modules/"
// ],
// Indicates which provider should be used to instrument code for coverage
coverageProvider: "v8",
// A list of reporter names that Jest uses when writing coverage reports
// coverageReporters: [
// "json",
// "text",
// "lcov",
// "clover"
// ],
// An object that configures minimum threshold enforcement for coverage results
// coverageThreshold: undefined,
// A path to a custom dependency extractor
// dependencyExtractor: undefined,
// Make calling deprecated APIs throw helpful error messages
// errorOnDeprecated: false,
// The default configuration for fake timers
// fakeTimers: {
// "enableGlobally": false
// },
// Force coverage collection from ignored files using an array of glob patterns
// forceCoverageMatch: [],
// A path to a module which exports an async function that is triggered once before all test suites
// globalSetup: undefined,
// A path to a module which exports an async function that is triggered once after all test suites
// globalTeardown: undefined,
// A set of global variables that need to be available in all test environments
// globals: {},
// The maximum amount of workers used to run your tests. Can be specified as % or a number. E.g. maxWorkers: 10% will use 10% of your CPU amount + 1 as the maximum worker number. maxWorkers: 2 will use a maximum of 2 workers.
// maxWorkers: "50%",
// An array of directory names to be searched recursively up from the requiring module's location
// moduleDirectories: [
// "node_modules"
// ],
// An array of file extensions your modules use
// moduleFileExtensions: [
// "js",
// "mjs",
// "cjs",
// "jsx",
// "ts",
// "tsx",
// "json",
// "node"
// ],
// A map from regular expressions to module names or to arrays of module names that allow to stub out resources with a single module
// moduleNameMapper: {},
// An array of regexp pattern strings, matched against all module paths before considered 'visible' to the module loader
// modulePathIgnorePatterns: [],
// Activates notifications for test results
// notify: false,
// An enum that specifies notification mode. Requires { notify: true }
// notifyMode: "failure-change",
// A preset that is used as a base for Jest's configuration
// preset: undefined,
// Run tests from one or more projects
// projects: undefined,
// Use this configuration option to add custom reporters to Jest
// reporters: undefined,
// Automatically reset mock state before every test
// resetMocks: false,
// Reset the module registry before running each individual test
// resetModules: false,
// A path to a custom resolver
// resolver: undefined,
// Automatically restore mock state and implementation before every test
// restoreMocks: false,
// The root directory that Jest should scan for tests and modules within
// rootDir: undefined,
// A list of paths to directories that Jest should use to search for files in
// roots: [
// "<rootDir>"
// ],
// Allows you to use a custom runner instead of Jest's default test runner
// runner: "jest-runner",
// The paths to modules that run some code to configure or set up the testing environment before each test
// setupFiles: [],
// A list of paths to modules that run some code to configure or set up the testing framework before each test
// setupFilesAfterEnv: [],
// The number of seconds after which a test is considered as slow and reported as such in the results.
// slowTestThreshold: 5,
// A list of paths to snapshot serializer modules Jest should use for snapshot testing
// snapshotSerializers: [],
// The test environment that will be used for testing
// testEnvironment: "jest-environment-node",
// Options that will be passed to the testEnvironment
// testEnvironmentOptions: {},
// Adds a location field to test results
// testLocationInResults: false,
// The glob patterns Jest uses to detect test files
// testMatch: [
// "**/__tests__/**/*.[jt]s?(x)",
// "**/?(*.)+(spec|test).[tj]s?(x)"
// ],
// An array of regexp pattern strings that are matched against all test paths, matched tests are skipped
// testPathIgnorePatterns: [
// "/node_modules/"
// ],
// The regexp pattern or array of patterns that Jest uses to detect test files
// testRegex: [],
// This option allows the use of a custom results processor
// testResultsProcessor: undefined,
// This option allows use of a custom test runner
// testRunner: "jest-circus/runner",
// A map from regular expressions to paths to transformers
// transform: undefined,
// An array of regexp pattern strings that are matched against all source file paths, matched files will skip transformation
// transformIgnorePatterns: [
// "/node_modules/",
// "\\.pnp\\.[^\\/]+$"
// ],
// An array of regexp pattern strings that are matched against all modules before the module loader will automatically return a mock for them
// unmockedModulePathPatterns: undefined,
// Indicates whether each individual test should be reported during the run
// verbose: undefined,
// An array of regexp patterns that are matched against all source file paths before re-running tests in watch mode
// watchPathIgnorePatterns: [],
// Whether to use watchman for file crawling
// watchman: true,
};
export default config;

193
make_variance_from_csv.ts Normal file
View File

@ -0,0 +1,193 @@
/**
prendre un CSV,
examiner toutes les colonnes et leurs valeurs,
garder en mémoire les valeurs uniques de chaque colonne
faire un nouveau csv qui ne montre que les valeurs uniques pour chacune des colonnes
et qui compte le nombre de valeurs
**/
import utils from './mappings/utils'
import {parse} from 'csv'
const fs = require('fs')
const minimist = require('minimist')
let mini_arguments: any = minimist(process.argv.slice(2))
// interface VarianceType {
// [key: string]: Array<string>
// }
let csv_content = 'variance de dataset\n';
let separator = '\t';
let separator_fallback = ',';
let data_variance: any = {};
let folder = 'irve_bornes_recharge'
let inputFile = 'latest.csv'
let fileNameOutput = `variance_${inputFile}`;
let inputPath = `./etalab_data/${folder}/${inputFile}`
// let inputPath = './etalab_data/toilettes/small_datas.csv'
let outputPath = `etalab_data/${folder}`
console.log('mini_arguments', mini_arguments)
if (mini_arguments['inputFile']) {
inputFile = `${mini_arguments['input-file']}`
}
if (mini_arguments['source']) {
inputPath = mini_arguments['source']
}
if (mini_arguments['separator']) {
separator = mini_arguments['separator']
}
let columns_headings: Array<string> = [];
let lines_count = 0;
let longest_variance_count = 0;
function getColumnsFromRow(row: string) {
let headings: any = []
console.log('elem', row)
headings = row.split(separator)
return headings
}
/**
* sort unique values ascending in each column
*/
function reorderValuesInDataVariance() {
columns_headings.forEach((heading: string) => {
data_variance[heading] = data_variance[heading].sort((a: any, b: any) => a - b)
})
}
console.log('read file ', inputPath)
fs.readFile(inputPath, function (err: any, fileData: any) {
if (err) {
throw new Error(err)
} else {
parse(fileData, {columns: false, trim: true}, function (err: any, lines: any) {
// Your CSV data is in an array of arrays passed to this callback as rows.
if (err) {
throw new Error(err)
}
console.log('line ', lines_count)
console.log('lines.length', lines.length)
lines.forEach((line: any) => {
line = line[0]
if (lines_count === 0) {
console.log('line', line)
if (line.indexOf(separator) === -1) {
console.log('separator not found: ', separator, 'trying other separator')
if (line.indexOf(separator_fallback) === -1) {
console.log('separator not found: ', separator_fallback)
throw new Error('no separator found in csv')
} else {
console.log('fallback separator found: ', separator_fallback, 'now using it ')
separator = separator_fallback
}
}
columns_headings = getColumnsFromRow(line)
console.log('columns_headings.length', columns_headings.length)
// console.log('columns_headings', columns_headings)
let headers = Object.keys(columns_headings)
columns_headings.forEach((header: string) => {
data_variance[header] = []
})
console.log('data_variance', data_variance)
} else {
// lignes suivantes
let column_index = 0
line.split(separator).forEach((value: string) => {
value = value.trim()
let column_header_current = columns_headings[column_index]
// console.log('column_index', column_index)
// dans chaque colonne, vérifier que la valeur n'est pas déjà présente
// dans les index de variance
// si la valeur est nouvelle, l'ajouter
if (data_variance[column_header_current].indexOf(value) === -1) {
data_variance[column_header_current].push(value)
if (
data_variance[column_header_current].length > longest_variance_count
) {
longest_variance_count = data_variance[column_header_current].length
}
} else {
console.log('value', value, ' déjà présente dans la collection', column_header_current)
}
column_index++
})
}
lines_count++
})
console.log('lines_count', lines_count)
console.log('longest_variance_count', longest_variance_count)
reorderValuesInDataVariance()
utils.writeFile(fileNameOutput, writeCSVVariance(), outputPath)
})
}
console.log('parsing done')
// console.log('data_variance', data_variance)
})
/**
* écrit un csv avec les données de variance du dataset donné
*/
function writeCSVVariance() {
let csv_content = `;variance de ${inputPath};généré le:;${new Date()};lignes du csv original:;${lines_count};fait avec make_variance_from_csv.ts de Tykayn
`
let columns = Object.keys(data_variance);
// add headings
columns_headings.forEach((heading: string) => {
csv_content = csv_content + separator + heading
})
csv_content = csv_content + '\n'
// add max length of variance for each column
let ii = 0
columns.forEach((column: string) => {
// console.log('column', column, data_variance[column].length)
csv_content = csv_content + separator + data_variance[column].length
ii++
})
csv_content = csv_content + '\n\n'
// add content of values
for (let ii = 0; ii < longest_variance_count; ii++) {
csv_content = csv_content + '\n'
columns.forEach((column: any) => {
if (ii < data_variance[column].length) {
let currentValue = data_variance[column][ii]
csv_content = csv_content + separator + currentValue
} else {
csv_content = csv_content + separator
}
})
}
return csv_content;
}

16
makefile Normal file
View File

@ -0,0 +1,16 @@
default: irve
get_files:
bash ./update_scripts/get_datasets.sh
echo "fichiers récupérés"
irve:
bash ./update_scripts/convert_irve.sh
echo " DONE "
echo " "
echo "you can open the IRVE converted osm file in JOSM : osm_output/bornes-irve-filetered_latest.osm "
echo "you can open the existing IRVE referenced in OSM : osm_output/bornes-irve-filetered_latest.osm "
osmose_irve:
pnpm i
ts-node convert_to_osm_tags.ts --osmose=true --engine=true --source="etalab_data/irve_bornes_recharge/osmose-item-irve-8411-intégrables.json" --outname="_irve-latest-osmose"
#geojson2osm "etalab_data/irve_bornes_recharge/osmose-item-irve-8411-intégrables.json" > "osm_output/osmose-item-irve-8411-intégrables.osm"
geojson2osm "output/my_converted_data_set__irve-latest-osmose.json" > "osm_output/my_converted_data_set__irve-latest-osmose.osm"

116
mappings/converters/configArbresEchirolles.ts Normal file
View File

@ -0,0 +1,116 @@
import MappingConfigType from "../mapping-config.type";
/**
* conversion des données concernant les arbres de la ville d'Echirolles
* lancement du script:
*
* ts-node convert_to_osm_tags.ts --source="etalab_data/abres/arbres_ech.json" --output-file="arbres-echirolles.geojson" --engine-config=MappingArbresEchirolles
*
* documentation sur les espèces d'arbres
* nom : item wikidata
* "Platanus acerifolia" Q24853030
* "Tilia cordata" Q158746
* "Liriodendron tulipifera" Q158783
*/
const MappingArbresEchirolles: MappingConfigType = {
// add_not_mapped_tags_too: false,
// source: undefined,
config_name: 'Mapping des arbres d\'Echirolles',
config_author: 'tykayn <contact+geojson2osm@cipherbliss.com>',
default_properties_of_point: {natural: 'tree', source : 'Échirolles Métropole'},
tags: {
// ******* booléens
// ******* nombres
// - ID_ARBRE : ref:FR:issy_les_mx:id_arbre identifiant interne à Issy les Moulineaux
"id_arbre": "ref:FR:Échirolles:id_arbre",
// ******* textes
"nom_latin": {
key_converted: "species",
conditional_values: {
"Platanus acerifolia": {'tags_to_add': {"species:wikidata": "Q24853030"}},
"Tilia cordata": {'tags_to_add': {"wikidata": "species:Q158746"}},
"Liriodendron tulipifera": {'tags_to_add': {"species:wikidata": "Q158783"}},
},
},
// - CADUC_PERS : leaf_cycle=evergreen pour persistant , deciduous pour caduque
// "caduc_pers": {
// conditional_values: {
// "Persistant ": {
// 'tags_to_add':{
// "leaf_cycle": "evergreen",
// }
// },
// "Caduc ": {
// 'tags_to_add':{
// "leaf_cycle": "deciduous",
// }
// },
// }
// },
// - FEUIL_CONI : feuillu leaf_type=broadleaved / connifère leaf_type=needleleaved
// "feuil_coni": {
// conditional_values: {
// "Feuillu ": {
// 'tags_to_add':{
// "leaf_type": "broadleaved",
// }
// },
// "Conifère ": {
// 'tags_to_add':{
// "leaf_type": "needleleaved",
// }
// },
// }
// },
// - PARTICULAR : Majeur, Remarquable : historic=monument
// "particular": {
// conditional_values: {
// "Majeur ": {
// 'tags_to_add':{
// "historic": "monument",
// },
// },
// "Remarquable ": {
//
// 'tags_to_add':{
// "historic": "monument",
// },
// },
// }
// },
// - FORME: tree_shape = curtain / free / half_free <= Architecturé, rideau / Libre / Semi-libre
// "forme": {
// key_converted: "tree_shape",
// conditional_values: {
// "Architecturé, rideau ": {
// 'tags_to_add':{
// "tree_shape": "curtain",
// },
// },
// "Semi-libre ": {
// 'tags_to_add':{
// "tree_shape": "half_free",
// },
// },
// "Libre ": {
// 'tags_to_add':{
// "tree_shape": "free",
// },
// },
// }
// },
},
add_not_mapped_tags_too: false,
source: {
geojson_path: 'https://data.issy.com/api/explore/v2.1/catalog/datasets/arbres-remarquables-issy-les-moulineaux/exports/geojson?lang=fr&timezone=Europe%2FBerlin',
url: 'https://www.data.gouv.fr/fr/datasets/arbres-remarquables'
},
filters: {
// offset:10
},
}
export default MappingArbresEchirolles

51
mappings/converters/configArbresHautsDeSeine.ts Normal file
View File

@ -0,0 +1,51 @@
import MappingConfigType from "../mapping-config.type";
const MappingArbres92: MappingConfigType = {
config_name: 'Mapping des arbres d\'Issy les Moulineaux',
config_author: 'tykayn <contact+geojson2osm@cipherbliss.com>',
default_properties_of_point: {natural: 'tree'},
tags: {
// ******* booléens
// ******* nombres
// - ID_ARBRE : ref:FR:issy_les_mx:id_arbre identifiant interne à Issy les Moulineaux
"idelement_vert": "ref:FR:cadastre_vert:id_element_vert",
"type": "ref:FR:cadastre_vert:type",
// ******* textes
// ******* textes
// - HAUTEUR est_height=* hauteur estimée en intervalle de mètres
//"class_hauteur": "est_height",
// - VERNACULAIRE : species:FR=*
//"vernaculaire": "species:FR",
// https://wiki.openstreetmap.org/wiki/Key:circumference
"diametre": "circumference",// tour du tronc à hauteur de poitrine
// "diametre": {// tour du tronc à hauteur de poitrine
// key_converted: "circumference",
// conditional_values: {
// "0 m": {
// ignore_this_data: true,
// }
// }
// },
"circonference": "diameter_crown",// tour de toutes les branches, la couronne
// "circonference": {// tour de toutes les branches, la couronne
// key_converted: "diameter_crown",
// conditional_values: {
// "0 m": {
// ignore_this_data: true,
// }
// }
//
// },
},
add_not_mapped_tags_too: false,
source: {
geojson_path: 'https://opendata.hauts-de-seine.fr/api/explore/v2.1/catalog/datasets/cadastre-vert-les-arbres/exports/geojson?lang=fr&timezone=Europe%2FBerlin',
url: 'https://opendata.hauts-de-seine.fr/explore/dataset/cadastre-vert-les-arbres'
},
filters: {},
}
export default MappingArbres92

103
mappings/converters/configArbresIssy.ts Normal file
View File

@ -0,0 +1,103 @@
import MappingConfigType from "../mapping-config.type";
const MappingArbresIssy: MappingConfigType = {
config_name: 'Mapping des arbres d\'Issy les Moulineaux',
config_author: 'tykayn <contact+geojson2osm@cipherbliss.com>',
default_properties_of_point: {natural: 'tree'},
tags: {
// ******* booléens
// ******* nombres
// - ID_ARBRE : ref:FR:issy_les_mx:id_arbre identifiant interne à Issy les Moulineaux
"id_arbre": "ref:FR:issy_les_mx:id_arbre",
"ordre": "ref:FR:issy_les_mx:ordre_arbre",
// - AN_PLANT : date de plantation
"an_plan": "date_start",
// ******* textes
// - HAUTEUR est_height=* hauteur estimée en intervalle de mètres
"class_hauteur": "est_height",
// - VERNACULAIRE : species:FR=*
"vernaculaire": "species:FR",
// - GENESPVAR : nom d'espèce species=*
"genespvar": "species",
// - CADUC_PERS : leaf_cycle=evergreen pour persistant , deciduous pour caduque
"caduc_pers": {
conditional_values: {
"Persistant ": {
'tags_to_add':{
"leaf_cycle": "evergreen",
}
},
"Caduc ": {
'tags_to_add':{
"leaf_cycle": "deciduous",
}
},
}
},
// - FEUIL_CONI : feuillu leaf_type=broadleaved / connifère leaf_type=needleleaved
"feuil_coni": {
conditional_values: {
"Feuillu ": {
'tags_to_add':{
"leaf_type": "broadleaved",
}
},
"Conifère ": {
'tags_to_add':{
"leaf_type": "needleleaved",
}
},
}
},
// - PARTICULAR : Majeur, Remarquable : historic=monument
"particular": {
conditional_values: {
"Majeur ": {
'tags_to_add':{
"historic": "monument",
},
},
"Remarquable ": {
'tags_to_add':{
"historic": "monument",
},
},
}
},
// - FORME: tree_shape = curtain / free / half_free <= Architecturé, rideau / Libre / Semi-libre
"forme": {
key_converted: "tree_shape",
conditional_values: {
"Architecturé, rideau ": {
'tags_to_add':{
"tree_shape": "curtain",
},
},
"Semi-libre ": {
'tags_to_add':{
"tree_shape": "half_free",
},
},
"Libre ": {
'tags_to_add':{
"tree_shape": "free",
},
},
}
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: 'https://data.issy.com/api/explore/v2.1/catalog/datasets/arbres-remarquables-issy-les-moulineaux/exports/geojson?lang=fr&timezone=Europe%2FBerlin',
url: 'https://www.data.gouv.fr/fr/datasets/arbres-remarquables'
},
filters: {
offset:10
},
}
export default MappingArbresIssy

67
mappings/converters/configAskAngela.ts Normal file
View File

@ -0,0 +1,67 @@
/**
* commerces adhérant à Ask Angela
* https://wiki.openstreetmap.org/wiki/FR:Tag:harassment_prevention%3Dask_angela
*/
import MappingConfigType from "../mapping-config.type";
const MappingAskAngela: MappingConfigType = {
config_name: "mappingRouenParkingVelos",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'harassment_prevention': 'ask_angela',
},
source: {
geojson_path: '',
url: ''
},
filters: {
// exclude_point_if_tag_not_empty: ['id_osm'], // on peut exclure des données converties celles qui sont déjà avec un identifiant openstreetmap afin de favoriser l'intégration san avoir à gérer les doublons
// offset: 1
},
add_not_mapped_tags_too: false,
boolean_keys: [],
tags_to_ignore_if_value_is: ['Non renseigne'],
tags: {
// ******* opendata de toulouse START **************
structure: {
remove_original_key: true,
conditional_values: {
"Hôtel": {
tags_to_add: {
'tourism': 'hotel'
}
},
"Restaurant": {
tags_to_add: {
'amenity': 'restaurant'
}
},
}
},
'id': 'ref:FR:Toulouse:ask_angela_id',
noms: {
key_converted: 'name',
convert_to_name: true,
},
nom : {
key_converted: 'name',
convert_to_name: true,
},
// ******* opendata de toulouse END **************
// ******* opendata de poitiers START **************
'_id': 'ref:FR:Poitiers:ask_angela_id',
// ******* opendata de poitiers END **************
// ******* opendata de bordeaux START **************
'gid': 'ref:FR:Bordeaux:ask_angela_id',
'libelle': 'name',
'url': 'contact:website',
'cdate': 'start_date:ask_angela',
'mdate': 'check_date:ask_angela',
// ******* opendata de bordeaux END **************
// "addresse": "",
}
}
export default MappingAskAngela;

154
mappings/converters/configFINESS.ts Normal file
View File

@ -0,0 +1,154 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const mappingFINESS: MappingConfigType = {
config_name: "mappingFINESS",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {},
source: {
geojson_path: "finess/finess_idf.json",
url: 'https://www.data.gouv.fr/fr/datasets/r/b9731c6f-c0d7-422e-9e1c-19edd51687ce'
},
filters: {
// exclude_point_if_tag_not_empty: ['id_osm'],
// offset: 1
},
add_not_mapped_tags_too: false,
boolean_keys: [],
tags_to_ignore_if_value_is: [],
tags: {
// ******* nombres
// capacite: 'capacity',
// ******* textes
siret: 'ref:FR:SIRET',
rslongue: {
key_converted: 'name',
convert_to_name: true,
},
nofinesset: 'ref:FR:FINESS:et',
nofinessej: 'ref:FR:FINESS:ej',
codemft: 'ref:FR:code_mft',
datemaj_structure: 'date:structure:update',
datemaj_geo: 'date:geolocalisation:update',
// categetab: 'ref:FR:categtab',
epci_code: 'ref:FR:code_EPCI',
epci_name: 'ref:FR:EPCI',
categagretab: 'ref:FR:FINESS:category_agreg',
codeape: 'ref:FR:NAF', // https://wiki.openstreetmap.org/wiki/FR:Key:ref:FR:NAF
// categtab: 'type:FR:FINESS',
dateouv: 'start_date',
dateautor: 'authorisation_date',
telephone: {
key_converted: 'operator:phone',
convert_to_phone: true,
},
telecopie: {
key_converted: 'operator:fax',
convert_to_phone: true,
},
categetab: {
remove_original_key: true,
key_converted: 'type:FR:FINESS',
conditional_values: {
// doc https://wiki.openstreetmap.org/wiki/FR:Key:type:FR:FINESS
// Centre Hospitalier Régional
101: {
tags_to_add: {
"amenity": "hospital",
"emergency": "yes",
"healthcare:speciality": "psychiatry",
}
},
// Centre Hospitalier ex Hôpital local
106: {
tags_to_add: {
"amenity": "hospital",
"emergency": "yes",
"healthcare:speciality": "psychiatry",
}
},
// Soins de Suite et Réadaptation
109: {
tags_to_add: {
"amenity": "hospital",
"healthcare:speciality": "psychiatry",
}
},
// Hôpital des armées
114: {
tags_to_add: {
"amenity": "hospital",
}
},
//Établissement de Soins Obstétriques Chirurgico-Gynécologiques
122: {
tags_to_add: {
"amenity": "hospital",
"healthcare:speciality": "obstetrics"
}
},
// Centre de Santé
124: {
tags_to_add: {
"healthcare": "center",
}
},
// Centre de Santé Dentaire
125: {
tags_to_add: {
"amenity": "dentist",
"healthcare": "dentist",
"healthcare:speciality": "dentist",
}
},
// Centre Médico-Psychologique (C.M.P.)
156: {
tags_to_add: {
"healthcare": "centre",
"healthcare:speciality": "psychiatry"
}
},
// "Laboratoires de Biologie Médicale": {
611: {
tags_to_add: {
"healthcare": "laboratory"
}
},
// Centre d'Accueil Th\u00e9rapeutique \u00e0 temps partiel (C.A.T.T.P.)
425: {
tags_to_add: {
"healthcare": "centre",
"healthcare:speciality": "psychiatry"
}
},
// "Centre Hospitalier (C.H.)": {
412: {
tags_to_add: {
"amenity": "hospital",
"healthcare": "hospital"
}
},
// "Pharmacie d'Officine": {
620: {
tags_to_add: {
"healthcare": "pharmacy"
}
},
// "Maison de santé (L.6223-3)": {
603: {
tags_to_add: {
"healthcare": "centre"
}
},
}
},
}
}
export default mappingFINESS;

175
mappings/converters/configIRVE.ts Normal file
View File

@ -0,0 +1,175 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const MappingIRVE: MappingConfigType = {
config_name: "IRVE config",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'charging_station'
},
source: {
geojson_path: "etalab_data/all.json",
url: 'https://www.data.gouv.fr/fr/datasets/r/7eee8f09-5d1b-4f48-a304-5e99e8da1e26'
},
/**
* select only certain points from the source
*/
filters: {
enable_coordinates_filter: false,
enable_properties_filter: true,
filter_points_lesser_than_NkW: 22 // ne pas sortir les points qui ont moins de ce nombre de puissance nominale
// add only geojson points who are found having this regex in the zipcode properties
// properties: {
// consolidated_code_postal: '^[76|27]'
// },
// bounding_box: [
// {}
// ]
},
add_not_mapped_tags_too: false,
boolean_keys: [
"prise_type_ef",
"prise_type_2",
"prise_type_combo_ccs",
"prise_type_chademo",
"gratuit",
"paiement_acte",
"paiement_cb",
"cable_t2_attache"
],
tags: {
// ******* nombres
nbre_pdc: 'capacity',
// ******* textes
amenity: 'amenity', // conserver le tag de base
capacity: 'capacity', // conserver le tag de base
nom_amenageur: 'operator',
siren_amenageur: 'owner:ref:FR:SIREN',
nom_operateur: 'operator',
telephone_operateur: {
key_converted: 'operator:phone',
convert_to_phone: true, // conversion en format international si possible
},
phone: {
key_converted: 'operator:phone',
convert_to_phone: true, // conversion en format international si possible
},
contact_operateur: 'operator:email', // ici, on souhaite convertir la clé contact_operateur=bidule en email=bidule
id_station_itinerance: {
key_converted: 'ref:EU:EVSE',
remove_stars: true,
},
id_station_local: 'ref',
/**
* l'info de gratuité a été mal renseignée par les opérateurs, ils mettent TRÈS souvent que c'est gratuit alors que ce n'est pas vrai.
*/
gratuit: {
key_converted: 'fee',
convert_to_boolean_value: true,
invert_boolean_value: true,
},
paiement_acte:
{
key_converted: 'authentication:none',
convert_to_boolean_value: true, // convertit en yes ou no
},
reservation: {
convert_to_boolean_value: true, // convertit en yes ou no
},
// observations: 'note',
nom_station: 'description',
nom_enseigne: 'network',
// ******* dates
date_mise_en_service: 'start_date',
// date_maj: 'source:date',
// ******** champs booléens
cable_t2_attache: {
key_converted: 'socket:type2_cable',
// socket:type2_cable vaut 1 dans OSM si vrai
truthy_value: '1',
ignore_if_falsy: true,
}
,
prise_type_ef: {
key_converted: 'socket:typee',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
prise_type_2: {
key_converted: 'socket:type2',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
prise_type_combo_ccs: {
key_converted: 'socket:type2_combo',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
prise_type_chademo: {
key_converted: 'socket:chademo',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
// ******** champs plus complexes
horaires: 'opening_hours', // déjà au bon format, enfin, en général. vérifier avec le validateur josm.
paiement_cb: {
key_converted: 'payment:credit_cards',
// ignore_if_falsy: true,
convert_to_boolean_value: true,
},
accessibilite_pmr: {
key_converted: "wheelchair",
conditional_values: {
"Accessibilité inconnue": {
ignore_this_data: true, // ne pas ajouter de tag si la valeur est égale à Accessibilité inconnue.
},
"Accessible mais non réservé PMR": {
value_converted: "yes"
},
"Réservé PMR": {
value_converted: "yes"
},
"Non accessible": {
value_converted: "no"
},
"Mo-Fr 08:30-12:00,Mo-Fr 14:00-19:00,Sat 09:00-18:30": {
value_converted: "Mo-Fr 08:30-12:00,Mo-Fr 14:00-19:00,Sat 09:00-18:30"
},
// On précise que l'on veut conserver cette valeur, par défaut tout ce qui n'est pas précisé en valeur conditionnelle n'est pas conservé
"24/7": {
value_converted: "24/7"
}
},
},
station_deux_roues: {
remove_original_key: true,
conditional_values: {
// ajout de trois tags si la valeur est yes
"yes": {
tags_to_add: [
{bicycle: "yes"},
{scooter: "yes"},
{motorcar: "no"},
]
}
}
},
puissance_nominale: {
key_converted: 'charging_station:output',
socket_output_find_correspondances: true,
}
},
}
export default MappingIRVE;

169
mappings/converters/configIRVESuisse.ts Normal file
View File

@ -0,0 +1,169 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const MappingIRVESuisse: MappingConfigType = {
config_name: "IRVE config",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'charging_station'
},
source: {
geojson_path: "etalab_data/all.json",
url: 'https://www.data.gouv.fr/fr/datasets/r/7eee8f09-5d1b-4f48-a304-5e99e8da1e26'
},
/**
* select only certain points from the source
*/
filters: {
enable_coordinates_filter: false,
enable_properties_filter: true,
// add only geojson points who are found having this regex in the zipcode properties
// properties: {
// consolidated_code_postal: '^[76|27]'
// },
// bounding_box: [
// {}
// ]
},
add_not_mapped_tags_too: false,
boolean_keys: [
"prise_type_ef",
"prise_type_2",
"prise_type_combo_ccs",
"prise_type_chademo",
"gratuit",
"paiement_acte",
"paiement_cb",
"cable_t2_attache"
],
tags: {
// ******* nombres
nbre_pdc: 'capacity',
// ******* textes
amenity: 'amenity', // conserver le tag de base
capacity: 'capacity', // conserver le tag de base
nom_amenageur: 'operator',
siren_amenageur: 'owner:ref:FR:SIREN',
nom_operateur: 'operator',
telephone_operateur: {
key_converted: 'operator:phone',
convert_to_phone: true, // conversion en format international si possible
},
phone: {
key_converted: 'operator:phone',
convert_to_phone: true, // conversion en format international si possible
},
contact_operateur: 'operator:email', // ici, on souhaite convertir la clé contact_operateur=bidule en email=bidule
id_station_itinerance: 'ref:EU:EVSE',
id_station_local: 'ref',
/**
* l'info de gratuité a été mal renseignée par les opérateurs, ils mettent TRÈS souvent que c'est gratuit alors que ce n'est pas vrai.
*/
gratuit: {
key_converted: 'fee',
convert_to_boolean_value: true,
invert_boolean_value: true,
},
paiement_acte:
{
key_converted: 'authentication:none',
convert_to_boolean_value: true, // convertit en yes ou no
},
reservation: {
convert_to_boolean_value: true, // convertit en yes ou no
},
// observations: 'note',
nom_station: 'description',
nom_enseigne: 'network',
// ******* dates
date_mise_en_service: 'start_date',
// date_maj: 'source:date',
// ******** champs booléens
cable_t2_attache: {
key_converted: 'socket:type2_cable',
// socket:type2_cable vaut 1 dans OSM si vrai
truthy_value: '1',
ignore_if_falsy: true,
}
,
prise_type_ef: {
key_converted: 'socket:typee',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
prise_type_2: {
key_converted: 'socket:type2',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
prise_type_combo_ccs: {
key_converted: 'socket:type2_combo',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
prise_type_chademo: {
key_converted: 'socket:chademo',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
// ******** champs plus complexes
horaires: 'opening_hours', // déjà au bon format, enfin, en général. vérifier avec le validateur josm.
paiement_cb: {
key_converted: 'payment:credit_cards',
// ignore_if_falsy: true,
convert_to_boolean_value: true,
},
accessibilite_pmr: {
key_converted: "wheelchair",
conditional_values: {
"Accessibilité inconnue": {
ignore_this_data: true, // ne pas ajouter de tag si la valeur est égale à Accessibilité inconnue.
},
"Accessible mais non réservé PMR": {
value_converted: "yes"
},
"Réservé PMR": {
value_converted: "yes"
},
"Non accessible": {
value_converted: "no"
},
"Mo-Fr 08:30-12:00,Mo-Fr 14:00-19:00,Sat 09:00-18:30": {
value_converted: "Mo-Fr 08:30-12:00,Mo-Fr 14:00-19:00,Sat 09:00-18:30"
},
// On précise que l'on veut conserver cette valeur, par défaut tout ce qui n'est pas précisé en valeur conditionnelle n'est pas conservé
"24/7": {
value_converted: "24/7"
}
},
},
station_deux_roues: {
remove_original_key: true,
conditional_values: {
// ajout de trois tags si la valeur est yes
"yes": {
tags_to_add: [
{bicycle: "yes"},
{scooter: "yes"},
{motorcar: "no"},
]
}
}
},
puissance_nominale: {
key_converted: 'charging_station:output',
socket_output_find_correspondances: true,
}
},
}
export default MappingIRVESuisse;

44
mappings/converters/configIRVE_osmose.ts Normal file
View File

@ -0,0 +1,44 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const MappingIRVEOsmose: MappingConfigType = {
config_name: "IRVE config for Osmose IRVE items",
osmose: true,
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'charging_station'
},
source: {
geojson_path: "etalab_data/all.json",
url: 'https://osmose.openstreetmap.fr/api/0.3/issues.geojson?full=true&status=open&item=8410&limit=20000'
},
/**
* select only certain points from the source
*/
filters: {
enable_coordinates_filter: false,
enable_properties_filter: true,
},
add_not_mapped_tags_too: true,
tags: {
// // ******* nombres
// // ******* textes
amenity: 'amenity', // conserver le tag de base
capacity: 'capacity', // conserver le tag de base
'owner:ref:FR:SIREN': 'owner:ref:FR:SIREN',
operator: 'operator',
'operator:phone': {
key_converted: 'operator:phone',
convert_to_phone: true, // conversion en format international si possible
},
'phone': {
key_converted: 'operator:phone',
convert_to_phone: true, // conversion en format international si possible
},
},
}
export default MappingIRVEOsmose;

64
mappings/converters/configIssy_OpenData_2roues.ts Normal file
View File

@ -0,0 +1,64 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const mappingIssy2Roues: MappingConfigType = {
config_name: "mappingIssy2Roues",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'bicycle_parking'
},
source: {
geojson_path: "cyclabilité/issy_les_mx_cyclabilité.json",
url: 'https://data.issy.com/explore/dataset/parkings-2-roues/table/?disjunctive.commune=&disjunctive.type_aire=&disjunctive.type_vehicule=&disjunctive.etat=&disjunctive.pres_abri=&disjunctive.acces_reglement=&disjunctive.position=&sort=nb_places'
},
filters: {
},
add_not_mapped_tags_too: false,
boolean_keys: [
"acces_reglement",
],
tags: {
// ******* nombres
nb_places: 'capacity',
nb_pl_velo: 'capacity:bike',
nb_pl_moto: 'capacity:motorcycle',
annee_crea: 'start_date',
// ******* textes
type_aire: {
"remove_original_key": true,
"conditional_values": {
"Poteaux": {
"tags_to_add": [
{
"bicycle_parking": "bollard"
},
]
},
"Arceaux": {
"tags_to_add": [
{
"bicycle_parking": "bollard"
},
]
},
"Anneaux": {
"tags_to_add": [
{
"bicycle_parking": "stands"
},
]
},
}
},
// type_aire: 'nb_pl_moto', // Poteaux => bicycle_parking=bollard, Arceaux, Anneaux => bicycle_parking=stands, Marquage, null, Non renseigné, Range-vélos, Sans, Trottoir
id_r2roues: 'ref:FR:r2roues_issy_les_mx',
// acces_reglement: 'access',
}
}
export default mappingIssy2Roues;

28
mappings/converters/configMuseums.ts Normal file
View File

@ -0,0 +1,28 @@
import MappingConfigType from "../mapping-config.type";
const MappingMuseums: MappingConfigType = {
config_name: 'Mapping des musées en Île de France',
config_author: 'tykayn <contact+geojson2osm@cipherbliss.com>',
default_properties_of_point: {tourism: 'museum'},
tags: {
// ******* booléens
// ******* nombres
"URL": "contact:website",
"Identifiant Muséofile": "ref:FR:museums:museofile",
"REF_Deps": "ref:FR:museums:deps",
// ******* textes
'Téléphone': {
key_converted: "contact:phone",
convert_to_phone: true,
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: '',
url: ''
},
filters: {
// offset:2 // uncomment to run small tests
},
}
export default MappingMuseums

50
mappings/converters/configPlanningFamilial.ts Normal file
View File

@ -0,0 +1,50 @@
/**
* commerces adhérant à Ask Angela
* https://wiki.openstreetmap.org/wiki/Tag:healthcare:speciality%3Dfamily_planning
*/
import MappingConfigType from "../mapping-config.type";
const MappingPlanningFamlial: MappingConfigType = {
config_name: "planning familial",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'healthcare:speciality': 'family_planning',
},
source: {
// source faite à partir de data scraping du site français
geojson_path: '',
url: 'https://www.planning-familial.org/fr',
overpass_query:`
[out:json][timeout:200];
{{geocodeArea:"France"}}->.searchArea;
nwr["healthcare:speciality"="family_planning"](area.searchArea);
out geom;
`
},
filters: {
// offset: 1
},
add_not_mapped_tags_too: false,
boolean_keys: [],
tags_to_ignore_if_value_is: ['Non renseigne'],
tags: {
adresse: "addr:full",
nom : {
key_converted: 'name',
convert_to_name: true,
},
telephone : {
key_converted: 'contact:phone',
convert_to_phone: true,
},
"contact:website":"contact:website",
"family_planning:handles:violences" : "family_planning:handles:violences",
"family_planning:handles:sexualities" : "family_planning:handles:sexualities",
"family_planning:handles:detection" : "family_planning:handles:detection",
"family_planning:handles:abortion" : "family_planning:handles:abortion",
"family_planning:handles:contraception" : "family_planning:handles:contraception",
// ******* opendata de toulouse END **************
}
}
export default MappingPlanningFamlial;

88
mappings/converters/configRouen_OpenData_velo_parkings.ts Normal file
View File

@ -0,0 +1,88 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const mappingRouenParkingVelos: MappingConfigType = {
config_name: "mappingRouenParkingVelos",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'bicycle_parking'
},
source: {
geojson_path: "cyclabilité/rouen_parking_velos.json",
url: 'https://data.metropole-rouen-normandie.fr/api/explore/v2.1/catalog/datasets/liste-des-stationnements-cyclables-metropole-rouen-normandie/exports/geojson?lang=fr&timezone=Europe%2FBerlin'
},
filters: {
exclude_point_if_tag_not_empty: ['id_osm'], // on peut exclure des données converties celles qui sont déjà avec un identifiant openstreetmap afin de favoriser l'intégration san avoir à gérer les doublons
// offset: 100
},
add_not_mapped_tags_too: false,
boolean_keys: [
"acces_reglement",
],
tags_to_ignore_if_value_is: ['Non renseigne'],
tags: {
// ******* nombres
capacite: 'capacity',
capacite_cargo: 'capacity:cargo_bike',
gestionnaire: {
key_converted: 'operator',
conditional_values: {
"Non renseigne": {
ignore_this_data: true, // ne pas ajouter de tag si la valeur est égale à
}
}
},
proprietaire: {
key_converted: 'owner',
conditional_values: {
"Non renseigne": {
ignore_this_data: true, // ne pas ajouter de tag si la valeur est égale à
}
}
},
date_maj: 'check_date',
// ******* textes
commentaire: 'note',
surveillance: {
key_converted: 'surveillance',
ignore_if_falsy: true,
convert_to_boolean_value: true,
},
mobilier: {
key_converted: 'mobilier',
remove_original_key: true, // TODO trouver ce qui empêche la conversion quand remove_original_key est a true
conditional_values: {
"POTELET": {
tags_to_add: [
{
"bicycle_parking": "bollard"
},
]
},
"ARCEAU": {
tags_to_add: [
{
"bicycle_parking": "stands"
},
]
},
"RATELIER": {
tags_to_add: [
{
"bicycle_parking": "rack"
},
]
},
}
},
id_local: 'ref:FR:rouen_veloparking_id',
// acces_reglement: 'access',
}
}
export default mappingRouenParkingVelos;

59
mappings/converters/configRouen_PAV.ts Normal file
View File

@ -0,0 +1,59 @@
/**
* points d'apport volontaire de Rouen
*/
import MappingConfigType from "../mapping-config.type";
const MappingRouenPAV: MappingConfigType = {
config_name: "mappingRouen points d'apport volontaire",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'recycling',
'recycling_type': 'container',
},
source: {
geojson_path: 'https://data.metropole-rouen-normandie.fr/api/explore/v2.1/catalog/datasets/donmetdec_pav/exports/geojson?lang=fr&timezone=Europe%2FBerlin',
url: 'https://data.metropole-rouen-normandie.fr/explore/dataset/donmetdec_pav/information/'
},
filters: {
// exclude_point_if_tag_not_empty: ['id_osm'], // on peut exclure des données converties celles qui sont déjà avec un identifiant openstreetmap afin de favoriser l'intégration san avoir à gérer les doublons
// offset: 50
},
add_not_mapped_tags_too: false,
boolean_keys: [
// "acces_reglement",
],
tags_to_ignore_if_value_is: ['Non renseigne'],
tags: {
// ******* nombres
pavtyp: {
remove_original_key:true,
conditional_values: {
"Emballages en verre": {
tags_to_add: {
'recycling:glass': 'yes'
}
},
"Emballages recyclables": {
tags_to_add: {
'recycling:pmd': 'yes' // https://wiki.openstreetmap.org/wiki/Key:recycling:pmd
}
},
"Ordures ménagères": {
tags_to_add: {
'amenity': 'waste_disposal'
}
}
}
},
date_maj: 'check_date',
// ******* textes
gml_id: 'ref:FR:Rouen:pav:gml_id',
objectid_1: 'ref:FR:Rouen:pav:object',
pavid: 'ref:FR:Rouen:pav:id',
pavnom: 'ref:FR:Rouen:pav:name',
commune: 'ref:FR:Rouen:commune',
}
}
export default MappingRouenPAV;

32
mappings/converters/configSurveillance.ts Normal file
View File

@ -0,0 +1,32 @@
/**
* caméras référencées sur le site de rouen
*
* certaines ont une précision "Caméra de sécurité sur borne escamotable"
* https://wiki.openstreetmap.org/wiki/Tag:man_made=surveillance
*/
import MappingConfigType from "../mapping-config.type";
const MappingSurveillanceRouen: MappingConfigType = {
config_name: "caméras de Rouen",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'man_made ': 'surveillance',
},
source: {
// source faite à partir de data scraping du site français
geojson_path: 'https://umap.openstreetmap.fr/fr/datalayer/44208/f923d80e-2ede-4ab8-b883-a7646491c506/',
url: 'https://rouen.fr/videoprotection'
},
filters: {
// exclude_point_if_tag_not_empty: ['id_osm'], // on peut exclure des données converties celles qui sont déjà avec un identifiant openstreetmap afin de favoriser l'intégration san avoir à gérer les doublons
// offset: 1
},
add_not_mapped_tags_too: false,
boolean_keys: [],
tags_to_ignore_if_value_is: ['Non renseigne'],
tags: {
//
}
}
export default MappingSurveillanceRouen;

24
mappings/converters/configTest.ts Normal file
View File

@ -0,0 +1,24 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const MappingTest: MappingConfigType = {
config_name: 'testing config MappingTest',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
nom_amenageur : 'name'
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export default MappingTest;

39
mappings/converters/config_toilettes.ts Normal file
View File

@ -0,0 +1,39 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
import MappingConfigType from "../mapping-config.type";
const MappingToilettes: MappingConfigType = {
config_name: "toilettes config for paris sanisettes",
config_author: "tykayn <contact@cipherbliss.com>",
default_properties_of_point: {
'amenity': 'toilets'
},
source: {
geojson_path: "etalab_data/toilettes_paris_datagouv.json",
url: 'https://opendata.paris.fr/api/explore/v2.1/catalog/datasets/sanisettesparis/exports/geojson?lang=fr&timezone=Europe%2FBerlin'
},
/**
* select only certain points from the source
*/
filters: {
enable_coordinates_filter: false,
enable_properties_filter: false,
// add only geojson points who are found having this regex in the zipcode properties
properties: {
// consolidated_code_postal: '^[75]'
}
},
add_not_mapped_tags_too: false,
tags: {
nom_operateur: 'operator',
telephone_operateur: 'phone',
contact_operateur: 'email', // ici, on souhaite convertir la clé
},
}
export default MappingToilettes;

45
mappings/converters/mappingConfigIRVE_simple.ts Normal file
View File

@ -0,0 +1,45 @@
/**
* plan de conversion des clés du jeu de données vers les tags OSM
* détail dans le tableau
* https://wiki.openstreetmap.org/wiki/France/data.gouv.fr/Bornes_de_Recharge_pour_V%C3%A9hicules_%C3%89lectriques
*/
const mappingIRVE:any = {
// ******* nombres
nbre_pdc: 'capacity',
amenity: 'amenity', // conserver le tag de base
capacity: 'capacity', // conserver le tag de base
nom_amenageur: 'operator',
siren_amenageur: 'owner:ref:FR:SIREN',
nom_operateur: 'operator',
telephone_operateur: 'phone',
// ici, on souhaite convertir la clé contact_operateur=bidule en email=bidule
contact_operateur: 'email',
id_station_itinerance: 'ref:EU:EVSE',
id_station_local: 'ref',
gratuit: 'fee',
paiement_acte: 'authentication:none',
reservation: 'reservation',
observations: 'note',
nom_station: 'name',
nom_enseigne: 'network',
// ******* dates
date_mise_en_service: 'start_date',
date_maj: 'source:date',
// ******** champs booléens
prise_type_ef: 'socket:typee',
prise_type_2: 'socket:type2',
prise_type_combo_ccs: 'socket:type2_combo',
prise_type_chademo: 'socket:chademo',
// ******** champs plus complexes
horaires: 'opening_hours', // déjà au bon format
}
export default mappingIRVE;

626
mappings/engine.ts Normal file
View File

@ -0,0 +1,626 @@
import custom_utils from './utils'
import MappingConfigType from "./mapping-config.type";
import Formatters from "./formatters";
const {debugLog} = custom_utils
let listOfBooleanKeys = [
"prise_type_ef",
"prise_type_2",
"prise_type_combo_ccs",
"prise_type_chademo",
"gratuit",
"paiement_acte",
"paiement_cb",
"cable_t2_attache"
]
function boolToAddable(someBooleanValue: boolean) {
return someBooleanValue ? 1 : 0
}
export default class {
mapping_config: any = {}
public stats: any;
truthyValues = [true, 'true', 'True', 'TRUE', '1', 'yes', 1]
falsyValues = [false, 'false', 'False', 'FALSE', '0', 'no', 0]
private jardinage = false;
private current_converted_geojson_point: any;
private current_geojson_point: any; // currently converting point
private list_of_points: any; // list of geojson points
constructor(mappingConfig: MappingConfigType) {
this.setConfig(mappingConfig)
this.stats = {
filtered_by_excluded_tags: 0,
phones_updated: 0,
power_output: 0,
phones_updated_list: [],
phones_not_updated: 0
}
}
setConfig(mappingConfig: MappingConfigType) {
debugLog('load config', mappingConfig.config_name)
debugLog('load config', mappingConfig.config_name)
this.mapping_config = mappingConfig
}
getConfig() {
return this.mapping_config
}
mapFeaturePoint(featurePointGeoJson: any) {
let geoJSONConvertedPoint: any = {}
geoJSONConvertedPoint.properties = {...this.mapping_config.default_properties_of_point}
geoJSONConvertedPoint.type = featurePointGeoJson.type
geoJSONConvertedPoint.geometry = featurePointGeoJson.geometry
this.current_converted_geojson_point = geoJSONConvertedPoint
return geoJSONConvertedPoint
}
/**
* TODO convert to mapping config property to transform_truthy
* @param pointKeyName
* @returns {boolean}
*/
isBooleanKey(pointKeyName: string): boolean {
return listOfBooleanKeys.indexOf(pointKeyName) !== -1
}
/**
* filter: reduce number of features
* @param offsetCount
* @param listOfFeatures
*/
filterFeaturesByOffset(offsetCount: number, listOfFeatures: any): Array<any> {
let filteredList = listOfFeatures
// TODO
return filteredList
}
/**
* filterFeaturesByPropertyRegex
* TODO
* @param propertyName
* @param criteriaRegex
* @param listOfFeatures
*/
filterFeaturesByPropertyRegex(propertyName: string, criteriaRegex: any, listOfFeatures: any) {
let filteredList = listOfFeatures.filter((feature: any) => {
return criteriaRegex.test(feature?.properties[propertyName])
})
return filteredList
}
/**
* filter a list of geojson points if one of the given exludedKeys is present in their properties.
* Example, we do not want to convert already present OSM point which have an osm_id value in their properties.
* @param list
* @param excludedKeys
*/
filterListOfPointsByExcludingIfKeyFilled(list: any, excludedKeys: Array<string>): any[] {
let newList: Array<any> = []
list.forEach((geojsonPoint: any) => {
let pointProperties = Object.keys(geojsonPoint.properties)
let addPoint = true;
excludedKeys.forEach((key: any) => {
debugLog(key, 'pointProperties[key]', pointProperties[key])
let foundProperty: string = pointProperties[key]
if (foundProperty && foundProperty !== 'null') {
addPoint = false
}
})
if (addPoint) {
// only add points that pass the not null filter
newList.push(geojsonPoint)
} else {
this.stats.filtered_by_excluded_tags++
}
})
return newList;
}
filterListOfPointsByExcludingIfMaxPowerIsLesserThan(minValue: number, list_of_points: any[]): any[] {
let newList: any[] = []
list_of_points.forEach((geojsonPoint: any) => {
let pointProperties = Object.keys(geojsonPoint.properties)
// trouver la valeur
// socket_output_find_correspondances
if (pointProperties.includes('puissance_nominale') &&
1 * (geojsonPoint.properties['puissance_nominale'].replace(' kW', '')) > minValue
) {
newList.push(geojsonPoint)
}
})
return newList;
}
/**
* retuns the converted element from mapping config if present, null otherwise
*/
mapElementFromConf(featurePoint: any): any {
debugLog('mapElementFromConf: mapElementFromConf', featurePoint)
if (!this.mapping_config) {
throw new Error('no config was loaded in the mapping engine. use setConfig(my_mapping_config) on this instance of mapping engine before using this. Your config should be typed to MappingConfigType Type.')
}
debugLog('mapElementFromConf: config_name', this.mapping_config.config_name)
let mappingKeys = Object.keys(this.mapping_config.tags)
let featurePointPropertiesKeys = []
if (this.mapping_config.osmose) {
// only creation of new points are handled by now [2023-10-07]
featurePointPropertiesKeys = Object.keys(featurePoint.properties.fixes[0][0].create)
// debugLog('featurePointPropertiesKeys', featurePointPropertiesKeys)
} else {
featurePointPropertiesKeys = Object.keys(featurePoint.properties)
}
debugLog('mapElementFromConf: ============= keys mappingKeys:', this.mapping_config.tags.length, mappingKeys.length)
debugLog('mapElementFromConf: ============= keys featurePointPropertiesKeys :', featurePoint.properties.length, featurePointPropertiesKeys.length)
let newProperties = {...this.mapping_config.default_properties_of_point}
// reinit properties of current point
let basePoint = Object.create(featurePoint)
basePoint.type = featurePoint.type
basePoint.geometry = featurePoint.geometry
basePoint.properties = {...this.mapping_config.default_properties_of_point}
// apply new properties if found in mapping config
featurePointPropertiesKeys.forEach(pointKeyName => {
debugLog('mapElementFromConf: convert', pointKeyName)
debugLog('mapElementFromConf: mapping keys:', mappingKeys)
this.convertProperty(pointKeyName, mappingKeys, featurePoint, newProperties)
})
basePoint.properties = newProperties
// debugLog('mapElementFromConf: basePoint', basePoint)
return basePoint
}
/**
* convertit une propriété en une autre selon la config de mapping
* @param pointKeyName
* @param mappingKeys
* @param featurePoint
* @param newProperties
*/
convertProperty(pointKeyName: string, mappingKeys: any, featurePoint: any, newProperties: any) {
this.current_geojson_point = featurePoint
let originalValue = ''
if (this.mapping_config.osmose) {
originalValue = featurePoint.properties.fixes[0][0].create[pointKeyName]
} else {
originalValue = featurePoint.properties[pointKeyName]
}
let intOriginalValue = parseInt(originalValue)
let mappingValueObject: any = '';
if (mappingKeys.indexOf(pointKeyName) !== -1) {
mappingValueObject = this.mapping_config.tags[pointKeyName]
debugLog('convertProperty: mappingValueObject ', mappingValueObject)
}
debugLog(' ------ convertProperty: pointKeyName', pointKeyName)
// debugLog('convertProperty: mappingKeys', mappingKeys)
let remove_original_key = false;
debugLog('tags_to_ignore_if_value_is', this.mapping_config.tags_to_ignore_if_value_is)
if (this.mapping_config.tags_to_ignore_if_value_is && this.mapping_config.tags_to_ignore_if_value_is.length && this.mapping_config.tags_to_ignore_if_value_is?.indexOf(originalValue) !== -1) {
debugLog('(x) => ignore', originalValue, ' in ', pointKeyName)
remove_original_key = true;
}
if (this.jardinage) {
debugLog(' ------ on fait du jardinage')
debugLog(' ------ mode mise en qualité activé')
debugLog(' ------ les données en entrée sont des infos geojson extraites depuis overpass turbo.')
debugLog(' ------ les clés des objets sont donc déjà dans le format de tag OSM,' +
'ne pas les convertir pour les mettre en qualité selon le modèle de mapping.')
}
if (this.mapping_config.add_not_mapped_tags_too && (mappingKeys.indexOf(pointKeyName) === -1)) {
/**
* add all unmapped tags is enabled
*/
debugLog(' ------ add all unmapped tags is enabled')
newProperties[pointKeyName] = originalValue;
} else {
/**
* only use existing keys
*/
debugLog("only use existing keys,", pointKeyName)
if (mappingKeys.indexOf(pointKeyName) !== -1) {
let valueConvertedFromMapping = featurePoint.properties[pointKeyName]
let keyConvertedFromMapping = mappingKeys[mappingKeys.indexOf(pointKeyName)]
let mappingConfigOfTag = this.mapping_config.tags[pointKeyName]
debugLog('========== mappingConfigOfTag', mappingConfigOfTag)
debugLog('convertProperty: found element', pointKeyName, '=>', keyConvertedFromMapping, 'value : ', valueConvertedFromMapping)
let convertedValue = originalValue
let typeOfConfigForKey = typeof mappingConfigOfTag
let isStringValue = typeOfConfigForKey === 'string'
let isConfigMappingObject = typeOfConfigForKey === 'object'
debugLog('convertProperty: - typeofValue', typeOfConfigForKey)
debugLog('convertProperty: - pointKeyName', pointKeyName)
debugLog('convertProperty: - valueConvertedFromMapping', valueConvertedFromMapping)
debugLog('typeof valueConvertedFromMapping === \'string\'', typeOfConfigForKey)
debugLog('convertProperty: isStringValue?', valueConvertedFromMapping, isStringValue)
debugLog('convertProperty: isStringValue?', valueConvertedFromMapping, isStringValue)
debugLog('mappingConfigOfTag', mappingConfigOfTag)
debugLog('typeOfConfigForKey', typeOfConfigForKey)
/**
* conversion si la clé à une config d'une string, on ne change que la clé, pas la valeur
*/
if (isStringValue) {
debugLog('convertProperty: -- string value')
debugLog('convertProperty: -- string value')
debugLog('convertProperty: -- simple conversion : ', pointKeyName, '=> ', mappingConfigOfTag, '_', originalValue, '=>', valueConvertedFromMapping)
debugLog('convertProperty: -- convertedValue', convertedValue)
convertedValue = valueConvertedFromMapping
if (convertedValue) {
newProperties[mappingConfigOfTag] = convertedValue
}
} else {
debugLog('convertProperty: no string value')
}
let configObject = mappingConfigOfTag
if (isConfigMappingObject) {
debugLog('convertProperty: is config object', configObject)
let newKey: any = '' + pointKeyName
if (configObject.key_converted) {
newKey = configObject.key_converted
debugLog('key_converted newKey', newKey)
}
if (configObject.transform_function) {
convertedValue = configObject.transform_function(originalValue)
}
if (configObject.truthy_value) {
// convertir la valeur, si elle est truthy, la transformer en ce que donne la propriété truthy_value
// exemple: le jeu de données dit que la colonne cable_t2_attache vaut "True", mais on veut le convertir en "1".
// on met donc truthy_value: '1'
debugLog('truthy_value', originalValue)
if (this.truthyValues.indexOf(originalValue) !== -1) {
convertedValue = configObject.truthy_value
}
}
if (configObject.falsy_value) {
if (this.falsyValues.indexOf(originalValue) !== -1) {
convertedValue = configObject.falsy_value
}
}
/**
* conversion booléenne
*/
if (mappingValueObject.convert_to_boolean_value) {
debugLog('convertProperty: is boolean_value_conversion')
convertedValue = this.convertToYesOrNo(originalValue)
} else {
debugLog('convertProperty: is NOT having boolean_value_conversion', mappingValueObject)
}
// gestion des puissances de bornes
// avec une fonction de transformation des valeurs
// parmi le domaine du jeu de données
// nécessite une clé conditionnelle à la valeur true d'autres clés converties.
if (configObject.socket_output_find_correspondances) {
// trouver à quel socket ça correspond
// si y'a plusieurs sockets, utiliser socket:max:output
let we_use_max_output = false;
let has_prise_type_2: boolean = this.isTruthyValue(this.current_geojson_point.properties.prise_type_2) || false
let has_prise_type_combo_ccs: boolean = this.isTruthyValue(this.current_geojson_point.properties.prise_type_combo_ccs) || false
let prise_type_chademo: boolean = this.isTruthyValue(this.current_geojson_point.properties.prise_type_chademo) || false
let prise_type_ef: boolean = this.isTruthyValue(this.current_geojson_point.properties.prise_type_ef) || false
let prise_type_e: boolean = this.isTruthyValue(this.current_geojson_point.properties.prise_type_e) || false
let prise_type_autre: boolean = this.isTruthyValue(this.current_geojson_point.properties.prise_type_autre) || false
let countOfSockets = (boolToAddable(has_prise_type_2) + boolToAddable(has_prise_type_combo_ccs) + boolToAddable(prise_type_chademo) +
boolToAddable(prise_type_ef) + boolToAddable(prise_type_autre) + boolToAddable(prise_type_e)
);
if (countOfSockets > 0) {
we_use_max_output = true;
}
// ajouter les tags de socket newProperties
let converted_value = originalValue.replace(/[^\d\.\,]/g, '').replace(',', '.')
let max_output = 401
// do not limit accepted values
let out = ''
if (intOriginalValue < max_output) {
// rajouter l'unité de puissance kW dans la valeur
out = converted_value + ' kW'
} else {
// prise en charge des valeurs en Watts et non en kW.
debugLog('too high kW value detected', originalValue)
if (intOriginalValue > 1000 && intOriginalValue < 401000) {
let kilowatts = (parseFloat(converted_value) / 1000).toFixed(2).replace('.00', '');
out = ('' + kilowatts + ' kW').replace('.00', '')
debugLog('valeurs en Watts out', out, 'original:', originalValue)
this.stats.power_output++
}
}
out = (out).replace('.00', '')
// debug land
if (has_prise_type_combo_ccs) {
newProperties['socket:type2_combo:output'] = out;
this.stats.power_output++
}
if (we_use_max_output) {
newProperties['charging_station:output'] = out;
} else {
if (has_prise_type_2 && prise_type_e) {
newProperties['socket:type_2:output'] = out;
this.stats.power_output++
debugLog('2 prises, attribuer la plus haute valeur à la type 2', out)
}
if (countOfSockets === 1) {
if (has_prise_type_2) {
newProperties['socket:type_2:output'] = out;
newProperties['socket:type_2'] = 1;
this.stats.power_output++
}
if (has_prise_type_combo_ccs) {
newProperties['socket:type2_combo:output'] = out;
newProperties['socket:type2_combo'] = 1;
this.stats.power_output++
}
if (prise_type_chademo) {
newProperties['socket:chademo:output'] = out;
newProperties['socket:chademo'] = 1;
this.stats.power_output++
}
if (prise_type_e) {
newProperties['socket:typee:output'] = out;
newProperties['socket:typee'] = 1;
this.stats.power_output++
}
} else {
debugLog('no sockets', this.current_geojson_point.properties.ref)
}
}
return out
}
if (configObject.invert_boolean_value) {
convertedValue = !this.convertToBoolean(originalValue) ? 'yes' : 'no'
debugLog('invert boolean', convertedValue, originalValue)
}
if (configObject.remove_stars) {
convertedValue = originalValue.replace('*', '')
debugLog('remove_stars', convertedValue, originalValue)
}
if (configObject.convert_to_phone) {
convertedValue = Formatters.convertToPhone(originalValue)
if (originalValue !== convertedValue) {
this.stats.phones_updated++
this.stats.phones_updated_list.push(convertedValue)
} else {
this.stats.phones_not_updated++
}
debugLog('convertedValue convert_to_phone', originalValue, '=>', convertedValue)
}
if (configObject.convert_to_name) {
convertedValue = Formatters.convertToName(originalValue)
}
if (configObject.remove_original_key) {
remove_original_key = true
}
if (configObject.ignore_if_falsy && this.falsyValues.indexOf(originalValue) !== -1) {
remove_original_key = true
}
if (configObject.ignore_if_truthy && this.truthyValues.indexOf(originalValue) !== -1) {
remove_original_key = true
}
/**
* config pour une clé
* nous pouvons renseigner une string ou un objet décrivant les transformations à réaliser
*/
if (configObject.conditional_values) {
// convert numbers from json to string to compare them correctly
originalValue = '' + originalValue
let keysConditionnalValues: any = Object.keys(configObject.conditional_values)
let isFoundValue = keysConditionnalValues.indexOf(originalValue)
let conditionnalConfig: any = configObject.conditional_values[keysConditionnalValues[isFoundValue]]
debugLog('convertProperty: conditional_values__________',
configObject.conditional_values)
debugLog('isFoundValue', isFoundValue, originalValue)
debugLog('keysConditionnalValues', keysConditionnalValues)
debugLog('-----++++++++ originalValue', originalValue)
debugLog('----------- isFoundValue', isFoundValue)
if (!remove_original_key) {
if (isFoundValue !== -1) {
debugLog('found condition', isFoundValue)
/** ----------------------
* gestion des valeurs conditionnelles
* ---------------------- */
debugLog('conditionnalConfig', conditionnalConfig)
if (conditionnalConfig.ignore_this_data) {
debugLog(`on ignore cette clé car sa valeur "${originalValue}" est à exclure: `, pointKeyName, '=>', newKey)
remove_original_key = true;
}
if (conditionnalConfig.truthy_value) {
// convertir la valeur, si elle est truthy, la transformer en ce que donne la propriété truthy_value
// exemple: le jeu de données dit que la colonne cable_t2_attache vaut "True", mais on veut le convertir en "1".
// on met donc truthy_value: '1'
if (this.truthyValues.indexOf(originalValue) !== -1) {
convertedValue = conditionnalConfig.truthy_value
}
}
if (conditionnalConfig.falsy_value) {
if (this.falsyValues.indexOf(originalValue) !== -1) {
convertedValue = conditionnalConfig.falsy_value
}
}
// use the value converted
else if (conditionnalConfig.value_converted) {
convertedValue = conditionnalConfig.value_converted
}
}
}
if (conditionnalConfig?.tags_to_add) {
debugLog('on ajoute des tags', conditionnalConfig.tags_to_add)
// on peut définir un ensemble de tags à rajouter
let tagKeys = Object.keys(conditionnalConfig.tags_to_add)
debugLog('conditionnalConfig.tags_to_add', conditionnalConfig.tags_to_add)
tagKeys.forEach((index: any) => {
debugLog('key', index)
debugLog('value', conditionnalConfig.tags_to_add[index])
newProperties[index] = conditionnalConfig.tags_to_add[index]
})
}
}
debugLog('convertProperty: convertedValue ==========> {', newKey, ':', convertedValue, '}')
debugLog(' =============== remove_original_key', newKey, remove_original_key)
let keysOfConfigObject = [];
let hasKeyIgnoreThisData = false;
if (configObject) {
keysOfConfigObject = Object.keys(configObject)
debugLog('keysOfConfigObject', keysOfConfigObject)
hasKeyIgnoreThisData = (keysOfConfigObject.indexOf('ignore_this_data') !== -1)
}
debugLog('remove_original_key && newKey && convertedValue && hasKeyIgnoreThisData', remove_original_key, newKey, convertedValue, hasKeyIgnoreThisData)
if (!remove_original_key && newKey && convertedValue && !hasKeyIgnoreThisData
) {
debugLog('convertedValue', convertedValue)
debugLog('convertProperty: added', newKey, (`${convertedValue}`).trim())
newProperties[newKey] = (`${convertedValue}`).trim()
}
}
} else {
debugLog('!!!!!! property not found in mappingKeys: ', pointKeyName)
}
}
debugLog('newProperties', newProperties)
return newProperties;
}
private isTruthyValue(someValue: string) {
let convertedValue;
if (this.truthyValues.indexOf(someValue) !== -1) {
convertedValue = true
}
if (this.falsyValues.indexOf(someValue) !== -1) {
convertedValue = false
}
return convertedValue
}
private convertToYesOrNo(originalValue: any) {
debugLog('convertProperty: ==========> original value', originalValue)
let convertedValue = '';
if (this.truthyValues.indexOf(originalValue) !== -1) {
convertedValue = 'yes'
} else {
debugLog('convertProperty: ==========> !!! NOT in truthy values', originalValue)
}
if (this.falsyValues.indexOf(originalValue) !== -1) {
convertedValue = 'no'
} else {
debugLog('convertProperty: ==========> !!! NOT in falsy values', originalValue)
}
return convertedValue;
}
private convertToBoolean(originalValue: any) {
debugLog('convertProperty: ==========> original value', originalValue)
let convertedValue;
if (this.truthyValues.indexOf(originalValue) !== -1) {
convertedValue = true
} else {
debugLog('convertProperty: ==========> !!! NOT in truthy values', originalValue)
}
if (this.falsyValues.indexOf(originalValue) !== -1) {
convertedValue = false
} else {
debugLog('convertProperty: ==========> !!! NOT in falsy values', originalValue)
}
return convertedValue;
}
}

13
mappings/extractors/askangela.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points
# "harassment_prevention=ask_angela"
# pour le monde entier
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:25];nwr["harassment_prevention"="ask_angela"];out+meta;'
export_file="ask_angela_points_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

11
mappings/extractors/cargo_bike_france.sh Normal file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# export depuis OSM des parking à vélo cargo
# cargo_bike=*
# en France
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["cargo_bike"](area.searchArea);out+geom;'
export_file="cargo_bike_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

13
mappings/extractors/centrales_france.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points
# tout système produisant de l'énergie, pas seulement élec.
# pour le monde entier
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;node["power"="generator"](area.searchArea);out+geom;'
export_file="centrales_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

11
mappings/extractors/coworking_france.sh Normal file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# export depuis OSM des points
# des espaces de coworking
# en France
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;node["amenity"="coworking_space"](area.searchArea);out+geom;'
export_file="coworking_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

60
mappings/extractors/geojson_to_csv.py Normal file
View File

@ -0,0 +1,60 @@
import argparse
import json
import pandas as pd
# définir le parseur d'arguments
parser = argparse.ArgumentParser(description="Convertir un fichier GeoJSON en CSV et ajouter des colonnes latitude et longitude")
parser.add_argument("geojson", help="Le chemin du fichier GeoJSON à convertir")
parser.add_argument("-o", "--output", default=None, help="Le nom du fichier de sortie CSV (par défaut: le même nom que le fichier GeoJSON avec l'extension CSV)")
args = parser.parse_args()
# charger le fichier GeoJSON en tant que liste de features
with open(args.geojson, "r") as f:
data = json.load(f)
# afficher le décompte des éléments
print("Nombre d'éléments dans le fichier GeoJSON :", len(data["features"]))
# initialiser un ensemble pour stocker les clés de toutes les features
all_keys = set()
# parcourir chaque feature pour trouver les clés de tag
for feature in data["features"]:
# ajouter les clés de la feature à l'ensemble
all_keys.update(feature["properties"]["tags"].keys())
# convertir l'ensemble en liste et trier les clés
columns = sorted(list(all_keys))
# initialiser le dataframe avec les colonnes définies ci-dessus
df = pd.DataFrame(columns=["id", "latitude", "longitude"] + columns)
# parcourir chaque feature pour ajouter une nouvelle ligne au dataframe avec les valeurs de chaque tag
for feature in data["features"]:
# définir la latitude et la longitude selon la première coordonnée si la géométrie est de type Linestring
if feature["geometry"]["type"] == "LineString":
latitude = feature["geometry"]["coordinates"][0][1]
longitude = feature["geometry"]["coordinates"][0][0]
else:
latitude = feature["geometry"]["coordinates"][1]
longitude = feature["geometry"]["coordinates"][0]
row = {
"id": feature["properties"]["id"],
"latitude": latitude,
"longitude": longitude
}
# ajouter les valeurs de chaque tag à la ligne
for key in columns:
row[key] = feature["properties"]["tags"].get(key, "")
df = df._append(row, ignore_index=True)
# définir le nom du fichier de sortie CSV
if args.output:
output_file = args.output
else:
output_file = os.path.splitext(args.geojson)[0] + ".csv"
print('fichier csv converti : ', output_file)
# convertir le dataframe en CSV en incluant toutes les colonnes
df.to_csv(output_file, sep=";", index=False)

13
mappings/extractors/hackerspace_france.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points
# leisure=hackerspace
# en France
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["leisure"="hackerspace"](area.searchArea);out+geom;'
export_file="hackerspace_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

10
mappings/extractors/hospitals_france.sh Normal file
View File

@ -0,0 +1,10 @@
#!/bin/bash
# export depuis OSM des hopitaux
# en France
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["amenity"="hospital"](area.searchArea);out+geom;'
export_file="hospitals_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

11
mappings/extractors/irve_france.sh Normal file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# export depuis OSM des stations et bornes de recharges
# "amenity"="charging_station"
# en France
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["amenity"="charging_station"](area.searchArea);out+geom;'
export_file="irve_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

13
mappings/extractors/irve_rouen.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points d'apport volontaire
# amenity=recycling
# à Rouen
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3600075628)->.searchArea;nwr["amenity"="charging_station"](area.searchArea);out+geom;'
export_file="irve_zone_rouen_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

10
mappings/extractors/mairies.sh Normal file
View File

@ -0,0 +1,10 @@
#!/bin/bash
# export depuis OSM des mairies
# en France
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["amenity"="townhall"](area.searchArea);out+meta;'
export_file="mairies_points_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

12
mappings/extractors/museums_france.sh Normal file
View File

@ -0,0 +1,12 @@
#!/bin/bash
# export depuis OSM des musées en France
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;node["tourism"="museum"](area.searchArea);out+geom;'
export_file="museum_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

13
mappings/extractors/pav_france.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points d'apport volontaire
# amenity=recycling
# en France
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["amenity"="recycling"](area.searchArea);out+geom;'
export_file="pav_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

13
mappings/extractors/pav_rouen.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points d'apport volontaire
# amenity=recycling
# à Rouen
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3600075628)->.searchArea;nwr["amenity"="recycling"](area.searchArea);out+geom;'
export_file="pav_zone_rouen_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

14
mappings/extractors/planning_familiaux.sh Normal file
View File

@ -0,0 +1,14 @@
#!/bin/bash
# export depuis OSM des points de planning familial du monde entier
# pour le monde entier
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:25];nwr["healthcare:speciality"="family_planning"];out+meta;'
export_file="planning_familial_points_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

12
mappings/extractors/ponts.sh Normal file
View File

@ -0,0 +1,12 @@
#!/bin/bash
# export depuis OSM des ponts qui ont un nom
# pour le monde entier
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["bridge"]["name"](area.searchArea);out+meta;'
export_file="ponts_points_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

12
mappings/extractors/restaurant_france.sh Normal file
View File

@ -0,0 +1,12 @@
#!/bin/bash
# export depuis OSM des restaurants
# "amenity"="restaurant" en France
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["amenity"="restaurant"](area.searchArea);out+geom;'
export_file="restaurant_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

12
mappings/extractors/surveillance_france.sh Normal file
View File

@ -0,0 +1,12 @@
#!/bin/bash
# export depuis OSM des caméras de surveillance
# "man_made"="surveillance"
# en France
url='https://overpass-api.de/api/interpreter?data=%5Bout%3Ajson%5D%5Btimeout%3A300%5D%3B%0Aarea(id%3A3602202162)-%3E.searchArea%3B%0Anode%5B%22man_made%22%3D%22surveillance%22%5D(area.searchArea)%3B%0Aout+geom%3B'
export_file="surveillance_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

13
mappings/extractors/toilets_france.sh Normal file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# export depuis OSM des points
# "amenity"="toilets"
# en France
echo -e "\n récupération de données depuis OpenStreetMap \n"
url='https://overpass-api.de/api/interpreter?data=[out:json][timeout:300];area(id:3602202162)->.searchArea;nwr["amenity"="toilets"](area.searchArea);out+geom;'
export_file="toilets_zone_france_from_openstreetmap"
source $www_folder/mapping-osm-geojson/update_scripts/functions.sh
extract_from_osm $url $export_file

86
mappings/formatters.ts Normal file
View File

@ -0,0 +1,86 @@
import custom_utils from "./utils";
const {debugLog,prefix_phone_fr_only} = custom_utils
/**
* Class that helps to convert values into predefined formats
*/
export default class Formatters {
static convertToPhone(originalValue: string) :string{
/**
* nettoyer les numéros de téléphone en ne gardant que les nombres et le préfixe de pays
*/
debugLog("convertToPhone:" , originalValue);
// debugLog('originalValue', originalValue.substring(1))
if (!originalValue) {
originalValue = ''
}
let original_without_spaces = originalValue.replace(' ', '')
let cleaned_value = `${original_without_spaces}`
cleaned_value = cleaned_value
.trim()
.replace('Stations-e', '')
.replace(/[a-zA-Zéèà]/ig, '')
.replace(/[\(\)\.\- ]/g, '')
let add_prefix = false;
if (
/^\d/.test(cleaned_value) &&
!/^\+33 /.test(original_without_spaces)
) {
add_prefix = true
}
cleaned_value = cleaned_value.replace('+33', '')
debugLog("convertToPhone: cleaned_value" , cleaned_value);
if (/^0/.test(cleaned_value)) {
cleaned_value = cleaned_value.substring(1)
}
let array_of_numbers = cleaned_value
.split('')
let ii = 0;
if (cleaned_value.length == 4) {
ii = 1
}
let convertedValue = ''
array_of_numbers.forEach((num: string) => {
if (ii % 2) {
convertedValue += ' ';
}
convertedValue += num;
ii++;
})
convertedValue = convertedValue.replace(' ', ' ').trim();
debugLog('convertedValue', convertedValue)
if (
/^\d/.test(convertedValue) &&
!/^\+33 /.test(convertedValue)
) {
add_prefix = true
}
if (add_prefix && prefix_phone_fr_only) {
convertedValue = `+33 ` + convertedValue
}
debugLog('phone: ', originalValue, '=>', convertedValue)
return ""+convertedValue;
}
static convertToName(originalValue: string) {
if (originalValue && originalValue.length) {
let tab = originalValue.toLowerCase().split('')
tab[0] = originalValue[0].toUpperCase()
return tab.join('');
}
return ''
}
}

106
mappings/mapping-config.type.ts Normal file
View File

@ -0,0 +1,106 @@
interface GeoJsonGeometry {
type: string,
coordinates: Array<number>,
}
interface GeoJsonProperties {
[key: string]: any,
}
interface GeoJsonFeature {
type: string,
geometry: GeoJsonGeometry,
properties: GeoJsonProperties,
}
export interface FeatureCollection {
type: string,
features: Array<GeoJsonFeature>,
}
export interface BoundingBoxCoordinatesType {
xMin: number,
xMax: number,
yMin: number,
yMax: number,
}
/**
* configuration to choose what point to exclude or include from geographic or properties hints
*/
export interface filteringConfig {
enable_coordinates_filter?: boolean;
enable_properties_filter?: boolean;
properties?: object;
bounding_box?: object;
offset?: number;
filter_points_lesser_than_NkW?: number; // filtrer les points qui ont moins de N kW dans la clé de puissance max
exclude_point_if_tag_not_empty?: Array<string>;
exclude_point_if_tag_truthy?: Array<string>;
exclude_point_if_tag_falsy?: Array<string>;
}
interface sourceConfig {
geojson_path: string; // the relative path to the geojson source file to analyse, from the root of this repository
url: string; // URL from where the geojson comes online, on a data platform. This URL should be fetchable to get the most recent data of the concerned dataset to convert.
overpass_query?: string; // query to get objects from OSM
}
export default interface MappingConfigType {
config_name: string, // descriptive name
config_author: string, // name and email for example
osmose?: boolean, // is the data from Osmose export
boolean_keys?: Array<string>, // what keys should be converted to boolean values
tags_to_ignore_if_value_is?: Array<string>, // list of strings for which we ignore the tags if they equal any of these
add_not_mapped_tags_too: boolean, // by default, we do not add tags from properties that we do not specify, set this to true to change it
default_properties_of_point?: object, // tag to add to every converted point by default
source: sourceConfig,
filters?: filteringConfig,
tags: FeaturePropertyMappingConfigType
}
/**
* configuration concernant toutes les valeurs
*/
export interface FeaturePropertyMappingConfigType {
convert_to_boolean_value?: boolean,
invert_boolean_value?: boolean,
remove_original_key?: boolean,
convert_to_phone?: boolean,
convert_to_name?: boolean,
ignore_if_falsy?: boolean,
ignore_if_truthy?: boolean,
remove_stars?: boolean,
conditional_values?: ConditionnalValuesConfigType,
transform_function?: Function,
[key: string]: any,
}
/**
* choix de conversion de la valeur originale selon des critères donnés
*/
export interface ConditionnalValuesConfigType {
key_converted?: string,
value_converted?: string,
truthy_value?: any,
falsy_value?: any, // si la valeur originale est falsy, la convertir en la valeur donnée ici
ignore_this_data?: boolean,
tags_to_add?: TagsToAddConfig,
transform_function?: Function,
}
export interface ConditionnalValuesType {
[key: string]: ConditionnalValuesConfigType,
}
interface OneOSMTag {
[key: string]: string,
}
export interface TagsToAddConfig {
tags_to_add: Array<OneOSMTag>
}

78
mappings/utils.ts Normal file
View File

@ -0,0 +1,78 @@
import * as fs from 'node:fs'
let show_debug = 0
// show_debug = 1
let output_folder = 'output';
const prefix_phone_fr_only = true
// console.log('----------------------show_debug', show_debug)
/**
* wrapper de log qui se montre uniquemnt si show_debug a été activé
* @param args
*/
function debugLog(...args: any[]) {
if (show_debug) {
console.log('### debug: ',)
args.map((elem: any) => console.log(' ', elem))
}
}
let listOfBooleanKeys = [
"prise_type_ef",
"prise_type_2",
"prise_type_combo_ccs",
"prise_type_chademo",
"gratuit",
"paiement_acte",
"paiement_cb",
"cable_t2_attache"
]
/**
*
* @param pointKeyName
* @returns {boolean}
*/
function isBooleanKey(pointKeyName: string): boolean {
return listOfBooleanKeys.indexOf(pointKeyName) !== -1
}
/**
* crée un fichier dans le dossier par défaut, output
* @param fileName
* @param fileContent
*/
function writeFile(fileName: string, fileContent: any, outputPathOverride: string = '') {
if (outputPathOverride) {
output_folder = outputPathOverride
} else {
console.log('pas de output', outputPathOverride
)
}
let destination = `./${output_folder}/${fileName}`.replace('//', '/');
console.log('write file ', destination)
return fs.writeFile(
destination,
fileContent,
'utf8',
(err) => {
if (err) {
console.log(`Error writing file: ${err}`)
} else {
console.log(`File ${fileName} is written successfully!`)
}
}
)
}
export default {
debugLog,
isBooleanKey,
writeFile,
prefix_phone_fr_only
}

1
mkdocs.yml Normal file
View File

@ -0,0 +1 @@
site_name: Mapping OSM geojson

1076668
output/_irve-latest-etalab.geojson Normal file
View File

File diff suppressed because it is too large Load Diff

82883
output/arbres-echirolles.geojson Normal file
View File

File diff suppressed because it is too large Load Diff

7051
package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

50
package.json Normal file
View File

@ -0,0 +1,50 @@
{
"name": "rangement",
"version": "1.0.0",
"description": "mapping geojson tags to osm tags from json config",
"main": "main.ts",
"author": "tykayn contact+rangement@cipherbliss.com",
"license": "AGPL-3.0-or-later",
"repository": {
"type": "git",
"url": "https://forge.chapril.org/tykayn/scripts.git"
},
"scripts": {
"start": "ts-node convert_to_osm_tags.ts --engine",
"departments": "ts-node build_departments.ts",
"variance": "ts-node make_variance_from_csv.ts --source=etalab_data/irve_bornes_recharge/latest.csv",
"example": "ts-node convert_to_osm_tags.ts --engine=true --department=974 --source=\"./etalab_data/irve_bornes_recharge/all.json\"",
"irve": "ts-node convert_to_osm_tags.ts --engine=true --source=\"./etalab_data/irve_bornes_recharge/all.json\"",
"irve:up": "bash update_scripts/convert_irve.sh",
"irve:jb": "ts-node convert_to_osm_tags.ts --engine=true --source=\"./etalab_data/irve_bornes_recharge/irve-junglebus.csv\"",
"simple": "ts-node convert_to_osm_tags.ts",
"chargemap": "ts-node chargemap.ts",
"filter": "node convert_to_osm_tags.ts --department=974 --engine=true",
"test": "jest --coverage --watch",
"testing": "ts-node convert_to_osm_tags.ts --source=\"./data_other/testing/testing.json\" --testingConfig --engine"
},
"dependencies": {
"@types/geojson2osm": "^0.0.3",
"csv": "^6.3.1",
"geojsontoosm": "^0.0.3",
"minimist": "^1.2.8",
"node-fs": "^0.1.7"
},
"devDependencies": {
"@babel/core": "^7.22.9",
"@babel/preset-env": "^7.22.9",
"@babel/preset-typescript": "^7.22.5",
"@jest/globals": "^29.6.2",
"@types/minimist": "^1.2.2",
"@types/node": "^20.4.7",
"babel-jest": "^29.6.2",
"jest": "^29.6.2",
"loglevel": "^1.8.1",
"nodemon": "^2.0.22",
"ts-jest": "^29.1.1",
"ts-node": "^10.9.1",
"tslib": "^2.6.1",
"typescript": "^5.1.6"
},
"packageManager": "pnpm@8.15.1+sha512.831cf4c5f8b8374af71521d4d153db49d7086de615c2af7cb5e9d7eb8ba630ddac46fea495d643e552ef2e68a3aa99a3e5e9fbee8696702967504df5c59cb273"
}

33
propose_mapping_from_data.py Normal file
View File

@ -0,0 +1,33 @@
import json
import sys
import json
# examiner un jeu de données geojson, et proposer des tags à mettre dans une config
# comme celle ci mappings/converters/configIRVE.ts
#
# exemple de lancement de commande:
# python propose_mapping_from_data.py mon_fichier.geojson
# Vérifie si un argument est fourni
if len(sys.argv) < 2:
print("Usage: python propose_mapping_from_data.py <geojson_file>")
sys.exit(1)
# Ouvre le fichier GeoJSON donné
with open(sys.argv[1]) as f:
data = json.load(f)
# Extraire les propriétés de tous les points de la FeatureCollection
properties = []
for feature in data['features']:
props = feature['properties']
properties.extend(props.keys())
# Créer un dictionnaire avec toutes les clés comme clés et des valeurs par défaut vides
defaults = {prop: '' for prop in properties}
# Remplacer les clés contenant "web" ou "téléphone" par "contact:website" ou "contact:phone"
defaults.update({prop: 'contact:website' if 'web' in prop else 'contact:phone' if 'téléphone' in prop else '' for prop in properties})
# Convertir le dictionnaire en JSON et l'afficher
print(json.dumps(defaults, indent=2))

148
tests/data/mappings_to_test.ts Normal file
View File

@ -0,0 +1,148 @@
import MappingConfigType from "../../mappings/mapping-config.type";
/**
* configurations de mapping pour les cas de tests
*/
export const mappingRemoveAll: MappingConfigType = {
config_name: 'testing config mappingRemoveAll',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingSame: MappingConfigType = {
config_name: 'testing config mappingSame',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
equal: 'equal'
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingName: MappingConfigType = {
config_name: 'testing config mappingName',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
add_not_mapped_tags_too: false,
tags: {
nom_amenageur: 'name'
},
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingPhone: MappingConfigType = {
config_name: 'testing config mappingPhone',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
telephone_operateur: {
key_converted: 'phone',
convert_to_phone: true, // convertit en yes ou no
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingBoolean: MappingConfigType = {
config_name: 'testing config mappingBoolean',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
consolidated_is_lon_lat_correct: {
convert_to_boolean_value: true,
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingTruthy: MappingConfigType = {
config_name: 'testing config mappingTruthy',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
consolidated_is_lon_lat_correct: {
truthy_value: "succès",
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingFalsy: MappingConfigType = {
config_name: 'testing config mappingFalsy',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
consolidated_city: {
falsy_value: "pas ouf succès",
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingIgnoreFalsy: MappingConfigType = {
config_name: 'testing config mappingIgnore',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
consolidated_city: {
ignore_if_falsy: true,
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}
export const mappingIgnoreTruthy: MappingConfigType = {
config_name: 'testing config mappingIgnore',
config_author: 'tykayn <contact@cipherbliss.com>',
default_properties_of_point: {},
tags: {
consolidated_is_lon_lat_correct: {
ignore_if_truthy: true,
},
},
add_not_mapped_tags_too: false,
source: {
geojson_path: "",
url: ""
},
filters: {},
}

24
tests/data/testing.json Normal file
View File

@ -0,0 +1,24 @@
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [
4.822159,
45.635079
]
},
"properties": {
"nom_amenageur": "Bob",
"siren_amenageur": "891624884",
"telephone_operateur": "0123456789",
"equal": "same value",
"consolidated_commune": "S\u00e9r\u00e9zin-du-Rh\u00f4ne",
"consolidated_is_lon_lat_correct": true,
"consolidated_city": false
}
}
]
}

164
tests/main.test.js Normal file
View File

@ -0,0 +1,164 @@
import mapping_engine from '../mappings/engine.ts'
import {
mappingPhone,
mappingRemoveAll,
mappingBoolean,
mappingName,
mappingSame, mappingTruthy, mappingFalsy, mappingIgnoreFalsy, mappingIgnoreTruthy
} from './data/mappings_to_test'
const testingGeoJson = require('./data/testing.json')
// import { describe, expect, test } from '@jest/globals'
describe('mapping properties with rich mapping engine', () => {
// test('do not add properties at all when there is nothing in tags of the mapping config', () => {
//
// })
let feature_to_test = testingGeoJson.features[0]
test('remove all properties when mapping says so', () => {
let Mapping_engine = new mapping_engine(mappingRemoveAll)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point).toBeTruthy()
expect(Object.keys(mapped_point.properties)).toStrictEqual([])
})
test('maps simple key to key, and keep the same value', () => {
let Mapping_engine = new mapping_engine(mappingSame)
let newProperties = Mapping_engine.convertProperty('equal',
Object.keys(mappingSame.tags),
feature_to_test,
mappingSame.default_properties_of_point )
expect(newProperties).toStrictEqual({
equal : "same value"
})
})
test('retrieve config name in mapping engine', () => {
let Mapping_engine = new mapping_engine(mappingRemoveAll)
expect(Mapping_engine.getConfig().config_name).toBe('testing config mappingRemoveAll')
})
test('maps nom_amenageur to name, and keep the same value', () => {
let Mapping_engine = new mapping_engine(mappingName)
let newProperties = Mapping_engine.convertProperty('nom_amenageur',Object.keys(mappingName.tags),feature_to_test,mappingName.default_properties_of_point )
expect(Mapping_engine.getConfig().config_name).toBe('testing config mappingName')
expect(newProperties).toStrictEqual({
name : "Bob"
})
})
test('ignore one value if it is truthy', () => {
let Mapping_engine = new mapping_engine(mappingIgnoreTruthy)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({})
})
test('ignore one value if it is falsy', () => {
let Mapping_engine = new mapping_engine(mappingIgnoreFalsy)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({})
})
// test('conditional value', () => { })
// test('conditional transform', () => { })
test('conditional truthy transform', () => {
let Mapping_engine = new mapping_engine(mappingTruthy)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ consolidated_is_lon_lat_correct: 'succès' })
})
test('conditional falsy transform', () => {
let Mapping_engine = new mapping_engine(mappingFalsy)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ consolidated_city: 'pas ouf succès' })
})
test('conditional boolean transform', () => {
let Mapping_engine = new mapping_engine(mappingBoolean)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ consolidated_is_lon_lat_correct: 'yes' })
})
test('conditional phone transform', () => {
let Mapping_engine = new mapping_engine(mappingPhone)
let mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
let expected_converted_phone = '+33 1 23 45 67 89'
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '3615'
expected_converted_phone = '+33 36 15'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '+33 6 12928883'
expected_converted_phone = '+33 6 12 92 88 83'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '+(33)-(9)-69366018'
expected_converted_phone = '+33 9 69 36 60 18'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '0811 69 06 06'
expected_converted_phone = '+33 8 11 69 06 06'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '+ 33 1 30 31 30 46 (numéro unique)'
expected_converted_phone = '+33 1 30 31 30 46'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '+ 3 3 1 3 0 3 13046 (numéro unique)'
expected_converted_phone = '+33 1 30 31 30 46'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '3 56 80 09'
expected_converted_phone = '+33 3 56 80 09'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '+ 33 1 30 31 30 46 (numéro unique)'
expected_converted_phone = '+33 1 30 31 30 46'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = 'Stations-e'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ })
feature_to_test.properties.telephone_operateur = '+33 0 7 66 38 74 96'
expected_converted_phone = '+33 7 66 38 74 96'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
feature_to_test.properties.telephone_operateur = '+ 33 1 30 31 30 46 (numéro unique)'
expected_converted_phone = '+33 1 30 31 30 46'
mapped_point = Mapping_engine.mapElementFromConf(feature_to_test)
expect(mapped_point.properties).toStrictEqual({ phone: expected_converted_phone })
// +19 0142056650;+19 0664534050
})
})
xdescribe('filters points', () => {
test('filter no points', () => { })
test('filter only one point', () => { })
test('filter X number of points', () => { })
test('filter city points', () => { })
test('filter bounding box', () => { })
test('filter combo, city', () => { })
test('filter combo, city + bbox', () => { })
test('filter combo, city + bbox + offset', () => { })
})
/**
* thats a roadmap, yes
*/
xdescribe('infer domain of values from csv file', () => {
test('gets the list of unique values in column', () => { })
})
xdescribe('infer domain of values from geojson file', () => {
test('gets the list of unique values in column', () => { })
})
xdescribe('build mapping engine config from unique values', () => {
test('builds a valid mapping config', () => { })
})

32
tsconfig.js Normal file
View File

@ -0,0 +1,32 @@
module.exports= {
"compilerOptions": {
"outDir": "build/dist",
"module": "esnext",
"target": "es2020",
"lib": ["esnext", "dom"],
"sourceMap": true,
"allowJs": true,
"jsx": "react",
"moduleResolution": "node",
"rootDir": "src",
"forceConsistentCasingInFileNames": true,
"noImplicitReturns": true,
"noImplicitThis": true,
"noImplicitAny": false, //here
"strictNullChecks": true,
"suppressImplicitAnyIndexErrors": true,
"noUnusedLocals": true,
"resolveJsonModule": true,
"esModuleInterop": true,
"noUnusedParameters": true
},
"exclude": [
"node_modules",
"build",
"scripts",
"acceptance-tests",
"webpack",
"jest",
"src/setupTests.ts"
]
}

15
update_scripts/convert_ask_angela.sh Normal file
View File

@ -0,0 +1,15 @@
#!/bin/bash
# convertir tous les jeux de données présents pour AskAngela
# toulouse
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/toulouse_askangela.json --output-file=toulouse_askangela.json --engine-config=MappingAskAngela
#bordeaux
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/bordeaux_askangela.json --output-file=bordeaux_askangela.json --engine-config=MappingAskAngela
# poitiers
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/poitiers_askangela.json --output-file=poitiers_askangela.json --engine-config=MappingAskAngela
# lyon
ts-node convert_to_osm_tags.ts --source=etalab_data/askangela/lyon_askangela.json --output-file=lyon_askangela.json --engine-config=MappingAskAngela

35
update_scripts/convert_data.sh Executable file
View File

@ -0,0 +1,35 @@
#!/bin/bash
# get the updated geojson
#wget https://www.data.gouv.fr/fr/datasets/r/7eee8f09-5d1b-4f48-a304-5e99e8da1e26 -P ./etalab_data -O ./etalab_data/all.json
#wget https://www.data.gouv.fr/fr/datasets/r/8d9398ae-3037-48b2-be19-412c24561fbb -P ./etalab_data -O ./etalab_data/all.csv
# https://www.data.gouv.fr/fr/datasets/r/8d9398ae-3037-48b2-be19-412c24561fbb pour le jeu de données irve schema v2, non utilisé dans ce script
# filter its tags
echo "refresh de la data"
echo " ce script utilise plusieurs dépendances: nvm, pnpm, geojsontoosm "
echo " c'est parti "
nvm use stable
pnpm i
pnpm start
# convert the new geojson to osm file
geojsontoosm etalab_data/all.json > osm_output/bornes-irve-all-from-etalab-opendata.osm
geojsontoosm output/my_converted_data_set_filtered_zipcode_91.json > osm_output/bornes-irve-filetered-from-etalab-opendata.osm
echo ' conversion faite.'
du -sch osm_output/bornes-irve-filetered-from-etalab-opendata.osm
echo ""
echo " vous pouvez ouvrir JOSM et mettre dedans les fichiers osm pour tenter une conflation
osm_output/bornes-irve-filetered-from-etalab-opendata.osm
osm_output/bornes-présentes-dans-osm.osm
pour mettre en valeur les données manquantes, vous pouvez sélectionner toutes les données existantes et rendre actif le calque de nouvelles données.
Cela fera des points de couleur sur fond de ronds plus gros en gris. Les points de couleur non entourés sur le fond de ronds plus gros sont à importer.
vérifiez ensuite les données du fichier osm en cliquant sur un point du calque bornes-irve-filetered-from-etalab-opendata.osm
"
# done

22
update_scripts/convert_irve.sh Executable file
View File

@ -0,0 +1,22 @@
#!/bin/bash
echo "convertir les fichiers de données depuis etalab_data/irve_bornes_recharge en fichier osm "
overpass_website="https://overpass-api.de/api/interpreter"
# personnaliser les scripts:
# aire Essonne: id 3600007401
# aire IDF: id 3600008649
# aire FR: id 3602202162
output_file_name='irve_osm_latest'
echo " - mapping du fichier IRVE publié par Etalab "
ts-node convert_to_osm_tags.ts --source="etalab_data/irve_bornes_recharge/latest.json" --output-file="_irve-latest-etalab.geojson" --engine-config=mappingConfigIRVE
echo " - mapping des analyses Osmose sur l'item 8411 (charging_station)"
ts-node convert_to_osm_tags.ts --osmose=true --source="etalab_data/irve_bornes_recharge/osmose-item-irve-8411-intégrables.json" --output-file="irve-latest-osmose.geojson" --engine-config=mappingConfigIRVE
echo "fichier généré: $PWD/output/converted__irve-latest-etalab.json"
cd update_scripts

4
update_scripts/convert_planing_familial.sh Normal file
View File

@ -0,0 +1,4 @@
#!/bin/bash
# convertir la liste de planing familial
ts-node convert_to_osm_tags.ts --source=etalab_data/planing_familial/antennes_planning_familial.json --output-file=antennes_planning_familial.geojson --engine-config=MappingPlanningFamlial

5
update_scripts/convert_recycling.sh Normal file
View File

@ -0,0 +1,5 @@
#!/bin/bash
# convertir tous les jeux de données présents pour AskAngela
# rouen
ts-node convert_to_osm_tags.ts --source=etalab_data/recycling/rouen_pav.geojson --output-file=rouen_pav.json --engine-config=MappingRouenPAV

11
update_scripts/convert_surveillance_rouen.sh Executable file
View File

@ -0,0 +1,11 @@
#!/bin/bash
echo "convertir les fichiers de données en fichier geojson pour JOSM "
output_file_name='rouen_surveillance_from_umap.geojson'
mappingType='MappingSurveillanceRouen'
ts-node convert_to_osm_tags.ts --source="etalab_data/surveillance/rouen_surveillance.json" --output-file="$output_file_name" --engine-config="$mappingType"
echo "fichier généré: $PWD/output/$output_file_name"

43
update_scripts/functions.sh Normal file
View File

@ -0,0 +1,43 @@
#!/usr/bin/bash
# fonction d'extraction d'une requête overpass et conversion du fichier de sortie
extract_from_osm(){
overpass_query=$1
file_name=$2
echo -e "extraction de $file_name"
echo -e "requête: \n $overpass_query"
echo -e "\n récupération de données depuis OpenStreetMap \n"
echo " export osm de: $export_file"
# Télécharger le fichier GeoJSON
curl -X GET -o "$export_file.json" -g $url
echo -e "\n Curl OK \n"
date
echo -e "nombre de points extraits depuis overpass turbo :"
jq '.elements | length' "$export_file.json"
# on garde la version OSM brute des données extraites
cp "$export_file.json" "$www_folder/mapping-osm-geojson/osm_output/$export_file.osm.json"
# on convertir en geojson
osm2geojson -q -i 2 --reader json "$export_file.json" "$export_file.geojson" -f
echo -e "\n -> fichier converti: $export_file.geojson "
# on déplace dans le dossier osm_output les versions brutes et converties
mv "$export_file.geojson" "$www_folder/mapping-osm-geojson/osm_output/$export_file.geojson"
echo " -> fichier placé dans $www_folder/mapping-osm-geojson/osm_output/$export_file.geojson "
# on convertit le geojson en csv
python3 "$www_folder/mapping-osm-geojson//mappings/extractors/geojson_to_csv.py" "$www_folder/mapping-osm-geojson/osm_output/$export_file.geojson" -o "$www_folder/mapping-osm-geojson/osm_output/$export_file.csv"
rm "$export_file.json"
echo "nombre de features dans l'export geojson :"
jq '.features | length' "$www_folder/mapping-osm-geojson/osm_output/$export_file.geojson"
}
echo "fonction d'extraction depuis OSM chargée"
export extract_from_osm

56
update_scripts/get_datasets.sh Executable file
View File

@ -0,0 +1,56 @@
#!/bin/bash
# get the updated geojson
# personnaliser les scripts:
# aire Essonne: id 3600007401
# aire IDF: id 3600008649
# aire FR: id 3602202162
##################
## IRVE data
##################
echo "- récupérer les infos de bornes de recharge électrique"
wget https://www.data.gouv.fr/fr/datasets/r/7eee8f09-5d1b-4f48-a304-5e99e8da1e26 -O "latest.json"
wget https://www.data.gouv.fr/fr/datasets/r/8d9398ae-3037-48b2-be19-412c24561fbb -O "latest.csv"
# wget https://www.data.gouv.fr/fr/datasets/r/b9731c6f-c0d7-422e-9e1c-19edd51687ce -O "finess_idf.json"
echo "- OK IRVE"
echo "- récupérer les données présentes dans OpenStreetMap"
curl --header "Content-Type: plain/text" --data @content_irve_geojson.txt --trace-ascii website-data.log "https://overpass-api.de/api/interpreter" > "irve_osm_latest.geojson"
echo "- récupérer les données présentes dans Osmose"
wget "https://osmose.openstreetmap.fr/api/0.3/issues.geojson?full=true&status=open&item=8410&limit=20000" -O "osmose-item-irve-8411-intégrables.json"
echo "- OK Osmose"
##################
# moving datasets to the source folder etalab_data
##################
echo " - déplacement des datasets des IRVE dans le dossier etalab_data/irve_bornes_recharge"
mv latest.json ../etalab_data/irve_bornes_recharge/
#mv finess_idf.json ../etalab_data/finess/
mv irve_osm_latest.geojson ../etalab_data/irve_bornes_recharge/
mv clean_french_irve.csv ../etalab_data/irve_bornes_recharge/
mv osmose-item-irve-8411-intégrables.json ../etalab_data/irve_bornes_recharge/
##################
# other sources of data should be placed in data_other folder
##################
echo "- récupérer les données de cyclabilité d'Issy"
wget "https://data.issy.com/api/explore/v2.1/catalog/datasets/parkings-2-roues/exports/geojson?lang=fr&timezone=Europe%2FBerlin" -O "issy_les_mx_cyclabilité.json"
echo "- récupérer les données des arbres d'Issy"
wget "https://data.issy.com/api/explore/v2.1/catalog/datasets/arbres-remarquables-issy-les-moulineaux/exports/geojson?lang=fr&timezone=Europe%2FBerlin" -O "issy_les_mx_arbres.json"
echo "- récupérer les données de cyclabilité de Rouen"
wget "https://data.metropole-rouen-normandie.fr/api/explore/v2.1/catalog/datasets/liste-des-stationnements-cyclables-metropole-rouen-normandie/exports/geojson?lang=fr&timezone=Europe%2FBerlin" -O "rouen_parking_velos.json"
mv "issy_les_mx_arbres.json" ../data_other/arbres/issy_les_mx_arbres.json
mv "geojson?lang=fr" ../data_other/cyclabilité/issy_les_mx_cyclabilité.json
mv "rouen_parking_velos.json" ../data_other/cyclabilité/rouen_parking_velos.json
# clean logs and finish
rm website-data.log
cd ..
echo "- refresh de la data OK"

88
update_scripts/make_variance_from_geojson.ts Normal file
View File

@ -0,0 +1,88 @@
/**
prendre un CSV,
examiner toutes les colonnes et leurs valeurs,
garder en mémoire les valeurs uniques de chaque colonne
faire un nouveau csv qui ne montre que les valeurs uniques pour chacune des colonnes
et qui compte le nombre de valeurs
**/
import utils from '../mappings/utils'
import {parse} from 'csv'
const fs = require('fs')
const minimist = require('minimist')
interface VarianceType {
[key: string]: Array<string>
}
let csv_content = 'variance de dataset\n';
let separator = ';';
// let columns_headings = [];
let data_variance: VarianceType = {};
const inputPath = '../etalab_data/arbres/arbres_ech.json'
// const inputPath = './etalab_data/small.csv'
let columns_headings: Array<string> = [];
let lines_count = 0;
let longest_variance_count = 0;
console.log('open file ', inputPath)
fs.readFile(inputPath, function (err: any, fileData: any) {
if (err) {
throw new Error(err)
} else {
parse(fileData, {columns: false, trim: true}, function (err: any, rows: any) {
if(err){
throw new Error(err)
}
console.log('line ', lines_count)
console.log('rows', rows)
console.log('longest_variance_count', longest_variance_count)
})
}
console.log('parsing done')
})
/**
* écrit un csv avec les données de variance du dataset donné
*/
function writeCSVVariance() {
let csv_content = ';variance de ' + inputPath + ';' + new Date() + '\n'
let columns = Object.keys(data_variance);
// add headings
columns_headings.forEach((heading: string) => {
csv_content = csv_content + separator + heading
})
csv_content = csv_content + '\n'
// add max length of variance for each column
let ii = 0
columns.forEach((column: string) => {
// console.log('column', column, data_variance[column].length)
csv_content = csv_content + separator + data_variance[column].length
ii++
})
csv_content = csv_content + '\n\n'
// add content of values
for (let ii = 0; ii < longest_variance_count; ii++) {
csv_content = csv_content + '\n'
columns.forEach((column: any) => {
if (ii < data_variance[column].length) {
let currentValue = data_variance[column][ii]
csv_content = csv_content + separator + currentValue
} else {
csv_content = csv_content + separator
}
})
}
return csv_content;
}

1
update_scripts/overpass_to_geojson.py Normal file
View File

@ -0,0 +1 @@
import osm2geojson

19
update_scripts/publish_to_datagouv.sh Normal file
View File

@ -0,0 +1,19 @@
#!/bin/bash
# hackerspaces : https://www.data.gouv.fr/fr/admin/dataset/671f6e8fa0f72ffb985b07b3/
# nécessite la package python pip "httpie"
source $HOME/Nextcloud/ressources/workflow_nextcloud/secrets_vars.sh
echo "mise à jour des hackerspaces"
http -f POST $API/datasets/$DATASET_HACKERSPACE/resources/$RESOURCE_HACKERSPACE/upload/ X-Api-Key:$API_KEY file@$HACKERSPACE_FILEPATH
echo "mise à jour des ask angela"
http -f POST $API/datasets/$DATASET_ASKANGELA/resources/$RESOURCE_ASKANGELA_GEOJSON/upload/ X-Api-Key:$API_KEY file@$ASKANGELA_FILEPATH_GEOJSON
http -f POST $API/datasets/$DATASET_ASKANGELA/resources/$RESOURCE_ASKANGELA_CSV/upload/ X-Api-Key:$API_KEY file@$ASKANGELA_FILEPATH_CSV
echo "mise à jour des musées"
http -f POST "$API/datasets/$DATASET_MUSEUMS/resources/$RESOURCE_MUSEUMS/upload/" X-Api-Key:$API_KEY file@$MUSEUMS_FILEPATH
echo "mise à jour des parking cargo"
http -f POST "$API/datasets/$DATASET_CARGO/resources/$RESOURCE_CARGO/upload/" X-Api-Key:$API_KEY file@$CARGO_FILEPATH

1
update_scripts/queries_overpass/content_irve_csv.txt Normal file
View File

@ -0,0 +1 @@
[out:csv(::id,::uid,::user,::changeset,::lon,::lat,'addr:postcode','addr:city',covered,layer,charging_type,'current:ac','current:dc','charging_station:output',capacity,voltage,amperage,socket,'note:location',access,charging_type,name,owner,network,operator,'operator:wikidata','operator:wikipedia','owner:ref:FR:SIREN',ref,'ref:EU:EVSE','email','contact:email','contact:phone',phone,contact,'socket:typee','socket:schuko','socket:type1','socket:type2','socket:type2:output','socket:type2_cable','socket:type2_combo','socket:type2_combo:output','socket:type3c','socket:type3c:output','socket:chademo','socket:chademo:output',fee,charge,'payment:contactless','parking:fee','payment:app','payment:credit_cards',reservation,authentication,'authentication:nfc','authentication:membership_card','authentication:none',bicycle,scooter,bus,hgv,motorcar,wheelchair,'survey:date',start_date,opening_hours,source,wikidata,description,note,fixme,shop,mapillary,panoramax)][timeout:25];area(id:3602202162)->.searchArea;(node["amenity"="charging_station"](area.searchArea););out meta;>;out meta qt;

1
update_scripts/queries_overpass/content_irve_geojson.txt Normal file
View File

@ -0,0 +1 @@
[out:json][timeout:25];area(id:3600008649)->.searchArea; (node["amenity"="charging_station"](area.searchArea);); out center;

10
update_scripts/queries_overpass/query-overpass.sh Executable file
View File

@ -0,0 +1,10 @@
#!/bin/bash
# Query Overpass API (https://wiki.openstreetmap.org/wiki/Overpass_API)
# Usage:
# ```sh
# echo "[out:json];(way(109849152); node(1985407312);); out;" | ./query-overpass.sh
# ```
curl -d @- -X POST http://overpass-api.de/api/interpreter
#> data_other/output_query_irve_from_overpass.json

13
update_scripts/run_all_converters.sh Executable file
View File

@ -0,0 +1,13 @@
#!/bin/bash
# chemin du dossier à parcourir
dir_to_search="."
# recherche tous les fichiers .sh dans le dossier et ses sous-dossiers
find "$dir_to_search" -type f -name "convert*.sh" -print0 | while IFS= read -r -d '' file; do
echo $file
# exécute chaque fichier .sh trouvé
bash $file
done

Some files were not shown because too many files have changed in this diff Show More