Python Forum
Python Project Structure for Modularity and Reusability with Multiple Entry Points
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python Project Structure for Modularity and Reusability with Multiple Entry Points
#1
I am massively confused about the approach I should take to organizing my project structure (see below) for my machine learning experiments. My projects generally have multiple individual experiments (i.e., multiple entry points) that I need to run (e.g., experiment 1, experiment 2, etc...) and all the experiments need to be able to import custom code I wrote for things like different machine learning models (resent, vgg, etc...) or utilities (metrics, common functions, etc...) to name a couple. I need to be able to run this code locally on my computer, share it with other people so they can run it locally on their computers, and execute it on server environments. So, it needs to be very modular and easy to configure.

My understanding is that I can put a blank init.py file in every directory and sub directory and then call each experiment from the proj_root folder to run each experiment as a module (e.g., python -m experiments.experiment_1.run_exp1).

Is this good/current practice in python to accomplish my goals of code reusability and project modularity? I would prefer to avoid the overhead of using tools like docker. Thank you!

proj_root/
|
|-- models
|   |
|   |-- resnet_model.py
|   |-- vgg_model.py
|
|-- utils
|   |
|   |-- metrics.py
|   |-- common.py
|
|-- experiments
|   |
|   |-- experiment_1
|   |   |
|   |   |-- run_exp1.py
|   |
|   |-- experiment_2
|   |   |
|   |   |-- run_exp2.py
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  python convert multiple files to multiple lists MCL169 6 1,620 Nov-25-2023, 05:31 AM
Last Post: Iqratech
  Project structure with a virtual environment gdbengo 1 1,472 Jan-26-2022, 03:22 PM
Last Post: snippsat
  How to define a variable in Python that points to or is a reference to a list member JeffDelmas 4 2,695 Feb-28-2021, 10:38 PM
Last Post: JeffDelmas
  Determine number of all the immediately adjacent points in python zackk 1 1,893 Feb-06-2021, 09:23 AM
Last Post: zackk
  How can I scroll over my data points when creating plots in Python? (I'm using Spyder moose 0 1,617 Nov-02-2020, 07:18 AM
Last Post: moose
  "Switch-to-spreadsheet" entry. Feasible in Python? whatspython 2 2,038 Sep-30-2020, 01:12 PM
Last Post: buran
  Calling main in project structure fails sandro4912 0 1,356 Aug-22-2020, 01:51 PM
Last Post: sandro4912
  data from multiple Entry widgets beevee 13 4,562 Aug-16-2020, 12:31 PM
Last Post: deanhystad
  python project structure anne 1 1,768 Aug-01-2020, 07:17 PM
Last Post: deanhystad
  how to specify options or tickle python to output points swept by the optimization? bsmile 3 2,707 Aug-02-2019, 10:31 PM
Last Post: scidam

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020