[RPC] Propose restructuring the repository folder structure

[wsmelton]

In conjunction with devcontainers being added in PR #6997, it is worth bringing up now that in order to get an automated build and release process set up we need to restructure the folders of this repository.

In most cases, an src folder contains the core base of the module. That would allow us to more easily build out the GitHub Action for doing a build script for releases and publishing to GitHub.

This would be moving the following folders/files into the src folder:

• bin
• functions
• internal
• optional
• xml
• dbatools.psd1
• dbatools.psm1

The new root folder of the module would be something like this:

..\DBATOOLS
|   .gitattributes
|   .gitignore
|   .markdownlint.json
|   allcommands.ps1
|   appveyor.yml
|   changelog.md
|   cleanup.ps1
|   codecov.yml
|   CODE_OF_CONDUCT.md
|   contributing.md
|   install.ps1
|   LICENSE
|   readme.md
+---.github
+---.vs
+---.vscode
+---src
|   |   dbatools.psd1
|   |   dbatools.psm1
|   +---bin
|   +---en-us
|   +---functions
|   +---internal
|   +---optional
|   \---xml
\---tests

[nvarscar]

I'm onboard with that idea - to separate module files from git/appveyor/younameit files.

If we're moving code to src, there is also a dbatools C# project hidden inside the /bin/projects/ folder. It would make sense to move that project to the /src/ folder, as that's where it would be expected to live. But it would need one more folder level to separate those.

+---src
|   +--- dbatools
|        |   dbatools.psd1
|        |   dbatools.psm1
|        |   ...
|   +--- dbatools-library
|        |   dbatools.sln
|        |   ...
...

[wsmelton]

Initially, we intentionally put the C# project in the current location because we offer features around rebuilding our library on import. Checking notes on the publishing process I see now that we no longer are publishing it to the PS Gallery.

If @potatoqualitee wants to continue excluding it there is a good bit of code that we can clean up and simply remove the functionality.

Reference/background:

It was originally found users wanted to include the raw project for security scanning our code for vulnerabilities. Where places also had security reviews required before "software" could be installed they needed easy access to the raw code. So when Fred had the project setup and ready a configuration item was added for the import process to allow a building that library on the fly to replace the dbatools.dll we publish with each release.

https://github.com/sqlcollaborative/dbatools/blob/cd7a897cf042666e3bdbd18b6fdefb77dd4991fd/bin/library.ps1#L74-L90

The build-project.ps was included for this purpose as it also a configurable item we have in the configuration area of code to allow them to always rebuild it (for security purposes):

https://github.com/sqlcollaborative/dbatools/blob/cd7a897cf042666e3bdbd18b6fdefb77dd4991fd/en-us/about_dbatools_importoptions.help.txt#L122-L127

Future

If we are going to no longer publish the raw project with the module. I'd recommend actually moving that to a new repository. Then we can include an automation process to create formal releases of the library and pull those into the module repository as needed. We rarely even make changes to that library nowadays.

[potatoqualitee]

While we are at it, if we can do public/private instead of functions/internal, i'd dig. i do want to make 100000% sure everyone keeps their commit messages and attribution though

[wsmelton]

i do want to make 100000% sure everyone keeps their commit messages and attribution though

Commit history is kept intact by the git database file. GitHub may or may not always read that information in when you are viewing the history of the file, but it is there.

[wsmelton]

@Stuart-Moore @FriedrichWeinmann @nvarscar

Any opinion on moving the project for the library to a dedicated repository?

We can get a formal build process setup and have a GitHub Action run it. Would also allow us to publish it to the Nuget gallery if we wanted to, then just have the module pull it down in a build process or something.

[nvarscar]

I'm fine with that. I don't think there is a good reason to ship the C# source code with the module, as there are plenty 3rd party libs that are shipped with the module already. One more wouldn't make a difference. Having a base lib moved to its own repo - with a build pipeline and unit teats (if any) sounds like a good plan.

[wsmelton]

There is so much breaking just from moving the source code of the module 🦀 . I had no idea it was this bad.

[wsmelton]

@lancasteradam FYI this is in the works. I'm going to try and get the PR merged this coming week.

Once that is done all forks of the repository will have to be updated. The cleanest way to do this:

Assuming upstream is pointed to sqlcollaborative/dbatools

git checkout development
git fetch upstream
git reset --hard upstream/development
git push origin development --force

This does a force reset instead of a merge, so the commit history stays cleaner.

[lancasteradam]

@wsmelton, ok, sounds good, thanks.

[wsmelton]

@potatoqualitee so renaming folders for public/private breaks our configuration structure in the module. I'm also seeing the TEPP breaks even after I've updated folder path references. I'm not sure if @FriedrichWeinmann can look at what I have in that or could say why this would break?

I get various messages about a function we have set as public, but lives in the private functions path (Join-DbaPath):

Join-DbaPath : The 'Join-DbaPath' command was found in the module 'dbatools', but the module could not be loaded. For more information, run 'Import-Module dbatools'.
At C:\git\dbatools\src\private\configurations\configuration.ps1:90 char:35
+     $script:path_FileUserShared = Join-DbaPath $([System.Environment] ...
+                                   ~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Join-DbaPath:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CouldNotAutoloadMatchingModule

A number of these as well; likely because the configurations failed to be brought into the session:

Register-DbatoolsConfigValidation : The term 'Register-DbatoolsConfigValidation' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ Register-DbatoolsConfigValidation -Name "consolecolor" -ScriptBlock {
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Register-DbatoolsConfigValidation:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Set-DbatoolsConfig : Input string was not in a correct format.
At line:6 char:1
+ Set-DbatoolsConfig -FullName 'assets.sqlbuildreference' -Value 'https ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Set-DbatoolsConfig], FormatException
    + FullyQualifiedErrorId : System.FormatException,Sqlcollaborative.Dbatools.Commands.SetDbatoolsConfigCommand

[FriedrichWeinmann]

Heya @wsmelton ,
it's approaching 2am here so I can't really make the time to take a look right now.
Guessing what's happening here based on memory (been some time and lots of other PowerShell in my head in the meantime, sorry), but those function-files were explicitly loaded as individual files for tasks that were offloaded into background runspaces.

So, if you rename the folders, it won't find them anymore and if you stop doing it in background runspaces, you get to see the errors.

[wsmelton]

I updated the paths where all files are referencing \internal... or \functions... so at a loss where it could be hidden.

[wsmelton]

I did notice the whole check for .git method with import by files is no longer valid with the new folder structure. The next plan for our build process will address this and make a moot point I believe (if I can get the approval on the next plan 🙊 ) .

[wsmelton]

Got it all figured out.

[potatoqualitee]

I do remove the source code before shipping! 🚢 I like the idea, but likely won't do it myself.

[wsmelton]

If anyone is interested in the result: https://github.com/wsmelton/dbatools/tree/folder-structure-1